Advanced Computing and Systems for Security [12]

Citation preview

Advances in Intelligent Systems and Computing 1136

Rituparna Chaki Agostino Cortesi Khalid Saeed Nabendu Chaki   Editors

Advanced Computing and Systems for Security Volume Twelve

Advances in Intelligent Systems and Computing Volume 1136

Series Editor Janusz Kacprzyk, Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland Advisory Editors Nikhil R. Pal, Indian Statistical Institute, Kolkata, India Rafael Bello Perez, Faculty of Mathematics, Physics and Computing, Universidad Central de Las Villas, Santa Clara, Cuba Emilio S. Corchado, University of Salamanca, Salamanca, Spain Hani Hagras, School of Computer Science and Electronic Engineering, University of Essex, Colchester, UK László T. Kóczy, Department of Automation, Széchenyi István University, Gyor, Hungary Vladik Kreinovich, Department of Computer Science, University of Texas at El Paso, El Paso, TX, USA Chin-Teng Lin, Department of Electrical Engineering, National Chiao Tung University, Hsinchu, Taiwan Jie Lu, Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, NSW, Australia Patricia Melin, Graduate Program of Computer Science, Tijuana Institute of Technology, Tijuana, Mexico Nadia Nedjah, Department of Electronics Engineering, University of Rio de Janeiro, Rio de Janeiro, Brazil Ngoc Thanh Nguyen , Faculty of Computer Science and Management, Wrocław University of Technology, Wrocław, Poland Jun Wang, Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Shatin, Hong Kong

The series “Advances in Intelligent Systems and Computing” contains publications on theory, applications, and design methods of Intelligent Systems and Intelligent Computing. Virtually all disciplines such as engineering, natural sciences, computer and information science, ICT, economics, business, e-commerce, environment, healthcare, life science are covered. The list of topics spans all the areas of modern intelligent systems and computing such as: computational intelligence, soft computing including neural networks, fuzzy systems, evolutionary computing and the fusion of these paradigms, social intelligence, ambient intelligence, computational neuroscience, artificial life, virtual worlds and society, cognitive science and systems, Perception and Vision, DNA and immune based systems, self-organizing and adaptive systems, e-Learning and teaching, human-centered and human-centric computing, recommender systems, intelligent control, robotics and mechatronics including human-machine teaming, knowledge-based paradigms, learning paradigms, machine ethics, intelligent data analysis, knowledge management, intelligent agents, intelligent decision making and support, intelligent network security, trust management, interactive entertainment, Web intelligence and multimedia. The publications within “Advances in Intelligent Systems and Computing” are primarily proceedings of important conferences, symposia and congresses. They cover significant recent developments in the field, both of a foundational and applicable character. An important characteristic feature of the series is the short publication time and world-wide distribution. This permits a rapid and broad dissemination of research results. ** Indexing: The books of this series are submitted to ISI Proceedings, EI-Compendex, DBLP, SCOPUS, Google Scholar and Springerlink **

More information about this series at http://www.springer.com/series/11156

Rituparna Chaki Agostino Cortesi Khalid Saeed Nabendu Chaki •

•

•

Editors

Advanced Computing and Systems for Security Volume Twelve

123

Editors Rituparna Chaki A.K. Choudhury School of Information Technology University of Calcutta Kolkata, West Bengal, India Khalid Saeed Faculty of Computer Science Bialystok University of Technology Bialystok, Poland

Agostino Cortesi DAIS Ca' Foscari University Venezia, Italy Nabendu Chaki Department of Computer Science and Engineering University of Calcutta Kolkata, West Bengal, India

ISSN 2194-5357 ISSN 2194-5365 (electronic) Advances in Intelligent Systems and Computing ISBN 978-981-15-2929-0 ISBN 978-981-15-2930-6 (eBook) https://doi.org/10.1007/978-981-15-2930-6 © Springer Nature Singapore Pte Ltd. 2020 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd. The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore

Preface

The seventh International Doctoral Symposium on Applied Computation and Security Systems (ACSS 2020) is held in Kolkata, India, during 28–29 February 2020. The University of Calcutta in collaboration with Ca' Foscari University of Venice, Italy, and Bialystok University of Technology, Poland, organized the symposium. This unique symposium is aimed specially to facilitate budding researchers in their pursuit of Ph.D., by providing a platform for the exchange of ideas. Over the years, we have been continuously updating the list of significant research areas so as to include the most significant research domains in the scope of the symposium each year. This helps ACSS to stay in tune with the evolving research trends. The seventh year of the symposium showed a significant improvement in the overall quality of papers, besides some very interesting papers in the domain of security and image analysis. We are grateful to the Programme Committee members for sharing their expertise and taking time off from their busy schedule to complete the review of the papers with utmost sincerity. The reviewers have pointed out the points of improvement for each paper they reviewed, and we believe that these suggestions would go a long way in improving the overall quality of research among the scholars. We have invited eminent researchers from academia and industry to chair the sessions which matched their research interests. As in previous years, the session chairs for each session had a prior go-through of each paper to be presented during the respective sessions. This results in interesting discussions during the sessions whereby the young researchers gather insightful advice for the improvement of their research work. We have invited researchers working in the domains of computer vision and applications, biometrics-based authentication, security for Internet of things, analysis and verification techniques, security in heterogeneous networks, large-scale networking, remote healthcare distributed systems, signal processing, routing and security in WSN intelligent transportation system, human–computer interaction, bioinformatics and system biology computer forensics, privacy and confidentiality access control, big data and cloud computing data analytics, VLSI design and embedded systems requirements, engineering software security algorithms, natural v

vi

Preface

language processing and quantum computing to submit their ongoing research works. The indexing initiatives from Springer have drawn high-quality submissions from scholars in India and abroad. ACSS continues with the tradition of the double-blind review process by the PC members and by external reviewers. The reviewers mainly considered the technical aspect and novelty of each paper, besides the validation of each work. This being a doctoral symposium, clarity of presentation was also given importance. The entire process of paper submission, review and acceptance process was done electronically. We thank the members of Programme Committee and Organizing Committee, whose sincere efforts before and during the symposium have resulted in a suite of strong technical paper presentations followed by effective discussions and suggestions for the improvement for each researcher. The Technical Programme Committee for the symposium selected only 13 papers for publication out of 32 submissions. We would like to take this opportunity to thank all the members of the Programme Committee and the external reviewers for their excellent and time-bound review works. We thank Springer for sponsoring the best paper award. We would also like to thank ACM for the continuous support towards the success of the symposium. We appreciate the initiative and support from Mr. Aninda Bose and his colleagues in Springer Nature for their strong support towards publishing this post-symposium book in the series “Advances in Intelligent Systems and Computing”. Last but not least, we thank all the authors without whom the symposium would not have reached up to this standard. On behalf of the editorial team of ACSS 2020, we sincerely hope that ACSS 2020 and the works discussed in the symposium will be beneficial to all its readers and motivate them towards even better works. Kolkata, India Venezia, Italy Bialystok, Poland Kolkata, India

Rituparna Chaki Agostino Cortesi Khalid Saeed Nabendu Chaki

Contents

Automatic Caption Generation of Retinal Diseases with Self-trained RNN Merge Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sanjukta Mishra and Minakshi Banerjee

1

Dimensionality Reduction of Hyperspectral Images: A Data-Driven Approach for Band Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Arati Paul, Putul Sahoo and Nabendu Chaki

11

COMMENT-MINE—A Semantic Search Approach to Program Comprehension from Code Comments . . . . . . . . . . . . . . . . . . . . . . . . . . Srijoni Majumdar, Shakti Papdeja, Partha Pratim Das and Soumya Kanti Ghosh

29

Biomarker Gene Identification Using a Quantum Inspired Clustering Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Srirupa Dasgupta, Arpita Das, Abhinandan Khan, Rajat Kumar Pal and Goutam Saha Shot Classification and Replay Detection in Broadcast Soccer Video . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Saikat Sarkar, Sazid Ali and Amlan Chakrabarti A Novel Automated Blood Cell Counting Method Based on Deconvolution and Convolution and Its Application to Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Joydeep Chatterjee, Semanti Chakraborty and Kanik Palodhi

43

57

67

A Citizen-Centred Sentiment Analysis Towards India’s Critically Endangered Avian and Mammalian Species . . . . . . . . . . . . . . . . . . . . . Inderjeet Singh Bamrah and Akshay Girdhar

79

Methodology for Generating Synthetic Time-Dependant Probabilistic Speed Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Lukáš Rapant, Daniela Szturcová, Martin Golasowski and David Vojtek

95

vii

viii

Contents

A Multi-Scale Patch-Based Deep Learning System for Polyp Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 Debapriya Banik, Debotosh Bhattacharjee and Mita Nasipuri Object Detection in Rainy Condition from Video Using YOLO Based Deep Learning Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 Ratnadeep Dey, Debotosh Bhattacharjee and Mita Nasipuri Visualization of the Evacuation Process Including People with Heart Diseases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Łukasz Szymkowski, Khalid Saeed and Piotr Szymkowski

About the Editors

Rituparna Chaki is a Professor of Information Technology at the University of Calcutta, India. She received her Ph.D. from Jadavpur University, India, in 2003 and has been with the University of Calcutta since 2013. Her research interests include optical networks, sensor networks, mobile ad hoc networks, the Internet of Things, data mining, etc. Agostino Cortesi Ph.D., is a Full Professor of Computer Science at Ca' Foscari University, Venice, Italy. He has previously served as Dean of Computer Science studies, as Department Chair, and as Vice-Rector for quality assessment and institutional affairs. His research interests include programming languages theory, software engineering, and static analysis techniques, with particular emphasis on security applications. Khalid Saeed is a Full Professor at the Faculty of Computer Science, Bialystok University of Technology, Poland. His research interests include biometrics, image analysis and processing, and computer information systems. Nabendu Chaki is a Professor at the Department of Computer Science & Engineering, University of Calcutta, India. Dr. Chaki completed his initial Physics studies at the legendary Presidency College in Kolkata, followed by Computer Science & Engineering studies at the University of Calcutta. He received his Ph.D. from Jadavpur University, India, in 2000. He holds six international patents, including four U.S. patents with his students. Prof. Chaki has been highly active in developing international standards for software engineering and cloud computing as a member of the Global Directory (GD) for ISO-IEC.

ix

Automatic Caption Generation of Retinal Diseases with Self-trained RNN Merge Model Sanjukta Mishra and Minakshi Banerjee

Abstract Retinal diseases may lead to blindness. So, early detection of the diseases may prevent the scenario. Automatic caption generation from the retinal images is the challenging method to diagnose the diseases. In the recent era, deep learning is very popular in analyzing the medical images effectively. In this paper, we propose an automatic caption generation method based on CNN and self-trained bidirectional LSTM model. We build the model based on merge architecture. The model is trained with normal and abnormal retinal images and their labeled diseases. The target is to generate the caption of true disease from a given test image. We use the STARE database to build the model. After the experiment, it can be said that our attempts are promising and experimental result shows that the model performs well in generating the caption from the new given retinal image. We also evaluate the score using BLEU metrics. Keywords Retinal diseases · Caption generation · Deep learning

1 Introduction The retinal diseases like diabetic retinopathy, choroidal neovascularization, glaucoma, drusen, artery occlusion, histoplasmosis, and macular degeneration may affect the eye if not diagnosed at an early stage. So automatic diagnosis is required to avoid the blindness. Recently, the number of patients and their diseases is increasing day by day. So, storing, retrieving, and analyzing the medical images are a challenging task. As the clinical database becomes large now a day, we need a model which

S. Mishra (B) Brainware University, Kolkata, India M. Banerjee RCC Institute of Information Technology, Kolkata, India © Springer Nature Singapore Pte Ltd. 2020 R. Chaki et al. (eds.), Advanced Computing and Systems for Security, Advances in Intelligent Systems and Computing 1136, https://doi.org/10.1007/978-981-15-2930-6_1

1

2

S. Mishra and M. Banerjee

extracts the true features from an image and also predicts one caption relevant to the image. Deep learning helps in diagnosis of the diseases effectively. Manual detection of the contents of images is difficult, so automatic caption generation from an image takes a very promising place in deep learning. The goal is to create a model which understands the contents of the input images and produces the caption or the name of the diseases as output. This technique gives proper information in less time which is necessary for early diagnosis. Several research works prove that deep learning increases the performances in images analysis and caption generation. Krizhevsky et al. [1] proposed a deep convolutional neural network to classify millions of images of high resolution in the ImageNet LSVRC-2010 contest. Kulkarni et al. [2] designed a system to generate automatic languages from images. It gives a good result. Simonyan et al. [3] experimented the performance of CNN for largescale database implementation. It has been noticed that pushing the depth to 16– 19 weight layers improved the performance. Donahue et al. [4] successfully build a model which recognized the visual descriptions. Jia et al. [5] proposed a work to extend the LSTM by adding semantic content. It successfully achieves a good result. Jin et al. [6] achieved good result by proposing an image captioning model which is based on parallel architecture between image and words. Chen et al. [7] successfully detected glaucoma using convolution neural network. Murthy et al. [8] discovered a good result by using convolution neural network method with word embedding vector to automatic image annotations. It has been noticed that CNN features win compared to many other feature extracted methods. Jozefowicz et al. [9] provided a brief description of several recurrent neural network architectures which are very useful for any further research work. You et al. [10] proposed an algorithm of image captioning by semantic attention which gives a better result than without attention-based approach. Zhou et al. [11] proposed a successful recurrent fusion procedure for image caption generation. Gulshan et al. [12] proposed a high sensible algorithm based on deep machine learning to evaluate diabetic retinopathy from retinal fundus images. Roberts et al. [13] proposed a successful method to detect the diseases from X-ray images and reports with cascading approach. Lyndon et al. [14] overcome the drawbacks of manual annotation-based model by proposing a CNN-RNN pairing-based captioning model. Jing et al. [15] build a promising method for caption generation of medical images. It is based on co-attention and hierarchical LSTM method. Allaouzi et al. [16] focused to give a successful overview of different method and evaluation metrics to perform medical image captioning. The paper produces lots of useful information related to medical image caption generation. Cao et al. [17] provided a Bag-LSTM method for medical image captioning. It is very powerful in semantic detection. In this paper, we use CNN with bidirectional LSTM and self-trained word embedding to automatic detection of the retinal diseases. Model is based on merge architecture.

Automatic Caption Generation of Retinal Diseases …

3

2 Methodology 2.1 Model In this paper, the model based on merge architecture [18] has been adopted where we have used a transfer learning by using Inception v3 model for image feature extraction through convolutional neural network and own dictionary to train the LSTM. In this model, the image has been fed to the CNN to extract the feature vectors. The word has been processed with LSTM using word embedding model. No pretrained word embedding model has been used here. The two input vectors are merged. In this model, RNN handles the linguistic information and CNN encodes the images. The model merge separately encoded image feature and description and then decoded to generate the next word in the sequence. The primary role of RNN is the encoder of word sequences. While training, the captions are divided into multiple tokens or prefixes where each prefix is converted to a fixed-size vector. LSTM is used to remember the previously generated words. The encoder generates word vector which combines with image feature vector in separate layer at the later stage. The below figure shows the model architecture (Fig. 1). Here, the prepared image and caption data have been loaded first. We trained the model with the images and captions belonging to the training set. The model will generate each word at a time not a whole sentence. So, the input is the past generated sequence of words. To start the process, we need a starting word, and to end the caption, we need an ending word. Whenever the descriptions are loaded, these tokens are added. At first, we map the words to unique integer values for encoding the data. Input sequences are fed into a word embedding layer followed by LSTM, and image features have been fed to other parts of the model. Each caption will split into words. Now as per the model, multiple iterations take place to give a probable next predicted word. The probability should be maximum. The sample iterations are shown in below table (Table 1). The input of the model is image vector, and one word generates next word as an output. Next, the two words with image vector have been given as input to get the next output, and so on. The model will train in this way. The model has two input arrays which are nothing but encoded text and the image features. Database and Image Preprocessing. In this paper, we use the STARE database. The database contains color fundus images which include several abnormal images like diabetic retinopathy, choroidal neovascularization, drusen, age-related macular degeneration, histoplasmosis, nevus, central retinal vein occlusion, arteriosclerotic retinopathy, epiretinal membrane, hypertensive retinopathy, retinitis, stellate maculopathy, branch retinal vein occlusion, branch retinal artery occlusion, and some normal images. The images are marked by the experts. We prepare a dictionary containing name of each image and creating five captions of same meaning for each image. An image can have several captions, so we make different relevant captions for an image. Table 2 shows the caption example. Now, for image caption generation,

4

S. Mishra and M. Banerjee

Caption Input

Partial caption sequence model

Self-trained Word Embedding Layer

Sequence of words

LSTM Layer (Bidirectional)

Decoder (Feed forward model) Dense Layer

Image feature vector (CNN)

Dense Layer

Softmax

Predicted word (Probability distribution)

Image Input

Image feature extractor model

Fig. 1 Model architecture

we need to take images as input. Here, we use the transfer learning method by using a pretrained model Inception v3. So, we reshape the image to three-channel 299 × 299 pixel image for the model. Bidirectional LSTM. The LSTM is a special type of RNN architecture. It is used whenever remembering or memory takes a part in the model. In the normal traditional RNN, the drawback is the vanishing gradient problem. Hence, LSTM overcomes the

Automatic Caption Generation of Retinal Diseases …

5

Table 1 Iteratively generation of caption “choroidal neovascularization disease” ITERATION 1

Input2

“Start”

Input1

Image

RNN

Feed forward+ softmax

“Choroidal”

vector Expected output

ITERATION 2 Input2

“Start

RNN

Choroidal” “NeovascularInput1

Feed forward+

Image

ization”

softmax

vector

Expected output

ITERATION 3 Input2

“Start Choroidal Neovascular-

RNN

ization”

Feed forward+ Input1

Image

“disease”

softmax

vector

Expected output

ITERATION 4

Input2

“Start Choroidal Neovascular-

RNN

ization disease”

Feed forward+ Input1

Image

“end”

softmax

vector

Expected output

problem. It contains a memory cell which remembers the value, input, output, and forget gate. The three gates controlled incoming and outgoing of the cell. Here, we use bidirectional LSTM which takes data from past as well as from future. The input sequences are processed in forward and backward directions using two sub-layers. BLSTM is stated as follows [19]:   ht = H Wx h xt + Whh ht−1 + bh

(1)

6

S. Mishra and M. Banerjee

Table 2 Caption example Abnormal retinal image

i. Choroidal neovascularization and arteriosclerotic retinopathy diseases found ii. Choroidal neovascularization and arteriosclerotic retinopathy together in a retinal image iii. The retinal image detects both choroidal neovascularization and arteriosclerotic retinopathy iv. This fundus image detects choroidal neovascularization with arteriosclerotic retinopathy v. Choroidal neovascularization fundus image with arteriosclerotic retinopathy

 ht = H W

←



←

← x t + W←← h t+1 + b←

xh

hh

h

←

yt = Wh y ht + W← h t + b y hy

←

(2) (3)

where ht is the forward hidden sequence and h t is the backward hidden sequence, bh is bias vector of hidden layer, by is the bias vector of output layer, the weight matrix between input and hidden layer is denoted as W xh , the recurrent weight matrix is denoted as W hh , the weight matrix between the hidden and output layer is W hy and H is the hidden layer activation function. The below figure (Fig. 2) shows the architecture of BLSTM. Fig. 2 BLSTM architecture

Automatic Caption Generation of Retinal Diseases …

7

3 Experiment 3.1 Result and Discussion We have used the STARE database for our training and testing the model. We have trained the model with the training database which consists of images and the captions, then find the result on the test images. The result will be a predictive output, i.e., the probability distribution function over the words in the caption dictionary. So, the result is based on maximum likelihood estimation. The meaning of this is that selection of word is the most likely word which is present for the specified input according to the model. This method is called the greedy method. The greedy selection of word takes place on maximum probability. Besides the greedy search, we also cross-check by applying the beam search algorithm. As a result, we find that some of the test images generate a caption with no error which is essential for perfect disease detection; some of them give a minor error and few give the error-prone captions. But in total the result is good. The above mentioned table (Table 3) shows the experimented results of some of the selected retinal images. Evaluation Metrics. Many evaluation metrics are there to evaluate the caption generation. Some of them are BLEU, Meteor, Rough, CIDer, etc. BLEU is popular among them. It is used to evaluate generative sentence to a true sentence. If the matches of the sentences are exact, then the score is 1.0, otherwise 0. In this model, we are using BLEU metric to evaluate the model. The reference and predicted captions are taken and evaluated using corpus BLEU score to test how much the generated text is similar to the actual text. We calculate BLEU scores for 1, 2, 3, and 4 cumulative ngrams. But we are focusing on the true contents not exact sentence. We are extracting the true disease name as the output which is our main goal, not the perfect sentence. BLEU score will be always between 0 and 1. BLEU score of our model has shown in Table 4.

4 Conclusion The paper produces an automatic retinal image captioning model based on CNNBLSTM merge model. We apply merge model because the performance of the inject model is poor compared to the merge model []. The inject model is more complex in the case of encoding in RNN. The size of vocabulary also increases in the inject model because each image and each word together form a single word. Here, we describe how we trained our model and what architecture we adopt. We also explain that we do not apply the pretrained model because pretrained word embedding models are trained by billions of words, but they are of maximum general words, not the medical terms. Next, we take the test images, apply the model, and then evaluate the results from the test images. Overall, the model gives a good result by finding accurate

8

S. Mishra and M. Banerjee

Table 3 Selective experimental result Original caption

Original disease

Generated caption

Result

Cilioretinal artery occlusion and central retinal artery occlusion

Cilioretinal artery occlusion and central retinal artery occlusion in retinal images

Caption with no error

Background diabetic retinopathy

Fundus image with background diabetic retinopathy

Caption with no error

Retinal drusen

Retinal image with drusen

Caption with no error

Cilioretinal artery occlusion and central retinal artery occlusion

Artery occlusion or central retinal artery occlusion

Caption with minor error

Choroidal neovascularization and age-related macular degeneration

Retinal drusen drusen drusen

Caption with total error

diseases of many of the abnormal retinal images. But we need to improve the result further. We only take the STARE database for testing and training the model. Further, we can augment the images and create a large database for training the model. We can also add the attention module to improve the quality of the result. So, we apply

Automatic Caption Generation of Retinal Diseases … Table 4 BLEU score

9

BLEU

Scores

BLEU-1

0.872347

BLEU-2

0.659928

BLEU-3

0.519264

BLEU-4

0.442572

the merge architecture for retinal disease caption generation that we will improve in the future for a better result.

References 1. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012) 2. Kulkarni, G., Premraj, V., Ordonez, V., Dhar, S., Li, S., Choi, Y., Berg, A.C., Berg, T.L.: Babytalk: understanding and generating simple image descriptions. IEEE Trans. Pattern Anal. Mach. Intell. 35(12), 2891–2903 (2013) 3. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: arXiv preprint, arXiv:1409.1556, pp. 1–14, Apr 2015 4. Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., Darrell, T.: Long-term recurrent convolutional networks for visual recognition and description. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2625–2634 (2015) 5. Jia, X., Gavves, E., Fernando, B., Tuytelaars, T.: Guiding the long-short term memory model for image caption generation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2407–2415 (2015) 6. Jin, J., Fu, K., Cui, R., Sha, F., Zhang, C.: Aligning where to see and what to tell: image caption with region-based attention and scene factorization. arXiv preprint, arXiv:1506.06272 (2015) 7. Chen, X., Xu, Y., Wong, D.W.K., Wong, T.Y., Liu, J.: Glaucoma detection based on deep convolutional neural network, engineering in medicine and biology society. In: IEEE, pp. 715– 718 (2015) 8. Murthy, V.N., Maji, S., Manmatha, R.: Automatic image annotation using deep learning representations. In: Proceedings of ACM ICMR (2015) 9. Jozefowicz, R., Zaremba, W., Sutskever, I.: An empirical exploration of recurrent network architectures. In: Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pp. 2342–2350 (2015) 10. You, Q., Jin, H., Wang, Z., Fang, C., Luo, J.: Image captioning with semantic attention. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4651– 4659 (2016) 11. Zhou, L., Xu, C., Koch, P., Corso, J.J.: Image caption generation with text-conditional semantic attention. arXiv preprint, arXiv:1606.04621 (2016) 12. Gulshan, V., Peng, L., Coram, M., Stumpe, M.C., Wu, D., et al.: Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316(22), 2402–2410 (2016) 13. Shin, H.-C., Roberts, K., Lu, L., Demner-Fushman, D., Yao, J., Summers, R.M.: Learning to read chest X-rays: Recurrent neural cascade model for automated image annotation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2497– 2506 (2016)

10

S. Mishra and M. Banerjee

14. Lyndon, D., Kumar, A., Kim, J.: Neural captioning for the Image CLEF 2017 medical image challenges (2017) 15. Jing, B., Xie, P., Xing, E.: On the automatic generation of medical imaging reports. arXiv preprint, arXiv:1711.08195 (2017) 16. Allaouzi, I., Ben Ahmed, M., Essaadi, A.: Automatic caption generation for medical images. In: Springer, SCA’18, Tetouan, Morocco © 2018 Association for Computing Machinery. ACM. ISBN 978-1-4503-6562-8/18/10. 10–11 Oct 2018 17. Cao, P., Yang, Z., Sun, L., Liang, Y., Yang, M.Q., Guan, R.: Image Captioning with Bidirectional Semantic Attention-Based Guiding of Long Short-Term Memory. Springer, Print ISSN 13704621 (2019) 18. Tanti, M., Gatt, A., Camilleri, K.P.: What is the role of recurrent neural networks (RNNs) in an im-age caption generator? In: Proceedings of The 10th International Natural Language Generation Conference, pp. 51–60, Santiago de Compostela, Spain, 4–7 Sept 2017 19. Mousa1 A., Schuller, B.: Contextual bidirectional long short-term memory recurrent neural network language models: a generative approach to sentiment analysis. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, vol. 1, pp. 1023–1032, Valencia, Spain, 3–7 Apr 2017

Dimensionality Reduction of Hyperspectral Images: A Data-Driven Approach for Band Selection Arati Paul, Putul Sahoo and Nabendu Chaki

Abstract Hyperspectral image (HSI) contains a large number of correlated bands. The processing of such image is associated with computational challenges like Hughes phenomenon, for which the classification accuracy decreases with increasing number of bands. Hence, dimensionality reduction (DR) of HSI is performed to enhance the efficiency of data processing. However, most of the DR techniques fail to suggest the optimum number of significant bands to achieve a good classification accuracy. In reality, user also cannot perceive the required number of bands before the analysis actually starts. In order to overcome this limitation, a multi-feature-based data-driven algorithm for unsupervised band selection is proposed in this paper. In the proposed method, bands are grouped using multi-feature analysis followed by band prioritization using signal-to-noise ratio (SNR). The performance of the proposed method is compared with other state-of-the-art methods using two benchmark hyperspectral dataset in terms of overall classification accuracy and execution time. The experimental results show that the proposed method achieved more accurate results without any user intervention in the band selection step. Keywords Hyperspectral image · Data driven · Multi-feature · Band selection · Dimensionality reduction

1 Introduction Advancement of sensor technology enables in capturing hyperspectral image from a remote platform. Hyperspectral images, captured in several narrow and contiguous spectral bands, are represented as three-dimensional (3D) data cube. The first two A. Paul (B) Regional Remote Sensing Centre—East, NRSC, ISRO, Kolkata, India P. Sahoo Maulana Abul Kalam Azad University of Technology, Kolkata West Bengal,, India N. Chaki University of Calcutta, Kolkata, India © Springer Nature Singapore Pte Ltd. 2020 R. Chaki et al. (eds.), Advanced Computing and Systems for Security, Advances in Intelligent Systems and Computing 1136, https://doi.org/10.1007/978-981-15-2930-6_2

11

12

A. Paul et al.

dimensions represent spatial features, and the third dimension carries spectral information. Remotely sensed hyperspectral data are successfully applied in many fields like geology [1], forestry [2], agriculture [3], urban [4], etc. However, the processing of huge number of bands produces many computational challenges. The Hughes phenomena [5] are often observed in HSI classification where overall accuracy of the classified output decreases with an increase in number of bands [6, 7]. The presence of highly correlated bands causes information redundancy, and therefore, dimensionality reduction (DR) is considered as an essential preprocessing step in the context of hyperspectral data analysis. DR algorithms are categorized into two major types: (1) feature extraction and (2) feature selection. Based on the usage of labelled patterns, again these categories are subdivided into supervised and unsupervised types. Principle component analysis (PCA) [8], independent component analysis (ICA) [9], local preserving projection (LLP) [10], and minimum noise fraction (MNF) [11, 12] are commonly used unsupervised feature extraction method. Unsupervised feature extraction can be done using Pooling [13], generative adversarial network [14], super-pixel-wise PCA approach [15], collaboration-competition graph [16], etc. The well-known methods that fall in supervised feature extraction category are Fisher’s linear discriminant analysis (LDA) [17], local Fisher’s discriminant analysis (LFDA) [18], etc. In feature extraction method, bands are mapped on a different subspace using a function where maximum information is concentrated into a few numbers of features. The given number of extracted features is subsequently used for analysis. Feature extraction modifies the original data and its properties, while the feature selection filters the redundant data without affecting original data. Therefore, feature selection algorithm is preferred over feature extraction when band’s physical meaning needs to be preserved. The supervised feature selection methods [19–21] depend on the labelled patterns, which are not readily available with good quality and quantity. Hence, unsupervised feature selection is generally preferred for DR of HSI. Clustering-based unsupervised methods include optimal clustering framework [22], local potential-based clustering algorithm [23], dual-clustering-based contextual analysis [24], etc. Clustering-based algorithms use statistical measures, viz. mutual information, similarity [23], contextual information [24], optimization criteria [22], etc., to group bands of similar nature into a given number of clusters, and subsequently, a representative band from each cluster is selected. The performance of clustering-based methods largely depends on the distribution of data and initial seed points. Sometimes, discriminating and informative bands are discarded due to their smaller presence. The ranking-based unsupervised band selection methods rank the band according to their importance or merit, and a given number of top-ranked bands are selected. Some of the recent ranking-based methods include manifold ranking [25], ranking-based clustering approach [26], band correlation and variance measurebased band ranking [27], and maximum variance principal component analysis [28]. In ranking-based methods, merits of individual band are evaluated at a time ignoring intra-band correlations. Therefore, redundant bands with more information easily get selected. To overcome this difficulty, the performance of a group of bands is evaluated at a time in optimization-based band selection methods. Here, the performance of a

Dimensionality Reduction of Hyperspectral Images …

13

given number of bands is optimized using an objective function through iterations. Some of the recent works in this category include robust multi-objective optimization via evolutionary algorithms [29, 30], genetic algorithms [31], firefly algorithm [32], multigraph determinantal optimization algorithm [33], etc. The optimization-based band selection involves high computational time, and some optimization approaches select the band with low signal-to-noise ratio that may degrade the classification performance. Though above-mentioned methods have their specific advantages in extracting or selecting significant and informative bands, however, they require user to decide the optimum number of bands to be finally extracted or selected. It has been observed in many experiments that optimum number of significant bands required to achieve good classification accuracy depends on the data [31]. Still, an efficient method for selecting required number of informative bands in an unsupervised manner in the present context is rare. To address this issue, an efficient multi-feature-based datadriven (MFDD) unsupervised band selection method is proposed in this paper. The proposed method includes two major steps, first, multi-feature-based band grouping and, second, band selection using signal-to-noise ratio (SNR). The band grouping does not include any heavy computing process like clustering; hence, the method is computationally efficient. Multi-feature analysis enables in selecting less correlated yet more informative bands. HSI data are generally affected by low SNR; hence in the proposed method, the SNR-based band selection is used to reduce the chance of selecting noisy bands. More importantly, the data-driven approach enables in removing dependency from user input required number of bands. The rest portion of the paper is arranged as follows. In Sect. 2, the proposed method has been explained. Section 3 details the datasets on which the proposed method is applied, and in Sect. 4, the experimental process is discussed. Finally, the conclusion is drawn in Sect. 5.

2 Proposed Method HSI is denoted by a 3D matrix W ∈ R M×N ×D where D is the number of spectral bands and M × N is the spatial dimension of each band image. The W can be represented as two-dimensional matrix X ∈ R L×D where L = M × N. The aim of this proposed DR method is to find a band subset d where d  D. The proposed method broadly consists of two steps: (a) band grouping based on some significant band properties, viz. variance [34], edge [35], and entropy [36], and (b) band selection based on signal-to-noise ratio (SNR). The detailed steps are explained in the following subsections.

14

A. Paul et al.

2.1 Multi-feature Analysis An informative band can be addressed by its ability to carry significant information. The significance of informative band depends on some band properties or features like entropy, variance, and presence of edge. So, in this proposed method, the band grouping process is done based on these three band features. The significance of these features is explained as follows. (i) Variance measures the average distance of data from the mean. So, more variance of a band indicates different earth features in that particular band varies more than the other one; this in turn helps in separating earth features in the data. Each band of hyperspectral image is considered as a set of data points. So, variance of the ith band of X can be calculated using Eq. 1.

var(X i ) =

L 1 (X i [n] − μi )2 , for each i = 1, . . . , D L n=1

(1)

where X i is the ith band of X, μi is the mean of X i , and it is given in Eq. 2. μi =

L 1 X i [n], for each i = 1, . . . , D L n=1

(2)

V ∈ R D is a vector that holds the variance of each band, i.e., Vi = var(X i ). Once the variance values of all bands are calculated, V is normalized using Eq. 3. NVi =

Vi − V min , for each i = 1, . . . , D V max − V min

(3)

where NVi denotes the normalized Vi ; V min and V max represent minimum and maximum variance value among all Vi , respectively. (ii) Edge represents the significant local changes in image intensity. Different land cover types (or classes) reflect differently in certain bands. Therefore, a clear edge is observed between two different classes in a band image where land cover reflectance varies from each other. Hence, more edge information of a band image indicates more usefulness of the band for discriminating different classes. In the proposed method, image gradients [37] are used to estimate edge information of each band of HSI. The gradient magnitude (gi ) of ith band image (Wi ) at a location (x, y) is calculated using Eq. 4.  gi (x, y) =



Wx+1,y,i − Wx,y,i

2

2  + Wx,y+1,i − Wx,y,i

(4)

Dimensionality Reduction of Hyperspectral Images …

15

Subsequently, the sum of all gradient magnitudes (G i ) of Wi is computed using Eq. 5. Gi =

M  N 

gi (x, y), for each i = 1, . . . , D

(5)

y=1 x=1

The normalized gradient magnitude (GN) is calculated using Eq. 6. NGi =

G i − G min , for each i = 1, . . . , D G max − G min

(6)

where NGi denotes the normalized G i ; G min and G max represent minimum and maximum value among all G i , respectively. (iii) In information theory, amount of information in a data depends on the degree to which the data content is unpredictable. Entropy of a signal [38] measures unpredictability; i.e., entropy is more when uncertainty is more. Therefore, entropy is used to measure the information content and can be estimated using Eq. 7 for a discrete random variable.

Hi = −

L  (P(X i [n]) × logP(X i [n])), for each i = 1, . . . , D

(7)

n=1

where Hi represents the entropy of ith band X i , and P(X i [n]) is the probability of each pixel of X i . The pixel probability is computed by calculating the relative frequency of occurrence of particular pixel value over ith band image. The normalized entropy (NH) is calculated using Eq. 8. NHi =

Hi − H min , for each i = 1, . . . , D H max − H min

(8)

where NHi denotes the normalized Hi , and H min and H max represent minimum and maximum value among all Hi , respectively. Once all the normalized feature vectors, viz. NV, NG, and NH, are computed, band grouping process commences. Values, ranging from 0 to 1, in each feature vector are assigned into ten groups using Eqs. 9–11. Thus, three grouped feature vectors, viz. GV, GG, and GH, are obtained which are used in the next stage. GVi = NVi × 10, for each i = 1, . . . , D

(9)

GGi = NGi × 10, for each i = 1, . . . , D

(10)

16

A. Paul et al.

GHi = NHi × 10, for each i = 1, . . . , D

(11)

2.2 Band Selection In the second stage of the proposed method, band selection is carried out using signalto-noise ratio (SNR) [39]. Signal-to-noise ratio can be defined as the signal power to noise power (unwanted signal power). Hence, high SNR means the signal strength is higher than noise strength. In the proposed methodology, a Gaussian noise [40] is introduced in every image and SNR is calculated using Eq. 12. SNRi =

P(X i ) , for each i = 1, . . . , D P(N )

(12)

where P(X i ) is power of the ith band signal X i (Eq. 13), and P(N) is power of Gaussian noise N. P(X i ) =

L 

X i [n]2 , for each i = 1, . . . , D

(13)

n=1

In each grouped vector, bands with the same group number are identified at a time, and their corresponding SNR values are calculated. The band having the highest SNR in each group is retained, and the rest are discarded. As this process is carried out for all three grouped vectors, there is a possibility of selecting a particular band more than once. Hence, unique bands, removing repetitions among all the feature vector groups, are finally selected. The schematic representation of the proposed method is depicted in Fig. 1, and the algorithm is given in Table 1.

Input 3D HSI

Band Features Characterization

Band grouping

WM×N×D

Output d number of selected bands

Band prioritization using SNR and Find unique bands

Fig. 1 Schematic representation of the proposed method

Dimensionality Reduction of Hyperspectral Images …

17

Table 1 Algorithm of the proposed method

Input: Hyperspectral image (W L×D

M×N×D

)

, L = M × N // Convert 3D HSI to 2D

M×N×D

1.

X

2.

Calculate grouped feature vectors GVD, GGD and GHD using equations 1 to 11.

3.

dD

4.

for each grouped feature vector FV i.

W

0 // initialization

for n = 1: 1: 10 a.

if (FV == n) 1.

m

indices of FV

2.

msnr

3.

dmsnr 1

index of the band with maximum SNR among m bands

b. end if ii. end for 5.

end for

6.

d

indices of d s.t. d>=1 Output: The selected band subset d, where d = l) 8. { 9. int mid = l + (r - l)/2; 10. if (arr[mid] == x) return mid; 11. if (arr[mid] > x) 12. return binS(arr, l, mid-1, x); 13. 14. // Else present in right half. 15. return binS(arr, mid+1, r, x); 16. } 17. return -1; 18. }

19. int main() { 20. // Enter the 2 digit vendor code 21. // List found in UI Screen 5 22. scanf("\%d", &n); 23. for(i = 0; i< n; i++){ 24. printf("Enter \%d vendor", i); 25. scanf("\%d", &arr[i]); 26. } 27. int vend = 15; //code for wholesalers 28. //introduced in SVN Commit#45S125 29. int result = binS(arr, 0, n-1, vend); 30. return 0; 31. }

Fig. 2 C code, P S : Search Key in an array, lifted from Geeks-for-Geeks and modified

36 Source Code (PS ) Symbol param defn line vend 27 arr 4 BinS arr, l, r, x 5 - 18

S. Majumdar et al. Code Comments(PS ) accn line type Token id line num POS tag 29 int vendor 3 6 NN (noun) 10-12, 15, 25, 29 int[] binary search 3 6 JJ(adj), NN 29 int

Fig. 3 Primitive extraction from source code and comments for P S (Fig. 2) Concept Token (Comment) id (Comment) line num POS tag vendor 3 6 NN (noun) Application specific binary search 3 6 JJ(adj), NN operation based on data-structure

Fig. 4 Concept inference based on ontology (algorithm and data structure) and listed concepts from code comments of P S (Fig. 2)

numbers), param (parameters) and type as primitives from source code. We employ NLP techniques (using the Stanford parser [26]) and finite automata to extract comment tokens with attributes as primitives from code comments (Example illustrated in Fig. 3). – Inference of concepts: As SD knowledge has a structured and finite vocabulary, we enumerate ontologies for concepts related to programming, architecture, exception and software life cycle as part of Comment- Mine. AD knowledge may be incomplete and unstructured, hence we list the relevant concepts. For project management, bug or version details, we manually inspected over 3000 comments of live projects and have used a rule-based algorithm to formulate indicator regular expressions consisting of phrases, numbers, special symbols and the like. We use similarity metrics with 3-gram matching to bind the extracted primitives to the enumerated ontology, listed concepts or the regular expressions (Illustrated in Figure 4). – Correlation: We correlate primitives with inferred concepts and with other primitives using a three-step process (briefly explained below). Detailed algorithms could not be enumerated due to paucity of space. Comment scope evaluation: To obtain the related list of the program symbols (source code primitives) for a comment, we develop a comment scope detection algorithm based on the placement of scope braces, type of symbol, symbol distance between comments and functions/blocks/classes and presence of symbols within comments. Association: Primitives are correlated with each other based on attributes (BinS correlates with comment#3 using line_no). Concepts inferred from primitives of comments are associated to the in scope program symbols with one or more logical relations based on decision rules and analysis of comment structure and symbol type. Like program symbol BinS of type function is mapped to mined concept Binary_Search from comment#3, with multiple associations implements, executes (Fig. 5). This also generates correlation between comments. Collision resolution: We identify and resolve natural language conflicts during correlation between concepts inferred from comments and source code primi-

Comment-Mine—A Semantic Search Approach …

37

Fig. 5 Portion of graph based on correlation of comment id#3 (P S ) with symbols Note: Scope of comment id#3 ={ binS, l, r, x, mid }. Solid, filled and dashed graph nodes symbolise primitives, inferred concepts and enumerated concepts (ontology) respectively

tives. First, using similarity metrics, we detect the conflict. Like, in {int search; // search is initialised later}, the concepts (programming related) inferred from the comment are search and initialized. search collides with concept inferred from program symbol. Hence, we tag search as redundant concept in comment. However, semantically in this comment, search is not a concept in the comment but acts only as a program symbol. To resolve this, we develop a Rule Model, based on the parse tree and dependencies generated for a comment using Stanford parser [26]. Hence, we identify search only as a program symbol which is correlated with concept initialized. – Knowledge graph construction and query processing: We add (a) enumerated concepts (ontologies) (b) primitives and their attributes and (c) inferred concepts with related associations to construct the knowledge graph. We show an example for the comment on line 6 and function binS of P S in Fig. 5. Concepts from ontologies provide more query options. Like, the query where linear data structure is used in function binS can be also supported, as array has been associated with generic concepts of data structure—linear and contiguous (Fig. 5). We use triple stores of RDF [15] to store the knowledge graph and the SPARQL [16] query language to code queries.

4 Testing and Validation of Comment- Mine In this section, we discuss the test and data set-up, results of testing Comment- Mine using queries and the approach to validate the knowledge graph. Data set-up: We crawled GitHub [7], code projects [17] and the like to extract C/C++ source with different Lines of Code and containing various types of comments

38

S. Majumdar et al.

Table 4 GT analysis for comment ( results compared with [20], shown for SD Knowledge) K tuple : , Sc short for Scenario Sc 0: Exact Match: → Machine GT Code Snippet: // binarySearch, function to sort salary int sBin(int p[], int x, int y, int z) { } Manual Annotation:

Machine Extraction:

Sc 1: Few inconsistencies with Human annotation → Mixed GT Code Snippet: //all permutations of size p Func1(int a1[], int p, int j) { } Manual Annotation:

Machine Extraction:

utility of Aggregated concepts like permutations of size p is still debatable Sc 2: No matches & incorrect Machine extraction → Human GT Code Snippet: taken from [20] int *x, y; // pointer to store address Manual Annotation:

Machine Extraction: Fixed in enhanced design of Comment- Mine Converts scenario to Machine GT using the type inferred for x and y from static analysis Machine Extraction: Sc 3: No matches and incorrect Human annotation → Machine GT Code Snippet: double x1[3], x2[3]; // inputs for Strassen’s algo Manual Annotation:

Machine Extraction:

Sc 4: No matches but correct machine and human annotations → Mixed GT Code Snippet: taken from [20] void qsort(int a[], int n) // sort using DAC Manual Annotation:

Machine Extraction:

Added in enhanced Comment- Mine, converts to Machine GT

Common Abbreviations and short forms related to DAC stored filtered based on algorithm (class of sort). Hence mapped correctly. Would fail for D&C Sc 5: Correct Machine and Human annotation (Matches for comment tokens only though) → Mixed GT Code Snippet: taken from [20] int a = 10; // initialize a Manual Annotation:

Machine Extraction:

Added in enhanced Comment- Mine, converts to Machine GT

using updated ontologies and associations Sc 6: Exact Matches and more extractions from Machine: → Machine GT Code Snippet: void MiST(double gr[i][i], int i) // Prim’s algorithm Manual Annotation:

Machine Extraction:



Comment-Mine—A Semantic Search Approach … Table 5 Validation results Scenario type No. of scenario (Table 4) Machine Scenario 0 Scenario 1 Scenario 2 Scenario 3 Scenario 4 Scenario 5 Scenario 6 Total

7841 3670 1001 3654 1310 1880 19,636 38,992

Human

39

% of MEC

7841 1302 1213 796 402 891 3498 15,943

20.1 9.4 2.6 9.4 3.4 4.8 50.3

Recall (%)

Accuracy (%)

85.12 83.00 82.40

83.00 81.50 83.00

85.65 81.23 83.98

84.03 82.10 83.31

MEC machine extracted concepts Table 6 Results of query-response Query type Precision (%) Comment-symbol correlation Entity 82.67 List 88.00 Template 88.78 Comment-comment correlation Entity 83.2 List 86.33 Template 85.13

and obtain a set of 672 source files. From the 672 source files, we extracted 5600 comments. A team of seven programmers (four of them with an experience ≥ 6 years in the software industry, the other three are graduate and undergraduate students) was formed for manual annotation of concepts from comments and their associations to related program symbols in a semi-supervised set-up. Test set-up: We collected 100 common developer queries (samples - Sect. 1) [18] for functional testing of Comment- Mine. The responses are lists consisting of one or more (a) inferred concept or (b) primitives with attributes or (c) correlated concept with multiple associations or combination of the above. Testing using queries: We test Comment- Mine with the set of 100 queries and evaluate the correctness and relevance of the responses. We achieve an average precision and recall of 85% and 83.5%, respectively (results in Table 6). The false positives are mostly due to improper or insufficient correlations between primitives and concepts. The associations to correlate inferred concepts with primitives (extracted from source code) are decided based on the attributes of the primitives. For example in code snippet { void raster (int r, double b) { } // does Quicksort }, concept Quicksort is correlated with symbol raster(of type function) using association implements,

40

S. Majumdar et al.

Table 7 Improving inference of existing approaches using Comment- Mine Comment & Code Existing approaches Comment- Mine Comment classification and quality evaluation (code lifted from [13] and modified) /* Check if applicable for big Steidl et al. [13]: mark Comment- Mine enhances endian? */ comment as not helpful as feature set by extracting long VHDLPos[arr_type]; question mark is present, with concept big endian no indicator phrases like fix, semantically using enumerated bug, task ontology (memory storage and data-type), which serves as an indicator to check the memory storage Detecting inconsistencies with source code (code lifted from [14] and modified) //SVN#45, [email protected] Tan et al. [14]: Cannot locate Comment- Mine enhances lock phrases in the while block feature set by extracting double CrRegion(void*){ before → next syntax, hence equivalent concept ..... while(p!=NULL) { reports a mismatch between barrier_wait semantically //call lock to traverse comment and code using enumerated ontology .... barrier_wait(& mic) g = (synchronisation). It provides p→next... } ..} aids for analysis like commit number, contact email through correlation between code comments

but r and b (of type built in) are correlated concept Quicksort using association used in, to ensure correctness. In some cases, a correlation with a related symbol may not be valid (not used in Quicksort). False negatives occur due to incorrect concept extractions. Validation of the knowledge graph: To analyse results of Table 6, we propose an approach based on semi-automatic analysis for ground truth (GT) conclusion of each comment and validation of the underlying knowledge graph. A total of 15,943 knowledge units have been annotated for set of 5600 comments. Alongside CommentMine extracts 38,992 knowledge units from the same set. We represent a knowledge unit as a tuple (K tuple ) of program symbol, the comment token, the inferred concept and related associations. If knowledge unit extracted from Comment- Mine matches the annotated version, it is a case of exact match, and we mark the machine (Comment- Mine) extracted set as the GT. Next, we identify six scenarios (characterised in Table 4), where automatic analysis cannot provide the final decision, and we include another round of manual review to decide on the GT. In Scenario 3 and Scenario 6, Comment- Mine extracts correct knowledge, and hence, is a case of Machine extracted GT coupled with exact matches. Scenario 2 (2.0%) represents Human annotated GT, which has been reduced from 4.1% (in [20]), using the rule model to resolve collisions and produce correct correlations (Table 5). Rest are Mixed GT cases, where both Comment- Mine extractions and human annotations are required for GT. In the present design, more ontologies have been enumerated to increase

Comment-Mine—A Semantic Search Approach …

41

correct extractions from Comment- Mine. Machine GT comprises 79.8% (Scenario 0, 3, 6) (increased from 72% in [20]) of the total machine extracted set (Table 6).

5 Conclusions We present a semantic search tool Comment- Mine for C/C++ applications, to extract knowledge related to design, implementation and evolution from comments and correlate the same (using our three-phase algorithm) to source code symbols in the form of a knowledge graph. We have formulated ten comprehensive ontologies based on programming concepts, exception types and software life cycle, which can be used for any concept mining tasks. We build a generic framework for manual annotation and for review and semi-automatic analysis of comments to conclude ground truth (released in https://sites.google.com/site/codepoapluginforeclipse/tool-for-comment-analysis). We test Comment- Mine using set of developer queries with an average precision and recall of 85% and 83.5%, respectively, on the evaluation of correctness of the responses. Apart from the primary aim to assist program comprehension using semantic search, Comment- Mine can improve inferences of existing comment analysis approaches by enhancing features sets or by providing additional support (illustrated for some major approaches in Table 7). Going forward, we plan to – Develop architecture for learning from ontologies, to enumerate more concepts and use the same to extract more relevant responses – Develop interface to support free form English language queries.

References 1. Dehaghani, S.M.H., Hajrahimi, N.: Which factors affect software projects maintenance cost more? Acta Inf. Med. Acad. Med. Sci. Bosnia Herzegovina 21(1), 63–72 (2013) 2. Koskinen, J.: Software maintenance costs. Information Technology Research Institute, University of Jyvaskyla, Tech. rep. (2003) 3. Singer, J., Lethbridge, T., Vinson, N.: An examination of software engineering work practices. In: CASCON High Impact Papers, pp. 174–188. IBM Corp. (2010) 4. Etzkorn, L.H., Davis, C.G., Bowen, L.L.: The language of comments in computer software: a sublanguage of english. J. Pragmatics 33(11), 1731–1756 (2001). (Elsevier) 5. Abebe, S.L., Haiduc, S., Marcus, A., Tonella, P., Antoniol, G.: Analyzing the evolution of the source code vocabulary. In: European Conference on Software Maintenance and Reengineering (ESMR), pp. 189–198. IEEE, New York (2009) 6. Stroustrup, B., Sutter, H.: C++ core guidelines. http://isocpp.github.io/CppCoreGuidelines/ CppCoreGuidelines. Last Accessed 1 Aug 2019 7. Github. https://github.com/ 8. Corazza, A., Maggio, V., Scanniello, G.: Coherence of comments and method implementations: a dataset and an empirical investigation. Software Qual. J. 26(2), 751–777 (2018) 9. de Souza, S.C.B., Anquetil, N., de Oliveira, K.M.: A study of the documentation essential to software maintenance. In: International Conference on Design of Communication, pp. 68–75. ACM, New York (2005)

42

S. Majumdar et al.

10. Aman, H., Amasaki, S., Yokogawa, T., Kawahara, M.: Empirical analysis of words in comments written for java methods. In: Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 375–379. IEEE, New York (2017) 11. Padioleau, Y., Tan, L., Zhou, Y.: Listening to programmers taxonomies and characteristics of comments in operating system code. In: International Conference on Software Engineering (ICSE), pp. 331–341. IEEE, New York (2009) 12. Haouari, D., Sahraoui, H., Langlais, P.: How good is your comment? a study of comments in Java programs. In: International Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 137–146. IEEE, New York (2011) 13. Steidl, D., Hummel, B., Juergens, E.: Quality analysis of source code comments. In: International Conference on Program Comprehension (ICPC), pp. 83–92. IEEE, New York (2013) 14. Tan, L., Yuan, D., Krishna, G., Zhou, Y.: icomment: bugs or bad comments? In: Association for Computing Machinery’s Special Interest Group on Operating Systems Review (SIGOPS), pp. 145–158. ACM, New York (2007) 15. Miller, E.: An introduction to the resource description framework. Bull. Am. Soc. Inf. Sci. Technol. 25(1), 15–19 (1998). (Wiley Online Library) 16. Prud, E., Seaborne, A., et al.: SPARQL query language for RDF (2006) 17. Codeproject. https://www.codeproject.com/ 18. Begel, A., Zimmermann, T.: Analyze this! 145 questions for data scientists in software engineering. In: International Conference on Software Engineering (ICSE), pp. 12–23. ACM, New York (2014) 19. Sillito, J., Murphy, G.C., De Volder, K.: Asking and answering questions during a programming change task. IEEE Trans. Software Eng. 34(4), 434–451 (2008). (IEEE) 20. Majumdar, S., Shakti, P., Das, P.P., Ghosh, S.: Smartkt: a search framework to assist program comprehension using smart knowledge transfer. In: International Conference on Software Quality, Reliability and Security (QRS), pp. 97–108. IEEE, New York (2019) 21. Pascarella, L., Bacchelli, A.: Classifying code comments in java open-source software systems. In: International Conference on Mining Software Repositories (MSR), pp. 227–237. IEEE, New York (2017) 22. Tan, L., Yuan, D., Zhou, Y.: Hotcomments: how to make program comments more useful? In: Conference on Programming Language Design and Implementation (SIGPLAN), pp. 20–27. ACM, New York (2007) 23. Howden, W.E.: Comments analysis and programming errors. IEEE Trans. Software Eng. 16(1), 72–81 (1990). (IEEE) 24. Krancher, O., Dibbern, J.: Knowledge in software-maintenance outsourcing projects: beyond integration of business and technical knowledge. In: Hawaii International Conference on System Sciences, pp. 4406–4415. IEEE, New York (2015) 25. Lattner, C., Adve, V.: The llvm compiler framework and infrastructure tutorial. In: International Workshop on Languages and Compilers for Parallel Computing (LCPC), pp. 15–16. Springer, Berlin (2004) 26. De Marneffe, M.C., Manning, C.D.: The Stanford typed dependencies representation. In: Proceedings of the Workshop on Cross-Framework and Cross-Domain Parser Evaluation, pp. 1–8. Association for Computational Linguistics (2008)

Biomarker Gene Identification Using a Quantum Inspired Clustering Approach Srirupa Dasgupta, Arpita Das, Abhinandan Khan, Rajat Kumar Pal and Goutam Saha

Abstract In this paper, we have implemented an unsupervised approach for finding out the significant genes from microarray gene expression datasets. The proposed method is based on implements a quantum clustering approach to represent geneexpression data as equations and uses the procedure to search for the most probable set of clusters given the available data. The main contribution of this approach lies in the ability to take into account the essential features or genes using clustering. Here, we present a novel clustering approach that extends ideas from scale-space clustering and support-vector clustering. This clustering method is used as a feature selection method. Our approach is fundamentally based on the representation of datapoints or features in the Hilbert space, which is then represented by the Schrödinger equation, of which the probability function is a solution. This Schrödinger equation contains a potential function that is extended from the initial probability function.The minima of the potential values are then treated as cluster centres. The cluster centres thus stand out as representative genes. These genes are evaluated using classifiers, and their performance is recorded over various indices of classification. From the experiments, it is found that the classification performance of the reduced set is much better than S. Dasgupta Government College of Engineering & Leather Technology, LB-11, Sector-III, Salt Lake, Kolkata 700106, India e-mail: [email protected] A. Das · A. Khan (B) · R. K. Pal University of Calcutta, Acharya Prafulla Chandra Roy Shiksha Prangan, JD-2, Sector-III, Salt Lake, Kolkata 700106, India e-mail: [email protected] A. Das e-mail: [email protected] R. K. Pal e-mail: [email protected] G. Saha North Eastern Hill University, Umshing Mawkynroh, Shillong 793022, Meghalaya, India e-mail: [email protected] © Springer Nature Singapore Pte Ltd. 2020 R. Chaki et al. (eds.), Advanced Computing and Systems for Security, Advances in Intelligent Systems and Computing 1136, https://doi.org/10.1007/978-981-15-2930-6_4

43

44

S. Dasgupta et al.

the entire dataset.The only free-scale parameter, sigma, is then altered to obtain the highest accuracy, and the corresponding biological significance of the genes is noted. Keywords k-NN · Microarray data · Quantum clustering · Random forest · Schrödinger equation · SVM

1 Introduction Microarray technology is a popular approach used by scientists to monitor the genome-level gene expression within different organisms. A huge amount of research is related with it [1, 10]. This is basically a high-dimensional array with a large number of features and very few samples on the other hand. Handling such data requires feature selection and feature extraction methods which eliminate noise and other irrelevant information from the data. The objective of feature selection works by removing irrelevant features and redundancy. The purpose of feature selection is to get rid of irrelevant and noisy genes from the input dataset, to speed up the processing of data by reducing the dimensionality [5]. Clustering techniques are usually used to form groups of members having similar nature or class. Clustering of genes revolves around the idea of creating groups of related genes, or genes having a very high correlation with each other [4]. Once clustered, the cluster centres are retained, and the other members of the clusters are eliminated as these centres can stand out as the representative, i.e. the most relevant gene. The others are considered redundant and hence eliminated. In our present work, we have reduced the initial number of genes from 2000 to 32 in the case of the first dataset considered, and in the case of the second dataset, from 218 to 14 genes.

1.1 Challenges in Gene Expression Data Analysis Initially, researchers focussed on genes co-expressing across sets of experimental conditions, implying, necessarily, the usage of clustering methods. More recently, scientists have been concentrating more on finding differentially expressed genes among the different classes of experiments, or related to different sources of clinical findings, as well as in extracting predictor genes. There is also a requirement for finding methods that allow the easy and quick conversion of biological information to the results of microarray experiments. Many distinct approaches, such as the Gene Ontology (GO) consortium, pathways databases, protein functional pattern, provide well-organised annotations for genes. Analysis of gene expression using microarray technology [2] has brought up a wide range of dimensions for investigating the biological nature of cells and organisms. Biomedical applications have initiated the use of existing technologies and the development and usage of tools. In terms of data analysis methodologies, it is

Biomarker Gene Identification Using a Quantum …

45

implied that, in addition to clustering, there is high demand for efficient methods for class prediction, which would include the response to drugs or therapies, or any phenotype or genotype defined independently of the gene expression profile. The novelty of our approach lies in the use of clustering for the purpose of feature selection.We have used quantum clustering other than traditional clustering methods as the other methods mostly require the number of clusters to be provided by the user whereas the use of quantum clustering generates the number of cluster centres based on the value of some parameters.

1.2 Traditional Clustering Methods and Their Limitations Clustering of data is a well-known problem of pattern recognition. A growing body of formal methods aiming to model, structure and/or classify data already exist, e.g. linear regression, principal component analysis, cluster analysis, support vector machines, and neural networks. Cluster analysis is an interesting tool because it typically does not make assumptions on the structure of the data. Since, often, the structure of the data is unknown, clustering techniques become particularly interesting for transforming this data into knowledge and discovering its underlying structure and patterns. Clustering is a hard problem, and a vast body of work on these algorithms exist. Typically, no single algorithm can respond to the specificities of all data. Different methods are suited to datasets of different characteristics and, often, the challenge of the researcher is to find the right algorithm for the task. Most of the clustering algorithms employed are distance-based. The most widely used clustering algorithms for gene expression data include hierarchical clustering (HC), k-means clustering, etc. These algorithms are quite simple and visually appealing, but their performances could be sensitive to noise. The k-means algorithm has a few limitations which are essential to keep in mind when using it and before choosing it. First of all, it requires a metric. The second main limitation of the k-means algorithm is its sensitivity to noise. Third limitation, the choice of initial centres influences the results. There exist heuristics to select the initial cluster centres, but none of them is perfect. For HC, once a decision is made to combine two clusters, it cannot be reversed. No objective function is directly minimised, sensitivity to noise and outliers, difficulty to handle different-sized clusters and convex shapes.

1.3 A New Clustering Approach—Quantum Clustering (QC) Currently, there exist several existing algorithms that are more robust than traditional algorithms by having broader applicability or being less dependent on input parameters, e.g. algorithms that do not take more than one parameters for performing an analysis. One such approach is quantum clustering (QC). This problem of unsupervised learning is in general ill-defined and has not been addressed much.

46

S. Dasgupta et al.

The problem was initially suggested by Horn [8]. Solutions to such related problems can be obtained from intuitive decisions derived from Physics. We present an alternative means that is also based on physical intuition and is derived from quantum mechanics. As an introduction to our approach, we start with the scale-space algorithm which assumes a Parzen-window estimator of the probability distribution of the data that has been provided. Euclidean space of dimension up to an overall normalisation by derived from data points in a and summing over all of them. The estimator is constructed by assuming Gaussian wave-functions. After this, we use the Schrödinger equation; from that equation, we will derive potentials for each data point in Gaussian state. We identify cluster centres with its minima, where the location of the cluster centres are the data points. We use Gradient descent algorithm to perform the clustering, i.e. moving the datapoint towards the cluster centres, and also we find the minima which are then termed as cluster centres. We know that elements are attracted towards the low potential elements and form clusters. The same concept is applied here by finding potential values for each data point and then find the minima of potential and moves the data points towards the minima. Now the actual goal of our experiment to find out the essential features from the dataset, we can say that the centre of each cluster is the representative of the corresponding clusters. Therefore, these centres are then meaningful features in the dataset. To validate the output, we are using classification methods to find out the accuracy over the selected features (centroid) through QC.

2 Preliminaries The techniques used in implementing the algorithm have been explained below.

2.1 Preprocessing Using MIN-MAX Normalisation Technique Min-Max is a technique that helps to normalise the data. It is used to scale the data between 0 and 1. This normalisation helps us to interpret the nature of the data easily. X represents the attributes or data points in the dataset, Min(X ) is the minimum value of X and Max(X ) is the maximum value of X . Scale =

x − Min(x) Max(x) − Min(x)

(1)

Biomarker Gene Identification Using a Quantum …

47

2.2 The Quantum Clustering Methodology The main clustering algorithm that we are going to use is the Quantum Clustering (QC) algorithm which was originally suggested by Horn and Gottlieb [6–8]. Parzen-Window Estimator: As an first step to our approach we start with the scalespace algorithm which uses a Parzen-window density function as the estimator of the probability distribution of the data at hand. The estimator is constructed by taking Gaussian of each of N data points in a Euclidean space of dimension d and summing them. This can be expressed, in a normalised form by: ψ(X ) =



  (x − xi )2 , ex p − 2σ 2

(2)

where xi is the i-th data point. So, {x1 , x2 , . . . , xn } are the collection of geometrically defined dataset. Sigma (σ ) is an input parameter. Schrödinger Potential: We introduce Schrödinger equation by the following:  σ2 2 H ψ = [T + V (x)] ψ = − ∇ + V (x) ψ(X ) = Eψ(X ), 2 

(3)

where T is the kinetic energy operator or free particle contribution as represented in Laplacian form, with the usual mass term m has been set to unity and the Planck constant term h is reproduced in the form of a variance σ . E eigenvalue, which is defined as: E = d2 , is the lowest possible eigenvalue of H . The Gaussian function thus represents the ground state of H . Solving for the potential V (X ), we get: V (X ) = E + =E−

 2  − σ2 ∇ 2 ψ

(4)

ψ

  d 1  (x − xi )2 2 + · ex p − − x ) (x i 2 2σ 2 ψ 2σ 2

(5)

Now, to find min(V ) = 0, this sets the value of: ( −σ2 ∇ 2 )ψ ψ 2

E = −min

(6)

The value of E should be then 0 < E ≤ d/2. V (X ) is determined uniquely. E has to be positive since V (X ) is always a non-negative function. After calculating V from (3). Now compute the local minima of the potential V by Gradient Descent Algorithm. Gradient Descent Algorithm: For computing local minima, we use gradient descent. After discovering local minima termed as cluster centre, allocating data points to different clusters.

48

Original Dataset

S. Dasgupta et al. Normalise Dataset using MIN-MAX Method

Feature Selection using Quantum Clustering

Classify the Data using the Selected Features

Evaluate Biological Significance

Fig. 1 Block diagram representation of the proposed work Table 1 Datasets used Datasets Number of samples Colon miRNA

Number of genes Class labels

Samples per class

62

2000

217

218

40 22 140 46 32

Tumour Normal Tumour Normal Cell-line

yi (0) = xi , yi (t + Δt) = yi (t) − η(t) · ∇V (yi (t)),

(7)

which takes η as the learning rate. Repeatedly calculating, and then using step-down operation, an objective function’s gradient like this constitutes gradient descent (GD). After performing GD over potential V , we find the local minima of V , which is termed as cluster centres. After finding cluster centres perform clustering by moving the data towards their cluster centres by calculating minimal distance from centres to the data points or by sliding the data points down the slopes of potential with gradient descent until they reach the centres.

3 Methodology The steps followed to retrieve the significant genes from the dataset are represented in Fig. 1. The method involves initial preprocessing followed by quantum clustering technique to identify the cluster centres as important genes and the final evaluation of the selected features with different classifiers. The datasets used are the colon cancer microarray dataset and another a microRNA dataset with three classes. Table 1 gives the details of the gene expression datasets used in the simulation. The dataset is a two-class gene expression profiles. The datasets consist of a matrix of gene expression vectors obtained from DNA microarrays for many patients. The dataset was obtained from cancerous or healthy colon tissues. Colon cancer gene expression information was extracted from DNA microarray data resulting, after preprocessing (normalisation), in a table of 2000 gene ×62 tissues expression values. The 62 tissues include 22 normal and 40 colon cancer tissues. The matrix contains the expression of the 2000 genes with the highest minimal intensity across the 62 tissues. The problem is to distinguish the cancer samples from the normal samples.

Biomarker Gene Identification Using a Quantum …

49

Algorithm 1 Feature Selection using Quantum Clustering (QC) Input: Gene expression dataset (X ), each datapoint in the input dataset (xi ), sigma value (σ ) Output: Cluster Centres (feature selection) 1: Set η = 0.1(η is the learning rate), r epetition = 100, extract total number of features, N 2: Compute q ← 1/2σ 2 3: for j = 1 to N do   4: Compute ψ(X ) ← ex p −(x − xi )2 /2σ 2 5: end for 6: Calculate the potential V for each data point after computing ψ(X ) 7: for i = 1 to N do   8: V (X i ) ← qψ · (x − xi )2 · ex p −q(x − xi )2 9: end for 10: Perform gradient descent to calculate local minima of V 11: for i = 1 to r epetition do 12: D ← D − η · dv (D being a matrix initialized to X ) 13: dv ← gradient o f V 14: η ← 0.5 · η 15: end for 16: Return D, where D contain the local minima of V , termed as cluster centres 17: Extract the centres and their locations, i.e. selected features

Gene/feature selection using quantum clustering is performed over genes to find out the local minima, which are termed as cluster centres. These local minima are termed as meaningful features/genes. Another dataset is miRNA dataset used in the simulation is the dataset with three class profiles. The datasets consist of a matrix of miRNA expression vectors. miRNA is microRNA; it is associated with normal and diseased tissues. miRNA expression information was extracted from microarray data, after preprocessing (normalisation), in a table of 217 miRNA ×218 Sample tissues expression values. The 218 tissues include 46 normal and 140 tumour tissues and 32 cell-line tissues. The problem is to distinguish the cancer samples from the normal samples. And also find out the significant miRNAs. Feature selection (important miRNA) using quantum clustering is performed over 217 miRNAs to find out the local minima, which are termed as cluster centres. These local minima are termed as meaning full miRNA.

3.1 Quantum Clustering Algorithm The performance of the above algorithm was tested using the following metrics: Positive (P): Observation is positive. Negative (N ): Observation is not positive. True Positive (TP): Observation positive, Prediction positive. False Negative (FN): Observation positive, Prediction negative. True Negative (TN): Observation negative, Prediction negative. False Positive (FP): Observation negative, Prediction positive.

50

S. Dasgupta et al.

Actual class

Predicted class

Positive

Negative

Total

Positive

TP

FP

TP + FP

Negative

FN

TN

FN + TN

Total

TP + FN

FP + TN

N

Fig. 2 Confusion matrix

Precision: To get the value of precision, we calculate the ratio of the total number of correctly classified positive examples to the total number of predicted positive examples. High Precision indicates an example labelled as positive indeed being positive (a small number of FP). Precision =

TP TP + FP

(8)

Recall: In binary classification, recall is called sensitivity. It measures the probability that a relevant document is retrieved by the query. Recall can be defined as the ratio of the total number of correctly classified positive examples to the total number of positive observations. High Recall shows that the class is correctly recognised (a small value of FN). TP Recall = (9) TP + FN F-measure: A measure that combines precision and recall is the harmonic meaning of precision and recall, the traditional F-measur e or balanced F-score. The F-measure will always be nearer to the smaller value of Precision or Recall. F-measure =

2 ∗ Recall ∗ Precision Recall + Precision

(10)

over the total number of possible cases. Accuracy =

TP + TN TP + FP + TN + FN

(11)

Confusion Matrix: A confusion matrix is a tabular representation of prediction results on a classification problem. The number of correct and incorrect predictions is summarised with count values and broken down individually for each class. This is the key to the confusion matrix (Fig. 2).

Biomarker Gene Identification Using a Quantum …

51

Fig. 3 Potential values of each datapoint of the colon cancer dataset

4 Results and Discussion This part presents the simulation results of the proposed approach using two cancer datasets. Initially, the quantum clustering algorithm is implemented as a solution to the Schrödinger’s Potential Equation to obtain the potential values of each point or features. The lowest values are considered as the cluster centres. Now with this new set of cluster centres, classifiers are run to evaluate the performance with the original dataset. The results returned are promising and represented below. The proposed approach is implemented in Python and executed in a PC with Intel Core i3 processor and 8 GB of RAM.

4.1 Case 1: Colon Cancer Data Quantum clustering is performed over 2000 genes/features, which contain both genes and UTRs, with varying values of sigma (σ ), considered as an input parameter. A different number of cluster centres are obtained in each case, which gives different classification accuracy. Figure 3 shows the potential values calculated across the number of genes where the lower potential values correspond to the cluster centres. The term sigma (σ ) is used to control the number of cluster centres that result from solving the potential equation of Schrödinger. This variable is then chosen by the user to satisfy subjective criteria, such as limiting the number of clusters. The corresponding features are then evaluated for accuracy. From Table 2, we can see that for σ = 0.85, there are 32 numbers of features selected through QC for which we are getting 84% of classification accuracy. So, the final chosen features/genes are 32.From the results, it can be observed that as σ increases the selected features also increase, but the best classification accuracy is not necessarily obtained at the highest

52 Table 2 Selected features using QC

Fig. 4 Comparative study based on the different metrics of evaluation using random forest

S. Dasgupta et al. Index Sigma value (σ )

Selected features Classification using QC accuracy

1 2 3 4 5

6 16 26 32 40

0.55 0.65 0.76 0.85 0.95

complete dataset

0.58 0.61 0.69 0.84 0.76

selected features

0.90

0.75

0.60 accuracy

precision

recall

f-score

value of σ considered. For classification, we need to split the dataset. Here, we have divided the dataset as 80% data for training and 20% data for testing purpose. The histogram in Fig. 4 shows the comparative performance of the total dataset with the reduced dataset after performing quantum clustering. The results are reported with the output of the Random Forest classifier on the indices of accuracy, pr ecision, r ecall, and f -measur e. In all the four cases considered, the performance of the reduced dataset on the factor precision is the best. Given a set of data points from repeated measurements of the same quantity, the set can be said to be precise if the values are close to each other, while the set can be said to be accurate if their average is close to the true value of the quantity being measured. Hence, it can be claimed definitely that the reduced set gives more stable result over a wide range of readings. The reason for such behaviour can be attributed to the presence of a large amount of unwanted information or genes which produce aberration in the performance of actual information, which again is the differentially expressed genes. Accuracy values across different classifiers are shown in Fig. 5 and the performance of random forest as the classifier is the best. Table 3 shows the number of genes annotated from the total number of genes found as cluster centres using DAVID ONTOLOGY, with their corresponding p-values. Biological Significance: The ultimate aim of the work is to find responsible genes for a disease. Therefore, the genes obtained earlier are further evaluated biologically using ontological tools like DAVID [3], which we have used to obtain the p-values.

Biomarker Gene Identification Using a Quantum … Fig. 5 Comparative study based on accuracy of different classifiers

0.90

53

complete dataset

selected features

0.75

0.60 svm

Table 3 Selected features and corresponding p-values

k-nn

random forest

Selected features through QC

Number of genes

p-value

6 16 26 32 40

2 9 12 27 34

NA 4.0e−2 5.6e−3 2.9e−7 3.8e−5

4.2 Case 2: miRNA Data The miRNA dataset is a microarray dataset containing 217 miRNA samples belonging to three different classes. The dataset is normalised. The resultant data is then clustered with Schrödinger’s Equation to obtain the lowest potential points which serve at the cluster centres or the important genes as represented in Fig. 6. Now, the parameters of the Schrödinger equation that controls the number of cluster centres produced is controlled by sigma. We vary it to obtain that value of sigma for which the accuracy obtained from the classifier is maximum. From Table 4, it is noted that the highest accuracy of 0.81 is obtained when σ = 0.15. As discussed for the colon dataset here to the lowest value of σ does not result in the best accuracy obtained. This could be because of the presence of spurious genes in the obtained set of genes. Figure 7 shows the performance of the selected dataset over the total dataset with Random forest classifier over for different performance metrics, whereas Fig. 8 is a comparison of accuracy over different classifiers. Biological Significance: The significant miRNAs found from the algorithm are searched in the mirDB (http://www.mirdb.org/database), to find genes affected by

54

S. Dasgupta et al.

Fig. 6 Potential value of each miRNA

Table 4 Selected features of miRNA using QC

Fig. 7 Comparative study based on the different metrics of evaluation using random forest

Index

Sigma value (σ )

Selected Classification features using accuracy QC

1 2 3 4 5

0.77 0.65 0.35 0.15 0.07

2 5 12 14 27

0.90

complete dataset

0.51 0.62 0.70 0.81 0.72

selected features

0.65

0.40 accuracy

Fig. 8 Comparative study based on accuracy of different classifiers

0.90

precision

recall

complete dataset

f-score

selected features

0.75

0.60 svm

k-nn

random forest

Biomarker Gene Identification Using a Quantum … Table 5 miRNA-gene association and corresponding p-values miRNA Genes associated hsa-let-7i hsa-mi R-134 hsa-mi R-214 hsa-mi R-186 hsa-mi R-181

DLX5, ADHFE1, DICER1, ATP2A2, CAMK2N1 ROBO2, STOX2, ABHD11, GPR137, SHARCD1 LSMI2, RC3H1, NAA15, AMMECR1L, ZPAND3 LIN54, GK5, MAP3K9, MAN2A1, DIPK2A G3BP2, IRS1, STK24, ZFHX4, CD69

55

p-values 3.5 × 10−6 7.0 × 10−6 3.5 × 10−6 3.5 × 10−5 3.5 × 10−3

them. Table 5 shows top five significant genes affected by the miRNAs and their p-values (obtained from DAVID ONTOLOGY), which are of the order of 1 × 10−6 and can be considered to be quite significant.

5 Conclusion The work has aimed at finding biomarker genes from microarray data, which are related to two types of data. The first type being microarray with genes versus samples and the other being microRNA data containing microRNA versus samples. We have used the idea of quantum clustering to identify the lowest potential region, which represents the cluster centres. The cluster centres are actually the most significant genes. The remaining genes are eliminated in the process as they are considered redundant and irrelevant. The reduced gene set is compared with the entire set to evaluate its classification efficiency, which stands out to be superior. The cluster centres are tuned using the only free-scale parameter σ to find the best set with maximum efficiency. The resultant gene set is then evaluated with different classifiers. In all cases, the reduced set outperforms the entire set. In the case of the colon dataset, it is found that at σ = 0.85, we are getting the highest classification accuracy as 84%. Also, the biological significance of the set is verified from DAVID Ontology [9], and pvalues found are quite satisfactory and of the order minus seven. The other dataset, which is the miRNA dataset, gives the highest accuracy of 81% with σ = 0.17. The significant miRNAs are noted, and their association with genes noted, and further, their significant p-values are noted. Here too, very low p-values are obtained which also justify the selection. The work is, however, not complete in the sense that it could be extended with other methods of gene selection.

References 1. Babu, M.M.: Introduction to microarray data analysis. Comput. Genomics: Theory Appl. 225, 249 (2004)

56

S. Dasgupta et al.

2. Dang, V.Q.: Evolutionary approaches for feature selection in biological data. Ph.D. thesis, Computer and Security Science, Edith Cowan University (2014) 3. Dennis, G., Sherman, B.T., Hosack, D.A., Yang, J., Gao, W., Lane, H.C., Lempicki, R.A.: David: database for annotation, visualization, and integrated discovery. Genome Biol. 4(9), R60 (2003) 4. Gupta, S., Singh, S.N., Kumar, D.: Clustering methods applied for gene expression data: a study. In: 2016 Second International Conference on Computational Intelligence & Communication Technology (CICT), pp. 724–728. IEEE, New York (2016) 5. Hira, Z.M., Gillies, D.F.: A review of feature selection and feature extraction methods applied on microarray data. Adv. Bioinform. 2015 (2015) 6. Horn, D., Axel, I.: Novel clustering algorithm for microarray expression data in a truncated SVD space. Bioinformatics 19(9), 1110–1115 (2003) 7. Horn, D., Gottlieb, A.: Algorithm for data clustering in pattern recognition problems based on quantum mechanics. Phys. Rev. Lett. 88(1), 018702 (2001) 8. Horn, D., Gottlieb, A.: The method of quantum clustering. In: Advances in Neural Information Processing Systems, pp. 769–776 (2002) 9. Jiao, X., Sherman, B.T., Huang, D.W., Stephens, R., Baseler, M.W., Lane, H.C., Lempicki, R.A.: DAVID-WS: a stateful web service to facilitate gene/protein list analysis. Bioinformatics 28(13), 1805–1806 (2012) 10. Marconi, D.: New approaches to open problems in gene expression microarray data. Ph.D. thesis, Alma Mater Studiorum, Università di Bologna (2008)

Shot Classification and Replay Detection in Broadcast Soccer Video Saikat Sarkar , Sazid Ali and Amlan Chakrabarti

Abstract In this work, we have classified the frames of a broadcast soccer video into four classes, namely long shot, medium shot, close shot and logo frame. A two-stream deep neural network (DNN) model is proposed for the shot classification. Along with static image features, player attributes like count of the players in a frame, area, width and height of the players are used as features for the classification. The heterogeneous features are fed into the DNN model through a late fusion strategy. In addition to shot classification, we propose a model to detect replay within a soccer video. The logo frames are used to decide the temporal boundary of a replay segment. A majority class assignment strategy is employed to improve the accuracy of replay detection. The experimental results show that our method is at least 12% better than that of similar approaches. Keywords Shot classification · Replay detection · Soccer video analysis · Two-stream DNN

1 Introduction During the broadcasting of a soccer match, the broadcast service providers often add different camera angles, zooming effects and animated frames in the video to make the video interactive to the viewers. Apart from capturing the global view of the soccer field, the camera is focused to track the player in control of the ball. Close-up S. Sarkar (B) Bangabasi College, University of Calcutta, Kolkata, India e-mail: [email protected] S. Ali Adamas University, Kolkata, India e-mail: [email protected] A. Chakrabarti A. K. Choudhury School of Information Technology, University of Calcutta, Kolkata, India e-mail: [email protected] © Springer Nature Singapore Pte Ltd. 2020 R. Chaki et al. (eds.), Advanced Computing and Systems for Security, Advances in Intelligent Systems and Computing 1136, https://doi.org/10.1007/978-981-15-2930-6_5

57

58

S. Sarkar et al.

(a) Long shot

(b) Medium shot

(c) Close shot

(d) Logo frame

Fig. 1 Different types of frames of a broadcast soccer video

views are shown to capture the emotions of the players and audiences. For better visual experience, replay segments of interesting events are shown. An automated soccer video processing system must differentiate the different types of frames to get insights from the video. For example, in an automatic ball possession statistics generator [9], close-up views and replays need to be discarded to generate the possession statistics. Shot classification and replay detection are also important activities for video summarization [4], event detection and content retrieval [11]. Based on the camera zooming effect, frames of a soccer video can be broadly classified into three categories, long shot, medium shot and close shot frames. The long shot frames have the lowest zooming effect. The long shot frames cover maximum portion of a soccer field, therefore, provides a global view of the field. The majority frames in a soccer video are of long shot type. An example of a few long shot frames is shown in Fig. 1a. Medium shot frames have zooming effect higher than long shot frames and lower than close shot frames. These types of frames are used to display one or more specific persons in the field. In a medium shot frame, a whole player is usually visible as shown in Fig. 1b. The close shot frames have the highest camera zoom. These frames are

Shot Classification and Replay Detection …

59

used to show close-up views of specific objects in the field. Close shot frames usually show the above waist view of a player, as shown in Fig. 1c. In addition to the three types of frames mentioned above, we introduce one more frame class, the logo frame. The logo frames contain the logo of the tournament, as shown in Fig. 1d. In a soccer game, important events like a goal, red card and injury of a player lead to break in the game. During this break, service providers repeat important events of the game to the audiences. These repeated events are called replays. The replays are normally shown with different camera zoom level or from different camera angles. To differentiate the replay segment from the live game, broadcasters usually add animated logos before and after the replay segment. In computer vision, different methods have been proposed for shot type and replay detection in soccer videos. In [6], the frames have been classified based on the detection of grass field pixels and the size of players. The soccer frame is decomposed into a binary image where zero represents the grass region and one represents foreground objects like a player. The grass field is identified by finding dominant color pixels. The players in the field are detected as the largest contour among all the foreground objects. Finally, the classification is done based on the grass field pixels and the size of the players. The shot classification method based on grass color pixels and edge distribution is proposed in [14]. The authors have used integrating grass color pixels to train a Gaussian mixture model (GMM) for predicting grass distribution. The edge distribution is calculated through a canny operator. The grass color pixels and edge distribution features are fed to a multi-class SVM for shot classification. An automatic technique for color model extraction of playing field and player uniform was proposed by [12]. The proposed methodology is capable of handling multi-colored patterns like striped uniforms and playing fields. A dominant frame color-based shot classification method is proposed in [4, 10, 13]. The authors have considered that a soccer frame is dominated by green colored soccer field and the distribution of the green color varies in different types of shots. A replay detection method based on the detection of logo templates in the frame is proposed in [6, 7]. Replay segments are identified by finding frames having a logo. The logos within the frame are detected based on the template matching technique. A generic replay detection method based on the difference of motion between replays and normal shots is proposed in [12]. In our case, we have used a deep neural network (DNN) model for shot classification. Unlike detecting the logos within a frame through a separate method, we consider frames with the logo as a separate frame class. The frames are classified using a DNN model with heterogeneous features, as discussed in Sect. 2.1. Based on the classification of the frames, the replay segment in a video is detected. In our case, the logo frames are used to decide the temporal boundary of a replay segment as discussed in Sect. 2.2. The rest of the paper is organized as follows. In Sect. 2, we have described our proposed methodology. Classification of frames and detection of highlights are also described in this section. The experimental results on open-source datasets are shown in Sect. 3. Finally, we conclude in Sect. 4.

60

S. Sarkar et al.

Fig. 2 Pipeline of our model

2 Proposed Method The overall strategy of our model is shown in Fig. 2. Our proposed method is divided into two steps. At first, we classify the soccer frames which are followed by the detection of the soccer highlights. We take a soccer frame as input. Next, the players are detected in the frame. Then, the input frame and the attributes of the players are passed to our proposed two-stream DNN network. The output of the network is the shot type of the input frame. Based on the shot type of the frames the replay segments are detected. Next, we discuss our shot classification strategy.

2.1 Shot Classification Inspired by the huge success of deep convolution neural network for image classification, we propose a two-stream deep neural network (DNN) for shot classification. The network is shown in Fig. 3. The top stream takes an RGB frame as input. The RGB frame is passed to a convolutional neural network (CNN) resulting in the generation of a V1 dimensional feature vector as output. Due to higher accuracy and small size, the Xception network [2] is used as the CNN network in our case. Along with the static image features, attributes of the players provide a strong clue for classification. As an example, due to the lowest zooming effects, the size (height, width) of players in a long shot frame is significantly different from a medium or close shot frame. Similarly, the number of players also varies in different types of frames. A logo frame is mostly covered by the logo. Therefore, players are fully or partially invisible in a logo frame. So we take four numerical features, the player count, area, width and height of players as an input to the network through the second stream of our model. The players within a frame are detected using the faster R-CNN method [8], shown in Fig. 4. The output of the player detector is a set of bounding boxes representing the possible location of the players and confidence score to each bounding box. Assuming

Soccer frame

Number of players, width height and size of the players Player attributes

61

Softmax

Shot Classification and Replay Detection …

CNN layer

FC7

FC1

FC2

FC6

FC8 Classificaion

Concatenation layer

Fully connected layers

Fig. 3 Pipeline of the two-stream DNN model

Fig. 4 Player detection in different types of frames. Detected players are marked with red rectangles

B number of bounding boxes detected by the player detector, the player count for the frame is set to N , where N is the number of bounding boxes whose confidence score is greater than or equal to the threshold γ . The width, height and area of the players are obtained by averaging width, height and area of all N bounding boxes. The input features are then passed through a series of six fully connected layers FC1 to FC6 and resulting in V2 dimensional feature. Between two fully connected layers, a dropout is added to avoid overfitting of the network. We combine both streams in a late fusion strategy. Features of both streams are concatenated in the concatenation layer, resulting in the feature vector of length V1 + V2 . The concatenated feature is then passed through FC7 and FC8 followed by the softmax layer for classification. Standard cross-entropy loss is used for training of the network. The procedure to detect replay is described next.

2.2 Replay Detection In a broadcast video, important events are re-displayed to the viewers as replays. Animated tournament logos are displayed at the beginning and end of a replay. In our case, the logo frame class represents the frame with a logo. An example of such sequence is shown in Fig. 5. Our replay detection strategy is guided by the fact

62

S. Sarkar et al.

Fig. 5 Example of a replay segment

that a replay segment always begins with a sequence of logo frames, followed by a sequence of long/medium/close shot frame called the replay segment. The replay ends with another sequence of logo frames.

Algorithm 1: Replay detection. Input : Frames of a soccer video. Output: Replay segment 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

while Video read not complete do Read a frame f Set c = shot t ype o f f Set f lag = 0 if c == logo frame then if f lag == 0 then Set f lag = 1 // Beginning of replay end else if f lag == 2 then Set f lag = 3 // End of replay end end if (c == long shot O R medium shot O R close shot) then if ( f lag == 1) then Mark f as replay // Replay segment Set f lag = 2 end else if f lag = 3 then Set f lag = 0 // Reset flag end end end

We first detect the sequence of logo frames, which denotes the starting point of a replay segment. We then wait for another sequence of logo frames, which denotes the end of the replay segment. The video segment between the starting and ending of a logo frame is marked as a replay segment. Our logo detection strategy is summarized in Algorithm 1. A variable f lag is used to mark different states of the video. Initially,

Shot Classification and Replay Detection …

63

we set f lag = 0. Once we get a sequence of logo frame, we change f lag = 1 which denotes the beginning of the replay. Detection of long/medium/close shot frame with f lag = 1 denotes the replay segment. On detection of a replay segment, we change f lag = 2. We set f lag = 3 on getting of another logo sequence which denotes the end of the replay. Finally, we reset f lag = 0 after getting non-logo frames at the end of a replay segment. Note that the accuracy of replay segment detection is dependent on the accuracy of the frame classifier. Due to high inter-frame similarity, a frame with a small logo icon is sometimes misclassified as a non-logo frame or vice versa. To avoid this misclassification error a majority class assignment strategy is proposed. We take a temporal window of frames of length w. We then find the majority class within w. Then, class labels of all the frames within w are changed to the class label of the majority class. The class of the ith frame L i is selected using (1). L i = maxlabelw ,

for i = 1, 2, ..., w,

(1)

where maxlabelw is the majority class within w. The temporal window is shifted by w frames.

3 Experimental Results To experimentally evaluate our model, we experiment with SoccerNet [5] dataset. Each of the videos is of 398 × 224 resolution and encoded with 25 frame rate per second. We choose six such videos for our experiment from the 2014-15 EPL tournament, each of which is about 45–50 minutes duration. To validate our model, we prepared ground truth data. Each frame of the video clips are marked with the pair {T, R}, where T ∈ {long shot, medium shot, close shot,

Fig. 6 Example of correctly classified shots

64

S. Sarkar et al.

Fig. 7 Example of wrongly classified shots

logo f rame} denotes the class of the frame and R ∈ {r eplay, nor eplay} denotes if the frame is part of a replay segment. We experimentally set player detection score threshold γ = 0.65. The window size for the majority class assignment strategy is set to w = 15 in our case. The size of FC1 is set to 64 followed by 512 and 512 for FC2 and FC3, respectively. The FC4 and FC5 are set to 1024 followed by FC6 of size 2048. The length of both V1 and V2 is 2048. So the concatenation layer size is 4096. The size of FC7 and FC8 are set to 1024. As we have four classes, the softmax layer size is set to 4. The dropout rate to all dropout layers is set to 30%. Our two-stream DNN model is trained with a total of 48,000 images, where each class contains 12,000 images. The images are resized to 398 × 224. We experimentally set the learning rate to 0.0001. The RMSprop optimizer is used for the training. The weights of the Xception network are initialized with ImageNet [3], whereas the weights for the fully connected layers are initialized randomly between 0 and 1. The training is done for total of 1000 epochs for which we got 98.23% training accuracy. The result of some correctly classified frames is shown in Fig. 6. A few examples of wrongly classified frames are shown in Fig. 7. The comparison of shot classification and replay detection accuracy of our method with [6] is shown in Table 1. Overall shot classification accuracy of our method is 92%, whereas 80% for [6]. The normalized confusion matrix is shown in Fig. 8. We see that we get higher classification accuracy for the long shot and logo frame classes. The medium shot frames are mainly misclassified by close shot frames. We assess our replay detection accuracy based on the intersection over union (IoU) on the temporal window. We say a replay is truly detected if I oU ≥ τ , where

Table 1 Comparison of shot classification accuracy Videos Shot classification accuracy (%) Our method Proposed by [6] Video 1 Video 2 Video 3 Video 4 Video 5 Video 6

90 93 90 91 92 95

81 79 80 78 80 81

Replay detection accuracy (%) Our method Proposed by [6] 96 95 97 96 90 92

89 83 84 84 90 85

Shot Classification and Replay Detection …

65

Fig. 8 Normalized confusion matrix

τ is a predefined threshold. We experimentally set τ = 0.2 in our case. The average replay detection accuracy of our method is 94%, whereas 86% for [6]. The proposed method takes .032 seconds on an average to process a frame with an unoptimized Python 3.5 code on a PC with Intel i5 2.3 GHz processor, 8 GB of RAM and Windows 10 operating system. For the implementation of the CNN model, we have used the TensorFlow-Keras framework [1]. The training is done on NVIDIA GTX 1070 Ti GPU with CUDA 10.1 and cuDNN 7.6.

4 Conclusions We propose a model for shot classification and replay detection in broadcast soccer videos. Experimental results show that our method is effective. Our method is also fast and can be used for real-time processing. In the future, we plan to extend our work for video event detection based on our shot classification and replay detection model.

References 1. Chollet, F., et al.: Keras. https://keras.io (2015) 2. Chollet, F.: Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1251–1258 (2017) 3. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE, New York (2009) 4. Ekin, A., Tekalp, A.M.: Shot type classification by dominant color for sports video segmentation and summarization. In: 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings (ICASSP’03), vol. 3, pp. III–173. IEEE, New York (2003)

66

S. Sarkar et al.

5. Giancola, S., Amine, M., Dghaily, T., Ghanem, B.: Soccernet: A scalable dataset for action spotting in soccer videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1711–1721 (2018) 6. Nguyen, N., Yoshitaka, A.: Shot type and replay detection for soccer video parsing. In: 2012 IEEE International Symposium on Multimedia, pp. 344–347. IEEE, New York (2012) 7. Pan, H., Li, B., Sezan, M.I.: Automatic detection of replay segments in broadcast sports programs by detection of logos in scene transitions. In: 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 4, pp. IV–3385. IEEE, New York (2002) 8. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015) 9. Sarkar, S., Chakrabarti, A., Mukherjee, P.D.: Generation of ball possession statistics in soccer using minimum-cost flow network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (2019) 10. Tabii, Y., Djibril, M.O., Hadi, Y., Thami, R.O.H.: A new method for video soccer shot classification. In: VISAPP, vol. 1, pp. 221–224 (2007) 11. Thomas, G., Gade, R., Moeslund, T.B., Carr, P., Hilton, A.: Computer vision for sports: current applications and research topics. Comput. Vis. Image Understanding 159, 3–18 (2017) 12. Wang, L., Zeng, B., Lin, S., Xu, G., Shum, H.Y.: Automatic extraction of semantic colors in sports video. In: 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 3, pp. iii–617. IEEE, New York (2004) 13. Xu, P., Xie, L., Chang, S.F., Divakaran, A., Vetro, A., Sun, H., et al.: Algorithms and system for segmentation and structure analysis in soccer video. In: ICME, vol. 1, pp. 928–931. Citeseer (2001) 14. Zhou, Y.H., Cao, Y.D., Zhang, L.F., Zhang, H.X.: An svm-based soccer video shot classification. In: 2005 International Conference on Machine Learning and Cybernetics, vol. 9, pp. 5398– 5403. IEEE, New York (2005)

A Novel Automated Blood Cell Counting Method Based on Deconvolution and Convolution and Its Application to Neural Networks Joydeep Chatterjee, Semanti Chakraborty and Kanik Palodhi

Abstract Blood cell counting and disease detection are very important in healthcare, biomedical research and pathology. Conventional cell counting techniques have problems of their own due to cost, complexity, skill requirement, and time consumption. Alternative image processing techniques are also challenging for huge computational load. Moreover, the modern image processing is quickly moving toward the neural network and machine learning-based smart systems rather than algorithms. They are, however, heavily resource-hungry and complex, essentially, games of humongous number crunching and monstrous computing workforce. The amount of data required to train the networks is also very large. Here, three differnt methods have been presented which provide relatively simple alternative to the above challgnes–A) blood cell counting using deconvolution–convolution algorithm, B) cell counting and disease detection using convolution and finally, C) blood cell counting using neural network aided by deconvolution–convolution method for clustering and classification. They are simpler, robust, faster and less resource-hungry as far as the requirement of the computational power is concerned. Keywords Blood cell · Deconvolution · Convolution · Neural networks · Clustering

1 Introduction Blood cell count is a major metric for health condition assessment. The primary blood cells are of the order of a few micrometers in size, and the world of blood cells J. Chatterjee · K. Palodhi (B) Department of Applied Optics and Photonics, University of Calcutta, Kolkata, India e-mail: [email protected] J. Chatterjee e-mail: [email protected] S. Chakraborty Department of Electronics and Photonics, Amity University, Kolkata, India e-mail: [email protected] © Springer Nature Singapore Pte Ltd. 2020 R. Chaki et al. (eds.), Advanced Computing and Systems for Security, Advances in Intelligent Systems and Computing 1136, https://doi.org/10.1007/978-981-15-2930-6_6

67

68

J. Chatterjee et al.

is quite rich in terms of diversity. It is needless to say that getting an accurate count of the blood cells is an important and challenging job. Generally applied methods such as flow cytometry and colony-forming unit are complicated and quite costly. Their operations not being user-friendly, skilled personnel are mandatory. To make the task of cell counting simpler and pocket-friendly, image processing-based cell counting plays a crucial role. Despite their significant contribution to this domain, they have some limitations as well. That is why modern image processing-based cell counting has ushered in the support of neural network and other machine learning techniques [1] where the computational system is trained with some sample data, and then it is used for the counting purpose. Again, these techniques are not perfect. In this proceeding, a useful image processing-based blood cell counting technique and its amalgamation with neural networks for further improvement purpose have been presented.

2 Challenges of Blood Cell Counting Over the years, blood cell counting technology has advanced significantly, yet there exist complexity and challenges of achievable accuracy. Optical microscope is one of the primitive tools still commonly used to visualize and quantify the blood cells. Blood slides are prepared using certain dyes such as Leishman stain and placed under the microscope to obtain cell count manually. It is necessary to employ a trained person for this manual count of number of cells scattered over certain zones of the slide. A specified grid pattern, marked on the slide, facilitates direct area and/or volumetric assessment. This is not only time consuming but susceptible to human errors also. As mentioned earlier methods like colony-forming unit and flow cytometry are difficult. In the case of colony-forming units, a number of colonies or chunks developed in certain media are determined. In flow cytometry, single cells are passed through the device and counted with the shining laser on the moving cells. To overcome the difficulties of many of the above-mentioned techniques, the help of several methods based on image processing is taken [2, 3]. Image processing is nothing but extraction or enhancement of the image data using some operations for quantitative evaluation maintaining the sanctity of the original data. Here, microscope images of the blood cells are subjected to operations such as Hough transform [4], watershed algorithm [5] and image segmentation [6] which require rigorous computational resources driven by exhaustive algorithms. On many occasions, the results contain significant degree of deviation from the desired data. Most of the above-mentioned algorithms are based on shape and threshold values. They might isolate the target from the background and noise signals, but generally they cannot determine the positions and count easily and simultaneously. Another problem is that they are unable to differentiate overlapped cells in the images which are a huge setback, and centroid determination is also challenging. Also, the computational

A Novel Automated Blood Cell Counting Method …

69

time and computational load are large. They are not suitable enough for real-time applications.

3 Proposed Methods 3.1 Deconvolution–Convolution Deconvolution is an important operation in the field of mathematics and engineering. In the case of image tailoring, it is used for cleansing, softening and deblurring pictures. With this operation, the edge information can also be improved to some extent. Mathematically, it is the reverse operation of convolution. Convolution basically represents the common area of the two participating patterns. The common area is maximum when the centers of the two patterns fall on each other. The more similar are the patterns, the higher is the peak. Keeping this in mind, the image is deconvolved with a Gaussian patch and then the result is convolved with a circular mask; sharp peaks centered at the centers of the blood cells are obtained and then they are isolated from the background with proper thresholding and counted using a simple blob detection algorithm, and the count is displayed. The main advantage is that this method can distinguish between overlapped cells. Centroids can also be determined easily. Mathematically, convolution can be stated as: ∞

a(x) ∗ b(x) = ∫ a(x1 )b(x − x1 )d x1 −∞

(1)

Typically, if q(x) is the result of convolution of a(x) and b(x), and Q(u), A(u) and B(u) are their Fourier transforms, respectively, then deconvolution is obtained by dividing Q(u) by B(u) and taking inverse Fourier transform of the result. Here, x and u are the coordinates in the object space and Fourier space, respectively. Some results using this method have been shown in Fig. 1. In Fig. 1, it can be clearly seen that blood cell—even some truncated ones—has been identified and counted. In Fig. 1c, overlapped cells have been distinguished.

3.2 Blood Cell Counting Using Convolution Method In another method, we have further simplified the algorithm. As it has been mentioned earlier, the convolution between two similar patterns gives a sharp peak at the central position. In this second method, we have made a circular mask and convolved it with the blood cell images. As a result, sharp peaks have been obtained at the positions of the centroids of the blood cells. The peaks have been isolated from the background by

70

J. Chatterjee et al. COUNT = 9

(a) 1 6 3

9

5 8 4

2 7 COUNT = 6

(b)

6 4

1

5 2 3 COUNT = 8

(c)

1 4 2 0 5 3

7

8 Fig. 1 Blood cell counting for scanning electron microscope (SEM) images using deconvolution– convolution algorithm

A Novel Automated Blood Cell Counting Method …

71

proper thresholding and have been identified and counted in a similar manner as the first method. The results have been shown in Fig. 2. In Fig. 2, it can be seen that the cells have been marked and counted very efficiently. In Fig. 2c, it can be seen that the overlapped cell has been identified as well. For the dark-field microscope images, annular circular ring-shaped mask has been used as the convolution mask. It is known that the sizes of the red blood cells (RBC) and WBC are different. The more is the match between the sizes of the cells and the mask, the higher is the convolution peak. With the help of this theory, in an image, the WBCs and the RBCs can be separately counted as shown in Fig. 3. In Fig. 3a, WBC count and in Fig. 3b, total cell count have been obtained. Hence, the RBC count is total count minus the WBC count (sizes of both the images are same).

3.3 Disease Detection Using Convolution Method Using this method, malaria and atypical lymphocytes have also been identified as shown in Fig. 4. Using a parasite or target as the mask in the convolution method, African sleeping sickness, bovine leukemia and leukemia in the SEM images have also been identified as presented in Fig. 5.

3.4 Blood Cell Counting Using Deconvolution–Convolution-Aided Neural Network-Based Method Although the above-mentioned two methods are giving good results, there is an inherent problem within them and that is the requirement of manual thresholding for images with different intensity values. This problem can be solved if neural network is incorporated in the processing. To have an understanding of the proposed neural network-aided deconvolution– convolution method, some basic knowledge on the neural network architecture is required. Some commonly used learning method has been shown in Table 1. Neural network is very similar to the nervous system of the human beings comprising of neurons, nodes and processing blocks. It is a simple machine learning technique. Human beings gather information from the outside world and it is carried to the brain through several nerves and neurons. After processing in the brain, response is visible to the world. Similarly, neural networks read the input data, carry it to the processing layers and display the response through the display block.

72 Fig. 2 Blood cell counting using circle convolution for a bright-field microscope image and b, c dark-field microscope images

J. Chatterjee et al.

(a)

NO. OF BLOOD CELLS = 8

(b)

NO. OF BLOOD CELLS = 12

(c)

A Novel Automated Blood Cell Counting Method …

73

Fig. 3 a WBC count and b total blood cell count obtained using convolution method [7]

The first operations of any neural network are classification and clustering of the input data. Depending upon certain logical conditions and data types, different types of data samples are divided into distinct groups. In the modern neural network algorithms, this is a huge challenge since large number of data input is required and humongous number crunching and iterative operations are to be performed for training the network. Here, in this third proposed method, this problem has been targeted. The deconvolution–convolution method has been used for this purpose to make life easier. This method drastically reduces the number of images to be used for the training purpose of the network. As shown earlier in this proceeding, the images are read, deconvolved with the Gaussian mask, convolved with the circular mask and the resulting intensity pattern is obtained. Then, the average values of the intensity patterns and the proper manual thresholding values are determined as the training parameters of the neural network, and the network is trained. The used neural network is a feedforward neural network with the forward data flow. This kind of network is not cyclic (Fig. 6). Levenberg-Marquardt optimization has also been used in the method with mean of squared error performance function. The Hessian matrix used is given by the

74 Fig. 4 a Malaria presence checking [8] and b atypical lymphocytes count using the convolution method [9]

J. Chatterjee et al.

(a)

Malaria Present

(b)

following equation:  −1 T J e ak+1 = ak − J T J + m I

(2)

Here, m is a scalar, J is Jacobian matrix containing first-order differentiation of the network demerit. For test images, using the deconvolution–convolution method, the intensity map is obtained and from the output of the trained neural network, the threshold value is

A Novel Automated Blood Cell Counting Method …

75 NO. OF BOVINE LEUKEMIA CELLS = 6

(a)

(c)

6

3

4

5

1 2 (d) (b)

Fig. 5 a, b African sleeping sickness detection [10], c bovine leukemia detection [11] and d leukemia detection in SEM image using the convolution method Table 1 Different learning techniques Name of the technique

Brief description

K-nearest neighbor (KNN) [12]

Weight assignment based on euclidean lengths among neighbors. Limitation is susceptibility to zonal settings. Very accurate and precise

Support vector machine (SVM) [13]

Kernel-based system. Recalling is good

Deep learning [14]

Several layers for information analysis. Require large data for training. Expert knowledge required

K-means clustering [14]

Iteration-based pixel operation for grouping

Naive Bayes classifier [14]

Formulated on Bayes theorem in probability

76

J. Chatterjee et al.

Fig. 6 Used neural network model

predicted; accordingly, the threshold is applied to obtain the bright intensity peaks at the positions of the cells and then they are counted. The results obtained have been shown in Fig. 7. Here, thirty images have been used for the training purpose. Blood cells have been quite accurately identified and counted in Fig. 7 without manual threshold COUNT = 3

(a) 1

2

3

COUNT = 6

(b)

6 4

1

5

2 3

Fig. 7 a, b Blood cell counting for SEM images using deconvolution–convolution-aided neural network method

A Novel Automated Blood Cell Counting Method …

77

manipulation. The time consumed for the entire process is very small (~6 s) even in ordinary computers. Again, blood cells for dark-field microscope images have been counted using the neural network aided by the deconvolution–convolution method with forty images being used for the training purpose. Here, an annular circular ring-shaped mask has been used as the convolution mask. The results have been shown in Fig. 8 (Table 2).

(b)

(a)

3 7 4

7

1 1

4 8

10 5

8

5 2

3

6

6 2

9

9

Fig. 8 a, b Blood cell counting for dark-field microscope images using deconvolution–convolutionaided neural network method

Table 2 Counting and disease detection results A. Cell count using deconvolution–convolution from images of SEM in Fig. 1, B. Cell count using convolution from images of bright-field microscope in Fig. 2(a), dark-field microscope in Fig. 2(b) and Fig. 2(c), and C. Cell count using deconvolution–convolution-aided neural network in Fig. 7 from SEM and Fig. 8 from dark-field

Disease detection from bright-field microscope images in Fig. 4 and Fig. 5 using convolution

All counted

All detected

78

J. Chatterjee et al.

4 Conclusions As shown in this paper, the deconvolution–convolution and convolution methods are very simple yet highly effective, especially for overlapped cell counting and disease detection. Using neural network and automatic thresholding based on pretrained data, the performance of the deconvolution–convolution method has been further enhanced. Acknowledgements The authors wish to thank University of Calcutta for lab facilities and darkfield RBC images. They also thank University Science Instrumentation Centre (USIC), Burdwan University and Dr. R. N. Dutta for SEM images. They also acknowledge The Internet Pathology Laboratory for Medical Education hosted by the University of Utah Eccles Health Sciences Library, Shutterstock, SSMJ and TVMDL for other images.

References 1. Chen, C.L., Mahjoubfar, A., Tai, L.C., Blaby, I.K., Huang, A., Niazi, K.R., Jalali, B.: Deep learning in label-free cell classification. Sci. Rep. 6, 1–16 (2016) 2. Theera-Umpon, N., Dhompongsa, S.: Morphological granulometric features of nucleus in automatic bone marrow white blood cell classification. IEEE Trans. Inf Technol. Biomed. 11, 353–359 (2007) 3. Ledwig, P., Sghayyer, M., Kurtzberg, J., Robles, F.E.: Dual-wavelength oblique back-illumination microscopy for the non-invasive imaging and quantification of blood in collection and storage bags. Biomed. Opt. Express. 9, 2743 (2018) 4. Maitra, M., Kumar Gupta, R., Mukherjee, M.: Detection and counting of red blood cells in blood cell images using hough transform. Int. J. Comput. Appl. 53, 13–17 (2012) 5. Tulsani, H., Saxena, S., Yadav, N.: Segmentation using morphological watershed transformation for counting blood cells. Int. J. C. Appl. Inf. Technol. 2, 28–36 (2013) 6. Safuan, S.N.M., Tomari, R., Zakaria, W.N.W., Othman, N.: White blood cell counting analysis of blood smear images using various segmentation strategies. In: AIP Conf. Proc. 1883 (2017) 7. The Internet Pathology Laboratory for Medical Education Hosted By The University of Utah Eccles Health Sciences Library: https://webpath.med.utah.edu/. Last accessed 24/08/2019 8. SSMJ: http://www.southsudanmedicaljournal.com. Last accessed 24/08/2019 9. Shutterstock: https://www.shutterstock.com/image-photo/atypical-lymphocyte-smear-denguefever-298524656. Last accessed 24/08/2019 10. Centers for Disease Control and Prevention’s Public Health Image Library CDC/Dr. Myron G. Sch. Identification number #613 (1970) 11. TVMDL: http://tvmdl.tamu.edu/2018/02/26/bovine-cbc-reveals-concurrent-blv-anaplasmosis/ blood-smear-depicting-concurrent-presence-of-bovine-leukemia-virus-blv-and-anaplasmosis/. Last accessed 24/08/2019 12. Sinha, P., Sinha, P.: Comparative study of chronic kidney disease prediction using KNN and SVM. Int. J. Eng. Res. V4 (2015) 13. Tai, W.L., Hu, R.M., Hsiao, H.C.W., Chen, R.M., Tsai, J.J.P.: Blood cell image classification based on hierarchical SVM. In: Proceedings of 2011 IEEE International Symposium on Multimedia, ISM 2011, pp. 129–136 (2011) 14. Poostchi, M., Silamut, K., Maude, R.J., Jaeger, S., Thoma, G.: Image analysis and machine learning for detecting malaria. Translational Res. 194, 36–55 (2018)

A Citizen-Centred Sentiment Analysis Towards India’s Critically Endangered Avian and Mammalian Species Inderjeet Singh Bamrah and Akshay Girdhar

Abstract Conservation Science (CS), nowadays, is a vital area for research and development because of its linkage with multiple domains. Sustainable goals as populated by the United Nations also emphasis on the need of the hour to protect biodiversity both on land and in water. In stimulating the research in this area, the role of citizen participation has evolved to a larger extent in this direction. Social media plays a significant role in providing citizen-centric data for analysing multifaceted problems. A lot of notable research in the past was carried out for determining illegal trade of animal species, threat and popularity assessment, animal identification in camera trap pictures, etc. using social media. However, analysis of people’s attitudes towards endangered species based upon different factors lacks in the number of studies and not has been covered in a well-versed manner. This research comprises the sentiment analysis of tweets concerned with the five most critically endangered Indian avian and mammalian Species. The study evaluates the variability in the sentiment scores and the intensity of the information shared for these different species using the Valence Aware Dictionary for sentiment reasoning algorithm. The lower number of tweets is due to less popularity of the species among citizens. However, the negative sentiments for even the most popular species signify the exasperation towards ineffectiveness of popular flagship programs. Keywords IUCN · VADER · Avian · Mammalian · Sentiment analysis · Citizen science · Conservation science

1 Introduction Anthropogenic activities in the past had led to cause a menace in destroying nature and its natural resources [1]. Comprehending these activities in a lucid manner is needed for developing crucial conservation solutions. Historically, only natI. S. Bamrah (B) · A. Girdhar I.K.G. Punjab Technical University, Kapurthala, India e-mail: [email protected] © Springer Nature Singapore Pte Ltd. 2020 R. Chaki et al. (eds.), Advanced Computing and Systems for Security, Advances in Intelligent Systems and Computing 1136, https://doi.org/10.1007/978-981-15-2930-6_7

79

80

I. S. Bamrah and A. Girdhar

ural science has contributed as a fundamental source of information for studying conservation action [2]. Data collection through field visits for analysis of different human–nature interactions became time-consuming and requires more effort than collecting digital information [3]. The current era is characterised by the presence of voluminous data generated online. Big data provides new research avenues in all the possible fields and social media acted as a primary source in generating the data needed for such researches. Social media from the past few years has proved to be a significant development for sharing and communicating information on multiple problems through large and diverse groups and communities. In this research article, we focused primarily on the role of social media in analysing the sentiment of common mass towards different critically endangered (CR) species of the Indian subcontinent. The quantitative analysis of the tweets uncovers the level of awareness towards these species. It explores the intensity of the popularity of animal species under the CR threat level. And finally, the study testifies the hypothesis which looks for a significant difference in the level of awareness for large- and small-sized species. Information collected from different social media applications serves as essential raw material for conducting several indispensable studies [4]. It served in the past to determine the illegal trading activities of endangered species [5]. Study of visitor’s pattern and movement in and out of parks, biodiversity area helped in determining the threat levels. Also, it played a major role in the analysis of geotagged sentiments of common mass towards specific parks [6]. A sharp rise in contribution to social media through citizen science improved the efficacy of a wide number of web applications working towards conservation of the endangered fauna and flora [7]. Citizen science data is capable of providing information related to conservation efforts through elaborated perspicacity into factors like population trends, migratory patterns and trends of species concerned with the need for potential conservation measures [8, 9]. Machine learning algorithms combined with the cloud platforms resulted in the development of several citizen-centric web applications. It all started from the Global Biodiversity Information Facility (GBIF) back in 1992. After which a lot of other significant web applications like Ocean Biogeographic Information System (OBIS), eBird, iNaturalist were designed and developed [10]. Although, these systems provide a rich and multimodal source of information yet it limits only to specific species/taxa of animal classification. Dynamic Ecological Information Management System-Site and Dataset Registry (DEIMS-SDR) is an attempt towards maintaining in situ observations related to field study and site observations. It comprises of a plethora of information in terms of the site location, ecosystems, etc [11]. However, when it comes to the formulation of policies concerning conservation action, public sentiments that emerged from citizen science data of textual form plays a significant role in elucidating the requirements. A study conducted on the sentiment analysis of the urban population towards green space through twitter data provided vital findings. These findings could be used to formulate policies catering to the desired need of natives [12]. Aesthetic benefit defines “appreciation of natural scenery” [13]. And these benefits serve as the inspiration source for initiating the conservation efforts [14]. The magni-

A Citizen-Centred Sentiment Analysis Towards …

81

tude with which the population communicates their feelings and emotions towards a natural site helps in inculcating desired proactive actions and formulation of policies for conservation. The study evaluated the expression of people in photographs taken in nature-connected areas along with the processing of textual information [15]. This article too focuses on an assessment of public emotions towards CR species of the Indian subcontinent through tweets. Multimodality offers a larger scope of assessment of services concerned with the cultural ecosystem. A recent study performed such an evaluation based on georeferenced pictures related to Brazil’s largest marine protected region [16]. Georeferencing helps a lot in understanding the attitude which changes significantly on the change of the region. The proposed research emphasised the sentiment evaluation with georeferencing as one of the vital criteria.

2 Brief Literature Study We conducted a well-organised literature review to understand the way online social media has emerged as one of the prominent sources of citizen-centric information. We also analysed how social media contributed to conservation culturomics. Identification of applications of conservation culturomics like expressing public engagement with different parts of the biotic ecosystem, novel metrics for an almost real-time environmental surveillance, assessment of the cultural influence of conservation arbitrations, etc. [17] paved the way for performing sentimental analysis through tweets for the assessment of public engagement towards critically endangered species of Indian subcontinent. Social media is one of the indispensable sources which contributed significantly to asses the conservation culturomics. Its extensive use in diverse fields proves its popularity. However, many of the research areas are still untapped. Minin et al. [4] delineated the way social media could be beneficial for the assessment of different dimensions of conservation science. Sentiment analysis is one such dimension, which helps to harness the true potential of citizen-centric information. Because of the timeconsuming nature of conventional procedures for data collection, social media offers a viable solution for the problem. Many recent studies also conducted in the direction of analysing the conservation status of the subject through sentiment analysis. Do [15] in his studies, evaluated the public attitude towards the national park for aesthetic interests. The researcher utilised images along with concise texts and search inclinations for study. Facial expressions assimilated in pictures, and sentiment analysis from textual data from multiple sources helped to accomplish the coveted aspirations. A surge in search amount depicted the enhanced public interests. The investigation symbolises the strength of the Internet to increase inferred aesthetic opinion based on varied spatial and ephemeral scales. Plunz et al. [6] also conducted a study to assess the correlation between wellbeing and urban green locality. The authors performed the study by selecting two parks in New York City. Sentiment analysis of the tweets with geotagging helped to

82

I. S. Bamrah and A. Girdhar

differentiate the scores of the public. The sentiment of the people who communicated inside the park turned out to be more positive than the other case. The research proposed a future direction where the analysis concerning a specific person, group could be significant. The proposed analysis in this research work caters to two specific critically endangered species. Roberts et al. [12] also analysed around over 10,000 tweets for the computation of positive sentiment and negative sentiment in order to interpret human well-being. Study focused on the need of such analysis in order to help different stakeholders in providing better living experience. Tourism is one of the areas which has a significant impact on foreseeing the future of conservation science. Such an impact further stimulates when it comes to visiting iconic places like the Great Barrier reef. In a research study by Curnock et al. [18], a survey among 4681 Australians was conducted to assess the sentiment towards the Great Barrier reef. The study presented the ability of iconic locations to vitalise the procedures being followed for conservation science. The study indicates the application of citizen science for sentiment analysis towards concerns related to the conservation of different iconic locations. The research findings can also be extended to analyse the interest of common mass towards different species. Although computational sentiment analysis-based researches are prevalent, Jost et al. [19] performed non-computational-based sentiment analysis of the word ’change’ for manifold areas concerning climate change. Authors analysed the results after convoying interviews with local government officials in eleven diverse associations. The research was destined to evaluate the absence of consensus among policy-makers and climate researchers over the interpretation of change. Several metrics were recognised as portraying positive and negative sentiments. As a scope of further enhancement, the study remarks the further addition of improved exchange of information among staff and local administration nurturing the knowledge and intensifying climate change policy. Along with the investigation of habitat and climate change for sentiment, the researches related to sentiment towards different endangered species paved the way. However, the studies were performed to assess either specific biomes or analysing threats associated with specific taxonomy only [20]. Meticulous use of different online sources for studying the pattern of human visitation to natural parks and other natural habitats by Hausmann et al. [20], and thus provided suggestions for the areas affected with high visitation risks. Research indicated the scope of sentiment analysis for a better understanding of citizen engagement. Researches also emphasised the effectiveness of sentiment scores towards the formulation of improved conservation science strategies [21]. Social media also provides the opportunity to study any illegal trade of exotic species. The richness of social media in multimodal information further stimulates the process like image recognition by analysing multiple contours and patterns involved can help to predict the type of crime involved. Different classifiers that help perform predictions can be improved with the help of many pre-existing repositories. However, the researches always emphasised the limitation in accessing social media data and ethical issues involved in following privacy terms [22].

A Citizen-Centred Sentiment Analysis Towards …

83

Variability in terms of attention given to different species is evident from the detailed information in the red list published by the International Union for Conservation of Nature and Natural resource (IUCN). As a consequence of which, Kidd et al. [23] try to assess this variability by analysing the citizen-centric engagement for different species in popular social media platforms. Towards the concluding part, authors shared five common characteristics shared by threatened species to raise the awareness of less popular species. Several significant research has also utilised the advanced machine learning approaches to cater to multiple issues in conservation science. Willi et al. [24] deployed a convolutional neural network-based classifier to recognise the animals in camera trap pictures. However, researchers elucidated the limitation of the study in terms of the inability to recognise rare species and the presence of non-trained image types. Conservation science is largely hindered by these illegal trade activities of animals. Such an analysis of illegal trade practices against Asian Otter is performed by Siriwat and Nijman in their research. The analysis included the monitoring of five Facebook group for social network linkages concerning illicit trade practices. Findings attribute to the weakened law and enforcement to combat such practices. As a consequence of which, poachers and other traders openly interact on the web and does not even require access to the dark web [5]. Several research studies as mentioned emphasised on the volatility in the data and the need to align it with citizen science [25, 26]. Social media has now become a dense repository of data, the true potential of which can be harnessed using different computational techniques. The use of citizen science in assessing the conservation efforts, species prevalence and other notable areas of conservation science saw a sudden surge in the recent research studies. Citizen science through popular social media platforms like Twitter has contributed significantly to evaluate public engagement with different animal species. This research study presents an analysis of public sentiments towards two critically endangered species.

3 Materials and Methods 3.1 Area of Interest India being in the tropical zone, boasts of its rich endemism for both flora and fauna. It hosts the major four biodiversity hotspots namely Western Ghats, the Himalayas, the Indo-Burma region and Sundaland [27] The study involves five most critically endangered Indian avian and mammalian species. The number of mature individuals in the case of each species is lower than 500 [28–37]. It includes Great Indian Bustard (Ardeotis nigriceps) [28], Jerdon’s Courser (Rhinoptilus bitorquatus) [29], Whitebellied Heron (Ardea insignis) [30], Himalayan Quail (Ophrysia superciliosa) [31] and Pink-headed Duck (Rhodonessa caryophyllacea) [32] from avian taxanomy. In mammalian taxanomy, the study incorporated the sentiment analysis towards

84

I. S. Bamrah and A. Girdhar

species, viz. Pygmy hog(Porcula salvania) [33], Andaman white-toothed shrew (Crocidura andamanensis) [34], Namdapha flying squirrel (Biswamoyopterus biswasi) [35], Malabar civet (Viverra civettina) [36] and Sumatran rhinoceros (Dicerorhinus sumatrensis) [37]. As rightly mentioned in a study by Roberge, gaining insights about public consciousness towards animal species is critical for conservation endeavours [38]. The number of messages exchanged on social media helps to examine the level of awareness the common mass contains and ultimately attributes to the effectiveness of citizen science data for conservation science. These species in the past showed a rapid decline in their adult population. Also, studies show evidence of biased engagement on social media by citizens towards large-sized species [38]. It is thus, the presented study is motivated towards a critical evaluation of bias using sentiment analysis of messages exchanged by common people over Twitter.

3.2 Twitter Data Retrieval Twitter is a microblogging platform and one of the dominant social media platforms that is capable of posting the short text. Twitter data can be accessed by the application programming interface (API) provided by twitter itself. This platform is growing rapidly and has an average generation of around 1 billion tweets every 3 days [39]. Python is found to be one of the widely acceptable languages when it comes to data analysis and it has allied areas like sentiment analysis. A collection of a large number of pre-built libraries helps it to reduce the cumbersomeness involved in complex computations [40]. Several libraries, like tweepy, python-twitter, etc., exist for Python language. However, all of them suffer from the basic limitation of the number of tweets accessible at a time. To counter this limitation, we used a web scrapping library named twint (https://github.com/twintproject/twint), which works in the same manner as a normal user would have accessed the twitter home page by scrolling. It is available as a complete package and can be installed using pip package installer of Python. Tweets can be accessed using a simple command from the command terminal. For conducting research work, we accessed the tweets with the general or local names of species as in its search criteria. The command looks like below which accessed the data through scrolling mechanism and saved it into a comma-separated version file. $ twint -s "species name" -o filename.csv --csv

Extracted data is then put forth to data cleansing procedure before it was used for sentiment analysis.

A Citizen-Centred Sentiment Analysis Towards …

85

3.3 Data Cleansing and Categorisation Tweets retrieved from twitter contains a lot of unnecessary information. It needs to be removed before it can be fed to a sentiment analyser. It might consist of direct linkage to the user profile using @ symbol, weblinks, re-tweet symbol and any special characters except hash, which in some cases represents important hashtags. To accomplish it, a Python library named NumPy (https://www.numpy.org/) along with a regular expression based library was used for removing the ineffectual information. Interpretation of the sentiment score was carried out on two distinct taxonomies namely, mammalian and avian. Also, the scores were analysed within the taxonomy for the anomalies observed within the taxonomy among different species.

3.4 Sentiment Analysis Sentiment analysis provides an overall picture of the attitude of citizen-centric data towards a subject. Several types of research attribute to the inclusion of computational aspects in determining the perception of common mass towards multiple aspects of conservation. Whether it is about the assessment of threats concerning the vital and significant bird species [20] or evaluating sentiment analysis as a wholistic measure in conservation culturomics [21], social media-based citizen-centric data has become a popular choice of researchers in studying, evaluating contemporary issues like biodiversity conservation.

Fig. 1 Flow diagram of sentiment analysis using VADER sentiment analyser

Data Extraction based on local name of species from Twitter using Twint Library

Reading CSV and removing redundancy

Data Cleaning Using NumPy and Pandas

VADER

Performing Sentiment Analysis using Valence Aware Dictionary for Sentiment Reasoning

Visualising the sentiment scores and Interpreting the results

86

I. S. Bamrah and A. Girdhar

Table 1 Summary of tweets depicting citizen engagement with different species Tweets summary No. of tweets Percentage of tweets Percentage of (within taxonomy) (%) tweets (both the taxonomies) (%) Great Indian bustard (Ardeotis 4961 nigriceps) Jerdon’s courser (Rhinoptilus 194 bitorquatus) White-bellied heron (Ardea 444 insignis) Himalayan quail (Ophrysia 235 superciliosa) Pink-headed duck (Rhodonessa 560 caryophyllacea) Pygmy hog (Porcula salvania) 931 Andaman white-toothed shrew 6 (Crocidura andamanensis) Namdapha flying squirrel 164 (Biswamoyopterus biswasi) Malabar civet (Viverra civettina) 207 Sumatran rhinoceros 783 (Dicerorhinus sumatrensis)

77.59

58.47

3.03

2.29

6.94

5.23

3.67

2.77

8.76

6.60

44.52 0.29

10.97 0.07

7.84

1.93

9.90 37.44

2.44 9.23

The majority of the researches relied upon a pre-built corpus when it comes to sentiment analysis. However, there was no such corpus available with any pre-assigned sentiment for training purposes. As a consequence of which, using a conventional sentiment analyser was not on the list. VADER which is an abbreviation for Valence Aware Dictionary for Sentiment Reasoning is a simple rule-based sentiment analyser. It is known for its computational accuracy without compromising the resource usage. It is also self-sustaining and agnostic and does not demand any comprehensive training dataset [41]. The ability of this analyser to measure the intensity of sentiment also makes it unprecedented. For example, sentiment represented by a statement “I love this species.” and through the statement “I love this species!” differs significantly due to the intensity involved by addition of exclamation sign. The scores get intensified on the addition of every extra exclamation mark. Figure 1 represents the workflow of the analysis.

A Citizen-Centred Sentiment Analysis Towards …

87

Table 2 Summary of polarity of tweets for different species Opinion Polarisation using VADER sentiment analyser Positive Negative Tweets with no polarised tweets polarised tweets sentiments Great Indian bustard (Ardeotis 4406 nigriceps) Jerdon’s courser (Rhinoptilus 44 bitorquatus) White-bellied heron (Ardea insignis) 191 Himalayan quail (Ophrysia 40 superciliosa) Pink-headed duck (Rhodonessa 122 caryophyllacea) Pygmy hog (Porcula salvania) 347 Andaman white-toothed shrew 1 (Crocidura andamanensis) Namdapha flying squirrel 51 (Biswamoyopterus biswasi) Malabar civet (Viverra civettina) 49 Sumatran rhinoceros (Dicerorhinus 168 sumatrensis)

275

280

23

127

51 34

202 161

110

328

102 5

482 0

18

95

14 267

144 348

4 Results 4.1 Twitter Data Summary The local name of species was used as a keyword to extract the data from twitter. A total of 8485 distinct tweets were extracted for both the taxonomy. Out of which around 6394 tweets belong to avian taxonomy and the rest 2091 to mammalian classification. Table 1 presents the share of tweets among different species of two taxonomies. Great Indian bustard shares a maximum number of tweets than other species within the same taxonomy and also among other species that belong to mammal taxonomy. This portrays the high awareness of citizens towards the species. On the contrary to this, Andaman white-toothed shrew is less talked about species and this is backed by the lowest share of tweets among all of the species. Within the avian taxonomy, it is Jerdon’s courser with the lowest share of tweets among other species. This, however, might be attributed to the low awareness among citizens due to the absence of any prevalent systemised monitoring scheme [29]. Similar is the case within the mammalian taxonomy in which the Andaman white-toothed shrew receives the lowest attention in terms of tweets. This again is due to the low recognition of the species due to the inadequacy of any flagship programme related to resource and habitat conservation [34].

88

I. S. Bamrah and A. Girdhar

4.2 Analysing Sentiment Polarity The sentiment scores of tweets were computed using Valence Aware Dictionary for Sentiment Reasoning (VADER). VADER in addition to positive and negative scores computes compound score which is computed based on analysis of both negative and positive polarities. The baseline for the compound score was set to 0.05 for classifying a tweet to either positive or negative polarity. The tweets with a compound score of more than or equal to 0.05 were classified under category of positive sentiments, tweets with compound score less than 0.05 and more than −0.05 were categorised as neutral and contains no significant emotions attached. And the tweets with the score below −0.05 were classified under category of negative sentiments. Table 2 is in alliance with the number of tweets percentage as shown in Table 1. Positive sentiments in case of flagship species like Great Indian bustard is due to the administrative interventions in drafting and implementing various protectionist approaches [42]. However, the outweighing negative sentiments by citizens even in the case of flagship species Great Indian bustard indicates the non-acceptability of the rate at which conservation efforts are materialised.

4.3 Analysing Variablity in Sentiment Scores Variability in the sentiment scores is shown for avian and mammalian species in Figs. 2 and 3 respectively. Dense plot for the Great Indian Bustard species and sparse plot for Jerdon’s Courser as evident from Fig. 2 is due to the variability in the level of consciousness amidst citizens. Government initiatives towards a species aid to elevate the level of awareness among the citizens. Also, Figure 3 shows the dense plot for Pygmy hog, a species the information about which is widely circulated in

A Citizen-Centred Sentiment Analysis Towards …

89

(a) Great Indian Bustard(Ardeotis nigriceps)

(b) Jerdon’s Courser(Rhinoptilus bitorquatus)

(c) White-bellied Heron(Ardea insignis)

(d) Himalayan Quail(Ophrysia superciliosa)

(e) Pink-headed Duck(Rhodonessa caryophyllacea)

Fig. 2 Sentiment scores of tweets searched against five Avian species

terms of conservation efforts as compared to other critically endangered species. However, other species like Sumatran Rhinoceros present a dense plot which might be attributed to the large size of the species as documented by one of the earlier studies [38].

5 Conclusion and Future Scope The analysis of data from social media and other web resources proven to be useful in studying human interaction among the multiple arenas of environment and

90

I. S. Bamrah and A. Girdhar

(a) Pygmy Hog(Porcula salvania)

(c) Namdapha Flying Squirrel(Biswamoyopterus biswasi)

(b) Andaman White-toothed Shrew(Crocidura andamanensis)

(d) Malabar Civet(Viverra civettina)

(e) Sumatran Rhino-ceros (Dicerorhinus sumatrensis)

Fig. 3 Sentiment scores of tweets searched against five Mammalian species

biodiversity conservation. Various studies revealed that it had proved to be of significant importance in the analysis of perceptions of common citizens towards the environment, environmental monitoring and governance [43]. Sentiment analysis aids the process of interpreting human interaction with the environment and biodiversity. Social media campaigns and other non-governmental movements foster the growth of awareness levels among citizens. However, the administration plays a significant role in this and the efforts put forth by them stimulate and influence the sentiments which citizens express on different social media websites. For example, popular species like tiger and elephant for which the government

A Citizen-Centred Sentiment Analysis Towards …

91

announces flagship programmes saw a surge in relevant social media posts shared and created by the common public. On the contrary, species which are either small-sized or less popularised by the government or other administrative machinery tend to be less popular among citizens, also. This study of sentiment analysis conducted for two different critically endangered taxonomies indicates the difference in terms of awareness shared among the public. The avian species which comprises of flagship species great Indian bustard holds the highest number of tweets for it as compared to other less popular species. However, size also plays a vital role and a larger share of tweets for species like Sumatran rhinoceros among critically endangered mammal species shows its popularity. This kind of analysis can be useful for administrative machinery to study which species need promotion among the common mass and needs more sensitisation. In spite of the usefulness of the study, it attributes to certain limitations also. The twitter data contributes towards a portion of large-sized social media, and many other sources can be used to improve the results in a significant way. Also, government policies can be analysed for the effectiveness and sentiments towards these policies provide better evaluation for the efficacy of the policies. This study does not take into account the sarcastic nature of tweets, which can be a tricky part to handle and requires much denser lexicon resources and evaluation to study the sarcastic tone involved in tweets.

References 1. Maxwell, S.L., Fuller, R.A., Brooks, T.M., Watson, J.E.M.: Biodiversity: the ravages of guns, nets and bulldozers. Nature 536(7615), 143–145 (2016) 2. Bennett, N.J., Roth, R., Klain, S.C., Chan, K., Christie, P., Clark, D.A., Cullman, G., Curran, D., Durbin, T.J., Epstein, G., Greenberg, A., Nelson, M.P., Sandlos, J., Stedman, R., Teel, T.L., Thomas, R., Veríssimo, D., Wyborn, C.: Conservation social science: understanding and integrating human dimensions to improve conservation. Biol. Conserv. 205, 93–108 (2017) 3. Waldron, A., Mooers, A.O., Miller, D.C., Nibbelink, N., Redding, D., Kuhn, T.S., Roberts, J.T., Gittleman, J.L.: Targeting global conservation funding to limit immediate biodiversity declines. Proc. Nat. Acad. Sci. 110(29), 12144–12148 (2013) 4. Toivonen, T., Heikinheimo, V., Fink, C., Hausmann, A., Hiippala, T., Järv, O., Tenkanen, H., Minin, E.D.: Social media data for conservation science: a methodological overview. Biol. Conserv. 233, 298–315 (2019) 5. Siriwat, P., Nijman, V.: Illegal pet trade on social media as an emerging impediment to the conservation of Asian otters species. J. Asia-Pacific Biodivers. 11(4), 469–475 (2018) 6. Plunz, R.A., Zhou, Y., Vintimilla, M.I.C., Mckeown, K., Yu, T., Uguccioni, L., Sutto, M.P.: Twitter sentiment in New York City parks as measure of well-being. Landscape Urban Plan. 189, 235–246 (2019) 7. Humphries, G.R.W.: How the Internet Can Know What You Want Before You Do: Web-Based Machine Learning Applications for Wildlife Management, pp. 335–351. Springer International Publishing, Cham (2018) 8. Sullivan, B.L., Phillips, T., Dayer, A.A., Wood, C.L., Farnsworth, A., Iliff, M.J., Davies, I.J., Wiggins, A., Fink, D., Hochachka, W.M., Rodewald, A.D., Rosenberg, K.V., Bonney, R., Kelling, S.: Using open access observational data for conservation action: a case study for birds. Biol. Conserv. 208, 5–14 (2017)

92

I. S. Bamrah and A. Girdhar

9. Horns, J.J., Adler, F.R., Ça˘gan, H.: Sekercio˘ ¸ glu: Using opportunistic citizen science data to estimate avian population trends. Biol. Conserv. 221, 151–159 (2018) 10. Humphries, G.R.W., Huettmann, F.: Machine Learning and ‘The Cloud’ for Natural Resource Applications: Autonomous Online Robots Driving Sustainable Conservation Management Worldwide? pp. 353–377. Springer International Publishing, Cham (2018) 11. Wohner, C., Peterseil, J., Poursanidis, D., Kliment, T., Wilson, M., Mirtl, M., Chrysoulakis, N.: DEIMS-SDR—a web portal to document research sites and their associated data. Ecol. Inform. 51, 15–24 (2019) 12. Roberts, H., Sadler, J., Chapman, L.: The value of twitter data for determining the emotional responses of people to urban green spaces: a case study and critical evaluation. Urban Stud. 56(4), 818–835 (2019) 13. de Groot, R., Alkemade, R., Braat, L., Hein, L., Willemen, L.: Challenges in integrating the concept of ecosystem services and values in landscape planning, management and decision making. Ecol. Complexity 7(3), 260 – 272 (2010), (Ecosystem Services – Bridging Ecology, Economy and Social Sciences) 14. Pearson, R.G.: Reasons to conserve nature. Trends Ecol. Evol. 31(5), 366–371 (2016) 15. Do, Y.: Valuating aesthetic benefits of cultural ecosystem services using conservation culturomics. Ecosyst. Serv. 36, 1–5 (2019) 16. Retka, J., Jepson, P., Ladle, R.J., Malhado, A.C., Vieira, F.A., Normande, I.C., Souza, C.N., Bragagnolo, C., Correia, R.A.: Assessing cultural ecosystem services of a large marine protected area through social media photographs. Ocean Coastal Manage. 176, 40–48 (2019) 17. Ladle, R.J., Correia, R.A., Do, Y., Joo, G.J., Malhado, A.C., Proulx, R., Roberge, J.M., Jepson, P.: Conservation culturomics. Front. Ecol. Environ. 14(5), 269–275 (2016) 18. Curnock, M.I., Marshall, N.A., Thiault, L., Heron, S.F., Hoey, J., Williams, G., Taylor, B., Pert, P.L., Goldberg, J.: Shifts in tourists’ sentiments and climate risk perceptions following mass coral bleaching of the great barrier reef. Nat. Clim. Change 9(7), 535–541 (2019) 19. Jost, F., Dale, A., Schwebel, S.: How positive is “change” in climate change? a sentiment analysis. Environ. Sci. Policy 96, 27–36 (2019) 20. Hausmann, A., Toivonen, T., Fink, C., Heikinheimo, V., Tenkanen, H., Butchart, S.H., Brooks, T.M., Minin, E.D.: Assessing global popularity and threats to important bird and biodiversity areas using social media data. Sci. Total Environ. 683, 617–623 (2019) 21. Lennox, R.J., Veríssimo, D., Twardek, W.M., Davis, C.R., Jari´c, I.: Sentiment analysis as a measure of conservation culture in scientific literature (2019) 22. Di Minin, E., Fink, C., Hiippala, T., Tenkanen, H.: A framework for investigating illegal wildlife trade on social media with machine learning. Conserv. Biol. 33(1), 210–213 (2019) 23. Kidd, L.R., Gregg, E.A., Bekessy, S.A., Robinson, J.A., Garrard, G.E.: Tweeting for their lives: visibility of threatened species on twitter. J. Nat. Convers. 46, 106–109 (2018) 24. Willi, M., Pitman, R.T., Cardoso, A.W., Locke, C., Swanson, A., Boyer, A., Veldthuis, M., Fortson, L.: Identifying animal species in camera trap images using deep learning and citizen science. Methods Ecol. Evol. 10(1), 80–91 (2018) 25. Rambonnet, L., Vink, S.C., Land-Zandstra, A.M., Bosker, T.: Making citizen science count: best practices and challenges of citizen science projects on plastics in aquatic environments. Mar. Pollut. Bull. 145, 271–277 (2019) 26. Xue, Y., Gomes, C.P.: Artificial Intelligence and Conservation, Engaging Citizen Scientists in Data Collection for Conservation, pp. 194–209. Cambridge University Press, Cambridge (2019) 27. Global biodiversity hotspots with special emphasis on Indian hotspots (2016) http://www. bsienvis.nic.in/Database/Biodiversity-Hotspots-in-India_20500.aspx 28. BirdLife International: Ardeotis nigriceps. The IUCN Red List of Threatened Species (2018). http://doi.org/10.2305/IUCN.UK.2018-2.RLTS.T22691932A134188105.en 29. BirdLife International: Rhinoptilus bitorquatus (amended version of 2016 assessment). The IUCN Red List of Threatened Species 2017 (2017). http://doi.org/10.2305/IUCN.UK.2017-3. RLTS.T22694103A117189206.en

A Citizen-Centred Sentiment Analysis Towards …

93

30. BirdLife International: Ardea insignis. The IUCN Red List of Threatened Species 2018 (2018). http://doi.org/10.2305/IUCN.UK.2018-2.RLTS.T22697021A134201407.en 31. BirdLife International: Ophrysia superciliosa. The IUCN Red List of Threatened Species 2018 (2018). http://doi.org/10.2305/IUCN.UK.2018-2.RLTS.T22679141A132051220.en 32. BirdLife International: Rhodonessa caryophyllacea (amended version of 2016 assessment). The IUCN Red List of Threatened Species 2018 (2018). http://doi.org/10.2305/IUCN.UK. 2016-3.RLTS.T22680344A125558688.en 33. Narayan, G., Deka, P., Oliver, W.: Porcula salvania. The IUCN Red List of Threatened Species 2008 (2008). http://doi.org/10.2305/IUCN.UK.2008.RLTS.T21172A9254675.en 34. Molur, S.: Crocidura andamanensis (errata version published in 2017). The IUCN Red List of Threatened Species 2016 (2016). http://doi.org/10.2305/IUCN.UK.2016-3.RLTS. T5595A22303530.en 35. Molur, S.: Biswamoyopterus biswasi (errata version published in 2017). The IUCN Red List of Threatened Species 2016 (2018). http://doi.org/10.2305/IUCN.UK.2016-3.RLTS. T2816A22271554.en 36. Mudappa, D., Helgen, K., Nandini, R.: Viverra civettina. The IUCN Red List of Threatened Species 2016 (2016). http://doi.org/10.2305/IUCN.UK.2016-1.RLTS.T23036A45202281.en 37. van Strien, N.J., Manullang, B., Sectionov, Isnan, W., Khan, M.K.M, Sumardja, E., Ellis, S., Han, K.H., Boeadi, Payne, J., Bradley Martin, E.: Dicerorhinus sumatrensis. The IUCN Red List of Threatened Species 2008 (2008). http://doi.org/10.2305/IUCN.UK.2008.RLTS. T6553A12787457.en 38. Roberge, J.M.: Using data from online social networks in conservation science: which species engage people the most on twitter? Biodivers. Conserv. 23(3), 715–726 (2014). Mar 39. Twitter: Twitter turns six (Mar 2012). http://blog.twitter.com/2012/03/twitter-turns-six.html 40. Rathee, N., Joshi, N., Kaur, J.: Sentiment analysis using machine learning techniques on python. In: 2018 Second International Conference on Intelligent Computing and Control Systems (ICICCS), pp. 779–785 (June 2018) 41. Hutto, C., Gilbert, E.: Vader: A parsimonious rule-based model for sentiment analysis of social media text. In: International AAAI Conference on Web and Social Media, pp. 216–225 (2014) 42. Khan, S., Chatterjee, N., Habib, B.: Testing performance of large-scale surveys in determining trends for the critically endangered Great Indian Bustard Ardeotis nigriceps. Sci. Rep. 9, 1–8 (2019) 43. Ghermandi, A., Sinclair, M.: Passive crowdsourcing of social media in environmental research: a systematic map. Glob. Environ. Change 55, 36–47 (2019)

Methodology for Generating Synthetic Time-Dependant Probabilistic Speed Profiles Lukáš Rapant, Daniela Szturcová, Martin Golasowski and David Vojtek

Abstract Traffic flow management of smart city is one of the most current topics in traffic modeling. We have developed a method based on traffic routing and reordering that is capable of performing this task. One of the inputs of this method is time-dependant probabilistic speed profile, i.e., speed profiles that take into account both the time and uncertainty of traffic speed due to various traffic events and peaks. However, the exact calculation of these profiles for each road is very difficult due to the huge amounts of real-world data required. Therefore, we propose a methodology, which should, by utilizing various available metadata about traffic network and Markov chain model, be capable of producing these probabilistic speed profiles synthetically. Keywords Synthetic traffic data · Markov chains · Probabilistic speed profiles

1 Introduction and Motivation One of the most challenging tasks of every intelligent traffic system (ITS) is the traffic flow management of a smart city. This management system can, to a certain degree, govern the route of individual cars in such a way, that resulting traffic is faster and more smooth. During our research in this area, we have developed an approach based on the utilization of multiple dynamic routing (more thorough description of our model that can be found in [9]. At the heart of our approach is the routing of a single car. This routing, if properly implemented, can improve the travel time of a given car by taking into account both the length of the path and dynamically changing traffic conditions. These traffic conditions can be derived from the data fusion of available data sources and, in a certain view, more importantly, historical data, i.e., how the traffic behaved at the same place in time in the past. The first part of this routing has L. Rapant (B) · D. Szturcová · M. Golasowski · D. Vojtek IT4Innovations, VŠB—Technical University of Ostrava, 17. listopadu 15/2172, 70833 Ostrava, Czech Republic e-mail: [email protected] © Springer Nature Singapore Pte Ltd. 2020 R. Chaki et al. (eds.), Advanced Computing and Systems for Security, Advances in Intelligent Systems and Computing 1136, https://doi.org/10.1007/978-981-15-2930-6_8

95

96

L. Rapant et al.

already been thoroughly researched, and several algorithms such as Dijkstra or A* have already been proposed. To capture the influence of dynamically changing traffic conditions, we utilized the probabilistic approach presented in our previous papers [10, 17] called probabilistic time-dependent routing algorithm (PTDR). This routing is then applied on a k alternative shortest routes and produces several probability distributions describing their duration. The last part of the proposed approach which follows after this computation is called reordering. This step allows us to extend the routing optimization to optimize all cars in the traffic network. It is realized by introducing one more element into the reordering calculation. This element is based on the number of cars that are traversing or are going to traverse, parts of a given route at the same time as the optimized car. This addition into the reordering step allows the individual cars to take into consideration the paths of other cars and chose the preferred road accordingly (possibly not the optimal one based on both usual traffic conditions and its length). One of the most important parts of this approach is PTDR. The PTDR is based on the computation of travel times and their probabilities on the selected path. Let G = {V, E} be a directed graph created from a road network, where vertices represent intersections and edges represent road segments. Our selected path starts at a vertex u and ends at a vertex v. We choose departure time τ0 and compute time-dependent shortest path duration with speed profiles of individual road segments serving as edge weights to obtain optimal path Po = {u = v0 , e0 , v1 , e1 , . . . , vn−1 , en−1 , vn = v} by travel time for τ0 , let us denote travel time of this path as t. Speed profiles are tables showing how the average speed on the segments develops during a week in each time interval. For example, speed profile can be seen in Table 1. However, if we ended here, the value of the resulting route would only depend on the time of departure and would not take into account a lot of dynamism introduced by other factors, such as traffic events. Also, it would not represent uncertainty connected to this route as all possible events on every segment would be smoothed into a single value. Therefore, we have proposed to introduce a probabilistic speed profile. Speed on each segment in each time would be represented not by a single speed but by several possible speeds and their corresponding probabilities. Example of such profile can be seen in Table 2. Utilizing the right method (which are more thoroughly described in [17]) together with these probabilistic profiles, instead of the single duration of a route, we can receive a complete probabilistic distribution q(τ0 , t). These distributions can then be readily utilized in reordering part. However, obtaining these probabilistic profiles is not a simple task. To capture as much of traffic behavior as possible, they would require very long time series of Table 1 Sample time-dependant speed profile

Time period

Average speed (km/h)

06:00–06:15 06:15–06:30 06:30–06:45 06:45–07:00

50 45 40 30

Methodology for Generating Synthetic Time-Dependant Probabilistic Speed Profiles Table 2 Sample probabilistic time-dependant profile Time period Speed 1 Prob. 1 Speed 2 (km/h) (km/h) 06:00–06:15 06:15–06:30 06:30–06:45 06:45–07:00

50 50 50 50

0.7 0.5 0.4 0.2

30 30 30 30

97

Prob. 2

Speed 3 (km/h)

Prob. 3

0.2 0.3 0.3 0.4

10 10 10 10

0.1 0.2 0.3 0.4

historical speed data from each segment to compute their weekly probabilistic speed profile. Also, our proposed system, which is aimed at the smart city, requires coverage of almost all road segments which is also very difficult to achieve. Therefore, it would be almost impossible to derive these profiles from real-world data. However, these profiles are necessary to perform the testing and optimization of our proposed system. There are several ways to obtain these profiles. The first is to utilize some sort of publicly available data set containing traffic speeds. There are many such sets, for example from New York [1] or UK motorways [2]. They, however, suffer from various deficiencies including those mentioned above. Usually, they have long enough time series to compute the probabilistic profile, however, resolution of these series may be too long to produce a profile of reasonable granularity (with a maximal window of 15 min). They also usually rely on a network of stationary sensors which has limitations of its own (as we have proven in [15]). Final problem is that they restrict potential experiments to a single locality, which may not be desirable in every case. Another possible solution is to use a traffic simulator like SUMO [6], MATSIM [11], TRANSIMS [16], or others. By running a large number of simulations with different settings, we can produce enough data to compute speed profiles. For example, Brinkhoff [7] proposes how to use his developed city traffic simulator to generate traffic speed time series. While it alleviates most of the previously mentioned problems with real data, it introduces some others. The most difficult one is to create a large number of realistic settings for the simulators which produce as realistic data as possible. Providing we are willing to invest a huge amount of time to calibrate several such simulations, we again end stuck to a single location. To perform experiments on another location, we are forced to calibrate a set of new models reflecting the conditions in a new location. Therefore, this approach is quite unsuitable in our case. One of the possible other solutions is to generate these probabilistic profiles from synthetically generated data. Suitability of this approach was proven in an article written by Li [12]. His approach, however, still requires some historical speed data, which we do not possess. Therefore, we have decided to develop our methodology for generating traffic speed data for probabilistic speed profile calculation. This task, however, is everything but simple. It requires solving several problems: – Finding all relevant data sources concerning a traffic network – Identifying and retrieving valuable information from these sources – Finding a correct mathematical model to generate these probabilistic profiles based on retrieved traffic network metadata.

98

L. Rapant et al.

This article presents a methodology to solve these problems. While this methodology is based on available data sources in the Czech Republic, these sources are certainly not unique to it and can be found in many different countries. The article has the following outline. After this introduction follows a theoretical background part which presents a Markov chain mathematical model, which is utilized in our generator. Next part presents our methodology and its premises, sources, and logic. Then follows practical results and the article is concluded by their discussion in the conclusion.

2 Markov Chains A Markov chain model can be described by the following statements. Let us have a set of states denoted S = s1 , s2 , ..., sn . The process starts in one of these states and at every time step it moves successively from one state to another. If the model is currently in state si , then it moves to another state s j at the next time step with a probability of pi j . This probability does not depend upon previous states of the model before the current state. This is called Markov property. The probabilities pi j are called transition probabilities and are usually stored in n × n matrix called transition matrix. Example transition matrix can look like: ⎛

0.6 ⎜ 0.4 p=⎜ ⎝0.25 0

0.4 0.6 0.25 0

0 0 0.25 0

⎞ 0 0 ⎟ ⎟ 0.25⎠ 1

(1)

where rows and columns correspond to individual states s1 , . . . , s4 . It is also possible for the process to remain in the state it is in with probability pii . Extreme variant of this is absorbing Markov chain. A state si of a Markov chain is called absorbing if it is impossible to leave it ( pii = 1). A Markov chain is absorbing if it has at least one absorbing state, and from every state it is possible to go to an absorbing state (not necessarily in one step). Starting state is specified by an initial probability distribution. Example of such distribution is: ⎛

⎞ 0.5 ⎜ 0.2 ⎟ ⎟ s=⎜ ⎝0.15⎠ 0.05

(2)

Markov chains can be described by a directed weighted graph, where vertices correspond to states si and edges to possible transitions between these states and their weights to probability of this transition.Example of such graph (corresponding to transition matrix in Eq. 1) can be seen in Fig. 1.

Methodology for Generating Synthetic Time-Dependant Probabilistic Speed Profiles

99

Fig. 1 Example graph representing Markov chain [3]

Markov chains are usually assumed to be time-homogeneous. This means that the graph and the matrix are independent of number of step k and are unique for the entire chain. However, in our case, we violate this assumption by having our Markov chain composed of several graph or matrices p1,...,n corresponding to the different times of a day. In this case, the edges of the k th graph are labeled by the probabilities of moving from the state at time k to the other states k + 1. This can be denoted as Pr(X k+1 = x | X k = xk ). This information can also be represented by the transition matrix from time k to time k + 1. More thorough information on Markov chains can be found here [8].

3 Methodology The first step in our proposed methodology for generating a synthetic probabilistic speed profiles is identification of relevant and available data sources. We expect these to describe both the qualities of road network and potential of traffic events. One of the most complete, freely available description of traffic network is contained in Open Street Maps (OSM) [14]. They contain all necessary static information about traffic network like maximum speed or road type (i.e., whether is the road highway or ordinary road). To cover the dynamic properties (i.e., time-dependant patterns or traffic events), we have to enrich these data. The time-dependant patterns can be mostly covered by empiric experiences from the literature. Figure 2 from the article written by Loumiotis et al. [13] clearly shows the basic trends during the week. The traffic is the heaviest during the morning and afternoon rush hour. Severity of this phenomenon may be influenced by type of the road (i.e., these jams are more likely to occur on road of higher class then on the ordinary street).

100

L. Rapant et al.

Fig. 2 Example of traffic speed development during the week [13]

To cover the traffic events, we utilized reports about traffic incidents from Police of Czech Republic [4]. By localization of these incidents, we may find that some segments are more likely to have some traffic problems than the other. In Czech Republic, Police freely shares these data, albeit with a lag of few month. However, we believe that such data are widely available in many countries around the globe. Next step is to determine what valuable information can be utilized from these data sources. From the OSM, we have chosen to utilize information about traffic speed, road type and whether the segment is inside or outside of some municipality. Need for the information about traffic speed is self-explanatory. Both types of road can heavily influence the severity of rush hours. For example, it is easily observable, that roads of higher class near the edges of the municipalities are experiencing some of the heaviest traffic. From the police data about traffic incidents, we have decided to utilize several wounded, number of deaths, and number of cars involved. Increases in all these variables most prominently influence the duration and effects of accidents. With all these data, now we can specify the mathematical model for the generating of probabilistic speed profiles. We have chosen to utilize the Markov chains S = s1 , s2 , . . . , sn as an underlying mathematical model for generating the synthetic traffic speed time series. States s1 , s2 , . . . , sn then correspond to discretization of traffic

Methodology for Generating Synthetic Time-Dependant Probabilistic Speed Profiles

101

speed which may be absolute (i.e. s1 = 120 km/h, s2 = 100 km/h and so on) or relative to the maximum speed on given segment (i.e., s1 = 100%vmax , s2 = 75%vmax , and so on . However, a single Markov chain model would be insufficient to produce data representing various effects described in previous paragraphs. Therefore, we must create several such models S1 , ..., Sn , each of which represents unique spatial and temporal properties of different segments. Based on empirical observations, we propose following temporal discretization (each time of the week has assigned special Markov chain model representing unique conditions during its duration): – Profile model—Working day 1. Night: 21:00–06:00: Very light or almost nonexistent traffic 2. Morning peak: 06:00–09:00: Very heavy traffic with higher probability of traffic problems 3. Off-peak: 09:00–15:00: Light to medium traffic with low probability of traffic problems 4. Afternoon peak: 15:00–18:00: Very heavy traffic with higher probability of traffic problems 5. Off-peak: 18:00–21:00: Light to medium traffic with low probability of traffic problems – Profile model—Weekend 6. Night: 20:00–07:00: Very light or almost nonexistent traffic 7. Morning peak weekend: 07:00–10:00: Medium traffic with low probability of traffic problems 8. Off-peak: 10:00–16:00: Light to medium traffic with low probability of traffic problems 9. Afternoon peak weekend: 16:00–20:00 Light to medium traffic with low probability of traffic problems We have decided to split the working days from weekend days as in most cases, the traffic behaves somewhat differently. We prepare a Markov chain model Si represented by transition matrix pi for each of these time periods. Without data, these models must be set based on an empiric knowledge of the traffic. However, it is entirely possible to train these models on reasonable batch of real-world data. For the sake of the space, our example models have 4 states: state 1 with 100% of maximum speed represents free flow, states 2 and 3 with 75% and 50 of maximum speed represent slowdown of traffic due to higher traffic intensity or other factors, and state 4 with 10% of maximum speed represents traffic jam. Then for example, transition matrices for the week afternoon peak p4 and week night p1 can look like ( where i corresponds to the labels in our list): ⎛

0.9 ⎜0.8 p1 = ⎜ ⎝0.7 0.1

0.075 0.1 0.15 0.1

0.02 0.075 0.1 0.3

⎞ 0.05 0.025⎟ ⎟ 0.05 ⎠ 0.5

⎛

0.2 ⎜ 0.1 p4 = ⎜ ⎝0.05 0

0.4 0.4 0.25 0.05

0.3 0.3 0.5 0.25

⎞ 0.1 0.2 ⎟ ⎟ 0.25⎠ 0.8

102

L. Rapant et al.

Another step is to introduce influence of location and type of the road. Each type influences can influence all periods or only specific ones, i.e., segments leading to and from the municipality are influenced by their type mainly during morning or afternoon peak, respectively. We have chosen to differentiate following types of segments (based on their functional classification ( f r c) from OSM and their relation to municipalities): 1. Municipality arteries—the busiest city roads ( f r c > 2 and inside municipalities): higher probability of slowdown and traffic jam, even heavier traffic during all peak hours 2. Municipality streets—ordinary streets ( f r c > 2 and inside municipalities): smaller influence of both peaks, small probability of traffic jam 3. Border municipality arteries IN—the busiest city roads leading into the municipality ( f r c > 2 and near border of municipality leading in the direction into the city): similar to arteries with greater load mainly during morning peak 4. Border municipality arteries OUT—the busiest city roads leading out of the municipality ( f r c > 2 and near border of municipality leading in the direction from the city): similar to arteries with greater load mainly during afternoon peak 5. Border municipality streets IN—ordinary streets leading into the city ( f r c > 6 and near border of municipality leading in the direction into the city): similar to streets with heavier traffic load during morning peak 6. Border municipality streets OUT—ordinary streets leading out of the city ( f r c > 2 and near border of municipality leading in the direction from the city): similar to streets with heavier traffic load during afternoon peak 7. Motorway—motorways and similar roads ( f r c > 2 and outside of municipality): higher probability of traffic jam, lower probability of slowdown (i.e., in Czech Republic, motorway traffic is usually either free flowing or jammed), smaller influence of peak hours 8. Trunk—main roads between bigger cities ( f r c > 4 and outside of municipality): similar to motorway with more equal state distribution (i.e. similar probability of slowdown and jam), smaller influence of peak hours 9. Ordinary roads—rest of the roads outside of cities ( f r c > 6 and outside of municipality): small probability of slowdown or traffic jam, smaller influence of peak hours. These influences are then realized by a set of matrix functions f ji ( pi ), where pi is a transition matrix corresponding to current time period i, j corresponds to label of the segment type and i denotes current time period (i.e., it is possible to have different function for the different time period for each king of segment type). Example of such function can look like:  f 44 (A) =

a11 −a11 ·0.5 a21 −a21 ·0.5 a31 −a31 ·0.5 a41 −a41 ·0.5

a12 −a12 ·0.5 a22 −a22 ·0.5 a32 −a32 ·0.5 a42 −a42 ·0.5

a13 +a11 ·0.25+a12 ·0.25 a23 +a21 ·0.25+a22 ·0.25 a33 +a31 ·0.25+a32 ·0.25 a43 +a41 ·0.25+a42 ·0.25

a14 +a11 ·0.25+a12 ·0.25 a24 +a21 ·0.25+a22 ·0.25 a34 +a31 ·0.25+a32 ·0.25 a44 +a41 ·0.25+a42 ·0.25

(3)

where A is input matrix and ax y are its elements corresponding to row x and column y. Please note that these functions must preserve a sums of each row.

Methodology for Generating Synthetic Time-Dependant Probabilistic Speed Profiles

103

Additionally, some segments can be marked as ones with frequent traffic incidents. These segments can be chosen based on data describing traffic incidents (in our case data from Police of Czech Republic). Each road segment is evaluated by sum of weights of all accidents that happened on said segment in a given day of the week. These weights wsd are calculated as wsd = (100 · Number of deaths + 10 · Number of seriously wounded + +Number of lightly wounded + 1) × Number of participating vehicles, where s is label of segment and d is day of the week. Segments with some incidents (i.e., w > 0) are then ordered by their w for each day of the week and divided into five groups by corresponding quantiles with the group 1 containing the segments with the lowest sums of weights and group five containing the highest. Each group is then assigned a function h dg ( p j ), where g corresponds to group number, d to day of the week and i a transition matrix for given type of segment and time period. The construction of this function is very similar to Eq. 3. Now, we can specify how to generate the probabilistic speed profile. Each of the profile is computed from a number of speed time series. By utilizing the models that we have specified in previous paragraphs, we can simply generate these sequences. Each weekly sequence is composed of secession of n Markov models (their length is given by a step of the models, we usually utilize steps from 5 min to 15 min) which are switched based on the time of the day and functions depending on type of the segments and traffic incident probability are applied: tsvk = (h 1g ( f j1 ( p11 ), . . . , h ig ( f ji ( pil ), . . . , h 7g ( f j6 ( p6n ) We consider all series to start at state 1. After generating a sufficient number of series (usually tens to low hundreds), these series are divided into the groups representing a number of bins for probabilistic profile and probability for each group is calculated. Progress of entire methodology can be summarized into Pseudocode 1.

4 Experimental Results Tests of this methodology were performed on two routes from Ostrava to Prague in the Czech Republic. These were chosen based on variety a types of segments and traffic accidents incidence. One of the roads passed along the main highway where the traffic is most heavy as it passes through the second biggest city in the Czech Republic, Brno. The other one passed mainly along the less frequented and slower roads. We have utilized the data from OSM and Police of Czech Republic. We have chosen to discretize the speed to four possible states in our profiles (i.e. s1 = 100%vmax , s2 = 75%vmax , s3 = 35%vmax and s4 = 10%vmax ). From the perspective of testing our traffic flow optimization, we are mainly interested in the last state of the traffic,

104

L. Rapant et al.

Algorithm 1 Methodology for generating random profiles 1: procedure Generate probabilistic profiles(segments, modelss patial, n, k, incidents) 2: for segments in segments do 3: for i = 1 : k do 4: for j = 1 : n do 5: p j = ComputeTransitionFromTime(s, j,modelst emporal) 6: p j = ModifyTrasitionFromType(s, j,modelss patial, p j ) 7: p j = ModifyTrasitionFromIncidents(s, j,incidents, p j ) 8: v kj = GenerateSpeed( p j , p j−1 ) 9: end for 10: end for pr o f iles(s=ComputeProfile(v kj ) 11: end for 12: return pr o f iles 13: end procedure

modelst emporal,

because it implies some problems, which must be optimized. Figures 3 (for highway) and 4 (for lesser quality road) show the probability of the state s4 from our generated probabilistic speed profiles along the route segments and during the entire week. Time discretizetion for the profiles was chosen to be 15 min, i.e., the week is divided into 672 time intervals. It is evident from the figures that there is significantly higher probability of traffic problems along the highway (and especially in the vicinity of segments 300–350 near Brno) than along the less frequented roads. This effect corresponds to the real

Fig. 3 Example of probability of 10%vmax along the slower road during the week

Methodology for Generating Synthetic Time-Dependant Probabilistic Speed Profiles

105

Fig. 4 Example of probability of 10%vmax along the highway road during the week

Fig. 5 Example of probability of 30%vmax for one highway segment during the week generated by our methodology

behavior of the traffic and gives us great tool for testing our traffic optimization (as it has to choose whether take faster but riskier road or slower but safer one). This also proves that our methodology approaches the real behavior of the traffic as closely as possible without real traffic data.

106

L. Rapant et al.

Fig. 6 Example of traffic speed time series for one UK highway segment during the week

Direct comparison is with real-world data is complicated by the fact, that they are very difficult to obtain in sufficient quality to generate genuine probabilistic speed profiles. We were able to obtain some data from UK highway network [5], which can be utilized for a certain level of comparison. Main problem of this data source is that it does contain only very rough measurements almost devoid of traffic incidents. However, it can be utilized for some time trend comparison. We compare an example speed time series covering one week from the UK highway with probability of the slow traffic (i.e. 30% of vm ax) for the similar segment generated by our model. It can be quite fairly expected that higher probability of lower speed should correspond to slower traffic in real life. Results can be seen in Figs. 5 and 6. From both figures, it is evident, that both time series experience growth of probability and decrease of speed at roughly same time. This behavior shows, that our approach is capable to capture the patterns that govern real traffic.

5 Conclusion In this article, a methodology for generating probabilistic speed profiles was presented. We have proven that utilizing the widely available data together with empiric knowledge and right mathematical mode, it is possible to generate realistic traffic speed data that can be utilized for testing of our reordering approach to traffic flow optimization. Huge advantage of our approach is that it does not necessarily require real traffic data. Instead, it is built on metadata about traffic network and reasonable amount of knowledge of traffic patterns. While we have used traffic network metadata specific for the Czech Republic, we believe that similar data are available for most

Methodology for Generating Synthetic Time-Dependant Probabilistic Speed Profiles

107

of the countries. Successful development of this approach allowed us to perform large-scale testing of reordering approach for traffic flow management developed in framework of Antarex H2020 project [9]. Without generated data from our proposed methodology, it would be practically impossible to perform any tests with traffic flow optimization system that is so robust and data demanding. In terms of future work, it would be highly desirable to compare our results to real data from traffic and even try to learn parameters of our models through some machine learning approach. Acknowledgements This work was supported by The Ministry of Education, Youth and Sports from the National Programme of Sustainability (NPS II) project ‘IT4Innovations excellence in science—LQ1602’, by the IT4Innovations infrastructure which is supported from the Large Infrastructures for Research, Experimental Development and Innovations project ‘IT4Innovations National Supercomputing Center LM2015070’, and partially by the SGC grant No. SP2019/108 ‘Extension of HPC platforms for executing scientific pipelines’, VŠB—Technical University of Ostrava, Czech Republic.

References 1. https://data.gov.uk/dataset/6a6aa71d-7871-4764-a706-4a96e90c9ba2/road-statistics-trafficspeeds-and-congestion. 22 Aug 2019 2. https://data.cityofnewyork.us/Transportation/Real-Time-Traffic-Speed-Data/qkm5-nuaq. 22 Aug 2019 3. https://brilliant.org/wiki/markov-chains/. 22 Aug 2019 4. https://www.policie.cz/clanek/statistika-nehodovosti-900835.aspx. 22 Aug 2019 5. https://roadtraffic.dft.gov.uk/summary. 22 Aug 2019 6. Behrisch, M., Bieker, L., Erdmann, J., Krajzewicz, D.: Sumo simulation of urban mobility an overview. In: SIMUL 2011, The Third International Conference on Advances in System Simulation, pp. 63–68 (2011) 7. Brinkhoff, T.: Generating traffic data. Bulletin of the Technical Committee on Data Engineering, vol. 26. IEEE Computer Society (2003) 8. Grinstead, C.M., Laurie Snell, J.: Introduction to Probability. American Mathematical Society (2003) 9. Golasowski, M., Beránek, J., Šurkovský, M., Rapant, L., Szturcová, D., Martinoviˇc, J., Slaninová, K.: Alternative paths reordering using probabilistic time-dependent routing. In: Barolli, L., Nishino, H., Enokido, T., Takizawa, M. (eds.) Advances in Networked-based Information Systems, pp. 235–246. Springer International Publishing, Cham (2020) 10. Golasowski, M., Tomis, R., Martinoviˇc, J., Slaninová, K., Rapant, L.: Performance evaluation of probabilistic time-dependent travel time computation. In: Computer Information Systems and Industrial Management, pp. 377–388 (2016) 11. Horni, A., Nagel, K., Axhausen, K.W.: Introducing MATSim, The Multi-Agent Transport Simulation MATSim. Ubiquity Press (2016) 12. Li, H.: Automatically generating empirical speed-flow traffic parameters from archived sensor data. Proc. Soc. Behavioral Sci. 138, 54–66 (2014) 13. Loumiotis, I., Demestichas, K., Adamopoulou, E., Kosmides, P., Asthenopoulos, V., Sykas, E.: Road traffic prediction using artificial neural networks. In: SEEDA-CECNSM, pp. 1–5 (09 2018) 14. OpenStreetMap contributors: Planet dump retrieved from https://planet.osm.org, https://www. openstreetmap.org (2017) 15. Rapant, L., Slaninova, K., Martinovic, J., Scerba, M., Hajek, M.: Comparison of ASIM traffic profile detectors and floating car data during traffic incidents. In: Proceedings of 14th IFIP

108

L. Rapant et al.

TC 8 International Conference on Computer Information Systems and Industrial Management. Lecture Notes in Computer Science, vol. 9339, pp. 120–131. Springer, Berlin (2015) 16. Smith, L., Beckman, R., Anson, D., Nagel, K., Williams, M.: Transims: Transportation analysis and simulation system. In: National Transportation Planning Methods Applications Conference (1995) 17. Tomis, R., Rapant, L., Martinoviˇc, J., Slaninová, K., Vondrák, I.: Probabilistic time-dependent travel time computation using Monte Carlo simulation. In: High Performance Computing in Science and Engineering, pp. 161–170. Springer International Publishing, Cham (2016)

A Multi-Scale Patch-Based Deep Learning System for Polyp Segmentation Debapriya Banik, Debotosh Bhattacharjee and Mita Nasipuri

Abstract Colorectal cancer (CRC) is one of the deadliest forms of cancer and is on the rise. Accurate segmentation of the precursor lesion, the polyp, can ensure the survival rate. Hence, there is a research trend to develop medical diagnosis support tool to assist clinicians. This study focuses on the development of deep learning-based CNN model for automated segmentation of polyp. As polyp varies frame to frame in terms of size, color, shape, and texture, segmentation is still an unsolved problem and a very challenging task. We have proposed a multi-scale patch-based CNN model for automatic segmentation of the polyp region. Local and global patches are extracted from each pixel of the input image and fed into two similar CNNs of which one is responsible for the extraction of local features and the other for extraction of global features that are being concatenated for accurately pixel label annotation of the polyp region. As in colonoscopy frames, there are some regions with the same intensity/texture as the polyp regions, so in the predicted segmentation map, some non-polyp regions are also considered as polyp regions, which are further refined by post-processing operation. The proposed model is evaluated on CVC-Clinic DB. The experimental result shows that our proposed method outperforms other baseline CNNs and state-of-the-art methods. Keywords Colorectal cancer · Polyp · Deep learning · Medical image segmentation

D. Banik (B) · D. Bhattacharjee · M. Nasipuri Department of Computer Science and Engineering, Jadavpur University, Kolkata 32, India D. Bhattacharjee e-mail: [email protected] M. Nasipuri e-mail: [email protected] © Springer Nature Singapore Pte Ltd. 2020 R. Chaki et al. (eds.), Advanced Computing and Systems for Security, Advances in Intelligent Systems and Computing 1136, https://doi.org/10.1007/978-981-15-2930-6_9

109

110

D. Banik et al.

1 Introduction Over the last decade, colorectal cancer (CRC) significantly threatens human health. The most cited risk factor for CRC is individual’s unhealthy diet [1]. It has seen a dramatic increase in incidence rate with the increasing urbanization. Usually, early diagnosis and treatment can effectively cure this disease, or it will deteriorate into cancer and spread to some other vital organs [2]. As per the reports published by GLOBOCAN report 2019, it has been observed that CRC is the fourth most common cause of cancers among men and women across the globe [3]. CRC often commences as a protrusion from the colon surface termed as a polyp, which is benign in its early stage but eventually develops into cancer if not treated appropriately. The survival rate of CRC solely depends on the stage in which it is detected, decreasing from rates higher than 95% in the earlier stages to rates lower than 35% in later stages [4]. A minimally invasive operator-dependent technique, namely colonoscopy, is considered as a “gold standard” for manual screening of the entire colon [5]. However, human factors such as lethargy, insufficient concentration, lack of proper illumination, and back to the back examination may lead to miss-diagnosis of a polyp during the exploration. This miss-diagnosis of the polyp may incline toward malignancy in the later stage of its growth. To compensate the miss-rates of the polyp, due to human uncertainty, the development of computer-aided diagnostic techniques has been an active area of research to support clinicians for early screening of CRC. Polyp segmentation has a consequential pertinence for clinical examination as it refers to the area enclosed by the polyp region that gives explicit information about the polyp shape suitable for in vivo analysis and guides clinicians for histopathological analysis to decide the degree of malignancy and decide the plan of treatment [6]. Despite significant progress over the last few years, computer-aided polyp segmentation is a challenging task due to the immense variation of polyp in terms of size, shape, color, and texture—Fig. 1 shows different variabilities of polyps. Numerous computational methods have been proposed over the past few years for accurate segmentation of the polyp regions. However, it is still underdeveloped and cannot mitigate the low recall rates [7]. We can broadly categorize the proposed segmentation techniques based on three approaches. The first approach is focused on image processing-based segmentation techniques, which includes region growing, active contour, watershed, etc. The second approach is focused on machine learning (ML) techniques where relevant hand-crafted features are extracted and trained using a classifier, viz. SVM, ANN, etc., for segmentation of the region of interest. Recently,

Fig. 1 Differnt variability of polyp in terms of size, shape, color, and texture

A Multi-Scale Patch-Based Deep Learning System …

111

the segmentation approach based on deep learning (DL) methods using the convolutional neural network (CNN), transfers learning using pre-trained models, has been an active area of research, and appears to give a more promising result. The rest of the paper is arranged as follows. In Sect. 2, we have reviewed some recent related research work. Sections 3 demonstrate the proposed system in detail. Section 4 describes the details of an experiment conducted to justify the performance of the proposed network, and the conclusion is drawn in Sect. 5.

2 Related Literature In this section, we have explored some of the recent state-of-the-art techniques related to the automatic segmentation of the polyp region. As discussed in the previous section, the segmentation techniques can be broadly clustered based on three different approaches. Recently, authors in [8] have proposed an automated segmentation method based on the first approach. In [8], authors have proposed an active contourbased polyp segmentation technique. Polyp frames are preprocessed using principal component pursuit (PCP) technique, which is followed by an active contour model, namely CV-LSM. The experimental result shows that the method can reasonably localize the polyp region, but the method fails to localize the polyp in low illumination condition. In contrast to the first approach, the second approach, which is based on ML techniques, has shown more promising results than the former approach. The extraction of domain-specific descriptors plays a vital role in this approach. Different authors [9, 10] have proposed different ML techniques with relevant descriptors for segmentation of the polyp region. Most recently, authors in [9] have proposed an automatized colon polyp segmentation via contour region analysis. In their study, they have applied some preprocessing steps to remove the artifacts in the polyp frame, which is followed by the extraction of edges for curvature calculation. Different descriptors are evaluated from the curvature and edge regions and finally concatenated and fed into different classifiers for the best segmentation result. This method fails to segment polyps with weak edges as the method solely depends on the edge information. Another study [10] has adopted a simple linear iterative clustering (SLIC) superpixel-based method for segmentation of polyps. Different SLIC values are determined, and the most relevant value for superpixel is chosen. Textural and color features are extracted from the SLIC frame and fed into an SVM for pixelwise classification of the polyp region and healthy region. However, infelicitous SLIC value leads to poor segmentation results. In recent years, the third approach, which is based on DL techniques, has gained lots of attention and emerged as a rapidly growing technique. Authors in [4, 5, 11, 12] have proposed different DL strategies for effective segmentation of the polyp region. A fully convolutional densenet is proposed by [5] for segmentation of polyps. A total number of 78 layers were used in the proposed network, which gives a considerable result in comparison to the stateof-the-art methods. In another study, authors in [11] have proposed a hybrid method using two variants of FCNN. The first variant combines the residual network and

112

D. Banik et al.

the dilation kernel layers within the FCNN, whereas the second proposed architecture is based on U-net augmented by dilation kernels. Before segmentation, frames were preprocessed and rescaled. Another exciting network is proposed by [4], where well-established network such as VGG and ResNets are restructured and fine-tuned for polyp segmentation. In the proposed network, an additional input channel to the RGB information is incorporated namely shape from shading (SfS) which recover the depth and provide a richer representation of the tissue’s structure. Qadir et al. [12] adapted different CNNs namely ResNet50, ResNet101, and Inception ResNet for extraction of features for Mask R-CNN. They have evaluated whether a more complex network is required or better dataset for training is required for the extraction of features. Finally, they have proposed an ensemble network for better performance. In their study, ResNet50 outperforms ResNet101 and ResNet Inception (v2) when a limited amount of training data is available. In this study, we have proposed a deep CNN network for the segmentation of the polyp region. As already discussed, polyps have immense variation in shape, size, and color. However, the same polyp may also appear in different scales depending on the distance between the polyp and the camera. This diverse representation of the same polyp in different scales motivated us to design a multi-scale patch-based CNN for in-depth learning. The segmentation map obtained by our method is further enhanced using some traditional post-processing techniques. We validate our method with the existing state-of-the-art method and also with the pre-trained baseline CNN models.

3 Proposed Method In this section, we have demonstrated our proposed end-to-end multi-scale patchbased learning CNN framework for the automatic segmentation of polyp. Figure 2 represents the workflow of our proposed methodology. Let us assume U :  → + be an RGB image where  ⊂ 3 signifies the image domain. The regions in the colonoscopy frame are typically classified as polyp regions and the non-polyp regions. In the following subsection, each of the components of the network is explained elaborately.

Fig. 2 Proposed CNN architecture

A Multi-Scale Patch-Based Deep Learning System …

113

3.1 Multi-Scale Patch Extraction In this scheme, patches are extracted from the input RGB image U H ×W of size 384 × 288 in different scales where H and W represent the height and width, respectively. A m number of patches Pam where m = {1, 2, 3, . . . , n} is extracted by sliding a window around each pixel, considering it as the center where the output would label the pixel. For each pixel, local and global patches are extracted simultaneously [13]. The local patch signifies the minute information, such as edges and corners, and can differentiate between a polyp and a non-polyp region based on local texture information. The local information can precisely segment the polyp boundary region. So, a local patch of size 34 × 34 is considered around each pixel in the input RGB image. In contrast to the local patch, a global patch is also extracted, keeping the same center as the local patch, which also carries vital information of the surrounding pixel region to decide the pixel label. The global patch can overlook the lack of proper illumination of a polyp and artifacts like specular highlights (shiny spots) [14], which result in miss-diagnosis of polyps as it is challenging to differentiate between the polyp and the non-polyp region. Hence, a global patch of size 68 × 68 is considered for each pixel surrounding the local patch keeping the same center. However, the global patch is resized to the size of the local patch. So for each input RGB image, 110, 592 pairs of local and global patches are extracted and fed into two similar CNNs. Different scales of patches for the same pixel will enable it to learn shape-invariant features at different scales. Hence, the robustness of CNN can be well improved with multi-scale extraction of patches [15]. The proposed CNN architecture is explained in the next subsection.

3.2 Proposed CNN Architecture The multi-scale patches (local patch and global patch) extracted from the input RGB images are fed into two similar parallel CNNs, as shown in Fig. 2. The resized global patch is fed into CNN through the upper path, and the corresponding local patch is fed into similar CNN through the lower path. The two CNNs have the same number of layers and other tunable hyperparameters. In our network, we have stacked three convolutional layers, and each of them is passed through rectified linear unit (ReLu) to impose nonlinearity to the output feature map. The first convolutional layer consists of 16 learnable kernels with a kernel size of 5 × 5. The second convolutional layer consists of 32 learnable kernels with a kernel size of 3 × 3, and the last convolutional layer consists of 64 learnable kernels with a kernel size of 3 × 3. The initial layer is mainly responsible for the extraction of generic features, so less number of kernels are used, and as the network progresses, deeper complex patterns are learned by the network, so more number of kernels are used in those layers [16]. Each of the convolutional layers is followed by maxpool of size 2 × 2. The maxpool layer

114

D. Banik et al.

downsamples the dimensionality of the feature maps, which are extracted from the convolutional layers. Finally, the feature maps from the two CNNs are concatenated and passed to the fully connected layer with 500, 100, which are densely connected, and dropout with probability p = 0.5 is used to enhance the performance and avert overfitting. The output layer is responsible for distinguishing between polyp and non-polyp region, so two neurons are used in the output layer, which is activated by a sigmoid function as it considers real values between 0 and 1.

3.3 Post-processing The segmentation map obtained by our proposed method extracts some non-polyp regions having the same probability distribution as the polyp region. To overcome this drawback, we have discarded connected regions having pixels value less than 400 as it has been experimentally evaluated polyp region have pixels at least higher than 400, which is followed by morphological hole filling operation, which gives the final segmentation output. The segmentation results are validated qualitatively and quantitatively, which are discussed in the following section.

4 Experimental Results and Discussion In this section, we have evaluated the performance of our proposed segmentation method. In Sect. 4.1, we have provided a detailed description of the dataset used for this study. In Sect. 4.2, we have qualitatively and quantitatively evaluated the efficacy of our method concerning the annotated ground truths to validate the robustness of the method. Our proposed model is implemented in Python 3.5 using Keras library and Tensorflow backend on a system with Intel Xeon Processor, 128 GB RAM powered by NVIDIA Quadro P5000 GPU of 16 GB. In this study, we have considered the dice loss function to tune the learnable kernels and optimize our network.

4.1 Database Description Our proposed segmentation technique is developed and evaluated on the CVCClinicDB database, a benchmark public dataset [17]. The dataset has been generated from 25 different colonoscopy videos, which is composed of 29 different sequences containing a polyp. In total, there are 612 SD frames each of size 384 × 288. We have randomly split the dataset into non-intersecting 70%–10%–20%, where 70% of the frames for training our model, 10% to validate the model, and the remaining 20% to test the model.

A Multi-Scale Patch-Based Deep Learning System …

115

4.2 Qualitative and Quantitative Evaluation Qualitative evaluation: In this subsection, we have shown the robustness of our method by a visual comparative analysis of our proposed method (PM) concerning the ground truth image (G t ). It may be noted that for evaluation of any medical image segmentation method, experts from the relevant field is responsible for manual delineation of the region of interest to generate the ground truth binary mask. We have considered frames with a polyp in various shapes, sizes, orientations, and colors to justify the efficacy of our proposed method. The 1st row in Fig. 3 shows the input RGB polyp frames, and the corresponding ground truth is shown in the next row, whereas the segmentation results by our proposed method are shown in row 3. In the 4th row, we have shown the TP, FP, and FN measure in terms of region overlay between G t and PM. The magenta color signifies the FP, whereas the green color signifies the FN. It can be visually perceived that our proposed method holds a good

Fig. 3 Comparision of our segmentation result. Row 1: input image. Row 2: ground truth. Row 3: proposed method. Row 4: overlay. Row 5: contour

116

D. Banik et al.

Fig. 4 Linear regression plot

agreement with the ground truth as very much less region is present in the FP and FN portion, and most of the region is part of TP (white portion). In the last row, we have additionally shown the contour variation between PM and the corresponding G t where the magenta color signifies the ground truth and the blue color signifies the proposed method. As there is much less variation between the two contours, it is a significant indicator that our PM gives diagnostically accepted results. Quantitative evaluation: In this subsection, we have quantitatively evaluated our proposed method with some well-known pre-trained baseline deep learning-based segmentation models and a state-of-the-art method. To judge the segmentation performance, the discrepancy evaluation method is being used where different metrics are compared between predicted segmentation results and the ground truth (G t ). Firstly, we have statistically evaluated the degree of similarity between the PM and the G t by a linear regression plot, as shown in Fig. 4. It can be observed that there is a good agreement between our PM and G t where the degree of correlation (R2 ) is 0.844. The performance of medical image segmentation result is well judged by the amount of overlap between the predicted segmentation results and the G t . So, to justify the potentiality of our PM, we have compared our PM with two well-known baseline CNN models, viz. U-net [18] and SegNet [19] in terms of region overlap measure specifically intersection over union (IoU), in which values ranges between 0 and 1. It can be well perceived from the bar graph in Fig. 5 that our PM significantly performs better than the two baseline models. Finally, we have compared our method with two baselines CNNs and a state-ofthe-art method in terms of dice similarity (DS), precision (Pre), and recall (Rec). The DS value justifies the region overlap between the segmentation result and the G t . However, a single-handed evaluation metric cannot be a criterion to evaluate the segmentation method, so we have also evaluated Pre and Rec. It can be witnessed from Table 1 that our PM outperforms the two baseline CNNs in terms of DS, Pre, and Rec. However, Pre for [20] is more than our PM, but there is a vast variation

A Multi-Scale Patch-Based Deep Learning System …

117

Fig. 5 Bar plot to compare the PM with other baseline CNNs in terms of IoU

Table 1 Quantitative evaluation of polyp segmentation results, best results are colored Dataset

Models

Segmentation performance DS↑

Pre↑

Rec↑

CVC-clinic DB

U-net

0.785

0.769

0.692

SegNet

0.738

0.704

0.666

Pogorelov et al. [20]

–

0.819

0.619

PM

0.813

0.809

0.786

between Pre and Rec for [20], which is clinically insignificant. The trade-off between Pre and Rec for our PM justifies the efficacy of our method.

5 Conclusion Accurate segmentation of the abnormal regions decides the plan for further diagnosis of a patient. In this study, we have presented a multi-scale patch-based CNN model for accurate segmentation of polyp from colonoscopy frames. Local and global patches are defined around each pixel, which is considered as the center for the two patches that are extracted from the input image. The extracted local and global patches which carry significant local and global textural information are fed into two similar CNNs having the same configuration. The output gives the label for the center pixel of the patch as a polyp region or non-polyp region. The proposed method is compared with some baseline CNNs and a state-of-the method where our PM outperforms other methods. As a part of future work, we will implement our PM on some benchmark datasets of other medical image modalities to further tune the model and justify the efficacy of the model.

118

D. Banik et al.

Acknowledgements The first author is grateful to the Council of Scientific and Industrial Research (CSIR) for providing Senior Research Fellowship (SRF) under the SRF-Direct fellowship program (ACK No. 143416/2K17/1, File No. 09/096(0922)2K18 EMR-I). The second author is thankful to the RUSA 2.0 at Jadavpur University for supporting this work.

References 1. Tabung, F.K., Brown, L.S., Fung, T.T.: Dietary patterns and colorectal cancer risk: a review of 17 years of evidence (2000–2016). Curr. Colorectal Cancer Rep. 13(6), 440–454 (2017) 2. Bernal, J., et al.: Comparative validation of polyp detection methods in video colonoscopy: results from the MICCAI 2015 endoscopic vision challenge. IEEE Trans. Med. Imaging 36(6), 1231–1249 (2017) 3. Globocaan: “Cancer Fact sheet,” 1383 [Online]. Available: http://gco.iarc.fr/today/home. Last accessed 2019/8/10 4. Brandao, P., et al.: Towards a computed-aided diagnosis system in colonoscopy: automatic polyp segmentation using convolution neural networks. J. Med. Robot. Res. 03(02), 1840002 (2018) 5. Yu, J., Pan, H., Yin, Q., Bian, X., Cui, Q.: Fully convolutional DenseNets for polyp segmentation in colonoscopy. In: 2019 IEEE 35th International Conference on Data Engineering Workshops, pp. 306–311 (2019) 6. Dijkstra, W., Sobiecki, A., Bernal, J., Telea, A.: Towards a single solution for polyp detection, localization and segmentation in colonoscopy images. In: Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, pp. 616–625 (2019) 7. Lequan, Y., Hao, C., Qi, D., Jing, Q., Pheng Ann, H.: Integrating online and offline threedimensional deep learning for automated polyp detection in colonoscopy videos. IEEE J. Biomed. Heal. informatics 21(1), 65–75 (2017) 8. Sasmal, P., Iwahori, Y., Bhuyan, M.K., Kasugai, K.: Active contour segmentation of polyps in capsule endoscopic images. In: 2018 International Conference on Signals and Systems, pp. 201–204 (2018) 9. Sánchez-gonzález, A., García-zapirain, B., Sierra-sosa, D., Elmaghraby, A.: Automatized colon polyp segmentation via contour region analysis. Comput. Biol. Med. 100, 152–164 (2018) 10. Maghsoudi, O.H.: Superpixel based segmentation and classification of polyps in wireless capsule endoscopy. In: 2017 IEEE Signal Processing in Medicine and Biology Symposium (SPMB), pp. 1–4 (2017) 11. Guo, Y., Matuszewski, B.: GIANA Polyp Segmentation with Fully Convolutional Dilation Neural Networks. CVML, 632–641 (2019) 12. Qadir, H.A., Shin, Y., Solhusvik, J., Bergsland, J., Aabakken, L., Balasingham, I.: Polyp detection and segmentation using mask R-CNN : does a deeper feature extractor CNN always perform better? In: 2019 13th International Symposium on Medical Information and Communication Technology, pp. 1–6 (2019) 13. Farabet, C., Couprie, C., Najman, L., LeCun, Y.: Learning hierarchical features for scene labeling. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1915–1929 (2012) 14. Sánchez-González, A., Soto, B.G.-Z.: Colonoscopy image pre-processing for the development of computer-aided diagnostic tools. In: Surgical Robotics, IntechOpen (2017) 15. Zhao, R., Ouyang, W., Li, H., Wang, X.: Saliency detection by multi-context deep learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1265– 1274 (2015) 16. Roy, K., Banik, D., Bhattacharjee, D., Nasipuri, M.: Patch-based system for classification of breast histology images using deep learning. Comput. Med. Imaging Graph. 71, 90–103 (2019)

A Multi-Scale Patch-Based Deep Learning System …

119

17. Bernal, J., Sánchez, F.J., Fernández-Esparrach, G., Gil, D., Rodríguez, C., Vilariño, F.: WMDOVA maps for accurate polyp highlighting in colonoscopy: validation vs. saliency maps from physicians. Comput. Med. Imaging Graph. 43, 99–111 (2015) 18. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234–241 (2015) 19. Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), 2481– 2495 (2017) 20. Pogorelov, K. et al.: Deep learning and hand-crafted feature based approaches for polyp detection in medical videos. In: Proceedings of IEEE Symposium on Computer-Based Medical Systems, pp. 381–386 (2018)

Object Detection in Rainy Condition from Video Using YOLO Based Deep Learning Model Ratnadeep Dey, Debotosh Bhattacharjee and Mita Nasipuri

Abstract Video surveillance is one of the primary and practical actions to prevent criminal and terrorist attacks. Nowadays all public areas are under video surveillance. Most of the surveillance cameras are installed in open spaces. However, the video data captured by the surveillance camera can be affected by the weather condition. In this paper, we are concentrating on the video data captured by surveillance cameras in rainy situation. We have proposed a deep learning-based method to detect object in rainy situations from videos. However, object detection in deep learning is the ubiquitous research topic in computer vision and analysis. Much work has been done in that area. However, deep learning-based object detection in rainy condition remains untouched. We have applied our proposed method for both daytime and nighttime. The method produces excellent results in all conditions. Keywords Object detection · Rainy condition · Deep learning · YOLO

1 Introduction In recent years, terrorism becomes a headache for security personnel in the world. In 2019, until August already 1259 attacks and 5436 casualties have been reported [1]. In 2018, the number of attacks was 2180 where the number of casualties was 9776. The statistics help to understand the alarming situation in the globe. However, countries take necessary actions to prevent the nuisance; still such fatal activities cannot be stopped. Nowadays, all crowded public areas are under video surveillance to prevent terrorist attacks. Each movement of objects, such as person, vehicle, and animals, is monitored thoroughly for 24 h. Manual monitoring of any public area is a tedious job, and there may be a possibility for misrecognition. Therefore, automation

R. Dey (B) · D. Bhattacharjee · M. Nasipuri Department of Computer Science & Engineering, Jadavpur University, Kolkata, India © Springer Nature Singapore Pte Ltd. 2020 R. Chaki et al. (eds.), Advanced Computing and Systems for Security, Advances in Intelligent Systems and Computing 1136, https://doi.org/10.1007/978-981-15-2930-6_10

121

122

R. Dey et al.

is needed for monitoring. An automatic surveillance system is needed to monitor public areas. Therefore, research in this domain becomes popular. Much work has been done in this area [2]. At first stage, traditional image processing techniques are applied in this object detection research [3]. However, tremendous progress in deep learning enforces researchers to apply deep learning in this domain. Deep learning seems useful in this domain. There are some deep convolutional models for object detection present in the literature. The models are mainly classified in two ways— (i) two-stage detection and (ii) one-stage detection. In case of two-stage detection, objects can be detected in two steps. In the first step, probable object can be identified. In the next step, bounding box is predicted. Finally, at optimization stage the optimal bounding boxes are adjusted around the objects. In single-stage object detection model, the images are divided into several grids. The grids containing objects are highlighted. At last, desired objects are detected. RCNN [4], Fast RCNN [5], and Faster RCNN [6] are examples of two-stage object detection where YOLO [7], SSD [8] are the examples of one stage detection. These models can detect an object very accurately. The object detection models are designed for object detection in reasonable visibility conditions. However, in a real-time application, the visibility of the scene may be degraded. Most of the surveillance cameras are installed in open spaces. Therefore, weather condition affects the vision. According to physical properties and effects, the weather conditions can be categorized into two categories [9]—(i) steady and (ii) dynamics. Fog, Haze, and Mist fall in the category of steady. In this case, the sizes of water droplets are small and vary between 1 and 10 μm. The droplets cannot be seen individually through human eyes. However, presence of small droplets makes changes in the intensity of the image. Many models are proposed to describe those effects in the image. In the case of dynamic weather conditions, the size of the droplets present in the air is between 0.1 and 10 mm. Therefore, the droplets can be seen through cameras. Rain, Snow comes into dynamic weather degradation category. In this case, the droplets are coming down with high velocity with complex trajectories. Motion blur is a widespread effect in this situation. In the case of steady weather degradation, the vision affected by aggregated effects of a large number of droplets. Therefore, modeling such weather condition is comparatively easy. On the other hand, modeling of dynamic weather conditions is very complicated because each droplet makes changes in vision with their different velocity and trajectory. In a country like India, where rain is widespread rather than snow. Therefore, we are concentrating on rainy weather conditions. Each raindrop is spherical by shape. They work as a lens. The lights are refracted and reflected through the droplets and make sharp intensity patterns in the image. The raindrops are in motion in time of rain. Depending on the velocity, the shape of the droplets changes. According to Beard and Chuang [10], the shape of the raindrop can be defined as  r (θ ) = a 1 +

10  n=1

 cn cos(nθ )

(1)

Object Detection in Rainy Condition from Video Using YOLO …

123

where a = undistorted shape, cn = coefficients depending on the radius and θ = polar angle of elevation. Different types of distortions can be observed by the changes in a droplet shape. The distortions caused by rain are categorized in two ways—(i) the presence of rain strikes. This can happen due to spatiotemporal correlation of a large number of falling water droplets on image plane and (ii) the visual effect of rain. The intensity changes caused by the droplets. The discussion until now makes the readers feel that rain causes a very complex degradation in vision. Therefore, object detection in rainy conditions is also a very tough job. As discussed earlier, many deep learning-based object detection models are present. In our case, we want to make a real-time system. Therefore, we are considering the object detection model, which can detect object in less time. The YOLO model suits our criteria rather than the other models. The reason why we choose the YOLO model over other models is as follows. 1. Very Fast model—The YOLO model is high-speed to detect objects. The training and testing time of the model are speedy than the other models. 2. Real-time usability—Because of its fastest computation ability, the YOLO model can be used in real-time applications very quickly. A pre-trained YOLO model can be used to detect objects from webcam streaming at current time. 3. Applicability—The YOLO model is convenient to use in system applications. The model is high-speed, and its ability to detect objects in real-time makes it useful in application domain. 4. Classification Score—The model can detect an object by drawing a rectangular box on it. Besides, the model calculates the classification score and shows this score with the rectangular box. The classification score is the probability of identifying an object in a particular class. The classification score is useful in application because according to the score an application system can decide the success of detecting object on that domain. These advantages make us interested in the YOLO model and inspire us to apply this model in our work for better detection of moving objects in rainy situations. The YOLO model mainly designed to detect objects in reasonable visible conditions. In this research work, we have used YOLO model to detect objects in rainy conditions. At best of our knowledge, we have applied the YOLO model in rainy conditions for the first time. We have modified YOLO v3 model using inductive transfer method to detect object in rainy conditions. This is the main contribution of this paper, and we have got satisfactory results. The rest of the paper is organized as follows. Section 2 enlists some related research works, Sect. 3 discusses the methodology of the research work, Sect. 4 shows the results we get and discuss them thoroughly, and finally, Sect. 5 concludes the paper.

124

R. Dey et al.

2 Related Work Many research works have been done to model dynamic weather conditions in computer vision [11–13]. In those works, authors tried to describe the effects of dynamic weather conditions in image mathematically. This modeling technique helps to analysis the rainy images. Some works have been reported for image de-raining. They tried to remove rain effects to recover the input image. They denoise the images and remove the rain strikes from the images. The deep learning model proves very effective in this work. Different deep networks have been used. DerainNet [14], ID-CGAN [15], SRDN [16], PReNet [17] are some examples in this regard. With the advancement of research in rain image modeling and image de-raining, deep learning-based object detection has some potential improvement. In Sect. 1, we have discussed the available deep learning-based object detection models. In this discussion, it is shown that YOLO model is better to use in our work. The YOLO model has already been used for object detection successfully. In [18], Peng et al. detect pedestrians in normal vision. They use the GMM to model the background to detect pedestrians primarily. Pedestrians are also detected by YOLO. Then the two results are combined to generate final detection results. The combined result is more accurate than the single detector by 20%. In another work [19], YOLO-based People Counting (YOLO-PC) is proposed to count persons from video. Lan et al. [20] propose YOLO-R model to identify pedestrians. Three Pass-through layers are added to the main YOLO network to connect shallow networks to deep networks. This modification helps authors to detect pedestrians in challenging situations. Lin et al. [21] count traffic using YOLO-based model. A system is proposed [22] using YOLO model for detecting small vehicles from aerial view angles. It includes dense topology and optimal pooling strategy. Visibility Enhancement Saliency YOLO (VESY) sensor is proposed in [23]. The foggy frames are analyzed through YOLO model to detect objects in low visibility.

3 Methodology In the previous sections, we have discussed the importance of object detection in rainy conditions. We have also started some of recent research works. Modeling of rainy image and image de-raining was the main research topic. A work on object detection in foggy situation has been reported in Sect. 2. However, until now no specific work has not been reported for object detection in rainy conditions. In this work, we are detecting objects from video data, which are distorted by rain. The input data directly feed into the network to detect an object. No preprocessing has been done. The basic deep learning block is mainly inspired by the YOLO v3 [24] model.

Object Detection in Rainy Condition from Video Using YOLO …

125

3.1 Basic Idea The YOLO model treats the object detection problem as a regression problem. The model contains one single convolutional neural network for detecting objects that can be trained end to end. A single image entirely feeds into the network. The network divides the image in grids of different sizes. If the center of an object falls into a grid, then the grid is responsible for detecting the object. Each grid further predicts B number of bounding boxes and confidence scores of each box. The confidence score is the probability of a box containing an object. The confidence is defined as Pr(Object) ∗ IOUtruth pred . The Pr(Object) is the probability of presence of object, and IOU is the intersection over union between predicted box and ground truth data. The confidence score is zero if the bounding box contains no object. Each bounding box contains five predictions—(i) X-coordinate, (ii) Y-coordinate, (iii) Height, (iv) Width, and (v) Confidence. Each grid cell computes class probability Pr(Classi |Object). At the time of testing, the class probability is multiplied with confidence score of each bounding box of that grid. That provides the classification accuracy of each object. Equation 2 shows how the classification score is calculated. truth Pr(Classi |Object) ∗ Pr(Object) ∗ IOUtruth pred = Pr(Classi ) ∗ IOUpred

(2)

3.2 Network Architecture The basic YOLO model has 24 convolutional layers, followed by two fully connected layers. The network consists of 1 × 1 reduction layers followed by 3 × 3 convolutional layers. The basic YOLO model is further modified for better accuracy and speed. In our work, we have used YOLO v3 model [24]. This model is more robust and accurate than the basic YOLO model. The YOLO v3 model is hybrid of some models. They are YOLO v2, Darknet-19, and some portion of residual network. There are shortcut connections in the layers. In this network, 3 × 3 and 1 × 1 convolutional layers are used like basic YOLO model. However, the total number of CNN layers used in the YOLO v3 model is 53. Which is very much higher than the layers used in YOLO basic model. The YOLO v3 is more powerful and accurate than YOLO basic model. Instead of containing more layers than the basic model, the YOLO v3 is faster than the basic model. Table 1 describes the details of network architecture of YOLO v3 model.

126

R. Dey et al.

Table 1 Network architecture of YOLO v3 model

1×

Type

No. of filters

Size of filters

Output

Convolution

32

3×3

256 × 256 128 × 128

Convolution

64

3 × 3/2

Convolution

32

1×1

Convolution

64

3×3

Convolution

128

3 × 3/2

Convolution

64

1×1

Convolution

128

3×3

Convolution

256

3 × 3/2

Convolution

128

1×1

Convolution

256

3×3

128 × 128

Residual 2×

64 × 64

Residual 8×

Convolution

512

3 × 3/2

Convolution

256

1×1

Convolution

512

3×3

Convolution

1024

3 × 3/2

Convolution

512

1× 1

Convolution

1024

3×3

16 × 16

16 × 16

Residual 4×

32 × 32

32 × 32

Residual 8×

64 × 64

8×8

8×8

Residual Average pooling

Global

Connected layer

1000

Softmax classifier

3.3 Transfer Learning Transfer learning is an approach of machine learning where one model generated for a specific task can be used in another task. This technique ensures the multi-tasking of a generated model. The form of transfer learning used in deep learning is called inductive transfer. In this approach, an existing deep neural model is used in a different but related task. In this work, we are using a pre-trained YOLO v3 model in our work. Our experimental database is very less according to in-depth learning requirements. Therefore, we are using the pre-trained model. The pre-trained model is further trained by the images distorted by the rain. The deep learning-based approaches are a data-driven procedure. The model gradually learns from training data samples. We have tried to learn YOLO v3 model by rain-distorted images. We have chosen the YOLO v3 model over basic YOLO model because the YOLO v3 model consists of

Object Detection in Rainy Condition from Video Using YOLO …

127

Residual layers, shown in Table 1. The layer computes de-convolution operations, which upsamples the tensor. As a result, the data that can be missed by downsampling is regained further. This mechanism is handy in case of object detection from distorted images, because the distorted images contain very less information. Therefore, it is necessary to upsample the tensor periodically to get proper results.

3.4 Training of the Network The final layer of the architecture predicts the bounding box and calculates class probability. Therefore, the linear activation function is used for the final layer. The Leaky rectified linear activation function is used for other layers. Equation 3 defines the Leaky rectified linear activation function.  ∅(x) =

x, if x > 0 0.1x, otherwise

(3)

This model uses a sum square error in the output for optimization. It equals weight errors for large boxes and small boxes. Therefore, the square root of the bounding box width and height is predicted instead of the width and height directly. The model predicts multiple bounding boxes from each grid cell. At the time of training, the model tries to find one box predictor to detect objects. The predictor with the highest IOU with the ground truth in current time is responsible for detection objects. This is the way for specialization of detectors. The predictors learn more and more features and become accurate by several pieces of training. Equation 4 shows multipart loss function used for optimization in the time of training. B S   2

λcoord

2  2 obj  πij xi − xˆi + yi − yˆt

i=0 j=0 S  B  2

+ λcoord

obj

πij

2 √ 2  wi − wˆ l + h i − hˆ l

i=0 j=0 B S   2

+

obj  πji ci

− cˆi

i=0 j=0 S  2

+

i=0

obj

πij

2

B S   2

+ λnoobj

i=0 j=0

  c ∈ classes

Pi (c) − Pˆι (c)

2

obj  πji ci

− cˆl

2

.

(4)

128

R. Dey et al.

4 Result and Discussion We have trained the model with video frames extracted from the video dataset. The dataset consists of surveillance videos of rain with duration 3–5 min each. The videos are captured in the rainy condition in daytime and nighttime. The database consists of 600 video clips. We have extracted frames from the video clips. Then makes ground truth using LabelImg [25] tool. We have trained the network with frames extracted from videos. We have used 10,000 frames to train the network. We have tested the model using rainy video clips from our dataset. Our proposed model correctly detects most of the objects present in the video clips. A bounding box is drawn on the detected objects. The colors of bounding boxes are specific for specific classes. The bounding box is blue for person; the box is magenta for car and violet for motorcycle. Classification scores of detected objects are shown with the bounding boxes. The value of the classification score is between 0 and 1. Figure 1 shows some screenshot of detection

(a) Objects detected in the day time

(b) Objects detected in the night time Fig. 1 Screenshot extracted in time of testing from video clips

Object Detection in Rainy Condition from Video Using YOLO … Table 2 Comparison of basic YOLO and our proposed model

129

Basic YOLO (%)

Proposed model inspired by YOLO v3 (%)

Precision

78.44

89.23

Recall

74.33

84.65

of objects from videos using our model. In the figure, it is shown that the model can correctly identify objects in both daytime and nighttime. We have calculated Precision and Recall of the output. This performance evolution metrics are useful to predict the performance of a system. We have calculated Precision and Recall by Eqs. 5 and 6 where FP = False Positive, TP = True Positive, FN = False Negative. Precision = Recall =

TP TP + FP

TP TP + FN

(5) (6)

We have compared the precision and recall of our model (YOLO v3) with basic YOLO model. Table 2 shows the comparison. It is shown that the proposed model inspired by YOLO v3 performs much better than basic YOLO model. Therefore, we can conclude the discussion by stating that our proposed deep learning-based model with YOLO v3 can detect objects successfully from video frames distorted by rain.

5 Conclusion We have detected objects in rainy conditions from videos. CCTV video footage is the input of our system, and by the proposed system we can detect objects successfully in rainy conditions. Object detection in normal vision is a broad research area where many researchers contributed as their way. However, object detection in rainy conditions is an untouched paradigm. Much work has been done on rain image modeling, image de-raining, etc. However, no specific work on object detection is found in the literature. In our approach, we have used YOLO v3 model as the basic model and use transfer learning approach to detect object from videos in rainy conditions. Our contribution is very prime in this area, and we get excellent results. However, there are such complex conditions where our system fails to detect objects. We are working on it, and we believe that in future we can get improvements in this research area. Acknowledgements The authors are thankful to the RUSA 2.0 at Jadavpur University for supporting this work.

130

R. Dey et al.

References 1. Terrorism—Statistics and Facts: Statista Research Department, 4 July 2019. https://www. statista.com/topics/2267/terrorism/. Accessed by 30 Aug 2019 2. Luna, E., San Miguel, J.C., Ortego, D., Martínez, J.M.: Abandoned object detection in videosurveillance: survey and comparison. Sensors 18, 4290. https://doi.org/10.3390/s18124290, 5 Dec 2018 3. Ojha, S., Sakhare, S.: Image processing techniques for object tracking in video surveillance—a survey. In: Proceedings of International Conference on Pervasive Computing (ICPC). https:// doi.org/10.1109/pervasive.2015.7087180, Jan 2015 4. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: ICCV’15 Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1440–1448, 07–13 Dec 2015 5. Girshick, R.: Fast R-CNN. In: ICCV’15 Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1440–1448, 07–13 Dec 2015 6. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: NIPS’15 Proceedings of the 28th International Conference on Neural Information Processing Systems, vol. 1, pp. 91–99, 07–12 Dec 2015 7. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 12 Dec 2016 8. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., Berg, A.C.: SSD: single shot multibox detector. https://doi.org/10.1007/978-3-319-46448-0_2, arXiv:1512.02325[cs.CV] 9. Garg, K., Nayar, S.K.: Vision and rain. Int. J. Comput. Vis. Springer. https://doi.org/10.1007/ s11263-006-0028-6 (2007) 10. Andsager, K., Beard, K.V., Laird, N.F.: Laboratory measurements of axis ratios for large raindrops. J. Atmos. Sci. 56, 2673–2683 (1999) 11. Langer, M.S., Zhang, L., Klein, A.W., Bhatia, A., Pereira, J., Rekhi, D.: A spectral-particle hybrid method for rendering falling snow. Rendering Tech., 217–226 (2004) 12. Wang, N., Wade, B.: Rendering falling rain and snow. In: SIGGRAPH (sketches 0186) (2004) 13. Starik, K., Werman, M.: Simulation of rain in videos. In: Texture Workshop, ICCV (2003) 14. Fu, X., Huang, J., Ding, X., Liao, Y., Paisley, J.: Clearing the skies: a deep network architecture for single-image rain removal. https://doi.org/10.1109/tip.2017.2691802, arXiv:1609.02087 (2017) 15. Zhang, H., Sindagi, V., Patel, V.M.: Image de-raining using a conditional generative adversarial network. arXiv:1701.05957 (2019) 16. Xia, H., Zhuge, R., Li, H., Song, S., Jiang, F., XU, M.: Single image rain removal via a simplified residual dense network. IEEE Access 6. https://doi.org/10.1109/access.2018.2879330 (2018) 17. Ren, D., Zuo, W., Hu, Q., Zhu, P., Meng, D.: Progressive image deraining networks: a better and simpler baseline. arXiv:1901.09221 (2019) 18. Peng, Q., Luo, W., Hong, G., Feng, M., Xia, Y., Yu, L., Hao, X., Wang, X., Li, M.: Pedestrian detection for transformer substation based on gaussian mixture model and YOLO. In: 2016 8th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC), 15 Dec 2016 19. Ren, P., Fang, W., Djahel, S.: A novel YOLO-Based real-time people counting approach. In: 2017 International Smart Cities Conference (ISC2), 02 Nov 2017 20. Lan, W., Dang, J., Wang, Y., Wang, S.: Pedestrian detection based on YOLO network model. In: 2018 IEEE International Conference on Mechatronics and Automation (ICMA), 08 Oct 2018 21. Yang, W., Jiachun, Z.: Real-time face detection based on YOLO. In: 2018 1st IEEE International Conference on Knowledge Innovation and Invention (ICKII), 10 Dec 2018 22. Widyastuti, R., Yang, C.K.: Cat’s nose recognition using you only look once (YOLO) and scale-invariant feature transform (SIFT). In: 2018 IEEE 7th Global Conference on Consumer Electronics (GCCE), 13 Dec 2018

Object Detection in Rainy Condition from Video Using YOLO …

131

23. Lin, J.P., Sun, M.T.: A YOLO-based traffic counting system. In: 2018 Conference on Technologies and Applications of Artificial Intelligence (TAAI), 27 Dec 2018 24. Redmon, J., Farhadi, A.: YOLOv3: an incremental improvement. arXiv:1804.02767 (2018) 25. https://github.com/tzutalin/labelImg. Last accessed 12 Aug 2019

Visualization of the Evacuation Process Including People with Heart Diseases Łukasz Szymkowski, Khalid Saeed and Piotr Szymkowski

Abstract Problem of conducting safe evacuations is complicated. When people are in foreign environment, they usually are not interested in the placement of furnishes in the facility or evacuation plans. Often, even in well-marked rooms, they are unable to indicate places where the fire extinguisher or AED device is located. During dangerous events such as fires or floods, they panic, which is making it difficult to safely leave the dangerous zones. Therefore, systems which can inform people about escape routes and how to move in a way that allows safe and efficient exit from the hazardous areas are essential. During the creation of such a system, there must be taken into account many factors such as: the type of threat, the impact of the obstacle on escape routes, a safe distance between people, the pace of movement, the flow of information between witnesses of the event, minimizing injury to participants. Presented solution allows crowd simulation, taking into account people with various heart diseases and establishing a safe way to evacuate. Keywords Evacuation process · Heart disease · Rescue process of fallen people

1 Introduction Currently more attention is being paid to safety in crowded places because of the higher risk of accidents. This is due to many factors, including: lack of information about available safety measures and their location, escape routes, how to behave during the evacuation, who is responsible for leading the evacuation process. In Poland, there were more than 7100 accidents during mass events in year 2018, according to

Ł. Szymkowski (B) · K. Saeed · P. Szymkowski Faculty of Computer Science, Bialystok University of Technology, Białystok, Poland K. Saeed e-mail: [email protected] © Springer Nature Singapore Pte Ltd. 2020 R. Chaki et al. (eds.), Advanced Computing and Systems for Security, Advances in Intelligent Systems and Computing 1136, https://doi.org/10.1007/978-981-15-2930-6_11

133

134

Ł. Szymkowski et al.

the BIM (Safety During Mass Events) report. Statistically, there are fewer accidents than in previous years; however, it does not mean that this result is satisfactory. The most important attribute is reaction time and speed of decision making during an emergency. When there is a predefined route to the exit, which is known to all the people, the sense of security and calm people involved in the accident can be provided. Therefore, a system that would solve the problem of determining the fastest, safe route to exit for each participant is required. The system should take into account the place where the threat occurs, its impact on evacuation, the speed of the phenomenon, the average speed of movement of a person and his position relative to other people. Unity is an engine adapted to create games, animations and simulations. It has many built-in functions and tools that streamline project creation. One of them is pathfinding based on the A* algorithm for searching the shortest path. In addition, it is possible to define elements of the world as obstacles, or zones with more or less weight, which tell us about whether a given path will be used. This algorithm is sufficient to set evacuation paths. Simulations of life-threatening situations help to identify deficiencies in the preparation of a secure event, building security, escape routes and allow for the elimination of errors earlier. People with heart disease need more attention during evacuation. Any greater pressure from the crowd can trigger the disease symptoms and in extreme cases, the person can block the passage without being able to move. These people can be monitored through smart bands that can be paired with a Bluetooth device. Information processing can be entrusted to a computer system under the supervision of a person with medical education and trained in the use of this system. ECG test The ECG signal is the most frequently interpreted signal showing cardiovascular defects (Fig. 1). The record of the study is a graph in which can be observed the oscillations that shows the stimulation of the heart. In recent years, technological advances have allowed for a simplified ECG test in miniaturized devices. Smartbands or smartwatches often have functions such as measuring blood pressure, pulse, and recording heart rhythm. These data can be transferred to an external device equipped with a Bluetooth module. People with a suspected heart defect can obtain such devices and share access to this data to institutions to prevent an accident in times of danger, or to provide assistance faster. ECG—waveform analysis Waves and intervals in ECG correspond to the following description: P wave PR interval

atrial depolarization interval between onset depolarization and onset of depolarization of ventricles QRS complex depolarization of ventricles Section E break in electrical activity of chambers before repolarization phase T wave repolarization ventricles

Visualization of the Evacuation Process Including People …

135

Fig. 1 Example of idealized ECG waveform fragment [1]

QT interval U-wave

the total time of ventricular depolarization and their repolarization unclear, it can be repolarization of the ventricular septum or slow ventricular repolarization.

Blood pressure and heart rate Blood pressure and heart rate are easier to measure than ECG signal, but it gives less information about the individual type of disease that people have. However, it can give an impulse to react if someone is in bad shape. The abnormalities in pressure and heart rate can lead to fainting or risk of heart attack. Normal blood pressure for an adult person is 120–139/80–89 mm Hg and heart rate is 66–76 bpm.

2 State of the Art Currently conducted research in the field of simulation of evacuation involves finding solutions on how people move during an emergency. Usually, a square grid of relatively small sizes is used (Fig. 2). This helps to control how pedestrians behave. The rooms prepared as examples of algorithms are usually one-level [2].

2.1 Area of Visibility Problem The phenomenon of a disaster can have important impact on the field of view to other participants in the evacuation. Basic safety measures like: visibility of exit signs, fire extinguishers and information about obstacles have an impact on human behavior.

136

Ł. Szymkowski et al.

Fig. 2 Visualization of square grid with possible directions of movement of the people

Very often the trends in movement of the crowd can be described—following the leader or the phenomenon of panic. The problem is solved in the following way [3]: • Each person can move only one space in grid, • When moving, he can occupy the adjacent space and choose the highest value field. If there is more than one field of the same value, it is selected at random, the • Field can be visited by only one person at a time. When there are more candidates for a given place, it is chosen randomly with the same probability. The rest are waiting for their positions, • When two people want to switch places, this is an acceptable operation, • When a person heads for the field marked with an exit, in the next step he leaves the room, leaves the room, the • When everyone gets out, program ends.

2.2 Information Exchange Problem The problem of information exchange between people during evacuation is essential for human safety. In a large area, it is important to recognize the threat, gather knowledge about blockades and obstacles on our way. The problem was solved by exchanging information between evacuation participants. Each of them, if they encountered an obstacle on their way, they are informing another person about it (Fig. 3). Methods used in the algorithms [4]: • Spray and Wait • Epidemic Routing • Max-Prop

Visualization of the Evacuation Process Including People …

137

Fig. 3 Example of communication process during evacuation [4]

This problem can also be solved by creating an Information Distribution System, which will inform the people in the dangerous areas. System should use wireless connection like Bluetooth [5]. The data would be sent to each device in short intervals, giving the required information about evacuation plan with updated positions of blocked paths.

2.3 Real-Time Simulation Problem Based on real-time simulation and route calculation is a key element for simulating the evacuation process on given area. During the movement, it is noticeable that people are attracted to each other by the social element of participating in the event and having a common goal in the group. The solution to this problem is to give strength parameter (attracting or repulsive) to individuals in the algorithm [6, 7].

138

Ł. Szymkowski et al.

3 Algorithm of the Authors The program has been prepared and executed in a way that allows to perform tasks in real time. It consists of the following parts: evacuating all of the healthy people (Fig. 4), checking the pulse and pressure of individual person, sound and light signaling of the injured ones and simulation of taking the people which are lying down out of the room by the rescue team (Fig. 5). Example scenarios are built in 2D space. To determine the paths that evacuation participants can navigate, he uses the navmesh grid and the A* algorithm. On one example scene, there are four exits with a width about twice the width of the human. During the evacuation, it is assumed that each participant knows the location of the exits and assesses which of them is closest. Everyone heads to the given exit, checking whether the given door is blocked (Fig. 6). When he is unable to pass through them, the next nearest exit is chosen. An injured person with heart disease is monitored—information of his pulse and blood pressure is collected. When it is inappropriate, a sound and light signal is sent to help locating and informing about danger to participants life. The algorithm includes a solution according to which help is provided by the two closest people. Evacuation participants or security personnel can be designated to help (Figs. 7 and 8). When the rescue people reach the victim, they lead him to the nearest exit (Fig. 9).

Fig. 4 Diagram of the healthy person’s response during evacuation

Visualization of the Evacuation Process Including People …

Fig. 5 Diagram of rescue for the fallen person

Fig. 6 Example of behavior during evacuation—injured person marked with an arrow

139

140

Fig. 7 Evacuation of an injured person

Fig. 8 Recrutation of the rescue team members (pseudocode)

Ł. Szymkowski et al.

Visualization of the Evacuation Process Including People …

141

Fig. 9 Gathering information about nearest exits and setting the escape path (pseudocode)

4 Conclusions The main aim of this study was to create an algorithm which can help in designing more secure environments for people with heart diseases. Program, which was a result of this scientific publication, is able to simulate dangerous situations—the customization of the room is needed. There is also an option for including obstacles and monitoring the heart rate of each person. The feature that differentiates this solution from other projects is the versatility of constructing the new rooms and ease of upgrade the code for new use cases. The knowledge gathered during this study can become a basis for further development. The next step in study will be gathering information about health during events and making evacuation process more accessible for visually and hearing-impaired people.

References 1. Gawronski, P., Saeed, K., Kulakowski, K.: Early warning of cardiac problems in a crowd, agent and multi-agent systems. In: Technologies and Applications 4th KES International Symposium, KES-AMSTA 2010, part 2, pp. 220–228 (2010) 2. Sarma, S.S., Das, S.R., Sinha, K., Sinha, B.P.: Fast transportation in a disaster situation along real-life grid-structured road networks. In: 2019 IEEE 90th Vehicular Technology Conference

142

Ł. Szymkowski et al.

(2019) 3. Hao, Y., Hongzhi, G., Chunfu, S., Yinhong, L.: Simulation of pedestrian evacuation with affected visual field and absence of evacuation signs. In: 2010 Sixth International Conference on Natural Computation (ICNC 2010), pp. 4286–4290 (2010) 4. Takayama, Y., Miwa, H.: Quick evacuation method for evacuation navigation system in poor communication environment at the time of disaster. In: International Conference on Intelligent Networking and Collaborative Systems, pp. 415–420 (2014) 5. Utsunomiya, E.: Information propagation that induces evacuation behavior at the time of disaster. In: IEEE Intelligence Conference on Dependable, Autonomic and Secure Computing, Intelligence Conference on Pervasive Intelligence and Computing, Intelligence Conference on Cloud and Big Data Computing, Intelligence Conference on Cyber Science and Technology Congress, pp. 681–686 (2019) 6. Yadong, M., Chongqiang, J.: Research and application of metro station evacuation simulation. In: 4th International Conference on Information Science and Control Engineering, pp. 1123–1125 (2017) 7. Wu, Q., Liu, S., Zhu, Z., Zhang, H.: Research on evacuation strategies to prevent stampedes in large-scale events. In: International Conference on Industrial Informatics—Computing Technology, Intelligent Technology, Industrial Information Integration, pp. 158–161 (2016)