Proceedings of ELM 2018 [1st ed.] 978-3-030-23306-8;978-3-030-23307-5

Table of contents :
Front Matter ....Pages i-viii
Random Orthogonal Projection Based Enhanced Bidirectional Extreme Learning Machine (Weipeng Cao, Jinzhu Gao, Xizhao Wang, Zhong Ming, Shubin Cai)....Pages 1-10
A Novel Feature Specificity Enhancement for Taste Recognition by Electronic Tongue (Yanbing Chen, Tao Liu, Jianjun Chen, Dongqi Li, Mengya Wu)....Pages 11-16
Comparison of Classification Methods for Very High-Dimensional Data in Sparse Random Projection Representation (Anton Akusok, Emil Eirola)....Pages 17-26
A Robust and Dynamically Enhanced Neural Predictive Model for Foreign Exchange Rate Prediction (Lingkai Xing, Zhihong Man, Jinchuan Zheng, Tony Cricenti, Mengqiu Tao)....Pages 27-36
Alzheimer’s Disease Computer Aided Diagnosis Based on Hierarchical Extreme Learning Machine (Zhongyang Wang, Junchang Xin, Yue Zhao, Qiyong Guo)....Pages 37-44
Key Variables Soft Measurement of Wastewater Treatment Process Based on Hierarchical Extreme Learning Machine (Feixiang Zhao, Mingzhe Liu, Binyang Jia, Xin Jiang, Jun Ren)....Pages 45-54
A Fast Algorithm for Sparse Extreme Learning Machine (Zhihong Miao, Qing He)....Pages 55-64
Extreme Latent Representation Learning for Visual Classification (Tan Guo, Lei Zhang, Xiaoheng Tan)....Pages 65-75
An Optimized Data Distribution Model for ElasticChain to Support Blockchain Scalable Storage (Dayu Jia, Junchang Xin, Zhiqiong Wang, Wei Guo, Guoren Wang)....Pages 76-85
An Algorithm of Sina Microblog User’s Sentimental Influence Analysis Based on CNN+ELM Model (Donghong Han, Fulin Wei, Lin Bai, Xiang Tang, TingShao Zhu, Guoren Wang)....Pages 86-97
Extreme Learning Machine Based Intelligent Condition Monitoring System on Train Door (Xin Sun, K. V. Ling, K. K. Sin, Lawrence Tay)....Pages 98-107
Character-Level Hybrid Convolutional and Recurrent Neural Network for Fast Text Categorization (Bing Liu, Yong Zhou, Wei Sun)....Pages 108-117
Feature Points Selection for Rectangle Panorama Stitching (Weiqing Yan, Shuigen Wang, Guanghui Yue, Jindong Xu, Xiangrong Tong, Laihua Wang)....Pages 118-124
Point-of-Interest Group Recommendation with an Extreme Learning Machine (Zhen Zhang, Guoren Wang, Xiangguo Zhao)....Pages 125-133
Research on Recognition of Multi-user Haptic Gestures (Lu Fang, Huaping Liu, Yanzhi Dong)....Pages 134-143
Benchmarking Hardware Accelerating Techniques for Extreme Learning Machine (Liang Li, Guoren Wang, Gang Wu, Qi Zhang)....Pages 144-153
An Event Recommendation Model Using ELM in Event-Based Social Network (Boyang Li, Guoren Wang, Yurong Cheng, Yongjiao Sun)....Pages 154-162
Reconstructing Bifurcation Diagrams of a Chaotic Neuron Model Using an Extreme Learning Machine (Yoshitaka Itoh, Masaharu Adachi)....Pages 163-172
Extreme Learning Machine for Multi-label Classification (Haigang Zhang, Jinfeng Yang, Guimin Jia, Shaocheng Han)....Pages 173-181
Accelerating ELM Training over Data Streams (Hangxu Ji, Gang Wu, Guoren Wang)....Pages 182-190
Predictive Modeling of Hospital Readmissions with Sparse Bayesian Extreme Learning Machine (Nan Liu, Lian Leng Low, Sean Shao Wei Lam, Julian Thumboo, Marcus Eng Hock Ong)....Pages 191-196
Rising Star Classification Based on Extreme Learning Machine (Yuliang Ma, Ye Yuan, Guoren Wang, Xin Bi, Zhongqing Wang, Yishu Wang)....Pages 197-206
Hand Gesture Recognition Using Clip Device Applicable to Smart Watch Based on Flexible Sensor (Sung-Woo Byun, Da-Kyeong Oh, MyoungJin Son, Ju Hee Kim, Ye Jin Lee, Seok-Pil Lee)....Pages 207-215
Receding Horizon Optimal Control of Hybrid Electric Vehicles Using ELM-Based Driver Acceleration Rate Prediction (Jiangyan Zhang, Fuguo Xu, Yahui Zhang, Tielong Shen)....Pages 216-225
CO-LEELM: Continuous-Output Location Estimation Using Extreme Learning Machine (Felis Dwiyasa, Meng-Hiot Lim)....Pages 226-235
Unsupervised Absent Multiple Kernel Extreme Learning Machine (Lingyun Xiang, Guohan Zhao, Qian Li, Zijie Zhu)....Pages 236-246
Intelligent Machine Tools Recognition Based on Hybrid CNNs and ELMs Networks (Kun Zhang, Lu-Lu Tang, Zhi-Xin Yang, Lu-Qing Luo)....Pages 247-252
Scalable IP Core for Feed Forward Random Networks (Anurag Daram, Karan Paluru, Vedant Karia, Dhireesha Kudithipudi)....Pages 253-262
Multi-objective Artificial Bee Colony Algorithm with Information Learning for Model Optimization of Extreme Learning Machine (Hao Zhang, Dingyi Zhang, Tao Ku)....Pages 263-272
Short Term PV Power Forecasting Using ELM and Probabilistic Prediction Interval Formation (Jatin Verma, Xu Yan, Junhua Zhao, Zhao Xu)....Pages 273-282
A Novel ELM Ensemble for Time Series Prediction (Zhen Li, Karl Ratner, Edward Ratner, Kallin Khan, Kaj-Mikael Bjork, Amaury Lendasse)....Pages 283-291
An ELM-Based Ensemble Strategy for POI Recommendation (Xue He, Tiancheng Zhang, Hengyu Liu, Ge Yu)....Pages 292-302
A Method Based on S-transform and Hybrid Kernel Extreme Learning Machine for Complex Power Quality Disturbances Classification (Chen Zhao, Kaicheng Li, Xuebin Xu)....Pages 303-318
Sparse Bayesian Learning for Extreme Learning Machine Auto-encoder (Guanghao Zhang, Dongshun Cui, Shangbo Mao, Guang-Bin Huang)....Pages 319-327
A Soft Computing-Based Daily Rainfall Forecasting Model Using ELM and GEP (Yuzhong Peng, Huasheng Zhao, Jie Li, Xiao Qin, Jianping Liao, Zhiping Liu)....Pages 328-335
Comparing ELM with SVM in the Field of Sentiment Classification of Social Media Text Data (Zhihuan Chen, Zhaoxia Wang, Zhiping Lin, Ting Yang)....Pages 336-344
Back Matter ....Pages 345-347

Citation preview

Proceedings in Adaptation, Learning and Optimization 11

Jiuwen Cao Chi Man Vong Yoan Miche Amaury Lendasse Editors

Proceedings of ELM 2018

Proceedings in Adaptation, Learning and Optimization Volume 11

Series Editors Meng-Hiot Lim, Nanyang Technological University, Singapore, Singapore Yew Soon Ong, Nanyang Technological University, Singapore, Singapore

The role of adaptation, learning and optimization are becoming increasingly essential and intertwined. The capability of a system to adapt either through modification of its physiological structure or via some revalidation process of internal mechanisms that directly dictate the response or behavior is crucial in many real world applications. Optimization lies at the heart of most machine learning approaches while learning and optimization are two primary means to effect adaptation in various forms. They usually involve computational processes incorporated within the system that trigger parametric updating and knowledge or model enhancement, giving rise to progressive improvement. This book series serves as a channel to consolidate work related to topics linked to adaptation, learning and optimization in systems and structures. Topics covered under this series include: • complex adaptive systems including evolutionary computation, memetic computing, swarm intelligence, neural networks, fuzzy systems, tabu search, simulated annealing, etc. • machine learning, data mining & mathematical programming • hybridization of techniques that span across artificial intelligence and computational intelligence for synergistic alliance of strategies for problem-solving • aspects of adaptation in robotics • agent-based computing • autonomic/pervasive computing • dynamic optimization/learning in noisy and uncertain environment • systemic alliance of stochastic and conventional search techniques • all aspects of adaptations in man-machine systems. This book series bridges the dichotomy of modern and conventional mathematical and heuristic/meta-heuristics approaches to bring about effective adaptation, learning and optimization. It propels the maxim that the old and the new can come together and be combined synergistically to scale new heights in problemsolving. To reach such a level, numerous research issues will emerge and researchers will find the book series a convenient medium to track the progresses made. ** Indexing: The books of this series are submitted to ISI Proceedings, DBLP, Google Scholar and Springerlink **

More information about this series at http://www.springer.com/series/13543

Jiuwen Cao Chi Man Vong Yoan Miche Amaury Lendasse •

•

•

Editors

Proceedings of ELM 2018

123

Editors Jiuwen Cao Institute of Information and Control Hangzhou Dianzi University Xiasha, Hangzhou, China Yoan Miche Nokia Bell Labs Cybersecurity Research Espoo, Finland

Chi Man Vong Department of Computer and Information Science University of Macau Taipa, Macao Amaury Lendasse Department of Information and Logistics Technology University of Houston Houston, TX, USA

ISSN 2363-6084 ISSN 2363-6092 (electronic) Proceedings in Adaptation, Learning and Optimization ISBN 978-3-030-23306-8 ISBN 978-3-030-23307-5 (eBook) https://doi.org/10.1007/978-3-030-23307-5 © Springer Nature Switzerland AG 2020 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Contents

Random Orthogonal Projection Based Enhanced Bidirectional Extreme Learning Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Weipeng Cao, Jinzhu Gao, Xizhao Wang, Zhong Ming, and Shubin Cai

1

A Novel Feature Specificity Enhancement for Taste Recognition by Electronic Tongue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yanbing Chen, Tao Liu, Jianjun Chen, Dongqi Li, and Mengya Wu

11

Comparison of Classification Methods for Very High-Dimensional Data in Sparse Random Projection Representation . . . . . . . . . . . . . . . . Anton Akusok and Emil Eirola

17

A Robust and Dynamically Enhanced Neural Predictive Model for Foreign Exchange Rate Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . Lingkai Xing, Zhihong Man, Jinchuan Zheng, Tony Cricenti, and Mengqiu Tao

27

Alzheimer’s Disease Computer Aided Diagnosis Based on Hierarchical Extreme Learning Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zhongyang Wang, Junchang Xin, Yue Zhao, and Qiyong Guo

37

Key Variables Soft Measurement of Wastewater Treatment Process Based on Hierarchical Extreme Learning Machine . . . . . . . . . . . . . . . . Feixiang Zhao, Mingzhe Liu, Binyang Jia, Xin Jiang, and Jun Ren

45

A Fast Algorithm for Sparse Extreme Learning Machine . . . . . . . . . . . Zhihong Miao and Qing He

55

Extreme Latent Representation Learning for Visual Classification . . . . Tan Guo, Lei Zhang, and Xiaoheng Tan

65

An Optimized Data Distribution Model for ElasticChain to Support Blockchain Scalable Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dayu Jia, Junchang Xin, Zhiqiong Wang, Wei Guo, and Guoren Wang

76

v

vi

Contents

An Algorithm of Sina Microblog User’s Sentimental Influence Analysis Based on CNN+ELM Model . . . . . . . . . . . . . . . . . . . . . . . . . . Donghong Han, Fulin Wei, Lin Bai, Xiang Tang, TingShao Zhu, and Guoren Wang Extreme Learning Machine Based Intelligent Condition Monitoring System on Train Door . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xin Sun, K. V. Ling, K. K. Sin, and Lawrence Tay

86

98

Character-Level Hybrid Convolutional and Recurrent Neural Network for Fast Text Categorization . . . . . . . . . . . . . . . . . . . . . . . . . . 108 Bing Liu, Yong Zhou, and Wei Sun Feature Points Selection for Rectangle Panorama Stitching . . . . . . . . . . 118 Weiqing Yan, Shuigen Wang, Guanghui Yue, Jindong Xu, Xiangrong Tong, and Laihua Wang Point-of-Interest Group Recommendation with an Extreme Learning Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 Zhen Zhang, Guoren Wang, and Xiangguo Zhao Research on Recognition of Multi-user Haptic Gestures . . . . . . . . . . . . 134 Lu Fang, Huaping Liu, and Yanzhi Dong Benchmarking Hardware Accelerating Techniques for Extreme Learning Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 Liang Li, Guoren Wang, Gang Wu, and Qi Zhang An Event Recommendation Model Using ELM in Event-Based Social Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 Boyang Li, Guoren Wang, Yurong Cheng, and Yongjiao Sun Reconstructing Bifurcation Diagrams of a Chaotic Neuron Model Using an Extreme Learning Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 Yoshitaka Itoh and Masaharu Adachi Extreme Learning Machine for Multi-label Classification . . . . . . . . . . . 173 Haigang Zhang, Jinfeng Yang, Guimin Jia, and Shaocheng Han Accelerating ELM Training over Data Streams . . . . . . . . . . . . . . . . . . . 182 Hangxu Ji, Gang Wu, and Guoren Wang Predictive Modeling of Hospital Readmissions with Sparse Bayesian Extreme Learning Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 Nan Liu, Lian Leng Low, Sean Shao Wei Lam, Julian Thumboo, and Marcus Eng Hock Ong Rising Star Classification Based on Extreme Learning Machine . . . . . . 197 Yuliang Ma, Ye Yuan, Guoren Wang, Xin Bi, Zhongqing Wang, and Yishu Wang

Contents

vii

Hand Gesture Recognition Using Clip Device Applicable to Smart Watch Based on Flexible Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 Sung-Woo Byun, Da-Kyeong Oh, MyoungJin Son, Ju Hee Kim, Ye Jin Lee, and Seok-Pil Lee Receding Horizon Optimal Control of Hybrid Electric Vehicles Using ELM-Based Driver Acceleration Rate Prediction . . . . . . . . . . . . . . . . . . 216 Jiangyan Zhang, Fuguo Xu, Yahui Zhang, and Tielong Shen CO-LEELM: Continuous-Output Location Estimation Using Extreme Learning Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 Felis Dwiyasa and Meng-Hiot Lim Unsupervised Absent Multiple Kernel Extreme Learning Machine . . . . 236 Lingyun Xiang, Guohan Zhao, Qian Li, and Zijie Zhu Intelligent Machine Tools Recognition Based on Hybrid CNNs and ELMs Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 Kun Zhang, Lu-Lu Tang, Zhi-Xin Yang, and Lu-Qing Luo Scalable IP Core for Feed Forward Random Networks . . . . . . . . . . . . . 253 Anurag Daram, Karan Paluru, Vedant Karia, and Dhireesha Kudithipudi Multi-objective Artificial Bee Colony Algorithm with Information Learning for Model Optimization of Extreme Learning Machine . . . . . 263 Hao Zhang, Dingyi Zhang, and Tao Ku Short Term PV Power Forecasting Using ELM and Probabilistic Prediction Interval Formation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 Jatin Verma, Xu Yan, Junhua Zhao, and Zhao Xu A Novel ELM Ensemble for Time Series Prediction . . . . . . . . . . . . . . . 283 Zhen Li, Karl Ratner, Edward Ratner, Kallin Khan, Kaj-Mikael Bjork, and Amaury Lendasse An ELM-Based Ensemble Strategy for POI Recommendation . . . . . . . . 292 Xue He, Tiancheng Zhang, Hengyu Liu, and Ge Yu A Method Based on S-transform and Hybrid Kernel Extreme Learning Machine for Complex Power Quality Disturbances Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303 Chen Zhao, Kaicheng Li, and Xuebin Xu Sparse Bayesian Learning for Extreme Learning Machine Auto-encoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319 Guanghao Zhang, Dongshun Cui, Shangbo Mao, and Guang-Bin Huang

viii

Contents

A Soft Computing-Based Daily Rainfall Forecasting Model Using ELM and GEP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328 Yuzhong Peng, Huasheng Zhao, Jie Li, Xiao Qin, Jianping Liao, and Zhiping Liu Comparing ELM with SVM in the Field of Sentiment Classification of Social Media Text Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336 Zhihuan Chen, Zhaoxia Wang, Zhiping Lin, and Ting Yang Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345

Random Orthogonal Projection Based Enhanced Bidirectional Extreme Learning Machine Weipeng Cao1, Jinzhu Gao2, Xizhao Wang1, Zhong Ming1(&), and Shubin Cai1 1

College of Computer Science and Software Engineering, Shenzhen University, Shenzhen 518060, China [email protected], {xzwang,mingz, shubin}@szu.edu.cn 2 School of Engineering and Computer Science, University of the Pacific, Stockton 95211, CA, USA [email protected]

Abstract. Bidirectional extreme learning machine (B-ELM) divides the learning process into two parts: At odd learning step, the parameters of the new hidden node are generated randomly, while at even learning step, the parameters of the new hidden node are obtained analytically from the parameters of the former node. However, some of the odd-hidden nodes play a minor role, which will have a negative impact on the even-hidden nodes, and result in a sharp rise in the network complexity. To avoid this issue, we propose a random orthogonal projection based enhanced bidirectional extreme learning machine algorithm (OEB-ELM). In OEB-ELM, several orthogonal candidate nodes are generated randomly at each odd learning step, only the node with the largest residual error reduction will be added to the existing network. Experiments on six real datasets have shown that the OEB-ELM has better generalization performance and stability than B-ELM, EB-ELM, and EI-ELM algorithms. Keywords: Extreme learning machine
Bidirectional extreme learning machine
Random orthogonal projection

1 Introduction The training mechanism of traditional single hidden layer feed-forward neural networks (SLFN) is that the input weights and hidden bias are randomly assigned initial values and then iteratively tuned with methods such as gradient descent until the residual error reaches the expected value. This method has several notorious drawbacks such as slow convergence rate and local minima problem. Different from traditional SLFN, neural networks with random weights (NNRW) train models in a non-iterative way [1, 2]. In NNRW, the input weights and hidden bias are randomly generated from a given range and kept fixed throughout the training

© Springer Nature Switzerland AG 2020 J. Cao et al. (Eds.): ELM 2018, PALO 11, pp. 1–10, 2020. https://doi.org/10.1007/978-3-030-23307-5_1

2

W. Cao et al.

process, while the output weights are obtained by solving a linear system of matrix equations. Compared with traditional SLFN, NNRW can learn faster with acceptable accuracy. Extreme learning machine (ELM) is a typical NNRW, which was proposed by Huang et al. in 2004 [3]. ELM inherits the advantages of NNRW and extends it to a unified form. In recent years, many ELM based algorithms have been proposed [4–6] and applied to various fields such as unsupervised learning [7] and traffic sign recognition [8]. Although ELM and its variants have achieved many interesting results, there are still several important problems that have not been solved thoroughly, one of which is the determination of the number of hidden nodes [1, 9]. In recent years, many algorithms have been proposed to determine the number of hidden nodes. We can group them into two categories: incremental and pruning strategies. For incremental strategy, the model begins with a small initial network and then gradually adds new hidden nodes until the desired accuracy is achieved. Some notable incremental algorithms include I-ELM [10], EI-ELM [11], B-ELM [12], EB-ELM [13], etc. For pruning strategy, the model begins with a larger than necessary network and then cuts off the redundant or less effective hidden nodes. Some notable destructive algorithms include P-ELM [14], OP-ELM [15], etc. This paper focuses on optimizing the performance of the existing B-ELM algorithm. In B-ELM [12], the authors divided the learning process into two parts: the odd and the even learning steps. At the odd learning steps, the new hidden node is generated randomly at one time, while at the even learning steps, the new hidden node is determined by a formula defined by the former added node parameters. Compared with the fully random incremental algorithms such as I-ELM and EI-ELM, B-ELM shows a much faster convergence rate. From the above analysis, we can infer that the hidden nodes generated at the odd learning steps (the odd-hidden nodes) play an important role in B-ELM models. However, the quality of the odd-hidden nodes cannot be guaranteed. Actually, some of them may play a minor role, which will cause a sharp rise in the network complexity. The initial motivation of this study is to alleviate this issue. Orthogonalization technique is one of the effective algorithms for parameter optimization. Wang et al. [16] proved that the ELM model with the random orthogonal projection has better capability of sample structure preserving (SSP). Kasun et al. [17] stacked the ELM auto-encoders into a deep ELM architecture based on the orthogonal weight matrix. Huang et al. [18] orthogonalized the input weights matrix when building the local receptive fields based ELM model. Inspired by the above works, in this study, we propose a novel random orthogonal projection based enhanced bidirectional extreme learning machine algorithm (OEBELM). In OEB-ELM, at each odd learning step, we first randomly generate K candidate hidden nodes and orthogonalize them into orthogonal hidden nodes based on the GramSchmidt orthogonalization method. Then we train it as an initial model for hidden nodes selection. After obtaining the corresponding value of residual error reduction for each candidate node, the one with the largest residual error reduction will be selected as the final odd-hidden node and added to the existing network. The even-hidden nodes are obtained in the same way as B-ELM and EB-ELM.

Random Orthogonal Projection

3

Our main contributions in this study are as follows. (1) The odd learning steps in B-ELM are optimized and better hidden nodes can be obtained. Compared with the B-ELM, the proposed algorithm achieves models with better generalization performance and smaller network structure. (2) The method to set the number of candidate hidden nodes in EB-ELM is improved. In OEB-ELM, the number of candidate hidden nodes is automatically determined according to data attributes, which can effectively improve the computational efficiency and reduce the human intervention to the model. (3) The random orthogonal projection technique is used to improve the capability of SSP of the candidate hidden nodes selection model, and thus the quality of the hidden nodes is further improved. Experiments on six UCI regression datasets have demonstrated the efficiency of our method. The organization of this paper is as follows: Sect. 2 briefly reviews the related algorithms. The proposed OEB-ELM is described in Sect. 3. The details of the experiment results and analysis are given in Sect. 4. Section 5 concludes the paper.

2 Review of ELM, I-ELM, B-ELM and EB-ELM A typical network structure of ELM with single hidden layer is shown in Fig. 1. The training mechanism of ELM is that the input weights x and hidden bias b are generated randomly from a given range and kept fixed throughout the training process, while the output weights b are obtained by solving a system of matrix equation.

Fig. 1. A basic ELM neural network structure

4

W. Cao et al.

The above ELM network can be modeled as L X

bi gðxi
xj þ bi Þ ¼ tj ; xi 2 Rn ; bi 2 R; j ¼ 1; . . .; N

ð1Þ

i¼1

where gð
Þ denotes the activation function, tj denotes the actual value of each sample, and N is the size of the dataset. Equation (1) can be rewritten as Hb ¼ T 0

where H ¼ B @

gðx1
x1 þ b1 Þ .. .

... .. . gðx1
xN þ b1 Þ




ð2Þ 3 bT1 7 , b¼6 4 ... 5

1

2

gðxL
x1 þ bL Þ C .. A .

gðxL
xN þ bL Þ

NL

bTL

2

3 T

t1 6 .. 7 , T¼4 . 5 . tNT Nm Lm

In Eq. (2), H represents the hidden layer output matrix of ELM and the output weight b can be obtained by b ¼ HþT

ð3Þ

where H þ is the Moore–Penrose generalized inverse of H. The residual error measures the closeness between the current network fn with n hidden nodes and the target function f , which can be summarized as en ¼ f n  f

ð4Þ

In the I-ELM algorithm [10], the random hidden nodes are added to the hidden layer one by one and the parameters of the existing hidden nodes stay the same after a new hidden node is added. The output function fn at the nth step can be expressed by fn ðxÞ ¼ fn1 ðxÞ þ bn Gn ðxÞ

ð5Þ

where bn denotes the output weights between the new added hidden node and the output nodes, and Gn ðxÞ is the corresponding output of the hidden node. The I-ELM can automatically generate the network structure; however, the network structure is often very complex because some of the hidden nodes play a minor role in the network. To alleviate this issue, the EI-ELM [11] and B-ELM [12] algorithms were proposed. The core idea of the EI-ELM algorithm is to generate K candidate hidden nodes at each learning step and only select the one with the smallest residual error. Actually, the I-ELM is a special case of the EI-ELM when K ¼ 1.

Random Orthogonal Projection

5

Different from the EI-ELM, the B-ELM divides the training process into two parts: the odd and the even learning steps. At each odd learning step (i.e., when the number of hidden nodes L 2 f2n þ 1; n 2 Zg), the new hidden node is generated randomly as in the I-ELM. At each even learning step (i.e., when the number of hidden nodes L 2 f2n; n 2 Zg), the parameters of the new hidden node are obtained by

^b2n ¼

^ 2n ¼ g1 ðuðH2n ÞÞx1 x

ð6Þ

pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi mseðg1 ðuðH2n ÞÞ  x2n xÞ

ð7Þ

^ 2n ¼ u1 ðg1 ðx2n x þ b2n ÞÞ H

ð8Þ

where g1 and u1 denote the inverse functions of g and u, respectively. From the training mechanism of the B-ELM mentioned above, we can infer that the hidden nodes generated at the odd learning steps have a significant impact on the model performance. However, the B-ELM cannot guarantee the quality of these hidden nodes. The odd-hidden nodes that play a minor role in the network will cause a sharp rise in the network complexity. To avoid this issue, we proposed the enhanced random search method to optimize the odd learning steps of B-ELM, that is, the EB-ELM algorithm [13]. In EB-ELM, at each odd learning step, K candidate hidden nodes are generated and only the one with the largest residual error reduction will be selected. EB-ELM can achieve better generalization performance than B-ELM. However, the number of candidate nodes K in EB-ELM is assigned based on experience, which makes it difficult to balance the computational efficiency and model performance.

3 The Proposed Algorithm In this section, we propose a random orthogonal projection based enhanced bidirectional extreme learning machine (OEB-ELM) for regression problems. Theorem 1. Suppose WKK is an orthogonal matrix, which satisfies W T W ¼ I, then for any X 2 Rn , jWXjj2 ¼ jjXjj2 . Proof. jWXjj2 ¼ X T W T WX ¼ X T IX ¼ jjXjj2 . From Theorem 1 and its proof, we can infer that the orthogonal projection can provide good capability of sample structure preserving for the initial model, which will improve the performance of the model and ensure the good quality of the candidate nodes. The proposed OEB-ELM can be summarized as follows:

6

W. Cao et al.

OEB-ELM Algorithm: N

n

Input: A training dataset D ={( xi , ti )}i =1 ⊂ R × R , the number of hidden nodes L , an activation function G , a maximum number of hidden nodes Lmax and an expected error ε . Output: The model structure and output weights matrix β . Step 1 Initialization: Let the number of hidden nodes L = 0 and the residual error be E = T . Set K = d , where K denotes the maximum number of trials of assigning candidate nodes at each odd learning step, d denotes the number of data attributes. Step 2 Learning step: while L < Lmax and E > ε do (a) Increase the number of hidden nodes L : L = L + 1 ; (b) If L ∈ {2n + 1, n ∈ Z } then Generate randomly the input weights matrix Wrandom = [ω(1) , ω(2) , ⋅⋅⋅, ω( j ) ]K *K and the random hidden bias matrix Brandom = [b(1) , b(2) , ⋅⋅⋅, b( j ) ]K *1 ;

Orthogonalize Wrandom and Brandom using the Gram-Schmidt orthogonalization method to obtain Worth and Borth , which satisfy

T

Worth Worth = I and

T

Borth Borth = 1 , respectively.

Calculate the temporal output weights β temp according to + β temp = H temp T. for j = 1 : K

Calculate the residual error E( j ) after pruning the

jth hidden node: E( j ) = T − H residual ⋅ β residual End for *

(c) Let j = { j | max 1≤ j ≤ k || E

( j)

||} . Set ω L = ωorth ( j ) , and bL = b( j ) . *

*

Update H L for the new hidden node and calculate the residual error E after adding the new hidden node Lth : E = E − H LβL .

End if (d) if L ∈ {2n, n ∈ Z } then Calculate the error feedback function sequence H L according to the equation

H 2 n = e2 n −1 ⋅ ( β 2 n −1 )

−1

Calculate the parameter pair (ω L , bL ) and update H L based on the equations (6), (7), and (8). Calculate the output weight β L according to

β2n =

〈 e2 n −1 , H 2 n 〉 H 2n

2

Calculate E after adding the new hidden node Lth : E = E − H LβL .

End if End while

Random Orthogonal Projection

7

4 Experimental Results and Analysis In this section, we present the details of our experiment settings and results. Our experiments are conducted on six benchmark regression problems from UCI machine learning repository [19] and the specification of these datasets is given in Table 1. We chose the Sigmoid function (i.e., Gðx; x; bÞ ¼ 1=ð1 þ expððxx þ bÞÞÞ) as the activation function of B-ELM, EB-ELM, EI-ELM, and OEB-ELM. The input weights x are randomly generated from the range of (−1, 1) and the hidden biases b are generated randomly from the range of (0, 1) using a uniform sampling distribution. For each regression problem, the average results over 50 trials are obtained for each algorithm. In this study, we did our experiments in the MATLAB R2014a environment on the same Windows 10 with Intel Core i5 2.3 GHz CPU and 8 GB RAM. Table 1. Specification of six regression datasets Name Training data Testing data Attributes Airfoil Self-noise 750 753 5 Housing 250 256 13 Concrete compressive strength 500 530 8 White wine quality 2000 2898 11 Abalone 2000 2177 8 Red wine quality 800 799 11

Our experiments are conducted based on the following two questions: (1) Under the same network structure, which algorithm can achieve the best generalization performance and stability? (2) With the increase of the number of hidden nodes, which algorithm has the fastest convergence rate? For Question (1), we set the same number of hidden nodes for the B-ELM, EB-ELM, EI-ELM, and OEB-ELM algorithms. The Root-Mean-Square Error of Testing (Testing RMSE), the Root-Mean-Square Error of Training (Training RMSE), Standard Deviation of Testing RMSE (SD), and learning time are selected as the indicators for performance testing. The smaller testing RMSE denotes the better generalization performance of the algorithm and the smaller SD indicates the better stability of the algorithm. The performance comparison of the four algorithms is shown in Table 2. It is noted that the close results are underlined and the best results are in boldface. From Table 2, we observe that the proposed OEB-ELM algorithm has smallest testing RMSE and standard deviation when applied on six regression datasets, which means that OEB-ELM can achieve better generalization performance and stability than B-ELM, EB-ELM, and EI-ELM. It is noted that the EI-ELM algorithm runs the longest time on all datasets, which shows that the OEB-ELM algorithm is more efficient than the EI-ELM algorithm.

8

W. Cao et al. Table 2. Performance comparison of the EB-ELM, B-ELM, EI-ELM, and OEB-ELM

Datasets

Algorithm

Airfoil self-noise

EB-ELM B-ELM EI-ELM OEB-ELM EB-ELM B-ELM EI-ELM OEB-ELM EB-ELM B-ELM EI-ELM OEB-ELM EB-ELM B-ELM EI-ELM OEB-ELM EB-ELM B-ELM EI-ELM OEB-ELM EB-ELM B-ELM EI-ELM OEB-ELM

Housing

Concrete compressive strength White wine

Abalone

Red wine

Learning time (s) 5.5403 1.2950 25.3281 5.7078 12.6897 2.9103 23.9253 15.8434 13.1275 2.8516 24.5106 13.3669 13.1369 3.1109 32.2019 16.7169 12.7944 3.0247 32.0638 11.1956 3.5128 0.8991 7.1844 4.1181

Standard deviation 0.0025 0.0032 0.2230 0.0025 0.0012 0.0060 0.1165 0.0010 0.0033 0.0225 0.0323 0.0012 0.0015 0.0071 75.0810 0.0013 0.0079 0.0149 0.0078 0.0071 0.0064 0.0044 0.0060 0.0020

Training RMSE 0.0709 0.0729 0.0469 0.0715 0.0182 0.0214 0.0043 0.0182 0.0216 0.0229 0.0075 0.0214 0.0150 0.0159 0.0107 0.0146 0.0106 0.0099

< HT H þ ILL HT T; N  L l b¼ ð5Þ  1 > : HT HT H þ INN T; N\L l where T ¼ ½t1 ; t2 ; . . .; tN T 2 RNC denotes the label matrix of training set, N is the number of training samples. Therefore, the output of ELM can be computed as: f ð xÞ ¼ hð xÞHT ðHT H þ

INN 1 Þ T l

ð6Þ

The KELM additionally introduces a kernel function when    calculating the output of the network [21], which denotes Kij ¼ hðxi Þ
h xj ¼ k xi ; xj . Thus, Eq. (6) can be expressed as: 2

3 k ðx; x1 Þ   INN 1 6 7 . .. f ð xÞ ¼ 4 T 5 Kþ l k ðx; xN Þ

ð7Þ

3 Experiments 3.1

Experimental Data

The data acquisition was performed on our own developed E-Tongue system [22]. Seven kinds of drinks including red wine, white spirit, beer, oolong tea, black tea, maofeng tea, and pu’er tea were selected as test objects. In the experiments, we formulate nine tests for each kind of drinks. Thus, a total of 63 (7 kinds  9 tests) samples were collected. 3.2

Experimental Results

In this section, we compare FSE method experimentally with three feature extraction methods: Raw (No treatment), principle component analysis (PCA) and discrete wavelet transform (DWT). After the original E-Tongue signals processed by different

14

Y. Chen et al.

feature extraction methods, support vector machine (SVM), random forest (RF) and KELM are implemented as recognition part for evaluation. In this section, we use leave-one-out (LOO) strategy for cross validation. The average accuracies of cross validation are reported in Table 1 and the total computation time for cross-validation training and testing is presented in Table 2. From both Tables 1 and 2, we can have the following observations: (1) When SVM is used for classification, the proposed FSE with SVM performs significantly better than other feature extraction methods. FSE with SVM can achieve 90.48%. In the view of execution time, FSE reaches the shortest time expense using SVM than other feature extraction methods. (2) When RF is used for classification, both FSE and “raw” methods get the highest average accuracy (82.54%) compared with other feature extraction methods. From time consumption of computation, FSE is nearly 90 times faster than Raw. (3) When KELM is adopted, FSE gets the highest accuracy 95.24%. Compared with raw feature (22.22%), PCA (69.84%) and DWT (88.89%), it is obvious that KELM shows better fitting and reasoning ability by using proposed FSE feature extraction method. Moreover, the specificity metric with Hilbert projection is more favorable to KELM than any other classifiers. As for time consumption, FSE coupled with KELM cost the least time expense in all methods. It indicates that KELM keeps the minimum amount of computation while providing excellent classification results. Table 1. Accuracy comparison Feature extraction Raw PCA RF 82.54% 73.02% SVM 84.13% 79.37% KELM 22.22% 69.84%

methods DWT 77.78% 77.78% 88.89%

FSE 82.54% 90.48% 95.24%

Table 2. Time consumption comparison Feature extraction Raw PCA RF 344.70 s 43.12 s SVM 48.69 s 37.32 s KELM 5.63 s 34.40 s

methods DWT 56.88 s 52.88 s 49.73 s

FSE 4.24 s 2.56 s 0.06 s

4 Conclusion In this article, we proposed a FSE method for nonlinear feature extraction in E-Tongue data and achieves taste recognition by using several typical classifiers such as SVM, RF and KELM. The proposed FSE coupled with KELM achieves the best results in both accuracy and computational efficiency on our collected data set by a self-developed

A Novel Feature Specificity Enhancement for Taste Recognition

15

E-Tongue system. We should admit that FSE seems to be effective in dealing with feature extraction from high dimensional data, especially LAPV signals. On the other hand, KELM can greatly promote the overall performance in accuracy and speed in recognition.

References 1. Legin, A., Rudnitskaya, A., Lvova, L., Di Nataleb, C., D’Amicob, A.: Evaluation of Italian wine by the electronic tongue: recognition, quantitative analysis and correlation with human sensory perception. Anal. Chim. Acta 484(1), 33–44 (2003) 2. Ghosh, A., Bag, A.K., Sharma, P., et al.: Monitoring the fermentation process and detection of optimum fermentation time of black tea using an electronic tongue. IEEE Sensors J. 15(11), 6255–6262 (2015) 3. Verrelli, G., Lvova, L., Paolesse, R., et al.: Metalloporphyrin - based electronic tongue: an application for the analysis of Italian white wines. Sensors 7(11), 2750–2762 (2007) 4. Tahara, Y., Toko, K.: Electronic tongues–a review. IEEE Sensors J. 13(8), 3001–3011 (2013) 5. Kirsanov, D., Legin, E., Zagrebin, A., et al.: Mimicking Daphnia magna, bioassay performance by an electronic tongue for urban water quality control. Anal. Chim. Acta 824, 64–70 (2014) 6. Wei, Z., Wang, J.: Tracing floral and geographical origins of honeys by potentiometric and voltammetric electronic tongue. Comput. Electron. Agric. 108, 112–122 (2014) 7. Wang, L., Niu, Q., Hui, Y., Jin, H.: Discrimination of rice with different pretreatment methods by using a voltammetric electronic tongue. Sensors 15(7), 17767–17785 (2015) 8. Apetrei, I.M., Apetrei, C.: Application of voltammetric e-tongue for the detection of ammonia and putrescine in beef products. Sens. Actuators B Chem. 234, 371–379 (2016) 9. Ciosek, P., Brzózka, Z., Wróblewski, W.: Classification of beverages using a reduced sensor array. Sens. Actuators B Chem. 103(1), 76–83 (2004) 10. Domínguez, R.B., Morenobarón, L., Muñoz, R., et al.: Voltammetric electronic tongue and support vector machines for identification of selected features in Mexican coffee. Sensors 14(9), 17770–17785 (2014) 11. Palit, M., Tudu, B., Bhattacharyya, N., et al.: Comparison of multivariate preprocessing techniques as applied to electronic tongue based pattern classification for black tea. Anal. Chim. Acta 675(1), 8–15 (2010) 12. Gutiérrez, M., Llobera, A., Ipatov, A., et al.: Application of an E-tongue to the analysis of monovarietal and blends of white wines. Sensors 11(5), 4840–4857 (2011) 13. Dias, L.A., et al.: An electronic tongue taste evaluation: identification of goat milk adulteration with bovine milk. Sens. Actuators B Chem. 136(1), 209–217 (2009) 14. Ciosek, P., Maminska, R., Dybko, A., et al.: Potentiometric electronic tongue based on integrated array of microelectrodes. Sens. Actuators B Chem. 127(1), 8–14 (2007) 15. Ivarsson, P., et al.: Discrimination of tea by means of a voltammetric electronic tongue and different applied waveforms. Sens. Actuators B Chem. 76(1), 449–454 (2001) 16. Winquist, F., Wide, P., Lundström, I.: An electronic tongue based on voltammetry. Anal. Chim. Acta 357(1–2), 21–31 (1997) 17. Tian, S.Y., Deng, S.P., Chen, Z.X.: Multifrequency large amplitude pulse voltammetry: a novel electrochemical method for electronic tongue. Sens. Actuators B Chem. 123(2), 1049–1056 (2007)

16

Y. Chen et al.

18. Palit, M., Tudu, B., Dutta, P.K., et al.: Classification of black tea taste and correlation with tea taster’s mark using voltammetric electronic tongue. IEEE Trans. Instrum. Meas. 59(8), 2230–2239 (2010) 19. Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995) 20. Huang, G.B., Zhu, Q.Y., Siew, C.K.: Extreme learning machine: Theory and applications. Neurocomputing 70(1), 489–501 (2006) 21. Huang, G.B., Zhou, H., Ding, X., et al.: Extreme learning machine for regression and multiclass classification. IEEE Trans. Syst. Man Cybern. Part B 42(2), 513–529 (2012) 22. Liu, T., Chen, Y., Li, D., et al.: An active feature selection strategy for DWT in artificial taste. J. Sens. 2018, 1–11 (2018)

Comparison of Classification Methods for Very High-Dimensional Data in Sparse Random Projection Representation Anton Akusok(B) and Emil Eirola Department of Business Management and Analytics, Arcada UAS, Helsinki, Finland {anton.akusok,emil.eirola}@arcada.fi

Abstract. The big data trend has inspired feature-driven learning tasks, which cannot be handled by conventional machine learning models. Unstructured data produces very large binary matrices with millions of columns when converted to vector form. However, such data is often sparse, and hence can be manageable through the use of sparse random projections. This work studies efficient non-iterative and iterative methods suitable for such data, evaluating the results on two representative machine learning tasks with millions of samples and features. An efficient Jaccard kernel is introduced as an alternative to the sparse random projection. Findings indicate that non-iterative methods can find larger, more accurate models than iterative methods in different application scenarios. Keywords: Extreme learning machines · Sparse data Sparse random projection · Large dimensionality

1

·

Introduction

Machine learning is a mature scientific field with lots of theoretical results, established algorithms and processes that address various supervised and unsupervised problems using the provided data. In theoretical research, such data is generated in a convenient way, or various methods are compared on standard benchmark problems – where data samples are represented as dense real-valued vectors of fixed and relatively low length. Practical applications represented by such standard datasets can successfully be solved by one of a myriad of existing machine learning methods and their implementations. However, the most impact of machine learning is currently in the big data field with the problems that are well explained in natural language (“Find malicious files”, “Is that website safe to browse?”) but are hard to encode numerically. Data samples in these problems have distinct features coming from a huge unordered set of possible features. Same approach can cover a frequent case of missing feature values [10,29]. Another motivation for representing data by c Springer Nature Switzerland AG 2020  J. Cao et al. (Eds.): ELM 2018, PALO 11, pp. 17–26, 2020. https://doi.org/10.1007/978-3-030-23307-5_3

18

A. Akusok and E. Eirola

abstract binary features is a growing demand for security, as such features can be obfuscated (for instance by hashing) to allow secure and confidential data processing. The unstructured components can be converted into vector form by defining indicator variables, each representing the presence/absence of some property (e.g., bag-of-words model [11]). Generally, the number of such indicators can be much larger than the number of samples, which is already large by itself. Fortunately, these variables tend to be sparse. In this paper, we study how standard machine learning solutions can be applied to such data in a practical way. The research problem is formulated as a classification of sparse data with a large number of samples (hundreds of thousands) and huge dimensionality (millions of features). In this work, the authors omit feature selection methods because they are slow on such large scale, they can reduce the performance if a poor set of features is used, and, most importantly, features need to be re-selected if the feature set changes. Feature selection is replaced by Sparse Random Projection [3] (SRP) that provides a dense low-dimensional representation of a high-dimensional sparse data while almost preserving relative distances between data samples [1]. All the machine learning methods in the paper are compared on the same SRP representation of the data. The paper also compares the performance of the proposed methods on SRP to find the suitable ones for big data applications. Large training and test sets are used, with a possibility to process the whole dataset at once. Iterative solutions are typically applied to large data processing, as their parameters can be updated by gradually feeding all available data. Such solutions often come at a higher computational cost and longer training time than methods where explicit solutions exist, as in linear models. Previous works on this topic considered neural networks and logistic regression [9]. Also, there is application research [27] without general comparison of classification methods. A wide comparison of iterative methods based on feature subset selection is given in the original paper for the publicly available URL Reputation benchmark [18]. The remainder of this paper is structured as follows. The next Sect. 2 introduces the sparse random projection, and the classification methods used in the comparison. The experimental Sect. 3 describes the comparison datasets and makes a comparison of experimental results. The final Sect. 4 discusses the findings and their consequences for practical applications.

2 2.1

Methodology Sparse Random Projection for Dimensionality Reduction

The goal of applying random projections is to efficiently reduce the size of the data while retaining all significant structures relevant to machine learning. According to Johnson–Lindenstrauss’ lemma, a small set of points can be projected from a high-dimensional to low-dimensional Euclidean space such that relative distances between points are well preserved [1]. As relative distances reflect the structure of the dataset (and carry the information related to neighborhood

Classification Methods with SRP-Encoded Data

19

and class of particular data samples), standard machine learning methods perform well on data in its low-dimensional representation. The lemma requires an orthogonal projection, that is well approximated by random projection matrix at high dimensionality. Johnson–Lindenstrauss lemma applies to the given case because the number of data samples is smaller than the original dimensionality (millions). However, computing such high-dimensional projection directly exceeds the memory capacity of contemporary computers. Nevertheless, a similar projection is obtained by using sparse random projection matrix. The degree of sparseness is tuned so that the result after the projection is a dense matrix with a low number of exact zeros. Denote the random projection matrix by W , and the original highdimensional data by X. The product W T X can be calculated efficiently for very large W and X, as long as they are sparse. Specifically, the elements of W are not drawn from a continuous distribution, but instead distributed as follows: ⎧  ⎪ ⎨− s/d with probability 1/2s wij = 0 (1) with probability 1 − 1/s ⎪ ⎩  + s/d with probability 1/2s where s = 1/density and d is the target dimension [15]. 2.2

Extreme Learning Machine

Extreme Learning Machine (ELM) [13,14] is a single hidden layer feed-forward neural network where only the output layer weights β are optimized, and all the weights wkj between the input and hidden layer are assigned randomly. With N input vectors xi , i ∈ [1, N ] collected in a matrix X and the targets collected in a vector y, it can be written as Hβ = y

where

H = h(W T X + 1T b)

(2)

Here W is a projection matrix with L rows corresponding to L hidden neurons, filled with normally distributed values, b is a bias vector filled with the same values, and h(·) is a non-linear activation function applied element-wise. This paper uses hyperbolic tangent function as h(·). Training this model is simple, as the optimal output weights β are calculated directly by ordinary least squares. Tikhonov regularization [30] is often applied when solving the least square problem in Eq. (2). The value of the regularization parameter can be selected by minimizing the leave-one-out cross-validation error (efficiently calculated via the PRESS statistic [20]). The model is easily adapted for sparse high-dimensional inputs by using sparse random matrix W as described in the previous section. ELM with this structure for the random weight matrix is very similar to the ternary ELM from [12].

20

A. Akusok and E. Eirola

ELM can incorporate a linear part by attaching the original data features X to the hidden neurons output H. A random linear combination of the original data features can be used if attaching all the features is infeasible, as in the current case of very high-dimensional sparse data. These features let ELM learn any linear dependencies in data directly, without their non-linear approximation. Such method is similar to another random neural network method called Random Vector Functional Link network (RVFL [23]), and is presented in this paper by the RVFL name. 2.3

Radial Basis Function ELM

An alternative way of computing the hidden layer output H is by assigning a centroid vector cj , j ∈ [1, L] to each hidden neuron, and obtain H as a distancebased kernel between the training/test set and a fixed set of centroid vectors. 2

H i,j = e−γj d

(xi ,cj )

, i ∈ [1, N ], j ∈ [1, L],

(3)

where γj is kernel width. Such architecture is widely known as Radial Basis Function (RBF) network [6,17], except that ELM-RBF uses fixed centroids and fixed random kernel widths γj . Centroid vectors cj are chosen from random training set samples to better follow the input data distribution. Distance function for dense data is Euclidean distance. 2.4

Jaccard Distance for Sparse Binary Data

Distance computations are a major drawback in any RBF network with Euclidean distances as they are slow and impossible to approximate for highdimensional data [2]. Jaccard distances can be used for binary data [8]. However, a naive approach for Jaccard distances is infeasible for datasets with millions of features. An alternative computation of Jaccard distance matrix directly from sparse data is considered in the paper and proved to be fast enough for practical purposes. Recall the Jaccard distance formulation for sets a and b: J(a, b) = 1 −

|a ∩ b| |a ∪ b|

(4)

Each column in sparse binary matrices A and B can be considered as a set of non-zero values, so A = [a1 , a2 , . . . am ] and B = [b1 , b2 , . . . bn ]. Their union and intersection can be efficiently computed with matrix product and reductions: |ai ∩ bj | = (AT B)ij , i ∈ [1, n], j ∈ [1, m]    |ai ∪ bj | = |ai | + |bj | − |ai ∩ bj | = 1T Aik + Bjk T 1 − AT B k

K

i,j

(5) (6)

Classification Methods with SRP-Encoded Data

21

The sparse matrix multiplication is the slowest part, so this work utilizes its parallel version. Note that the runtime of a sparse matrix product AT B scales sub-linearly in the number of output elements n·m, so the approach is inefficient for distance calculation between separate pairs of samples (ai , bj ) not joined in large matrices A, B.

3 3.1

Experiments Datasets

The performance of the various methods is compared on two separate, large datasets with related classification tasks. The first dataset concerns Android application packages, with the task of malware detection. Features are extracted using static analysis techniques, and the current data consists of 6,195,080 binary variables. There are 120,000 samples in total, of which 60,000 are malware, and this is split into a training set of 100,000 samples and a fixed test set of 20,000 samples. The data is very sparse – the density of nonzero elements is around 0.0017%. Even though the data is balanced between classes, the relative costs of false positives and false negatives are very different. As such, the overall classification accuracy is not the most useful metric, and the area under the ROC curve (AUC) is often preferred to compare models. More information about the data can be found in [22,26,27]. Second, the Web URL Reputation dataset [19] contains a set of 2,400,000 websites that can be malicious or benign. The dataset is split into 120 daily intervals when the data was collected; the last day is used as the test set. The task is to classify them using the given 3,200,000 sparse binary features, as well as 65 dense real-valued features. This dataset has 0.0035% nonzero elements, however, a small number of features are real-valued and dense. For this dataset, the classification accuracy is reported in comparison with the previous works [32,33]. 3.2

Additional Methods

Additional methods include Kernel Ridge Regression (KRR), k-Nearest Neighbors (kNN), Support Vector Machine for binary classification (SVC), Logistic regression and Random Forest. Of these methods, only SVC and logistic regression have iterative solutions. Kernel Ridge Regression [21,25] combines Ridge regression, a linear least squares solution with L2-penalty on weights norm, with the kernel trick. Different kernels may be used, like the Jaccard distance kernel for sparse data proposed above. k-Nearest Neighbors (kNN) method is a simple classifier that looks at k closest training samples to a given test sample and runs the majority vote between them to predict a class. The value of k is usually odd to avoid ties. It can use different distance functions, and even a pre-computed distance matrix (with the Jaccard distance explained in the Methodology section).

22

A. Akusok and E. Eirola

Support Vector Machine [7] constructs a hyperplane in a kernel space, that separates the classes. It’s a native binary classifier with an excellent performance, that can be extended to regression as well. There is a significant drawback of the computational complexity that is between quadratic and cubic in the number of samples [16]. Logistic regression [4] is a binary classifier that utilizes the logistic function in its optimization problem. The logistic function is non-linear and prevents Logistic Regression from having a direct one-step solution like linear least squares in ELM or RVFL. Its weights are optimized iteratively [28], with an L2 penalty helping to speed up the convergence and improve the prediction performance. Random Forest [5,31] is an ensemble method consisting of many random tree classifiers (or regressors). Each tree is trained on bootstrapped data samples and a small subset of features, producing a classifier with large variance but virtually no bias. Then Random Forest averages the predictions of all trees, reducing the variance of their estimations. It can be considered a non-iterative method because parameters of each leaf of a tree are computed in a closed form and never updated afterward. The method can work directly with sparse high-dimensional data. It obtains good results already with 10 trees. Increasing this number yields little improvement at a significant price in the training time. Drawbacks of Random Forest are its inability to generalize beyond the range of feature values observed in the training set, and slower training speed with a large number of trees. All methods are implemented in Python using scikit-learn routines [24]. The experiments are run on the same workstation with a 6-core Intel processor and 64 GB RAM. 3.3

Results on Large Data

The effect of various Sparse Random Projection dimensionality on classification performance for large datasets is examined here. A range of 4–10,000 features is evaluated, sampled uniformly on a logarithmic scale. Only “fast” methods with sub-quadratic runtime scaling in the number of training samples are considered.

Fig. 1. The effect of varying number of Sparse Random Projection features on the performance for Android Malware dataset. Vertical line corresponds to 5000 features.

Classification Methods with SRP-Encoded Data

23

Fig. 2. The effect of varying number of Sparse Random Projection features on the performance for Web URL Reputation dataset. Full dataset is tested with a maximum 650 features due to the memory constraints. Vertical line corresponds to 5,000 features.

Performance evaluation for Android Malware dataset is presented on Fig. 1. Random Forest methods are the best performers with little SRP features, but other methods catch up after 2,000 features in SRP and outperform Random Forest with even higher SRP dimensionality. The number of trees in Random Forest has a small positive effect of only +0.2% AUC for 10 times more trees. The same evaluation for Web URL Reputation dataset is shown in Fig. 2. Random Forest is again better with fewer SRP features and performs similarly with more features. More trees in Random Forest reduce is performance fluctuations. Interesting that all the methods except Random Forest perform very similarly despite their different formulation; a fact that is probably connected to the nature of Sparse Random Projection. 3.4

Sparse Random Projection Benchmark

A larger variety of classification methods are compared on a reduced training set of only 1,000 samples, randomly selected from the training set. All experiments use the same test set of 20,000 samples. Sparse Random Projection includes 5,000 features that provide highest performance with reasonable runtime in the previous experiments (vertical line on Figs. 1, 2). The experimental results are summarized in Table 1. A total of 101 fixed training sets are generated – one for tuning hyperparameters, and the rest for 100 runs of all the methods. All methods use tuned L2-regularization with the regularization parameter selected on a logarithmic scale of [2−20 , 2−19 , . . . , 220 ], except for kNN that has a validated k = 1 and Random Forest with 100 trees for a reasonable runtime. Comparison results on Web URL Reputation dataset achieved 94.4% accuracy in rule-based learning [32], and 97.5% accuracy in SVM-based approach [19]. The latter is comparable to the proposed results, but the exact comparison depends on a particular point of the ROC curve.

24

A. Akusok and E. Eirola

Table 1. Mean area under ROC curve in % (with the standard deviation in parentheses) and runtime in seconds for all methods on the two benchmark datasets, using 1,000 training samples and summarized over 100 runs. Bold font denotes the best result for each dataset, and any other not statistically significantly different values (paired t-test at the significance level 0.05). Method

Android malware Web URL reputation AUC (std.), % time, s AUC (std.), % time, s

ELM, SRP

99.41 (0.08)

99.29 (0.16)

2.9

RVFL, SRP

99.34 (0.08)

2.2

98.13 (0.74)

2.1

RBF-ELM, SRP

99.12 (0.11)

2.1

97.53 (1.81)

2.1

KRR, SRP

92.61 (0.24)

1.3

99.15 (0.22)

1.3

kNN, SRP

86.55 (1.00)

13.4

83.35 (2.25)

13.5

SVC, SRP

99.26 (0.10)

44.2

99.17 (0.21)

34.6

Logistic Regression, SRP 99.21 (0.11)

4

3.1

0.6

99.17 (0.23)

0.1

Random Forest

98.54 (0.42)

30.6

95.64 (1.65)

4.9

RBF-ELM, Jaccard

86.34 (5.15)

18.9

79.67 (3.62)

1.7

KRR, Jaccard

99.48 (0.06)

18.3

99.31 (0.06)

1.6

kNN, Jaccard

91.38 (0.50)

18.0

84.62 (0.95)

1.3

Conclusion

This study provides useful insights on the nature of a very high-dimensional sparse data and the utility of Sparse Random Projection for its processing. The original high-dimensional sparse data representation is best combined with Random Forest if the data is abundant. Random Forest efficiently learns to discriminate between classes on a huge dataset and has the fastest training speed if run in parallel with a small number of trees. However, it underperforms on smaller datasets. The original sparse data can be used in kernel matrix calculation for Kernel Ridge regression, that excels in smaller datasets. However, the kernel computation runtime is a significant drawback, and the KRR itself cannot scale to huge datasets. Sparse Random Projection efficiently represents a high-dimensional sparse data given a sufficient number of features (at least 1,000 in the tested datasets). In that case, it provides good results with very different methods: based on neural networks, logistic regression, kernel methods like KRR and SVM. An interesting fact is that the choice of a particular method is not significant. Of the aforementioned methods, ELM and RVFL are the most versatile. They provide best results in a short runtime, for any training set size.

Classification Methods with SRP-Encoded Data

25

Acknowledgements. This work was supported by Tekes – the Finnish Funding Agency for Innovation – as part of the “Cloud-assisted Security Services” (CloSer) project.

References 1. Achlioptas, D.: Database-friendly random projections: johnson-lindenstrauss with binary coins. J. Comput. Syst. Sci. 66(4), 671–687 (2003). Special Issue on PODS 2001 2. Beyer, K., Goldstein, J., Ramakrishnan, R., Shaft, U.: When Is “Nearest Neighbor” Meaningful? In: Beeri, C., Buneman, P. (eds.) Database Theory – ICDT 1999: 7th International Conference Jerusalem, Israel, 10–12 January 1999 Proceedings, pp. 217–235. Springer, Heidelberg (1999) 3. Bingham, E., Mannila, H.: Random projection in dimensionality reduction: Applications to image and text data. In: Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2001, pp. 245–250. ACM, New York, NY, USA (2001) 4. Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Boston (2006) 5. Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001) 6. Broomhead, D.S., Lowe, D.: Multivariable functional interpolation and adaptive networks. Complex Syst. 2(3), 321–355 (1988) 7. Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995) 8. Czarnecki, W.M.: Weighted tanimoto extreme learning machine with case study in drug discovery. IEEE Comput. Intell. Mag. 10(3), 19–29 (2015) 9. Dahl, G.E., Stokes, J.W., Deng, L., Yu, D.: Large-scale malware classification using random projections and neural networks. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3422–3426. IEEE (2013) 10. Eirola, E., Lendasse, A., Vandewalle, V., Biernacki, C.: Mixture of gaussians for distance estimation with missing data. Neurocomputing 131, 32–42 (2014) 11. Harris, Z.S.: Distributional Structure, pp. 3–22. Springer, Dordrecht (1981) 12. van Heeswijk, M., Miche, Y.: Binary/ternary extreme learning machines. Neurocomputing 149, 187–197 (2015) 13. Huang, G.B., Zhou, H., Ding, X., Zhang, R.: Extreme learning machine for regression and multiclass classification. IEEE Trans. Syst. Man Cybern. Part B: Cybern. 42(2), 513–529 (2012) 14. Huang, G.B., Zhu, Q.Y., Siew, C.K.: Extreme learning machine: theory and applications. Neural Networks Selected Papers from the 7th Brazilian Symposium on Neural Networks (SBRN 2004) 7th Brazilian Symposium on Neural Networks, vol. 70, no. 1–3, pp. 489–501, December 2006 15. Li, P., Hastie, T.J., Church, K.W.: Very sparse random projections. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 287–296. ACM (2006) 16. List, N., Simon, H.U.: General polynomial time decomposition algorithms. J. Mach. Learn. Res. 8, 303–321 (2007) 17. Lowe, D.: Adaptive radial basis function nonlinearities, and the problem of generalisation. In: 1989 First IEE International Conference on Artificial Neural Networks, Conf. Publ. No. 313, pp. 171–175. IET (1989)

26

A. Akusok and E. Eirola

18. Ma, J., Saul, L.K., Savage, S., Voelker, G.M.: Identifying suspicious URLs: an application of large-scale online learning. In: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 681–688. ACM, Montreal, Quebec, Canada (2009) 19. Ma, J., Saul, L.K., Savage, S., Voelker, G.M.: Identifying suspicious URLs: an application of large-scale online learning. In: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 681–688. ACM (2009) 20. Miche, Y., van Heeswijk, M., Bas, P., Simula, O., Lendasse, A.: TROP-ELM: a double-regularized ELM using LARS and Tikhonov regularization. Neurocomputing 74(16), 2413–2421 (2011) 21. Murphy, K.P.: Machine Learning: A Probabilistic Perspective. The MIT Press, Cambridge (2012) 22. Palumbo, P., Sayfullina, L., Komashinskiy, D., Eirola, E., Karhunen, J.: A pragmatic android malware detection procedure. Comput. Secur. 70, 689–701 (2017) 23. Pao, Y.H., Park, G.H., Sobajic, D.J.: Learning and generalization characteristics of the random vector functional-link net. Neurocomputing 6(2), 163–180 (1994). Backpropagation, Part IV 24. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825–2830 (2011) 25. Saunders, C., Gammerman, A., Vovk, V.: Ridge regression learning algorithm in dual variables. ICML 98, 515–521 (1998) 26. Sayfullina, L., Eirola, E., Komashinsky, D., Palumbo, P., Karhunen, J.: Android malware detection: Building useful representations. In: 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 201–206, December 2016 27. Sayfullina, L., Eirola, E., Komashinsky, D., Palumbo, P., Miche, Y., Lendasse, A., Karhunen, J.: Efficient detection of zero-day android malware using normalized bernoulli naive bayes. IEEE TrustCom 2015, 198–205 (2015) 28. Schmidt, M., Le Roux, N., Bach, F.: Minimizing finite sums with the stochastic average gradient. Mathematical Programming, pp. 1–30 (2013) 29. Sovilj, D., Eirola, E., Miche, Y., Bj¨ ork, K.M., Nian, R., Akusok, A., Lendasse, A.: Extreme learning machine for missing data using multiple imputations. Neurocomputing 174, 220–231 (2016) 30. Tikhonov, A.N.: On the stability of inverse problems. Dokl. Akad. Nauk. SSSR 39(5), 195–198 (1943) 31. Zhang, L., Ren, Y., Suganthan, P.N.: Towards generating random forests via extremely randomized trees. In: 2014 International Joint Conference on Neural Networks (IJCNN), pp. 2645–2652. IEEE (2014) 32. Zhang, P., Zhou, C., Wang, P., Gao, B.J., Zhu, X., Guo, L.: E-tree: an efficient indexing structure for ensemble models on data streams. IEEE Trans. Knowl. Data Eng. 27(2), 461–474 (2015) 33. Zhou, Z., Zheng, W.S., Hu, J.F., Xu, Y., You, J.: One-pass online learning: a local approach. Pattern Recogn. 51, 346–357 (2016)

A Robust and Dynamically Enhanced Neural Predictive Model for Foreign Exchange Rate Prediction Lingkai Xing(&), Zhihong Man, Jinchuan Zheng, Tony Cricenti, and Mengqiu Tao Swinburne University of Technology, Hawthorn, VIC 3122, Australia [email protected]

Abstract. In today’s highly interlinked international economy, accurate and timely real-time predictions of foreign exchange (fx) market offer tremendous business, social and political values to our community. In this work, an effective and accurate neural predictive model based on recurrent neural network and Regularised Extreme Learning Machine is developed. Hidden layer of the developed model is designed to map complex financial input patterns into a higher dimensional hidden feature space where input patterns are represented is a clear and delicate way. Output feedback loops and regularisation are implemented in the model to facilitate strong robustness and capability of capturing complex dynamical characteristics of the fx system. Experimental results show superior performance of the developed model comparing to benchmark models under varying market conditions. Keywords: Recurrent neural networks Extreme Learning Machine




Foreign exchange rate prediction




1 Introduction Foreign exchange (fx) market is one of the most complex dynamic systems in the world and provides the fundamental facilitations to today’s international economy. As a result, it is of great interests for scientists, economists, governments and various financial entities to develop accurate and effective models to predict the system. However, due to the system’s complex and chaotic nature, modeling and predicting the system is one of the most challenging problems which are yet to be solved. Traditionally, dynamical behaviors of the fx market have been mainly studied by financial economists, and financial econometrics models have been implemented [1–3] to predict fx rates. However, due to their restrictive parametric assumptions and lacking of sophisticated dynamical structures, conventional financial econometrics models are incapable of capturing intrinsic dynamical structures embedded in big data. Therefore, there is a great potential for using more advanced and intelligent analytical and modeling tools and machine learning methods in study of the fx system. In particular, Artificial neural networks (ANNs), known as biologically inspired signal processing systems are considered to be a promising method for tackling such task due to their © Springer Nature Switzerland AG 2020 J. Cao et al. (Eds.): ELM 2018, PALO 11, pp. 27–36, 2020. https://doi.org/10.1007/978-3-030-23307-5_4

28

L. Xing et al.

data-driven nature, universal approximation capability, strong capability in capturing data nonlinearity and parallelism [4, 5]. In fx market modeling and forecasting literature, the mostly widely studied and implemented ANNs are the multilayer feedforward neural networks and recurrent neural networks trained with back-propagation based algorithm (BP) [5–10]. However, in today’s financial arena, predictive models incorporated into real-world automated trading systems are required to be able to update on-the-fly at high frequency and hence to be able to produce more accurate predictions under turbulent market conditions. In such case, slow convergence and high sensitivity to initialisation becomes are prohibiting effective applications of BP based neural predictive models for financial forecasting tasks [11, 12]. Hence, faster and more efficient neural predictive models are required. Extreme Learning Machine (ELM) offers a promising alternative to overcome limitations of BP based neural models, given its elegant way of converting ANN training problem from a complex nonconvex optimisation problem to a simple normminimising linear optimization problem while keeping the netowork’s universal approximation capability [13]. With ELM, good generalization and much faster learning can be achieved at the same time [13]. Due to its merits, several ELM based RNN models have been developed for fx rate forecasting purposes [15–17]. In particular, an ELM based recurrent neural network with output feedback connections called Dynamic Neural Network (DNN) was proposed in [17] showing promising performance in capturing complex dynamic characteristics of a dynamical process. In this paper, an robust and accurate neural predictive model is developed for fx market forecasting tasks based on a modified DNN. The developed model is consist of a single-hidden layer feedforward network (SLFN) structure and tapped-delaymemories (TDMs) placed at both input and output layer to generate temporal-spatial input vectors and output feedbacks, respectively. In such way, complex relationship between various social economic factors and the targeted fx rate can be captured by the neural model in a dynamical way. The hidden layer is specially designed with hyperbolic tangent activation function, controlled narrow interval for randomised input weights and bias terms and. In that, hidden layer of the proposed model is ensured to be highly sensitive to the delicate components in financial input data stream. In order to improve the network’s comprehensiveness in modeling dynamical features of the fx market and reduce number of hidden nodes required, a generalised hidden output matrix is then generated by combining the hidden outputs with the output feedbacks. To uplift the model’s robustness against outliners, Regularised ELM algorithm (R-ELM) [18] is implemented to compute the optimal generalised output weights. In our comprehensive simulation study, the proposed model has shown general superior forecasting accuracy and robustness against the benchmark models. The remaining sections of this paper are organized as follows: In Sect. 2, the fx market modeling problem is formulated, and the developing procedure for our proposed predictive model is described. In Sect. 3, simulations and performance comparisons between our predictive model and other incumbent models are carried out. Section 4 concludes the paper and presents direction for future work.

A Robust and Dynamically Enhanced Neural Predictive Model

29

2 Problem Formulation and the Proposed DNN Based Predictive Model As mentioned above, financial market can be considered as a complex dynamical system whose underlying complex operating mechanisms of the fx market system can be represented by difference Equation (5) below. yð pÞ ¼ f ðyðp  1Þ; yðp  2Þ


yðp  krate Þ;


; us ðp  1Þ; us ðp  2Þ


us ðp  ks ÞÞ þ eðp  1Þ

ð1Þ

where y and fu1  us g represent the targeted fx rate and other s exogenous factors, is an unknown complex dynamical relationship. p  1 as the present point of time with p  maxfkrate ; k1


ks ; cg [ 0, krate ; k1


ks [ 0 represent respective number of lags for fx rate and exogenous factor, s is the number of exogenous influencing factors, and c [ 0 represents the number of the network’s output feedback loops. The term eðp  1Þ represents market disturbances in the financial market, which can catastrophic events, unforeseen political changes and so on. Therefore, our aim is to let the neural network to learn the underlying input-output dynamical relationship which is represented by the difference function (1). To do that, a series of temporally successive training dataset with a period length of M at time point p  1 as U ¼ ½ uðp  MÞ


uðp  1Þ T and t ¼ ½ tðp  MÞ


tðp  1Þ T are presented to the network. As shown in Fig. 1, the input layer is comprised of a series of TDMs which are utilized to present various financial and economic data in a temporalspatial manner to the network through the temporal-spatial vectors. In particular, uðp  1Þ represents the most recent temporal input pattern vector within the training input pattern vector batch at time point p  1 which can be expressed as: Bias

D

D

D

D

D

D

Layer A

Layer B

Fig. 1. The neural network structure for the proposed neural predictive model

30

L. Xing et al.

uðp  1Þ ¼ ½yðp  1Þ; yðp  2Þ


yðp  krate Þ;


; us ðp  1Þ; us ðp  2Þ


us ðp  ks Þ:

ð2Þ

In practice, the choice of factors x1


xs is determined from empirical financial market research results and expert’s understanding about the targeted fx rate. yðpÞ is the network’s prediction corresponding to the one-day ahead forecasting target exchange rate tðpÞ at time point p  1. Note that, in this study, we only focus on one-day ahead forecast problems and hence there is only one output node in the output layer. Remark 3.1: The choice of delay lags for each input factors is a critical aspect in our modeling procedure which has direct impact on the model’s prediction performance. Although theoretical lag determination rules have been developed based on Embedding theorem, such method is based on one’s understanding about the underlying attractors allocation of the modeled dynamical system. For financial market modeling, such method is unsuitable due to the lack of understanding of the market’s dynamical structure. As an effective heuristic alternative, genetic algorithm (GA) based lag selection method for multivariate dynamic modeling problems are widely implemented due to its simplicity and effectiveness [19, 20]. Therefore, we implement GA to find optimal lags for each factor in this work. Then, the hidden layer outputs for the given training input matrix can be computed as: 2

h1 ð p  M Þ 6 .. H¼4 . h1 ðp  1Þ

3


hh1 ðp  MÞ 7 .. .. 5 . .




ð3Þ

hh1 ðp  1Þ

where   hj ðp  mÞ ¼ G uðp  mÞwj þ bj

ð4Þ

for m ¼ 1. . .M and j ¼ 1; . . .; n1. Input weights W ¼ ½ w1


wh  and hidden biases b ¼ ½ b1


bh  are randomly generated from uniform distributions as wnj  i:i:d:U ðLWL ; LWU Þ and bq  i:i:d:U ðLBL ; LBU Þ with wj ¼ ½ w1j


wnj T for j ¼ 1; . . .; h1 and q ¼ 1; . . .; n1. LWL ; LWU ; LBL and LBU are the controlled lower and upper bounds for the uniform. For activation function GðÞ of hidden nodes, a modified sigmoid function is implemented. Remark 3.2: From our studies, it was found that the overall level of hidden neurons’ activity is generally positively related to the model’s prediction precision. Therefore, in this work, hyperbolic tangent activation function rather than conventional sigmoid is implemented in the hidden neurons to uplift the intensity of hidden activations, and controlled random distribution intervals are designed to ensure neurons are working at the active region of the hyperbolic tangent.

A Robust and Dynamically Enhanced Neural Predictive Model

31

Remark 3.3: The high dependencies between various financial and economic data are well known phenomenon in financial literature [21–23]. In such case, the distribution of input data in the input space for highly correlated variables are probably distributed in an entangled way and hence increase risk of biased estimation. To address this issue, we use a larger size of hidden layer to perform decomposition of the original data set from the input pattern space to a higher dimensional feature space where the data points will become more spread-out and a clearer representation of the input patterns can be obtained as proven in the Cover’s theorem [4]. However, large size of hidden layer also means large computational and memories requirements in practice. Hence, how to provide same quality of performance with a smaller size of hidden layer is a interesting and practical topic needing more attention. From Fig. 1, it is noted that feedbacks at the output layer of the RNN model are generated by a set of output tapped-delay-memories with a length of c based on the previous DNN model. Then, the generalized hidden layer outputs can be expressed as: 2

h1 ð p  M Þ


6 .. .. H ¼ 4 . . h 1 ð p  1Þ




hn1 ðp  MÞ tðp  M  1Þ .. .. . . hn1 ðp  1Þ

t ð p  2Þ

3


t ð p  M  cÞ 7 .. .. 5: . .


t ð p  c  1Þ

ð5Þ

Remark 3.4: In [17], it is shown that the DNN was able to better capture the complex dynamical characteristics of targeted dynamic process with additional dynamic information presented in the generalized hidden layer outputs matrix. In our study, by injecting additional dynamical information about the underlying fx rate into the hidden output matrix, the network is able to perform accurate predictions with significantly reduced number of hidden neurons. Also, feedbacks enable the network to capture not only the dynamical relationship between input patterns and the targeted fx rate, but also the dynamical delicacies existing among the future fx rate and its most recent histories. Finally, the generalized hidden layer output matrix in (5) is utilised to calculate the optimal generalized output weights b which can be obtained by performing a batchlearning type of linear least square optimization using a regularization technique proposed in [18] as: 

c 1 min kek2 þ kbk2 2 2 s:t:

 ð6Þ

e ¼ t  o ¼ t  H b

and hence,  1 1 T  IþH H HT t: b ¼ c 

ð7Þ

Note that linear activation function is used for the network’s output node. Remark 3.6: In (5–9), it is seen that the ELM based neural computing method has the advantages that the long convergence and high sensitivity to initialization of BP based

32

L. Xing et al.

methods can be avoided and hence the training is much faster and minimum norm and the globally optimised solution can be achieved at the same time. Remark 3.7: Due to the high level of outliners in financial data, we include a regularization term as shown in (6) to reduce both the empirical and structural risks of the trained model and hence to improve its robustness against disturbances. More specifically, sensitivity of the optimised output weight vector b (i.e. the model’s structural risk) can be controlled by adjusting the regularization parameter c.

3 Simulations and Experimental Results 3.1

Data Description

Our dataset for the simulation is collected from three main online data bases including Thomas Reuters MarketPsych [24], Quandl and Federal Reserve Bank of St. Louise. The input data set for the proposed model comprises 35 strings of daily time series data including market sentimental index aggregated from professional financial news and social media, commodity future prices and economic fundamentals for the period between 06 June 2011 and 25 May 2018. For the same time period, the daily AUD/USD exchange rate string is collected as the target data set. In preliminary data screening, several issues including large difference in levels between various data strings, considerable proportions of missing values, nonstationarity, the stylised fattailed and negatively skewed distributions of input and output strings are with heavy outliers. To address some undesirable issues, several data pre-processing procedures are conducted including interpolation, normalization and smoothing. 3.2

Experimental Setup and Model Parameter Sensitivity Analysis

In this work, all models are trained based on the above-mentioned dataset during the exact same period, synchronously. All experimental results are generated for one-day ahead forecasting task. Moreover, three typical financial phases (i.e. conditions) are chosen to for comparing our models’ prediction capability against other models under more realistic environment. These include (1) Steady period where volatility of the targeted exchange rate is low and hence its movement is less fluctuated; (2) Volatile period where volatility of the targeted exchange rate is high and variations of the fx rate are dramatic, but there is not significant conditional mean shift; (3) Transition period around which sudden or gradual shifts of the targeted exchange rate happens. For each of these three periods, we choose 500 days of data as our training, testing and validation (i.e. out-ofsample) dataset. The proportional splits for the training, validation and testing datasets are 50%, 20% and 30%. To ensure all models including our proposed model are optimally tuned for the comparative analysis. A set of 10 sampled mini-training and testing batches with a length of 50 days are randomly selected across the entire period of the whole dataset.

A Robust and Dynamically Enhanced Neural Predictive Model

33

For performance measurement, we implemented the Mean Absolute Percentage Error (MAPE) and Correct Direction Score Percentage (CDSP). Definitions for these two measurements are shown as below: MAPE ¼

100 XN kpi  ti k i¼1 ti N

ð8Þ

and 100 XN CDSP ¼ i¼1 N



0 0

signðpi Þ 6¼ signðti Þ : signðpi Þ ¼ signðti Þ

ð9Þ

In searching for optimal unit-delay for the chosen input factors for all neural based models, GA optimization technique is implemented. The basic steps to implement GA for selecting optimal input delays can be found in [19, 20]. In our case, a CDSP threshold value of 70% is used as the convergence stopping criteria. Note that, in our testing experiment, the abovementioned 10 sampled mini-training and testing batches are used during the process to generate the CDSP criteria. According to the results, the correspondent unit-delay for each factor is generated optimally giving a sum of 480 units in the input layer of the network.

Fig. 2. Sensitivity analysis of hidden nodes number with averaged MAPE and averaged CDSP

In Fig. 2, effect of hidden node number on the averaged MAPE and CDSP performance of our proposed model is presented. The averaged metrics are the mean value of ten 50-day-mini-batches described in the previous subsection. It is seen that the effect of high dimensional space on improving forecasting accuracy is apparent while the averaged MAPE decreased sharply with the increase of hidden node number from 100 to 500. On the other hand, we observe that significant performance improvement happens when hidden nodes increase from 100 to 600, an increase of CDSP from 34.64% to 57.16%. Therefore, 500 is chosen to be the optimal number. From Table 1, it is seen that number of feedback loops z can significantly improve the MAPE performance of the neural predictive model against noisy and varying data environment. However, it is also seen that with the increase of z, CDS is adversely

34

L. Xing et al.

Table 1. Averaged performance testing for number of output feedback loops of the proposed model. Performance measurements Number of model’s output feedback loops z 1 2 3 4 5 MAPE 3.01 2.94 2.69 2.82 2.85 CDS 28.71 28.36 27.93 27.51 27.52

affected when z reached over 3. The cause for such case is still under the authors’ investigation and remedy will be provided. Hence, for now, the optimal value for z is set to 3. Table 2. Averaged performance testing for regularization parameter of the proposed model. Performance measurements Values of model’s regularization parameter 1c 0.0005 0.00005 0.000005 4.51 2.36 3.41 24.19 28.67 30.13

MAPE CDS

From Table 2, it is observed that with the inclusion of the regularisation parameter 1=c, MAPE performance is further improved. This result suggests under randomly sampled data environment the model’s robustness against disturbances is enhanced. The optimal value for d=c is chosen as 0.00005. 3.3

Comparative Performance Analysis

In this subsection, comparative performance analysis and simulation results for the proposed model, NARX, ELM, ESTAR and ARIMA are presented over the three typical fx market phases. Table 3. Out-of-sample best average performance comparisons of the proposed model, BPNARX, ELM and AR-GARCH model for one-day-ahead AUD/USD foreign exchange rate forecast Phases

Models Proposed model (480100-1, FB = 3) MAPE CDS Steady 1.68 102.45 Volatile 2.75 93.69 Transition 2.65 86.21

BP-NARX (480-75-1, FB = 3) MAPE CDS 2.28 99.76 3.52 93.59 4.53 85.58

ELM (480-500-1)

ESTAR (3–2) AR-GARCH (3-2-2-0)

MAPE CDS MAPE CDS MAPE 2.36 100.41 17.34 62.46 9.53 4.18 95.70 15.59 66.32 13.09 6.53 88.74 9.67 2.02 15.91

CDS 78.31 70.22 64.82

A Robust and Dynamically Enhanced Neural Predictive Model

35

It is seen from Table 3, the proposed model has the lowest testing averaged MAPE among all models across all three financial phases, which means that in terms of variation prediction results of proposed model shows robustness and overall high accuracy. Also, the proposed model has the higher CDS comparing to all three competing models across all three phases, except with ELM during unstable periods. It is noted that our model’s CDS performance is adversely affected by the output feedback loops. The reason for such results is under authors’ investigation. The training time for proposed model is much shorter than NARX’s training time and is almost the same as ELM.

4 Conclusion and Discussions In this work, we have successfully developed a timely and accurate neural predictive model for foreign exchange rate prediction purpose based on ELM neural computing approach. In the proposed model, temporal input patterns are mapped into a higher dimensional feature space with an efficient number of sensitivity enhanced hidden neurons. In addition, output feedback loops are used to facilitate more delicate modelling of dynamics of the fx rate and improve the computational efficiency. Moreover, R-ELM is implemented to train the network to enhance its robustness against both internal and external disturbances. Further works will be conducted to further improve the neural model’s accuracy for directional prediction under volatile market phases. Acknowledgements. The authors would like to express sincere gratitude to Dr. Richard Peterson and Mr. Changjie Liu from MarketPsych LLC, US for their generous contribution of providing resourceful data asset used in this study and our other research works and their kindness for assisting us in the data retrieving and wrangling process.

References 1. Bollerslev, T.: Financial econometrics: past developments and future challenges. J. Econ. 100(1), 41–51 (2001) 2. Teräsvirta, T.: Specification, estimation, and evaluation of smooth transition autoregressive models. J. Am. Stat. Assoc. 89(425), 208–218 (1994) 3. Engle, R.F.: Autoregressive conditional heteroscedasticity with estimates of the variance of united kingdom inflation. Econometrica 50(4), 987–1007 (1982) 4. Haykin, S.: Neural Networks: A Comprehensive Foundation, p. 842. Prentice Hall, Upper Saddle River (1998) 5. Huang, W., Lai, K.K., Nakamori, Y., Wang, S., Yu, L.: Neural networks in finance and economics forecasting. Int. J. Inf. Technol. Decis. Making 06(01), 113–140 (2007) 6. Chung-Ming, K., Tung, L.: Forecasting exchange rates using feedforward and recurrent neural networks. J. Appl. Econ. 10(4), 347–364 (1995) 7. Kaastra, I., Boyd, M.: Designing a neural network for forecasting financial and economic time series. Neurocomputing 10(3), 215–236 (1996) 8. Saad, E.W., Prokhorov, D.V., Wunsch, D.C.: Comparative study of stock trend prediction using time delay, recurrent and probabilistic neural networks. IEEE Trans. Neural Netw. 9(6), 1456–1470 (1998)

36

L. Xing et al.

9. Tino, P., Schittenkopf, C., Dorffner, G.: Financial volatility trading using recurrent neural networks. IEEE Trans. Neural Netw. 12(4), 865–874 (2001) 10. Man, Z., Lee, K., Wang, D., Cao, Z., Khoo, S.: Robust single-hidden layer feedforward network-based pattern classifier. IEEE Trans. Neural Netw. Learn. Syst. 23(12), 1974–1986 (2012) 11. Xie, L., Yang, Y., Zhou, Z., Zheng, J., Tao, M., Man, Z.: Dynamic neural modeling of fatigue crack growth process in ductile alloys. Inf. Sci. 364–365, 167–183 (2016) 12. Huang, G.-B., Zhu, Q.-Y., Siew, C.-K.: Extreme learning machine: theory and applications. Neurocomputing 70(1), 489–501 (2006) 13. Man, Z., Lee, K., Wang, D., Cao, Z., Miao, C.: A new robust training algorithm for a class of single-hidden layer feedforward neural networks. Neurocomputing 74(16), 2491–2501 (2011) 14. Park, J.M., Kim, J.H.: Online recurrent extreme learning machine and its application to timeseries prediction. In: 2017 International Joint Conference on Neural Networks (IJCNN), pp. 1983–1990 (2017) 15. Ertugrul, Ö.F.: Forecasting electricity load by a novel recurrent extreme learning machines approach. Int. J. Electr. Power Energy Syst. 78, 429–435 (2016) 16. Zhi, L., Zhu, Y., Wang, H., Xu, Z., Man, Z.: A recurrent neural network for modeling crack growth of aluminium alloy. Neural Comput. Appl. 27(1), 197–203 (2016) 17. Huang, G.-B., Zhou, H., Ding, X., Zhang, R.: Extreme learning machine for regression and multiclass classification. IEEE Trans. Syst. Man Cybern. Part B Cybern. 42(2), 513–529 (2012) 18. Lingras, P., Mountford, P.: Time delay neural networks designed using genetic algorithms for short term inter-city traffic forecasting, pp. 290–299: Springer, Heidelberg (2001) 19. Kim, H.-J., Shin, K.-S.: A hybrid approach based on neural networks and genetic algorithms for detecting temporal patterns in stock markets. Appl. Soft Comput. 7(2), 569–576 (2007) 20. Granger, C.W.J., Huangb, B.-N., Yang, C.-W.: A bivariate causality between stock prices and exchange rates: evidence from recent Asianflu. Q. Rev. Econ. Financ. 40(3), 337–354 (2000) 21. Zhou, B.: High-frequency data and volatility in foreign-exchange rates. J. Bus. Econ. Statistics 14(1), 45–52 (1996) 22. Bauwens, L., Ben Omrane, W., Giot, P.: News announcements, market activity and volatility in the euro/dollar foreign exchange market. J. Int. Money Financ. 24(7), 1108–1125 (2005) 23. Sentiment Regimes. In: Trading on Sentiment 24. Diebold, F., Lee, J.-H., Weinbach, G.C.: Regime switching with time-varying transition probabilities, Federal Reserve Bank of Philadelphia (1993). https://EconPapers.repec.org/ RePEc:fip:fedpwp:93-12 25. Bollen, N.P.B., Gray, S.F., Whaley, R.E.: Regime switching in foreign exchange rates: evidence from currency option prices. J. Econ. 94(1), 239–276 (2000) 26. Balcilar, M., Hammoudeh, S., Asaba, N.-A.F.: A regime-dependent assessment of the information transmission dynamics between oil prices, precious metal prices and exchange rates. Int. Rev. Econ. Financ. 40, 72–89 (2015)

Alzheimer’s Disease Computer Aided Diagnosis Based on Hierarchical Extreme Learning Machine Zhongyang Wang2 , Junchang Xin1,3(B) , Yue Zhao2 , and Qiyong Guo4

3

1 School of Computer Science and Engineering, Northeastern University, Shenyang 110169, China [email protected] 2 Sino-Dutch Biomedical and Information Engineering School, Northeastern University, Shenyang 110169, China Key Laboratory of Big Data Management and Analytics (Liaoning Province), Northeastern University, Shenyang 110169, China 4 Shengjing Hospital of China Medical University, Shenyang 110004, China

Abstract. The usual computer aided diagnosis approaches of Alzheimer’s disease patients based on fMRI often require a lot of manual intervention. By contrast, H-ELM needs only less manual intervention and can extract features by a multi-layer feature representation framework. Therefore, an AD CADx model based on H-ELM is proposed. First, the common spatial pattern is used to extract information from the BOLD signals, and then the features are encoded and trained by H-ELM. H-ELM is used to realize the expression of deep feature of the brain, so as to further improve the diagnostic accuracy. Finally, experimental evaluation proved the effectiveness of the proposed algorithms. Keywords: Alzheimer’s disease · Computer aided diagnosis BOLD signals · Brain functional network · H-ELM

1

·

Introduction

Alzheimer’s disease (AD) is a neurodegenerative disease commonly seen in the elderly people, after heart disease, tumor and cerebrovascular disease [1,2]. A large number of studies have shown that the functional structure of brain in patients with AD can be displayed by resting-state functional magnetic resonance imaging (rs-fMRI) [3]. Compared with other neuroimaging data, fMRI shows more functional changes in the brain than the structure of the brain [4]. The blood oxygenation level dependent (BOLD) signals collected by rs-fMRI have a certain delay compared with the change of neuron signal [5]. It is generally believed that abnormal areas of brain function have been appeared in AD patients [6]. The BOLD signal of the patient’s fMRI data usually shows the difference from the c Springer Nature Switzerland AG 2020  J. Cao et al. (Eds.): ELM 2018, PALO 11, pp. 37–44, 2020. https://doi.org/10.1007/978-3-030-23307-5_5

38

Z. Wang et al.

healthy population, and the analysis of this difference is mostly artificial, which makes the AD study include the influence of subjective factors [7,8]. Research has begun to make use of deep learning for computer aided diagnosis (CADx) of AD patients [9,10], while common deep learning methods, such as DNN, usually require a large number of parameters. By contrast, H-ELM needs only less manual intervention and can extract features by a multi-layer feature representation framework, and the higher layers represent more abstract information than those from the lower ones [11]. Therefore, the feature expression ability of H-ELM is used to realize the multi-layer feature encoding of the brain BOLD signal of the patients of AD and the healthy population, and to classify them by this features to realize the computer aided diagnosis of Alzheimer’s disease, i.e. the AD CADx model based on H-ELM. First, the Common spatial pattern (CSP) method [12] is used to extract the information from the fMRI BOLD signals, and all the data of the experimental control group are converted to the structure that can be used for H-ELM, and then the fMRI data are encoded and classified by H-ELM. The remainder of the paper is organized as follows. Section 2 introduces the proposed approaches. Then, we tested the proposed methods by experiment as shown in Sect. 3. Finally the proposed methods is concluded in Sect. 4.

2 2.1

Methods fMRI Data Preprocessing

For each subject, the first 7 volumes of the functional images were discarded for signal equilibrium and to allow the participant’s adaptation to the circumstances, leaving rest volumes for next steps. Standard preprocessing was applied on rs-fMRI data set of all patients using Data Processing Assistant for Resting State fMRI (DPARSF) toolbox [13] and Statistical Parametric Mapping Software Package (SPM12) package [14]. Slice-timing correction to the last slice was performed. fMRI time-series realigned using a six-parameter rigid-body spatial transformation to compensate for head movement effects. Then all images were normalized into the Montreal Neurological Institute (MNI) space and re-sampled to 3–mm isotropic voxels. A band-pass filtered (0.01–0.08 Hz) was used to retain effective information as much as possible. 2.2

AD CADx Based on H-ELM with BOLD Signals

Overview. For preprocessed fMRI data, the BOLD signal contains information about brain oxygen change related to brain activity, so this information can be used to analyze the changes of the function of brain activity. In the usual signal analysis method, it is necessary to analyze the abnormal part of the signal, which leads to a great deal of computation. However, for Alzheimer’s disease, the abnormality of its brain activity is not concentrated in a certain type of

Alzheimer’s Disease CADx Based on H-ELM

39

Fig. 1. The overview of CADx approaches by BOLD signals

signal, so it can be expressed in the form of all the signals encoded. Therefore, in this paper, an AD CADx model based on H-ELM is proposed, and the framework is given in Fig. 1. Firstly, extract all BOLD signals information from fMRI images data. Then, through Common spatial pattern (CSP), the data can be transformed into the dimension that can be used for ELM encoding and analysing. After this, the information encoding process and training of BOLD signal are realized by H-ELM, and the trained model is used for the auxiliary diagnosis of AD, and finally the classification result of AD is obtained. Data Dimensionality Reduction. In order to facilitate the computation of H-ELM, dimension reduction of fMRI data is needed. Common spatial pattern (CSP) [12] has proven to be an effective method for feature extraction in classifying two classes. CSP is a mathematical procedure used in signal processing for separating a multivariate signal into additive subcomponents which have maximum differences in variance between two windows. Let Xi,1 ∈ RC×P and Xi,2 ∈ RC×P denote fMRI samples of two classes recorded from the i − th trial with C and P being the number of BOLD signals and Time sampling points, respectively. Assume both the fMRI samples have been bandpass filtered at a specified frequency band and mean-removed. CSP aims at finding spatial filter w ∈ RC to transform the fMRI data so that the ratio of data variance between the two classes is maximized  WT W s.t.W2 = 1 (1) w = arg max T 1 W W 2W  Nl  T where l = i=1 Xil Xil Nl with Nl being the number of trials belonging to class l. A matrix W = [w1 , w2 , . . . w2M ] ∈ C×2M including the spatial filters is formed by the eigenvectors corresponding to the M largest and smallest eigenvalues. For a given fMRI sample x, the feature vector is constructed as x = [x1 , x2,... x2M ] with entries T X)), m = 1, 2, . . . 2M xm = log(var(wm

where var(.) denotes the variance operation.

(2)

40

Z. Wang et al.

AD CADX Based on H-ELM with BOLD Signals. The proposed H-ELM by Tang et al. [11] is built in a multilayer manner. The H-ELM training architecture is structurally divided into two separate phases: (1) unsupervised hierarchical feature representation and (2) supervised feature classification. Since the supervised training is implemented by the original ELM, we just introduce the unsupervised hierarchical feature representation. The training set (X, T) is based on the dimensionality reduction. The matrix H1 of the image obtained by eigenvector parameters. Then, the N-layer unsupervised learning is performed to eventually obtain the high-level sparse features. Mathematically, the output of each hidden layer can be represented as Hi = g(Hi−1 β)

(3)

where Hi is the output of the ith layer (i ∈ [1, K]), Hi−1 is the output of the (i − 1)th layer, g() denotes the activation function of the hidden layers, and β represents the output weights. the resultant outputs of the Kth layer, i.e., HK , are viewed as the high-level features extracted from the input data. The next step is to implement the classification of feature data using the original ELM. The feature representation and training process for H-ELM is shown in Algorithm 1. First, calculate the CSP of the data set (line 1). Then the encoding features are calculated (lines 2–5). The parameters (wj , bj ) of the hidden layer nodes are randomly generated (lines 6–7). Then, the matrix H of the image obtained by eigenvector parameters wj and bj . Finally, parameter βout is obtained through H and classification results T (lines 8–9). After training process based on H-ELM, training parameters can be obtained. This means that the H-ELM has been able to detect AD. The specific process of detection is shown in Algorithm 2. The first step is to extract feature vectors from the data (lines 1–4). And then calculate H according to w and b (line 6), The last step is to get the result based on β (line 7).

Algorithm 1. Training process of H-ELM with BOLD.

1 2 3 4 5 6 7 8 9 10

input : Xtrain , Ttrain : fMRI data train set; K: Hidden layer numbers; L: Hidden layer node numbers; output: w, b, βout : Three parameters of ELM. X =CSP (Xtrain ); H0 =X ; for i = 1 to K Hi = g(Hi−1 β); F eature=HK ; for j = 1 to L do Randomly generate hidden node parameters (wj , bj ); Calculating the output matrix H based on(wj , bj , F eature); Calculation βout = H† T; Return wj , bj and βout ;

Alzheimer’s Disease CADx Based on H-ELM

41

Algorithm 2. CADx model based on H-ELM with BOLD. input : Xtest , Ttest : fMRI data train set; w, b, βout : Hidden layer numbers; K: Hidden layer numbers; L: Hidden layer node numbers; output: Tresult : Classification results. X =CSP (Xtest ); H0 =X ; for i = 1 to K Hi = g(Hi−1 β); F eature=HK ; Calculating the output matrix H based on(wj , bj , F eature); Calculation Tresult =HT; Return Tresult ;

1 2 3 4 5 6 7 8

3

Experiments Evaluation

3.1

Experimental Data

The accuracy of AD classification is verified by data obtained from Alzheimers Disease Neuroimaging Initiative (ADNI) database1 . Ethical statements are present by ADNI. A total of 400 rs-fMRI data subjects including 200 AD patients and 200 health control (HC) subjects were obtained. The details are shown in Table 1. Table 1. Details of ADNI data set Dataset Subjects (Male/Female) Ages

3.2

Slices

TE

TR

AD

105/95

73.6 ± 15 3.31 mm 30 ms 3.0 s

HC

103/97

75.1 ± 13 3.31 mm 30 ms 3.0 s

Experimental Set-Ups

The experiment is ran on the computer with the Intel Core i3 2.7 GHz CPU, 6 GB RAM and MATLAB2013a platform. In this section, we will demonstrate the selection of hyper parameters in H-ELM and ELM schemes. Compared with the traditional neural networks training algorithms, much less parameters need to be chosen in the training process of H-ELM and ELM. We can see that two user specified parameters are required, i.e., the parameter C for the regularized least mean square calculation, and the number of hidden nodes L. In the simulations, C is set as 10−10 , 10−9 , ..., 109 , 1010 , and L is set as 100, 200, ..., 2000. The hiddenlayer number n of the encoding progress in H-ELM is set 1, 2, ..., 20. And the number of hidden nodes N of encoding layer is set as 100, 200, ..., 2000. 1

http://adni.loni.usc.edu/.

42

Z. Wang et al.

3.3

Experimental Results

The performance of CADx of Alzheimer’s disease by BOLD signals between H-ELM and ELM is compared. In this section, we will demonstrate the selection of hyperparameters in H-ELM and ELM schemes. As [11], the parameter C and the number of hidden nodes L are required. Figure 2(a) and (d) shows the learning accuracies of H-ELM and ELM in the L subspace, where the parameter C is prefixed. As shown in Figure, H-ELM can follow a similar convergence property of ELM but with higher accuracy, and the performances tend to be quite stable. Meanwhile, in Fig. 2(b) and (e), the learning accuracies of H-ELM and ELM in the C subspace is shown by the parameter prefixed L. As C increases, the accuracy of H-ELM changes rapidly until convergence to a better performance than ELM. The impacts of C and L are different in H-ELM and ELM, there are intersections under the action of the two parameters. The 3-D accuracy curves of H-ELM and ELM are further shown in Fig. 2(c) and (f) the learning performance of H-ELM is all affected by the parameters C and L. And the H-ELM has a better performance than ELM. As shown in all Fig, the L can be selected from 1000 to 2000, and C can be selected from 100 to 1010 . In addition, the hidden-layer number of the encoding progress n, and the number of hidden nodes of encoding layer N in H-ELM can also affect the accuracy. Figure 3(a) shows the learning accuracies of H-ELM and ELM in the N subspace, where the parameter n is prefixed. Figure 3(b) shows the learning accuracies of H-ELM and ELM in the n subspace, where the parameter N is prefixed. The 3-D accuracy curves of H-ELM and ELM are further shown in Fig. 3(c). The simultaneous effect of N and N affects the accuracy. When n and N increase to a certain extent, accuracy is maintained in a relatively stable state. 0.85

0.9 0.8

0.8

1

0.7

0.65

Testing accuracy

0.7

0.5 0.4

C=1e0 C=1e2 C=1e4 C=1e6 C=1e8

200

(a)

400

600

800

1000 1200 L/H-ELM

1400

1600

1800

0.1

10

-8

(b)

0.75

0.6 0.5 0.4

10

10

10 5

2000 1500

10 0 10

0

H-ELM in terms of L

0.7

0.2

C=1e0 C=1e2 C=1e4 C=1e6 C=1e8

0.2

2000

0.8

0.3

0.3

0.6

0.55

0.9

0.6 Testing accuracy

Testing accuracy

0.75

10

-6

10

-4

10

-2

0

10 10 C/H-ELM

2

10

4

10

6

10

8

10

(c)

H-ELM in terms of C

1000

-5

C/H-ELM

10

10

-10

500 100

L/H-ELM

H-ELM in terms of (C, L)

0.9 0.8

0.7

1

0.7 Testing accuracy

0.6

0.5 0.4

C=1e0 C=1e2 C=1e4 C=1e6 C=1e8

0.5

200

400

(d)

600

800

1000 1200 L/ELM

1400

1600

ELM in terms of L

1800

0.7 0.6 0.5 0.4 0.2

C=1e0 C=1e2 C=1e4 C=1e6 C=1e8

0.2 0.1

2000

0.8

0.3

0.3

0.55

0.45

0.9

0.6 Testing accuracy

Testing accuracy

0.65

10 10 10

5

2000 10

1500

0

10

0 10

-8

(e)

10

-6

10

-4

10

-2

0

10 10 C/ELM

2

10

4

10

6

ELM in terms of C

10

8

10

C/ELM

10

(f)

1000

-5

10 -10

500 100

L/ELM

ELM in terms of (C, L)

Fig. 2. Accuracy by BOLD signals in (C, L) subspace for the H-ELM and ELM

Alzheimer’s Disease CADx Based on H-ELM 1

1

0.9

0.9

0.8

0.8

43

1

0.7

0.6

0.7

1 5 10 15 20

200

(a)

400

600

800

1000 1200 N/H-ELM

1400

1600

1800

H-ELM in terms of N

0.4

0.7 0.6 0.5 0.4 0.2 20

100 500 1000 1500 2000

0.5

2000

0.8

0.3

0.6

0.5

0.4

Testing accuracy

Testing accuracy

Testing accuracy

0.9

2

(b)

4

6

8

10 12 n/H-ELM

14

16

18

H-ELM in terms of n

15

2000 1500

10

1000

5 n/H-ELM

20

(c)

1

500 100

N/H-ELM

H-ELM in terms of (n, N)

Fig. 3. Accuracy by BOLD signals in (n, N) subspace for the H-ELM

4

Conclusion

In order to solve the problem of a large number of artificial parameters in computer-aided diagnosis of Alzheimer’s disease. H-ELM is used to realize the encoding of BOLD signal characteristics in fMRI data, so as to realize computeraided diagnosis of Alzheimer’s disease. Finally, our extensive experimental studies using real data show that our proposed approach can fulfill the requirements of real-world applications. Acknowledgments. This research was partially supported by the National Natural Science Foundation of China (Nos. 61472069, 61402089 and U1401256), the Fundamental Research Funds for the Central Universities (Nos. N161602003, N171607010, N161904001, and N160601001), the Natural Science Foundation of Liaoning Province (No. 2015020553).

References 1. Cass, S.P.: Alzheimer’s disease and exercise: a literature review. Curr. Sport. Med. Rep. 16(1), 19–22 (2017) 2. Bassiony, M.M., Lyketsos, C.G.: Delusions and hallucinations in alzheimer’s disease: review of the brain decade. Psychosomatics 44(5), 388–401 (2003) 3. Rathore, S., Habes, M., Iftikhar, M.A., et al.: A review on neuroimaging-based classification studies and associated feature extraction methods for alzheimer’s disease and its prodromal stages. Neuroimage 155, 530–548 (2017) 4. Logothetis, N.K.: What we can do and what we cannot do with fMRI. Nature 453(7197), 869–878 (2008) 5. Hennig, J., Speck, O., Koch, M.A., et al.: Functional magnetic resonance imaging: a review of methodological aspects and clinical applications. J. Magn. Reson. Imaging 18(1), 1–15 (2003) 6. Grossman, M., Peelle, J.E., Smith, E.E., et al.: Category-specific semantic memory: converging evidence from bold fmri and alzheimer’s disease. Neuroimage 68, 263– 274 (2013) 7. Galvin, J.E., Price, J.L., Yan, Z., et al.: Resting bold fMRI differentiates dementia with lewy bodies vs alzheimer disease. Alzheimers Dement. J. Alzheimers Assoc. 7(4), e69–e69 (2011)

44

Z. Wang et al.

8. Cantin, S., Villien, M., Moreaud, O., et al.: Impaired cerebral vasoreactivity to CO2 in alzheimer’s disease using BOLD fMRI. Neuroimage 58(2), 579–587 (2011) 9. Liu, S., Liu, S., Cai, W., et al.: Early diagnosis of alzheimer’s disease with deep fearning. In: IEEE 11th International Symposium on Biomedical Imaging, pp. 1015–1018. IEEE Press, Beijing (2014) 10. Sarraf, S., Tofighi, G.: Deep Learning-based pipeline to recognize alzheimer’s disease using fMRI data. In: Proceedings of 2016 Future Technologies Conference, pp. 816–820. IEEE Press, San Francisco, CA (2017) 11. Tang, J., Deng, C., Huang, G.B.: Extreme learning machine for multilayer perceptron. IEEE Trans. Neural Netw. Learn. Syst 4, 809–821 (2017) 12. Zhang, Y., Wang, Y., Zhou, G., et al.: Multi-kernel extreme learning machine for EEG classification in brain-computer interfaces. Expert Syst. Appl. 96, 302–310 (2018) 13. Yan, C.G., Zane, Y.F.: DPARSF: a MATLAB toolbox for ‘pipeline’ data analysis of resting-state fMRI. Front. Syst. Neurosci. 4(13), 13 (2010) 14. Eickhoff, S.B., et al.: A new SPM toolbox for combining probabilistic Cytoarchitectonic maps and functional imaging data. Neuroimage 25(4), 1325C–1335C (2005) 15. Tzouriomazoyer, N., Landeau, B., Papathanassiou, D., et al.: Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain. Neuroimage 15(1), 273–289 (2002) 16. Khazaee, A., Ebrahimzadeh, A., Babajaniferemi, A.: Classification of patients with MCI And AD from healthy controls using directed graph measures of resting-state fMRI. Behav. Brain Res. 322(PtB), 339–350 (2017)

Key Variables Soft Measurement of Wastewater Treatment Process Based on Hierarchical Extreme Learning Machine Feixiang Zhao1 , Mingzhe Liu1(B) , Binyang Jia2 , Xin Jiang1 , and Jun Ren3 1

State Key Laboratory of Geohazard Prevention and Geoenvironment Protection, Chengdu University of Technology, Chengdu, Sichuan, China [email protected] 2 Chengdu Academy of Environmental Sciences, Chengdu, Sichuan, China [email protected] 3 Institute of Natural and Mathematical Sciences, Massey University, Palmerston North, New Zealand [email protected]

Abstract. Many process variables are included in the wastewater treatment process. The realization of real-time detection of as many process variables as possible is of great significance for improve the quality of wastewater treatment. Due to cost constraints, it is difficult to achieve real-time monitoring of some important process variables. Soft measurement scheme, which modeling the relationship between easy-to-measure variables and hard-to-measure variables to achieve the best estimate of the latter is a common solution. This paper proposes a key variables soft measurement scheme for wastewater treatment process based on hierarchical extreme learning machine (HELM). The proposed scheme takes some of the known variables that are easy to measure as inputs, and then implement a best estimate of unknown variables that are difficult to measure. At the same time, the selective moving window strategy (SMW) is used to update the training datasets. Experiments show that the proposed scheme has excellent performance on many key indicators. Keywords: Hierarchical extreme learning machine Wastewater treatment · Feature extraction

1

·

Introduction

The wastewater treatment process contains a variety of complex chemical reactions. Meanwhile a large number of electrical automation equipments and human operations are included. Wastewater treatment process is easily affected by unfavorable factors such as equipment failures and manipulate error, resulting in secondary environmental pollution and other issues. The more types of process variables in wastewater treatment are detected, the easier it is to detect failures. However, due to cost constraints, only a limited number of process variables c Springer Nature Switzerland AG 2020  J. Cao et al. (Eds.): ELM 2018, PALO 11, pp. 45–54, 2020. https://doi.org/10.1007/978-3-030-23307-5_6

46

F. Zhao et al.

can be detected. In order to detect as many process variables as possible, soft measurement scheme is a commonly used solution [1,11]. Since the 1990s, many classical high-performance algorithms are applied in the field of soft measurement of key variables in wastewater treatment processes, such as generalized damped least squares (GDLS) [14], stacked neural networks (SNN) [15], polynomial neural network (PNN) [16], artificial neural network (ANN) [12,13] and support vector machine (SVM) [17]. The wastewater treatment process can be viewed as a time variable system, the data stored long ago may lose timeliness. Therefore, the training datasets of the model must be update in time, and the old samples should be deleted while adding new samples. Xiong et al. [10] proposed a novel selective update strategy, which determines whether the training datasets needs updating based on the prediction error. Users can set the error threshold manually to achieve trade-off between update efficiency and model performance. Xiong et al. [10] combines it with probabilistic principle component analysis (PPCA) algorithm and gaussian process regression (GPR) algorithm to form a soft sensor called SMW-PPCA-GPR. The extreme learning machine (ELM) proposed by Huang et al. [5] is an algorithm for solving single hidden layer neural networks. For the traditional neural network, especially the single-hidden layer feed-forward neural network (SLFN), ELM is faster than the traditional learning algorithm on the premise of guaranteeing the learning accuracy, which is the most prominent feature of ELM. Because of its advantages such as fast learning speed, strong generalization ability and effectively avoid over-fitting, etc, ELM algorithm has been widely used in recent years. Its application areas include computer vision, video tracking, stock trend analysis and so on. The original ELM has a single-layer structure. Even if a large number of hidden layer nodes are selected, the learning effect on some complex features is still not ideal. For this issue, Tang et al. [4] proposed an improved ELM algorithm and names it as hierarchical extreme learning machine (HELM). HELM extends ELM from a single layer to multiple layers. HELM builds a sparse autoencoder module based on the original ELM and adds it to the traditional ELM framework, this module is used to extract features from the raw data. Through its application in a variety of commonly used datasets, HELM exhibits greater performance than traditional ELM. This paper combines the HELM algorithm with the SMW proposed by Xiong et al. [10], proposes a soft measurement scheme for key variables of the wastewater treatment process. The HELM architecture in this scheme contains three layers of feature extraction. The rest of this paper is given as follows. The second part elaborates the theory of ELM and HELM in detail. The third part compares the modeling performance of the two schemes. Finally, we make our conclusion in the fourth part.

Key Variables Soft Measurement

2 2.1

47

Theory Extreme Learning Machine

The single hidden layer feed-forward neural network is applied in many fields with its excellent learning ability. However, the disadvantages of traditional training algorithms such as slow training speed and easy to fall into local minimum limit their further development. Extreme learning machine algorithm randomly generates the connection weights among the input layer, the hidden layer and the bias of the hidden layer. To obtain the only optimal solution, only the number of neurons in the hidden layer needs to be set manually [2,3]. Compared with traditional algorithms such as BP algorithm, this algorithm has faster learning speed and better generalization performance. Given a single hidden layer feed-forward neural network. The number of input layer neurons is n. The number of hidden layer neurons is L, and the number of output layer neurons is m. The weights matrix between the input layer and the hidden layer is W : ⎡ ⎤ ⎡ T⎤ w11 w12 · · · w1n w1 ⎢ w21 w22 · · · w2n ⎥ ⎢w2T ⎥ ⎢ ⎥ ⎢ ⎥ (1) =⎢ . W =⎢ . ⎥ .. . . .. ⎥ ⎣ .. ⎣ .. ⎦ . . ⎦ . T wL

wL1 wL2 · · · wLn

L×n

The weight matrix between the hidden layer and output layer is β: ⎡ T⎤ ⎡ ⎤ β1 β11 β12 · · · β1m ⎢β2T ⎥ ⎢ β21 β22 · · · β2m ⎥ ⎢ ⎥ ⎢ ⎥ β=⎢ . ⎥ =⎢ . .. . . .. ⎥ ⎣ .. ⎦ ⎣ .. . . ⎦ . T βL1 βL2 · · · βLm βL L×m

(2)

Offset vector of hidden layer is b: ⎡

⎤ b1 ⎢ b2 ⎥ ⎢ ⎥ b=⎢ . ⎥ ⎣ .. ⎦ bL

(3)

L×1

Given a dataset containing N samples: (xj , tj ), j = 1, 2, ..., N . For any sample xj = (xj1 , xj2 , ..., xjn )T ∈ Rn , the corresponding expected output vector is tj = (tj1 , tj2 , ..., tjm )T ∈ Rm . Combine all expected output vectors into matrix T : ⎡ T⎤ ⎡ ⎤ t1 t11 t12 · · · t1m ⎢ tT2 ⎥ ⎢ t21 t22 · · · t2m ⎥ ⎢ ⎥ ⎢ ⎥ (4) T =⎢ . ⎥ =⎢ . .. . . .. ⎥ ⎣ .. ⎦ ⎣ .. . . ⎦ . tTN

N ×m

tN 1 t N 2 · · · t N m

48

F. Zhao et al.

Assuming that the output of sample xj through this architecture is oj , j = 1, 2, ..., N . The standard mathematical model of the neural network architecture can be expressed as formulas (5), (6): L 

g(wi · xj + bi )βi = oj , j = 1, 2, ..., N,

(5)

i=1 N 

oj − tj  = 0.

(6)

i=1

Where g(·) is the activation function. Combine (5) and (6) into formula (7): L 

g(wi · xj + bi )βi = tj , j = 1, 2, ..., N

(7)

i=1

From (7), it is necessary to solve the matrix W and b before solving the matrix β. Rewrite (7) as follows: Hβ = T Where:

⎤ g(w1 · x1 + b1 ) · · · g(wL · x1 + bL ) ⎥ ⎢ .. .. .. =⎣ ⎦ . . .

(8)

⎡



H = H(W, b) = hij N ×L

(9)

g(w1 · xN + b1 ) · · · g(wL · xN + bL )

(8) is equivalent to the following optimization problem: β = arg min Hβ − T 

(10)

β = H † T

(11)

β

The solution is: †

T

−1

T

H = (H H) H is the Moore-Penrose generalized inverse matrix of the hidden layer output matrix H. When HH T is not singular, H † = H T (HH T )−1 . When H T H is not singular, H † = (H T H)−1 H T . To obtain a more stable solution, according to the principle of ridge regression [6], formula (10) is usually added with a constraint factor. When H T H is not singular, it turns into: 1 (12) β = ( + H T H)−1 H T H c After deeply researching the single hidden layer feed-forward neural network, Huang et al. [3,7] proposes and proves the following theories: Given a single hidden-layer feed-forward neural network with n-L-m structure and a dataset containing N samples (xj , tj ), j = 1, 2, ..., N . For any sample (xj = (xj1 , xj2 , ..., xjn )T ∈ Rn ), the corresponding expected output vector is tj = (tj1 , tj2 , ..., tjm )T ∈ Rm . Given the activation function g : R −→ R is infinite differentiable on any interval. If randomly generated i and bi are distributed over any interval of Rn and R with any continuous probability, there are:

Key Variables Soft Measurement

49

Conclusion 1: Hidden layer output matrix H is reversible with probability 1; Conclusion 2: For any  > 0, Hβ −T F =  is established with probability 1. According to this theory, the general steps of the ELM algorithm are: Set inputs, including the number of hidden neurons L, constraint parameter c, training dataset and activation function g(·); According to this theory, the general steps of the ELM algorithm are: Set inputs, including the number of hidden neurons L, constraint parameter c, training dataset and activation function g(·); Step 1: Randomly arrange the input weight matrix W and the hidden layer bias vector b; Step 2: Calculate hidden layer output matrix H; Step 3: Calculate the output weight matrix β according to formula (11). 2.2

Hierarchical Extreme Learning Machine

The inherent structure of the traditional ELM leads to its inability to exhibit consistently high performance when dealing with certain complex natural signals. In view of this, Tang et al. [4] proposed a novel hierarchical extreme learning machine (HELM) architecture. The architecture consists of two basic modules: the feature extraction module and the classification/regression module based on the traditional ELM. Specifically, the first module consists of an unsupervised sparse auto-encoder based on ELM. Tang et al. [4] applies it to the design of auto-encoder based on the maximum approaching capability of ELM. At the same time L1 regularization was introduced into the design of auto-encoder to obtain more sparse implicit information. The module can extract sparse features of the raw data. In general, the purpose of an auto-encoder is to construct a function h(A, b, x) that satisfies h(A, b, x) ≈ x. In other words, an auto-encoder is a neural network that reproduces the input signal as much as possible [8]. Unlike traditional auto-encoder algorithms, the unsupervised sparse auto-encoder in the HELM architecture have random weights. The construction of the unsupervised sparse auto-encoder can be expressed using the optimization formula (13). oβ = arg min Hβ − X2 + λ1 β1 β

(13)

Where X represents the original data, H represents the random mapping output matrix, and β is the hidden layer weight. This optimization problem is solved by the fast iterative shrinkage-threshold (FIST) algorithm [18]. 2.3

Selective Moving Window Strategy

Due to the time-variation of each process variable in the wastewater treatment process, the historical data that exist for a long time in the training dataset may no longer reflect the precise working conditions. It is necessary to update

50

F. Zhao et al.

the training dataset used for soft measurement. But excessive update frequency will also increase the time cost. In view of this, Xiong et al. [10] proposed the selective moving window strategy (SMW) on the basis of the moving window strategy (MW) [9], and determined whether to update the training datasets by setting a threshold for the prediction error.

3 3.1

Experiments UCI Dataset

The wastewater treatment process dataset from the UCI machine learning repository is used to evaluate the performance of the SMW-PPCA-GPR proposed by Xiong et al. [10]. The dataset contains 380 complete samples and each sample has 38 attributes.

Fig. 1. Flow chat for SMW-HELM.

Key Variables Soft Measurement

51

For fairness, this paper combines SMW and HELM algorithm to form SMWHELM scheme, and then compares it with the SMW-PPCA-GPR scheme in UCI wastewater treatment process dataset. The flow chart of SMW-HELM is shown in Fig. 1. In order to compare the performance of the above two schemes, this paper selects some representative evaluation indicators, including root mean square error (RMSE) and maximum absolute error (MAE). The formulas are as shown in (14) and (15).

N

1  (yi − yi )2 (14) RM SE =  N i=1 M AE = max | yi − yi |, i = 1, 2, ..., N

(15)

Where yi is the estimate of the process variable, yi is the true value of the process variable. Conventional wastewater treatment generally adopts activated sludge method. The flow chart is shown in Fig. 2. The process includes a number of process variables, of which biochemical oxygen demand (BOD) is an important indicator of the content of inorganic pollutants in water. The accurate measurement of this variable is of great significance for monitoring the quality of the wastewater treatment process. However, the process of oxidative decomposition of inorganic pollutants by microorganisms is very slow. At the same time, the measurement of biochemical oxygen demand based on hardware can easily receive interference. This leads to time-consuming and inaccurate measurement of biochemical oxygen demand. Therefore, the biochemical oxygen demand of primary settler and secondary settler is used as the indices to be estimated. The UCI dataset is divided into two parts, which are the training dataset (numbers:300) and the test dataset (numbers:80). The remaining 36 attributes in the 38 attributes are used as input variables after being centralized and normalized, and then two regression schemes above is used to estimate the biochemical oxygen demand.

Fig. 2. Wastewater treatment process based on activated sludge process.

52

F. Zhao et al.

The predicted results of biochemical oxygen demand of the above two schemes are shown in Fig. 3 and Table 1. From the experimental results, we can see that compared with SMW-PPCA-GPR, the SMW-HELM proposed in this paper has the best performance. 100

110

90

100 90

70

Concentration of BOD

Concentration of BOD

80

60 50 40 30

70 60 50 40 30

20 10

80

20

0

10

20

30

40

50

60

70

80

10

0

10

Test samples

20

30

40

50

60

70

80

Test samples

Real Content SMW-PPCA-GPR SMW-HELM

Real Content SMW-PPCA-GPR SMW-HELM

(a)

(b)

Fig. 3. Prediction results of BOD. (a) Comparison among SMW-PPCA-GPR and SMW-HELM in primary settler. (b) Comparison among SMW-PPCA-GPR and SMWHELM in secondary settler.

Table 1. The performance of two schemes Location

Models

Primary settler

SMW-PPCA-GPR 10.1519 17.8621 SMW-HELM 2.8247 4.8839

Secondary settler SMW-PPCA-GPR SMW-HELM

4

RMSE

MAE

9.0772 14.9791 1.1462 1.9618

Conclusion

This paper combines the SMW method with the HELM algorithm and proposes a soft measurement scheme for key process variables that are difficult to measure in the wastewater treatment process. This scheme was applied to the UCI sewage treatment dataset, and the BOD were predicted. Compared with SMW-PPCAGPR, the scheme showed better performance. The experimental result proves the actual engineering value of SMW-HELM, and further proves the superiority of HELM in features extraction relative to PPCA.

Key Variables Soft Measurement

53

References 1. Huang, M., Ma, Y., Wan, J., Chen, X.: A sensor-software based on a genetic algorithm-based neural fuzzy system for modeling and simulating a wastewater treatment process. Appl. Soft Comput. 27, 1–10 (2015). https://doi.org/10.1016/ j.asoc.2014.10.034 2. Trends in extreme learning machines: a review. Neural Netw. 61, 32–48 (2015). https://doi.org/10.1016/j.neunet.2014.10.001 3. Huang, G.B., Zhu, Q.Y., Siew, C.K.: Extreme learning machine: a new learning scheme of feedforward neural networks. In: 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No. 04CH37541). vol. 2, pp. 985–990, July 2004. https://doi.org/10.1109/IJCNN.2004.1380068 4. Tang, J., Deng, C., Huang, G.: Extreme learning machine for multilayer perceptron. IEEE Trans. Neural Netw. Learn. Syst. 27(4), 809–821 (2016). https://doi.org/10. 1109/TNNLS.2015.2424995 5. Huang, G.B., Zhu, Q.Y., Siew, C.K.: Extreme learning machine: theory and applications. Neurocomputing 70(1), 489–501 (2006). https://doi.org/10.1016/j. neucom.2005.12.126. Neural Networks 6. Marquardt, D.W., Snee, R.: Ridge regression in practice. American Statistician - AMER STATIST 29, 3–20 (1975). https://doi.org/10.1080/00031305.1975. 10479105 7. Huang, G.B.: Learning capability and storage capacity of two-hidden-layer feedforward networks. IEEE Trans. Neural Netw. 14(2), 274–281 (2003). https://doi. org/10.1109/TNN.2003.809401 8. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006). https://doi.org/10.1126/science. 1127647 9. Xianghua, C., Ouguan, X., Hongbo, Z.: Recursive PLS soft sensor with moving window for online PX concentration estimation in an industrial isomerization unit. In: 2009 Chinese Control and Decision Conference, pp. 5853–5857, June 2009. https://doi.org/10.1109/CCDC.2009.5195246 10. Xiong, W., Shi, X.: Soft sensor modeling with a selective updating strategy for gaussian process regression based on probabilistic principle component analysis. J. Frankl. Inst. 355(12), 5336–5349 (2018). https://doi.org/10.1016/j.jfranklin. 2018.05.017 11. Kadlec, P., Gabrys, B., Strandt, S.: Data-driven soft sensors in the process industry. Comput. Chem. Eng. 33(4), 795–814 (2009). https://doi.org/10.1016/ j.compchemeng.2008.12.012 12. Zhang, Z., Wang, T., Liu, X.: Melt index prediction by aggregated rbf neural networks trained with chaotic theory. Neurocomputing 131, 368–376 (2014). https:// doi.org/10.1016/j.neucom.2013.10.006 13. Gonzaga, J., Meleiro, L., Kiang, C., Filho, R.M.: Ann-based soft-sensor for realtime process monitoring and control of an industrial polymerization process. Comput. Chem. Eng. 33(1), 43–49 (2009). https://doi.org/10.1016/j.compchemeng. 2008.05.019 14. Yoo, C.K., Lee, I.B.: Soft sensor and adaptive model-based dissolved oxygen control for biological wastewater treatment processes. Environ. Eng. Sci. 21(3), 331–340 (2004). https://doi.org/10.1089/109287504323066978 15. Zyngier, D., Araujo, O., Lima, E.: Soft sensors with white- and black-box approaches for a wastewater treatment process. Braz. J. Chem. Eng. 17(4–7), 433–440 (2000). https://doi.org/10.1590/S0104-66322000000400008

54

F. Zhao et al.

16. Kim, Y., Bae, H., Poo, K., Kim, J., Moon, T., Kim, S., Kim, C.: Soft sensor using PNN model and rule base for wastewater treatment plant. In: Wang, J., Yi, Z., Zurada, J.M., Lu, B.L., Yin, H. (eds.) Advances in Neural Networks - ISNN 2006, pp. 1261–1269. Springer, Heidelberg (2006) 17. Kaneko, H., Funatsu, K.: Adaptive soft sensor based on online support vector regression and Bayesian ensemble learning for various states in chemical plants. Chemom. Intell. Lab. Syst. 137, 57–66 (2014). https://doi.org/10.1016/j.chemolab. 2014.06.008 18. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)

A Fast Algorithm for Sparse Extreme Learning Machine Zhihong Miao1(B) and Qing He2,3 1

2

Department of management, China People’s Police University, Langfang 065000, China [email protected] Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing 100190, China [email protected] 3 University of Chinese Academy of Sciences, Beijing 100049, China

Abstract. As a more compact network, sparse ELM is an alternative model of extreme learning machine (ELM) for classification, which requires less memory space and testing time than conventional ELMs. In this paper, a fast training algorithm (FTA-sELM) specially developed for sparse ELM is proposed, which improves training speed, achieves better generalization performance and further promotes the application of sparse ELM in large data problem. The proposed algorithm breaks the large quadratic programming (QP) problem of sparse ELM into a series of two-dimensional sub-QP problems, specifically. In every iteration, Newton’s method is employed to solve the optimal solution for each sub-QP problem. Moreover, a new clipping scheme for Lagrange multipliers is presented, which improves convergence performance. Keywords: Sparse extreme learning machine (sparse ELM) Classification · Quadratic programming (QP) · Sequential minimal optimization (SMO)

1

·

Introduction

Differing from the early studies for ELM [1–10], in [11], a unified ELM has been proposed, which provides a unified framework to accommodate a wide range of feature mappings and different learning methods. Moreover, both LSSVM and PSVM can also be simplified under this unified framework [11]. Based on the Karush-Kuhn-Tucker (KKT) theorem [12], the constrained optimization problem for a unified ELM can be converted into a dual optimization problem, which can be solved by analytic method via a matrix equation [13–15]. However, in consideration of using equality constraints in unified ELM, sparsity would Supported by a grant from China National Natural Science Foundation under Project of China (No. 61573335, U1811461, 91546122) and also a grant from China National Key Research and Development Program (No. 2017YFB1002104). c Springer Nature Switzerland AG 2020  J. Cao et al. (Eds.): ELM 2018, PALO 11, pp. 55–64, 2020. https://doi.org/10.1007/978-3-030-23307-5_7

56

Z. Miao and Q. He

be lost [16], i.e. almost all Lagrange variables of dual problem are non-zero. This leads to need a large amount of memory space and plenty of testing time, specifically for large scale applications. Instead of using equality constraints as the primal unified ELM, an alternative optimization problem based ELM was suggested in [8,16] for classification, and a comprehensive sparse ELM was proposed. Compared with the primal unified ELM, a more compact network is constructed by this sparse ELM, which requires much less memory space and testing time. Furthermore, it has been proved that the sparse ELM unifies several classification methods, such as SLFNs, conventional SVM and RBF networks, and that both random hidden nodes and kernels can be used in sparse ELM [16]. Seeing that inequality constraints are employed in the sparse ELM aforementioned, it is necessary to adopt one iterative method to search the optimal solution. Hence, in [16], by using the principle of sequential minimal optimization (SMO) [17], a training algorithm based on iterative processes was proposed, in which the large QP problem of sparse ELM is divided into a series of smallest possible sub-QP problems with each sub-problem only needing to update one Lagrange multiplier. The global optimal solution of the optimization problem would be approximately achieved in a finite number of iterations. However, since only one Lagrange multiplier is updated in each iteration, many more iterations may be needed to find the optimal solution, the training time would become longer. Inspired by the sequential minimal optimization (SMO) for SVM and the training algorithm for sparse ELM proposed in [16], this paper presents another training algorithm (FTA-sELM) for sparse ELM. Instead of that only one Lagrange multiplier is replaced at every iteration in [16], a pair of Lagrange multipliers are simultaneously updated in our training algorithm. Moreover, a new clipping scheme which maps a point in two-dimensional space into a square region [0, C] × [0, C] is proposed. The training speed of the proposed algorithm could be faster than that of the primal algorithm (proposed in [16]) for sparse ELM, especially when dealing with large sample problems.

2

Briefs of Sparse ELM

In [8–16], based on an optimization scheme, a sparse ELM for classification problem has been proposed, in which inequality constraints are used. The Lagrange multipliers αi ’s can be formed as a sparse representation. A typical classification problem of sparse ELM with a single-output node can be formulated as Minimize: LP s = 12 β2 + C

N  i=1

Subject to: ti h(xi )β ≥ 1 − ξi ,

ξi

(1)

ξi ≥ 0, (i = 1, 2, · · · , N )

where β = [β1 , · · · , βM ]T is the vector of the output weights between the M hidden-layer nodes and the output node, ξi is the training error with respect

A Fast Algorithm for Sparse Extreme Learning Machine

57

to the training sample (xi , ti ) (xi ∈ Rp , ti ∈ {1, −1}), C is a user-specified parameter that balances between structural risk and minimal training errors, and h(x) is a random feature mapping from input space to a higher dimensional space. The Lagrange function of the above optimization problem (1) can be formed as LDs =

N N N    1 β2 + C ξi − μi ξi − αi (ti h(xi )β − (1 − ξi )) 2 i=1 i=1 i=1

(2)

where αi ’s are Lagrange multipliers that correspond to training samples (xi , ti )’s, respectively. N It is easy to prove that β = i=1 αi ti h(xi ) and C = αi +μi would be satisfied at the optimal solution of function LDs [16]. Then, according to KKT theorem [12], the above optimization problem (1) could be solved through the following dual quadratic optimization problem: Minimize: LDs =

1 2

N  N 

αi αj ti tj Ω(xi , xj ) −

i=1 j=1

Subject to: 0 ≤ αi ≤ C, (i = 1, 2, · · · , N )

N 

αi

i=1

where Ω is the ELM kernel matrix:  h(xi ) · h(xj )T , feature mapping; Ω(xi , xj ) = kernel method. K(xi , xj ),

(3)

(4)

and K(·, ·) is a kernel function, which must satisfy Mercer’s conditions [16]. Then, the KKT optimality conditions of (3) are μi ξi = 0, αi (ti h(xi )β − (1 − ξi )) = 0 (i = 1, · · · , N )

(5)

If αs1 , αs2 , · · · , αsM are all non-zero Lagrange multipliers solutions of above dual problem, then, the output of sparse ELM is f (x) = h(x)

M 

αsi tsi h(xsi )T =

i=1

M 

αsi tsi Ω(x, xsi )

(6)

i=1

where xsi is support vector (SV), and M is the number of SVs. Consequently, the last term of above formula is corresponding to a sparse network. In order to get the optimal solution of above dual problem, a sequential minimal optimization (SMO)-based optimization algorithm has been proposed in [16]. In that algorithm, the following iteration formula is used to update Lagrange multiplier αi : αi∗ = αi +

1 − ti f (xi ) Ω(xi , xi )

(7)

58

Z. Miao and Q. He

Since all αi ’s must be bounded in [0, C], the following clipping scheme was employed in [16]. ⎧ ⎨ 0, αi∗ < 0 clip αi = αi∗ , 0 ≤ αi∗ ≤ C (8) ⎩ C, αi∗ > C Moreover, a vector d = (d1 , d2 , · · · , dN )T , which is a direction for αi to be updated, was defined as follows (Definition 5.1 in [16]): ⎧ αi = 0 ⎨ 1, di = −sign(gi ), 0 < αi < C (9) ⎩ −1, αi = C where gi =

3

∂LDs ∂αi .

Fast Training Algorithm for Sparse ELM Problem

Essentially, sparse ELM (1) is a quadratic programming (QP) problem. Similar to conventional SVM, some iterative computation strategies such as SMO can be used to search the optimal solution. In the primal algorithm proposed in [16] (mentioned in Sect. 2), as only one Lagrange multiplier is updated in each iteration, it may lead to expend much more number of iteration to search the optimal solution, then, training process would be time-consuming. In this section, an alternative training algorithm is proposed for sparse ELM. 3.1

Iteration Strategy

Differing from the previous algorithm, two Lagrange multipliers are selected to be simultaneously updated at the current iteration in our algorithm. In other words, the large QP problem of sparse ELM is specifically divided into a series of two-dimensional sub-QP problems, and in each of which, optimal solution is obtained by using Newton’s method [18]. Specifically, the search direction for optimal solution is usually not along a coordinate axis in each iteration as the algorithm in [16] or SMO. This scheme will speed up the solving process and use less number of iteration than that in the previous algorithm in [16]. The global optimal solution of the optimization problem would be achieved approximately with less number of iterations. In each iteration, selecting two multipliers from all Lagrange multipliers is based on a criteria discussed later. Suppose αa , αb are selected to be updated at the current iteration. Then the gradient vector and the Hessian matrix of the objective function LDs with respect to αa and αb are   t f (xa ) − 1 G(a,a) G(a,b) (∇LDs )(a,b) = a , (∇2 LDs )(a,b) = (10) G(a,b) G(b,b) tb f (xb ) − 1 where G(a,b) = ta tb Ω(xa , xb ).

A Fast Algorithm for Sparse Extreme Learning Machine

According to Newton’s method [18], we have  ∗  αa αa = − (∇2 LDs )−1 (a,b) (∇LDs )(a,b) αb∗ αb

59

(11)

As the objective function of the dual optimal problem (3) is a convex quadratic polynomial [18], the objective function of (3) with respect to the variables αa and αb (other Lagrange multipliers αi ’s are temporarily fixed) is also a convex quadratic polynomial. Consequently, the optimal point is achieved at (αa∗ , αb∗ ) according to Newton iterative method theory [18]. However, the Hessian matrix (∇2 LDs )(a,b) may not always be nonsingular. Even if (∇2 LDs )(a,b) is nonsingular, a near singular one may lead to numerically unstable solution. Therefore, we introduce a switch parameter δ and modify the above iterative formula as follows:  ∗  αa αa (12) = − Θ(a,b) (∇LDs )(a,b) αb∗ αb where

 Θ(a,b) =

1

G(a,a) G(b,b) −δG2(a,b)

G(b,b) −δG(a,b) −δG(a,b) δG(a,a)

and the switch parameter δ is

1, G(a,a) G(b,b) − G2(a,b) =  0 δ= 2 0, G(a,a) G(b,b) − G(a,b) = 0

(13)

(14)

From above iterative formula, it can be seen that when (∇2 LDs )(a,b) is singular, the only one Lagrange multiplier αa is updated, and another Lagrange multiplier αb is invariant at the current iteration. As a result, it is the same as the algorithm in [16] for this case. Thus, the proposed iterative strategy is a generalization of the algorithm in [16]. 3.2

Clipping Optimal Point (α∗a , α∗b )

For the dual optimal problem (3), there are constraint conditions 0 ≤ αi ≤ C for all Lagrange multipliers αi ’s. Generally, the solution αa∗ , αb∗ from (12) may not be located in the interval [0, C]. Therefore, after obtaining the solutions αa∗ and αb∗ , we must clip the point (αa∗ , αb∗ ) into the square region [0, C] × [0, C]. A straightforward way to clip the optimal solution αa∗ and αb∗ is to utilize (8), respectively. Nevertheless, the convergence effect would become poor by using this scheme. For this reason, a new clipping scheme will be proposed in following discussion.

60

Z. Miao and Q. He

In order to clip the optimal point (αa∗ , αb∗ ) into the square region [0, C]×[0, C], we introduce some notes as following: SC = {(x, y)T ∈ R2 |0 ≤ x ≤ C, 0 ≤ y ≤ C} clip ∗ , α(a,b) = (αaclip , αbclip )T , α(a,b) ∗ − α(a,b) α(a,b) ∗ α(a,b) − α(a,b) 2

T

α(a,b) = (αa , αb )

n = (nx , ny )T =  Cx =

0, nx < 0 , C, nx ≥ 0



Cy =

0, ny < 0 C, ny ≥ 0

(15) =

(αa∗ , αb∗ )T

(16) (17) (18)

where SC means a square region in two-dimensional space R2 as show in Figs. 1 and 2, α(a,b) is a point in the coordinate system αa Oαb , n is a unit vector which indicates the direction of a half-line l from the original point α(a,b) to the ∗ as shown in Figs. 1 and 2. The half-line l indicates the search optimal point α(a,b) direction of the Newton’s method. Cx and Cy are determined by the vector n. ∗ The main goal in the following discussion is to clip the optimal point α(a,b)

clip into the square region Sc , i.e. to find a clipping point α(a,b) ∈ Sc , then α(a,b) will

clip clip be updated by α(a,b) in the next iteration. How to find a reasonable point α(a,b) in Sc ? It is natural to choose a point from Sc such that the distance between ∗ is minimum. Therefore,it is equivalent to this point and the optimal point α(a,b) solve the following optimization problem. ∗ Minimizeα∈Sc : α − α(a,b) 

(19)

Instead of straightforward using the clipping scheme (8) as in [16], the proposed clipping scheme in this paper is sorted into two cases as follows: (1) The intersection set of l and SC only has one point, i.e. α(a,b) ( Fig. 1 illuminates one situation for this case). (2) The intersection set of l and SC has more than one point (as shown in Fig. 2).

Fig. 1. A situation for Case 1: where αa = Cx

Fig. 2. A situation for Case 2: where Cx = Cy = C, and P1 is the clipped point

A Fast Algorithm for Sparse Extreme Learning Machine

61

For the case 1, the half-line l does not lie on the square region Sc except the initial point α(a,b) . Moreover, α(a,b) is on boundary of SC (i.e. α(a,b) ∈ ∂SC ) and the direction of the vector n is outward from SC . For the sake of algorithm implementation, the condition of the case 1 can be converted as logical expression as given in the following theorem. Theorem 1. The condition that the intersection set of l and SC only has one point is equivalent to the logical expression: “αa = Cx OR αb = Cy ”. Therefore, for the first case (i.e.the intersection set of l and SC only has one point), the logical expression: “αa = Cx OR αb = Cy ” can be employed as the discriminate criterion of this case. For this special situation, we clip the optimal clip ∗ into α(a,b) by straightforward using the clipping scheme (8) for αa∗ point α(a,b) ∗ and αb , respectively. In a similar way for the second case (i.e. the intersection set of l and SC has more than one point), we have the following theorem. Theorem 2. The condition that the intersection set of l and SC has more than one point is equivalent to the logical expression: “αa = Cx AND αb = Cy ”. ∗ Consequently, in order to judge whether the current state (α(a,b) ,α(a,b) ) correspond to this case, the logical expression: “αa = Cx AND αb = Cy ” can be employed. As the part of l which starts from the initial point α(a,b) to the optimal point ∗ partially or entirely lies on SC in this case (as shown in Fig. 2), and the α(a,b) search direction of the Newton iterative method is a descent one for the objective function, we can choose a point P on the half-line l such that P ∈ SC and the ∗ is minimum. Based on this principle, distance between P and the optimal α(a,b) there are four points to choose from, i.e.

P0 : (αa , αb ), P1 : (αa(1) , αb ), P2 : (αa(2) , αb ), P3 : (αa∗ , αb∗ ) (1)

(2)

where the coordinates of P1 and P2 can be calculated as follows:



n (C −α ) n (C −α ) (αa + x nyy b , Cy ), if ny = 0 (Cx , αb + y nxx a ), if nx = 0 P1 = , P2 = P3 , if nx = 0 if ny = 0 P3 , Only one of the three points P1 , P2 and P3 can be selected as the clipping clip . It is easy to know that the point which is the closest to P0 must point α(a,b) ∗ in be in SC . It is evident that this point is the closest to the optimal α(a,b)

clip Sc . Therefore, the clipping point α(a,b) can be appointed through the following formula. clip α(a,b) = PkT , k = arg min{P0 − Pj 2 , j = 1, 2, 3} j

(20)

62

3.3

Z. Miao and Q. He

Selection Criterion

According to the dual problem theory [12], KKT conditions (5) can be used to determine whether a solution is the optimal one. Therefore, in the training process, if one data violates the KKT conditions, then the Lagrange multiplier corresponding to this data should be updated in this iteration. If all data satisfy KKT conditions (5), then the optimal solution has been generated and the training algorithm would be terminated. In [16], whether or not the KKT conditions are satisfied can be determined by using a vector J defined as follows [16]: J = (J1 , J2 , · · · , JN ), Ji =

∂LDs di ∂αi

(21)

where Ji is selection parameter, di is defined as (9) (or Definition 5.1 in [16]). According to Theorem 5.1 in [16], if the data corresponding to αv violates KKT conditions, then Jv < 0, and vice versa. Until Ji > 0 for all i, the optimal solution is obtained and the training process would be terminated. Therefore, in the training process, the minimum value of Ji is negative [16]. Consequently, we can select the Lagrange multiplier which is corresponding to the minimal selection parameter as the first Lagrange multiplier αa , i.e. a = arg

min

i=1,2,··· ,N

Ji

(22)

Generally, once the first Lagrange multiplier αa is chosen, the second Lagrange multiplier can be chosen from the rest of Lagrange multipliers corresponding to the selection parameter Ji which satisfies Ji < 0. In order to speed convergence, the heuristics method proposed by Platt in SMO [17] can be employed to choose the second Lagrange multiplier. It is desirable to choose the second Lagrange multiplier such that it maximizes the size of the step taken during joint optimization. In a similar way as that using in SMO, the step size can be approximated by |(f (xa ) − ta ) − (f (xb ) − tb )|. According to Osuna’s theorem [19], the proposed training algorithm is convergent to the global optimal solution in a finite number of iterations. Therefore, we have the following result. Theorem 3. The proposed training algorithm will converge to the global optimal solution in a finite number of iterations. Proof. The process of proof is similar to that of Theorem 5.2 in [16]. 3.4

Training Algorithm

Since the proposed training algorithm is based on iterative method, the KKT conditions may not be satisfied accurately. Thus, KKT conditions can be checked within a tolerance ε, i.e. if mini=1,··· ,N Ji > −ε, then the training algorithm is terminated, and an approximative optimal solution is obtained. Consequently,the

A Fast Algorithm for Sparse Extreme Learning Machine

63

proposed training algorithm (named as FTA-sELM for short) is summarized in Ds Algorithm 1. where g = (g1 , g2 , · · · , gN ), gi = ∂L ∂αi . In FTA-sELM, there is no dependency between Lagrange multipliers, and the search space of two Lagrange multipliers is a square region [0, C] × [0, C] for each sub-QP problem. Hence, the FTA-sELM has a larger search space than SMO in each iteration, as a result, our method can get the ideal solution faster. Algorithm 1. Fast Training Algorithm of Sparse ELM for Classification (FTA-sELM) Input:Input: a training samples set {(xi , ti )|xi ∈ Rp , ti ∈ {1, −1}, i = 1, · · · , N }, hidden node number M , an appropriate ELM kernel matrix Ω and parameter C Output: a trained instance of Sparse ELM 1. Initialization: α = 0,g = −1, J = g,d = 1 (α, g, J, d ∈ RN ) 2. While mini=1,··· ,N Ji ≤ −ε: 1) Choose two Lagrange multipliers αa ,αb , according to the following subscript: a = arg mini=1,··· ,N Ji b = arg maxi∈{j,|Jj