Blind Equalization in Neural Networks: Theory, Algorithms and Applications 9783110450293, 9783110449624

Table of contents :
Preface
Contents
1. Introduction
2. The Fundamental Theory of Neural Network Blind Equalization Algorithm
3. Research of Blind Equalization Algorithms Based on FFNN
4. Research of Blind Equalization Algorithms Based on the FBNN
5. Research of Blind Equalization Algorithms Based on FNN
6. Blind Equalization Algorithm Based on Evolutionary Neural Network
7. Blind equalization Algorithm Based on Wavelet Neural Network
8. Application of Neural Network Blind Equalization Algorithm in Medical Image Processing
Appendix A: Derivation of the Hidden Layer Weight Iterative Formula in the Blind Equalization Algorithm Based on the Complex Three-Layer FFNN
Appendix B: Iterative Formulas Derivation of Complex Blind Equalization Algorithm Based on BRNN
Appendix C: Types of Fuzzy Membership Function
Appendix D: Iterative Formula Derivation of Blind Equalization Algorithm Based on DRFNN
References
Index

Citation preview

Liyi Zhang, Yunshan Sun, Xiaoqin Zhang, Yu Bai Blind Equalization in Neural Networks

Also of Interest Text Mining of Web-Based Medical Content A. Neustein (Ed.), 2014 ISBN 978-1-61451-541-8, e-ISBN 978-1-61451-390-2, e-ISBN (EPUB) 978-1-61451-976-8, Set-ISBN 978-1-61451-391-9

Speech and Automata in Health Care A. Neustein (Ed.), 2014 ISBN 978-1-61451-709-2, e-ISBN 978-1-61451-515-9, e-ISBN (EPUB) 978-1-61451-960-7, Set-ISBN 978-1-61451-516-6

Oscillatory Neural Networks M. Kuzmina et al, 2013 ISBN 978-3-11-026835-5, e-ISBN 978-3-11-026920-8, Set-ISBN 978-3-11-026921-5

Computer Interpretation of Metaphoric Phrases S. Weber Russell, 2015 ISBN 978-1-5015-1065-6, e-ISBN 978-1-5015-0217-0, e-ISBN (EPUB) 978-1-5015-0219-4, Set-ISBN 978-1-5015-0218-7

Liyi Zhang, Yunshan Sun, Xiaoqin Zhang, Yu Bai

Blind Equalization in Neural Networks Theory, Algorithms and Applications

Author Prof. Liyi Zhang Tianjin University of Commerce School of Information Engineering 409 Guangrong Road Beichen District 300134 Tianjin China

ISBN 978-3-11-044962-4 e-ISBN (PDF) 978-3-11-045029-3 e-ISBN (EPUB) 978-3-11-044967-9 Set-ISBN 978-3-11-045030-9 Library of Congress Cataloging-in-Publication Data A CIP catalog record for this book has been applied for at the Library of Congress. Bibliographic information published by the Deutsche Nationalbibliothek The Deutsche Nationalbibliothek lists this publication in the Deutsche Nationalbibliografie; detailed bibliographic data are available on the Internet at http://dnb.dnb.de. © 2018 Walter de Gruyter GmbH, Berlin/Boston Typesetting: Integra Software Services Pvt. Ltd. Printing and binding: CPI books GmbH, Leck Cover image: adventtr/gettyimages @ Printed on acid-free paper Printed in Germany www.degruyter.com

Preface Blind equalization (BE) technology is a new adaptive technology. BE only uses the prior information of received signals to equalize the channel characteristics, so training sequence is not needed. The output sequence is close to the transmitted sequence. Inter-symbol interference is overcome effectively and the quality of communication is improved by BE. Neural network (NN) is a cross-edge discipline of neural science, information science, and computer science. NN has the following abilities such as massively parallel, distributed storage and processing, self-organizing, adaptive, self-learning, and highly fault tolerant. The combination of NN and BE can improve convergence performance and equalization effect. The combination of NN and BE is a hot research topic in communication, signal, and information processing. It has important theoretical significance and practical value. This book was written by the author and his doctors and masters, namely, Yunshan Sun, Xiaoqin Zhang, Rui Lu, Xiaowei Niu, Yu Bai, Haiqing Cheng, Fengmei Jia, Yanling Jia, Yuan Li, Yong Liu, and Yanqi Kang. This book was also supported by the following research funds: Shanxi Province Natural Science Fund project “Mobile communication blind equalizer” (20011035), China Postdoctoral Project Science Foundation “Fuzzy neural network used in blind equalization technology” (20060390170), Shanxi Provincial Natural Science Foundation Project “The blind equalization technique based on neural network” (20051038), Tianjin High School Science and Technology Fund Project “Research on evolution neural network blind equalization algorithm” (20060610), and “Medical image blind restoration algorithm based on Zigzag transform” (20110709) Tianjin Research Program of Application Foundation and Advanced Technology “Research on Integration issues of Medical MRI Image Three-Dimensional Implementation” (13JCYBJC15600), and “Low-dose Medical CT Image Blind Restoration Reconstruction Algorithm based on Bayesian Compressed Sensing” (16JCYBJC28800). This book was translated by Yu Bai (Chapters 1–2), Xiaoqin Zhang (Chapters 3–5), and Yunshan Sun (Chapters 6–8). The NN and BE algorithms are combined, and the new neural network is systematically studied. Some research results have been published in important academic journals and also in international and domestic conferences. This book is a summary of the results of these studies. The latest research trends and frontiers in neural network blind equalization algorithm in domestic and international are reflected basically in the book. This book is divided into eight chapters. The first chapter is introduction. The significance and application fields of blind equalization are given. The classification and research status of NN blind equalization algorithms are summarized. The research background and main work in this book are pointed out.

DOI 10.1515/9783110450293-202

VI

Preface

The second chapter describes the fundamental theory of NN. The concept, structure, algorithms, and equalization criterion of blind equalization are introduced. The fundamental principles and learning methods of NN blind equalization algorithm are elaborated. The evaluation of blind equalization algorithm is analyzed. The third chapter is about the research of blind equalization algorithms based on FFNN. BE algorithm based on feed-forward neural networks (four-layer, three-layer, and five-layer) are studied. BE algorithms based on momentum term, time-varying momentum term, and time-varying step are studied, respectively. The fourth chapter is about the research of blind equalization algorithms based on FBNN. BE algorithms based on bilinear recurrent NN, diagonal recurrent NN, and quasi-diagonal recurrent NN are studied, respectively. The blind equalization algorithms based on mean square error nonlinear function with time-varying step diagonal recurrent NN and with time-varying step quasi-diagonal recurrent NN are studied, respectively. The fifth chapter is the research of blind equalization algorithms based on FNN. The blind equalization algorithm based on fuzzy NN filter, fuzzy NN controller, and fuzzy NN classifier is studied, respectively. The sixth chapter is blind equalization algorithm based on evolutionary neural network. The blind equalization algorithms based on optimization NN weights and structure optimized by genetic algorithm are studied, respectively. The seventh chapter describes blind equalization algorithm based on wavelet NN. The blind equalization algorithms based on feed-forward NN and feedback wavelet NN are studied, respectively. The eighth chapter provides the application of blind equalization algorithm neural network in medical image processing. The application of blind equalization in CT image restoration is mainly studied. We would like to express our sincere thanks to Professor Jianfu Teng, the doctoral tutor in Tianjin University; Professor Dingguo Sha, the doctoral tutor in Beijing Institute of Technology; and Professor Huakui Wang, the doctoral tutor in Taiyuan University of Technology for their help and support. We wish to thank Yanqin Li who is responsible for proofreading and revision. We are also grateful to the scholars at home and abroad whose published literature are referred to in this book. Due to the limited level of the author, there are some oversights and inadequacies inevitably in the book, the readers are welcome to give suggestions. The author August 2016

Contents 1 1.1 1.2 1.2.1 1.2.2 1.2.3 1.2.4 1.2.5 1.2.6 1.3 1.3.1 1.3.2 1.3.3 1.3.4 1.3.5 1.4 1.4.1 1.4.2

Introduction 1 The research significance of the BE technology 1 The application of BE technology 3 The application in digital television 3 The application in CATV system 3 The application in smart antenna 4 The application in software radio 4 The application in blind image restoration 5 The application in radiofrequency identification 5 The research progress of neural network BE algorithm The FFNN BE algorithm 6 The BE algorithm based on FBNN 10 The FNN BE algorithm 11 The ENN BE algorithm 13 BE algorithm based on WNN 14 The research background and structure 14 The research background 14 The structure of the book 15

2

The Fundamental Theory of Neural Network Blind Equalization Algorithm 17 The fundamental principle of blind equalization 17 The concept of blind equalization 17 The structure of blind equalizer 18 The basic algorithm of blind equalization 19 The equalization criteria of blind equalization 22 The fundamental theory of neural network 27 The concept of artificial neural network 27 The development of ANN 28 The characteristics of ANN 29 Structure and classification of ANN 30 The fundamental principle of neural network blind equalization algorithm 32 The principle of blind equalization algorithm based on neural network filter 32 The principle of blind equalization algorithm based on neural network controller 32 The principle of blind equalization algorithm based on neural network controller classifier 33

2.1 2.1.1 2.1.2 2.1.3 2.1.4 2.2 2.2.1 2.2.2 2.2.3 2.2.4 2.3 2.3.1 2.3.2 2.3.3

6

VIII

2.4 2.4.1 2.4.2 2.5 2.5.1 2.5.2 2.5.3 2.5.4 2.5.5 2.5.6 2.5.7 2.6 3 3.1 3.1.1 3.1.2 3.1.3 3.2 3.2.1 3.2.2 3.2.3 3.3 3.3.1 3.3.2 3.3.3 3.4 3.4.1 3.4.2 3.4.3 3.5 3.5.1 3.5.2 3.5.3 3.6 3.6.1 3.6.2 3.6.3 3.7

Contents

The learning method of neural network blind equalization 34 algorithm The BP algorithm 34 The improved BP algorithm 36 The evaluation index of the neural network blind equalization algorithm 42 Convergence speed 43 The computational complexity 43 The bit error characteristics 43 The ability of tracking the time-varying channel 43 The ability of anti interference 43 The convexity of the cost function 44 The steady-state residual error 45 Summary 50 51 Research of Blind Equalization Algorithms Based on FFNN Basic principles of FFNN 51 Concept of FFNN 51 Structure of FFNN 51 Characteristics of FFNN 52 Blind equalization algorithm based on the three-layer FFNN 52 Model of the three-layer FFNN 52 Real blind equalization algorithm based on the three-layer FFNN 53 Complex blind equalization algorithm based on the three-layer FFNN 56 Blind equalization algorithm based on the multilayer FFNN 63 Concept of the multilayer FFNN 63 Blind equalization algorithm based on the four-layer FFNN 64 Blind equalization algorithm based on the five-layer FFNN 70 Blind equalization algorithm based on the momentum term FFNN 78 Basic principles of algorithm 78 Derivation of algorithm 80 Computer simulation results 82 Blind equalization algorithm based on the time-varying momentum term FFNN 84 Basic principles of algorithm 84 Derivation of algorithm 86 Computer simulation results 87 Blind equalization algorithm based on variable step-size FFNN 88 Basic principles of algorithm 88 Derivation of algorithm 90 Computer simulation results 91 Summary 92

IX

Contents

4.6.1 4.6.2 4.6.3 4.7

94 Research of Blind Equalization Algorithms Based on the FBNN Basic principles of FBNN 94 Concept of FBNN 94 Structure of FBNN 94 Characteristics of FBNN 94 Blind equalization algorithm based on the bilinear recurrent NN Basic principles of algorithm 95 Real blind equalization algorithm based on BLRNN 96 Complex blind equalization algorithm based on BLRNN 99 Blind equalization algorithm based on the diagonal recurrent NN Model of diagonal recurrent NN 103 Derivation of algorithm 105 Computer simulation results 106 Blind equalization algorithm based on the quasi-DRNN 107 Model of quasi-DRNN 107 Derivation of algorithm 110 Computer simulation results 112 Blind equalization algorithm based on the variable step-size DRNN 115 Basic principles of algorithm 115 Derivation of algorithm 115 Computer simulation results 116 Blind equalization algorithm based on the variable step-size QDRNN 119 Basic principles of algorithm 119 Derivation of algorithm 119 Computer simulation results 120 Summary 123

5 5.1 5.1.1 5.1.2 5.1.3 5.1.4 5.1.5 5.2 5.2.1 5.2.2 5.2.3 5.3 5.3.1

124 Research of Blind Equalization Algorithms Based on FNN Basic principles of FNN 124 Concept of FNN 124 Structure of FNN 125 The choice of fuzzy membership function 126 Learning algorithm of FNN 128 Characteristics of FNN 128 Blind equalization algorithm based on the FNN filter 129 Basic principles of algorithm 129 Derivation of algorithm 129 Computer simulation results 133 Blind equalization algorithm based on the FNN controller 134 Basic principles of the algorithm 134

4 4.1 4.1.1 4.1.2 4.1.3 4.2 4.2.1 4.2.2 4.2.3 4.3 4.3.1 4.3.2 4.3.3 4.4 4.4.1 4.4.2 4.4.3 4.5 4.5.1 4.5.2 4.5.3 4.6

95

103

X

5.3.2 5.3.3 5.4 5.4.1 5.4.2 5.4.3 5.5 6 6.1 6.1.1 6.1.2 6.1.3 6.1.4 6.1.5 6.1.6 6.2 6.2.1 6.2.2 6.2.3 6.3 6.3.1 6.3.2 6.3.3 6.4 7 7.1 7.1.1 7.1.2 7.1.3 7.2 7.2.1 7.2.2 7.2.3 7.3

Contents

Derivation of algorithm 135 Computer simulation results 139 Blind equalization algorithm based on the FNN classifier Basic principles of algorithm 141 Derivation of algorithm 141 Computer simulation results 146 Summary 147

141

149 Blind Equalization Algorithm Based on Evolutionary Neural Network Basic principles of evolutionary neural networks 149 The concept of GA 149 Development of GA 150 GA parameters 151 The basic process of GA 157 Characteristics of GA 158 The integration mechanism of GA and neural network 158 Blind equalization algorithm based on neural network weights optimized by GA 159 The basic algorithm principle 159 Feed-forward neural network weights optimized by GA blind equalization algorithm (GA-FFNNW) in binary encoding 160 Real encoding GA-FFNNW blind equalization algorithm 167 GA optimization neural network structure blind equalization algorithm (GA-FFNNS) 172 The basic algorithm principle 172 Algorithm derivation 173 Computer simulation 175 Summary 179 180 Blind equalization Algorithm Based on Wavelet Neural Network Basic principle of wavelet neural network 180 The concept of wavelet neural network 180 The structure of wavelet neural network 181 The characteristics of wavelet neural network 182 Blind equalization algorithm based on feed-forward wavelet neural network 182 Algorithm principle 182 Blind equalization algorithm based on feed-forward wavelet neural network in real number system 183 Blind equalization algorithm based on FFWNN in complex number system 187 Blind equalization algorithm based on recurrent wavelet neural network 194

Contents

194 Principle of algorithm Blind equalization algorithm based on BLRWNN in real number system 194 Blind equalization algorithm based on BLRWNN in complex system 198 Summary 204

7.3.1 7.3.2 7.3.3 7.4 8 8.1 8.1.1 8.1.2 8.1.3 8.2 8.2.1 8.2.2 8.2.3 8.2.4 8.3 8.3.1 8.3.2 8.3.3 8.4

Application of Neural Network Blind Equalization Algorithm in Medical Image Processing 205 Concept of image blind equalization 205 Imaging mechanism and degradation process of medical CT image 205 The basic principle of medical CT images blind equalization 207 Quantitative measurements 208 Medical CT image neural network blind equalization algorithm based on Zigzag 209 Basic principle of algorithm 209 Iterative formula derivation 211 Analysis of algorithm convergence performance 213 Experimental simulation 215 Medical CT image neural network blind equalization algorithm based on double zigzag encoding 219 The basic principle of algorithm 219 Formula derivation of iterative algorithm 220 Experimental simulation 224 Summary 226

Appendix A: Derivation of the Hidden Layer Weight Iterative Formula in the Blind Equalization Algorithm Based on the Complex Three-Layer FFNN 229 Appendix B: Iterative Formulas Derivation of Complex Blind Equalization Algorithm Based on BRNN 231 Appendix C: Types of Fuzzy Membership Function

235

Appendix D: Iterative Formula Derivation of Blind Equalization Algorithm Based on DRFNN 239 References Index

XI

251

243

1 Introduction Abstract: In this chapter, the research significance and the application field of blind equalization (BE) are analyzed. The classification and research status of the neural network BE algorithm are summarized. The research background and the main work of this book are pointed out.

1.1 The research significance of the BE technology The concept of BE (called as the self-recovering equalization at that time) was proposed by the Japanese scholar professor Y. Sato [1] first in 1975. It has gradually become a key technology of digital communication, and also a frontier and hot research topic of communication and signals processing, because it can overcome the inter-symbol interference (ISI) effectively, reduce the bit error rate (BER), and improve reception and the quality of communication. BE is set up on the basis of overcoming the defects of the adaptive equalization. BE only uses the prior information of received sequence itself to equalize channel characteristics, instead of sending the training sequence. As a result, the output sequence of the equalizer approximates the transmitted sequence as far as possible. Before data transmission, a training sequence known by the receiver needs to be transmitted in the adaptive equalization. Then, the changes or errors of the sequence passing through the channel are measured by the receiver. According to the error information, parameters of equalizer are adjusted. Eventually, the channel characteristic is compensated by the equalizer. As a result, the almost undistorted signals are obtained from the sequence of equalizer output, and the reliable data transmission is guaranteed. The process is called as automatic equalization. At this time, the equalizer is in training mode. When the training process is over, the adjustment of equalizer parameter gets convergence, the reliability of decision signals is higher, and the error rate is less. After the training process, the data begin to transmit. At that time, the transmitted signals are unknown. In order to track possible changes of channel characteristics dynamically, the receiver takes output decision signals of the equalizer as the reference signals. These reference signals are used to measure the errors produced by channel changes and to adjust the equalizer’s output signals continuously. At this time, the above-mentioned process is called as decision-directed equalization. According to the theory of adaptive filter, the condition for the equalizer to work properly under decision-directed mode is that the eye pattern opens to a certain extent in advance. The above condition can ensure equalizer-reliable convergence. If the condition is not satisfied, a training sequence known by the receiver will be sent by the sending end to train the equalizer again, and make it get convergence. Thus, the training process DOI 10.1515/9783110450293-001

2

1 Introduction

is also called as the learning process of the equalizer. For the general communication system, the training process is indispensable. The development and application of the equalization technology improve the performance of communication system greatly. Just as R.D. Gitlin et al. [2] said, the revolution of the data communications can be traced back to the discovery of automatic and adaptive equalization technology in the late 1960s. However, with the development of digital communication technology to wide band, high speed, and large capacity, the shortcomings and defects of the adaptive equalization technology are increasingly exposed, mainly in the following [3]: (1) The training sequence does not transmit useful information, so the information transmission rate of the communication system is reduced. For example, in the Global System for Mobile Communication (GSM), there are 26 of every 148 symbols occupied for training, which led to a loss of 18% capacity [4]. In the high-frequency communication system, the training sequence even accounts for the 50% total transmission capability [5]. (2) For a fast time-varying channel, the training sequence must be transmitted frequently in order to continuously update the channel estimate and track the channel change. (3) In a broadcast or point-to-multipoint communication network, if a branch channel is temporarily disabled and need to resume work, it is necessary to equalize the branch receiver again. In this case, the branch receiver may not get the training signal. Thus, it will require the central station to interrupt the communication with other branch channels and send a training sequence to it. Or the transmitted signals of the central station have always contained the training signal. The digital HDTV is a typical example of broadcast communication. (4) Due to the interference or other factors in a channel, a receiver may fail to track the channel, and thus the communication is interrupted. In order to rebuild the communication, the sending end is required to transmit the training sequence again. Thus, the feedback channel must be added to the system to transmit the “request training signal,” which makes the system complex and difficult to realize. (5) In some special applications, a receiver cannot get the training sequence, such as information interception, reconnaissance system, image reconstruction, and so on. Therefore, BE can overcome the shortcomings of the adaptive equalization effectively. Furthermore, there are still much interference in the process of communication, such as recurrent fading, serious nonlinear and time-varying characteristics, and the multipath propagation of channel. And the communication may be interrupted as the receiver fails to track the channel characteristics. For the above interferences, BE can adapt equalization, adjust parameters, track channel characteristics, and complete the best estimate of the signal.

1.2 The application of BE technology

3

1.2 The application of BE technology Now, the BE technology has been widely applied in communication, radar, sonar, control engineering, seismic exploration, biomedical engineering, and so on. Especially in the field of communication, it can be said that the BE technology has penetrated into various industries [6, 7].

1.2.1 The application in digital television The high-definition television (HDTV) has become a popularization trend for television broadcast. The United States, Japan, etc. have developed many transmission schemes of digital broadcast television. These transmission schemes cover the quadrature amplitude modulation (QAM) transmission scheme, vestigial sideband modulation (VSB) transmission scheme and orthogonal frequency division multiplexing (OFDM) transmission scheme, and so on. At present, three international standards have been formed: DVB-T (Digital Video Broadcasting—Terrestrial) standard, the ATSC (Advanced Television Systems Committee) standard, and the ISDB-T (Integrated Services Digital Broadcasting) standard. The DVB-T standard, proposed by the European DVB organization, takes coded OFDM as core technology. The ATSC standard, proposed by the American big alliance, takes eight VSBs as the core technology. The ISDB-T standard, proposed by Japan, takes the band segmented transmission OFDM as the core technology. The major difference between them and the existing analog broadcast television is that all transmitted information, including images, audio, additional information, forward error correction, synchronous information, and so on, is digital information. In the transmission process of this digital information, the channel interference, such as recurrent fading, multipath propagation, and so on, can lead to ISI. In order to eliminate these interferences and reduce BER, the BE technology is adopted generally. For example, the DigiCipher and CC-DigiCipher systems [8] in the United States adopt the scheme of BE with transversal filter. In the above scheme, the complex channel equalizer is composed of four groups of 256 taps transversal filter, and adopted 32QAM/16QAM. Its transmission rate is 19.2 Mbit/s. Yang yong [9] designs a dual-mode blind equalizer based on the ATSC digital television receiver.

1.2.2 The application in CATV system In 1995, the J.83 recommendation is called Digital Multi-programme System for Television, Sound and Data Services for Cable Distribution given by the International Telecommunication Union Telecommunication Standardization Sector (ITU-T). It was the first relevant standard for digital cable TV system set by the ITU. The J.83

4

1 Introduction

recommendation reflects the practical application level of digital video transmission system. Because of the authority and the technical feasibility of the ITU standards, technical schemes for digital video transmission system formulated by every country must be compatible with J.83 recommendation. Four technical schemes of digital video transmission system are put forward in the standard. The former three schemes (A, B, C) use QAM mode. In QAM mode, the local carrier recovery and BE are employed, which make it to be good at real time. And the 64 QAM special chips have been commercialized [10].

1.2.3 The application in smart antenna The smart antenna is an antenna array composed of N antenna units. Each unit has M sets of weighted devices, which can form M beams with different direction. In practical application, the pattern of antenna array can be changed by adjusting the weight matrix, which makes the beam to follow users. In this way, the interference is suppressed and the signal-to-noise ratio is improved. In the implementation process of smart antenna technology, a variety of different algorithms are employed. Of them, the adaptive algorithm (such as least mean square [LMS] algorithm and recursive least squares [RLS] algorithm) and BE algorithm (the constant modulus algorithm is the most common)are important [11]. For example, the ATR photoelectric communication institute in Japan develops a multi-beam smart antenna based on the beam space handling. Its array elements form a 16-element planar square matrix with half wavelength spacing. Its working radiofrequency (RF) is 1.545 GHz. After receiving signals, first, array elements finish the analog-to-digital conversion; second carry out the fast Fourier transform; third, form orthogonal beam; fourth, use the constant modulus algorithm (when the co-channel interference is larger) or maximum ratio combining diversity algorithm (when the noise is larger); finally, use field-programmable gate array (FPGA) to realize real-time antenna configuration and complete intelligent processing. The digital signal processing of smart antenna is completed by 10 FPGA chips. The size of whole circuit board is 23.3 cm × 34.0 cm [12].

1.2.4 The application in software radio The software radio is to construct a generic and reprogramming hardware platform. In addition, the working frequency, modulation and demodulation method, business types, data rate and data format, control protocol, and so on of the platform are reconfigurable and controllable. Thus, the different types and functions of radio stations can be realized by selecting different software modules. The core idea of software radio is to use the broadband D/A and A/D converter at the place as close to the antenna as possible, and to use software to define the radio function [13].

1.2 The application of BE technology

5

The digital signal processing module is the core part of the software radio. It is mainly used for real-time digital signal processing after conversion, and to realize a large number of radio functions by software, such as decoding, modulation, demodulation, filtering and synchronization, BE, detection, data encryption, transmission error correction, frequency-hopping spread spectrum, de-spreading and de-hopping, communication environmental assessment and channel selection, and so on. The digital signal processing module can be expanded flexibly, so as to satisfy the requirements of different wireless communication system for digital signal processing in speed and computational complexity. Among them, BE is used to compensate the effect of nonideal properties of channel and to eliminate ISI [14].

1.2.5 The application in blind image restoration The blind image restoration (BIR) algorithm is a restoration technology of original image. Under the premise that the image degradation process is unknown, the BIR algorithm just only uses a degraded image to eliminate the influence of point spread function. Now, BIR has been widely applied in the astronomy imaging, medical diagnosis, military, public security, and other fields [15, 16]. Now, there have been a lot of BIR algorithms. Liu tao et al. [17], first, extended the one-dimensional BE algorithm and applied it into two-dimensional image processing. In their algorithm, the transmission of image signals is equivalent to a linear time-invariant system with single input single output. And then, the BE algorithm is used for image restoration. Sun Yunshan et al. converted two-dimensional medical images into one-dimensional signal by row-column transformation, orthogonal transformation, zigzag encoding method, and so on. Based on the above-mentioned conversions, they propose many blind image equalization algorithms. The validity of these algorithms is verified by the computer simulation [18–25].

1.2.6 The application in radiofrequency identification Radiofrequency identification (RFID) technology, rose in the 1990s, is a new kind of automatic noncontact identification technology. RFID realizes the two-way noncontact target recognition and data exchange between the reader and radio tag by RF. With advantages of noncontact, fast reading/identifying, no wear, free from the influence of environment, long service life, convenient for user, and so on, the RFID technology has been widely applied in many fields, such as industrial automation, business automation, logistics management, transportation control management, and so on [26]. But in the process of receiving and transmitting RF signal, the influence of the band-limited transmission and the multipath effect led to ISI. The ISI can affect the accuracy of detection and identification seriously. For the problem of channel ISI, Song

6

1 Introduction

Weiwei [27] proposed the constant modulus algorithm (CMA) BE. Zhang Liyi et al. [28] proposed a neural network BE algorithm based on a time-varying step. In addition, Bai Yu [29] applied the Bussgang BE algorithm into the high-performance ultra high frequency (UHF) RFID system. The computer simulations under Rayleigh fading channel and Rician fading channel show that the algorithm can effectively reduce the BER of system.

1.3 The research progress of neural network BE algorithm The neural network has many characteristics, such as massive parallel, distributed storage and processing, self-organizing, adaptive, self-learning, ability of high fault tolerance. Based on these characteristics, the neural network BE algorithm can approximate channel characteristics and recover the transmitted signals by utilizing characteristics of neural network. At present, there are many classification methods for neural network BE algorithm. According to the fundamental principle of neural network BE algorithm, there are mainly cost function method and the energy function method [30]. According to the combination mechanism of the neural network and BE, there are mainly the BE algorithm based on neural network filter, the BE algorithm based on neural network classifier, and BE algorithm based on neural network controller [31]. According to the different structure of neural network, there are mainly feed-forward neural network (FFNN) BE algorithm, feedback neural network (FBNN) BE algorithm, the wavelet neural network (WNN) BE algorithm, the fuzzy neural network (FNN) BE algorithm, and the evolutionary neural network (ENN) BE algorithm.

1.3.1 The FFNN BE algorithm The FFNN generally consists of the input layer, output layer, and hidden layer. In FFNN, each neuron only receives the output of the previous layer as its input and sends its output to the next layer. There is no feedback in the whole FFNN. Each neuron can have any number of inputs, but only one output. FFNN has the advantages of simple structure, low computation, and so on. In 1991, N. Benvenuto et al. [32] proposed a complex multilayer FFNN blind equalizer for mobile satellite communication system. In their equalizer, the traditional transversal filter is instead by multilayer FFNN, but the equalizer only applies to the Doppler signal. In 1998, Cheolwoo You et al. [33] proposed an improvement of the above algorithm for QAM signal. Its cost function is 󵄨󵄨2 󵄨 ̃ 󵄨󵄨2 ̃ ["sgn (󵄨󵄨󵄨󵄨dec (x(n)) ̃ J(n) = x(n) 󵄨󵄨 ) – 󵄨󵄨󵄨x(n) 󵄨󵄨 ]

(1.1)

1.3 The research progress of neural network BE algorithm

7

The transfer function is f (x) = f (x̃R (n)) + jf (x̃I (n))

(1.2)

f (x) = x + ! sin 0x

(1.3)

̃ is the output signal of the equalizer; " is the undetermined parameters; In eq. (1.1), x(n) ̃ x̃R (n) and x̃I (n) are the real part and imaginary part of x(n), respectively; !(! > 0) is the scale factor. In 1994, S. Mo et al. [34] proposed a BE algorithm based on FFNN and the higherorder cumulant. In their algorithm, the fourth-order spectrum of the output signals is used to identify channel, and then the nonlinearity of neural network is employed to construct an inverse channel of the channel. The algorithm can be applied to linear or nonlinear channel, and can overcome the influence of channel order uncertainty, and also has a certain fault tolerance to the additive noise. However, its convergence speed is slow and can only be applied to pulse code modulation (PAM) signal. The convexity of the cost function is not discussed in their paper, so their algorithm cannot guarantee convergence to global optimal. Liang Qilian et al. [35] proposed a BE algorithm based on multilayer neural network and higher-order cumulant. In their algorithm, Rosario algorithm is combined with Solis’ random algorithm. The former can find the local minimum point with relatively less iterations and speed up the convergence, and the latter can guarantee unidirectional convergence to the global minimum. The cost function is J(n) =

1 2 ̃ [y(n) – x(n)] 2

(1.4)

In eq. (1.4), y(n) is a received signal, that is, input signal of the blind equalizer. In 2003, Zhang Liyi et al. [36] proposed a BE algorithm. The three-layer FFNN structure and the cost function of the traditional CMA BE algorithm are adopted in their algorithm. The cost function is J(n) =

2 1 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨2 – R2 ] [󵄨󵄨x(n) 2

(1.5)

Transfer function is a deformation of the hyperbolic tangent function. Its expression is f (x) = x + !

ex – e–x ex + e–x

(1.6)

In eq. (1.46), the definition of R2 is consistent with that of the constant modulus BE algorithm. It is defined as R2 =

󵄨 󵄨4 E [󵄨󵄨󵄨x(n)󵄨󵄨󵄨 ] 󵄨 󵄨2 E [󵄨󵄨󵄨x(n)󵄨󵄨󵄨 ]

(1.7)

8

1 Introduction

By changing the value of the scale factor !, the transfer function can be applied to the equalization of different amplitude signals. For signals with larger amplitude interval, ! should take larger value. For signals with smaller amplitude interval, ! should take smaller values, and it is helpful to classify the output signal. The computer simulation shows that the convergence speed of the algorithm is improved. In the same year, on the basis of analyzing the influence of the step-size factor on the convergence performance of the BE algorithm, Zhang Liyi et al. [37] proposed the variable step-size neural network BE algorithm. In the algorithm, the kurtosis of error signals is adopted as control factor of step size as follows: 󵄨 󵄨 ,(n) = ! 󵄨󵄨󵄨K [e(n)]󵄨󵄨󵄨 󵄨2 󵄨 󵄨4 󵄨 󵄨2 󵄨 K[e(n)] = E [󵄨󵄨󵄨e(n)󵄨󵄨󵄨 ] – 2E2 [󵄨󵄨󵄨e(n)󵄨󵄨󵄨 ] – 󵄨󵄨󵄨󵄨E [e2 (n)]󵄨󵄨󵄨󵄨

(1.8) (1.9)

In eq. (1.8), ! is the scale factor that can control the step size ,(n), and K [e(n)] is the kurtosis of error signal e(n). In the initial period, the algorithm uses a larger step size to speed up the convergence rate. When the convergent conditions are nearly satisfied, the algorithm gradually reduces the step size to decrease the steady residual error. Thus, it effectively solves the contradiction between the convergence speed and convergence accuracy. In 2005, Xiao Ying et al. [38] proposed a variable step-size neural network BE algorithm for underwater acoustic communication. In their algorithm, a small step size is set first. And then, if J(n) increases after one iteration, the step size will be multiplied by a constant a (a is less than 1). Then the next iteration point is recalculated along the original direction. If J(n) is decreased after one iteration, the step size will be multiplied by a constant b (b greater than 1), that is, the change of iteration step is {a, a < 1, ,(n) = { b, b > 1, {

when BJ(n) > 0 when BJ(n) < 0

(1.10)

Its neural network structure is the three-layer FFNN. The simulation results show that when compared with the neural network blind equalizer with traditional BP algorithm, the variable step-size BP algorithm has faster convergence speed. In addition, the equalization performance of the variable step-size BP algorithm is improved significantly. In 2005, according to the characteristics of the FFNN, Cheng Haiqing et al. [39] constructed a new cost function by analyzing the constant modulus BE algorithm: J(n) =

2 2 1 󵄨󵄨 ! ̃ 󵄨󵄨󵄨󵄨2 – R2 ] + [󵄨󵄨󵄨󵄨x(n) ̂ 󵄨󵄨󵄨󵄨2 – 󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 ] [󵄨󵄨x(n) 2 2

(1.11)

1.3 The research progress of neural network BE algorithm

9

2 󵄨 ̃ 󵄨󵄨2 In eq. (1.11), 21 [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2 ] is a cost function of the traditional CMA, 2 ! 󵄨󵄨 ̂ 󵄨󵄨󵄨󵄨2 – 󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 ] is the mean square error function, x(n) ̂ [ x(n) is the recovered sig2 󵄨󵄨 nal from the decision device output, and ! is employed to adjust the proportion of the error function in the whole cost function. The transfer function is shown in eq. (1.3). In the process of weight update, the traditional cost function of the constant modulus BE algorithm and the error function of neural network are decreased by the above method, simultaneously. Thus, the convergence is speed up, and the steady residual error and the BER are reduced. Therefore, the algorithm gets a better effect of equalization. But the scale factor ! of the cost function can only be determined by experiment. In 2006, Jia Yanling [40] adopted the sigmoid function and the tongue-like curve as the control factor of step size, respectively; see eqs. (1.12) and (1.13). In this way, two variable step-size neural network BE algorithms are proposed: 2

(n)]]

be2 (n)

+1

,(n) = "[1 – e–!E[[e ,(n) = a[1 –

1

]

(1.12)

]

(1.13)

In eqs. (1.12) and (1.13), ! is an undetermined parameter and used to control the variation rate of step size. The constant " is the scale factor that can control the range of ,(n), a is the amplitude adjustment coefficient, and b is the waveform control coefficient of the tongue-like curve. In the same year, Zhang Xiaoqin et al. [41] studied the application of the threelayer FFNN BE algorithm in QAM and other complex systems. They adopted the cost function of the traditional CMA BE algorithm. To see eq. (1.15). The transfer function of their algorithm is f (x) = x + E [e2 (n)] sin 0x

(1.14)

̂ – x(n) ̃ is an error function. In eq. (1.14), e(n) = x(n) The computer simulations show that the algorithm can reduce the error rate and steady residual error. In 2006 and 2007, Zhang Liyi et al. [42, 43] and Cheng Haiqing et al. [44] analyzed the influence of momentum factor on the performance of BE algorithm. Then, they proposed the BE algorithm based on momentum term and the neural network BE algorithm based on variable momentum factor, respectively. The computer simulations show that these algorithms are available. In 2007, Kang Yanqi [45] studied the four-layer and five-layer FFNN BE algorithm. When compared with the three-layer FFNN BE algorithm, their algorithms improved the accuracy of convergence at the cost of lowering convergence speed and increasing computational complexity.

10

1 Introduction

1.3.2 The BE algorithm based on FBNN In FBNN, namely recurrent neural network, the information transfer between neurons of the neural network is no longer from one layer to another layer, but there is a link between any two neurons. In FBNN, each neuron is a processing unit and can receive input and take output to the outside. Because the feedback mechanism is introduced, FBNN is a nonlinear dynamic system. It has the characteristics of small size, fast convergence, and so on. In 1994, G. Kechriotis et al. [46] successfully applied the recursive neural network to the BE and proposed a real-time recursive algorithm. In the algorithm, the highorder statistical characteristics of signals are used to construct the cost function. And the network weights are adjusted by minimizing the cost function. The cost function is 4

4

i=1

i=1

J(n) = ∑ !i e2i (n) = ∑ !i {E[x̃ i (n)] – E[yi (n)]}

2

(1.15)

In eq. (1.15), !i > 0 is a scale constant, which defined the weight of the corresponding error. When compared with the traditional BP algorithm, their algorithm greatly improves the convergence rate of the blind equalizer. But their algorithm is easy to fall into local optimal and has too many parameters in the cost function. These disadvantages led to a higher calculation complexity of algorithm. In 1997, Liang Qilian et al. [47] proposed an improved algorithm, which greatly reduced the computation. The new cost function is 2

J(n) =

∑ !i eii (n) i=1

i

E[y2i (n)] } { = ∑ !i {E[x̃ i (n)] – E[yi (n)] } i=1 { } 2

(1.16)

In 1996, Liao Riping et al. [48] proposed the optimized FBNN BE algorithm. By using the second-order and fourth-order cumulative moments, they derived a linear optimal cost function of BE based on the high-order cumulative moment: 2

r2

q

–r1

r2

2

J(n) = !1 ∑ [ ∑ wj (n)C24 ( j, k)] + !2 ∑ [ ∑ wj (n)C4 ( j + k)] k=0 [j=r1 k=q+1 [j=r1 ] ] –1

r2

2

(1.17)

2 + !3 ∑ [ ∑ wj (n)C4 ( j + k)] + !4 [x̃ 2 (n) – R󸀠2 ] k=–r2 [j=r1 ]

In eq. (1.17), !i (i = 1, 2, 3, 4) is the scale factor, wj (n) is the coefficient of weight, r1 and r2 are the order of neural network, 0 ≤ n ≤ q, R󸀠2 = E [x2 (n)], C2 and C4 are the second-order and fourth-order cumulates of received signals, C24 is defined as

1.3 The research progress of neural network BE algorithm

11

q

C24 ( j, k) = ∑ [C4 (n + j) C2 (k – n) – C2 (n + j) C4 (k – n)], (–q ≤ k ≤ 2q)

(1.18)

n=0

In their algorithm, the two-layer FBNN is adopted to optimize the weights of linear equalizer. The error signal is calculated in the first layer of neural network. The summation and integral are completed in the second layer of the neural network. The outputs of the second layer of the neural network are the weights of the linear equalizer. The algorithm has the advantages of fast convergence, good adaptability for nonstationary signal, no sensitive to the initial weights, and so on, but its computation amount is large because of the high-order cumulates. In 2003, on the basis of bilinear recurrent neural network adaptive equalization algorithm, Lu Rui [49] combined the traditional CMA BE algorithm with the FBNN to get a neural network BE algorithm. The cost function and transfer function, eqs. (1.5) and (1.6), respectively, can accelerate the convergence, reduce the steady residual error, and enhance the capability of phase recovery simultaneously. In 2006 and 2007, Jia Fengmei et al. [50–52] applied the recurrent neural network into BE algorithm and proposed BE algorithms based on diagonal recurrent neural network and quasi-diagonal recurrent neural network. The cost function and the transfer function are shown in eqs. (1.5) and (1.6), respectively. Compared with the three-layer FFNN BE, their algorithm is better at convergence speed and error rate, but has basically similar steady residual error.

1.3.3 The FNN BE algorithm FNN is a new neural network model. It is the organic combination of fuzzy information processing technique and artificial neural network technology. In FNN, the neural network simulates the hardware of human brain, and the fuzzy technique simulates the software of human brain. FNN has the characteristics of learning, association, identification, adaptive, and fuzzy information processing. In 1993, Wang Lixin et al. [53] proposed an adaptive fuzzy equalizer for the nonlinear channel equalization, which used LMS and RLS algorithms to adjust the parameter. The build steps are as follows. First, a fuzzy set in the input space of equalizer is defined. And then, the membership functions of fuzzy set are made to cover the whole input space. Finally, a set of fuzzy rules in the form of “if–then” are constructed. On the basis of fuzzy rules, the fuzzy adaptive equalizer is constructed, which finally use LMS and RLS algorithms to update the parameters of the equalizer. In 1997, Chin-Teng Lin et al. [54] proposed a new adaptive neural fuzzy filter (ANFF) based on the learning ability of neural network and fuzzy “if–then” rules. It is essentially a multilayer feed-forward network and uses the fuzzy rules of FNN to solve the “black box” problem of the neural network. It has two characteristics. First is the combination of prior knowledge and ANFF. The other is to find the optimal

12

1 Introduction

structure and parameters. It is suitable for online work. At the same year, Ki Yong Lee et al. [55] proposed a neural network equalizer based on generalized probabilistic descent algorithm. It utilized the fuzzy decision learning rule to solve the problem of overlapping of decision range caused by the noise. In 1999, Ramain Pichevar et al. [56] used fuzzy rules to adjust the learning rate of the artificial neural network equalizer and the momentum of back-propagation network. In this way, the convergence speed of equalizer is improved. They think that traditional methods, such as linear transversal equalizer, can only equalize the common channel and nonlinear channel can only depend on the neural network equalizer. Their algorithm combines the back-propagation network and fuzzy rules to deal with nonlinear problems. Meanwhile, the convergence rate of their algorithm is speeded up by using fuzzy rules to adjust the parameter. In 2002, Kumar Sahu Prasanna et al. [57] proposed an FNN equalizer with low computational cost. Its performance is close to that of maximum posterior probability. But its computational complexity is significantly reduced. It can employ the supervised clustering algorithm for training. In 2003, Rafael Ferrari et al. [58] proposed a fuzzy error-prediction equalizer. It is an unsupervised nonlinear equalization. Its performance is close to that of the Bayesian (Bayes) equalizer and suitable for real-time processing. In 2004, Zu Jiakui [59] proposed nonsingleton fuzzy logic equalizer. The equalizer adopted the steepest gradient descent method as parameter optimization algorithm. Thus, the initial parameter has great influence on the equalizer. In their article, the clustering algorithm is used to choose the initial parameters, which effectively improve the convergence rate. But the problem of stability still exists. In 2005, Ching-Hung Lee et al. [60] proposed the equalization of the nonlinear time-varying channel by periodic FNN equalizer. The periodic FNN is based on the learning ability and fuzzy rules structure. And in the case of given limited information, Ching-Hung Lee’s algorithm uses periodic FNN to estimate the periodic signal, and adopts a feedback learning algorithm to accelerate the learning speed. For the equalization of the time-varying channel, the above method is simple at structure. At the same year, Zhang Liyi et al. [61] analyzed the combination mechanism of FNN and proposed the three kinds of BE algorithms: BE algorithm based on FNN filter, FNN controller, and FNN classifier, respectively. Bai Yu [62] constructed several kinds of FNN BE algorithms by replacing FFNN with the FNN. Sun Yunshan [63–66] structured FNN as classifier and proposed several BE algorithms based on fuzzy classifier. Zhang Xiaoqin [67, 68] structured FNN as controller to control the iteration step of the FNN blind equalizer. Her algorithm has better convergence effect. In 2006, Heng Song et al. [69] proposed an FNN decision feedback equalization algorithm based on optional fuzzy rules. In the algorithm, the neural network employs decision feedback structure, and the steepest gradient descent method is used for training. It has strong ability of anti-interference and nonlinear classification. The

1.3 The research progress of neural network BE algorithm

13

simulation results show that its error rate is smaller compared with other nonlinear channel equalization method.

1.3.4 The ENN BE algorithm The ENN is a new neural network model. In the ENN, evolutionary algorithm (EA) and neural network are complementary. The EA can simulate the natural evolutionary law and swarm intelligence optimization search. And the neural network can complete the weight learning and the structure optimization. The ENN organically integrated the above superiorities to obtain a strong robust adaptive neural network. The ENN is a new research hotspot in recent years, but the application of evolutionary neural network in communication channel equalization started fairly late. In 2003, Chen Jinzhao et al. [70], first, applied the genetic algorithm into BE. They analyzed as to how the genetic algorithm can be used to solve the problem of BE based on high-order cumulant. Its cost function is 󵄨󵄨 󵄨󵄨 ̃ 󵄨C (x(n)) 󵄨󵄨 J(n) = 󵄨󵄨 4 󵄨󵄨2 󵄨󵄨C4 (x(n)) ̃ 󵄨󵄨 󵄨

(1.19)

̃ ̃ and C4 (x(n)) are the second-order cumulant and fourth-order In eq. (1.19), C2 (x(n)) cumulant of the equalizer output signals, respectively. The algorithm adopted genetic algorithms to solve the maximum of cost function and avoided being trapped into local extremum during gradient search. In the genetic algorithm, the real number encoding, roulette rules, and nonuniform mutation are employed. The fitness function is F=

1 󵄨󵄨 C x(n) 󵄨󵄨 C x(n) 2 󵄨󵄨 | 4 ( ̃ )| 󵄨󵄨 ( )| | 4 ̃ – x(n)) ̂ !1 󵄨󵄨 ] 2 – 2 󵄨󵄨 – !2 E [(x(n) ̃ )| 󵄨󵄨 |C2 (x(n) |C2 (x(n))| 󵄨󵄨

(1.20)

In eq. (1.20), C2 (x(n)) and C4 (x(n)) are the second-order cumulant and fourth-order cumulant of the transmitted signals. !1 and !2 are the scale factors. The simulation results show that the algorithm has fast convergence speed and strong anti-interference ability. In 2006, Li Yuan [71, 72] combined the genetic algorithm with the neural network BE algorithm and proposed a two-stage learning scheme. In the scheme, the genetic algorithm is used first to do a rapid global search for the weights, and to make the performance of networks reach a certain requirement by controlling the genetic algebra. And then the BP algorithm is adopted for the local optimal search, until the requirements of BE is met. In the same year, Xiao Ying et al. [73] proposed an underwater acoustic channel BE algorithm based on the genetic algorithm optimization neural network. They optimized the topological structure and network weights of

14

1 Introduction

three-layer FFNN simultaneously. Their algorithm effectively overcomes the defects of the traditional FFNN BE algorithm, improves the generalization performance of FFNN BE algorithm, and strengthens the ability of tracking time-varying channel and adapting channel mutation. But further research is still needed for the reasonable selection of parameters of genetic operators and the decreasing calculation amount of algorithms. In this algorithm, the binary encoding, ranking selection rules are used. Its cost function is shown by eq. (1.5), and the fitness function is F=

2 2 󵄨󵄨 ̃ 󵄨󵄨2 [󵄨󵄨x(n)󵄨󵄨 – R2 ]

(1.21)

In 2007, Liu Yong [74, 75] adopted the genetic algorithm to optimize weights and structure of the neural network, and obtained the BE algorithm based on genetic algorithm optimization neural network weights and BE algorithm based on genetic algorithm optimization neural network structure.

1.3.5 BE algorithm based on WNN WNN is a kind of neural network structure. It combines the wavelet theory with neural network theory and can make full use of the good localized performance of wavelet transform and the self-learning function of neural network. WNN has the strong ability of approximation and fault tolerance. It also avoids the nonlinear optimization problems such as the blindness in design of the neural network structure and local optimum. In 1997, Shichun He and Zhenya He [76] applied the feedback WNN into BE of nonlinear communication channel. In 2004, Niu Xiaowei [77], respectively, proposed a BE algorithm based on three-layer feed-forward WNN and a BE algorithm based on bilinear feedback WNN. She derived the iterative formulas of real and complex systems, respectively. In 2009, Xiao Ying et al. [78] proposed a cascaded hybrid WNN adaptive blind equalizer for the channel with severe nonlinear distortion. Gao min and Guo Yecai et al. [79–81] introduced orthogonal wavelet transform and orthogonal wavelet packet transform theory into neural network BE algorithm.

1.4 The research background and structure 1.4.1 The research background Since BE came out 30 years ago, it has become a hot research topic gradually as the continuous development and wide application of digital communication technology. Famous foreign publications, such as “IEEE Transactions on Communication,”

1.4 The research background and structure

15

“IEEE Transactions on Signal Process,” “IEEE Transactions on Information,” “Signal Processing,” and so on, report on the latest research progress of the BE in every year and even every issue. Many colleges, universities, and research institutions in China also engage in the research of the subject. The National Natural Science Foundation, the 863 projects and so on, also provided a lot of investment and financing on the research. Especially, the emergence of new optimization theories has injected new vitality into the research of BE. Since the beginning of 2000, under the support of many funded projects, such as Shanxi Province Natural Science Foundation “Research on the blind equalization for mobile communication” (20011035), Chinese Postdoctoral Science Foundation “Research on the application of fuzzy neural network in blind equalization” (20060390170), Shanxi Province Natural Science Foundation “Research on blind equalization technology based on neural network” (20051038), Subject of Science and Technology Development Fund of Tianjin College “Research on blind equalization algorithm based on evolutionary neural network” (20060610), “Research on medical image blind restoration algorithm based on zigzag transform” (20110709), etc., the author of this book and his research team have achieved many combinations of the neural network and the BE algorithm, and research on the BE algorithms based on neural network, especially the BE algorithm based on the new neural network. On the basis of above research results, they have proposed many algorithm forms. These algorithms have advantages of low computational cost, better convergence performance, and strong practicability. These algorithms have important theoretical significance and practical value for compensation of the nonideal characteristics of channels. These algorithms can improve the reception and communication quality. This book is a summary and refinery of the research findings.

1.4.2 The structure of the book This book is divided into eight chapters. This chapter provided the significance and application fields of BE are analyzed. The classification and research status of the neural network BE algorithms are summarized. And the research background and main work of this book are pointed out. The second chapter introduces the concept, structure, algorithms, and equalization criterion of BE. The fundamental principles and learning methods of neural network BE algorithm are elaborated. The improvement methods of the traditional BP algorithm are summarized. The evaluation index of BE algorithms, especially the parameter of convexity of cost function and steady residual error and so on, is analyzed. In the third chapter, on the basis of analyzing the basic principle of FFNNs, the BE algorithm based on three-layer, four-layer, and five-layer FFNN are studied. BE

16

1 Introduction

algorithms based on momentum term and time-varying momentum term are also studied. The iterative formulas of the above algorithms are derived. The computer simulations are performed to validate the effectiveness of algorithms. In the fourth chapter, on the basis of analyzing the basic principle of FBNN, five kinds of BE algorithm are studied: BE algorithm based on bilinear recurrent neural network, diagonal recurrent neural network, quasi-diagonal recurrent neural network, the time-varying step diagonal recurrent neural network with mean square error nonlinear function, and time-varying step quasi-diagonal recurrent neural network. The iterative formulas of these algorithms are derived. The computer simulations are done to validate the effectiveness of algorithms. In the fifth chapter, on the basis of analyzing the basic principle of FNN, three kinds of BE algorithms are studied, respectively: BE algorithm based on FNN filter, FNN controller, and FNN classifier. The iterative formulas of the algorithms are derived. The computer simulations are done to validate the effectiveness of algorithms. In the sixth chapter, on the basis of analyzing the basic theory of ENN, the BE algorithms based on optimization neural network weights and structure by genetic algorithm are studied. The parameter selection and algorithm steps are described. The computer simulations are carried out. In the seventh chapter, on the basis of analyzing the basic principle of WNN, BE algorithms based on the three-layer FFNN and on the bilinear feedback WNN are studied. The iterative formula of the algorithm is derived. The computer simulations are done. In the eighth chapter, the imaging mechanism of medical computed tomographic (CT) image, degenerative process, the basic principle of image BE algorithm, and quantitative measure index are analyzed. On the basis of the above analysis, the medical CT image neural network BE algorithm based on zigzag encoding and dual zigzag encoding are studied. The iterative formulas of these algorithms are derived. The convergence performances are analyzed and the computer simulations are carried out.

2 The Fundamental Theory of Neural Network Blind Equalization Algorithm Abstract: In this chapter, the concept, structure, algorithm form, and equalization criterion of the blind equalization are first introduced. And then, the fundamental principle and learning method of the neural network blind equalization are expounded. Then, in order to overcome the shortcomings of traditional BP algorithms, some improved methods are summarized. Finally, the evaluation indexes of the blind equalization algorithm are analyzed. Among these evaluation indexes, the convex of cost function and steady residual error are analyzed emphatically.

2.1 The fundamental principle of blind equalization 2.1.1 The concept of blind equalization The blind equalization can just only use the prior information of the received sequence itself to equalize the channel character. That is, it makes the output sequence of equalizer to approximate the transmitted sequence as far as possible, without the aid of training sequence. Its functional block diagram is shown in Figure 2.1. In Figure 2.1, h(n) is the impulse response of discrete time transmission channel (including transmit filter, transmission medium, and receive filter). Depending on what the modulation method is used, h(n) can be a real value or a complex value. w(n) is the impulse response of equalizer. The equalizer generally employs a finitelength transversal filter with length L. x(n) is a transmitted sequence of system; y(n) is a received sequence after channel transmission and is also the input sequence of ̃ equalizer; n(n) is additive noise of the channel; x(n) is the recovery sequences after equalizing. According to Figure 2.1 y(n) = h(n) ∗ x(n) + n(n) = ∑ h(i)x(n – i) + n(n)

(2.1)

i

where ∗ expresses the convolution operation. In order to get x(n) from y(n), it is necessary to carry out a deconvolution algorithm. The deconvolution algorithm is equivalent to identify the inverse channel h–1 (n) of a transmission channel h(n). When y(n) and x(n) are known, it is not difficult to solve this problem. The training of adaptive equalizer belongs to the case. But when x(n) is unknown, that is, when only y(n) is known in the three parameters, it is very difficult to solve h–1 (n). The mathematical model of such a problem is called blind deconvolution. The blind equalization is an application of the blind deconvolution in the communications field. DOI 10.1515/9783110450293-002

18

2 The Fundamental Theory of Neural Network Blind Equalization Algorithm

Noise n(n) Transmitted sequence

Transmission channel h(n)

x (n)

Received sequence

+

y (n)

Recovery sequence

Blind equalizer w (n)

~ x(n)

Figure 2.1: The functional block diagram of blind equalization.

y (n)

w0 (n)

Z −1

×

y (n – 1)

w1 (n)

Z −1

×

y(n − L + 1)

Z −1 wL−2(n)

wL−1(n)

×

×

Σ

~ x(n) Figure 2.2: The structure of transversal filter.

In practical application, the solution of h(n) or x(n) will be uncertain if only the received sequence is known. An identical observation sequence can be the convolutions between two different quantities. The solution of the problem is a challenge for the international blind processing [82].

2.1.2 The structure of blind equalizer Now, the classic blind equalizer mainly adopts the finite tapped transversal filter. Its structure diagram is shown in Figure 2.2. The input sequence vector Y(n) of the transversal filter is T

Y(n) = [y(n), y(n – 1), ⋅ ⋅ ⋅ , y (n – L + 1)]

(2.2)

The weight vector (or coefficient vector) W(n) of the filter is T

W(n) = [w0 (n), w1 (n), ⋅ ⋅ ⋅ , wL–1 (n)]

(2.3)

̃ of the transversal filter can be expressed as Then, the output x(n) L–1

̃ x(n) = ∑ wi (n)y(n – i) = YT (n)W(n) = WT (n)Y(n) i=0

where L is the length of transversal filter.

(2.4)

2.1 The fundamental principle of blind equalization

19

2.1.3 The basic algorithm of blind equalization Because the blind equalizer is developed on the basis of the adaptive equalizer, its algorithms are basically similar with the adaptive equalization algorithms, except for no training sequences. For the adaptive equalizer, the most common algorithms are the least mean square (LMS) algorithm and the recursive least squares (RLS) algorithm [83]. 2.1.3.1 The LMS algorithm Suppose that d(n) is the expected response signal of system. It is also known as the ̃ of the filter and d(n), i. e. training signal. e(n) is the error between the output x(n) ̃ e(n) = d(n) – x(n) = d(n) – YT (n)W(n)

(2.5)

According to the minimum mean square error (MMSE), the mean square error (MSE) between the output of filter and the expected response is defined as the cost function, i. e. 2

J(n) = E[e2 (n)] = E[(d(n) – YT (n)W(n)) ]

(2.6)

= E[d2 (n)] – 2E[d(n)YT (n)] W(n) + WT (n)E[Y(n)YT (n)] W(n) Defining R = E[Y(n)YT (n)] as autocorrelation matrix of the equalizer input sequence, which is an L × L square matrix, and P = E[d(n)YT (n)] as a cross-variance matrix, eq. (2.6) can be expressed as follows: J(n) = E[d2 (n)] – 2WT (n)P + WT (n)RW(n)

(2.7)

According to the MMSE criterion, let the gradient of eq. (2.7) at W(n) be zero, i. e. ∇=

𝜕J(n) = 2RW(n) – 2P = 0 𝜕W(n)

(2.8)

Then, the optimal value W∗ (n) of W(n) should satisfy eq. (2.9): W∗ (n) = R–1 P

(2.9)

where W∗ (n) is the Wiener solution of the transversal filter, namely, it is the optimal coefficient vector of filter. Because the MSE function (namely the cost function) is a quadratic equation of the filter coefficients W(n), it forms a multi-dimensional ultraparabolic surface. It seems to be a bowl-shaped surface with only minimum point located at the bottom of the bowl. Usually, it is called as the error performance surface of the filter. When the initial value of W(0) is given, the MSE is located at a point

20

2 The Fundamental Theory of Neural Network Blind Equalization Algorithm

of the error performance surface. The adaptive adjustment of the coefficients makes the MSE to move toward the minimum point at the bottom of bowl, and finally reach the minimum point. Thus, the optimum Wiener filter is achieved. For the adaptive equalization, eq. (2.9) needs to be changed into adaptive algorithm. At present, many gradient estimation methods have been proposed. One of the most famous and most widely applied is the LMS algorithm proposed by B. Widrow. The core idea of the algorithm is to replace the MSE with the square error. That is, change eq. (2.8) as follows: ∇=

𝜕e2 (n) = –2e(n)Y(n) 𝜕W(n)

(2.10)

According to the steepest descent method, the formula of LMS adaptive equalization algorithm is W(n + 1) = W(n) + 2,e(n)Y(n)

(2.11)

where , is the step factor. 2.1.3.2 The RLS algorithm The key of the RLS algorithm is to use the minimization criterion of time average of the square to replace the LMS criterion of LMS algorithm. The minimization criterion is to average the square of the error signals from the initial time to the current time, and then to minimize the average value. The cost function is defined as n

J(n) = ∑ +n–i e2 (i)

(2.12)

i=0

where 0 < + < 1 is called as forgetting factor. It can allocate the heavier weight for the data that are closer to time n and the lesser weight for the data that are farther from time n. That is, for the error of every moment, + has a certain forgetting effect. According to the minimization criterion, the derivative of the cost function is solved. And let it equal to zero, namely, ∇=

n 𝜕J(n) = –2 ∑ +n–i e(i)Y(i) = 0 𝜕W(n) i=0

(2.13)

By substituting eq. (2.5) into eq. (2.13), eq. (2.14) can be obtained: n

n

i=0

i=1

∑ +n–i d(i)Y(i) = [∑ +n–i Y(i)YT (i)] W(n)

(2.14)

21

2.1 The fundamental principle of blind equalization

Let n

n

i=0

i=0

R(n) = ∑ +n–i Y(i)YT (i) and P(n) = ∑ +n–i d(i)Y(i) And then, by substituting them into eq. (2.14), eq. (2.15) can be obtained: W(n) = R–1 (n)P(n)

(2.15)

As shown in eq. (2.15), the solution obtained by the least square criterion is still the Wiener solution. The recursive estimation formulas of R(n) and P(n) can be expressed as follows: R(n) = +R(n – 1) + Y(n)YT (n)

(2.16)

P(n) = +P(n – 1) + d(n)Y(n)

(2.17)

By applying the lemma of matrix inversion for eq. (2.16), eq. (2.18) can be obtained: 1 R–1 (n – 1)Y(n)YT (n)R–1 (n – 1) [R–1 (n – 1) – ] + + + YT (n)R–1 (n – 1)Y(n) 1 = [R–1 (n – 1) – k(n)YT (n)R–1 (n – 1)] +

R–1 (n) =

(2.18)

R–1 (n – 1)Y(n)

= R–1 (n)Y(n) is the Kalman gain vector. + + YT (n)R–1 (n – 1)Y(n) By substituting eqs. (2.17) and (2.18) into eq. (2.15), eq. (2.19) can be obtained:

Here k(n) =

W(n) = +–1 [R–1 (n – 1) – k(n)YT (n)R–1 (n – 1)] × [+P(n – 1) + Y(n)d(n)] = R–1 (n – 1)P(n – 1) + +–1 d(n)R–1 (n – 1)Y(n) – k(n)YT (n)R–1 (n – 1)P(n – 1) – +–1 d(n)k(n)YT (n)R–1 (n – 1)Y(n)

(2.19)

= W(n – 1) + k(n) [d(n) – YT (n)W(n – 1)] Let e (n, n – 1) = d(n) – YT (n)W(n), then the iterative formula of the RLS algorithm can be obtained: W(n) = W(n – 1) + k(n)e (n, n – 1)

(2.20)

2.1.3.3 The comparison of several classes of algorithms In the blind equalization algorithms, LMS algorithm and RLS algorithm have their own characteristics. The former has the advantages of low computational complexity and easy programming. But the convergence speed of LMS algorithm is slow, and the ability of tracking the channel change is poor. The latter is good at convergence speed and

22

2 The Fundamental Theory of Neural Network Blind Equalization Algorithm

Table 2.1: Performance comparison of several algorithms Sequence number

Names of algorithms

Number of multiplication

Advantages

Shortcomings

1

LMS algorithm

2L + 1

Slow convergence, poor tracking ability

2

RLS algorithm

2.5L2 + 4.5L

3

Fast transversal filter algorithm

7L + 14

4

Gradient lattice algorithm

13L – 8

Low computational complexity and easy to implement Fast convergence, strong ability of tracking Fast convergence, strong ability of tracking, low computational complexity Stabilization, low computational complexity

High computational complexity Programming is complex, instability

Programming is complex, poor performance

the ability of tracking the channel change, but its computational complexity is high. In Table 2.1, the performances of several algorithms are compared.

2.1.4 The equalization criteria of blind equalization According to the current research progress, there are main three equalization criteria of blind equalizer [84], namely zero-forcing criterion, kurtosis criterion, and normalized criterion.

2.1.4.1 The zero-forcing criterion ̃ of the blind equalizer is In Figure 2.1, the output sequence x(n) ̃ x(n) = w(n) ∗ y(n)

(2.21)

̃ The blind equalizer is designed to make x(n) become the best estimate of x(n). Therefore, ̃ x(n) = x (n – D) e j6 where D is an integer delay and 6 is a constant phase shift.

(2.22)

2.1 The fundamental principle of blind equalization

23

In order to satisfy eq. (2.22), on the premise that the channel noise superposition is not considered requires w(n) ∗ h(n) = $(n – D) e j6

(2.23)

where $(n) is the Kronecker function. To calculate the Fourier transformation of eq. (2.23), the following equation can be obtained: +∞

W(9)H(9) = ∑ $(n – D) e j6 e–j9n = e j(6–9D)

(2.24)

n=–∞

namely, W(9) =

1 e j(6–9D) H(9)

(2.25)

According to the above analysis, the relationship between the equalizer transmission function and channel transfer function should satisfy eq. (2.25) in the design process of blind equalizer. T Let s(n) = [s0 , s1 , ⋅ ⋅ ⋅ , sL–1 ] be the response function of the combined system, which is a combination of the transmission channel and the blind equalizer. Then s(n) = h(n) ∗ w(n)

(2.26)

̃ x(n) = s(n) ∗ x(n) = ∑ si (n)x(n – i) = x (n – D) e j6

(2.27)

namely

i

According to the above equations, when the response function of the combined system is a finite dimensional vector (the length is L), the vector has only a nonzero element (its mode is 1), that is, 0, ⋅ ⋅ ⋅ , 0, e j6 , 0, ⋅ ⋅ ⋅ , 0]T s(n) = [s0 (n), s1 (n), ⋅ ⋅ ⋅ , sL–1 (n)]T = [0, ⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

(2.28)

D–1

where the number of zero before the nonzero element is D–1, and D ≤ L. and eq. (2.28) is called as the zero-forcing criterion of blind equalizer [85]. 2.1.4.2 The kurtosis criterion The kurtosis criterion is also known as SW theorem, which was formulated by O. Shalvi and E. Weinstein [86] in 1990.

24

2 The Fundamental Theory of Neural Network Blind Equalization Algorithm

Let x(n) be the independent identical distribution. From eq. (2.27), the following equation can be obtained: 󵄨󵄨 󵄨󵄨2 󵄨 󵄨 ̃ 2 ] = E[󵄨󵄨󵄨󵄨∑ si (n)x(n – 1)󵄨󵄨󵄨󵄨 ] E[|x(n)| 󵄨󵄨 󵄨󵄨 󵄨] [󵄨 i = E[∑ ∑ si (n)s∗j (n)x(n – i)x∗ (n – j)] ] [ i j = ∑ ∑ E[x(n – i)x∗ (n – j)] si (n)s∗j (n) i

(2.29)

j

= E [|x(n)|2 ] ∑ |si (n)|2 i

In addition, to take the square on both sides of eq. (2.27) x̃ 2 (n) = ∑ s2i (n)x2 (n – i) + 2 ∑ si (n)sj (n)x(n – i)x (n – j) i, j, i = ̸ j

i

=

∑ s2i (n)x2 (n i

(2.30)

– i)

By calculating the mathematical expectation of eq. (2.30), the following equation can be obtained: E[x̃ 2 (n)] = E[∑ s2i (n)x2 (n – i)] = E[x2 (n)] ∑ s2i (n) i

(2.31)

i

And then, by using eq. (2.27), eq. (2.32) can be obtained: 󵄨󵄨 󵄨󵄨4 󵄨 ̃ 󵄨󵄨4 󵄨󵄨󵄨 󵄨󵄨󵄨 ] [ ] = E s (n)x(n – i) E[󵄨󵄨󵄨x(n) ∑ 󵄨󵄨 󵄨󵄨 󵄨󵄨 i 󵄨󵄨 󵄨󵄨 󵄨 ] [󵄨 i = E[∑ ∑ ∑ ∑ si (n)s∗j (n)sm (n)s∗k (n)x(n – i)x∗ (n – j) x (n – m) x∗ (n – k)] ] [ i j m k = ∑ ∑ ∑ ∑ si (n)s∗j (n)sm (n)s∗k (n)E[x(n – i)x∗ (n – j) x (n – m) x∗ (n – k)] i

j

m k

(2.32) In eq. (2.32) 󵄨 󵄨4 E[󵄨󵄨󵄨x(n)󵄨󵄨󵄨 ] i = j = m = k { { { { { {E2 [󵄨󵄨󵄨x(n)󵄨󵄨󵄨2 ] i = j ≠ m = k, i = k ≠ j = m E[x(n – i)x∗ (n – j) x (n – m) x∗ (n – k)] = {󵄨 󵄨2 󵄨󵄨2 󵄨󵄨E[x (n)]󵄨󵄨 { i = m ≠ j = k { 󵄨 󵄨󵄨 { { {󵄨 otherwise {0 (2.33)

2.1 The fundamental principle of blind equalization

25

Substituting eq. (2.33) into eq. (2.32), eq. (2.34) can be obtained: 2

󵄨 󵄨4 󵄨4 󵄨2 { 󵄨2 󵄨4 } 󵄨 ̃ 󵄨󵄨4 󵄨 󵄨 󵄨 2 󵄨 E[󵄨󵄨󵄨x(n) 󵄨󵄨 ] = E[󵄨󵄨󵄨x(n)󵄨󵄨󵄨 ] ∑ 󵄨󵄨󵄨si (n)󵄨󵄨󵄨 + 2E [󵄨󵄨󵄨x(n)󵄨󵄨󵄨 ] {[∑ 󵄨󵄨󵄨si (n)󵄨󵄨󵄨 ] – ∑ 󵄨󵄨󵄨si (n)󵄨󵄨󵄨 } i i { i } 2 󵄨󵄨 󵄨󵄨 󵄨󵄨 󵄨2 󵄨󵄨 󵄨 󵄨4 󵄨 + 󵄨󵄨󵄨󵄨E[x2 (n)]󵄨󵄨󵄨󵄨 [󵄨󵄨󵄨∑ s2i (n)󵄨󵄨󵄨 – ∑ 󵄨󵄨󵄨si (n)󵄨󵄨󵄨 ] 󵄨󵄨 󵄨󵄨 i 󵄨 [󵄨 i ]

(2.34)

Substituting eqs. (2.29) and (2.31) into eq. (2.34), (2.35) can be obtained: 2

󵄨 ̃ 󵄨󵄨4 󵄨 󵄨4 󵄨4 󵄨2 󵄨2 󵄨 󵄨 2 󵄨 E[󵄨󵄨󵄨x(n) 󵄨󵄨 ] = E[󵄨󵄨󵄨x(n)󵄨󵄨󵄨 ] ∑ 󵄨󵄨󵄨si (n)󵄨󵄨󵄨 + 2E [󵄨󵄨󵄨x(n)󵄨󵄨󵄨 ] [∑ 󵄨󵄨󵄨si (n)󵄨󵄨󵄨 ] i

i

󵄨󵄨2 󵄨 󵄨󵄨 󵄨2 󵄨󵄨󵄨 󵄨 󵄨2 󵄨4 󵄨 󵄨 – 2E2 [󵄨󵄨󵄨x(n)󵄨󵄨󵄨 ] ∑ 󵄨󵄨󵄨si (n)󵄨󵄨󵄨 + 󵄨󵄨󵄨󵄨E[x2 (n)]󵄨󵄨󵄨󵄨 󵄨󵄨󵄨∑ s2i (n)󵄨󵄨󵄨 󵄨󵄨 󵄨󵄨 i 󵄨 󵄨 i 󵄨󵄨 2 󵄨󵄨2 󵄨󵄨 󵄨󵄨4 󵄨 󵄨 – 󵄨󵄨E[x (n)]󵄨󵄨 ∑ 󵄨󵄨si (n)󵄨󵄨

(2.35)

i

󵄨 󵄨4 󵄨4 󵄨 ̃ 󵄨󵄨2 󵄨2 󵄨4 󵄨 󵄨 2 󵄨 = E[󵄨󵄨󵄨x(n)󵄨󵄨󵄨 ] ∑ 󵄨󵄨󵄨si (n)󵄨󵄨󵄨 + 2E2 [󵄨󵄨󵄨x(n) 󵄨󵄨 ] – 2E [󵄨󵄨󵄨x(n)󵄨󵄨󵄨 ] ∑ 󵄨󵄨󵄨si (n)󵄨󵄨󵄨 i

i

󵄨 󵄨2 󵄨 󵄨2 󵄨 󵄨4 + 󵄨󵄨󵄨󵄨E[x̃ 2 (n)]󵄨󵄨󵄨󵄨 – 󵄨󵄨󵄨󵄨E[x2 (n)]󵄨󵄨󵄨󵄨 ∑ 󵄨󵄨󵄨si (n)󵄨󵄨󵄨 i

By moving the mathematical expectations of transmitted sequence and recovery sequence to both sides of the equal sign, respectively, eq. (2.36) can be obtained: 󵄨 󵄨2 󵄨 ̃ 󵄨󵄨4 2 󵄨 ̃ 󵄨󵄨󵄨󵄨2 ] – 󵄨󵄨󵄨E[x̃ 2 (n)]󵄨󵄨󵄨 E[󵄨󵄨󵄨x(n) 󵄨󵄨 ] – 2E [󵄨󵄨󵄨x(n) 󵄨 󵄨 󵄨 󵄨4 󵄨󵄨4 󵄨 󵄨2 󵄨4 󵄨󵄨 󵄨 󵄨 󵄨 = E[󵄨󵄨x(n)󵄨󵄨 ] ∑ 󵄨󵄨si (n)󵄨󵄨 – 2E2 [󵄨󵄨󵄨x(n)󵄨󵄨󵄨 ] ∑ 󵄨󵄨󵄨si (n)󵄨󵄨󵄨 i

󵄨 󵄨2 󵄨 󵄨4 – 󵄨󵄨󵄨󵄨E[x2 (n)]󵄨󵄨󵄨󵄨 ∑ 󵄨󵄨󵄨si (n)󵄨󵄨󵄨

i

(2.36)

i

󵄨2 󵄨 󵄨4 󵄨 󵄨2 󵄨 󵄨4 󵄨 = {E[󵄨󵄨󵄨x(n)󵄨󵄨󵄨 ] – 2E2 [󵄨󵄨󵄨x(n)󵄨󵄨󵄨 ] – 󵄨󵄨󵄨󵄨E[x2 (n)]󵄨󵄨󵄨󵄨 } ∑ 󵄨󵄨󵄨si (n)󵄨󵄨󵄨 i

̃ To define that eqs. (2.37) and (2.38) are the kurtosis of the output sequence x(n) and the input sequence x(n), respectively, 󵄨 󵄨2 󵄨 ̃ 󵄨󵄨4 2 󵄨 ̃ ̃ 󵄨󵄨󵄨󵄨2 ] – 󵄨󵄨󵄨E[x̃ 2 (n)]󵄨󵄨󵄨 K [x(n)] = E[󵄨󵄨󵄨x(n) 󵄨󵄨 ] – 2E [󵄨󵄨󵄨x(n) 󵄨 󵄨 󵄨2 󵄨󵄨2 󵄨󵄨󵄨 2 󵄨󵄨 󵄨󵄨4 2 󵄨󵄨 K [x(n)] = E[󵄨󵄨x(n)󵄨󵄨 ] – 2E [󵄨󵄨x(n)󵄨󵄨 ] – 󵄨󵄨E[x (n)]󵄨󵄨󵄨󵄨

(2.37) (2.38)

According to the value of kurtosis, signals can be divided into three types: the Gaussian signal (kurtosis is zero), sub-Gaussian signal (kurtosis is negative), and super-Gaussian signal (kurtosis is positive).

26

2 The Fundamental Theory of Neural Network Blind Equalization Algorithm

Substituting eqs. (2.37) and (2.38) into eq. (2.36), the following equation can be obtained: 󵄨4 󵄨 ̃ K [x(n)] = K [x(n)] ∑ 󵄨󵄨󵄨si (n)󵄨󵄨󵄨

(2.39)

i

According to the above analysis, the SW theorem can be obtained as follows: 󵄨 ̃ 󵄨󵄨2 󵄨 󵄨2 If E[󵄨󵄨󵄨x(n) 󵄨󵄨 ] = E[󵄨󵄨󵄨x(n)󵄨󵄨󵄨 ], then 󵄨󵄨 󵄨 󵄨 󵄨 󵄨󵄨 ≤ 󵄨󵄨K [x(n)]󵄨󵄨; ̃ (1) 󵄨󵄨K [x(n)] 󵄨 󵄨 󵄨 󵄨󵄨 󵄨󵄨 󵄨 󵄨󵄨 󵄨 ̃ (2) 󵄨󵄨K [x(n)] 󵄨󵄨 = 󵄨󵄨K [x(n)]󵄨󵄨, it can be true if and only if the vector s(n) meets eq. (2.28). The equalization criterion of a blind equalizer given by the theorem requires to max̃ imize the kurtosis of recovery sequence x(n) under the constraint condition that the ̃ variances of x(n) and x(n) are equivalent, that is, under the constraint conditions 󵄨 ̃ 󵄨󵄨2 󵄨 󵄨2 󵄨 󵄨󵄨 ̃ E[󵄨󵄨󵄨x(n) 󵄨󵄨 ] = E[󵄨󵄨󵄨x(n)󵄨󵄨󵄨 ], to maximize 󵄨󵄨󵄨K [x(n)] 󵄨󵄨. The SW theorem provides a necessary and sufficient condition of channel blind equalization, and boils down the problem of the channel equalization as a maximization problem under a constraint condition. 2.1.4.3 The normalization criterion The normalization criterion, also called as Gadzow theorem, is another criterion of blind equalization. It was an extension of the SW theorem and proposed by J.A. Gadzow [87] in 1996. Suppose that the transmitted sequence x(n) is non-Gaussian, independent and identically distributed (i.i.d.) stationary process, and its N order cumulates are written ̃ of blind equalizer is expressed as cNx , the N order cumulates of output sequence x(n) as cN x̃ . BBR formula is derived by Bartlett, Brilinger, and Rosenblatt, so the formula is called the BBR formula. According to the BBR formula, eq. (2.40) can be obtained: cN x̃ = cNx ∑ sNi (n)

(2.40)

i

The (M, N)-order normalized cumulates of the transmitted sequence x(n) and the ̃ are, respectively, defined as follows: output sequence x(n) kx (M, N) =

cMx M/N

[cNx ]

, kx̃ (M, N) =

cMx̃ M/N

[cN x̃ ]

(2.41)

Here, suppose cN x̃ ≠ 0, cNx ≠ 0. According to eqs. (2.40) and (2.41), eq. (2.40) can be obtained: kx̃ (M, N) =

∑i sM i (n)

k M/N x

[∑i sNi (n)]

(M, N)

(2.43)

2.2 The fundamental theory of neural network

27

From eq. (2.43), the Gadzow theorem can be obtained. The Gadzow theorem is that if the input sequence x(n) of channel is non-Gaussian, i.i.d. stationary process, then its ̃ is also non-Gaussian, i.i.d. stationary process. The relationship output sequence x(n) of the normalized cumulates of the input and output is as follows: (1) If N is an even number, and M > N, then kx̃ (M, N) ≤ kx (M, N) (2)

(2.44)

If N is an even number, and M < N, then kx̃ (M, N) ≥ kx (M, N)

(2.45)

The necessary and sufficient condition for the above two equations to be established is that there is just one nonzero element in s(n), i. e., to satisfy eq. (2.28). The Gadzow theorem provides the channel blind equalization with another necessary and sufficient condition, and transforms the channel equalization problem into an unconstrained maximization problem. Among the three criteria, the zero-forcing criterion has only theoretical significance, but no practical value. It is the basis of other criteria. The kurtosis criterion is an equalization criterion with constraint condition, which requires that the input power and output power of system are equal, i. e., the gain of system is 1. It does not conform to the actual situation. In the actual communication system, the system is only required to be nondistortion, and no special requirements for gain, so the criterion has a certain practical value, but has limitations. The normalized criterion is an equalization criterion without the constraints condition. It is a simple optimization criterion. In the normalized criterion, the order number of cumulates can be chosen arbitrarily, so the equalization criterion is not one but a cluster. Thus, it has a wide application value [84].

2.2 The fundamental theory of neural network 2.2.1 The concept of artificial neural network The artificial neural network (ANN) is a term relative to the biological neural network in biology. ANN is an adaptive nonlinear dynamic system. It is composed of many basic computing units known as neurons. These neurons are interconnected widely. The information processing of network is realized by the interactions between neurons. The learning and recognition of network depend on the dynamic evolution process of neurons. It was put forward on the basis of research achievements of the modern biological neuroscience and reflected the basic features of human brain function.

28

2 The Fundamental Theory of Neural Network Blind Equalization Algorithm

2.2.2 The development of ANN The research of ANN began in 1943. It experiences the development history from rise to depression and from depression to prosperous. Its development history can be roughly divided into the following stages: (1) The start-up stage: in 1943, French psychologist W.S. McCulloch and mathematical logician W. Pitts proposed the mathematical model of neurons (i. e., M-P neural network model) based on the analysis and summary of the basic properties of neurons. Their work opened a prelude of the ANN research. In 1949, psychologist D.O. Hebb proposed the hypothesis of synaptic strength adjustment rule between neurons, namely the Hebb rule. The rule still plays an important role in the ANN model. (2) The initial boom stage: in this stage, for the research of neural network theory, the foundation of researching the ANN from the system perspective was basically established. In 1957, F. Rosenblatt proposed the perceptron model. It makes the study of neural network from theoretical exploration into practice application for the first time, and provokes the first climax of the neural network research. In 1960, B. Widrow et al. proposed the adaptive linear element (Adline) network. The model gets a good application in adaptive systems, such as adaptive filtering, prediction and pattern recognition, and other fields. (3) The dismal state stage: in the 1960s, due to the rapid development of computer, coupled with the limited function of the single-layer neural network, and lack of effective learning algorithm, the neural network research went into the dismal state stage. Especially in 1969, M. Minsky et al., who are artificial intelligence scholars at the Massachusetts Institute of Technology, demonstrated the functional limitations of a simple linear perceptron in their published monograph Perceptrons. They pointed out that the perceptron can’t solve xor problem, and the effective calculation method of multilayer network hasn’t been found. Thus, many researchers lose faith in neural network research. (4) The revival stage: in 1982, the biological physicist Dr. J.J. Hopfield [88] at California Institute proposed a discrete neural network model, which marked the beginning of recovery of the study on neural networks. In 1984, Dr. J.J. Hopfield [89] proposed the continuous neural network model and realized the electronic circuit simulation of the neural network. His work opens up a new way for the computer application in neural network, and successfully solved the famous optimal combination problem (traveling salesman problem). His research achievements get a wide attention of the researchers in related fields. In 1986, D.E. Rumelhart et al. [90] proposed the error back-propagation algorithm (BP algorithm) for the multilayer neural network. Their work refutes M. Minsky et al.’s wrong conclusions. The BP algorithm has become the most widely used, most extensively studied, and fastest growing algorithm.

2.2 The fundamental theory of neural network

(5)

29

The high-tide stage: since J.J. Hopfield proposed the ANN model and D.E. Rumelhar et al. proposed the BP algorithm, there are more and more academic activities, research institutions, monographs, and special issues related to the neural network. The related research results also continue to emerge. Similarly, in 1988, L.O. Chua [91, 92] proposed the cellular neural network model, in 1984, G.E. Hinton et al. [93] proposed the Boltzmann machine, and so on. In June 21–24, 1987, the first world neural network conference was held in San Diego, USA, which marked that the research on neural network has been performed all over the world.

Now, ANN has been widely applied and has made great progress in computer image processing, speech processing, optimization, intelligent control, and other fields.

2.2.3 The characteristics of ANN Because ANN is designed by simulating the human brain, it has the characteristics as follows: 2.2.3.1 Distributed associative memory When compared with the general computer unit, the storage mode of ANN is a distributed associative memory. The weight value of the connection between neurons is the storage unit, which represents the current state of the knowledge stored in the network. The storage of ANN is associative. To input an incomplete information into the network, the ANN will automatically search among the weights of the network, until the joint unit that best matches with input information is found. The biggest advantage of associative memory is able to store many patterns of information, and also can quickly classify these information into the existing storage mode. 2.2.3.2 Better fault tolerance In the traditional computer unit, if a portion of the memory cell is destroyed, the whole system will be paralyzed. However, the information stored in the neural network is not stored in a fixed place, so the efficacy of neural network will be only reduced a little if some neurons are damaged. It does not lead the whole system to collapse. 2.2.3.3 Strong adaptive ability The adaptive ability refers to the ability of neural networks to adjusting by itself. If a connection weight of network changes, the network can automatically recover by

30

2 The Fundamental Theory of Neural Network Blind Equalization Algorithm

learning (algorithm). On the other hand, the neural network has the ability to learn from the external environment, and the function to improve itself by learning. The learning can be achieved by either the training based on the input samples or not. The purpose of the learning is to extract the relevant characteristics or inherent law from a set of data.

2.2.4 Structure and classification of ANN 2.2.4.1 The basic structure of ANN ANN is composed of some neurons on the basis of certain rules. Under the effect of the input signal, each neuron can transform from the current state into another state, and produce the corresponding output signal. The neurons decide the characteristics and functions of the whole neural network, and its basic structure is shown in Figure 2.3. In Figure 2.3, x1 , x2 , ⋅ ⋅ ⋅ , xn are the input signals of the neuron i, which can be an external input and also can be the output of other neurons.wi1 , wi2 , ⋅ ⋅ ⋅ , win are the weights of connection; (i is the threshold of neuron i; ui is the linear combination of input signals of neuron i; vi is a net input of neuron i; f (⋅) is the transfer function of neuron i, and also known as activation function; yi is the output of neuron i. The output characteristic of the neuron can be expressed as n

ui = ∑ wij xj

(2.46)

j=1

v i = ui – ( i

(2.47)

yi = f (vi )

(2.48)

In a neural network, the transfer function decides the correspondence relation between the input and output of the neuron. The transfer functions commonly used are mainly threshold function, piecewise linear function, hyperbolic tangent function, and Sigmoid function.

x1

wi1

x2

wi2

xn

win

θi

Σ

ui

− v i ++

f (·)

yi Figure 2.3: Schematic diagram of the basic structure of neuron.

2.2 The fundamental theory of neural network

31

The representation form of the threshold function is {1 yi = f (vi ) = { 0 {

vi ≥ 0 vi < 0

(2.49)

The representation form of the piecewise linear function is 1 vi ≥ 1 { { {1 yi = f (vi ) = { 2 (1 + vi ) –1 < vi < 1 { { vi ≤ –1 {0

(2.50)

The representation form of the hyperbolic tangent function is v 1 – e–vi yi = f (vi ) = th ( i ) = 2 1 + e–vi

(2.51)

The representation form of the Sigmoid function is yi = f (vi ) =

1 1 + e–!vi

(2.52)

where the parameter ! > 0 is used for the control of the slope. 2.2.4.2 The classification of ANN According to the topology structure and connecting mode, ANN can be divided into two categories: the feed-forward neural network (FNN) and the feedback neural network (FBN). The feed-forward neural network The FNN generally consists of input layer, output layer, and hidden layer. The hidden layer can be a single layer and also can be a multilayer. Each neuron only receives the output of the previous layer as the input of itself, and transmits its output to the next layer. There is no feedback in the whole network. Each neuron can have any number of inputs, but only one output. From the perspective of its effect, the FNN is mainly a function mapping, which can be used for function approximation. The feedback neural network In FBN, the information transmission between neurons is no longer from one layer to another layer. But a link can exist between any two of the neurons. Each neuron is a processing unit and receives the input and gives output to the outside simultaneously. Because of the feedback mechanism, FBN is a nonlinear dynamic system. FBN can be described by the differential equation model and difference equation. According to which model is used, it can be divided into two categories, i. e., continuous system

32

2 The Fundamental Theory of Neural Network Blind Equalization Algorithm

and discrete system. From the perspective of its effect, the FBN utilizes the minimum value of the energy function to solve the optimization problem.

2.3 The fundamental principle of neural network blind equalization algorithm There are many combinations of neural network and blind equalization algorithm. Among them, the common one is to structure neural network as a filter, a controller, and a classifier. And then they are applied into blind equalization algorithm. In this way, the blind equalization algorithm based on the neural network filter, the neural network controller, or the neural network classifier can be obtained. One of the most commonly used is the first one.

2.3.1 The principle of blind equalization algorithm based on neural network filter There are two main principles of blind equalization algorithm based on neural network filter. One is based on the method of the cost function. Here, first, a network structure is selected, and then puts forward a cost function for the selected network structure. And according to the cost function, the recursive equation of weights (the equation contains the characteristics of input and output signals) is determined. Finally, the aim of weight adjustment is achieved by minimizing the cost function. The other is based on energy function method. The neural network always gets into the steady state from the direction that the energy function decreases. The above characteristic is the basic idea of the energy function method. In practice, by comparing the cost function of the traditional algorithm with the energy function, the former becomes the energy function of the network after being properly modified. Then, the state equation of the network based on the new energy function (mainly to design the parameters related to the weight in equation) is designed. In this way, the original network is transformed, and the purposes of blind equalization can be achieved. The functional block diagram of blind equalization algorithm based on neural network filter is shown in Figure 2.4.

2.3.2 The principle of blind equalization algorithm based on neural network controller The blind equalization algorithm based on neural network controller takes the neural network as a controller to control some parameters of the traditional constant modulus blind equalization algorithm, such as the iteration step size and momentum

2.3 The fundamental principle of neural network blind equalization algorithm

33

n(n) x(n)

Channel h(n)

+

y(n)

Neural network w(n)

~ x(n)

Decision device

ˆ x(n)

Blind equalization algorithm Figure 2.4: The functional block diagram of blind equalization algorithm based on neural network filter.

n (n) x (n)

Channel h(n)

+

y (n)

Neural network w (n)

~ x(n)

Decision device

xˆ(n)

Neural network controller Figure 2.5: The functional block diagram of blind equalization algorithm based on neural network controller.

factor. By controlling these parameters, the time-variable step size or the time-variable momentum factor blind equalization algorithms are obtained. The two kinds of algorithms can solve the contradiction between convergence speed and convergence accuracy. Thus, they improve the convergence performance. The functional block diagram of the blind equalization algorithm based on neural network controller is shown in Figure 2.5.

2.3.3 The principle of blind equalization algorithm based on neural network controller classifier The blind equalization algorithm usually adopts a decision device with a fixed threshold, which led to a larger decision error and higher bit error rate (BER). The defects of fixed decision threshold can be eliminated by using neural network as classifier to replace the original decision device. Each decision region does not necessarily belong to its own samples. But the neural network can overcome the deficiency that the signals within same region must be judged as the same signal. The functional block diagram of the blind equalization algorithm based on neural network classifier is shown in Figure 2.6.

34

2 The Fundamental Theory of Neural Network Blind Equalization Algorithm

n(n) x(n)

Channel h(n)

+

y(n)

Blind identification

Neural ~ Deconvolution x(n) xˆ(n) network w(n) classifier

Figure 2.6: The functional block diagram of blind equalization algorithm based on neural network classifier.

2.4 The learning method of neural network blind equalization algorithm 2.4.1 The BP algorithm The neural networks generally use the error BP algorithm. As long ago as 1969, Bryson proposed the theory of BP algorithm. In 1974 and 1985, Werbos and Paker proposed independently the BP algorithm, respectively. However, their work got no deserved concern. In 1986, the PDP Research Group comprising Bumelhart and McCleland et al. systemically proposed the theory of neural network based on BP algorithm, after an in-depth study on the microstructure of parallel distributed processing. The research findings of the group receive people’s universal approval and attention. The BP neural network is the most mature and most common network structure in these application fields such as signal processing and adaptive control [94]. The BP algorithm is a supervised learning algorithm. It belongs to a generalization of $ learning algorithms. Its purpose is to minimize the error sum of squares of the network output layer. The BP algorithm consists of two parts, i. e., information forward transmission and error BP. In forward process, by calculating layer by layer, the information inserted into the input layer propagates to the output layer through the hidden layer. And the outputs of neurons of each layer have effect on the input of neurons of next layer. If the desired outputs are not obtained at the output layer, the error variation value of the output layer would be calculated and be propagated backward. The error signals are propagated back along the original connection path through the network, and to modify the weights of the neurons in each layer until the desired goal is obtained. 2.4.1.1 The forward propagation The structure of the three-layer FNN is shown in Figure 2.7. In Figure 2.7, only the forward outputs are allowed in the neural network. Neurons of different layers are connected by weights. wij (n)(i = 1, 2, . . . , I; j = 1, 2, . . . , J) are the connection weights between the input layer and hidden layer, wjk (n)(k = 1, 2, . . . , K) are the connection weights between the hidden layer and output layer. u and v are the input and output of neurons, respectively. The superscript and the subscript of u and v indicate a layer and a neuron of the layer, respectively. yi (n) and x̃k (n) are the input

2.4 The learning method of neural network blind equalization algorithm

wij(n)

35

wjk(n)

y1 (n)

x~1(n)

y2 (n)

~ x 2(n)

yI (n)

~ x K(n) Input layer I

Output layer K

Hidden layer J

Figure 2.7: The structure of the three-layer feed-forward neural network.

and output of whole neural network, respectively. Thus, the transmission rule of the signal, i. e., the state equation is uIi (n) = yi (n) viI (n)

=

uIi (n)

(2.53) = yi (n)

(2.54)

I

I

i=1

i=1

uJj (n) = ∑ wij (n)viI (n) = ∑ wij (n)yi (n)

(2.55)

I

vjJ (n) = f1 (uJj (n)) = f1 (∑ wij (n)yi (n))

(2.56)

i=1 J

uKk (n) = ∑ wjk (n)vjJ (n)

(2.57)

j=1 J

x̃k (n) = vkK (n) = f2 (uKk (n)) = f2 (∑ wjk (n)vjJ (n))

(2.58)

j=1

In eqs. (2.56) and (2.58), f1 (n) and f2 (n) are the transfer function of the hidden layer and the output layer, respectively. 2.4.1.2 The back-propagation The error function of BP algorithm is generally defined as the sum of square errors. The errors are the difference between the actual output and the desired output. Let the Kth desired outputs of the output layer are dk (n), respectively. Then the MSE of the single output is ek (n) = dk – x̃k (n) The MSE of the output layer of the whole neural network is

(2.59)

36

2 The Fundamental Theory of Neural Network Blind Equalization Algorithm

E(n) =

1 K 2 1 K 2 ∑ ek (n) = ∑ [dk (n) – x̃k (n)] 2 k=1 2 k=1

(2.60)

In order to achieve the purpose of learning, the connection weights in the network need to be adjusted based on the error function. The BP algorithm adopts the steepest descent method to adjust weights, and the adjustment amount at each time is BW(n) = –,

𝜕E(n) 𝜕W(n)

(2.61)

where 0 < , < 1 is the learning rate, or adjusting step size, or iteration step size. For the output layer, the adjustment amount of weight is Bwjk (n) = –,1

𝜕E(n) = ,1 [dk (n) – x̃k (n)] f2󸀠 (uKk (n)) vkK (n) 𝜕wjk (n)

(2.62)

= ,1 ek (n)f2󸀠 (uKk (n)) vkK (n) The weight adjustment formula of output layer is wjk (n + 1) = wjk (n) + ,1 ek (n)f2󸀠 (uKk (n)) vkK (n)

(2.63)

where ,1 is the learning rate of output layer. For the hidden layer, the weight adjustment amount is Bwij = –,2

𝜕E(n) = ,2 [dk (n) – x̃k (n)] f2󸀠 (uKk (n)) wjk (n)f1󸀠 (uJk (n)) yi (n) 𝜕wij (n)

(2.64)

= ,2 ek (n)f2󸀠 (uKk (n)) wjk (n)f1󸀠 (uJj (n)) yi (n) The weight adjustment formula of hidden layer is wij (n + 1) = wi j(n) + ,2 ek (n)f1󸀠 (uJj (n)) f2󸀠 (uKk (n))wjk (n)yi (n)

(2.65)

where ,2 is the learning rate of hidden layer.

2.4.2 The improved BP algorithm The core of the BP algorithm is the steepest descent method. It is an error descent algorithm based on gradient. It not only has some advantages, such as simple principle, easy realization, and so on, but also has many deficiencies [95, 96]: 2.4.2.1 Lower-speed of convergence The reason of the slower convergence of BP algorithm is mainly the fixed learning rate (also known as iteration step size). The essence of BP algorithm is the optimization

2.4 The learning method of neural network blind equalization algorithm

37

calculation of the gradient descent method. The BP algorithm uses the information of the first-order derivative of the error function to guide the adjustment of weights at next iteration. In this way, the objective of minimum error can be obtained. In order to guarantee the convergence of the algorithm, the learning rate generally must be less than the maximum eigenvalue of input vector autocorrelation matrix [97], which determines that the convergence speed of the algorithm is slow. The learning rate is usually determined by experience and remains fixed during the training process. Thus, the adjustment magnitude of a network weight is decided by the partial derivative of error function with respect to the weight. At the relatively flat part of the error surface, the partial derivative is smaller. So the adjustment magnitude of weights is smaller. Therefore, many times adjusting is needed to descend the surface of the error function in the BP algorithm. At the part of the error surface with the larger surface curvature, the partial derivative is larger. So the adjustment magnitude of weights is larger. Thus, the overshoot phenomena may occur in the vicinity of the minimum point of the error function. The overshoot phenomena make the path of weight adjustment to be zigzag, and difficult to converge to the minimum. Hence, it leads the convergence speed of the algorithm to be slow. 2.4.2.2 Easy to fall into local minimum points The training of BP algorithm is to make the algorithm to arrive at the minimum point of the error along the surface of the error function from a starting point. As the error surface is a complex multidimensional one, there may exist several local minima, which will inevitably make the training to be trapped in local minimum. From the mathematical point of view, the BP algorithm belongs to a nonlinear optimization problem. It inevitably has the problem of local minima. Meanwhile, in the gradient descent method, the training is to start from a point and to adjust along the slope of the error function, and gradually reach the minimum of error. For complex networks, the error function is a multidimensional space surface. It is just like a bowl. The minimum is at the bottom of bowl. But the surface of the bowl is uneven. So during the training process, the algorithm may fall into a small trough area which is a local minimum. If changing from local minimum to any direction, the error will increase. Thus, the training cannot escape from the local minima. Therefore, the starting point of the learning model is very important for the convergence of the network. In view of the above defects of BP algorithm, many improved algorithms have been proposed in recent years. They can be mainly divided into three categories, such as the BP algorithm with heuristic information technology, the BP algorithm with numerical optimization technique, and the BP algorithm based on modern optimization theory. 2.4.2.3 The BP algorithm with heuristic information technology The essence of the BP algorithm with heuristic information technology is to increase the learning rate when the change of the error gradient is slow, and to reduce the

38

2 The Fundamental Theory of Neural Network Blind Equalization Algorithm

learning rate when the change is drastic [98]. There are mainly the variable learning rate method, adding momentum method, and so on. The variable learning rate method The variable learning rate refers to the learning rate that is variable in the process of network training. That is, in the initial stage of training, a larger learning rate is adopted to accelerate the convergence speed. When close to convergence, the learning rate is reduced to avoid that the amplitude of weight adjustment is so large to generate oscillation or divergence. In the variable learning rate method, there are many forms of changing the learning rate. They are mainly the following. The Vogel algorithm. The Vogel algorithm [99] is a common fast learning algorithm. It is an algorithm based on the retreat method of the optimization theory. The learning rate changing mode of the Vogel algorithm is W(n + 1) = W(n) + 𝛾 [W(n) – W(n – 1)] – ,(n)

𝜕E(n) 𝜕W(n)

(2.66)

where 𝛾 is the added momentum term or momentum factor, ,(n) is the variable learning rate, or the time-varying iteration step size. Its changing rule is ,(n) = {

,(n – 1)m ,(n – 1)n

𝛾=𝛾 𝛾=0

if BE(n) < 0 if BE(n) > 0

(2.67)

where BE(n) = E(n + 1) – E(n) is called as the variation of the error. m > 1 and n < 1 refer to the positive and negative learning factors, respectively. According to eq. (2.67), the learning rate is determined based on the specific situations. When the current amendment direction of error gradient is correct (namely BE(n) < 0), the learning rate is increased (m > 1), and the momentum term is added (𝛾 is not zero). Otherwise, the learning rate is reduced (n < 1) and the momentum term is discarded (𝛾 is zero). In this way, the learning efficiency is improved greatly. But in practical applications, there still expose many problems. For example, the adjustment of learning rate is determined by m and n. If m and n are close to 1, then the amendment amplitude at each iteration is smaller. If the initial step size is not set properly, then the process of getting an appropriate value by adjusting is very slow. If the amendment amplitude in iteration is larger, then the step size may be increased to a large value or reduced to very small value to adjust too frequently. At the same time, the alternation of the increasing and decreasing of step is very frequent, and the oscillations between increasing and decreasing will reduce the number of effective correction. Thus, the efficiency is decreased. In view of the problem, in 2003, Wang Xiaoping et al. [100] made an improvement. They set up upper and lower bounds for the step variation range to limit the amendment amplitude of step. If BE(n) > 0, then

2.4 The learning method of neural network blind equalization algorithm

39

weight modification is invalid. Only when BE(n) < 0 for several successive iterations, the step is allowed to adjust. In this way, the step oscillations are prevented. The other variable learning rate algorithm. According to the changing trend of error function, the method of variable learning rate is proposed in the literature [101], namely ,(n + 1) = ,(n)

E(n + 1) E(n)

(2.68)

Here, when the weights make E(n) far away from the minimum point, the learning rate is increased. When it is close to the minimum point, the learning rate is reduced. A practical adjustment method of step size is proposed based on the positive and negative of the variation of error in literature [102]. In the method, an initial step size is set first, and if the error variation BE increases after one iteration, the step will be multiplied by a constant n (n is less than 1), and then to recalculate the next iterative point along the original direction. If the error variation BE decreases after one iteration, the step will be multiplied by a constant m (m is more than 1). In this way, the step size can be reasonably adjusted at the cost of increasing computing efforts slightly, namely, {m,(n – 1) ,(n) = { n,(n – 1) {

if BE(n) < 0

(2.69)

if BE(n) > 0

At present, there are many improved BP algorithms based on the variable learning rate. The detailed content can be found in the literature [103–105]. The method of additional momentum term. The essence of the additional momentum term method is to transmit the weight adjustment effect of several recent iterations to the change of the current weight. During the modification of weights, both the error effect on the gradient and the change trend effect on the error surface need to be considered. In this way, the sensitivity of network to the local details of error surface is reduced, the defect of the network trapping in a local minimum is effectively inhibited, and the convergence of the algorithm is sped up. At present, there are many methods of additional momentum term. Many literatures have put forward different methods. The single momentum term method. The single momentum term method refers that there is just only one momentum factor in the weight adjustment process. In the single momentum term methods proposed by the literature [102, 106], a part of weight adjustment amount at last iteration is added to the amount of weight adjustment at current iteration. The iterative formula is W(n + 1) = W(n) + 𝛾 [W(n) – W(n – 1)] – ,

𝜕E(n) 𝜕W(n)

(2.70)

40

2 The Fundamental Theory of Neural Network Blind Equalization Algorithm

where W(n) – W(n – 1) is the amount of weight adjustment at last iteration. 0 ≤ 𝛾 ≤ 1 is the momentum factor. When 𝛾 = 0, eq. (2.70) turns into the ordinary BP algorithm. The momentum term method proposed in the literature [98] is a deformation of eq. (2.70): W(n + 1) = W(n) + 𝛾 [W(n) – W(n – 1)] – , (1 – 𝛾)

𝜕E(n) 𝜕W(n)

(2.71)

In this equation, when the momentum factor is zero, the weight change is decided by the gradient descent method. When the value of the momentum factor is 1, the new weights change amount is that of the last iteration, and the part of changing amount produced by the gradient method is ignored. When the momentum factor is between 0 and 1, the weight adjustment changes toward the average direction of the error surface bottom. In the momentum term method proposed in literature [96], the momentum factor not only affects the partial derivatives of the previous moment but also affects the partial derivative of the present moment: W(n + 1) = W(n) – , (1 – 𝛾)

𝜕E(n) 𝜕E(n) –𝛾 𝜕W(n) 𝜕W(n – 1)

(2.72)

The double momentum terms method. In the double momentum terms method, the weight adjustment process is affected by two momentum factors. Its characteristic is the better effect. But the values of two momentum factors need to be determined. The double momentum term method proposed in literature [107] is shown as follows: W(n + 1) = W(n) – , (1 – 𝛾1 )

𝜕E(n) 𝜕E(n) – 𝛾1 + 𝛾2 [W(n) – W(n – 1)] 𝜕W(n) 𝜕W(n – 1)

(2.73)

where 𝛾1 , 𝛾2 are momentum factors. The third term can be regarded as the learning experience of the last iteration. When the third term has the same gradient direction with the current iteration, it plays an accelerating role. Otherwise it is equivalent to a damping term. The damping term can reduce the oscillation trend during learning and improve network stability. The fourth term plays the role of smoothing learning process and skipping the local minima. The performance of the network can be greatly improved by using the two terms simultaneously. When the momentum factor 𝛾1 = 0, eq. (2.73) is transformed into eq. (2.63). The two momentum terms method given by literature [108] is shown below: W(n + 1) = W(n) + 𝛾1 BW(n) + 𝛾2 BW(n – 1) – , (1 – 𝛾1 ) (1 – 𝛾2 )

𝜕E(n) 𝜕W(n)

= W(n) + 𝛾1 [W(n) – W(n – 1)] + 𝛾2 [W(n – 1) – W (n – 2)] – , (1 – 𝛾1 ) (1 – 𝛾2 )

𝜕E(n) 𝜕W(n)

(2.74)

2.4 The learning method of neural network blind equalization algorithm

41

The essence of the algorithm is to transmit the weights changing influence of the last two iterations to the changing weight of the current iteration by two momentum factors. When 𝛾1 = 0 and 𝛾2 = 0, the algorithm is the traditional BP algorithm. When 𝛾1 = 1 and 𝛾2 = 1, the increment of the new weight is turned into the sum of weight increment of the last two iterations. After adding the two items, the weight is urged to change toward the average direction of the error surface bottom. The method is helpful for avoiding convergence getting into a local minimum point. Meanwhile, the method greatly improves the speed of convergence. 2.4.2.4 The BP algorithm with numerical optimization technique The BP learning process of neural network is essentially a numerical optimization problem. It can be improved by using numerical optimization techniques, such as the Newton method, conjugate gradient method, and Levenberg–Marquardt algorithm. The detailed content can be found in literatures [88, 109–111]. 2.4.2.5 The BP algorithm based on modern optimization theory The BP algorithms based on modern optimization theory are to combine neural network with the genetic algorithm (GA), colony algorithm (CA), or simulated annealing (SA) algorithm. By using these combinations, the defects of BP algorithm are overcome, and the global convergence performance of the algorithm is improved. The BP algorithm optimized by GA The GA is a global probability search algorithm based on biological evolutionary mechanisms of natural selection and genetic variance, an so on. It has the characteristics of strong robustness, randomness, global, as well as suitable for parallel processing. From the macro-perspective, the global search ability of the GA is very strong. From the micro-perspective, its local search ability is poor. Therefore, when using GA to optimize BP algorithm, the first thing is to find good initial weights of a neural network by utilizing the global search ability of the GA. And then train the neural network by BP algorithm from the initial weights. Finally, find the best connection weights of the neural network. In this way, not only the local minima are avoided but also the convergence speed is accelerated [112–114]. The BP algorithm optimized by SA The SA algorithm is a method of solving the multi-extremum global optimization. It can effectively get rid of the local minima, and can not only “down the mountain,” but also “mountain climbing.” Therefore, to take SA algorithm as a weight factor and to add it to the BP algorithm can help BP algorithm not only go through the “flat area”

42

2 The Fundamental Theory of Neural Network Blind Equalization Algorithm

of the error curve quickly but also jump out the “local little bottom.” In this way, the training speed and the error precision of the BP algorithm are improved [115–117]. The BP algorithm optimized by CA The CA is a novel simulated evolutionary algorithm. It has some characteristics of positive feedback, distributed computing, heuristic convergence, strong robustness, fast convergence speed, and easy obtaining of the global optimal solution. The convergence speed can be accelerated and the local minimum can be avoided by using CA to train the weight of neural network [118, 119]. The method of optimizing the fuzzy neural network by the particle swarm optimizers (PSO) is proposed in literature [120]. The position vector of PSO is corresponded to the weight vector of the fuzzy neural network, and the adaptive function is corresponded to the objective function (i. e., cost function). And then the aim of optimizing the fuzzy neural network is obtained by the evolution of PSO. The BP algorithm optimized by immune algorithm The immune algorithm (IA) is a global optimization algorithm. It draws lessons from the unique information processing mechanism of the biological immune system and can achieve the self-regulation similar to biological immune system and generate different antibody functions. For neural networks, it can realize the weight learning. The weights of neural network designed by IA can achieve good results in terms of global convergence and convergence speed [121–123]. And the computer simulations validate its effectiveness. In addition, the BP algorithm is optimized by the fish swarm algorithm [124, 125], the support vector machine [126, 127], bee CA [128, 129], and so on.

2.5 The evaluation index of the neural network blind equalization algorithm There are mainly three evaluation indexes of neural network blind equalization algorithm. The first one is the convergence speed. It decides whether the algorithm can be used in real-time system or not. The second one is decides whether the algorithm can obtain the optimal solution. That is, whether the cost function is convex or not. The last one is the size of the steady residual error after convergence. Under general circumstances, the related parameters characterizing the blind equalizer performance are basically same as that of the adaptive equalizer. They are the convergence speed, computational complexity, error characteristics, steady residual error, ability of tracking variable channel, and anti-interference ability.

2.5 The evaluation index of the neural network blind equalization algorithm

43

2.5.1 Convergence speed The convergence speed refers to, for a constant input, the iteration number of the algorithm when it gets convergence. The convergence refers to that the iterative result of the algorithm is sufficiently close to the optimal solution. The algorithm with fast convergence speed not only can quickly adapt a stable environment, but also can timely track characteristic change of nonstable environment.

2.5.2 The computational complexity The computational complexity refers to the operation number required to complete the iterative algorithm. Many equalization algorithms are faster at convergence speed, but their applications are subject to a certain restriction because their operational burden is too large. The requirement of these algorithms for hardware and software is very high. Therefore, it is of great significance to reduce the computational complexity under the premise that the requirement of BER is satisfied.

2.5.3 The bit error characteristics Under the premise that the computational complexity of the algorithm is not increased and the convergence speed meets the requirements, to reduce the error rate of equalizer can effectively improve the communication quality. The BER is defined as the ratio of the error symbol number to the total transmission symbol number in a fairly long period.

2.5.4 The ability of tracking the time-varying channel The ability of tracking the time-varying channel is mainly reflected by the problem that whether the algorithm can converge and keep stability in the case of the time variant occurring in channel. The tracking ability of the algorithm is restricted by the complexity of the algorithm and the realization method.

2.5.5 The ability of anti interference The ability of anti-interference refers to the resistance ability of the algorithm to the noise superposed on the channel, especially to the sudden strong noise. When subject to a noise interference, the convergence performance of algorithms with a poor antiinterference ability would be poor, and even cannot converge.

44

2 The Fundamental Theory of Neural Network Blind Equalization Algorithm

2.5.6 The convexity of the cost function The convexity of the cost function is the key that whether the algorithm can converge to the global optimum or not. By taking equalizer tap coefficients as the parameters to draw the cost function figure, the performance surface figure can be obtained. Let the number of the equalizer taps be L, then the performance surface figure is (L+1) dimensional. In order to facilitate analysis, in ideal channel, the relationship between the cost function of LMS and two orthogonal main tap coefficients is given in Figure 2.8. And the relationship between the cost function of Stop-and-Go and two orthogonal main tap coefficients is given in Figure 2.9. In Figures 2.8 and 2.9, the vertical coordinate is the logarithm of the cost function, and the horizontal coordinate is the main tap coefficients. From Figure 2.8, the cost function of the LMS algorithm has only a trough, so it is a convex function and has only a minimum value point. According to Figure 2.9, the cost function of Stop-and-Go algorithm is a nonconvex function, because its error-curved surface has local minima except the global minimum value. These local minima make Stop-and-Go algorithm possible to fall into local minima and result in error convergence. There are two reasons for the existence of local minimum of equalizer cost function. One reason is that the length of equalizer is limited so that blind equalization

2 0 –2 –4 –6 –8 5

5

0

0

Figure 2.8: The picture of cost function of LMS.

–5 –5

0

–5 –10 5 0 –5 –4

–2

0

2

4 Figure 2.9: The picture of cost function of Stop-and-Go.

45

2.5 The evaluation index of the neural network blind equalization algorithm

algorithm converges to a local minimum. For the equalizer with an integer number of taps, the case will occur. So the local minima values are called “unavoidable local minima.” The other is that the cost function selected is not good. In this case, even if the equalizer is of infinite length, there will always exist local minima. So the local minima values are called “inherent local minima.” This kind of minima can be avoided by selecting a good cost function [130]. All the blind equalization algorithms realized by the finite impulse response filter are unable to avoid “unavoidable local minima.” And all Stop-and-Go algorithms have “inherent local minima,” and only Godard algorithm does not have “inherent local minimum” [86]. The “unavoidable local minima” can be decreased by choosing suitable initial values. That is, the selection of initial value is related to whether it converges to the global optimal solution or not. As shown in Figure 2.10, the form of the function is as follows: J(n) = 3 (1 – x)2 e[–x

2

–(y+1)2 ]

2 2 2 1 1 – 10 ( x – x3 – y5 ) e[–x –y ] – e[–(x+1) ] 5 3

(2.75)

According to Figure 2.10, if the initial value is w2󸀠 , then it is very easy to find the global optimal solution from w2󸀠 . If the initial value is w2 , then it can only converge to w1 . The w1 is obviously a local minimum point. Therefore, in practical applications, the high-order cepstrum [131–133] and GA [134, 135] are often used for obtaining better initial weights.

2.5.7 The steady-state residual error [136] The steady-state residual error is defined as the residual error after the blind equalizer gets convergence: ̃ – x(n) e(n) = x(n)

(2.76)

Because the x(n) cannot be known generally, it is often replaced by the decision ̂ sequence x(n), i. e. (2.77)

Cost function

̃ – x(n) ̂ e(n) = x(n)

w 2′

w 1′

w1 w2 Weight distributions

Figure 2.10: Schematic diagram of cost function.

46

2 The Fundamental Theory of Neural Network Blind Equalization Algorithm

Its MSE can be obtained as follows: 2

2

̃ – x(n)) ̂ ̂ E[e2 (n)] = E[(x(n) ] = E[(WT (n)Y(n) – x(n)) ] T ̂ (n)] W(n) + E[x̂ 2 (n)] = WT (n)E[Y(n)YT (n)] W(n) – 2E[x(n)Y

(2.78)

= WT (n)RW(n) – 2PW(n) + E[x̂ 2 (n)] In eq. (2.78), R = E[Y(n)YT (n)] is the autocorrelation matrix of the equalizer input seT ̂ (n)] is the correlation matrix. The MSE is generally composed quence and P = E[x(n)Y of two parts, such as theoretical error and excess MSE. 2.5.7.1 The theoretical error The theoretical error refers to the error produced by adopting finite length transversal filter to replace the infinite length transversal filter. For the infinite filter, if the noise superimposed in channel is ignored, then +∞

+∞

+∞

̃ x(n) = WT (n)Y(n) = ∑ wi (n)y(n – i) = ∑ ∑ wi (n)hj (n)x(n – i – j) i=–∞ +∞

i=–∞ j=–∞

+∞

(2.79)

+∞

= ∑ x(n – l) ∑ wi (n)hl–i (n) = ∑ $l (n)x(n – l) = x(n) l=–∞

i=–∞

l=–∞

and required +∞

∑ wi (n)hl–i (n) = $l (n)

(2.80)

i=–∞

The ideal inverse filter given by eq. (2.80) should have an infinite number of taps. However, the ideal inverse filter is not realistic in practical application and usually replaced by finite length filters. For a finite-length transversal filter, let its length be L, then the output of the filter is L–1

̃ x(n) = ∑ wi (n)y(n – i) i=0 +∞

–1

+∞

= ∑ wi (n)y(n – i) – ∑ wi (n)y(n – i) – ∑ wi (n)y(n – i) i=–∞

i=–∞

(2.81)

i=L

= x(n) – v1 (n) – v2 (n) –1

+∞

i=–∞

i=L

In eq. (2.81), v1 (n) = ∑ wi (n)y(n – i) and v2 (n) = ∑ wi (n)y(n – i) are called as convolution errors, i. e., the theoretical error. They are the residual intersymbol interference for using truncated filter.

2.5 The evaluation index of the neural network blind equalization algorithm

47

In practical applications, in order to reduce the theoretical error, the filter with enough number of taps should be adopted as far as possible. It can make the filter impulse response to approximate the inverse of the channel impulse response. 2.5.7.2 The excess MSE The excess MSE refers to the error that the steady-state weight vector randomly fluctuates in the vicinity of the optimal weight vector after getting convergence. And ̃ x(n) = ∑ sk (n)x(n – i) = sD (n)x (n – D) + ∑ sk (n)x(n – i)

(2.82)

k=D ̸

k

where s(n) = h(n) ∗ w(n) is the response function of the combined system of the transmission channel and the equalizer. In eq. (2.82), the first item is the desired output signal and the second item is the intersymbol interference. According to the zero-forcing criterion of blind equalization, it can be known that when eq. (2.83) is true, the blind equalizer converges to the optimal solution. At this time, the intersymbol interference is zero: sD (n) = e j> ,

sk (n) = 0 (k ≠ D)

(2.83)

But in practical applications, it is possible that the tap coefficient is always perturbation in the vicinity of the optimum when the equalizer has converged to the extreme point. Thus, the excess MSE occurs. The reason for this problem is the influence of the gradient noise and can be expressed as %B = ,trR%min

(2.84)

where trR is the trace of the autocorrelation matrix of the equalizer input sequence and %min = E[x̂ 2 (n)] – PT R–1 P is the LMS error of the LMS algorithm. 2.5.7.3 Factors affecting the steady-state residual error Since the perturbation process of the tap coefficient is a very complex nonstationary process, it is difficult to quantitatively analyze. Therefore, only qualitative analysis can be carried out. Tap coefficient In practical applications, to reduce the theoretical error, a filter with a sufficient number of taps should be used as far as possible. A filter with a sufficient number of taps can make its impulse response to approach the inverse of channel impulse response.

48

2 The Fundamental Theory of Neural Network Blind Equalization Algorithm

But the increase in the tap coefficient number is bound to increase the algorithm complexity. At the same time, as tap coefficient number increases, the trR raises. It leads to the increase of excess MSE. Therefore, to increase the tap coefficients, the number of the transversal filter does not guarantee that the steady-state residual error can be reduced. For the specific input signal, there exists an optimal tap coefficient. The optimal tap coefficient makes the steady-state residual error to minimize. Regardless of increase or decrease in the tap coefficient number, it is possible to make the steady-state residual error increase. The conditions for the convergence curve shown in Figure 2.11 are as follows: 4PAM is used as the input signal; 11- and 31-order transversal filters are adopted, respectively; the step size is 0.01; the input signal-to-noise ratio is 20 dB; and the typical telephone channel is used. The conditions for the convergence curve shown in Figure 2.12 are as follows: 8PAM is used as the input signal; 11- and 31-order transversal filters are adopted, respectively; the step size is 0.0.01; the input signal-to-noise ratio is 20 dB; and the typical telephone channel is used. The transfer function [137] is H1 (z) = 0.005 + 0.009z–1 – 0.024z–2 + 0.854z–3 – 0.218z–4 + 0.049z–5 – 0.016z–6

(2.85)

According to Figure 2.11 and 2.12, the less the tap coefficients of transversal filter are, the fast the convergence speed is, but the larger the steady residual error is. The more the tap coefficients are, the smaller the steady residual error is, but the slower the

0.08 1 L = 11

MSE

0.06

2 L = 31

0.04 2 0.02 1 0

0

2,000

4,000

6,000

8,000

Iterative number

0.10

1 L = 11

0.08 MSE

Figure 2.11: The convergence curves of 4PAM signal with different filter orders.

2 L = 31

0.06 2

0.04

1

0.02 0

0

0.5

1.5 1 Iterative number

2

× 104

Figure 2.12: The convergence curves of 8PAM signal with different filter orders.

2.5 The evaluation index of the neural network blind equalization algorithm

49

convergence speed is. However, the tap coefficient number should not be too large. According to the simulation results, when the tap coefficient number increases to more than 50, its steady residual error is larger than the steady residual error of the 11 tap coefficients. Therefore, for different applications, the tap coefficient number has an optimal value. Step-size factor According to eqs. (2.11) and (2.84), the step-size factor plays a very important role in the convergence process. If a large step-size factor is adopted, the adjustment magnitude of tap coefficients at each iterative is large. From the convergence performance view, the large step-size factor can make the convergence speed and tracking speed of algorithm fast. However, when the tap coefficient of the equalizer is close to the optimal value, the tap coefficient will jitter in a larger range. And the tap coefficient cannot further converge. So a larger excess MSE occurs. Conversely, if a small stepsize factor is adopted, the adjustment magnitude of tap coefficients at each iterative is small, and the convergence speed and tracking speed of algorithm are slow. But when the tap coefficient of the equalizer is close to the optimal value, the tap coefficients will be jitter in a smaller range. Thus, there will be a smaller excess MSE. The conditions for the convergence curve shown in Figure 2.13 are as follows: 4 PAM is used as the input signal; 11-order transversal filters are adopted; the step sizes are 0.015 and 0.005, respectively; the input signal-to-noise ratio is 20 dB; and the typical telephone channel is used. And its transfer function is shown in eq. (2.47). The conditions for the convergence curve shown in Figure 2.11 are as follows: 8 PAM 0.10 0.08 MSE

1 0.06

0.02 0

2 step = 0.005

2

0.04

step = 0.015

1 0

1,000

2,000

3,000

4,000

Iterative number

Figure 2.13: The convergence curve of 4PAM signal with different step size factor.

0.10 0.08 MSE

1 step = 0.002 0.06 2

2 step = 0.008

0.04 1

0.02 0

0

2,000 4,000 6,000 8,000 10,000 Iterative number

Figure 2.14: The convergence curve of 8PAM signal with different step size factor.

50

2 The Fundamental Theory of Neural Network Blind Equalization Algorithm

is used as the input signal; 11-order transversal filters are adopted; the step sizes are 0.002 and 0.0008, respectively; the input signal-to-noise ratio is 20 dB; and the typical telephone channel is used. And its transfer function is shown in eq. (2.47). It can be seen from the above two figures that the influences of the step-size factor on the convergence speed and the residual error of the steady state are contradictory. In practical application, the step-size factor should be selected based on the requirements. Types of transmitted signal The relevant research shows that the excess MSE is related to the statistical characteristics of the input sequence. Under the same conditions, different input sequences can lead to different excess MSEs [138, 139].

2.6 Summary In the chapter, the basic concepts, common structural forms, and algorithms of the blind equalization are introduced in detail. In addition, the zero-forcing criterion, kurtosis criterion, and normalized criterion are analyzed. The three combination methods of neural network and blind equalization algorithm are described, namely to construct neural network as a filter, as a controller, and as a classifier, and then apply them to the blind equalization algorithm. Among them, to construct neural network as the filter to replace the transversal filter of the traditional blind equalization algorithm is the most common one. On this basis, the shortcomings of BP algorithm are analyzed. Furthermore, various improved BP algorithms, such as variable learning rate BP algorithm and momentum BP algorithm, the BP algorithm optimized by GA, the BP algorithm optimized by CA, and the BP algorithm optimized by SA optimization are summarized. They lay the foundation for further studies. In the chapter, the evaluation indexes of the blind equalization algorithm are also analyzed. Two important parameters, namely the convexity of the cost function and the steady-state residual error, are mainly studied. The convexity of the cost function directly affects the global convergence of the algorithm. Its core is the selection of function form and initial weights. The steady residual error directly affects the decision and the BER. It can be reduced by the determination of the filter length and the selection of the step factor.

3 Research of Blind Equalization Algorithms Based on FFNN Abstract: In this chapter, the basic principle of feed-forward neural network (FFNN) is analyzed. First, blind equalization algorithms based on the three-layer FFNN, fourlayer FFNN, and five-layer FFNN are studied. Then iteration formulas of algorithms are derived. Computer simulations are done. The theoretical analysis and experimental results verify that with the increase of layer number, the algorithm convergence rate becomes slow and the computational complexity increases. But the steady residual error decreases after the algorithm converged, that is, the approximation ability enhances. Second, the improved BP algorithm is applied to the blind equalization algorithm, then blind equalization algorithms based on the momentum term, time-varying momentum term, and variable step size are studied. When these new algorithms are compared with the blind equalization algorithm based on the traditional BP algorithm, the performances of the new algorithms can be improved.

3.1 Basic principles of FFNN 3.1.1 Concept of FFNN FFNN is one of the simplest neural networks. The neurons are arranged hierarchically, and each neuron is only connected with the neurons of the previous layer. It receives outputs of the previous layer and then outputs to the next layer, there is no feedback between layers. At present, FFNN is one of the most widely used and rapidly developed artificial neural networks. Research work began in the 1960s, and the current theoretical researches and practical applications have reached a very high level.

3.1.2 Structure of FFNN A sort of one-way multilayer structure is used in FFNN, which is shown in Figure 3.1. Each layer contains some neurons, and the neurons in the same layer are not connected to each other, and the transfer of information between the layers is only along one direction. The first layer is the input layer, the last layer is the output layer, and the middle layer is the hidden layer. The hidden layer can be a single-layer or multilayer. In the structure of FFNN, the input vector of the network is Y(n) = T

[ y1 (n) y2 (n) ⋅ ⋅ ⋅ yI (n) ] . The weights between layers are wij (n), wjp (n), and wpk (n). The T ̃ output vector is X(n) = [ x̃ (n) x̃ (n) ⋅ ⋅ ⋅ x̃ (n) ] . 1

DOI 10.1515/9783110450293-003

2

K

52

3 Research of Blind Equalization Algorithms Based on FFNN

wjp(n)

wij(n)

wpk(n)

y1 (n)

~ x 1(n)

y2 (n)

~ x 2(n)

yI (n)

~ x K (n) Input layer I

The first hidden layer J

The second hidden layer P

Output layer K

Figure 3.1: Structure of FFNN.

3.1.3 Characteristics of FFNN FFNN has the simple structure and the wide application. It can approximate any continuous function and square integrable function by any precision, and it can be used to accurately realize arbitrary finite training sample set. From the system point of view, the feed-forward network is a kind of static nonlinear mapping, which can obtain the complex nonlinear processing ability through the complex mapping of the simple nonlinear processing unit. From the computation point of view, the feed-forward network lacks abundant dynamic behavior. Most of feed-forward networks are learning networks, and the classification ability and pattern recognition ability of FFNN are generally better than those of feedback network.

3.2 Blind equalization algorithm based on the three-layer FFNN 3.2.1 Model of the three-layer FFNN The model of the three-layer FFNN is shown in Figure 3.2. It is composed of input layer, output layer, and a hidden layer. Suppose the connection weight between the input layer and the hidden layer is wij (n), where i is the neuron of the input layer (i = 0, 1, . . . , m), and j is the neuron of the hidden layer (j = 0, 1, . . . , k). The connection weight between the hidden layer and the output layer is wj (n). The input of the input layer is Y(n) = T

[ y(n) y(n – 1) ⋅ ⋅ ⋅ y (n – m) ] . The input of the hidden layer is uj (n) and the output is Ij (n), The output of the output layer is v(n). The output of the whole neural network is ̃ x(n). So the state functions of the three-layer FFNN can be expressed as

3.2 Blind equalization algorithm based on the three-layer FFNN

y(n)

53

w00(n) w0(n)

y(n−1)

f (·)

w1(n)

y(n−m+1)

y(n−m)

~ x(n)

wk(n)

wmk(n)

Input layer

Hidden layer

Out layer

Figure 3.2: Structure of the three-layer FFNN. m

uj (n) = ∑ wij (n)y(n – i)

(3.1)

i=0

Ij (n) = f [uj (n)]

(3.2)

k

v(n) = ∑ wj (n)Ij (n)

(3.3)

j=0

̃ x(n) = f [v(n)]

(3.4)

where f (⋅) is the transfer function between the input and output of the hidden layer, at the same time it is also the transfer function between the input and output of the output layer. f (⋅) is a nonlinear function. Two keys of blind equalization based on FFNN are how to determine the connection weights between neurons and how to choice transfer function. The connection weights are usually obtained by using some learning algorithms or training the input-output of the system.

3.2.2 Real blind equalization algorithm based on the three-layer FFNN 3.2.2.1 Derivation of algorithm In the research work of the traditional FFNN, the threshold function, the sigmoid function, and the hyperbolic tangent function are generally selected as the transfer function. In this chapter, the hyperbolic tangent function is selected as the basis of the transfer function to design a new network transfer function. Because the

54

3 Research of Blind Equalization Algorithms Based on FFNN

characteristics of the hyperbolic tangent function are smooth, gradual, and monotonous, it is beneficial to distinguish the input sequence. The new transfer function is defined as f (x) = x + !

ex – e–x ex + e–x

(3.5)

where ! is the scale factor. If the signal has large amplitude interval, it should be assigned a large value. If the signal has small amplitude interval, it should be assigned a small value. This is beneficial to classify the output signal. According to the training methods of the traditional CMA [140] and FFNN, a new cost function is defined as J(n) =

2 1 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨2 – R2 ] [󵄨󵄨x(n) 2

(3.6)

̃ is the output sequence of the FFNN equalizer, and it is the real signal. The where x(n) 4 E[|x(n)| ] definition of R2 is the same as that of the traditional CMA, so R2 = 2 . E[|x(n)| ] According the steepest descent method, the new weights iterative formula of the blind equalization algorithm based on FFNN can be obtained: W(n + 1) = W(n) – ,∇ ∇=

(3.7)

̃ 𝜕x(n) 𝜕J(n) ̃ = 2 [x̃ 2 (n) – R2 ] x(n) 𝜕W(n) 𝜕W(n)

(3.8)

The three-layer FFNN contains the hidden layer and the output layer, therefore the iteration functions of weights are different.

Weight iterative formula of the output layer The connection weight between the output layer and the hidden layer is wj (n), so ̃ 𝜕x(n) = f 󸀠 [v(n)]Ij (n) 𝜕wj (n)

(3.9)

Plugging eqs. (3.9) and (3.8) into eq. (3.7), the weight iterative formula of the output layer is obtained. wj (n + 1) = wj (n) – ,1 k(n)Ij (n)

(3.10)

where ,1 is the iterative step-size factor of the output layer, and k(n) 󸀠 ̃ 2 [x̃ 2 (n) – R2 ] x(n)f [v(n)].

=

3.2 Blind equalization algorithm based on the three-layer FFNN

55

Weight iterative formula of the hidden layer The connection weight between the hidden layer and the input layer is wij (n), so ̃ 𝜕x(n) 𝜕v(n) = f 󸀠 [v(n)] = wj (n)f 󸀠 [v(n)] f 󸀠 [uj (n)] y(n – i) 𝜕wij (n) 𝜕wij (n)

(3.11)

Plugging eqs. (3.11) and (3.8) into eq. (3.7), the weight iterative formula of the hidden layer is obtained. wij (n + 1) = wij (n) – ,2 kj (n)y(n – i)

(3.12)

where ,2 is the iterative step-size factor of the hidden layer, and kj (n) = f 󸀠 [uj (n)] wj (n)k(n). According to the above weight iterative formulas, the neural network can be used to blind equalize. When the number of network layer increases (i. e., more than one hidden layer), the iterative formula (3.12) of the hidden layer is still available [141]. 3.2.2.2 Computer simulation results The input signal is 8PAM. The signal-to-noise ratio (SNR) is 20 dB and the filter order is 11. The simulation channels adopt the typical digital telephone channel and the ordinary channel. The z-transform of the typical digital telephone channel is shown in eq. (2.84). The z-transform of the ordinary channel is [142] H2 (z) = 1 + 0.5z–1 + 0.25z–2 + 0.125z–3

(3.13)

The convergence curves of 8PAM through two channels using the real blind equalization algorithm based on the three-layer FFNN (TFFNNR) and the blind equalization algorithm proposed by Cheolwoo You [141] (MFFNN) are shown in Figures 3.3 and 3.4, where ,1 = ,2 = 0.001. Simulation results have shown that the three-layer FFNN algorithm has faster convergence speed than MFFNN algorithm, but after convergence steady residual error is slightly greater than the MFFNN algorithm. 0.10

MSE

0.08

1

2

1

TFFNNR

2

MFFNN

0.06 0.04 0.02 0

0

1,000 2,000 3,000 4,000 5,000 Iterative number

Figure 3.3: The convergence curves through the typical telephone channel.

56

3 Research of Blind Equalization Algorithms Based on FFNN

0.10

MSE

0.08

2

0.06

1

TFFNNR

2

MFFNN

1

0.04 0.02 0

0

2,000 4,000 6,000 8,000 10,000 Iterative number

Figure 3.4: The convergence curves through the ordinary channel.

3.2.3 Complex blind equalization algorithm based on the three-layer FFNN 3.2.3.1 Derivation of algorithm Two problems need to be solved when the algorithm is extended to the complex range. First, a suitable complex-valued function must be designed as the network transfer function. Second, when the complex signals through the complex channel, the phase deflection will inevitably be produced. Therefore, the algorithm should not only correct the amplitude distortion of signals but also recover the phase information of signals. The complex is composed of a real part and an imaginary part, so the network structure needs to change appropriately. Signal is divided into two parts before entering the nonlinear transfer function. The real part and the imaginary part are transmitted by two identical channels respectively. At the output side of the nonlinear transfer function, the real part and the imaginary part are re-synthesized a complex signal. The network structure is shown in Figure 3.5 [141]. The cost function under the complex condition is still defined by the formula (3.6). Because the nonlinear transfer function exists in the neural network, the weight form of the network should be divided into the real part and the imaginary part: W(n) = WR (n) + jWI (n)

󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 𝜕J(n) 󵄨 ̃ 󵄨󵄨2 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) = 2 [󵄨󵄨󵄨x(n) +j 󵄨 ] ∇= 󵄨󵄨 – R2 ] 󵄨󵄨󵄨x(n) 󵄨󵄨 [ 𝜕W(n) 𝜕WR (n) 𝜕WI (n)

(3.14) (3.15)

fR (yR , yI) Y(n)

+ fI (yR , yI)

j

Figure 3.5: The complex processing unit model in neural network.

57

3.2 Blind equalization algorithm based on the three-layer FFNN

At the same time, in order to facilitate formula derivation, the network signal is also written as the complex form: y(n – i) = yR (n – i) + jyI (n – i)

(3.16)

wij (n) = wij,R (n) + jwij,I (n)

(3.17)

uj (n) = ∑ wij (n)y(n – i) i

= ∑ [wij,R (n)yR (n – i) – wij,I (n)yI (n – i)]

(3.18)

i

+ j ∑ [wij,R (n)yI (n – i) + wij,I (n)yR (n – i)] i

Ij (n) = f [uj,R (n)] + jf [uj,I (n)] wj (n) = wj,R (n) + jwj,I (n)

(3.19) (3.20)

v(n) = ∑ wj (n)Ij (n) = ∑ [wj,R (n)Ij,R (n) – wj,I (n)Ij,I (n)] j

j

+ j ∑ [wj,R (n)Ij,I (n) + wj,I (n)Ij,R (n)]

(3.21)

j

̃ x(n) = f [vR (n)] + jf [vI (n)]

(3.22)

Similar to the real situation, due to the complex three-layer FFNN has the hidden layer and the output layer, so the iterative formulas of weights are different. Weight iterative formula of the output layer The connection weight between the output layer and the hidden layer is wj (n), so 󵄨 ̃ 󵄨󵄨 ̃ x̃ ∗ (n)] 𝜕 󵄨󵄨󵄨x(n) 󵄨󵄨 1 𝜕 [x(n) = 󵄨󵄨 󵄨 ̃ 󵄨󵄨󵄨 𝜕wj,R (n) 𝜕wj,R (n) 2 󵄨󵄨x(n) =

2 2 1 𝜕 {f [vR (n)] + f [vI (n)]} 󵄨 ̃ 󵄨󵄨 𝜕wj,R (n) 2 󵄨󵄨󵄨x(n) 󵄨󵄨

(3.23)

1 = 󵄨󵄨 {f [vR (n)] f 󸀠 [vR (n)] Ij,R (n) + f [vI (n)] f 󸀠 [vI (n)] Ij,I (n)} ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n) 󵄨 ̃ 󵄨󵄨 ̃ x̃ ∗ (n)] 𝜕 󵄨󵄨󵄨x(n) 󵄨󵄨 1 𝜕[x(n) = 󵄨󵄨 󵄨 ̃ 󵄨󵄨󵄨 𝜕wj,I (n) 𝜕wj,I (n) 2 󵄨󵄨x(n) =

2 2 1 𝜕 {f [vR (n)] + f [vI (n)]} 󵄨 ̃ 󵄨󵄨 𝜕wj,I (n) 2 󵄨󵄨󵄨x(n) 󵄨󵄨

1 = 󵄨󵄨 {–f [vR (n)] f 󸀠 [vR (n)] Ij,I (n) + f [vI (n)] f 󸀠 [vI (n)] Ij,R (n)} ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n)

(3.24)

58

3 Research of Blind Equalization Algorithms Based on FFNN

According to eqs. (3.23) and (3.24), we obtain 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨x(n) 𝜕 󵄨󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 1 +j 󵄨 = {f [vR (n)] f 󸀠 [vR (n)] Ij,R (n) + f [vI (n)] f 󸀠 [vI (n)] Ij,I (n)} ̃ 󵄨󵄨󵄨󵄨 𝜕wj,R (n) 𝜕wj,I (n) 󵄨󵄨󵄨󵄨x(n) 1 󸀠 󸀠 + j 󵄨󵄨 󵄨󵄨 {f [vI (n)] f [vI (n)] Ij,R (n) – f [vR (n)] f [vR (n)] Ij,I (n)} ̃ 󵄨󵄨x(n)󵄨󵄨 1 = 󵄨󵄨 {f [vR (n)] f 󸀠 [vR (n)] + jf [vI (n)] f 󸀠 [vI (n)]} Ij∗ (n) ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n)

(3.25)

Plugging eq. (3.25) into eqs. (3.14) and (3.15), the weight iterative formula of the output layer is obtained: wj (n + 1) = wj (n) – 2,1 k(n)Ij∗ (n)

(3.26)

󵄨 ̃ 󵄨󵄨2 where k(n) = [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2 ] {f [vR (n)] f 󸀠 [vR (n)] + jf [vI (n)] f 󸀠 [vI (n)]}, and ,1 is the iterative step-size factor of the output layer. Weight iterative formula of the hidden layer The connection weight between the hidden layer and the input layer is wij (n), so wij (n + 1) = wij (n) – 2,2 kj (n)y∗ (n – i)

(3.27)

where ,2 is the iterative step-size factor of the hidden layer, and 󵄨 ̃ 󵄨󵄨2 󸀠 󸀠 󸀠 ∗ kj (n) = 2 [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2 ] f [uj,R (n)] Re {[f [vR (n)] f [vR (n)] + jf [vI (n)] f [vI (n)]] wj (n)} 󵄨 ̃ 󵄨󵄨2 󸀠 󸀠 󸀠 ∗ + j2 [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2 ] f [uj,I (n)] Im {[f [vR (n)] f [vR (n)] + jf [vI (n)] f [vI (n)]] wj (n)} (3.28) According to the above iterative formulas, the FFNN blind equalization algorithm can be extended to the complex domain. Similar to the real situation, when the number of the hidden layer increases, the hidden layer iterative formula (3.27) is still available. The specific derivation process is shown in Appendix A. 3.2.3.2 Selection of the transfer function When the algorithm is extended to the complex domain, the transfer function of the network must have a series of special features. G.M. Georgiou et al. [[143] had analyzed that the complex-valued transfer function should have the general form and conditions.

3.2 Blind equalization algorithm based on the three-layer FFNN

59

The complex-valued transfer function should have the general formula F(x) = . (xR , xI ) + j& (xR , xI )

(3.29)

where . (⋅) and =(⋅) are real-valued functions. Complex-valued transfer function must have the following conditions: (1) In the complex domain, for the real part and imaginary part of the signal, the transfer function F(⋅) is the nonlinear function. (2) In order to ensure the system stability, F(⋅) should be a bounded function in a given range and there is no singularity in this range (the discontinuous point). In addition, if F(⋅) is nonanalytic, F(⋅) is continuous within the given range. (3) F(⋅) should have continuous one-order partial derivatives, that is, 𝜕. (xR , xI ) 𝜕. (xR , xI ) 𝜕= (xR , xI ) 𝜕= (xR , xI ) , , , 𝜕xR 𝜕xI 𝜕xR 𝜕xI

(4)

all exist. For complex signals, the real part and imaginary part of the transfer function should have the following relation. 𝜕. (xR , xI ) 𝜕= (xR , xI ) 𝜕= (xR , xI ) 𝜕. (xR , xI ) ≠ 𝜕xR 𝜕xI 𝜕xR 𝜕xI

(5)

(3.30)

Because the modulation method of QAM determines that the signal in the complex plane is symmetrical (i. e., constellation diagram distributes symmetrically), the output signals . (xR , xI ) and = (xR , xI ) should have the same dynamic range. In a general communication system, the no distortion complex signal is symmetrical about the x-axis and y-axis, and the interval between signals is equal.

According to the above limits, the transfer function has two choices. (1) In the real channel, the hyperbolic tangent function is used as the base of the transfer function. When the function is used as the complex-valued transfer function, the above five limit conditions are satisfied. So the hyperbolic tangent function can be used in the complex-valued blind equalization algorithm based on FFNN. However, it is found in the computer simulation that if the transmission channel is a real channel, the algorithm performance using the hyperbolic tangent function is better than that of the algorithm proposed by Cheolwoo You [141]. Because the signal phase information does not change, when the complex signal is transmitted through the real channel, the performance is similar to that of the real-valued system. However, when the hyperbolic tangent transfer function is used in the complex channel, the ability to correct the phase information is worse

60

(2)

3 Research of Blind Equalization Algorithms Based on FFNN

than the transfer function proposed by You Cheolwoo. Therefore, the hyperbolic tangent transfer function is not applicable in the complex-valued channel. There are five limit conditions for complex transfer function. The amplitude and phase of the QAM signals have two dimensions in the complex system. According to the characters above, Cheolwoo You had proposed a transfer function successfully applied to the complex multilayer FFNN blind equalization algorithm (MFFNN); the function form is as follows: f (x) = f (x̃R ) + jf (x̃I )

(3.31)

where f (x) = x + ! sin 0x, and it is a real-valued function. ! is a ratio factor and ! > 0. In the MFFNN algorithm simulation experiments, it is found that the value of the ratio factor ! has a great influence on the algorithm convergence performance. The larger the value is, the faster convergence rate the algorithm has, but after convergence, the Steady-state Residual Error (SRE) is larger. If the ! value is small, the SRE after convergence is small, but the convergence rate becomes slow. Because ! affects the algorithm convergence performance, the nonlinear part of the transfer function is modified in this chapter. In the beginning of the algorithm, the bigger value of ! is chosen to speed up the convergence rate. Along with the iterative process, the value of ! should be decreased, so that after convergence the SRE is reduced. The variable ! used in the algorithm not only accelerates convergence speed but also improves the convergence precision. In order to make the value of ! vary along with the algorithm convergence process, the mean square error (MSE) is used instead of !. The corrected transfer function is f (x) = x + E [e2 (n)] sin 0x

(3.32)

3.2.3.3 Computer simulation results x

–x

–e The transfer function is f (x) = x + ! eex +e –x The input signal is 16QAM. SNR is 20 dB. The simulation channels adopt the typical digital telephone channel and the ordinary channel. Their z-transforms are shown in eqs. (2.84) and (3.13). The convergence curves of 16QAM through the typical telephone channel (,1 = ,2 = 0.02) and the ordinary channel (,1 = ,2 = 0.015) are shown in Figure 3.6, where TFFNNC is the complex blind equalization algorithm based on the three-layer FFNN. MFFNN is the blind equalization algorithm proposed by Cheolwoo You [141]. It can be seen from the figures that the convergence rate of TFFNNC algorithm has been significantly improved, but after convergence, the steady-state residual errors of TFFNNC and MFFNN are similar. The bit error rate (BER) curves of two algorithms through two channels are shown in Figures 3.7 and 3.8. Simulation results have shown that in the same SNR, the

61

3.2 Blind equalization algorithm based on the three-layer FFNN

0.10 1

1

TFFNNC

0.10

2

MFFNN

0.08

0.06

MSE

MSE

0.08

2

0.04

0.06

TFFNNC MFFNN

2

0.04

0.02 0

1

1 2

0.02 0

(a)

1,000 2,000 3,000 4,000 5,000 Iterative number

0 (b)

0

1,000

2,000 3,000 4,000 Iterative number

5,000

Figure 3.6: The convergence curves through the typical telephone channel and the ordinary channel: (a) The convergence curves through the typical telephone channel and (b) The convergence curves through the ordinary channel.

TFFNNC MFFNN

log10 (BER)

–1.5 –1.6 –1.7 –1.8 –1.9 –2 –2.1 10

15 20 SNR (dB)

25

Figure 3.7: The BER curves through the typical telephone channel.

–1.4 TFFNNC MFFNN

log10 (BER)

–1.6 –1.8 –2 –2.2 –2.4 10

15

20

SNR (dB)

25

Figure 3.8: The BER curves through the ordinary channel.

TFFNNC algorithm has lower BER than the MFFNN algorithm. So the BER performance of TFFNNC algorithm is better than that of MFFNN algorithm. The 16QAM constellations after 5,000 iterations are shown in Figure 3.9. After the convergence of the MFFNN algorithm, the constellation diagram has a certain offset near the center point, but the convergence of the TFFNNC algorithm does not have the offset phenomenon, so TFFNNC algorithm overcomes the phenomenon of uneven constellation of MFFNN algorithm.

62

3 Research of Blind Equalization Algorithms Based on FFNN

2

2

1

1

0

0

–1

–1

–2 –2

–1

0

1

–2 –2

2

–1

0

1

2

(b)

(a)

Figure 3.9: The convergence constellations of MFFNN and TFFNNC: (a) the constellation of MFFNN algorithm and (b) the constellation of TFFNNC algorithm.

The transfer function is f (v) = x + E [e2 (n)] sin 0x The input signal is 16QAM. SNR is 20 dB. The simulation channel adopts the wireless digital communication channel. Its z-transform is shown as follows [141]: H3 (z) = (0.0410 + j0, 0109) z0 + (0.0495 + j0.0123) z–1 + (0.0672 + j0.0170) z–2 + (0.0919 + j0.0235) z–3 + (0.7920 + j0.1281) z–4 + (0.3960 + j0.0871) z–5 + (0.2715 + j0.0498) z–6 + (0.2291 + j0.0414) z–7 + (0.1287 + j0.0154) z–8 + (0.1032 + j0.0119) z–9 (3.33) The convergence curves of 16QAM through the wireless digital communication channel (,1 = ,2 = 0.004) are shown in Figure 3.10. It can be seen from the figure that the convergence performance of TFFNNC algorithm is improved, that is, the TFFNNC algorithm has faster convergence speed and lower SRE than the MFFNN algorithm. So the TFFNNC algorithm can improve the convergence performance. It verifies that the parameter selection of the transfer function is reasonable.

1

0.4 MSE

2

1

TFFNNC

2

MFFNN

0.2 0

0

1,000 2,000 3,000 4,000 5,000 Iterative number

Figure 3.10: The convergence curves through the wireless digital communication channel.

3.3 Blind equalization algorithm based on the multilayer FFNN

63

–1.3 TFFNNC MFFNN

log10 (BER)

–1.4 –1.5 –1.6 –1.7 10

15

20

25

SNR (dB)

Figure 3.11: The BER curves through the wireless digital communication channel.

4

4

2

2

0

0

–2

–2

–4 –4

–4 –2

0

2

4

–4

–2

0

2

4

Figure 3.12: The convergence constellations of MFFNN and TFFNNC: (a) the constellation of MFFNN algorithm and (b) the constellation of TFFNNC algorithm.

The BER curves after 5,000 iterations are shown in Figure 3.11. In the same SNR situation, the BER of the TFFNNC algorithm is obviously reduced. The 16QAM constellations after 5,000 iterations are shown in Figure 3.12. It can be seen from the figure that the TFFNNC algorithm is better in the concentration degree than the MFFNN algorithm.

3.3 Blind equalization algorithm based on the multilayer FFNN 3.3.1 Concept of the multilayer FFNN The multilayer FFNN is a kind of feed-forward neural with more than one hidden layer.

64

3 Research of Blind Equalization Algorithms Based on FFNN

In 1987, Robert Hecht Nielsen [144] has proved that a BP network with only one hidden layer can approximate any continuous function in a closed interval. This shows that a three-layer neural network can be used to complete the mapping of any n -dimension to m -dimension. It is proved that FFNN with only one hidden layer is a universal function approximator. This means that FFNN with one hidden layer has sufficient approximation ability, but does not mean it is the best approximator. Sometimes the FFNN with two hidden layers has better approximation ability. It also has been proved in literature [106] that if BP neural network maps a continuous function, a hidden layer is sufficient. If BP neural network learns the discontinuous function, two hidden layers are needed. Increasing the number of hidden layer can increase the processing ability of the artificial neural network, further reduce the error and improve the accuracy, but it will make the training complex, the number of training samples increase and the training time increase.

3.3.2 Blind equalization algorithm based on the four-layer FFNN 3.3.2.1 Model of the four-layer FFNN The four-layer FFNN is a neural network with two hidden layers, as shown in Figure 3.13. There is only the forward output in the neural network, and each layer is connected with the weights. The connection weight between the input layer and the first hidden layer is wij (n)(i = 1, 2, . . . , I; j = 1, 2, . . . , J), and the connection weight between the first hidden layer and the second hidden layer is wjk (n)(k = 1, 2, . . . , K), and the connection weight between the second hidden layer and the output layer is wk (n). The input of the neuron is u, the output of the neuron is v. uIi is the ith input neuron of the Ith layer; viJ is the ith output neuron of the Jth layer. The input of the ̃ whole neural network is y(n – i), and the output is x(n). So the state functions of the four-layer FFNN are defined as

wij(n)

wjk(n)

y(n − 1) wk(n) y(n − 2) ~ x(n) Output layer

y(n − I) Input layer I The first hidden layer J Figure 3.13: Structure of four-layer FFNN.

The second hidden layer K

3.3 Blind equalization algorithm based on the multilayer FFNN

65

uIi (n) = y(n – i)

(3.34)

viI (n) = uIi (n) = y(n – i)

(3.35)

I

I

i=1

i=1

uJj (n) = ∑ wij (n)viI (n) = ∑ wij (n)y(n – i)

(3.36)

I

vjJ (n) = f1 (uJj (n)) = f1 (∑ wij (n)y(n – i))

(3.37)

i=1 J

uKk (n) = ∑ wjk (n)vjJ (n)

(3.38)

j=1 J

vkK (n) = f2 (uKk (n)) = f2 (∑ wjk (n)vjJ (n))

(3.39)

j=1 K

u(n) = ∑ wk (n)vkK (n)

(3.40)

k=1 K

̃ v(n) = x(n) = f3 (u(n)) = f3 ( ∑ wk (n)vkK (n))

(3.41)

k=1

3.3.2.2 Derivation of algorithm The transfer function is chosen as follows [141]: f1 (x) = f2 (x) = f3 (x) = f (x) = x + ! sin 0x

!>0

(3.42)

In order to guarantee the monotone of the transfer function, it is required that the derivative of the function is always more than zero. So, f 󸀠 (x) = 1 + !0 cos 0x > 0

(3.43)

The transfer function is used to modify the signal value within a certain range. Then the signal value can be more close to the original signal. The nonlinear function ! sin 0x is the nonlinear correction term. ! sin 0x makes the signal value converged to original signal. The input–output characteristic curves of f (x) and its derivative characteristic curves are shown in Figures 3.14 and 3.15, respectively. In the original literature, 0 < ! < 1/0 ≈ 0.318. But in the actual simulation, ! is a fixed value within (0, 01 ). The experiments proved that ! affects the algorithm greatly. When ! is big, f (x) accelerates to adjust the output signal, so the algorithm convergence speed is fast, but SRE is large. On the contrary, when ! is small, f (x) slows down the adjustment speed, after convergence SRE is small, but the convergence speed is slow.

66

3 Research of Blind Equalization Algorithms Based on FFNN

1.0

f(x)

0.5 0

α1 = 0.25 α2 = 0.15 α3 = 0.05

–0.5 –1.0 –1

–0.5

0 x

0.5

1.0

Figure 3.14: The input–output characteristic curves of f (x).

1.0

Figure 3.15: The input–output characteristic curves of f 󸀠 (x).

2.0

f′(x)

1.5

α1 = 0.25

α3 = 0.05

α2 = 0.15

1.0 0.5 0 –1

–0.5

0 x

0.5

The cost function is selected as eq. (3.6). The steepest descent method is used in the weight iteration formula of network, so 𝜕J(n) 𝜕W(n) ̃ 𝜕J(n) 𝜕x(n) 󵄨 ̃ [󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] = 2x(n) 𝜕W(n) 𝜕W(n)

W(n + 1) = W(n) – ,

(3.44) (3.45)

where , is the weight iterative step size. Because there are two hidden layers and an output layer in the four-layer FFNN, the weight iteration formulas are different. Weight iterative formula of the output layer The connection weight between the output layer and the second hidden layer is wk (n), so ̃ 𝜕x(n) ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] wk (n + 1) = wk (n) – 2,1 x(n) 𝜕wk (n) K ̃ 𝜕x(n) = f3󸀠 ( ∑ wk (n)vkK (n)) vkK (n) 𝜕wk (n) k=1

(3.46) (3.47)

K

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f3󸀠 ( ∑ wk (n)vkK (n)) vkK (n) wk (n + 1) = wk (n) – 2,1 x(n) k=1

where ,1 is the weight iterative step size of the output layer.

(3.48)

3.3 Blind equalization algorithm based on the multilayer FFNN

67

Weight iterative formula between the first hidden layer and the second hidden layer The connection weight between the first hidden layer and the second hidden layer is wjk (n), so ̃ 𝜕x(n) ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] wjk (n + 1) = wjk (n) – 2,2 x(n) 𝜕wjk (n)

(3.49)

K ̃ 𝜕x(n) 𝜕u(n) = f3󸀠 ( ∑ wk (n)vkK (n)) 𝜕wjk (n) 𝜕w jk (n) k=1 K

= f3󸀠 ( ∑ wk (n)vkK (n)) wk (n) k=1

𝜕vkK (n) 𝜕wjk (n)

K

J

k=1

j=1

(3.50)

= f3󸀠 ( ∑ wk (n)vkK (n)) wk (n)f2󸀠 (∑ wjk (n)vjJ (n)) vjJ (n) K

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f3󸀠 ( ∑ wk (n)vkK (n)) wjk (n + 1) = wjk (n) – 2,2 x(n) k=1

J

× f2󸀠 (∑ wjk (n)vjJ (n)) wk (n)vjJ (n)

(3.51)

j=1

where ,2 is the weight iterative step size between the first hidden layer and the second hidden layer. Weight iterative formula between the input layer and the first hidden layer The connection weight between the input layer and the first hidden layer is wij (n), so ̃ 𝜕x(n) ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] wij (n + 1) = wij (n) – 2,3 x(n) 𝜕wij (n)

(3.52)

J

K I ̃ 𝜕x(n) = f3󸀠 ( ∑ wk (n)vkK (n)) f2󸀠 (∑ wjk (n)vjJ (n)) f1󸀠 (∑ wij (n)y(n – i)) 𝜕wij (n) k=1 j=1 i=1

× wk (n)wjk (n)y(n – i)

(3.53) K

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f3󸀠 ( ∑ wk (n)vkK (n)) wij (n + 1) = wij (n) – 2,3 x(n) k=1

×

J I f2󸀠 (∑ wjk (n)vjJ (n)) f1󸀠 (∑ wij (n)y(n j=1 i=1

(3.54) – i)) wk (n)wjk (n)y(n – i)

where ,3 is the weight iterative step size between the input layer and the first hidden layer.

68

3 Research of Blind Equalization Algorithms Based on FFNN

3.3.2.3 Computer simulation results The input signals are 4PAM and 8PAM. SNR is 20dB. The simulation channels adopt the typical digital telephone channel and the ordinary channel, and their z-transforms are shown in eqs. (2.84) and (3.13). The superposed Gauss white noise is zero mean and unit variance. When 4PAM and 8PAM go through the typical digital telephone channel, the iterative step sizes are ,1 = ,2 = ,3 = 0.000028 and ,1 = ,2 = ,3 = 0.0000015 respectively. When 4PAM and 8PAM go through the ordinary channel, the iterative step sizes are ,1 = ,2 = ,3 = 0.000018 and ,1 = ,2 = ,3 = 0.000001 respectively. In the simulation of 4PAM signals, the 11-3-2-1 type FFNN is adopted. The initialization weights are designed as below. T

(1)

(2) (3)

00000100000 [ ] From the input layer to the first hidden layer wij (n) = [ 0 0 0 0 0 0 0 0 0 1 0 ] , [0 0 0 0 0 1 0 0 0 0 0] T 110 ] , From the first hidden layer to the second hidden layer wjk = [ 010 T

From the second hidden layer to the output layer wk = [ 1 0.1 ] .

In the simulation of 8PAM signals, the 5-4-4-1 type FFNN is adopted. The initialization weights are designed as below. 0 1 0 0

1 0 0 0

0 0 0 0

T

(1)

0 [0 [ From the input layer to the first hidden layer wij (n) = [ [0 [0

0 0] ] ] , 0] 0]

(2)

1 [0 [ From the first hidden layer to the second hidden layer wjk (n) = [ [0 [1

(3)

From the second hidden layer to the output layer wk (n) = [ 1.1 –0.1 –0.1 –0.1 ] .

0 0 0 1

0 1 1 0

T

0 0] ] ] , 0] 0]

T

When two kinds of signals go through two channels using three-layer FFNN (TFFNN) and four-layer FFNN (FOFFNN), the convergence curves and the BER curves are shown respectively from Figures 3.16 to 3.19. The BER curves are obtained in different SNRs after 40,000 iterations. Simulation results have shown that the four-layer FFNN blind equalization algorithm is compared with the three-layer FFNN blind equalization algorithm, the SRE and BER are smaller, but the convergence rate become slower.

69

3.3 Blind equalization algorithm based on the multilayer FFNN

0.12

0.10 (1) TFFNN

0.10

0.06

MSE

0.08 MSE

(1) TFFNN 0.08

(2) FOFFNN

(2)

0.04 (1)

0.02 0

0

(a)

(2) FOFFNN

0.06

(1)

0.04

(2)

0.02

5,000 10,000 Iterative number

0

15,000

0

5,000 10,000 Iterative number

(b)

15,000

Figure 3.16: The convergence curves of 4PAM and 8PAM through the typical telephone channel: (a) The convergence curves of 4PAM and (b) The convergence curves of 8PAM.

5

1 (1) TFFNN (2) FOFFNN

0.6 (1) (2)

0.4

(1) TFFNN (2) FOFFNN

4 BER × 10–5

BER × 10–5

0.8

3

(1)

2 (2) 1

0.2 0 19 (a)

20

21 SNR (dB)

22

0 19

23

20

(b)

21

22

23

SNR (dB)

Figure 3.17: The BER curves of 4PAM and 8PAM through the typical telephone channel: (a) The BER curves of 4PAM and (b) The BER curves of 8PAM.

0.1

0.1 (1) TFFNN (2) F0FFNN

0.06

(1)

(1)

0.04 (2)

0.02 0 (a)

0

5,000

(1) TFFNN (2) F0FFNN

0.08 MSE

MSE

0.08

0.06 (2)

0.04 0.02

10,000

Iterative number

15,000

0 (b)

0

5,000

10,000

15,000

Iterative number

Figure 3.18: The convergence curves of 4PAM and 8PAM through the ordinary channel: (a) The convergence curves of 4PAM and (b) The convergence curves of 8PAM.

70

3 Research of Blind Equalization Algorithms Based on FFNN

5 (1) TFFNN (2) FOFFNN

(1)

1.0

0.5

19

20

21 SNR (dB)

(a)

(1)

3 2 1

(2) 0

(1) TFFNN (2) FOFFNN

4 BER × 10–5

BER × 10–5

1.5

22

23

0

(2) 19

20

(b)

21 SNR (dB)

22

23

Figure 3.19: The BER curves of 4PAM and 8PAM through the ordinary channel: (a) The BER curves of 4PAM and (b) The BER curves of 8PAM.

3.3.3 Blind equalization algorithm based on the five-layer FFNN 3.3.3.1 Model of the five-layer FFNN The five-layer FFNN is a neural network with three hidden layers, and its structure is shown in Figure 3.20. There is only the forward output in the neural network, and each layer is connected with the weights. The connection weight between the input layer and the first hidden layer is wij (n)(i = 1, 2, . . . , I; j = 1, 2, . . . , J), between the first hidden layer and the second hidden layer is wjp (n)(p = 1, 2, . . . , P), between the second hidden layer and the third hidden layer is wpk (n)(k = 1, 2, . . . , K), and that between the third hidden layer and the output layer is wk (n). The input of the neuron is u, the output of the neuron is v. uIi is the ith input neuron of the Ith layer; viJ is the ith output neuron of the Jth layer. The input of the ̃ whole neural network is y(n – i), and the output is x(n), so the state functions of the five-layer FFNN are defined as below.

wij(n)

wjp(n)

wpk(n)

y(n−1)

wk(n)

y(n− 2) ~ x(n) Output layer

y(n− I) Input layer I The first The second hidden layer J hidden layer P Figure 3.20: The structure of five-layer FFNN.

The third hidden layer K

71

3.3 Blind equalization algorithm based on the multilayer FFNN

uIi (n) = y(n – i) viI (n)

=

uIi (n)

(3.55)

= y(n – i)

(3.56)

I

I

i=1

i=1

uJj (n) = ∑ wij (n)viI (n) = ∑ wij (n)y(n – i)

(3.57)

I

vjJ (n) = f1 (uJj (n)) = f1 (∑ wij (n)y(n – i))

(3.58)

i=1 J

uPp (n) = ∑ wjp (n)vjJ (n)

(3.59)

j=1 J

vpP (n) = f2 (uPp (n)) = f2 (∑ wjp (n)vjJ (n))

(3.60)

j=1 P

uKk (n) = ∑ wpk (n)vpP (n)

(3.61)

p=1 P

vkK (n) = f3 (uKk (n)) = f3 ( ∑ wpk (n)vpP (n))

(3.62)

p=1 K

u(n) = ∑ wk (n)vkK (n)

(3.63)

k=1 K

̃ v(n) = x(n) = f4 (u(n)) = f4 ( ∑ wk (n)vkK (n))

(3.64)

k=1

3.3.3.2 Derivation of algorithm The cost function is selected as eq. (3.6) and the transfer function is chosen as follows [141]: f1 (x) = f2 (x) = f3 (x) = f4 (x) = f (x) = x + ! sin 0x

!>0

(3.65)

The steepest descent method is used in the weight iteration formula of network, so 𝜕J(n) 𝜕W(n) ̃ 𝜕x(n) 𝜕J(n) ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] = 2x(n) 𝜕W(n) 𝜕W(n)

W(n + 1) = W(n) – ,

(3.66) (3.67)

where , is the weight iterative step size. Because there are three hidden layers and one output layer in the five-layer FFNN, the weight iteration formulas are different. Weight iterative formula of the output layer The connection weight between the output layer and the third hidden layer is wk (n), so

72

3 Research of Blind Equalization Algorithms Based on FFNN

̃ 𝜕x(n) ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] wk (n + 1) = wk (n) – 2,1 x(n) 𝜕wk (n)

(3.68)

K ̃ 𝜕x(n) = f4󸀠 ( ∑ wk (n)vkK (n)) vkK (n) 𝜕wk (n) k=1

(3.69) K

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f4󸀠 ( ∑ wk (n)vkK (n)) vkK (n) wk (n + 1) = wk (n) – 2,1 x(n)

(3.70)

k=1

where ,1 is the weight iterative step size of the output layer. Weight iterative formula between the second hidden layer and the third hidden layer The connection weight between the second hidden layer and the third hidden layer is wpk (n), so ̃ 𝜕x(n) ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] wpk (n + 1) = wpk (n) – 2,2 x(n) 𝜕wpk (n)

(3.71)

K ̃ 𝜕x(n) 𝜕u(n) = f4󸀠 ( ∑ wk (n)vkK (n)) 𝜕wpk (n) 𝜕w pk (n) k=1 K

= f4󸀠 ( ∑ wk (n)vkK (n)) wk (n) k=1

𝜕vkK (n) 𝜕wpk (n)

K

P

k=1

p=1

(3.72)

= f4󸀠 ( ∑ wk (n)vkK (n)) wk (n)f3󸀠 ( ∑ wpk (n)vpP (n)) vpP (n) K

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f4󸀠 ( ∑ wk (n)vkK (n)) wpk (n + 1) = wpk (n) – 2,2 x(n) k=1

P

× f3󸀠 ( ∑ wpk (n)vpP (n)) wk (n)vpP (n)

(3.73)

p=1

where ,2 is the weight iterative step size between the second hidden layer and the third hidden layer. Weight iterative formula between the first hidden layer and the second hidden layer The connection weight between the first hidden layer and the second hidden layer is wjp (n), so ̃ 𝜕x(n) ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] wjp (n + 1) = wjp (n) – 2,3 x(n) 𝜕wjp (n)

(3.74)

73

3.3 Blind equalization algorithm based on the multilayer FFNN

K ̃ 𝜕x(n) 𝜕u(n) = f4󸀠 ( ∑ wk (n)vkK (n)) 𝜕wjp (n) 𝜕w jp (n) k=1 K

= f4󸀠 ( ∑ wk (n)vkK (n)) wk (n) k=1

𝜕vkK (n) 𝜕wjp (n)

K

P

J

k=1

p=1

j=1

= f4󸀠 ( ∑ wk (n)vkK (n)) wk (n)f3󸀠 ( ∑ wpk (n)vpP (n)) wpk (n)f2󸀠 (∑ wjp (n)vjJ (n)) vjJ (n) (3.75) K

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f4󸀠 ( ∑ wk (n)vkK (n)) wjp (n + 1) = wjp (n) – 2,3 x(n) k=1

(3.76) P

J

p=1

j=1

× f3󸀠 ( ∑ wpk (n)vpP (n)) f2󸀠 (∑ wjp (n)vjJ (n)) wk (n)wpk (n)vjJ (n) where ,3 is the weight iterative step size between the first hidden layer and the second hidden layer.

Weight iterative formula between the input layer and the first hidden layer The connection weight between the input layer and the first hidden layer is wij (n), so ̃ 𝜕x(n) ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] wij (n + 1) = wij (n) – 2,4 x(n) 𝜕wij (n)

(3.77)

K ̃ 𝜕x(n) 𝜕u(n) = f4󸀠 ( ∑ wk (n)vkK (n)) 𝜕wij (n) 𝜕w ij (n) k=1 K

= f4󸀠 ( ∑ wk (n)vkK (n)) wk (n) k=1

𝜕vkK (n) 𝜕wij (n)

K

P

k=1

p=1

= f4󸀠 ( ∑ wk (n)vkK (n)) wk (n)f3󸀠 ( ∑ wpk (n)vpP (n)) J

× wpk (n)f2󸀠 (∑ wjp (n)vjJ (n)) j=1

𝜕vjJ (n) 𝜕wij (n)

K

P

J

k=1

p=1

j=1

= f4󸀠 ( ∑ wk (n)vkK (n)) wk (n)f3󸀠 ( ∑ wpk (n)vpP (n)) wpk (n)f2󸀠 (∑ wjp (n)vjJ (n)) I

× wjp (n)f1󸀠 (∑ wij (n)y(n – i)) y(n – i) i=1

(3.78)

74

3 Research of Blind Equalization Algorithms Based on FFNN

K

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f4󸀠 ( ∑ wk (n)vkK (n)) wij (n + 1) = wij (n) – 2,4 x(n) k=1

P

J

p=1

j=1

× f3󸀠 ( ∑ wpk (n)vpP (n)) f2󸀠 (∑ wjp (n)vjJ (n))

(3.79)

I

× f1󸀠 (∑ wij (n)y(n – i)) wk (n)wpk (n)wjp (n)y(n – i) i=1

where ,4 is the weight iterative step size between the input layer and the first hidden layer.

3.3.3.3 Computer simulation results The input signals are 4PAM and 8PAM. SNR is 20 dB. The simulation channels adopt the typical digital telephone channel and the ordinary channel, and their z-transforms are shown in eqs. (2.84) and (3.13). The superposed Gauss white noise is zero mean and unit variance. When 4PAM signals go through the typical digital telephone channel, the iterative step sizes are ,1 = ,2 = ,3 = ,4 = 0.000034. When 8PAM signals go through the typical digital telephone channel, the iterative step sizes are ,1 = ,2 = ,3 = ,4 = 0.000023. When 4PAM signals go through the ordinary channel, the iterative step sizes are ,1 = ,2 = ,3 = ,4 = 0.000028. When 8PAM signals go through the ordinary channel, the iterative step sizes are ,1 = ,2 = ,3 = ,4 = 0.0000018. In the simulation of 4PAM signals, the 7-5-3-2-1 type FFNN is adopted. The initialization weights are designed as below. 0 0 1 1 0

0 0.1 0 0 1

0 0 0 0 1

0 0 0.1 0 0

0 0 0 0 1

T

(1)

0.1 [ 1 [ [ From the input layer to the first hidden layer wij (n) = [ 0 [ [ 1 [ 0

1 0] ] ] 0] ; ] 0] 0]

(2)

1 0.1 0 0 0 [ ] From the first hidden layer to the second hidden layer wjp (n) = [ 0.1 0.5 0 0 0 ] ; [ 1 1 0 0 1]

(3)

10 0 ] ; From the second hidden layer to the third hidden layer wpk (n) = [ 0 1 0.1

(4)

From the third hidden layer to the output layer wk (n) = [ 1 0.1 ] .

T

T

T

In the simulation of 8PAM signals, the 5-4-4-2-1 type FFNN is adopted. The initialization weights are designed as follows:

3.3 Blind equalization algorithm based on the multilayer FFNN

(1)

(2) (3)

0.1 [ 0 [ From the input layer to the first hidden layer wij (n) = [ [ 0.1 [ 0

0 1 0 0.1

1 0 0 0

0 0 0.1 0.1

75

T

0 0] ] ] ; 0] 0]

1 [ 0 [ From the first hidden layer to the second hidden layer wjp (n) = [ [ 0.1 [ 1 From the second hidden layer to the third hidden layer

T

0.1 0 0 0 1 0] ] ] ; 0 1 0] 1 0 0] wpk (n) =

T

1 0.5 0 0.1 ] ; [ 0.1 0 0.1 1 (4)

T

From the third hidden layer to the output layer wk (n) = [ 0.1 0.1 ] .

When two kinds of signals go through two channels using three-layer FFNN (TFFNN) and five-layer FFNN (FIFFNN), the convergence curves and the BER curves are shown respectively from Figures 3.21 to 3.24. The BER curves are obtained in different SNR after 40,000 iterations. Simulation results have shown that the FIFFNN blind equalization algorithm is compared with the three-layer FFNN blind equalization algorithm, the SRE and BER are smaller, but the convergence rate becomes slower. When two kinds of signals go through two channels using FOFFNN and FIFFNN, the convergence curves and the BER curves are shown respectively from Figures 3.25 to 3.28. When 4PAM signals go through the typical digital telephone channel, the iterative step sizes are ,1 = ,2 = ,3 = ,4 = 0.000025. When 8PAM signals go through the typical digital telephone channel, the iterative step sizes are ,1 = ,2 = ,3 = ,4 = 0.000013.

0.12

0.12 (1) TFFNN (2) FIFFNN

0.10

0.08 MSE

MSE

0.08 0.06 0.04 (1)

0.02 0 (a)

(1) TFFNN (2) FIFFNN

0.10

0

(1)

0.06

(2)

0.04

(2)

0.02 5,000 10,000 Iterative number

15,000

0 (b)

0

5,000 10,000 Iterative number

15,000

Figure 3.21: The convergence curves of 4PAM and 8PAM through the typical telephone channel: (a) The convergence curves of 4PAM and (b) The convergence curves of 8PAM.

76

3 Research of Blind Equalization Algorithms Based on FFNN

0.10

0.10 (1) TFFNN (2) FIFFNN

(1)

0.06 (2)

(1) 0.04 0.02 0

(1) TFFNN (2) FIFFNN

0.08 MSE

MSE

0.08

0.06 (2) 0.04 0.02

0

(a)

5,000

10,000

0

15,000

Iterative number

0

(b)

5,000

10,000

15,000

Iterative number

Figure 3.22: The convergence curves of 4PAM and 8PAM through the ordinary channel: (a) The convergence curves of 4PAM and (b) The convergence curves of 8PAM.

1

5 (1) TFFNN (2) FIFFNN

0.6

(1)

0.4 (2)

0.2 0 19 (a)

4 BER × 10–5

BER × 10–5

0.8

20

(1)

3 2

(2)

1

21 SNR (dB)

22

0

23

(1) TFFNN (2) FIFFNN

19

20

(b)

21 SNR (dB)

22

23

Figure 3.23: The BER curves of 4PAM and 8PAM through the typical telephone channel: (a) The BER curves of 4PAM and (b) The BER curves of 8PAM.

5 (1) TFFNN (2) FIFFNN

1.0

(1) 0.5

(1) TFFNN (2) FIFFNN

4 BER × 10–5

BER × 10–5

1.5

(2)

(1)

3 2

(2) 1

0 19 (a)

20

21 SNR (dB)

22

23

0 19 (b)

20

21 22 SNR (dB)

23

Figure 3.24: The BER curves of 4PAM and 8PAM through the ordinary channel: (a) The BER curves of 4PAM and (b) The BER curves of 8PAM.

77

3.3 Blind equalization algorithm based on the multilayer FFNN

0.12

0.12 (1) FOFFNN (2) FIFFNN

0.10

0.08 MSE

MSE

0.08 0.06 (1)

0.04

(2)

0

(2)

0.02

5,000 10,000 Iterative number

(a)

(1)

0.06 0.04

0.02 0

(1) FOFFNN (2) FIFFNN

0.10

0

15,000

0

(b)

5,000 10,000 Iterative number

15,000

Figure 3.25: The convergence curves of 4PAM and 8PAM through the typical telephone channel: (a) The convergence curves of 4PAM and (b) The convergence curves of 8PAM.

0.10

0.10 (1) FOFFNN (2) FIFFNN

0.08

(1) MSE

MSE

0.06 (2) 0.04

0

0.06

(1)

0.02

(1) FOFFNN (2) FIFFNN

0.08

(2)

0.04 0.02

0

5,000 10,000 Iterative number

(a)

0

15,000

0

(b)

5,000 10,000 Iterative number

15,000

Figure 3.26: The convergence curves of 4PAM and 8PAM through the ordinary channel: (a) The convergence curves of 4PAM and (b) The convergence curves of 8PAM.

12 (1) F0FFNN (2) FIFFNN

4

(1)

2

0 19 (a)

(2)

20

(1) F0FFNN (2) FIFFNN

10 BER × 10–6

BER × 10–6

6

8 (2)

(1)

6 4

21 SNR (dB)

22

23

2 19 (b)

20

21 SNR (dB)

22

23

Figure 3.27: The BER curves of 4PAM and 8PAM through the typical telephone channel: (a) ) The BER curves of 4PAM and (b) The BER curves of 8PAM.

78

3 Research of Blind Equalization Algorithms Based on FFNN

1

2.5

BER × 10–5

(1) F0FFNN (2) FIFFNN

0.6 0.4

(2)

0.2 0

19 (a)

(1)

2.0 BER × 10–5

(1)

0.8

1.5 1.0 0.5

20

21 SNR (dB)

22

23

(1) F0FFNN (2) FIFFNN

0

(2) 19

20

(b)

21 22 SNR (dB)

23

Figure 3.28: The BER curves of 4PAM and 8PAM through the ordinary channel: (a) The BER curves of 4PAM and (b) The BER curves of 8PAM.

When 4PAM signals go through the ordinary channel, the iterative step sizes are ,1 = ,2 = ,3 = ,4 = 0.000018. When 8PAM signals go through the ordinary channel, the iterative step sizes are ,1 = ,2 = ,3 = ,4 = 0.000001. Simulation results have shown that the FIFFNN blind equalization algorithm is compared with the FOFFNN blind equalization algorithm, the SRE and BER are smaller, but the convergence rate become slower. From the analysis and simulations in this section, it can be seen that with the increase of the neural network hidden layer number, the convergence rate and computational complexity increase, but after convergence the SRE and BER are reduced.

3.4 Blind equalization algorithm based on the momentum term FFNN 3.4.1 Basic principles of algorithm According to the analysis in Section 2.4.2.1, in order to accelerate the convergence speed, the momentum term can be introduced. Its weight variation is BW(n) = 𝛾BW(n – 1) – ,

𝜕J(n) 𝜕W(n)

(3.80)

where 0 ≤ 𝛾 < 1 is the added momentum term, and 𝛾BW(n – 1) memories the weight modification direction of previous moment. That is, the weight modification direction at (n + 1) moment is the directions superposition at (n – 1) moment and n moment. 𝜕J(n) When the sign of –, 𝜕W(n) at n moment is the same as the sign of (n – 1) moment,

𝜕J(n) BW(n) increases, so adjustment speed is accelerated. When the sign of –, 𝜕W(n) at n moment is different from the sign of (n – 1) moment, BW(n) decreases, so W(n + 1) is stable. If 𝛾 = 0, eq. (3.80) becomes the traditional BP algorithm.

3.4 Blind equalization algorithm based on the momentum term FFNN

79

The search paths of the traditional BP algorithm, momentum term BP algorithm and variable learning rate momentum term BP algorithm on the same performance surface are shown from Figures 3.29 to 3.31. The 1-2-1 type FFNN structure is adopted, and the initial weights are all the same. The weight from the hidden layer to the output layer is w = [1, 1]T , the weight from the input layer to the hidden layer is w = [1, 1]T . It can be seen from the figures that for the same performance surface, the search paths of the variable learning rate momentum BP algorithm and momentum BP algorithm are shorter than that of the traditional BP algorithm, that is ,their convergence speeds are faster.

10

10 w2(1,1)

Sum sq. Error

15

5

5

0 –5 0

–5

0 0

5 5

10 w2(1,1)

10 15

–5

15 w1(1,1)

–5

0

5 w1(1,1)

10

15

Figure 3.29: The convergence trajectory of the traditional BP algorithm in the same performance surface.

15

10

5

w2(1,1)

Sum sq. Error

10

0 –5 –5

0

5

0

0

5 5

10 w2(1,1)

10 15

15 w1(1,1)

–5 –5

0

5 w1(1,1)

10

15

Figure 3.30: The convergence trajectory of the momentum term BP algorithm in the same performance surface.

80

3 Research of Blind Equalization Algorithms Based on FFNN

10

10

5

w2(1,1)

Sum sq. Error

15

0 –5 –5

0

0

0

5 5

10 w2(1,1)

5

10 15

15 w1(1,1)

–5 –5

0

5 w1(1,1)

10

15

Figure 3.31: The convergence trajectory of the variable learning rate momentum term BP algorithm in the same performance surface.

wij(n)

y(n−1)

wj(n)

y(n−2)

~ x(n) Output layer

y(n−I) Input layer I Hidden layer J Figure 3.32: Structure of three-layer FFNN.

3.4.2 Derivation of algorithm In this algorithm, the three-layer FFNN is adopted. The structure is shown in Figure 3.32. The connection weight between the input layer and the hidden layer is wij (n)(i = 1, 2, ⋅ ⋅ ⋅ , I; j = 1, 2, ⋅ ⋅ ⋅ , J), and the connection weight between the hidden layer and the output layer is wj (n). The input of the neuron is u, the output of the neuron is v. uIi is the ith input neuron of the Ith layer; viJ is the ith output neuron of the ̃ Jth layer. The input of the whole neural network is y(n – i), and the output is x(n), so the state functions are defined as follows: uIi (n) = y(n – i)

(3.81)

viI (n)

(3.82)

=

uIi (n)

= y(n – i)

81

3.4 Blind equalization algorithm based on the momentum term FFNN

I

I

i=1

i=1

uJj (n) = ∑ wij (n)viI (n) = ∑ wij (n)y(n – i)

(3.83)

I

vjJ (n) = f1 (uJj (n)) = f1 (∑ wij (n)y(n – i))

(3.84)

i=1 J

u(n) = ∑ wj (n)vjJ (n)

(3.85)

j=1 J

̃ v(n) = x(n) = f2 (u(n)) = f2 (∑ wj (n)vjJ (n))

(3.86)

j=1

The cost function is selected as eq. (3.6). The transfer function is chosen as eq. (3.42). Using the recurrence formula, eq. (3.80) becomes n

BW(n) = –, ∑ 𝛾n–m m=0

n 𝜕J (m) 𝜕x̃ (m) = –2, ∑ 𝛾n–m x̃ (m) [|x̃ (m)|2 – R2 ] 𝜕W (m) 𝜕W (m) m=0

(3.87)

So, the weight iteration formulas of each layer are obtained.

3.4.2.1 Weight iterative formula of the output layer The connection weight between the output layer and the hidden layer is wj (n), so n

wj (n + 1) = wj (n) – 2,1 ∑ 𝛾n–m x̃ (m) [|x̃ (m)|2 – R2 ] m=0

𝜕x̃ (m) 𝜕wj (m)

(3.88)

J

𝜕x̃ (m) = f 󸀠 (∑ wj (m) vjJ (m)) vjJ (m) 𝜕wj (m) j=1

(3.89)

n

J

m=0

j=1

wj (n + 1) = wj (n) – 2,1 ∑ 𝛾n–m x̃ (m) [|x̃ (m)|2 – R2 ]f 󸀠 (∑ wj (m) vjJ (m)) vjJ (m) (3.90) where ,1 is the weight iterative step size of the output layer.

Weight iterative formula of the hidden layer The connection weight between the hidden layer and the input layer is wij (n), so n

wij (n + 1) = wij (n) – 2,2 ∑ 𝛾n–m x̃ (m) [|x̃ (m)|2 – R2 ] m=0

𝜕x̃ (m) 𝜕wij (m)

(3.91)

J

I 𝜕x̃ (m) = f 󸀠 (∑ wj (m) vjJ (m)) wj (m) f 󸀠 (∑ wij (m) y (m – i)) y (m – i) 𝜕wij (m) j=1 i=1

(3.92)

82

3 Research of Blind Equalization Algorithms Based on FFNN

n

J

m=0

j=1

wij (n + 1) = wij (n) – 2,2 ∑ 𝛾n–m x̃ (m) [|x̃ (m)|2 – R2 ]f 󸀠 (∑ wj (m) vjJ (m)) (3.93)

I

× f 󸀠 (∑ wij (m) y (m – i)) wj (m) y (m – i) i=1

where ,2 is the weight iterative step size of the hidden layer.

3.4.3 Computer simulation results The input signals are 4PAM and 8PAM. SNR is 20 dB. The simulation channels adopt the typical digital telephone channel and the ordinary channel, and their z-transforms are shown in eqs. (2.84) and (3.13). When 4PAM signals go through the typical digital telephone channel, the iterative step sizes are ,1 = ,2 = 0.00018, 𝛾 = 0.6. When 8PAM signals go through the typical digital telephone channel, the iterative step sizes are ,1 = ,2 = 0.00001, 𝛾 = 0.2. When 4PAM signals go through the ordinary channel, the iterative step sizes are ,1 = ,2 = 0.00018, 𝛾 = 0.4. When 8PAM signals go through the ordinary channel, the iterative step sizes are ,1 = ,2 = 0.00001, 𝛾 = 0.25. In the simulation of 4PAM signals, the 11-3-1 type FFNN is adopted. The initialization weights are designed as follows: T

(1) (2)

00000100000 [ ] From the input layer to the hidden layer wij (n) = [ 0 0 0 0 0 1 0 0 0 0 0 ] , [0 0 0 0 0 1 0 0 0 0 0] T From the hidden layer to the output layer wj (n) = [ 1 0.1 1 ] .

In the simulation of 8PAM signals, the 5-4-1 type FFNN is adopted. The initialization weights are designed as follows: 0 1 0 0

1 0 0 0

0 0 0 0

T

(1)

0 [0 [ From the input layer to the hidden layer wij (n) = [ [0 [0

0 0] ] ] , 0] 0]

(2)

From the hidden layer to the output layer wj (n) = [ 1.1 –0.1 –0.1 –0.1 ] .

T

When two kinds of signals go through two channels using the traditional BP algorithm (TBP) and the momentum term BP algorithm (MTBP), the convergence curves and the BER curves are shown from Figures 3.33 to 3.36, respectively. The BER curves are obtained in different SNRs after 25,000 iterations. From the figures, it can be seen that the MTBP has faster convergence speed and lower BER, but SRE almost unchanged.

3.4 Blind equalization algorithm based on the momentum term FFNN

0.12

0.10 (1) TBP (2) MTBP

0.10

MSE

MSE

0.06

(1)

0.04

(2)

(2)

0.04

0

1,000 2,000 3,000 4,000 5,000 Iterative number

(a)

(1)

0.06

0.02

0.02 0

(1) TBP (2) MTBP

0.08

0.08

0

83

0

1,000 2,000 3,000 4,000 5,000 Iterative number

(b)

Figure 3.33: The convergence curves of 4PAM and 8PAM through the typical telephone channel: (a) The convergence curves of 4PAM and (b) The convergence curves of 8PAM.

0.12

0.10 (1) TBP (2) MTBP

0.10

(1) (1)

MSE

MSE

0.08 0.06

(2) 0.04

0.06 (2)

0.04 0.02

0.02 0

(1) TBP (2) MTBP

0.08

0

0

1,000 2,000 3,000 4,000 5,000

(a)

0

1,000 2,000 3,000 4,000 5,000

(b)

Iterative number

Iterative number

Figure 3.34: The convergence curves of 4PAM and 8PAM through the ordinary channel: (a) The convergence curves of 4PAM and (b) The convergence curves of 8PAM.

12

4

8 6 (1)

4 0

(a)

19

20

2 1

21 SNR (dB)

22

23

(1) MTBP (2) TBP

(2)

3 BER × 10–5

BER × 10–6

(1) MTBP (2) TBP

(2)

10

0

(b)

(1) 19

20

21

22

23

SNR (dB)

Figure 3.35: The BER curves of 4PAM and 8PAM through the typical telephone channel: (a) The BER curves of 4PAM and (b) The BER curves of 8PAM.

84

3 Research of Blind Equalization Algorithms Based on FFNN

4

6 (1) MTBP (2) TBP

(2)

(2)

BER × 10–5

BER × 10–5

3 2 (1)

4

2

(1)

1 0 (a)

19

20

21

22

23

SNR (dB)

(1) MTBP (2) TBP

0

19

(b)

20

21 SNR (dB)

22

23

Figure 3.36: The BER curves of 4PAM and 8PAM through the ordinary channel: (a) The BER curves of 4PAM and (b) The BER curves of 8PAM.

3.5 Blind equalization algorithm based on the time-varying momentum term FFNN 3.5.1 Basic principles of algorithm In the blind equalization algorithm based on the momentum term, the momentum term has a great influence on the algorithm performance. According to the weight iteration formula (3.80), the current weights update and previous weights update are in the same direction, the function of momentum term is to increase learning rates (i. e., iterative step size), which is beneficial for speeding up the convergence rate. But when the algorithm is closing to convergence, large SRE and BER will be produced. Therefore, the time-varying momentum term is used, that is, at the beginning of the iterative algorithm, the momentum is increased to accelerate the convergence speed, and when approaching convergence, the momentum is decreased to reduce the SRE and BER. 3.5.1.1 Selection of momentum term control factor The convergence process of the FFNN blind equalization algorithm is the process of the neural network weight vector gradually closing to the optimal weight vector, that ̂ ̃ is, the process of the residual error e(n) = x(n) – x(n) gradually decreasing. It can be seen that the steady residual error transformation rule is basically identical with the required time-varying momentum item transformation rule, but there are some defects if the SRE directly is used to control momentum item. (1) In the convergence process of the algorithm, the residual error is sensitive to the interference signal. When the algorithm closes to convergence, if there is a strong burst interference signal in the channel, the residual error will increase, which leads the momentum term increasing. It is easy to cause the inaccurate adjustment and the algorithm divergence.

85

3.5 Blind equalization algorithm based on the time-varying momentum term FFNN

(2)

The time-varying momentum algorithm needs to use larger momentum item before the algorithm convergence, which can accelerate the convergence speed. But if the momentum item is directly adjusted by the residual error, the residual error will fall sharply, which leads to the momentum item rapidly decrease, and convergence speed slow.

Therefore, the nonlinear function of the MSE is used as the momentum term control factor. The function of the momentum term is 𝛾(n) = ! [1 – e1–MSE(n) ]

(3.94)

where ! is the scale factor, and it is used to control the size of the function 𝛾(n) and determine the speed of the curve rise. The parameter 1 is a constant, and it used to control function 𝛾(n) plus and minus signs change. When the algorithm converges, the value of 1 is appropriately selected to make the momentum term is a negative value, which can reduce the iterative step size, so that reduce state residual error and the BER. However, both parameters need experiment to determine. 3.5.1.2 Algorithm performance analysis The time-varying optimal weight vector from the input layer to the hidden layer is ŵ 11 (n) .. . ̂ w (n) J1 [

[ ŵ ij (n) = [ [

⋅ ⋅ ⋅ ŵ I1 (n) .. .. ] ] . . ] ⋅ ⋅ ⋅ ŵ JI (n) ]

(3.95)

The time-varying optimal weight vector from the hidden layer to the output layer is T

ŵ j (n) = [ ŵ 1 (n) ⋅ ⋅ ⋅ ŵ J (n) ]

(3.96)

̂ x(n) = Y(n)ŵ ij (n)ŵ j (n) + . (n)

(3.97)

So,

where . (n) is the interference signal with zero mean and independent identically distributed. Plugging eq. (3.97) into the expression of residual error (2.76), we obtain ̂ – x(n) ̃ e(n) = x(n) = Y(n)ŵ ij (n)ŵ j (n) – Y(n)wij (n)wj (n) + . (n) = Y(n) [ŵ ij (n)ŵ j (n) – wij (n)wj (n)] + . (n) = Y(n)V(n) + . (n)

(3.98)

where V(n) is the weight error vector. In the algorithm convergence process, the neural network weight vector gradually closes to the optimal weight vector, that is, V(n) gradually reduces until it tends to zero. . (n) is the interference signal.

86

3 Research of Blind Equalization Algorithms Based on FFNN

The above theoretical analysis shows that the change tendency of the residual error signal is from big to small. At the start of the algorithm, neural network weight vector is far from the optimal weight vector, and the residual error is the maximum. The algorithm gradually converges while the residual error decreases gradually, and after convergence it reaches the minimum. Therefore, the transformation rule of residual error is identical to the requirement of time-varying momentum item. 3.5.1.3 Selection of rectangular window length In the iteration process of algorithm, the real MSE is unavailable, so it is commonly substituted by its estimation. MSE estimation is obtained by the average of D residual errors’ squares and D is the length of the rectangular window. The selection of D has a great impact on the algorithm. The smaller the D is, the more sensitive the momentum term to the channel mutation and burst noise is, that is, the stronger the ability to track the time-varying channel is. However, the momentum term is sensitive to burst noise, it is possible to inaccurately adjust the equalizer, because these are mutually contradictory. The selection of D should be determined according to actual applications. When the channel time-varying is serious and strong interference noise is smaller, a smaller D should be selected in order to improve the capability of tracking the time-varying channel. When the channel is relatively stable and interference noise is large, a bigger D should be selected in order to reduce the misadjustment caused by strong noise. When the channel time-varying is serious and interference noise is large, the value of D can adopt a compromised value. In the simulation, generally, D = 100.

3.5.2 Derivation of algorithm Plugging eq. (3.94) into eqs. (3.90) and (3.93), the weight iteration formulas of the FFNN blind equalization algorithm based on the time-varying momentum term are obtained. (1) Weight iterative formula of the output layer n

J

m=0

j=1

wj (n + 1) = wj (n) – 2,1 ∑ 𝛾n–m (m) x̃ (m) [|x̃ (m)|2 – R2 ]f 󸀠 (∑ wj (m) vjJ (m)) vjJ (m) (3.99) (2)

Weight iterative formula of the hidden layer n

J

m=0

j=1

wij (n + 1) = wij (n) – 2,2 ∑ 𝛾n–m (m) x̃ (m) [|x̃ (m)|2 – R2 ]f 󸀠 (∑ wj (m) vjJ (m)) I

× f 󸀠 (∑ wij (m) y (m – i)) wj (m) y (m – i) i=1

(3.100)

3.5 Blind equalization algorithm based on the time-varying momentum term FFNN

87

3.5.3 Computer simulation results The input signals are 4PAM and 8PAM. SNR is 20 dB. The simulation channels adopt the typical digital telephone channel and the ordinary channel, and their z-transforms are shown in eqs. (2.84) and (3.13). The structure of neural network, the initial weights and the iterative step size are all identical with those of the FFNN blind equalization algorithm based on the momentum term. Two parameters in the momentum term function are ! = 10, 1 = 0.4. When two kinds of signals go through two channels using the momentum term BP algorithm (MTBP) and the time-varying momentum item BP algorithm (VMTBP), the convergence curves are shown in Figures 3.37 and 3.38. It can be seen from the figures that the convergence performance of the blind equalization algorithm based on the time-varying momentum term is slightly better than that of the blind equalization algorithm based on the momentum term.

0.12

0.10 (1) MTBP (2) VMTBP

0.10

MSE

MSE

0.08 0.06 0.04 (1)

0.02 0 0 (a)

(1) MTBP (2) VMTBP

0.08 0.06 0.04 (1)

(2)

(2)

0.02 0

1,000 2,000 3,000 4,000 5,000 Iterative number

0 (b)

1,000 2,000 3,000 4,000 5,000 Iterative number

Figure 3.37: The convergence curves of 4PAM and 8PAM through the typical telephone channel: (a) The convergence curves of 4PAM and (b) The convergence curves of 8PAM.

0.10

0.12 (1) MTBP (2) VMTBP

0.10

MSE

MSE

0.08 0.06 0.04 (1)

0.02 0 (a)

0

(2)

1,000 2,000 3,000 4,000 5,000 Iterative number

(1) MTBP (2) VMTBP

0.08 0.06 (1)

(2)

0.04 0.02 0 (b)

0

1,000 2,000 3,000 4,000 5,000 Iterative number

Figure 3.38: The convergence curves of 4PAM and 8PAM through the ordinary channel: (a) The convergence curves of 4PAM and (b) The convergence curves of 8PAM.

88

3 Research of Blind Equalization Algorithms Based on FFNN

4 (1)

1.0

(1) MTBP (2) VMTBP

3 BER × 10–5

BER × 10–5

1.5

0.5

19

20

(a)

2 (2) 1

(2) 0

(1) MTBP (2) VMTBP

(1)

21 SNR (dB)

22

0

23

19

20

21

(b)

22

23

SNR (dB)

Figure 3.39: The BER curves of 4PAM and 8PAM through the typical telephone channel: (a) The BER curves of 4PAM and (b) The BER curves of 8PAM.

4

6 (1)

2 1 0

(a)

(1) MTBP (2) VMTBP

(1)

BER × 10–5

BER × 10–5

3

(1) MTBP (2) VMTBP 4

2

(2)

(2) 19

20

21 SNR (dB)

22

23

0 19 (b)

20

21 SNR (dB)

22

23

Figure 3.40: The BER curves of 4PAM and 8PAM through the ordinary channel: (a) The BER curves of 4PAM and (b) The BER curves of 8PAM.

The BER curves of two kinds of signals through two channels are shown in Figures 3.39 and 3.40. The BER curves are obtained in different SNR after 25,000 iterations. It can be seen from the figures that the BER of the blind equalization algorithm based on the time-varying momentum term is lower than that of the blind equalization algorithm based on the momentum term.

3.6 Blind equalization algorithm based on variable step-size FFNN 3.6.1 Basic principles of algorithm According to the analysis in Section 2.3.2, the variable step size is substituted for the fixed step size in order to accelerate the convergence speed. In the blind equalization

89

3.6 Blind equalization algorithm based on variable step-size FFNN

algorithm, the variable step size ,(n) controlled by the MSE nonlinear function is substituted for the fixed step size ,, which can increase the step-size factor and improve the convergence speed at the beginning of the algorithm, and following the iteration proceed, the step-size factor gradually decreases until the minimum so that the steady residual error is minimum. The variable step size is ,(n) = ! {1 – e–"E[e

2

(n)]

}

(3.101)

where " is the undetermined parameter, and it is used to control step-size change speed. ! is the scale factor, and it is used to control the change range of ,(n). Because 0 ≤ 1 – exp {–"E [e2 (n)]} ≤ 1, from eq. (3.101), we obtain 0 ≤ ,(n) ≤ !

(3.102)

In the case of channel interference, the change of step size is always in the range of [0, !], which avoids the algorithm misadjustment or divergent caused by the sudden circumstances, and ensures the good tracking performance. At the same time, in order to ensure the convergence of the algorithm, the step-size factor must be satisfied as follows [145]: 0 ≤ ,(n) ≤

2 3tr(R)

(3.103)

where R is the autocorrelation matrix of the neural network input signal and tr(R) is the trace of R, so !≤

2 3tr(R)

(3.104)

Equation (3.104) is the selecting principle of !. The changing curves of e(n) and ,(n) are shown in Figures 3.41 and 3.42. In Figure 3.41, ! is fixed and " is variable, and in Figure 3.42, ! is variable and " is fixed. 󵄨 󵄨 It can be seen from the figures that with the decrease of 󵄨󵄨󵄨e(n)󵄨󵄨󵄨, the step-size factor 0.20

μ(n)

0.15 0.10 0.05

β=5

β=8 α = 0.2

β=2 β = 0.5

0 –1.0

–0.5

0 e(n)

0.5

1.0

Figure 3.41: The changing curves of e(n) and ,(n) when ! is fixed and " is variable.

90

3 Research of Blind Equalization Algorithms Based on FFNN

0.20 0.15

α = 0.20 α = 0.15

μ(n)

α = 0.10 0.10

β=8

0.05 α = 0.04 0 –1.0 –0.5

0 e(n)

0.5

1.0

Figure 3.42: The changing curves of e(n) and ,(n) when " is fixed and ! is variable.

,(n) decreases. ,(n) is monotonous, so it satisfies the requirements of the step-size variation.

3.6.2 Derivation of algorithm The three-layer FFNN is adopted in the algorithm, and its structure is shown in Figure 3.32. The cost function is selected as eq. (3.6). The transfer function is chosen as eq. (3.42). Using the time-varying step size of eq. (3.101) substitutes the fixed step size, and the iterative formulas of each layer are obtained. 3.6.2.1 Weight iterative formula of the output layer The connection weight between the output layer and the hidden layer is wj (n), so J

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f2󸀠 (∑ wj (n)vjJ (n)) vjJ (n) wj (n + 1) = wj (n) – 2,1 (n)x(n)

(3.105)

j=1

where ,1 (n) is the weight iterative variable step size of the output layer. 3.6.2.2 Weight iterative formula of the hidden layer The connection weight between the hidden layer and the input layer is wij (n), so J

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f2󸀠 (∑ wj (n)vjJ (n)) wij (n + 1) = wij (n) – 2,2 (n)x(n) j=1

I

× f1󸀠 (∑ wij (n)y(n – i)) wj (n)y(n – i) i=1

where ,2 (n) is the weight iterative variable step size of the hidden layer.

(3.106)

3.6 Blind equalization algorithm based on variable step-size FFNN

91

3.6.3 Computer simulation results

0.10

0.10

0.08

0.08 TBP

0.06 0.04

MSE

MSE

The input signals are 4PAM and 8PAM. SNR is 20dB. The simulation channels adopt the typical digital telephone channel and the ordinary channel, and their z-transforms are shown in eqs. (2.84) and (3.13). In the simulation, the 3-3-1 type FFNN is adopted, and the rectangular window length is D = 100. When two kinds of signals go through two channels using the traditional BP algorithm (TBP) and the variable step-size BP algorithm (VSBP), the convergence curves and BER curves are shown from Figure 3.43 to 3.46. In the variable step-size algorithm, the parameters of 4PAM are ! = 0.01, " = 1. And the parameters of 8PAM are ! = 0.01, " = 0.05. In the traditional fixed step-size algorithm, the parameter of 4PAM is , = 0.0002. Moreover, the parameter of 8PAM is , = 0.00001.

VSBP

0.02 0.00

TBP

0.06

VSBP

0.04 0.02

0

(a)

0.00

1,000 2,000 3,000 4,000 5,000 Iterative number

0

(b)

1,000 2,000 3,000 4,000 5,000 Iterative number

Figure 3.43: The convergence curves of 4PAM and 8PAM through the typical telephone channel: (a) The convergence curves of 4PAM and (b) The convergence curves of 8PAM.

0.10

0.10 0.08 TBP

0.06 0.04

VSBP

0.02

TBP

0.06

VSBP

0.04 0.02

0.00

0.00 0

(a)

MSE

MSE

0.08

1,000 2,000 3,000 4,000 5,000 Iterative number

0 (b)

1,000 2,000 3,000 4,000 5,000 Iterative number

Figure 3.44: The convergence curves of 4PAM and 8PAM through the ordinary channel: (a) The convergence curves of 4PAM and (b) The convergence curves of 8PAM.

92

3 Research of Blind Equalization Algorithms Based on FFNN

10–2

10–2

10–3

10–3

TBP BER

BER

TBP

10–4

10–5 11

10–4

VSBP

12

13

14

10–5

15

VSBP

11

12

SNR (dB)

13

14

15

SNR (dB)

Figure 3.45: The BER curves of 4PAM and 8PAM through the typical telephone channel. (a) The BER curves of 4PAM and (b) The BER curves of 8PAM.

10–2

10–2

10–3

10–3

TBP

BER

BER

TBP

10–4

10–5 11

12

13 SNR (dB)

VSBP 10–4

VSBP

14

15

10–5

11

12

13

14

15

SNR (dB)

Figure 3.46: The BER curves of 4PAM and 8PAM through the ordinary channel: (a) The BER curves of 4PAM and (b) The BER curves of 8PAM.

Simulation results have shown that the variable step-size FFNN blind equalization algorithm is compared with the fixed step-size FFNN blind equalization algorithm, the convergence rate speed become faster, the SRE and BER are smaller.

3.7 Summary In this chapter, first, the blind equalization algorithms based on the three-layer, fourlayer and five-layer FFNN are studied and the iteration formulas of algorithms are derived, then computer simulations are done. The five-layer and four-layer FFNN blind equalization algorithms are compared with the three-layer FFNN blind equalization algorithm, the SRE and BER are reduced, but the computational complexity is significantly increased. Therefore, the four-layer and five-layer FFNN blind equalization algorithms only have theoretical significance, but they are not practical.

3.7 Summary

93

Second, the improved BP algorithm is used in neural network blind equalization algorithm, then the blind equalization algorithms based on the momentum term time-varying momentum term and variable step size are studied. These algorithms are compared with the blind equalization algorithm based on the traditional BP neural network algorithm, and the convergence performance is improved.

4 Research of Blind Equalization Algorithms Based on the FBNN Abstract: In this chapter, the basic principles and structure types of feedback neural network (FBNN) are introduced. The blind equalization algorithms based on bilinear FBNN, diagonal recursive neural network, and quasi diagonal recursive neural network are studied. The principle characteristics and state functions of these algorithms are analyzed. The iteration formulas of algorithms are deduced and the computer simulations are done. At the same time, the idea of variable step size is introduced in these new algorithms. The mean square error nonlinear function is used as step-size control factor, so the blind equalization algorithm based on the variable step-size diagonal recursive neural network and variable step-size quasi diagonal recursive neural network are obtained.

4.1 Basic principles of FBNN 4.1.1 Concept of FBNN The FBNN is also known as the recursive neural network. The information transmission between neurons is no longer from one layer to another layer, but there is a link between every neuron. Each neuron is a processing unit, at the same time it can receive and output signals. Because the feedback mechanism is introduced, the time delay network is contained in the variable. It is a nonlinear dynamic system.

4.1.2 Structure of FBNN At present, in order to adapt to a variety of different dynamic performance requirements, dozens of types of FBNN structures have been proposed. Because the different structures lead to different input–output relations, different dynamic performances are showed. FBNN is roughly classified as in Figure 4.1 [146].

4.1.3 Characteristics of FBNN FBNN becomes a nonlinear dynamic system due to the existence of internal feedback. It can directly reflect the dynamic characteristics of the system, and it is suitable for industrial process modeling, simulation, and controlling [147]. Its main characteristic is that the network system has several stable states; when the network starts from

DOI 10.1515/9783110450293-004

95

4.2 Blind equalization algorithm based on the bilinear recurrent NN

Feedback NN

Global Feedback NN

Forward Feedback NN

Global Forward Feedback NN

Hybrid Feedback NN

Local Forward Feedback NN

Figure 4.1: Classification of FBNN.

a certain initial state, it always can converge to another stable equilibrium state. By designing the network weight, the equilibrium state of the system can be stored in the network.

4.2 Blind equalization algorithm based on the bilinear recurrent NN 4.2.1 Basic principles of algorithm Bilinear recurrent neural network (BLRNN) was proposed by Dong-Chul Park and Taekyun Jung Jeong [148] in 2002 and it is used in adaptive equalization algorithm. The network structure not only has the feed-forward term and feedback term but also includes a linear feedback term. This network has the advantages of both high-order neural network and recurrent neural network, so it can be used to approximate many kinds of nonlinear functions. In the high-order neural network algorithm, when the order increases, the calculation quantity exponentially increases, but BLRNN can overcome this problem so that it is easier to implement in hardware. In this section, BLRNN is combined with traditional constant modulus algorithm (CMA), and a new blind equalization algorithm based on BLRNN is studied. The principle diagram of BLRNN is shown in Figure 4.2 [148]. T

In Figure 4.2, the input of the network is Y(n) = [ y(n) y(n – 1) . . . y (n – k) ] , the ̃ output of the output layer is v(n), the total output of the network is x(n), the weight of the feedback unit is ai (i = 1, 2), the weight of feed-forward unit is cj (j = 0, 1, . . . , k), the weight of the linear feedback unit is bij , and the number of input neurons in the input unit is k + 1 (the number of the linear feedback units is k + 1). The input–output relationship function of BLRNN is

96

4 Research of Blind Equalization Algorithms Based on the FBNN

y(n) Feed-forward unit

y(n–1)

cj

ν(n) Σ

~ x(n)

f (.)

y(n–k)

bij

Linear feedback unit

Feedback unit

ai

z –1

z –1 Figure 4.2: The principle diagram of BLRNN. 2

2

k

k

̃ – i) + ∑ ∑ bij (n)y (n – j)x(n ̃ – i) + ∑ cj (n)y (n – j) v(n) = ∑ ai (n)x(n i=1

i=1 j=0

(4.1)

j=0

̃ x(n) = f [v(n)]

(4.2)

where f (⋅) is the transfer function of neural network. In the adaptive equalization algorithm, Dong-Chul Park and Tae-kyun Jung Jeong [148] adopted the sigmoid function.

4.2.2 Real blind equalization algorithm based on BLRNN 4.2.2.1 Derivation of algorithm The cost function and the transfer function are the same as those of the three-layer FFNN in eqs. (3.5) and (3.6). Using the steepest descent method, the iterative formula of network weights can be obtained: W(n + 1) = W(n) – ,∇

(4.3)

So, ∇=

̃ 𝜕J(n) 𝜕x(n) ̃ = 2 [x̃ 2 (n) – R2 ] x(n) 𝜕W(n) 𝜕W(n)

(4.4)

97

4.2 Blind equalization algorithm based on the bilinear recurrent NN

The bilinear recurrent neural network contains three different network structures: the feed-forward unit, the feedback unit, and the linear feedback unit, therefore the iteration formulas of weights are different. Weight iterative formula of the feed-forward unit The connection weight between the feed-forward unit and the output unit is cj (n), so ̃ 𝜕x(n) = f 󸀠 [v(n)] y(n – i) 𝜕cj (n)

(4.5)

The weight iterative formula of the feed-forward unit is ̃ cj (n + 1) = cj (n) – 2,c [x̃ 2 (n) – R2 ] f 󸀠 [v(n)] x(n)y (n – j)

(4.6)

where ,c is the weight iterative step size of the feed-forward unit. Weight iterative formula of the feedback unit The connection weight between the feedback unit and the output unit is ai (n), so ̃ 𝜕x(n) ̃ – i) = f 󸀠 [v(n)] x(n 𝜕ai (n)

(4.7)

The weight iterative formula of the feedback unit is ̃ x(n ̃ – i) ai (n + 1) = ai (n) – 2,a [x̃ 2 (n) – R2 ] f 󸀠 [v(n)] x(n)

(4.8)

where ,a is the weight iterative step size of the feedback unit. Weight iterative formula of the linear feedback unit The connection weight between the linear feedback unit and the output unit is bij (n), so ̃ 𝜕x(n) ̃ – i)y(n – j) = f 󸀠 [v(n)] x(n 𝜕bij (n)

(4.9)

The weight iterative formula of the linear feedback unit is ̃ x(n ̃ – i)y(n – j) bij (n + 1) = bij (n) – 2,b [x̃ 2 (n) – R2 ] f 󸀠 [v(n)] x(n)

(4.10)

where ,b is the weight iterative step size of the linear feedback unit. The BLRNN discussed above is a two-layer structure. When the hidden layer unit is contained, the derivation method is similar to the feed-forward neural network.

98

4 Research of Blind Equalization Algorithms Based on the FBNN

0.10

0.10 1

2

1 BLRNNR 2 CMA

1 BLRNNR 2 CMA

0.08

0.06

MSE

MSE

0.08

0.04

1

0.06

2

0.04 0.02

0.02

0

0 0 (a)

0

2,000 4,000 6,000 8,000 10,000 (b)

Iterative number

2,000 4,000 6,000 8,000 10,000 Iterative number

Figure 4.3: The convergence curves through the typical telephone channel and the ordinary channel: (a) the convergence curves through the typical telephone channel and (b) the convergence curves through the ordinary channel.

–1.0

CMA

–1.2 lg (BER)

–1.3 lg (BER)

BLRNNR CMA

BLRNNR

–1.2

–1.4 –1.5 –1.6

–1.4

–1.6

–1.7 –1.8 10 (a)

15

20 SNR (dB)

25

–1.8 10 (b)

15

20

25

SNR (dB)

Figure 4.4: The BER curves through the typical telephone channel and the ordinary channel: (a) the BER curves through the typical telephone channel and (b) the BER curves through the ordinary channel.

4.2.2.2 Computer simulation results The input signal is 8PAM. SNR is 20 dB. The simulation channels adopt the typical digital telephone channel and the ordinary channel. Their z-transforms are shown in eqs. (2.84) and (3.13). The convergence curves of 8PAM through two channels using the real bilinear feedback neural network (BLRNNR) algorithm and the traditional CMA are shown in Figure 4.3, where ,a = ,b = 5×10–7 , ,c = 0.001. Simulation results have shown that the BLRNNR algorithm has obvious faster convergence speed than the traditional CMA. The BER curves of 8PAM through two channels using BLRNNR algorithm and the traditional CMA are shown in Figure 4.4. The curves are after 10,000 iterations.

99

4.2 Blind equalization algorithm based on the bilinear recurrent NN

Simulation results have shown the BLRNNR algorithm has obvious lower BER than the traditional CMA.

4.2.3 Complex blind equalization algorithm based on BLRNN 4.2.3.1 Derivation of algorithm The algorithm derivation is similar to the complex three-layer feed-forward neural network (TFFNN). The network signal can be expressed in complex form: y(n) = yR (n) + jyI (n) 2

2

k

(4.11) k

̃ – i) + ∑ ∑ bij (n)y (n – j)x(n ̃ – i) + ∑ cj (n)y (n – j) v(n) = ∑ ai (n)x(n i=1

i=1 j=0

j=0

2

2

i=1

i=1

= ∑ [ai,R (n)x̃R (n – i) – ai,I (n)x̃I (n – i)] + j ∑ [ai,R (n)x̃I (n – i) + ai,I (n)x̃R (n – i)] k

2

+ ∑ ∑ bij,R (n) [yR (n – j) x̃R (n – i) – yI (n – j) x̃I (n – i)] i=1 j=0 2

k

– ∑ ∑ bij,I (n) [yR (n – j) x̃I (n – i) + yI (n – j) x̃R (n – i)] i=1 j=0 2

k

+ j ∑ ∑ bij,I (n) [yR (n – j) x̃R (n – i) – yI (n – j) x̃I (n – i)] i=1 j=0 2

k

+ j ∑ ∑ bij,R (n) [yR (n – j) x̃I (n – i) + yI (n – j) x̃R (n – i)] i=1 j=0 k

k

+ ∑ [cj,R (n)yR (n – j) – cj,I (n)yI (n – j)] + j ∑ [cj,R (n)yI (n – j) + cj,I (n)yR (n – j)] j=0

j=0

(4.12) ̃ x(n) = x̃R (n) + jx̃I (n) = f [vR (n)] + jf [vI (n)]

(4.13)

Defining the cost function of the complex BLRNN algorithm is eq. (3.6), so the weight iteration formula is W(n) = WR (n) + jWI (n) 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 𝜕J(n) 󵄨 ̃ 󵄨󵄨2 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) = 2 [󵄨󵄨󵄨x(n) +j 󵄨 ] ∇= 󵄨󵄨 – R2 ] 󵄨󵄨󵄨x(n) 󵄨󵄨 [ 𝜕W(n) 𝜕WR (n) 𝜕WI (n)

(4.14) (4.15)

100

4 Research of Blind Equalization Algorithms Based on the FBNN

Weight iterative formula of the feed-forward unit The connection weight between the feed-forward unit and the output unit is cj (n), so 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 𝜕 󵄨󵄨x(n) 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 = +j 󵄨 𝜕cj (n) 𝜕cj,R (n) 𝜕cj,I (n)

(4.16)

According to eqs. (4.11)–(4.13), the weight iterative formula of the feed-forward unit is cj (n + 1) = cj (n) – 2,c Ky∗ (n – j)

(4.17)

where ,c is the weight iterative step size of the feed-forward unit. And K 󵄨 ̃ 󵄨󵄨2 [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2 ] {f [vR (n)] f 󸀠 [vR (n)] + jf [vI (n)] f 󸀠 [vI (n)]}. The specific derivation process is shown in Appendix B.

=

Weight iterative formula of the feedback unit The connection weight between the feedback unit and the output unit is ai (n), so 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 𝜕 󵄨󵄨x(n) 𝜕 󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 󵄨󵄨 = 󵄨 +j 󵄨 𝜕ai (n) 𝜕ai,R (n) 𝜕ai,I (n)

(4.18)

According to eqs. (4.11)–(4.13), the weight iterative formula of the feedback unit is ai (n + 1) = ai (n) – 2,a K x̃ ∗ (n – i)

(4.19)

where ,a is the weight iterative step size of the feedback unit. The specific derivation process is shown in Appendix B. Weight iterative formula of the linear feedback unit The connection weight between the linear feedback unit and the output unit is bij (n), so 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 𝜕 󵄨󵄨x(n) 𝜕 󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 󵄨󵄨 = 󵄨 +j 󵄨 𝜕bij (n) 𝜕bij,R (n) 𝜕bij,I (n)

(4.20)

According to eqs. (4.11)–(4.13), the weight iterative formula of the linear feedback unit is bij (n + 1) = bij (n) – 2,b Ky∗ (n – j) x̃ ∗ (n – i)

(4.21)

where ,b is the weight iterative step size of the linear feedback unit. The specific derivation process is shown in Appendix B. If there is a hidden layer unit in the complex FBNN, the derivation method is similar to the complex blind equalization algorithm based on the feed-forward neural network.

4.2 Blind equalization algorithm based on the bilinear recurrent NN

1.0

2.0 1 BLRNNC 2 CMA

1

0.6

1.5 MSE

MSE

0.8

2

0.4

1 BLRNNC 2 CMA

1 2

1.0 0.5

0.2 0

101

0

(a)

1,000

2,000

3,000

0 (b)

Iterative number

0

1,000 2,000 3,000 4,000 5,000 Iterative number

Figure 4.5: The convergence curves through the typical telephone channel and the ordinary channel: (a) the convergence curves through the typical telephone channel and (b) the convergence curves through the ordinary channel.

–1.0 BLRNNC CMA lg (BER)

–1.5

–2.0

–2.5 10

15

20

25

SNR (dB)

Figure 4.6: The BER curves through the typical telephone channel.

4.2.3.2 Computer simulation results x

–x

–e The transfer function is f (x) = x + ! eex +e –x The input signal is 16QAM. SNR is 20 dB. The simulation channels adopt the typical digital telephone channel and the ordinary channel. Their z-transforms are shown in eqs. (2.84) and (3.13). The convergence curves of 16QAM through two channels using the complex bilinear feedback neural network (BLRNNC) algorithm and the traditional CMA are shown in Figure 4.5. When BLRNNC algorithm through the typical telephone channel, ,a = ,b = 3.0 × 10–7 , ,c = 0.001. When BLRNNC algorithm through the ordinary channel, ,a = ,b = 3.0 × 10–7 , ,c = 0.002. Simulation results have shown that the BLRNNC algorithm has obvious faster convergence speed than the traditional CMA. The BER curves of 16QAM through two channels using the BLRNNC algorithm and the traditional CMA are shown in Figures 4.6 and 4.7 respectively. The iterative number is 10,000. Simulation results have shown that in the same SNR condition,

102

4 Research of Blind Equalization Algorithms Based on the FBNN

BLRNNC

–2.0

CMA

lg (BER)

–2.2 –2.4 –2.6 –2.8 –3.0 10

15

20

SNR (dB)

1 BLRNNC 2 CMA

1.0 0.8 MSE

Figure 4.7: The BER curves through the ordinary channel.

1

2

0.6 0.4 0.2 0

0

1,000 2,000 3,000 4,000 5,000 Iterative number

Figure 4.8: The convergence curves through the wireless digital communication channel.

BLRNNC algorithm has much lower BER than the traditional CMA. This shows that BLRNNC algorithm has better ability of overcoming the intersymbol interference than the traditional CMA. The transfer function is f (x) = x + E [e2 (n)] sin 0x The input signal is 16QAM. SNR is 20 dB. The simulation channel adopts the wireless digital communication channel. Its z-transform is shown in eq. (3.33). The convergence curves of 16QAM through the wireless digital communication channel using the complex bilinear feedback neural network (BLRNNC) algorithm and the traditional CMA are shown in Figure 4.8. Where ,a = ,b = 1.0 × 10–5 , ,c = 0.004. Simulation results have shown that the BLRNNC algorithm has faster convergence speed and smaller SRE than the traditional CMA. The BER curves of 16QAM through the wireless digital communication channel using the BLRNNC algorithm and the traditional CMA are shown in Figure 4.9. The iterative number is 10,000. Simulation results have shown that in the same SNR condition, BLRNNC algorithm has lower BER than the traditional CMA. The convergence constellations of 16QAM through the wireless digital communication channel using the BLRNNC algorithm and the traditional CMA

4.3 Blind equalization algorithm based on the diagonal recurrent NN

103

–1.2 BLRNNC

lg (BER)

–1.4

CMA

–1.6 –1.8 –2.0 –2.2 10

15

20

25

SNR (dB)

Figure 4.9: The BER curves through the wireless digital communication channel.

4

4

2

2

0

0

–2

–2

–4 –4 (a)

–2

0

2

4

–4 –4

–2

0

2

4

(b)

Figure 4.10: The convergence constellations of the BLRNNC algorithm and the CMA. (a) the constellation of BLRNNC algorithm and (b) the constellation of CMA algorithm.

are shown in Figure 4.10. The iterative number is 10,000. Simulation results have shown that BLRNNC algorithm can recover the phase deflection but CMA can’t.

4.3 Blind equalization algorithm based on the diagonal recurrent NN 4.3.1 Model of diagonal recurrent NN The diagonal recurrent neural network (DRNN) was proposed in 1996 by C.C. Ku and Lee [149], which is a simplified form of the Elman network. In DRNN, each neuron in the hidden layer only receives feedback from own output, but doesn’t receive feedback

104

4 Research of Blind Equalization Algorithms Based on the FBNN

w1d(n) h (n) w11

y(n–1)

w1o(n) y(n–2) ~ x (n)

y(n–I + 1)

wJo(n)

wIJh(n)

y(n–I ) Input layer

wJd(n) Hidden layer

Out layer

Figure 4.11: The structure of DRNN.

from other neurons. Therefore, it has the characteristics of simple structure, dynamic memory, and dynamic mapping ability. Its structure is shown in Figure 4.11. In Figure 4.11, suppose that the connection weight between the input layer and the hidden layer is wijh (n)(i = 1, 2, . . . , I; j = 1, 2, . . . , J), the connection weight between the hidden layer and the output layer is wjo (n), the feedback weight of the recurrent layer is wjd (n). The input of the neuron is u, the output of the neuron is v. uIi is the ith input neuron of the Ith layer; viJ is the ith output neuron of the Jth layer. The input of ̃ the whole neural network is y(n – i), and the output is x(n), so the state functions of DRNN are defined as follows: uIi (n) = y(n – i)

(4.22)

viI (n)

(4.23)

=

uIi (n)

= y(n – i)

I

uJj (n) = ∑ wijh (n)y(n – i) + wjd (n)vjJ (n – 1)

(4.24)

i=1 I

vjJ (n) = f1 (uJi (n)) = f1 (∑ wijh (n)y(n – i) + wjd (n)vjJ (n – 1))

(4.25)

i=1 J

u(n) = ∑ wjo (n)vjJ (n)

(4.26)

j=1 J

̃ v(n) = x(n) = f2 (u(n)) = f2 (∑ wjo (n)vjJ (n))

(4.27)

j=1

where f1 (⋅) is the input–output transfer function of the hidden layer and f2 (⋅) is the input–output transfer function of the output layer.

105

4.3 Blind equalization algorithm based on the diagonal recurrent NN

4.3.2 Derivation of algorithm When the DRNN is used to solve the problem of the blind equalization, the cost function adopts eq. (3.6), and the transfer function adopts eq. (3.42). According to the steepest descent method, the iterative formula of network weights can be obtained: W(n + 1) = W(n) – ,

𝜕J(n) 𝜕W(n)

(4.28)

where , is the iteration step size ̃ 𝜕J(n) 𝜕x(n) ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] = 2x(n) 𝜕W(n) 𝜕W(n)

(4.29)

The DRNN has the input layer, the hidden layer, the self-feedback layer, and the output layer. So there are different weight iterative formulas.

4.3.2.1 Weight iteration formula of the output layer The connection weight between the output layer and the hidden layer is wjo (n), so ̃ 𝜕x(n) ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] o wjo (n + 1) = wjo (n) – 2,1 x(n) 𝜕wj (n)

(4.30)

J

̃ 𝜕x(n) = f2󸀠 (∑ wjo (n)vjJ (n)) vjJ (n) o 𝜕wj (n) j=1

(4.31) J

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f2󸀠 (∑ wjo (n)vjJ (n)) vjJ (n) wjo (n + 1) = wjo (n) – 2,1 x(n)

(4.32)

j=1

where ,1 is the weight iteration step size of the output layer.

4.3.2.2 Self-feedback weight iteration formula of the hidden layer The self-feedback connection weight of the hidden layer is wjd (n), so ̃ 𝜕x(n) ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] wjd (n + 1) = wjd (n) – 2,2 x(n) 𝜕wjd (n)

(4.33)

J

I ̃ 𝜕x(n) 󸀠 o J o 󸀠 = f ( w (n)v (n) ) w (n)f ( wijh (n)y(n – i) + wjd (n)vjJ (n – 1)) vjJ (n – 1) ∑ ∑ 2 j j 1 j 𝜕wjd (n) j=1 i=1

(4.34)

106

4 Research of Blind Equalization Algorithms Based on the FBNN

J

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f2󸀠 (∑ wjo (n)vjJ (n)) wjd (n + 1) = wjd (n) – 2,2 x(n) j=1

(4.35)

I

× f1󸀠 (∑ wijh (n)y(n – i) + wjd (n)vjJ (n – 1)) wjo (n)vjJ (n – 1) i=1

where ,2 is the self-feedback weight iteration step size of the hidden layer. 4.3.2.3 Weight iteration formula of the hidden layer The connection weight between the input layer and the hidden layer is wijh (n), so ̃ 𝜕x(n) ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] wijh (n + 1) = wijh (n) – 2,3 x(n) h 𝜕wij (n)

(4.36)

J

I ̃ 𝜕x(n) 󸀠 o J o 󸀠 = f ( w (n)v (n) ) w (n)f ( wijh (n)y(n – i) + wjd (n)vjJ (n – 1)) y(n – i) ∑ ∑ 2 j j 1 j 𝜕wijh (n) j=1 i=1

(4.37)

J

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f2󸀠 (∑ wjo (n)vjJ (n)) wijh (n + 1) = wijh (n) – 2,3 x(n) j=1

×

I f1󸀠 (∑ wijh (n)y(n i=1

– i) +

wjd (n)vjJ (n

–

(4.38) 1)) wjo (n)y(n

– i)

where ,3 is the weight iteration step size of the hidden layer.

4.3.3 Computer simulation results The input signals are 4PAM and 8PAM. The additive Gauss white noise is zero mean and unit variance. The simulation channels adopt the typical telephone channel and the ordinary channel. Their z-transforms are shown in eqs. (2.84) and (3.13). When 4PAM and 8PAM signals go through the typical digital telephone channel, the iterative step sizes are ,1 = ,2 = ,3 = 0.00008 and ,1 = ,2 = ,3 = 0.000002 respectively. When 4PAM and 8PAM signals go through the ordinary channel, the iterative step sizes are ,1 = ,2 = ,3 = 0.00008 and ,1 = ,2 = ,3 = 0.0000025, respectively. In the simulation of 4PAM signal, the 3-3-1 type neural network is adopted. The initialization weights are designed as below. T

(1)

0.04 0.00 0.15 [ ] From the input layer to the hidden layer [ 0.01 0.81 0.76 ] ; [ 0.03 –0.40 0.90 ]

4.4 Blind equalization algorithm based on the quasi-DRNN

107

0.10 1 TFFNN 2 DRNN 1

0.04

(a)

0.06 2

0.04

2

0.02 0

1

0.08

0.06

1 TFFNN 2 DRNN

0.10

MSE

MSE

0.08

0.02 0

1,000 2,000 3,000 4,000 5,000 Iterative number

0 (b)

0

1,000 2,000 3,000 4,000 5,000 Iterative number

Figure 4.12: The convergence curves of 4PAM and 8PAM through the typical telephone channel: (a) The convergence curves of 4PAM and (b) The convergence curves of 8PAM. T

(2)

The recurrent layer [ 0.07 0.50 0.90 ] ;

(3)

From the hidden layer to the output layer [ 1.10 0.70 0.90 ] .

T

In the simulation of 8PAM signal, the 3-3-1 type neural network is adopted. The initialization weights are designed as below. T

(2)

0.05 0.00 0.10 [ ] From the input layer to the hidden layer [ 0.01 0.80 0.70 ] ; [ 0.01 –0.50 0.90 ] T The recurrent layer [ 0.10 0.50 0.90 ] ;

(3)

From the hidden layer to the output layer [ 1.00 0.60 0.90 ] .

(1)

T

When two kinds of signals go through two channels, the convergence curves and the BER curves are shown from Figure 4.12 to 4.15, respectively. The BER curves are obtained in different SNR after 5,000 iterations. Simulation results have shown that DRNN algorithm has faster convergence speed and lower BER than the TFFNN algorithm. But the steady residual errors are similar in two algorithms.

4.4 Blind equalization algorithm based on the quasi-DRNN 4.4.1 Model of quasi-DRNN The structure of the DRNN is simple, but there is no interconnection between neurons of hidden layer. Now it has been proved that there are interconnections between the biological neurons, and connections between different neurons are very complex and diverse ways. There are close connections, loose connections, and no connection

108

4 Research of Blind Equalization Algorithms Based on the FBNN

0.10

0.12 1 TFFNN 2 DRNN

0.08

2

0.06 2

0.04

0.02 0

1

0.08

0.06 0.04

1 TFFNN 2 DRNN

0.10

MSE

MSE

1

0.02 0

0

1,000 2,000 3,000 4,000 5,000

(a)

Iterative number

0

1,000 2,000 3,000 4,000 5,000

(b)

Iterative number

Figure 4.13: The convergence curves of 4PAM and 8PAM through the ordinary channel: (a) the convergence curves of 4PAM and (b) the convergence curves of 8PAM.

1.0

6

BER × 10–5

0.6

1 TFFNN 2 DRNN BER × 10–5

1 TFFNN 2 DRNN

0.8 1

0.4 0.2 0 19

4 1 2

2 20

(a)

2 22

21

0 19

23

SNR (dB)

20

21

22

23

SNR (dB)

(b)

Figure 4.14: The BER curves of 4PAM and 8PAM through the typical telephone channel: (a) the BER curves of 4PAM and (b) the BER curves of 8PAM.

12

6

BER × 10–6

10 8

BER × 10–6

1 TFFNN 2 DRNN 1

6

2

4

1 2

4

2

2 19 (a)

1 TFFNN 2 DRNN

20

21 SNR (dB)

22

23

0 19 (b)

20

21

22

23

SNR (dB)

Figure 4.15: The BER curves of 4PAM and 8PAM through the ordinary channel: (a) the BER curves of 4PAM and (b) the BER curves of 8PAM.

4.4 Blind equalization algorithm based on the quasi-DRNN

109

w1d(n) h (n) w11

y(n–1)

w1o(n)

y (n–2)

d w12 (n)

~ x (n)

y (n–I + 1)

wJo(n)

wIJh(n)

y (n–I ) Input layer

wJd(n) Hidden layer

Out layer

Figure 4.16: Structure of QDRNN.

between neurons at all, which led to the behavior diversely. Therefore, Li Hong-ru et al. [150] improved the DRNN structure according to the biological neuron characteristics and the neural network application need. The connections between the hidden layer adjacent neurons were added, and then the quasi-DRNN (QDRNN) was obtained. Its structure is shown in Figure 4.16. In Figure 4.16, suppose that the connection weight from the input layer to the hidden layer is wijh (n)(i = 1, 2, . . . , I; j = 1, 2, . . . , J), the connection weight from the hidden layer to the output layer is wjo (n), the weights of the recurrent layer are wjd (n), d d wj,j+1 (n)( j = 1, 2, . . . , J – 1) and wj–1,j (n)( j = 2, 3, . . . , J). The input of the neuron is u,

the output of the neuron is v. uIi is the ith input neuron of the Ith layer; viJ is the ith output neuron of the Jth layer. The input of the whole neural network is y(n – i), and ̃ the output is x(n), so the state functions of QDRNN are defined as below. uIi (n) = y(n – i)

(4.39)

viI (n) = uIi (n) = y(n – i)

(4.40)

I

d J (n)vj+1 (n – 1) uJj (n) = ∑ wijh (n)y(n – i) + wjd (n)vjJ (n – 1) + wj,j+1 i=1 d J (n)vj–1 (n – 1) + wj–1,j

(4.41)

I

vjJ (n) = f1 (uJi (n)) = f1 (∑ wijh (n)y(n – i) + wjd (n)vjJ (n – 1) i=1

+

d J (n)vj+1 (n wj,j+1

– 1) +

(4.42) d J wj–1,j (n)vj–1 (n

– 1))

110

4 Research of Blind Equalization Algorithms Based on the FBNN

J

u(n) = ∑ wjo (n)vjJ (n)

(4.43)

j=1 J

̃ v(n) = x(n) = f2 (u(n)) = f2 (∑ wjo (n)vjJ (n))

(4.44)

j=1

where f1 (⋅) is the input–output transfer function of the hidden layer and f2 (⋅) is the input–output transfer function of the output layer.

4.4.2 Derivation of algorithm In the blind equalization algorithm based on QDRNN, the cost function and the transfer function are identical with those of DRNN algorithm. They are shown in eqs. (3.6) and (3.42). The QDRNN has the input layer, the hidden layer, the self-feedback layer, and the output layer. So there are different weight iterative formulas. 4.4.2.1 Connection weight iteration formula of the output layer The connection weight between the hidden layer and the output layer is wjo (n), so ̃ 𝜕x(n) ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] o wjo (n + 1) = wjo (n) – 2,1 x(n) 𝜕wj (n)

(4.45)

J

̃ 𝜕x(n) = f2󸀠 (∑ wjo (n)vjJ (n)) vjJ (n) 𝜕wjo (n) j=1

(4.46) J

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f2󸀠 (∑ wjo (n)vjJ (n)) vjJ (n) wjo (n + 1) = wjo (n) – 2,1 x(n)

(4.47)

j=1

where ,1 is the weight iteration step size of the output layer. 4.4.2.2 Feedback weight iteration formula of the hidden layer d (n), and The feedback connection weights of the hidden layer are wjd (n), wj,j+1 d wj–1,j (n), so

̃ 𝜕x(n) ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] wjd (n + 1) = wjd (n) – 2,2 x(n) d 𝜕wj (n)

(4.48)

d d ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] wj,j+1 (n + 1) = wj,j+1 (n) – 2,3 x(n)

̃ 𝜕x(n) d 𝜕wj,j+1 (n)

(4.49)

d d ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] wj–1,j (n + 1) = wj–1,j (n) – 2,4 x(n)

̃ 𝜕x(n) d 𝜕wj–1,j (n)

(4.50)

4.4 Blind equalization algorithm based on the quasi-DRNN

111

J

̃ 𝜕x(n) = f2󸀠 (∑ wjo (n)vjJ (n)) wjo (n) 𝜕wjd (n) j=1 J

× f1󸀠 (∑ wijh (n)y(n – i) + wjd (n)vjJ (n – 1)

(4.51)

j=1 d J d J + wj,j+1 (n)vj+1 (n – 1) + wj–1,j (n)vj–1 (n – 1)) vjJ (n – 1)

J

̃ 𝜕x(n) = f2󸀠 (∑ wjo (n)vjJ (n)) wjo (n) d 𝜕wj,j+1 (n) j=1 J

× f1󸀠 (∑ wijh (n)y(n – i) + wjd (n)vjJ (n – 1)

(4.52)

j=1 d J d J J + wj,j+1 (n)vj+1 (n – 1) + wj–1,j (n)vj–1 (n – 1)) vj+1 (n – 1)

J

̃ 𝜕x(n) = f2󸀠 (∑ wjo (n)vjJ (n)) wjo (n) d (n) 𝜕wj–1,j j=1 J

× f1󸀠 (∑ wijh (n)y(n – i) + wjd (n)vjJ (n – 1)

(4.53)

j=1 d J d J J + wj,j+1 (n)vj+1 (n – 1) + wj–1,j (n)vj–1 (n – 1)) vj–1 (n – 1)

J

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f2󸀠 (∑ wjo (n)vjJ (n)) wjd (n + 1) = wjd (n) – 2,2 x(n) j=1

I

d J × f1󸀠 (∑ wijh (n)y(n – i) + wjd (n)vjJ (n – 1) + wj,j+1 (n)vj+1 (n – 1)

(4.54)

i=1 d J (n)vj–1 (n – 1)) wjo (n)vjJ (n – 1) + wj–1,j

J

d d ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f2󸀠 (∑ wjo (n)vjJ (n)) (n + 1) = wj,j+1 (n) – 2,3 x(n) wj,j+1 j=1

I

d J × f1󸀠 (∑ wijh (n)y(n – i) + wjd (n)vjJ (n – 1) + wj,j+1 (n)vj+1 (n – 1) i=1 d J J (n)vj–1 (n – 1)) wjo (n)vj+1 (n – 1) + wj–1,j

(4.55)

112

4 Research of Blind Equalization Algorithms Based on the FBNN

J

d d ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f2󸀠 (∑ wjo (n)vjJ (n)) wj–1 (n + 1) = wj–1 (n) – 2,4 x(n) j=1

I

d J × f1󸀠 (∑ wijh (n)y(n – i) + wjd (n)vjJ (n – 1) + wj,j+1 (n)vj+1 (n – 1)

(4.56)

i=1 d J J (n)vj–1 (n – 1)) wjo (n)vj–1 (n – 1) + wj–1,j

where ,2 is the self-feedback weight iteration step size of the hidden layer. ,3 and ,4 are the mutual-feedback weight iteration step sizes of the hidden layer. 4.4.2.3 Connection weight iteration formula of the hidden layer The connection weight between the input layer and the hidden layer is wijh (n), so ̃ 𝜕x(n) ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] wijh (n + 1) = wijh (n) – 2,5 x(n) h 𝜕wij (n)

(4.57)

J

̃ 𝜕x(n) = f2󸀠 (∑ wjo (n)vjJ (n)) wjo (n) 𝜕wijh (n) j=1 I

× f1󸀠 (∑ wijh (n)y(n – i) + wjd (n)vjJ (n – 1)

(4.58)

i=1 d J d J (n)vj+1 (n – 1) + wj–1,j (n)vj–1 (n – 1)) y(n – i) + wj,j+1 J

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f2󸀠 (∑ wjo (n)vjJ (n)) wijh (n + 1) = wijh (n) – 2,5 x(n) j=1

I

d J × f1󸀠 (∑ wijh (n)y(n – i) + wjd (n)vjJ (n – 1) + wj,j+1 (n)vj+1 (n – 1)

(4.59)

i=1 d J (n)vj–1 (n – 1)) wjo (n)y(n – i) +wj–1,j

where ,5 is the weight iteration step size of the hidden layer.

4.4.3 Computer simulation results The input signals are 4PAM and 8PAM. The additive Gauss white noise is zero mean and unit variance. The simulation channels adopt the typical telephone channel and the ordinary channel. Their z-transforms are shown in eqs. (2.84) and (3.13).

4.4 Blind equalization algorithm based on the quasi-DRNN

113

When 4PAM and 8PAM signals go through the typical digital telephone channel, the iterative step sizes are ,1 = ,2 = ,3 = ,4 = ,5 = 0.00016 and ,1 = ,2 = ,3 = ,4 = ,5 = 0.0000055, respectively. When 4PAM and 8PAM signals go through the ordinary channel, the iterative step sizes are ,1 = ,2 = ,3 = ,4 = ,5 = 0.00006 and ,1 = ,2 = ,3 = ,4 = ,5 = 0.000007 respectively. In the simulation of 4PAM signal, the 3-3-1 type neural network is adopted. The initialization weights are designed as follows. T

(2)

0.04 0.00 0.15 [ ] From the input layer to the hidden layer [ 0.00 0.80 0.70 ] , [ 0.00 –0.50 0.90 ] T The recurrent layer [ 0.07 0.50 0.90 ] ,

(3)

Between recurrent layers [ 1.00 1.00 1.00 1.00 ],

(4)

From the hidden layer to the output layer [ 1.00 0.60 0.90 ] .

(1)

T

In the simulation of 8PAM signal, the 3-3-1 type neural network is adopted. The initialization weights are designed as follows. T

(2)

0.04 –0.10 0.15 [ ] From the input layer to the hidden layer [ 0.01 0.80 0.70 ] , [ 0.03 –0.50 0.90 ] T The recurrent layer [ 0.05 0.50 0.85 ] ,

(3)

Between recurrent layers [ 1.00 1.00 1.00 1.00 ],

(4)

From the hidden layer to the output layer [ 1.00 0.50 1.00 ] .

(1)

T

When two kinds of signals go through two channels, the convergence curves and the BER curves are shown respectively from Figures 4.17 to 4.20, respectively. The BER

0.10

0.12 1 TFFNN 2 QDRNN

0.08

0.06 1

0.04

(a)

1

0.06

2

0.04 2

0.02 0

1 TFFNN 2 QDRNN

0.10

MSE

MSE

0.08

0

1,000 2,000 3,000 4,000 5,000 Iterative number

0.02 0 (b)

0

1,000 2,000 3,000 4,000 5,000 Iterative number

Figure 4.17: The convergence curves of 4PAM and 8PAM through the typical telephone channel: (a) the convergence curves of 4PAM and (b) the convergence curves of 8PAM.

114

4 Research of Blind Equalization Algorithms Based on the FBNN

0.10

0.12 (1) TFFNN (2) QDRNN (1)

0.06

0.08

0.04

(1)

0.06

(2)

0.04

(2) 0.02 0 0

(1) TFFNN (2) QDRNN

0.10

MSE

MSE

0.08

0.02 0

1,000 2,000 3,000 4,000 5,000

(a)

Iterative number

0

1,000 2,000 3,000 4,000 5,000

(b)

Iterative number

Figure 4.18: The convergence curves of 4PAM and 8PAM through the ordinary channel: (a) The convergence curves of 4PAM and (b) The convergence curves of 8PAM.

1.0

5 (1) TFFNN (2) QDRNN

0.6

(1) (2)

0.4 0.2 0 19

20

(a)

21

22

(1) TFFNN (2) QDRNN

4 BER × 10–5

BER × 10–5

0.8

23

SNR (dB)

(1)

3 2 1

(2)

0 19

20

21

(b)

22

23

SNR (dB)

Figure 4.19: The BER curves of 4PAM and 8PAM through the typical telephone channel: (a) The BER curves of 4PAM and (b) The BER curves of 8PAM.

12

6 (1) TFFNN (2) QDRNN

8

(1)

6 4 2

(a)

(1) TFFNN (2) QDRNN

(1) BER × 10–5

BER × 10–6

10

4 (2) 2

(2) 19

20

21 SNR (dB)

22

23

0 19 (b)

20

21

22

23

SNR (dB)

Figure 4.20: The BER curves of 4PAM and 8PAM through the ordinary channel: (a) the BER curves of 4PAM and (b) the BER curves of 8PAM.

4.5 Blind equalization algorithm based on the variable step-size DRNN

115

curves are obtained in different SNR after 5,000 iterations. Simulation results have shown that QDRNN algorithm has faster convergence speed and lower BER than the TFFNN algorithm [17]. But the steady residual error is not obviously improved.

4.5 Blind equalization algorithm based on the variable step-size DRNN 4.5.1 Basic principles of algorithm In the DRNN blind equalization algorithm, the fixed step size is adopted. If the variable step size ,(n) is used to replace the fixed step size, the blind equalization algorithm based on the variable step-size DRNN is obtained. ,(n) is a nonlinear function about mean square error.

4.5.2 Derivation of algorithm Plugging expression (3.101) of ,(n) into iterative formulas (4.32), (4.35), and (4.38) of the blind equalization algorithm based on DRNN, the iterative formulas of each layer are shown as follows. 4.5.2.1 Connection weight iteration formula of the output layer J

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f2󸀠 (∑ wjo (n)vjJ (n)) vjJ (n) wjo (n + 1) = wjo (n) – 2,(n)x(n)

(4.60)

j=1

4.5.2.2 Self-feedback weight iteration formula of the hidden layer J

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f2󸀠 (∑ wjo (n)vjJ (n)) wjd (n + 1) = wjd (n) – 2,(n)x(n) j=1

I

×

f1󸀠 (∑ wijh (n)y(n i=1

– i) +

wjd (n)vjJ (n

–

1)) wjo (n)vjJ (n

(4.61) – 1)

4.5.2.3 Connection weight iteration formula of the hidden layer J

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f2󸀠 (∑ wjo (n)vjJ (n)) wijh (n + 1) = wijh (n) – 2,(n)x(n) j=1

I

× f1󸀠 (∑ wijh (n)y(n – i) + wjd (n)vjJ (n – 1)) wjo (n)y(n – i) i=1

(4.62)

116

4 Research of Blind Equalization Algorithms Based on the FBNN

4.5.3 Computer simulation results The input signals are 4PAM and 8PAM. The additive Gauss white noise is zero mean and unit variance. The simulation channels adopt the typical telephone channel and the ordinary channel. Their z-transforms are shown in eqs. (2.84) and (3.13). When 4PAM and 8PAM signals go through the typical telephone channel, the variable step-size parameters are ! = 0.0002, " = 35 and ! = 0.00002, " = 41 respectively. The fixed step size are ,1 = ,2 = ,3 = 0.00008 and ,1 = ,2 = ,3 = 0.000002, respectively. When 4PAM and 8PAM signals go through the ordinary channel, the variable stepsize parameters are ! = 0.00015, " = 30 and ! = 0.00001, " = 30, respectively. The fixed step size are ,1 = ,2 = ,3 = 0.00008 and ,1 = ,2 = ,3 = 0.0000025, respectively. In the simulation of 4PAM signal, the 3-3-1 type neural network is adopted. The initialization weights are designed as follows: T

(2)

0.04 0.00 0.15 ] [ ] From the input layer to the hidden layer [ [ 0.01 0.81 0.76 ] , [ 0.03 –0.40 0.90 ] T The recurrent layer [ 0.07 0.50 0.90 ] ,

(3)

From the hidden layer to the output layer [ 1.10 0.70 0.90 ] .

(1)

T

In the simulation of 8PAM signal, the 3-3-1 type neural network is adopted. The initialization weights are designed as follows: T

0.05 0.00 0.10

(1)

] [ ] From the input layer to the hidden layer [ [ 0.01 0.80 0.70 ] ; [ 0.01 –0.50 0.90 ] T

(2)

The recurrent layer [ 0.10 0.50 0.90 ] ;

(3)

From the hidden layer to the output layer [ 1.00 0.60 0.90 ] .

T

The variable curves of ,(n) are shown in Figures 4.21 and 4.22. When two kinds of signals go through two channels, the convergence curves and the BER curves are shown from Figures 4.23 to 4.26 respectively. Simulation results have shown that with the increase of the iteration number, the iterative step size decreases gradually, which meets the design requirements. At the same time, the blind equalization algorithm based on the variable step-size DRNN (VSDRNN) has faster convergence speed, lower BER, and smaller SRE than the blind equalization algorithm based on the DRNN.

4.5 Blind equalization algorithm based on the variable step-size DRNN

117

Step size μ × 10–5

Step size μ × 10–4

2.0

1

0

0

1.5 1.0 0.5 0

1,000 2,000 3,000 4,000 5,000 Iterative number

(a)

0

(b)

1,000 2,000 3,000 4,000 5,000 Iterative number

Step size μ × 10–5

Step size μ × 10–4

Figure 4.21: The learning rate change curves of 4PAM and 8PAM through the typical telephone channel : (a) the step-size change curves of 4PAM and (b) the step-size change curves of 8PAM.

1

0

0

1

0

1,000 2,000 3,000 4,000 5,000 Iterative number

(a)

2

0

(b)

1,000 2,000 3,000 4,000 5,000 Iterative number

Figure 4.22: The learning rate change curves of 4PAM and 8PAM through the ordinary channel: (a) The step-size change curves of 4PAM and (b) The step-size change curves of 8PAM.

0.08

0.10 (1) DRNN (2) VSDRNN

0.04 (1) 0.02 0 (a)

(2) 0

1,000 2,000 3,000 4,000 5,000 Iterative number

(1) DRNN (2) VSDRNN

0.08 MSE

MSE

0.06

0.06 (2)

(1)

0.04 0.02 0 (b)

0

1,000 2,000 3,000 4,000 5,000 Iterative number

Figure 4.23: The convergence curves of 4PAM and 8PAM through the typical telephone channel: (a) the convergence curves of 4PAM and (b) the convergence curves of 8PA.

118

4 Research of Blind Equalization Algorithms Based on the FBNN

0.08

0.10 (1) DRNN (2) VSDRNN

0.06 (1)

0.04

(2)

0.02 0

0

0.04

(1)

(2)

0.02 0

1,000 2,000 3,000 4,000 5,000

(a)

(1) DRNN (2) VSDRNN

0.06 MSE

MSE

0.08

Iterative number

0

1,000 2,000 3,000 4,000 5,000

(b)

Iterative number

Figure 4.24: The convergence curves of 4PAM and 8PAM through the ordinary channel: (a) the convergence curves of 4PAM and (b) the convergence curves of 8PAM.

6

8 (1)

(1) DRNN (2) VSDRNN BER × 10–5

BER × 10–6

6 4 2 0 19

(1) DRNN (2) VSDRNN

4 (1)

2

(2) 20

(a)

21

22

0 19

23

SNR (dB)

(2) 20

21

(b)

22

23

SNR (dB)

Figure 4.25: The BER curves of 4PAM and 8PAM through the typical telephone channel: (a) The BER curves of 4PAM and (b) The BER curves of 8PAM.

4

10 (1) DRNN (2) VSDRNN (1) 6

(2)

(a)

(1) 2 1

4 2 19

(1) DRNN (2) VSDRNN

3 BER × 10–5

BER × 10–6

8

20

21 SNR (dB)

22

23

0 19 (b)

(2) 20

21

22

23

SNR (dB)

Figure 4.26: The BER curves of 4PAM and 8PAM through the ordinary channel: (a) the BER curves of 4PAM and (b) the BER curves of 8PAM.

4.6 Blind equalization algorithm based on the variable step-size QDRNN

119

4.6 Blind equalization algorithm based on the variable step-size QDRNN 4.6.1 Basic principles of algorithm In the QDRNN blind equalization algorithm, the fixed step size is adopted. If the variable step size ,(n) is used to replace the fixed step size, the blind equalization algorithm based on the variable step-size QDRNN is obtained. ,(n) is a nonlinear function about mean square error.

4.6.2 Derivation of algorithm Plugging expression (3.101) of ,(n) into iterative formulas (4.47), (4.54)–(4.56), and (4.59) of the blind equalization algorithm based on QDRNN, the iterative formulas of each layer are shown as follows:

4.6.2.1 Connection weight iteration formula of the output layer J

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f2󸀠 (∑ wjo (n)vjJ (n)) vjJ (n) wjo (n + 1) = wjo (n) – 2,(n)x(n)

(4.63)

j=1

4.6.2.2 Feedback weight iteration formula of the hidden layer J

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f2󸀠 (∑ wjo (n)vjJ (n)) wjd (n + 1) = wjd (n) – 2,(n)x(n) j=1

I

d J × f1󸀠 (∑ wijh (n)y(n – i) + wjd (n)vjJ (n – 1) + wj,j+1 (n)vj+1 (n – 1)

(4.64)

i=1 d J (n)vj–1 (n – 1)) wjo (n)vjJ (n – 1) + wj–1,j J

d d ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f2󸀠 (∑ wjo (n)vjJ (n)) (n + 1) = wj,j+1 (n) – 2,(n)x(n) wj,j+1 j=1

I

d J × f1󸀠 (∑ wijh (n)y(n – i) + wjd (n)vjJ (n – 1) + wj,j+1 (n)vj+1 (n – 1) i=1 d J J (n)vj–1 (n – 1)) wjo (n)vj+1 (n – 1) + wj–1,j

(4.65)

120

4 Research of Blind Equalization Algorithms Based on the FBNN

J

d d ̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f2󸀠 (∑ wjo (n)vjJ (n)) wj–1 (n + 1) = wj–1 (n) – 2,(n)x(n) j=1

I

d J × f1󸀠 (∑ wijh (n)y(n – i) + wjd (n)vjJ (n – 1) + wj,j+1 (n)vj+1 (n – 1)

(4.66)

i=1 d J J (n)vj–1 (n – 1)) wjo (n)vj–1 (n – 1) + wj–1,j

4.6.2.3 Connection weight iteration formula of the hidden layer J

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f2󸀠 (∑ wjo (n)vjJ (n)) wijh (n + 1) = wijh (n) – 2,(n)x(n) j=1

I

d J × f1󸀠 (∑ wijh (n)y(n – i) + wjd (n)vjJ (n – 1) + wj,j+1 (n)vj+1 (n – 1)

(4.67)

i=1 d J (n)vj–1 (n – 1)) wjo (n)y(n – i) +wj–1,j

4.6.3 Computer simulation results The input signals are 4PAM and 8PAM. The additive Gauss white noise is zero mean and unit variance. The simulation channels adopt the typical telephone channel and the ordinary channel. Their z-transforms are shown in eqs. (2.84) and (3.13). When 4PAM and 8PAM signals go through the typical telephone channel, the variable step-size parameters are ! = 0.0003, " = 35 and ! = 0.00002, " = 40 respectively. The fixed step sizes are ,1 = ,2 = ,3 = ,4 = ,5 = 0.00016 and ,1 = ,2 = ,3 = ,4 = ,5 = 0.0000055, respectively. When 4PAM and 8PAM signals go through the ordinary channel, the variable stepsize parameters are ! = 0.0002, " = 40 and ! = 0.00001, " = 30, respectively. The fixed step sizes are ,1 = ,2 = ,3 = ,4 = ,5 = 0.00006 and ,1 = ,2 = ,3 = ,4 = ,5 = 0.000007, respectively. In the simulation of 4PAM signal, the 3-3-1 type neural network is adopted. The initialization weights are designed as follows: T

(2)

0.04 0.00 0.15 [ ] From the input layer to the hidden layer [ 0.00 0.80 0.70 ] , [ 0.00 –0.50 0.90 ] T The recurrent layer [ 0.07 0.50 0.90 ] ,

(3)

Between the recurrent layers [ 1.00 1.00 1.00 1.00 ],

(4)

From the hidden layer to the output layer [ 1.00 0.60 0.90 ] .

(1)

T

4.6 Blind equalization algorithm based on the variable step-size QDRNN

121

In the simulation of 8PAM signal, the 3-3-1 type neural network is adopted. The initialization weights are designed as follows. T

(2)

0.04 –0.01 0.15 [ ] From the input layer to the hidden layer [ 0.01 0.80 0.70 ] , [ 0.03 –0.50 0.90 ] T The recurrent layer [ 0.05 0.50 0.85 ] ,

(3)

Between the recurrent layers [ 1.00 1.00 1.00 1.00 ],

(4)

From the hidden layer to the output layer [ 1.00 0.50 1.00 ] .

(1)

T

The variable step-size curves of ,(n) are shown in Figures 4.27 and 4.28. When two kinds of signals go through two channels, the convergence curves and the BER curves are shown from Figures 4.29 to 4.32, respectively.

Step size μ × 10–5

Step size μ × 10–4

2.0

2

1

0

0 (a)

1.5 1.0 0.5 0

1,000 2,000 3,000 4,000 5,000 Iterative number

0

(b)

1,000 2,000 3,000 4,000 5,000 Iterative number

Figure 4.27: The learning rate change curves of 4PAM and 8PAM through the typical telephone channel: (a) the step-size change curves of 4PAM and (b) the step-size change curves of 8PAM.

Step size μ × 10–5

Step size μ × 10–4

1.0

1

0

0 (a)

0.8 0.6 0.4 0.2 0

1,000 2,000 3,000 4,000 5,000 Iterative number

0 (b)

1,000 2,000 3,000 4,000 5,000 Iterative number

Figure 4.28: The learning rate change curves of 4PAM and 8PAM through the ordinary channel: (a) The step-size change curves of 4PAM and (b) The step-size change curves of 8PAM.

122

4 Research of Blind Equalization Algorithms Based on the FBNN

0.10

0.10 (2) (1)

0.06

(1) QDRNN (2) VSQDRNN

0.08 (2) MSE

MSE

0.08

(1) QDRNN (2) VSQDRNN

0.04

0.06 0.04 (1)

0.02

0.02 0 0

0 0

1,000 2,000 3,000 4,000 5,000

(a)

Iterative number

1,000 2,000 3,000 4,000 5,000

(b)

Iterative number

Figure 4.29: The convergence curves of 4PAM and 8PAM through the typical telephone channel: (a) the convergence curves of 4PAM and (b) the convergence curves of 8PAM.

0.06

0.10 (1) QDRNN (2) VSQDRNN

0.05

MSE

MSE

0.04 0.03 (2)

0.02

0

0.06

0.02 0

1,000 2,000 3,000 4,000 5,000

(a)

(2)

0.04

(1)

(1) 0.01 0

(1) QDRNN (2) VSQDRNN

0.08

Iterative number

0

1,000 2,000 3,000 4,000 5,000

(b)

Iterative number

Figure 4.30: The convergence curves of 4PAM and 8PAM through the ordinary channel: (a) the convergence curves of 4PAM and (b) the convergence curves of 8PAM.

5

8 (1) QDRNN (2) VSQDRNN (1)

4 2

(a)

19

20

3 2

(1)

1

(2) 0

(1) QDRNN (2) VSQDRNN

4 BER × 10–5

BER × 10–6

6

21 SNR (dB)

22

23

0 (b)

(2) 19

20

21

22

23

SNR (dB)

Figure 4.31: The BER curves of 4PAM and 8PAM through the typical telephone channel: (a) the BER curves of 4PAM and (b) the BER curves of 8PAM.

4.7 Summary

1.0

3 (1) QDRNN (2) VSQDRNN BER × 10–5

(1) QDRNN (2) VSQDRNN

0.8 BER × 10–5

123

(1)

0.6 0.4

(2)

2 (1) 1

(2)

0.2 0 (a)

19

20

21 SNR (dB)

22

23

0 19 (b)

20

21

22

23

SNR (dB)

Figure 4.32: The BER curves of 4PAM and 8PAM through the ordinary channel: (a) The BER curves of 4PAM and (b) the BER curves of 8PAM.

Simulation results have shown that the blind equalization algorithm based on the variable step-size QDRNN (VSQDRNN) has better convergence performance than the blind equalization algorithm based on the QDRNN.

4.7 Summary In this chapter, the blind equalization algorithm based on the FBNN is mainly studied. The BLRNN, DRNN, and QDRNN are applied to the blind equalization algorithm, replacing the traditional transversal filter, so the blind equalization algorithms based on the BLRNN, DRNN, and QDRNN are obtained. At the same time, the variable stepsize method is introduced into the blind equalization algorithm based on the FBNN. The nonlinear function about MSE is used as the step-size control factor, so the blind equalization algorithms based on the variable step-size DRNN and variable step-size QDRNN are obtained. The simulation results verify the effectiveness.

5 Research of Blind Equalization Algorithms Based on FNN Abstract: In this chapter, the concept, development, structures, learning algorithms, and characteristics of the fuzzy neural network (FNN) are summarized. The methods of how to select and determine the fuzzy membership function are introduced. The blind equalization algorithms based on FNN filter, FNN controller, and FNN classifier are researched. The structures adopt the dynamic recurrent FNN, five-layer FNN, and three-layer FNN, respectively. The iterative formulas of algorithms are deduced. The simulation results verify the effectiveness of the proposed algorithms.

5.1 Basic principles of FNN Fuzzy neural network (FNN) is an adaptive and self-learning fuzzy system which combines fuzzy technology with neural network. It can process the fuzzy information automatically. The neural network imitates the hardware of the human brain, and the fuzzy technique imitates the software of the human brain. FNN set learning, association, identification, adaptation, and fuzzy information processing into one. Currently, it is widely used in the fields of fuzzy control, fuzzy decision-making, expert system, pattern recognition, signal processing, and so on.

5.1.1 Concept of FNN In 1965, Professor L. A. Zadeh, California University, Berkeley [151], published the famous paper “Fuzzy Sets”, and the fuzzy theory is created. The core of “Fuzzy Sets” is to establish a mathematical model of language analysis for the complex system or process, which can transform natural language into an acceptable computer language. The birth of fuzzy sets theory provides a powerful tool for dealing with a class of fuzzy problems in the objective world. In 1987, B. Kosko [152] firstly combined fuzzy theory with neural network organically. Many scholars have studied the combination methods, mainly in three ways: (1) Neural fuzzy system. It is a fuzzy system based on neural network learning algorithm. It is constructed hierarchically according to the operation steps of fuzzy logic, and the basic functions of the fuzzy system are not changed. The implemented method is to extract the fuzzy rules first, and then the neural network learning algorithm is used to adjust the parameters of the neural fuzzy system. The learning process can be data driven or knowledge driven, which reflects the inherent characteristics of FNN. DOI 10.1515/9783110450293-005

5.1 Basic principles of FNN

(2)

(3)

125

FNN. It is called neural network introduced fuzzy operation or narrow FNN in some literatures. In this method, the FNN retains the basic properties and structure of the neural network, and only some elements and the conventional neural network model are “fuzzified.” This network is still a neural network essentially because the network structure is unchanged, and only the connection weight is changed. Fuzzy neural hybrid system. The fuzzy technique and neural network are combined to form a hybrid system, and the neural network learning algorithm is improved by using fuzzy logic. First, the heuristic knowledge is obtained by analyzing the network performance, then the learning parameters are adjusted to speed up the learning convergence rate. The research time in this field is not long, so the research work is not much.

5.1.2 Structure of FNN The FNN has a wide variety of structures, but its basic structure is composed of the fuzzification layer, the fuzzy inference layer, and the defuzzification layer, which is shown in Figure 5.1. 5.1.2.1 The fuzzification layer This layer is the necessary part of every kind of the FNN. In this layer, the input variables are fuzzified , and the calculation of a membership function is completed. 5.1.2.2 The fuzzy inference layer This layer is the connection between the premise and conclusion of fuzzy inference. In this layer, the network is fuzzy mapped. The structures of inference layer are various, such as the back-propagation (BP) network, radial basis function network and other types of networks. 5.1.2.3 The defuzzification layer In this layer, the distribution pattern basic fuzzy state of the inference conclusion part is translated into the determined state. The determined output is obtained so that the system performs. In order to enhance the adaptability of the FNN, the fuzzification layer, the fuzzy inference layer, and the defuzzification layer are all usually multilayer networks. In Input

Fuzzification

Fuzzy inference

Figure 5.1: The topology structure of FNN.

Defuzzification

Output

126

5 Research of Blind Equalization Algorithms Based on FNN

this way, the membership functions and fuzzy rules of fuzzy inference model can be adjusted automatically through the network learning.

5.1.3 The choice of fuzzy membership function The relationship between fuzzy signal and accurate mathematical expression is established by fuzzy membership function. The fuzziness of the signal lies in the uncertainty of signal. The membership function is the characteristic function of fuzzy sets, and it is the base of using fuzzy theory to solve practical problems. In fuzzy signal sets, it is impractical to clearly define the signal 0 and 1. Any signal may belong to multiple fuzzy subsets simultaneously. The membership relation is usually denoted by the membership degree. Fuzzy systems are built on the fuzzy “if-then” rules easily accepted. However, it is almost impossible to automatically generate and adjust the membership function, and it is not realistic to find a unified membership degree calculation method. So how to effectively structure the membership function is a key problem. The determination process of membership function is essentially objective and has a certain objective regularity. But in some degree, it also has subjectivity and experience. Its structure generally follows the following principles: (1) The membership function is a real convex function. That is, the membership function has a single peak characteristic. (2) The membership function should be in accordance with the semantic order of the people. The inappropriate overlap should be avoided. (3) For the same input, no two membership functions will have the maximum membership degree at the same time. (4) Every point in the domain should belong to the region of one membership function at least. (5) When two membership functions are overlapped, the maximum membership degrees of the two membership functions should not be overlapped. At present, the common methods used to determine membership function include fuzzy statistical method, exemplification, expert experience method, dualistic contrast compositor method, and so on. 5.1.3.1 Fuzzy statistical method Fuzzy statistical method is also known as the fuzzy method [153]. Select a domain U, such as a set of “people”, and select a fixed element u ∈ U, such as “Zhang San,” then suppose there is a set A moving in U, such as “tall”, and A’s boundary is variable. But the degree of tall is a fuzzy concept, and its value varies with the condition, places, and different views. Let different people comment that “Zhang San” is tall or not. If Zhang San is tall, then u ∈ A; if Zhang San isn’t tall, then u ∉ A. The membership function of “tall” , (u) is expressed as

5.1 Basic principles of FNN

, (u) = lim

n→∞

the number of u ∈ A n

127

(5.1)

where n is the total number of trials, and it need large enough. With the increase of n, the membership degree , (u) tend to a value in the interval [0, 1]. The fuzzy statistical method can directly reflect the membership degree of the fuzzy concept, but the computation is large.

5.1.3.2 Exemplification method The exemplification method is proposed by Zadeh [154] in 1972. Suppose that the fuzzy subset A belongs to the domain U, the exemplification method’s main idea is to estimate the membership function of A by using known finite ,A values. If the domain U is all human, and A is “the tall person,” then A is a fuzzy subset. In order to determine ,A , given the value of “tall” h in advance, select one of a few linguistic truth values (the degree of true) to answer “a person is tall or not.” Supposing linguistic truth values are “true,” “roughly true,” “seem true and seem false,” “roughly false,” and “false,” then the linguistic truth values are represented numerically, such as 1, 0.75, 0.5, 0.25, and 0. The discrete representation of membership functions ,A can be obtained through answering different samples h1 , h2 , . . . , hn . 5.1.3.3 Expert experience method The expert experience method is based on the practical experience of the experts to determine membership functions. In the process of determination, some mathematical methods can be used. In many cases, the expert experience method can obtain the initial membership function, then the function needs to be modified and improved by learning and practice tests constantly. The actual effect is the basis of test and adjusting the membership functions.

5.1.3.4 Dualistic contrast compositor method Dualistic contrast compositor method is a more practical method to determine membership function. It determines a certain characteristic order through pairwise comparison among many things, then the membership function general shapes of these things to these characteristics are determined. According to the contrast measurement, dualistic contrast compositor method can be divided into the relative comparison method, the contrast average method, the preference sequence technique, and the similar priority ratio method [155]. Commonly used fuzzy membership functions are mainly Z-type, S-type, the intermediate symmetry type, the axial symmetry type, and so on. Further details are shown in Appendix C.

128

5 Research of Blind Equalization Algorithms Based on FNN

5.1.4 Learning algorithm of FNN The learning of FNN mainly refers to the adjustment of connection weights through a certain learning algorithm so that the network has the functions of memory, recognition, classification, information processing, problem optimal solving, and so on. At present, there are many ways to classify the learning algorithm of FNN. According to having the teacher or not, the learning algorithm can be divided into teacher learning and no teacher learning. In the teacher learning algorithm, the actual output and the expected output of the network are compared, and then the error is used to adjust the network weights constantly until the network is optimal. In the no teacher learning algorithm, the network automatically adjust the weights to optimize the network according to the preset rules (such as competition rules), so that the network eventually have functions such as classification, storage, and others. According to the learning content, the learning algorithm can be divided into structure learning and parameter learning. In the structure learning, the number, type, and initial parameters of input and output space membership functions, the number of rules and the parameters of each rule are determined. In the parameter learning, the initial parameters of FNN determined by the structure learning are fine-tuned. Commonly used structure learning algorithms have many methods such as the grid partition method, the space division method, modified mountain clustering, the fuzzy curve method, the cutting method, the similarity comparison method, and so on. Commonly used parameter learning algorithms are BP algorithm, genetic algorithm, the hybrid learning algorithm of least squares algorithm and gradient method, modified recursive least squares algorithm, and so on.

5.1.5 Characteristics of FNN In addition to the natures and characteristics of the general neural network, the FNN has some special properties [156]. (1) Using the calculation method of fuzzy mathematics, the calculation of FNN processing units becomes more convenient, so the speed of processing information is accelerated; (2) Due to the fuzzy operation mechanism, the fault tolerance ability of the system is enhanced; (3) The FNN has greatly enhanced the means of processing information, which makes the system process information more flexibly; (4) The FNN has expanded the scope and ability of the system dealing with the information, so that the system can deal with the certain information and uncertain information at the same time. (5) The FNN has solved the “black box” inside the neural network, and the internal of network is transparent.

5.2 Blind equalization algorithm based on the FNN filter

(6)

129

Local nodes or weights are used to determine and adjust the generation and modification of fuzzy rules and membership functions in fuzzy systems, so the learning speed is fast.

But there are some difficulties and problems in the realization of FNN. There are many problems to solve, such as the structure is too complicated, the physical meanings of the node and connection are not clear, the learning algorithm is lengthy, the program is difficult to achieve, the network convergence is poor, the trained network’s generalization ability is restricted, how to extract the fuzzy rules and how to automatically generate the membership functions, and so on.

5.2 Blind equalization algorithm based on the FNN filter 5.2.1 Basic principles of algorithm In the blind equalization algorithm based on FNN filter, FNN is used to replace the transversal filter of traditional blind equalization algorithm. Then the center and bandwidth of the Gaussian fuzzy membership function and connection weights are adjusted, so that the output sequences of the equalizer are approximate to the sending sequences. First, the structure of a FNN is selected, then a cost function is constructed according to the structure. Second, the recursive equations of center, bandwidth, and weights are determined according to the cost function. Finally, the above parameters are adjusted by minimizing the cost function. The principle block diagram is shown in Figure 5.2.

5.2.2 Derivation of algorithm 5.2.2.1 Structure of FNN Dynamic recurrent FNN (DRFNN) [156] is used in the blind equalization algorithm based on the FNN filter. The structure is shown in Figure 5.3. It is composed of five n(n) x(n)

Channel h(n)

+

y(n)

FNN Filter

~ x (n)

Decision

x̂ (n)

Blind Equalization Algorithm Figure 5.2: The principle diagram of blind equalization algorithm based on the FNN filter.

130

5 Research of Blind Equalization Algorithms Based on FNN

wij(n)

1 2 ...

y(n – 1)

wj (n)

J

~ x (n) ...

...

...

y(n – I)

...

y(n – 2)

Figure 5.3: The structure of DRFNN.

layers: the input layer, the fuzzification layer, the rule layer, the normalized layer, and the output layer. The second layer has a recurrent link, which not only can use the object’s prior knowledge well but also has a good response to the dynamic system. Suppose that I is the input of the neuron, O is the output of the neuron, Ii(k) is the ith

input neuron of the kth layer. O(p) is the jth output neuron of the pth layer. The input j ̃ of the whole neural network is y(n – i) (i = 1, 2, . . . , I), and the output is x(n). The state functions of DRFNN are defined as below. The first layer is the input layer: (1) Ii (n) = y(n – i)

(5.2)

(1) (1) Oi (n) = Ii (n) = y(n – i)

(5.3)

The second layer is the fuzzification layer (i. e., the membership function layer). (n) + wij (n) ⋅ O(2) (n – 1) Iij(2) (n) = O(1) i ij

2

(n) O(2) ij

=f

[Iij(2) (n)]

(2) { { [Iij (n) – mij (n)] = exp {– { 3ij2 (n) {

} } ( j = 1, 2, . . . , J) } } }

(5.4)

(5.5)

where wij (n) is the connection weight of the feedback unit, mij (n) is the center of the Gaussian membership function, and 3ij (n) is the width of the Gaussian membership function.

5.2 Blind equalization algorithm based on the FNN filter

131

The third layer is the rule layer. The number of neuron is J in this layer. The jth neuron only receives the outputs of each jth neuron in the ( j – 1)th layer. I

Ij(3) (n) = ∏ O(2) (n) ij

(5.6)

i=1

O(3) (n) = Ij(3) (n) j

(5.7)

The fourth layer is the normalized layer. (n) Ij(4) (n) = O(3) j (n) = O(4) j

(5.8)

Ij(4) (n) ∑Jj=1 Ij(4) (n)

(5.9)

The fifth layer is the output layer (i. e., the defuzzification layer ). J

(4)

I (5) (n) = ∑ wj (n)Oj (n)

(5.10)

j=1 (5)

̃ x(n) = O(5) (n) = Ij (n)

(5.11)

where wj (n) is the connection weight from the normalized layer to the output layer.

5.2.2.2 Derivation of algorithm In the blind equalization algorithm based on FNN, the choice of the cost function is related to the global convergence and convergence rate. Because the cost function of the constant modulus blind equalization algorithm (CMA) is only related to the amplitude of the received sequence, and it is unrelated to the phase, CMA is not sensitive to the carrier phase. CMA has been used widely at present. So the cost function of traditional CMA is used here: J(n) =

2 1 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨2 – R2 ] [󵄨󵄨x(n) 2

(5.12)

󵄨 󵄨4 󵄨 󵄨2 where R2 = E [󵄨󵄨󵄨x(n)󵄨󵄨󵄨 ] /E [󵄨󵄨󵄨x(n)󵄨󵄨󵄨 ]. The steepest descent method is adopted: W(n + 1) = W(n) – ,

̃ 𝜕J(n) 𝜕x(n) 󵄨 ̃ 󵄨󵄨2 ̃ = W(n) – 2, [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2 ] x(n) 𝜕W(n) 𝜕W(n)

(5.13)

132

5 Research of Blind Equalization Algorithms Based on FNN

wj (n) iteration formula

̃ 𝜕x(n) = O(4) (n) j 𝜕wj (n)

(5.14)

󵄨 ̃ 󵄨󵄨2 (4) ̃ (n) wj (n + 1) = wj (n) – 2,1 [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2 ] x(n)O j

(5.15)

where ,1 is the connection weight iteration step size from the normalized layer to the output layer.

mij (n) iteration formula J

∑ Ij(4) (n) + Ij(4) (n)

̃ j=1 𝜕x(n) = 2O(3) (n)wj (n) j 𝜕mij (n)

2

J

[ ∑ Ij(4) (n)]

Iij(2) (n) – mij (n)

(5.16)

3ij2 (n)

j=1

J

∑ Ij(4) (n) + Ij(4) (n)

j=1 󵄨 ̃ 󵄨󵄨2 (3) ̃ (n)wj (n) mij (n + 1) = mij (n) – 4,2 [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2 ] x(n)O j

2

J

[ ∑ Ij(4) (n)]

Iij(2) (n) – mij (n) 3ij2 (n)

j=1

(5.17) where ,2 is the center iteration step-size of the Gaussian membership function. 3ij (n) iteration formula

J

∑ Ij(4) (n) + Ij(4) (n)

̃ j=1 𝜕x(n) (n)wj (n) = 2O(3) j 𝜕3ij (n)

J

2

[ ∑ Ij(4) (n)]

2

[Iij(2) (n) – mij (n)]

(5.18)

3ij3 (n)

j=1

J

󵄨󵄨2

󵄨 ̃ (3) ̃ 3ij (n + 1) = 3ij (n) – 4,3 [󵄨󵄨󵄨x(n) (n)wj (n) 󵄨󵄨 – R2 ] x(n)O j

∑ Ij(4) (n) + Ij(4) (n)

j=1

J

2

[ ∑ Ij(4) (n)]

[Iij(2) (n) – mij (n)]

2

3ij3 (n)

j=1

(5.19) where ,3 is the width iteration step size of the Gaussian membership function.

5.2 Blind equalization algorithm based on the FNN filter

133

wij (n) iteration formula 󵄨 ̃ 󵄨󵄨2 wij (n + 1) = wij (n) – 4,4 [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2 ] J

(3) ̃ x(n)O wj (n)O(2) (n j ij

∑ Ij(4) (n) + Ij(4) (n)

– 1)

j=1

J

[∑ j=1

2 Ij(4) (n)]

[Iij(2) (n) – mij (n)] 3ij2 (n)

(5.20)

where ,4 is the connection weight iteration step size of the feedback unit. The specific derivation process is shown in Appendix D.

5.2.3 Computer simulation results The input signals are 4PAM and 8PAM. The signal-to-noise ratio (SNR) is 20 dB. The additive Gauss white noise is zero mean. The simulation channel adopts typical digital telephone channel. Its z-transform is shown in eq. (2.84). When 4PAM and 8PAM signals go through the typical telephone channel, the convergence curves and bit error rate (BER) curves of using DRFNN and the traditional three-layer BP feed-forward neural network (TBPFFNN) [49] are shown from Figures 5.4 to 5.7, where the iterative step sizes of 4PAM are ,1 = 0.008, ,2 = 0.003, ,3 = 0.002, and ,4 = 0.008. The iterative step sizes of 8PAM are ,1 = 0.008, ,2 = 0.002, ,3 = 0.0015, and ,4 = 0.001. Simulation results have shown that the DRFNN algorithm has faster convergence speed, lower BER, smaller SRE than the traditional TBPFFNN algorithm. But in actual operation, the DRFNN algorithm has more calculation.

0.10 1 TBPFFNN 2 DRFNN

MSE

0.08 0.06 0.04 1

0.02 2 0

0

1,000 2,000 3,000 Iterative number

4,000 Figure 5.4: The convergence curves of 4PAM.

134

5 Research of Blind Equalization Algorithms Based on FNN

0.10 1 TBPFFNN 2 DRFNN

MSE

0.08 0.06 0.04 1 0.02 2 0

0

1,000

2,000

3,000

4,000

Iterative number

10–1

1

Figure 5.5: The convergence curves of 8PAM.

2

BER

10–2

10–3

10–4

1 TBPFFNN 2 DRFNN 0

5

10 SNR (dB)

15

20 Figure 5.6: The BER curves of 4PAM.

10–1 1 2

BER

10–2

10–3

1 TBPFFNN 2 DRFNN

10–4 0

5

10 SNR (dB)

15

20 Figure 5.7: The BER curves of 8PAM.

5.3 Blind equalization algorithm based on the FNN controller 5.3.1 Basic principles of the algorithm The basic principle of blind equalization algorithm based on FNN controller is shown in Figure 5.8. In the traditional CMA, the iterative step size has a great influence on the algorithm performance. The big step size can accelerate the convergence speed and tracking speed, but it produces the bigger SRE. The small step size can diminish SRE,

5.3 Blind equalization algorithm based on the FNN controller

135

n(n) x(n)

Channel h(n)

y(n)

+

Blind Equalizer

~ x (n)

Decision

x̂ (n)

FNN Controller Figure 5.8: The principle diagram of blind equalization algorithm based on the FNN controller.

but it decelerates the convergence speed and tracking speed. When the fixed step size is used, the algorithm convergence speed and convergence accuracy are restricted to each other. So the CMA can only make a compromise between the SRE and convergence speed, which greatly restricts its application. The blind equalization algorithm based on the FNN controller can solve this problem well. The FNN controller is used to control the step size of the blind equalization algorithm. In the initial stage of the algorithm, a large step size is adopted to speed up the convergence rate. The step size is reduced gradually along with the algorithm convergence to improve the convergence precision.

5.3.2 Derivation of algorithm 5.3.2.1 Structure of the FNN Five-layer feed-forward FNN is used in blind equalization algorithm based on the FNN controller. Its structure is shown in Figure 5.9. It is composed of five layers: the input layer, the fuzzification layer, the rule layer, the normalized layer, and the defuzzification layer. For the two-dimensional fuzzy controller, the inputs are the error E(n) and the error change CE(n), the output is the change value of the control quantity B,(n). This controller has the characteristics of good controlling effect, easy to be realized by computer. So it is widely used at present. In Figure 5.9, there are NB-Negative Big, NS-Negative Small, ZE-Zero, PS-Positive Small, PB-Positive Big, S-Small, M-Medium, B-Big, P-Positive, and N-Negative. E(n) and CE(n) are defined as E(n) = MSE(n) CE(n) = MSE(n) – MSE(n – 1) The state functions of feed-forward FNN controller are defined as follows.

(5.21) (5.22)

136

5 Research of Blind Equalization Algorithms Based on FNN

Δμ(n)

PB

PS

P

ZE

ZE

N

NS

NB

B

M

CE(n)

S

E(n)

Figure 5.9: The structure of feed-forward FNN controller.

The first layer is the input layer: I1(1) (n) = CE(n), I2(1) (n) = E(n)

(5.23)

(n) = Ii(1) (n)(i = 1, 2, j = 1, 2, 3) O(1) ij

(5.24)

The second layer is the fuzzification layer: (n) Iij(2) (n) = O(1) ij

(5.25) 2

(2) [ (Iij (n) – mij (n)) ] (2) Ojm (n) = exp [– ] (m = 1, 2, 3) 3ij2 (n) [ ]

(5.26)

The third layer is the rule layer: (3) (n) = O(2) (n) Ijm jm

(5.27)

(3) (n) = ∏ Ijm (n) (l = 1, 2, ⋅ ⋅ ⋅ , 9) O(3) l

(5.28)

The fourth layer is the normalized layer: 7

(4) (4) (4) (4) (4) (n), O(4) Il(4) (n) = O(3) 1 (n) = I1 (n), O2 (n) = I2 (n), O3 (n) = ∑ Il (n) l l=3 (4) (4) (4) O(4) 4 (n) = I8 (n), O5 (n) = I9 (n)

(5.29)

5.3 Blind equalization algorithm based on the FNN controller

137

The fifth layer is the defuzzification layer, and the output is B,(n): Ih(5) (n) = O(4) h (n)

(5.30)

5

B,(n) = O(5) (n) = ∑ wh (n)Ih(5) (n)

(5.31)

h=1

where mij (n) and 3ij (n) are the expectation and variance of the space fuzzy domain, respectively, wh (n) is the connection weight of the fifth layer. 5.3.2.2 Fuzzy inference rules CE(n) and E(n) are used as the controller inputs to control the step size of the FNN controller, then Gauss membership function is used to fuzzification inputs. The output is B,(n). The iterative step size formula of equalizer tap coefficients is ,(n + 1) = ,(n) + B,(n). The fuzzy inference rules are shown as follows: (1) If E(n) is big and CE(n) is positive, then B,(n) is PB; (2) If E(n) is big and CE(n) is zero, then B,(n) is ZE; (3) If E(n) is big and CE(n) is negative, then B,(n) is NB; (4) If E(n) is medium and CE(n) is positive, then B,(n) is PS; (5) If E(n) is medium and CE(n) is zero, then B,(n) is ZE; (6) If E(n) is medium and CE(n) is negative, then B,(n) is NS; (7) If E(n) is small and CE(n) is positive, then B,(n) is ZE; (8) If E(n) is small and CE(n) is zero, then B,(n) is ZE; (9) If E(n) is small and CE(n) is negative, then B,(n) is ZE. 5.3.2.3 Derivation of algorithm In order to make the controller work better, the parameters of the controller must be trained and learned. The aim of training is to minimize the cost function. According to the training methods of the CMA algorithm and feed-forward neural network, a new cost function is constructed: J(n) =

2 1 󵄨󵄨 󵄨2 [󵄨B,(n)󵄨󵄨󵄨 – R2 ] 2 󵄨

(5.32)

󵄨 󵄨4 󵄨 󵄨2 where R2 = E [󵄨󵄨󵄨CE(n)󵄨󵄨󵄨 ] /E [󵄨󵄨󵄨CE(n)󵄨󵄨󵄨 ]. In the FNN controller, three parameters mij (n), 3ij (n), and wh (n) will be iterated. The steepest descent method is used. wh (n) iteration formula 𝜕J(n) 󵄨 󵄨2 = 2 [󵄨󵄨󵄨B,(n)󵄨󵄨󵄨 – R2 ] B,(n)Ih(5) (n)E(n) 𝜕wh (n)

(5.33)

138

5 Research of Blind Equalization Algorithms Based on FNN

so, 󵄨 󵄨2 wh (n + 1) = wh (n) – ,1 [󵄨󵄨󵄨B,(n)󵄨󵄨󵄨 – R2 ] B,(n)Ih(5) (n)E(n)

(5.34)

where ,1 is the iteration step size of wh (n).

mij (n) iteration formula 𝜕B,(n) 𝜕J(n) 󵄨 󵄨2 = 2 [󵄨󵄨󵄨B,(n)󵄨󵄨󵄨 – R2 ] B,(n) 𝜕mij (n) 𝜕mij (n) (2) 𝜕B,(n) 𝜕Ojm (n) 󵄨 󵄨2 = 2 [󵄨󵄨󵄨B,(n)󵄨󵄨󵄨 – R2 ] B,(n) (2) 𝜕Ojm (n) 𝜕mij (n)

𝜕B,(n) 󵄨 󵄨2 = 4 [󵄨󵄨󵄨B,(n)󵄨󵄨󵄨 – R2 ] B,(n) (2) 𝜕Ojm (n) 𝜕B,(n) 𝜕O(2) (n) jm

=

𝜕B,(n) (3) 𝜕Ijm (n)

=

(3) 𝜕B,(n) 𝜕Ol (3) 𝜕O(3) (n) 𝜕Ijm (n) l

=

(n) 𝜕B,(n) O(3) l

=

(n) 𝜕B,(n) 𝜕O(4) (n) O(3) l h

(3) Ijm (n) 𝜕O(3) (n) l

=

(5.35)

2 (2) [I (n)–mij (n)] ij – 32 (n) ij e

Iij(2) (n) – mij (n) 3ij2 (n)

(n) 𝜕B,(n) O(3) l

=

(3) Ijm (n) 𝜕O(3) (n) l

O(3) (n) 𝜕B,(n) l

(5.36)

(3) Ijm (n) 𝜕Il(4) (n)

(3) Ijm (n) 𝜕O(4) (n) 𝜕Il(4) (n) h

where 𝜕O(4) 1 (n) 𝜕I1(4) (n) 𝜕O(4) 4 (n) 𝜕I8(4) (n)

= 1,

= 1,

𝜕O(4) 2 (n) 𝜕I2(4) (n) 𝜕O(4) 5 (n) 𝜕I9(4) (n)

= 1,

= 1,

𝜕O(4) 3 (n) 𝜕I3(4) (n) 𝜕B,(n) 𝜕O(4) (n) h

=

𝜕O(4) 3 (n) 𝜕I4(4) (n)

=

𝜕O(4) 3 (n) 𝜕I5(4) (n)

=

𝜕O(4) 3 (n) 𝜕I6(4) (n)

=

𝜕O(4) 3 (n) 𝜕I7(4) (n)

= wh (n)

= 1,

(5.37)

So,

󵄨 󵄨2 mij (n + 1) = mij (n) – 4,2 [󵄨󵄨󵄨B,(n)󵄨󵄨󵄨 – R2 ] B,(n)e

–

2 (2) (n)–mij (n)] ij 32 ij

[I

(2) O(3) (n) Iij (n) – mij (n) l (3) Ijm (n)

3ij2 (n)

wh (n) (5.38)

where ,2 is the iteration step size of mij (n).

139

5.3 Blind equalization algorithm based on the FNN controller

3ij (n) iteration formula

𝜕J(n) 𝜕3ij(2) (n)

2

= 4 [[B,(n)] – R2 ] B,(n)

The derivation of

𝜕B,(n)

𝜕O(2) (n) jm

–

𝜕B,(n) 𝜕O(2) (n) jm

e

2 (2) (n)–mij (n)] ij 32 ij

[I

[Iij(2) (n) – mij (n)]

2

(5.39)

3ij3 (n)

is similar to the above algorithm. So,

󵄨 󵄨2 3ij (n + 1) = 3ij (n)–4,3 [󵄨󵄨󵄨B,(n)󵄨󵄨󵄨 – R2 ] B,(n)e

–

2 (2) [I (n)–mij (n)] ij 32 ij

2

(2) O(3) (n) [Iij (n) – mij (n)] l (3) Ijm (n)

3ij3 (n)3

wh (n) (5.40)

where ,3 is the iteration step size of 3ij (n). The FNN controller can be obtained according to the above parameters’ iterative formulas. The iterative step size of the traditional CMA is controlled by FNN controller. The weight iteration formula of CMA is ̃ [R2 – 󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 ] Y(n) W(n + 1) = W(n) + ,(n)x(n)

(5.41)

where ,(n) is the variable step size and 5

,(n + 1) = ,(n) + B,(n) = ,(n) + ∑ wh (n)Ih(5) (n)

(5.42)

h=1

This is the blind equalization algorithm based on the FNN controller.

5.3.3 Computer simulation results The input signals are 4PAM and 8PAM. SNR is 20 dB. The simulation channels adopt typical digital telephone channel and the ordinary channel. Their z-transforms are shown in eqs. (2.84) and (3.13). When 4PAM and 8PAM signals go through the typical telephone channel and the ordinary channel, the convergence curves and BER curves of using FNN controller algorithm (FNNCA) and the CMA based on MSE variable step-size are shown from Figures 5.10 to 5.13. When 4PAM and 8PAM go through the typical digital telephone channel, the iterative step sizes are ,1 = 0.03, ,2 = 0.04, ,3 = –0.06 and ,1 = 0.03, ,2 = 0.02, ,3 = –0.09, respectively. When 4PAM and 8PAM go through the ordinary channel, the iterative step sizes are ,1 = 0.03, ,2 = 0.05, ,3 = –0.06 and ,1 = 0.03, ,2 = 0.04, ,3 = –0.09, respectively. Simulation results have shown that the FNNCA algorithm has faster convergence speed and lower SRE than CMA based on MSE variable step-size algorithm,

140

5 Research of Blind Equalization Algorithms Based on FNN

0.08

1 CMA 2 FNNCA

MSE

0.06 1 0.04 0.02 2 0

0

1,000 2,000 Iterative number

3,000

Figure 5.10: The convergence curves of 4PAM through the typical telephone channel.

0.10 1 CMA 2 FNNCA

MSE

0.08 0.06 0.04

1

0.02 2 0

0

2,000 4,000 6,000 8,000 10,000 Iterative number

Figure 5.11: The convergence curves of 8PAM through the typical telephone channel.

0.10 1 CMA 2 FNNCA

MSE

0.08 0.06

1

0.04 2

0.02 0

0

1,000 2,000 Iterative number

3,000

Figure 5.12: The convergence curves of 4PAM through the ordinary channel.

0.10 1 CMA 2 FNNCA

MSE

0.08 0.06 0.04

1

0.02 2

0 0

2,000 4,000 6,000 8,000 10,000 Iterative number

Figure 5.13: The convergence curves of 8PAM through the ordinary channel.

5.4 Blind equalization algorithm based on the FNN classifier

141

but the effect is not obvious. Therefore, comprehensively considering the algorithm complexity and performance improvement, blind equalization algorithm based on the FNN controller is only a method; it is impractical.

5.4 Blind equalization algorithm based on the FNN classifier 5.4.1 Basic principles of algorithm In the blind equalization algorithm based on the FNN classifier, FNN is used as a classifier to replace the decision device of traditional blind equalization algorithm. The traditional decision has defects of large decision error caused by the fixed threshold decision. But the blind equalization algorithm based on the FNN classifier can effectively overcome this problem. It has advantages of simple learning algorithm, and short training time. Essentially FNN classifier algorithm combines the input of certain information and input of fuzzy information, so that the nonlinear characteristics of the system are more abundant, and the classification ability of the system is improved. The fuzzy system is applied to the channel equalization, which overcomes the oneness of the original decision method, improves the accuracy of the decision, and reduces the BER. The block diagram is shown in Figure 5.14. First of all, the received signals are used to blind estimate channel, and then the mathematics convolution theorem is used to preliminary recover the input signals, finally FNN is used to classify signals.

5.4.2 Derivation of algorithm 5.4.2.1 Blind channel estimation Without sending any training sequence, the blind channel estimation only uses the statistical characteristics of the received signal to obtain the channel characteristics. n(n) x(n)

Channel h(n)

+

y(n)

Deconvolution

~ x (n)

FNN Classifier

x̂ (n)

Blind Estimate Channel Figure 5.14: The principle block diagram of blind equalization algorithm based on the FNN classifier.

142

5 Research of Blind Equalization Algorithms Based on FNN

In this section, the high-order statistics estimator [157, 158] is used. The algorithm computation is larger, but the realization is simple and intuitive. Suppose input signal x(n) is zero mean and statistically independent sequence, n(n) is Gauss white noise sequence, hk (0 ≤ k ≤ L) is the channel impulse response sequence, and h0 = 1. The receiving signal is L

y(n) = ∑ hk x (n – k) + n(n)

(5.43)

k=0

The fourth-order statistics of the received signal and input signal are C4y (41 , 42 , 43 ) = E[y(n)y∗(n + 41 ) y (n + 42 ) y∗(n + 43 )] – E[y(n)y∗(n + 41 )] E [y (n + 42 ) y∗(n + 43 )] – E[y(n)y(n + 42 )] E [y∗(n + 41 ) y∗(n + 43 )]

(5.44)

– E[y(n)y∗(n + 43 )] E [y (n + 42 ) y∗(n + 41 )] C4x (41 , 42 , 43 ) = E [x(n)x∗(n + 41 ) x(n + 42 ) x∗(n + 43 )] – E[x(n)x∗(n + 41 )] E[x (n + 42 ) x∗(n + 43 )] – E[x(n)x (n + 42 )] E[x∗(n + 41 ) x∗(n + 43 )]

(5.45)

– E[x(n)x∗(n + 43 )] E[x (n + 42 ) x∗(n + 41 )] According to eqs. (5.43)–(5.45), we obtain L

L

L

L

C4y (41 , 42 , 43 ) = ∑ ∑ ∑ ∑ hk1 h∗k2 hk3 h∗k4 k1 =0 k2 =0 k3 =0 k4 =0

× {E [x (n – k1 ) x∗(n + 41 – k2) x (n + 42 – k3 ) x∗(n + 43 – k4 )] – E [x (n – k1 ) x∗(n + 41 – k2 )] E [x (n + 42 – k3 ) x∗(n + 43 – k4 )] – E [x (n – k1 ) x(n + 42 – k3 )] E [x∗(n + 41 – k2 ) x∗(n + 43 – k4 )] – E [x (n – k1 ) x∗(n + 43 – k4 )] E [x (n + 42 – k3 ) x∗(n + 41 – k2 )]} (5.46) If 41 = L, 42 = 4, 43 = 41 , there is L

L

L

L

C4y (L, 4, 41 ) = ∑ ∑ ∑ ∑ hk1 h∗k2 hk3 h∗k4 k1 =0 k2 =0 k3 =0 k4 =0

× {E [x (n – k1 ) x∗(n + L – k2 ) x (n + 4 – k3 ) x∗(n + 41 – k4 )] – E [x (n – k1 ) x∗(n + L – k2 )] E [x (n + 4 – k3 ) x∗(n + 41 – k4 )] – E [x (n – k1 ) x (n + 4 – k3 )] E [x∗(n + L – k2 ) x∗(n + 41 – k4 )] –E [x (n – k1 ) x∗(n + 41 – k4 )] E [x (n + 4 – k3 ) x∗(n + L – k2 )]}

(5.47)

5.4 Blind equalization algorithm based on the FNN classifier

143

where 󵄨 󵄨 E [󵄨󵄨󵄨󵄨x(n)4 󵄨󵄨󵄨󵄨] k = l = m = r { { { { { {E2 [󵄨󵄨󵄨x(n)󵄨󵄨󵄨2 ] k = l ≠ m = r, k = r ≠ l = m E [x (n – k) x∗(n – l) x (n – m) x∗(n – r)] = {󵄨 󵄨 2 󵄨 󵄨2 { {󵄨󵄨󵄨󵄨E [x (n)]󵄨󵄨󵄨󵄨 k = m ≠ l = r { { { others {0 If k1 = 0, L = k2 , 4 = k3 , 41 = k4 , there is C4y (L, 4, 41 ) = h0 h∗41 h4 h∗L C4x (0, 0, 0) C4y (L, 0, 41 ) =

h0 h∗41 h0 h∗L C4x

(5.48)

(0, 0, 0)

C4y (L, 0, 41 ) h4 = C4y (L, 4, 41 ) h0

for

(5.49) 0 ≤ 41 ≤ L

(5.50)

So, ∗ ∗ C4y (L, 0, 41 ) C4y (L, 0, 41 ) h4 = C4y (L, 0, 41 ) C4y (L, 4, 41 ) h0 L

L

41 =0

41 =0

(0 ≤ 41 ≤ L)

∗ ∗ (L, 0, 41 ) C4y (L, 0, 41 ) h4 = ∑ C4y (L, 0, 41 ) C4y (L, 4, 41 ) h0 ∑ C4y L

L

41 =0

41 =0

∗ ∗ (L, 0, 41 ) C4y (L, 0, 41 ) = h0 ∑ C4y (L, 0, 41 ) C4y (L, 4, 41 ) h4 ∑ C4y

h4 = =

(5.51) (5.52)

(5.53)

∗ (L, 0, 41 ) C4y (L, 4, 41 ) h0 ∑L41 =0 C4y ∗ (L, 0, 4 ) C (L, 0, 4 ) ∑L41 =0 C4y 1 4y 1 ∗ (L, 0, 41 ) C4y (L, 4, 41 ) h0 ∑L41 =0 C4y

󵄨 󵄨2 ∑ 󵄨󵄨󵄨󵄨C4y (L, 0, 41 )󵄨󵄨󵄨󵄨 4 =0 L

, (1 ≤ 4 ≤ L)

(5.54)

1

The estimated value of the channel impulse response can be obtained by eq. (5.54).

5.4.2.2 Design of the FNN classifier Three-layer feed-forward FNN structure is adopted in the FNN classifier. Its structure is shown in Figure 5.15 [159]. The three-layer feed-forward FNN is composed of the fuzzification layer, the rule layer, and the defuzzification layer. Its state function is given below. The first layer is the fuzzification layer: ̃ ̃ Ii(1) = x(n), O(1) = ,i [x(n)] i

(i = 1, 2, . . . , k)

(5.55)

144

5 Research of Blind Equalization Algorithms Based on FNN

~ x (n)

x̂1 (n) x̂2 (n) m1

m2

...

x̂k (n)

...

mk

... ~ x (n)

~ x (n)

...

~ x (n) Figure 5.15: The principle diagram of blind equalization algorithm based on the FNN classifier.

̃ where , (x(n)) is the fuzzy membership function. In this algorithm, the membership function is chosen as 󵄨 ̃ 󵄨 if ( 󵄨󵄨󵄨x(n) – mi 󵄨󵄨󵄨 ≤ ri )

{1 ̃ ,i (x(n)) ={ ̃ x(n)–m exp (– | r i | %) i {

other

(5.56)

where mi is the center of divided areas by the ith rule; ri is the radius of the corresponding input signal area. % is the sensitive parameter, and it can reflect reduce speed of the membership degree when the input sample is far away from the ball center. So, when the signal does not belong to any regular hyper ellipsoid, the membership degree ̃ ,i (x(n)) given by formula (5.56) is not zero, it is reduce to a smaller value according to the distance from the cluster center [160]. The second layer is the rule layer: Ii(2) (n) = O(1) (n), O(2) (n) = i i

̃ ,i (x(n)) k

= x̂i (n)

(5.57)

̃ ∑ ,i (x(n))

i=1

The third layer is the defuzzification layer: k

Ii(3) (n) = O(2) (n), O(3) (n) = i i

̃ ∑ ,i (x(n)) x̂i (n)

i=1

k

̃ ∑ ,i (x(n))

i=1

where x̂i (n) is the output of the ith rule. FNN contains k fuzzy rules. Each rule is represented in the following form: ̃ is (mi , ri ), then x̂i (n) = ui (x(n)) ̃ Ri : if x(n) (i = 1, 2, . . . , k)

(5.58)

5.4 Blind equalization algorithm based on the FNN classifier

145

̃ is the expected where, Ri is the ith rule; x̂i (n) is the local output of the ith rule; ui (x(n)) ̃ doesn’t output of the ith rule; k is the number of the fuzzy rules. From the rules, if x(n) ̂ belong to the area specified by (mi , ri ), then xi (n) = 0. The learning algorithm of FNN classifier uses fuzzy competitive learning algorithm [161]. Through competitive learning, the input signal is classified into the corresponding category, then the membership degree of certain class is obtained by fuzzy membership function, i.e., the match degree of signal belonging to each rule is obtained. The steps are (1) Selecting k, it is the number of clusters, and k is the number of signal types. The initial signal center is mi (i = 1, 2, . . . , k); (2) Determining the center coefficient for any input signal 󵄨 󵄨󵄨 ̃ 󵄨x(n) – mi 󵄨󵄨󵄨 di = 󵄨 k 󵄨 ̃ 󵄨 ∑ 󵄨󵄨󵄨x(n) – mi 󵄨󵄨󵄨

(5.59)

i=1

(3)

Modifying the value of the signal center mi (i = 1, 2, . . . , k). ̃ – mi (n)] mi (n + 1) = mi (n) + ,d2i [x(n)

(4)

(5.60)

where , is the iteration step size. Fuzzy competitive learning method is to modify the center vector of the input signal successively, so the calculation time becomes shorter, and the convergence speed is faster. Modifying the input area ri (i = 1, 2, . . . , k) 󵄨󵄨 󵄨 󵄨󵄨mi – mj 󵄨󵄨󵄨 󵄨 , (0 < + < 1) ri = min 󵄨 j=1,2,...,k +

(5.61)

i=j̸

(5)

The input area radius ri not only can determine the size of input area but also can determine the degree of overlap between regions. Each region contains all belonging to own samples, and in a certain degree it also contains some samples of adjacent area. This reflects the fuzzy of rules. With the adjustment of the network, it is little possibility that the new data does not belong to the union of the divided regional. Therefore, how to choose the radius of each region is very important. ̃ ̃ Using eq. (5.56), for any input sample, ,i (x(n)) is calculated. Then ,i (x(n)) is normalized. By learning and adjusting the divided space and input region radius of FNN, the match degree to each rule for each input signal is obtained, so the signals are divided. FNN classifies the signals to effectively reduce the error rate of signal transmission.

146

5 Research of Blind Equalization Algorithms Based on FNN

5.4.3 Computer simulation results The input signals are 4PAM and 8PAM. SNR is 20 dB. The additive Gauss white noise is zero mean. The simulation channels adopt the typical telephone channel and the ordinary channel. Their z-transforms are shown in eqs. (2.84) and (3.13). When 4PAM and 8PAM signals go through the typical telephone channel and the ordinary channel, the convergence curves and BER curves of using FNN classifier algorithm (C-FNN) and the feed-forward neural network (FF-NN) [49] are shown from Figures 5.16 to 5.21. Simulation figures have shown that C-FNN algorithm has better convergence property, faster convergence speed, smaller MSE and lower BER than FF-NN algorithm proposed in literature [49]. 0.10 (1) FF-NN (2) C-FNN

MSE

0.08 0.06

(1)

0.04

(2)

0.02 0

0

1,000 2,000 3,000 Iterative number

4,000

Figure 5.16: The convergence curves of 4PAM through the typical telephone channel.

0.12 (1) FF-NN (2) C-FNN

0.10 MSE

0.08

(1) (2)

0.06 0.04 0.02 0

0

1,000 2,000 3,000 Iterative number

4,000

Figure 5.17: The convergence curves of 8PAM through the typical telephone channel.

4,000

Figure 5.18: The convergence curves of 4PAM through the ordinary channel.

0.10 (1) FF-NN (2) C-FNN

MSE

0.08 0.06

(1)

(2)

0.04 0.02 0

0

1,000 2,000 3,000 Iterative number

5.5 Summary

147

0.10 (1) FF-NN (2) C-FNN

MSE

0.08 0.06 (1) 0.04

(2)

0.02 0

0

1,000 2,000 3,000 Iterative number

4,000

Figure 5.19: The convergence curves of 8PAM through the ordinary channel.

10–1

BER

10–2

10–3 10–4

FF-NN C-FNN 0

5

10 SNR (dB)

15

20

Figure 5.20: The BER curves of 8PAM through the typical telephone channel.

15

20

Figure 5.21: The BER curves of 8PAM through the ordinary channel.

10–1

BER

10–2

10–3 10–4

FF-NN C-FNN 0

5

10 SNR (dB)

5.5 Summary In this chapter, according to the combination of the FNN and blind equalization algorithm, the blind equalization algorithms based on FNN filter, FNN controller, and FNN classifier are studied. The different FNN structures are adopted. The iterative formula and fuzzy inference rules are derived and the computer simulation is carried out. The blind equalization algorithm based on FNN filter uses FNN to replace the transversal filter of the traditional blind equalization algorithm, so a new equalization algorithm is proposed. The new algorithm is compared with the traditional

148

5 Research of Blind Equalization Algorithms Based on FNN

blind equalization algorithm based on the BP neural network, steady residual error is smaller, the speed of convergence is faster, but the computation is larger. The blind equalization algorithm based on FNN controller controls the step size of CMA, and then the variable step-size blind equalization algorithm is obtained. FNNCA compares with the MSE variable step-size blind equalization algorithm, performance has a slight improvement, but the system complexity and computation will increase obviously. Therefore, the algorithm is of no value in practical application. The blind equalization algorithm based on FNN classifier uses the FNN classifier to replace the fixed threshold decision classifier of traditional blind equalization algorithm. And the blind channel estimation technique is adopted. The structure is complex and the computation is increased, but convergence performance is significantly improved.

6 Blind Equalization Algorithm Based on Evolutionary Neural Network Abstract: This chapter includes the following contents: the basic theory and development of genetic algorithm (GA), the parameter coding, the initial population setting, the design of adapt function, genetic operator choice, the controlling parameter setting method, and the GA and neural network combination mechanism. Furthermore, blind equalization algorithm based on GA optimization neural network weights and structure is studied and computer simulation is carried out. The results show that the convergence performance of the two new algorithms is faster compared with the traditional neural network blind equalization algorithm.

6.1 Basic principles of evolutionary neural networks Evolutionary neural network is the evolutionary algorithm (EA) applied to neural network construction and learning [162]. Evolutionary neural network integrates the advantages of neural networks and EA, so it has very strong robustness. As a result, evolutionary neural network is a new interdisciplinary research field in recent years. (EA) is a random search algorithm which utilizes biological natural selection and genetic mechanism. EA can get near-optimal solution without error function gradient information, so it is an efficient tool in search, optimization, and machine learning EA [163]. EA includes evolutionary programming (EP) (or evolutionary programming), evolutionary strategy (ES), and genetic programming (GP). Each method emphasizes different natural evolution aspects. GA emphasizes on gene chromosome operation (genetic information between parent and offspring); GP is a standard GA expansion; EP and ES emphasize the behavioral changes of the species, namely, the behavior connection between parent and offspring. Each method solves practical problems with the biological evolution ideas and principles. Continuous or differentiable cost function is not needed in problems calculated by EA. The global optimal solution is easy to obtain since the search always pervades all of space. Therefore, EAs used in neural network optimization and design are a former method. In this chapter, the optimization and design of the neural network by GA is studied and the application of EA used in the blind equalization algorithm is also given.

6.1.1 The concept of GA GA is an adaptive global optimization algorithm based on natural evolution by Darwin and genetic variation theory by Mendel. Only the fittest can survive in Darwin’s DOI 10.1515/9783110450293-006

150

6 Blind Equalization Algorithm Based on Evolutionary Neural Network

theory. Each species adapt to the environment step by step, the basic characteristics of each individual species are inherited by the offspring, but the offspring will produce some different changes. So only those individual characteristics adapting to the environment can be retained. The genetic is the most important principles in Mendel genetic theory. Genetic passwords exist in cells, which contains in the chromosome. Each gene has special position and controls a special property, so the individual produced by each gene has specific adaptability to the environment. The generations which adapt to the environment better can be produced by gene mutation and hybridization. Only genes with high adaptability can be preserved after natural selection.

6.1.2 Development of GA The GA has developed for more than 50 years. The development process can be divided into three stages: 6.1.2.1 The rise in the 1960s–1970s GA is proposed by Fraser in the early 1960s, which highlights evolution process of mutation and selection by simulation [164]. Bionic optimization algorithm idea was proposed by Professor J.H. Holland in Michigan University in the United States since the similarity between the natural biological genetic phenomenon and artificial adaptive behavior is recognized. The “GA” was first proposed by J.D. Bagley [165], J.H. Holland’s student, in his doctoral thesis in 1967. And then the first paper on the GA application is published. Although it suffers from doubts and objections, this research is insisted by Professor J.H. Holland and his students. The GA has two important breakthroughs in 1975. The one is the monograph by Professor J.H. Holland [166] named Adaptation in Natural and Artificial Systems. The monograph contains the GA basic theory and method, schema theory, and the implicit parallelism of GA. Meanwhile, GA is applied in adaptive system simulation, function optimization, machine learning and automatic control, and other fields; the other is the doctoral thesis by K.A. Dejong [167] named “An Analysis of the Behavior of a Class of Genetic Adaptive System.” It contains the famous Dejong five function test platform and the performance evaluation criteria. As an example to function optimization, the analysis and simulations on six kinds of schemes of GA are also given. From then on, GA has been widely used in many fields and some basic GA theories are also developed. 6.1.2.2 The stage of development in the 1980s When the traditional artificial intelligence simulated by symbol system got into temporary trouble, the intelligence simulation research from the biological system

6.1 Basic principles of evolutionary neural networks

151

revives, such as underlying the revived neural network, machine learning, and GA simulation of intelligence. Classifier System (CS), proposed by Professor J.H.Holland, is the first GA-based machine learning classifier system. A complete framework is constructed in classifier system. International Society of Genetic Algorithms (ISGA) held an international conference on GAs. ISGA was established in 1985. “GA in Search, Optimization and Machine Learning” was published by D.J. Goldberg [168] who is H. Holland’s student. The main achievements and application of GA are summarized in the book. The modern GA base [169] is established. Generally speaking, GA had developed from the classical stage to the modern stage in this period.

6.1.2.3 The prosperous stage in the 1990s Since the GA is practical, efficient, good robust, it has been used in machine learning, pattern recognition, neural network, control system optimization, and other fields in 1990s. The research of GA in China mainly began in the 1990s. At present, GA, expert system, and neural network have become hot topics in artificial intelligence field.

6.1.3 GA parameters The realization of GA includes the following steps: parameters encoding, initial group settings, designing of fitness function, genetic operation designing, and control parameters setting.

6.1.3.1 Parameter encoding Coding is the basic operation of the GA. Because GA can not directly deal with the problem space parameters, the encoding aim is to change the problem space parameters into chromosomes which are composed by genes with certain structure in space. The GA has good robustness, so the encode requirement is not harsh. However, the coding strategy of GA has great influence on genetic operators, especially on crossover and mutation operator. The encoding rules are as follows [170]: (1) Completeness: all points in the problem space (feasible solutions) can become the coding points (chromosome) (2) Soundness: the coding points in the space can correspond to all the points in the problem space (3) Non-redundancy: all points before encoding and coding should be correspondence respectively Besides these basic rules, other encoding principles proposed by K.A. Dejong [167] are as follows:

152

(1)

(2)

6 Blind Equalization Algorithm Based on Evolutionary Neural Network

The meaningful building blocks (building blocks) coding principles: Coding should easily generate short distance and low-order building blocks (building block). The minimum character encoding rule: in order to get the simple problem description, the smallest character set should be used.

Coding includes binary coding, gray coding, sequence coding, real coding (floating point coding), tree coding, adaptive coding, chaotic coding, multiparameter coding, and so on. The common coding is binary and real. Binary encoding The original solution for the problem is mapped into the sequences composed of 0,1 in binary encoding. Solutions are got by decoding. The adaptation degree is calculated. The most patterns can be expressed by minimum character set encoding principle. At the same time, model theorem can be used in algorithm analysis. However, there are the following shortcomings [171]: 1) The adjacent encodes may have large Hamming distance such as 15 and 16 and these can be expressed as 1111 and 10000, respectively. When 15 is reversed to 16, all the bits must be changed. The searching efficiency of the genetic operators is reduced. The disadvantage called the Hamming cliff appears. 2) Accuracy determines the string length binary encoding, so the algorithm has no fine-tuning function. The high precision produces long bit string and low efficiency. Gray encoding can be used to overcome the Hamming cliff. 3) The binary encoding bit string is very long and inefficient in high dimensional optimization problems. The real number encoding Decimal encoding is directly used in real number coding, so the solution can be directly operated. The GA search ability is increased since the heuristic information related to the problem areas is provided. The main advantages are the following: the use in high precision optimization, large space search, fast speed, convenient combination with traditional optimization methods and to deal with optimization of constraints. 6.1.3.2 Population scale and initial population setting The number of individuals in the population is called the population scale. Because of the need for the GA group operation, there must be an initial population composed by initial solutions first. The larger the population scale is, the higher the diversity of the population is. As a result, the risk falling into local solution is smaller. However, individual competition is small and computational complexity increases. Therefore, the population scale should be suitable. Population scale is determined by the length of the individual

6.1 Basic principles of evolutionary neural networks

153

which is proved by D.E. Goldberg [172]. Assuming that the length of the individual is L, the population scale is 2L/2 . The optimal scale optimization problem of real number coding is studied by Yan Feng sun [173]. The following strategies [170] are adopted by the initial population setting: (1) Estimating the distribution range of the optimal solution in the whole problem space and setting the initial population in the range of this distribution (2) Continuously choosing the best individual to join in the group from random generating individuals until the population scale is satisfied In addition, for the problem with constrained domains, it is necessary to take into account that the random initialization of the points is in the feasible region. So feasible region is used to generate the initial population. 6.1.3.3 Fitness function design In biological heredity and evolution, there will be more opportunities to reproduce for species which adapt to environment better. The concept “degree” is used to measure each individual to reach the optimal solution. The individuals with higher degree are more possible to pass on to the next generation. Fitness function is used to measure the individual fitness degree. It is very important to select fitness function which directly affects the convergence rate and the optimal solution obtained. The fitness function should be single valued, continuous, nonnegative, maximum, consistent, high universality, and so on. 6.1.3.4 Genetic operation design Genetic operators including selection, crossover, and mutation constitute the core of GA and make GA to have strong search ability. The crossover operator is the most important in GA. Selection operator Selection operator (reproduction or reproduction operator) is a process that chooses strong chromosomes from population. Choosing rules are chromosome fitness values. The greater the fitness is, the bigger the probability to be selected. As a result, the number of descendants is bigger. The choosing methods include fitness proportional model, expected value model, rank-based model, and so on. Fitness proportional model. Fitness proportional model (Roulette wheel or Monte Carlo selection method) is the most common selection method in GA. In this method, the selection probability of each individual is proportional to the adaptation value. That is

154

6 Blind Equalization Algorithm Based on Evolutionary Neural Network

Psi =

fi N

(6.1)

∑ fi

i=1

where Psi represents the ith selection probability; fi is the ith fitness value. The proportion of the individual fitness value in the whole individuals is reflected in choosing selection probability. The greater the fitness value is, the bigger the probability to be selected is. However, individuals with high fitness value may also be eliminated when individuals are small. Namely, the individuals with low fitness value may also be selected. Expected Value Model. In order to overcome the defects of fitness proportional model, expected value model can be adopted, which first calculate the expected number of each individual in the next generation: M=

fi Nfi = N f̄ ∑ fi

(6.2)

i=1

where f ̄ =

1 N

N

∑ fi is the fitness mean.

i=1

Second, if an individual is selected to pairing and crossover, the desired number of the next generation is expected to reduced by 0.5; if not, the number minus 1. Third, in the above two cases, if an individual’s expectations value is less than zero, then the individual does not participate in the selection. Rank-based model. Each individual adaptation value is calculated and ranked from large to small in Rank-based Model. Then the pre-designed probability table is assigned to individuals according to former order, which is called choosing possibility. Since all the individuals are ranked according to their fitness values, the selection probability is not related to the fitness value but to the ordinal sort. However, the relationships between selection probability and the serial number need to be determined in advance. At the same time, there is a statistical error. Crossover operator New generation offspring chromosomes are generated by crossover operator (recombination or breeding operator) through exchanging a pair of parent chromosome genes. It is the core of the GA, which is the symbol difference from other evolutionary optimization method. The excellent characteristics in the original group are maintained in a certain extent and new groups diversity is got. Crossover operator is divided into one-point crossover, two-point crossover, and uniform crossover.

6.1 Basic principles of evolutionary neural networks

155

One-point crossover. A random crossover point is set in one-point crossover (simple crossover or one point crossover). On the right of the random point, the partial gene structure of two individuals is exchanged. Two new individuals are generated. Assuming that the chromosome length is L, there is L – 1 intersection position. So a different crossover result can be achieved. Two-point crossover. Two random crossover points are set in two-point crossover (Double-point Crossover). Two intersection points of the individual gene structure are swapped. Two new individuals are generated respectively. If the chromosome length is L, there may be (L – 1) (L – 2) /2 cross point set. Uniform crossover. Two parent individual gene strings are exchanged respectively in uniform crossover. The specific method is as follows: first, two parent individuals A and B are selected; second, the binaural rings A󸀠 and B󸀠 , which have the same length with parent individual gene, are randomly generated as a mask word or template. How A󸀠 and B󸀠 inherit two parent string genes is judged. If the mask code is 1, A and B genes are inherited by A󸀠 and B󸀠 , respectively. If the mask code is 0, A and B genes are inherited by A󸀠 and B󸀠 crosshatched. Two new individuals with new genes for offspring are got. The location is not considered the exchange position in uniform crossover, so the failure mode probability is larger. However, it can search the new model which cannot be obtained by the previous cross methods based on points. Mutation operator Mutation originates from gene mutation which simulates the biological evolution process. Gene mutations are the cause of the species diversity. The chromosome gene values are randomly changed according to a certain possibility in mutations. The basic steps are as follows: 1) Randomly determine the gene location in all individual codes of the group 2) The variation of these genes is carried out with the prior probability Pm Mutation can be divided into simple mutation, uniform mutation, nonuniform mutation, Gauss mutation, and so on. Basic bit mutation operator. The basic mutation operator is applied to the binary encoding. Binary encoding is randomly reversed by a certain possibility, that is, 0 is reversed to 1 or 1 to 0. Uniform mutation operator. Uniform mutation is suitable to the real number encoding. Gene values in the original real encoding are replaced by another gene values

156

6 Blind Equalization Algorithm Based on Evolutionary Neural Network

according to mutation probability Pm . If M = m1 , m2 , . . . , mL is the individual, mk ∈ [mk min , mk max ] is the mutation point. The variation formula is m󸀠k = mk min + ! (mk max – mk min )

(6.3)

where m󸀠k is the variation value of mk ; ! is the random variable in [0, 1].

Nonuniform mutation operator. The nonuniform mutation is a random disturbance to the original genetic value. The variation value is regarded as the new genetic value.

Normal mutation operator. Normal mutation operator (Gauss mutation operator) is suitable to real coding. It is used in evolution strategies first. Different from uniform mutation operator, the gene mk , randomly selected from M, is mutated according to the normal distribution possibilities N (mk , 32 ). In GA, the crossover operator is used as the main operator because of good global search ability while the mutation operator is used as an auxiliary operator because of the good local search ability.

6.1.3.5 Control parameter setting In the GA, the control parameters mainly include the crossover probability Pc and the mutation probability Pm .

The crossover probability The crossover operator is the main operator to generate a new individual which determines the global search ability. The performance is affected by the value of crossover operator. The smaller the crossover probability is, the slower the new individuals are. The excellent mode is destroyed by bigger crossover probability. Therefore, the general value is between 0.4 and 0.99.

The mutation probability The convergence and the final solution are directly affected by the size of the mutation probability. The bigger mutation probability makes the algorithm constantly explore new solution space and increases the diversity. Premature convergence is prevented. However, the population gene is easy to be destroyed. When the mutation probability is small, the convergence can be accelerated in the neighborhood near the optimal solution. However, inadequate variation and stagnation appear. Therefore, the general value is between 0.001 and 0.1 [171].

6.1 Basic principles of evolutionary neural networks

157

Coding and generating initial population

Individual fitness testing and evaluation

Selection and replication

Crossover

Mutation

N

Assessing fitness Y End

Figure 6.1: The basic process of genetic algorithm.

6.1.4 The basic process of GA GA is a kind of search algorithm with the mode “generation + examination.” The problem parameter space is instead by the coding space. The fitness function is regarded as the evaluated standard. The population coding is the genetic basis. Genetic mechanism is realized by genetic operation in groups and an iterative process [170]. The process is shown in Figure 6.1. The chart above can be explained as follows: (1) Randomly choose N individual as the initial group from individual group. Each individual is regarded as chromosome. A new group is generated by N chromosomes; (2) Calculate the fitness value of each individual in the group; (3) Select the desired individual from the group by selection and reproduction, and put them into the mating pool; (4) The crossover and mutation operators are used to form N individuals in the next generation group, and the fitness value of each new individual is calculated; (5) If the ending condition is satisfied, stop loop; otherwise go to step (3).

158

6 Blind Equalization Algorithm Based on Evolutionary Neural Network

6.1.5 Characteristics of GA 6.1.5.1 The advantages of GA The main advantages of GA [174] are as follows: (1) Global stability: GA is not easy to fall into the local extreme points in the search process. That is, the optimal solution is easier to get even in the case of noncontinuous function and noise. So GA has the strong capability of noise immunity. (2) Parallel and high efficiency and effectiveness: GA has the characteristics of largescale global search and parallelism, so it is suitable for parallel computing. (3) Robust: Robustness means that groups instead of initial value play an important role in GA search. It is tested that the same results for a similar problem are got by different iterations. (4) Universal and easy scalability: Natural evolution mechanism is used to represent complex phenomenon in GA. The function form can be different. A variety of optimization search problem can be solved by GA. New problems can be solved by adjusting operator parameters, which is beyond the reach of the other optimization methods. (5) Simplicity: The basic idea of GA is simple and easy to understand. 6.1.5.2 The deficiency of GA The main deficiencies of GA are the following [175]: (1) The weak local search ability; (2) Weak adaptability to space change search. (3) The premature convergence phenomenon. (4) The optimal individual in the offerings individuals is worse than the optimal one in parent individual. This phenomenon is called “degradation.”

6.1.6 The integration mechanism of GA and neural network Neural network and GA are computation intelligence methods based on bionics. The advantages are neural network and GA are used to solve some nonlinear optimization problems. It has become an important part of intelligence information processing in future. The combination methods between GA and neural network are various. Many combination methods between GA and neural network have been developed. There exist three forms in neural network optimized by GA [175]: 6.1.6.1 The neural network connection weights optimized by GA All information about neural network system is contained in the location of neural network connection. A good weight distribution can be obtained by certain weight

6.2 Blind equalization algorithm based on neural network weights optimized by GA

159

iterative rules in conventional approaches. It may lead to long training time, and even fall into a local extreme. As a result, the proper weight distribution cannot be got. GA is used to overcome the above defects.

6.1.6.2 The structure of neural network optimized by GA The structure of neural network includes network topology composition, which is network connection, and node transfer function. The processing ability is obviously affected by the network structure. The problem can be successfully solved by a good structure and there is no redundancy nodes and connections. However, the design of neural network structure still relies on the experiences. There is no theoretical guidance. Structure optimization can be transformed into biological evolution process. The optimal solution of structure optimization problem is got by hybridization and mutation. GA is parallel and derivative is not needed. Mathematical programming method used in structure optimization can be overcome. Therefore, compared with other optimization algorithms, GA is more suitable for the complicated structure optimization problem.

6.1.6.3 Neural network learning rules optimized by GA Neural network system function is decided by learning rules in a certain sense. Currently the learning rules are set in advance before training. The GA is adopted to optimize the neural network learning rules to satisfy the problems and environment.

6.2 Blind equalization algorithm based on neural network weights optimized by GA 6.2.1 The basic algorithm principle The principle block diagram of the blind equalization algorithm based on neural network weights optimized by GA is shown in Figure 6.2. To some extent, the final solution of the training is determined by the initial weights of neural networks. The training time and convergence speed are largely different because of the different initial weights. So the distribution of the initial weights is very important. In order to find a suitable initial weight and avoid the premature, the global search ability of GA and fast local search speed of BP are utilized to obtain the suitable initial weights. First, the initial weights of the neural network are got by the good global search ability of GA; Second, the neural network is trained by

160

6 Blind Equalization Algorithm Based on Evolutionary Neural Network

n(n) x(n)

Channel h(n)

+

y(n)

Neural network

~ x (n)

Judgment

x̂ (n)

GA blind equalization Figure 6.2: The principle block diagram of blind equalization algorithm based on neural network weights optimized by GA.

BP algorithm; Finally, the best connection weights of the neural network are searched. The local minimum is avoided and the convergence speed is faster. In the blind equalization algorithm based on neural network weights optimized by GA, the choice of chromosome encoding, the design of fitness function, and the structure of genetic operators are key parts.

6.2.2 Feed-forward neural network weights optimized by GA blind equalization algorithm (GA-FFNNW) in binary encoding 6.2.2.1 Algorithm form In GA-FFNNW blind equalization algorithm, the three-layer feed-forward structure is used, as shown in Figure 3.20. The state equation is shown in eqs. (3.51)–(3.56). Given the random initial weights, the training process is divided into two stages. We use GA for fast global search. The network performance is achieved by controlling the iterative number of GA. The second stage is the local optimization stage of weights. BP algorithm is used to search for the local optimal solution until to meet precision of system. A. The optimization stage of initial weight The main steps of initial weights optimized by GA are as follows: (1) Randomly generate the initial group for GA; encode initial weight of each individual by the binary encoding program; construct the gene code chain. Each gene code chain is corresponding to a specific value of neural network. (2) Design the fitness function and calculate the fitness of each individual. The individual with the largest fitness is connected to the next generation, and the individuals who have small fitness are eliminated. (3) Generate the next generation group by dealing with the current generation of the population by crossover and mutation operators. (4) Repeat steps (2) and (3) until the training objectives are satisfied.

6.2 Blind equalization algorithm based on neural network weights optimized by GA

(5)

161

When the iteration stop condition is satisfied, the individual is the optimal individual. The decoding is regarded as the parameters of the neural network. The network is trained by the traditional algorithm to realize the local optimization.

Binary encoding In the binary coding of neural network, each connection weight is represented by the fixed length binary bits. For each individual, the optimal individual is obtained by genetic operation. The set of parameters in the solution space is restored by the decoding process. The connection weights are arranged closely at the hidden layer nodes which are regarded as the center. Neural network feature is easier to be extracted by the implicit layer node. If the location distance of hidden layer nodes in chromosomes is large, the good characteristics in the evolutionary process may be destroyed by crossover operator. So it is important to design the crossover operator. In the binary encoding, the digit selection depends on the range of weight value. A binary three-layer feed-forward neural network is shown in Figure 6.3. Because of the smaller range of weight, each connection weight can be represented by four-bit binary digits. The binary coding is simple and intuitive. The hardware implementation is also very convenient, because the computer is processing binary bits directly. There are two problems with binary encoding. The one is that the precision be inversely proportional to the coding length. The other one is there is a many-to-one mapping between encoding (genotype) and the actual network (phenotype). The combinatorial problem of crossover operation lead to low efficiency of algorithm and difficultly produce the worthy generation, as shown in Figure 6.3(b).

Design of the fitness function Cost function iteration to a minimum value is the goal of neural network blind equalization algorithm. The largest adaption function value is the goal of GA calculation. So the minimum value should be transformed into maximum problem. Neural network blind equalization algorithm is to find the optimum value of a cost function. The goal of GA is to obtain the maximum value of fitness function. As a result, the minimum problem should be transformed into a maximum problem.

6 9 4 5 7 (a)

10

0110

0101

1001 0100

0111

1010

0101

0110

1001 0100

0111

1010

0110

0101

1001

0111

0100 1010

0101

0110

1001

0111

0100 1010

(b)

Figure 6.3: The case of binary encoding: 4-bit binary encoding is adopted in this section.

162

6 Blind Equalization Algorithm Based on Evolutionary Neural Network

In this chapter, the multiplicative inverse of constant modulus cost function is regarded as the fitness function, namely F(n) =

1 2 = 2 2 󵄨 J(n) [󵄨󵄨󵄨x(n) 󵄨 󵄨 ̃ 󵄨󵄨 – R2 ]

(6.4)

where J(n) is the cost function of constant modulus blind equalization algorithm, as shown in eq. (3.11).

Selection of genetic operator The genetic operator includes the reproduction operator, crossover operator, and mutation operator. In general, the roulette operator is used for the selection operator. The choice probability of each individual is proportional to its fitness value, as shown in eq. (6.1). Single point crossover is used for crossover operator. Bit mutation operator is used for mutation operator.

Stop of the estimation iteration In the weights iteration of neural network optimized by GA, GA iteration times Gen and maximum fitness value Fmax are used to determine the stop of the estimation iteration. Gen = 50 is used as the termination condition of the algorithm in the chapter. B. The stage of weight local optimization A global optimized initial weight is obtained when the termination condition is satisfied. The local optimal search is carried out by the traditional BP algorithm until the accuracy requirements are met. In BP training algorithm, the cost function is shown in eq. (3.11) and the transfer function is f (x) = x + !

ex – e–x ex + e–x

(6.5)

where ! is the proportion parameter. The purpose is to make the network applicable to different modulation signals. In order to guarantee the monotonicity of transfer function, the derivative of transfer function must be positive. If the signal amplitude is larger, ! should be larger, if not, ! should be smaller. According to the steepest descent method, the iteration formula of each layer weight can be obtained. For three-layer neural network, the iterative formula weights of hidden layer and output layer are different. From eqs. (3.51) to (3.56), the following function can be obtained:

6.2 Blind equalization algorithm based on neural network weights optimized by GA

163

J

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f2󸀠 (∑ wj (n)vjJ (n)) vjJ (n) wj (n + 1) = wj (n) – 2,1 x(n)

(6.6)

j=1 J

I

j=1

i=1

̃ [󵄨󵄨󵄨󵄨x(n) ̃ 󵄨󵄨󵄨󵄨2 – R2 ] f2󸀠 (∑ wj (n)vjJ (n)) f1󸀠 (∑ wij (n)y(n – i)) wij (n + 1) = wij (n) – 2,2 x(n) × wj (n)y(n – i)

(6.7)

where ,1 and ,2 are the iterative step sizes of output layer and hidden layer, respectively.

6.2.2.2 Computer simulation 2PAM signals are adopted in computer simulation. The 3-layer (or 3-5-1) neural network is performed in the simulation. Typical telephone channel and common channel are used as simulation channel, respectively. The transmission functions are shown in eqs. (2.85) and (3.13). The GA initial population size is 80; Gen is 50; crossover probability is Pc = 0.85; mutation probability is Pm = 0.01; selection probability is Ps = 0.06. The individual optimization processes by GA in common and typical telephone channels are shown in Figures 6.4 and 6.5, respectively. The iterative trend is the same as adaptation degree increase, so the direction of the algorithm evolution is correct.

Fitness value

0.4 0.3

Largest fitness value

0.2 Average fitness value

0.1 0

0

10 20 30 40 Iterative number of GA

50

Figure 6.4: Optimization process of GA in common channel.

Fitness value

0.20 Largest fitness value

0.15 0.10

Average fitness value

0.05 0

0

10 20 30 40 Iterative number of GA

50

Figure 6.5: Optimization process of GA in typical telephone channel.

164

6 Blind Equalization Algorithm Based on Evolutionary Neural Network

101

Training error

100

10–1

10–2

10–3

0

50

100 150 Training cycle

200

Figure 6.6: Training curve of neural network optimized by GA.

500 Training cycle

1,000

Figure 6.7: Training curve of neural network unoptimized by GA.

101

Training error

100

10–1

10–2

10–3

0

The BP neural network weight curves are given in Figures 6.6 and 6.7. Figure 6.6 shows the neural network weight curves after the initial weights optimized by GA. Figure 6.7 shows the neural network weight curves before the initial weights optimized by GA. The predetermined error accuracy is 10–3 . As shown in the curves, the neural network weights optimized by GA only need 234 iterations to achieve the predetermined precision while the neural network weight unoptimized by GA need 1,198 iterations. It shows that the introduction of GA greatly improves the training speed of the neural network weights. The GA is used to determine the initial weight of the neural network in advance. As a result, the convergence rate is obviously accelerated. To test algorithm validity, white Gaussian noise is added into the input. The signal-to-noise ratio (SNR) is 25 dB. The number of iteration is 2,000. Convergence comparisons between BP-FFNN and GA-FFNNW in the common channel and typical telephone channel are shown in Figures 6.8 and 6.9, respectively . The iterative step is ,1 = ,2 = 0.02.

6.2 Blind equalization algorithm based on neural network weights optimized by GA

165

0.030 0.025 BP-FFNN

MSE

0.020 0.015 GA-FFNNW 0.010 0.005 0

0

500 1,000 1,500 Iterative number

2,000

Figure 6.8: Convergence curve of 2PAM signal in common channel.

0.5 BP-FFNN

MSE

0.4

0.3 GA-FFNNW 0.2

0.1

0

0

500 1,000 1,500 Iterative number

2,000

Figure 6.9: Convergence curve of 2PAM signal in typical telephone channel.

As shown in the curves, the GA-FFNNW is superior to the traditional BP-FFNN blind equalization algorithm in terms of convergence speed and steady-state residual error [49]. The simulation points are 20,000. The unequalized and equalized signals in common and typical telephone channels are shown in Figures 6.10 and 6.11, respectively. The unequalized signals are shown in the upper part of Figures 6.10 and 6.11, respectively, while the equalized signals are shown in the lower part. As shown in the graph, the unequalized signals are not recognizable and hardly restored. The equalized signals in lower half part are converged and restored easily.

Effect after equalization

Effect before equalization

Effect after equalization

Effect before equalization

166

6 Blind Equalization Algorithm Based on Evolutionary Neural Network

5

0

–5

0

2,000

4,000 6,000 Time k

8,000

10,000

0

2,000

4,000 6,000 8,000 Number of samples

10,000

0

2,000

4,000

8,000

10,000

0

2,000

4,000 6,000 8,000 Number of samples

10,000

1

0

–1

Figure 6.10: Constellation before and after equalization in the common channel.

5

0

–5

6,000

1

0

–1

Figure 6.11: Constellation before and after equalization in the typical telephone channel.

The bit error rate changes with SNR (BER-SNR) comparison curves are shown in Figures 6.12 and 6.13, respectively. Figure 6.12 gives the BER-SNR comparison curves between BP-FFNN and GA-FFNNW in common channel. Figure 6.13 gives the BER-SNR comparison curves between BP-FFNN and GA-FFNNW in typical telephone channel. It can be seen that the GA-FFNNW blind equalization algorithm reduces the BER.

6.2 Blind equalization algorithm based on neural network weights optimized by GA

167

100

BP-FFNN

BER

10–1

10–2

GA-FFNNW

10–3

10–4

5

10

15

20 25 SNR (dB) Iterative number

30

35

30

35

Figure 6.12: SNR-BER comparison between BP-FFNN and GA-FFNNW blind equalization algorithms in the common channel.

100

BP-FFNN

BER

10–1

10–2

GA-FFNNW

10–3

10–4

5

10

15

20 25 SNR (dB) Iterative number

Figure 6.13: SNR-BER comparison between BP-FFNN and GA-FFNNW blind equalization algorithm in typical telephone channel.

6.2.3 Real encoding GA-FFNNW blind equalization algorithm 6.2.3.1 Algorithm form In order to overcome the shortcomings of the binary encoding, real encoding can be used in the section.

168

6 Blind Equalization Algorithm Based on Evolutionary Neural Network

6 9 4 5

6.0 7

(a)

NN

5.0

9.0 4.0

7.0 10.0

10 (b)

Decimal network

Figure 6.14: The case of decimal coding. (a) NN and (b) decimal network.

The real encoding In the real encoding of neural network, each connection weight is represented by a fixed-length decimal representation, as shown in Figure 6.14. Real encoding is used to reduce the encoding length. The optimal individual directly obtained by the genetic operation is the weight of the neural network. This method is simple because of no decoding process. Design of fitness function Similar with the binary encoding program, the reciprocal of constant modulus algorithm cost function is used as the fitness function, as shown in eq. (6.4). Selection of genetic operator The roulette operator is used as selection operator. Each individual choice probability is proportional to the fitness value. The expression is shown in eq. (6.1); the singlepoint crossover is adopted in the chapter. The normal mutation operator is used as mutation operator. Iteration terminal condition In this chapter, the iteration terminal condition is given as Gen = 50. The iterative formulas of weight local optimization are shown in eqs. (6.6) and (6.7). Therefore, the main steps for the initial weight optimization are as follows: (1) Randomly generate N individuals as the initial population. Each individual is a two-dimensional vector. Suppose that popsize = {(wi , +i ), i = 1, 2, . . . , N}, where wi is the connection weights, + is the Gauss variable. A complete neural network is represented by each individual. (2) k is the number of genetic iterations. Suppose that k = 1, the individual variation is carried out according to the normal mutation operator. (3) Calculate the individual fitness values according to the fitness function shown in eq. (6.4). (4) Compare the random individual with the individual selected randomly from (wi , +i ) and (wi+1 , +i+1 ). If the fitness is large, the individual is copied to the next generation; repeat it until popsizei = N. (5) Go to step (2) if the termination condition is met. Otherwise, let k = k + 1.

6.2 Blind equalization algorithm based on neural network weights optimized by GA

169

When the iterative terminal condition is satisfied, the individual is both the optimal individual and the connection weight of neural network. Then, the local optimization is obtained by BP algorithm. 6.2.3.2 Computer simulation 4PAM is used as the input signal. The 3-layer (or 3-5-1) neural network is performed in the simulation. The typical telephone channel and common channel are adopted. The transmission functions are shown in eqs. (2.85) and (3.13), respectively. The initial population size of GA is 80. The iterative number of GA is 50 and selection probability is Ps = 0.06. Crossover probability is Pc = 0.85; mutation probability is Pm = 0.01. The individual processes optimized by GA in common and typical telephone channels are shown in Figures 6.15 and 6.16, respectively.

Fitness value

8

Largest fitness value

6

4 Average fitness value 2

0

0

20 40 Iterative number of GA

60

Figure 6.15: Individual optimized by GA in the common channel.

Fitness value

2

Largest fitness value 1

Average fitness value

0

0

20 40 60 Iterative number of GA

Figure 6.16: Individual optimized by GA in the typical telephone channel.

170

6 Blind Equalization Algorithm Based on Evolutionary Neural Network

In Figures 6.15 and 6.16, the upper curve is the largest fitness values while the lower curve is the average fitness values. The directions of iterative and fitness increasing are the same, so the new algorithm is tested to be right. At the same time, the fitness values do not increase when the iteration is 10 for the common channel and 25 for the typical telephone channel. The real coding optimization neural network is more easily to seek to the optimizations. The system is added with the white Gaussian noise (SNR = 20). The iterations is 10,000 and 15,000 while the step size factors are ,1 = ,2 = 0.02. The convergence curves of 4PAM signals in the two channels are given in Figures 6.17 and 6.18, respectively.

0.10 0.09 0.08

BP-FFNN

0.07 MSE

0.06 0.05 0.04 GA-FFNNW

0.03 0.02 0.01 0

0

2,000 4,000 6,000 8,000 10,000 Iterative number

Figure 6.17: Convergence curve of 4PAM signal in common channel.

0.10 0.09

GA-FFNNW

0.08 0.07 MSE

0.06 0.05 0.04 BP-FFNN

0.03 0.02 0.01 0

0

5,000 10,000 Iterative number

15,000

Figure 6.18: Convergence curve of 4PAM signal in a typical telephone channel.

6.2 Blind equalization algorithm based on neural network weights optimized by GA

171

Effect before equalization

In Figures 6.17 and 6.18, the steady-state residual error convergence rate by GAFFNNW blind equalization algorithm is faster compared with the traditional FFNN blind equalization algorithm [49]. The unequalized and equalized signals constellations in common and typical telephone channels are given in Figures 6.19 and 6.20, respectively. The simulation signal is 4PAM; the iteration step size is 0.02; the population size is 120; the probability of selection is Ps = 0.06; the crossover probability is Pc = 0.85; the mutation probability is Pm = 0.01; the maximum iterative number is 1,000; and the simulation point is 40,000. Equalization performance is satisfied by GA-FFNNW in both common and typical telephone channels. However, the equalization performance for 4PAM is worth than it for 2PAM. That is because the signal bit number is increased. The GA-FFNNW in signal multiamplitude modulated equalization need to be improved. The BER-SNR curves in different channels are shown in Figures 6.21 and 6.22, respectively. The sampling length is 5,000 bits. From the curves, the BER is almost the same in low SNR for FFNN and GA-FFNNW, the BER by GA-FFNNW is lower when SNR is more than 20 dB. So GA-FFNNW blind equalization algorithm is better at high SNR cases.

5

0

–5

0

2,000

0

2,000

8,000

10,000

4,000 6,000 8,000 Number of samples

10,000

4,000

6,000

Effect after equalization

5

0

–5

Figure 6.19: Constellation before and after equalization in the common channel for 4PAM.

172

6 Blind Equalization Algorithm Based on Evolutionary Neural Network

Effect before equalization

5

0

–5

0

2,000

0

2,000

8,000

10,000

4,000 6,000 8,000 Number of samples

10,000

4,000

6,000

Effect after equalization

5

0

–5

Figure 6.20: Constellation before and after equalization in the typical telephone channel for 4PAM.

100 BP-FFNN

BER

10–1

10–2

GA-FFNNW

10–3

10–4 10

15

20 25 SNR (dB)

30

35

Figure 6.21: Comparison of BER for 4PAM signal in common channel.

6.3 GA optimization neural network structure blind equalization algorithm (GA-FFNNS) 6.3.1 The basic algorithm principle There is no complete theoretical guidance for the structural design of the neural network so far. The more the hidden layer are, the stronger the network approximation

6.3 GA optimization neural network structure blind equalization algorithm (GA-FFNNS)

173

100

BP-FFNN

BER

10–1

10–2 GA-FFNNW 10–3 10

15

20 25 SNR (dB)

30

35

Figure 6.22: Comparison of BER for 4PAM signal in typical telephone channel.

n(n) x(n)

Channel h(n)

+

y(n)

Neural network

~ x (n)

Judgment

x̂ (n)

Genetic algorithm blind equalization algorithm Figure 6.23: The principle of GA-FFNNS blind equalization algorithm.

ability is. However, the complexity increase leads to over-fitting input sequences. So the proper design of the neural network structure is very important. In this section, the neural network structure is optimized by GA. The principle diagram is shown in Figure 6.23. When the topology and connection weights of the neural network are combined in encoding, the structure of the neural network and the connection weights can be optimized by GA simultaneously.

6.3.2 Algorithm derivation 6.3.2.1 Encoding In order to realize the dynamic coordination optimization of network topology and weight training, each chromosome of network topology and weight vector coding should be included. Encoding mode can be binary or real. All the nodes and connection weights in neural network can be encoded into the chromosomes. Namely, all connections in the structure can be expressed by binary

174

6 Blind Equalization Algorithm Based on Evolutionary Neural Network

bits. If there are K nodes, the network structure and weight distribution can be expressed by K-rank square matrix W: W = [wij (n)]K×K

(6.8)

where wij (n) is the connection weight from node i to j. If there exists network connection, wij = 1; Otherwise, wij = 0. The final optimization chromosome is the optimal network topology. A simple TFNN is shown in Figure 6.24 (a). There are five weight nodes: w13 = 4, w14 = 6, w23 = 7, w24 = 5, w45 = 9, and w35 = 10 respectively and the rest is zero. The K-rank square matrix W is shown in Figure 6.24 (b). The matrix structure is shown in Figure 6.24 (c). The structure weight matrix is shown in Figure 6.24 (d). The network structure encoding is 0 0 1 1 0 0 0 1 1 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 The encoding of the network structure and weight is 0 0 4 6 0 0 0 7 5 0 0 0 0 0 10 0 0 0 0 9 0 0 0 0 0 This kind of coding method is more intuitive and easy to be realized. The connection weight is only corresponding to encoding matrix element. It is more convenient to add or delete connection weights. So the neural network structure is easier to be optimized by encoding. If the binary bit is replaced by real, real encoded occurs. In the neural network structure optimized by GA, the difference between binary encode and real encode is just the operator.

6

1

4

9

4

5

5 2

7

3

10

w11 w21 w31

w12 w13 w22 w23 w32 w33

w41

w42 w43 w44 w45

w51

w52 w53

w14 w15 w24 w25 w34 w35 w54 w55

(b)

(a) 0 0 1 1 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 (c)

0 0 1 1 0

0 0 0 0 0 (d)

0 0 0 0 0

4 7 0 0 0

6 0 5 0 0 10 0 9 0 0

Figure 6.24: The neural network structure and weight binary encoding structure.

6.3 GA optimization neural network structure blind equalization algorithm (GA-FFNNS)

175

6.3.2.2 Fitness function design The reciprocal of constant modulus cost function is used as the fitness function, as shown in eq. (6.4). 6.3.2.3 Selection of genetic operator The expected value operator is used as the selection operator, shown in eq. (6.2). The uniform crossover operator is used as the crossover operator. The uniform mutation operator is used as the mutation operator. 6.3.2.4 Iteration terminal conditions In this case, the iteration terminal condition is given as Gen = 50.

6.3.3 Computer simulation The input sequence is 4PAM signal. The simulation channels are typical telephone channel and common channel. The transmission functions are shown in Eqs. (2.85) and (3.13). The initializing neural network structure is generated randomly. In the common channel, K = 100 is neural network node. The initial population size is 80. The iterative number is 50. Pc = 0.85 is the crossover probability; Pm = 0.0025 is the mutation probability. Individual search process by GA-FFNNS is shown in Figure 6.25. From the figure, it is easy to get the conclusion that search direction of the algorithm is toward the optimal direction. The convergence comparison curves by GA-FFNNS and FFNN [49] in different channels are shown in Figures 6.26 and 6.27, respectively. The simulation signal is 4PAM and the SNR is 20 dB. From the chart, steady residual error in GA-FFNNS blind equalization algorithm is lower than the one in FFNN blind equalization algorithm. However, the convergence 0.10

Fitness value

0.08 Largest fitness value

0.06

Average fitness value

0.04 0.02 0

0

20

40 60 80 Iterative number of GA

100

Figure 6.25: Individual optimization process by GA-FFNNS.

176

6 Blind Equalization Algorithm Based on Evolutionary Neural Network

0.10

MSE

0.08

Structure optimization FNN blind equalization

0.06 0.04

FNN blind equalization

0.02 0

0

2,000

4,000 6,000 Iterative number

8,000

10,000

Figure 6.26: The convergence curve of GA-FFNNS in the common channel.

0.12 0.10 Structure optimization FNN blind equalization

MSE

0.08 0.06

FNN blind equalization

0.04 0.02 0

0

5,000

10,000

Iterative number

15,000

Figure 6.27: The convergence curve of GA-FFNNS in the typical telephone channel.

speed in GA-FFNNS is reduced either. The reason is GA divergent search. Although the algorithm is ensured to tend to the optimal, search time is extended inevitably. The computational complexity increases due to the network complexity. So the slow convergence problem occurs. Especially when the population size and genetic algebra increase, the problem will become slower. The problem needs to be solved in the GA application. There are three algorithms: FFNN blind equalization algorithm, GA-FFNNW blind equalization algorithm, and GA-FFNNS blind equalization algorithm. The comparisons convergence curve in the tree algorithms above in different channels are given in Figures 6.28 and 6.29, respectively. In the three algorithms, the steady residual error of FFNN is the largest while the steady residual error of GA-FFNNS blind equalization algorithm is the smallest. The converse speed of GA-FFNNS is the slowest. The converse speed of GA-FFNNW is the fastest, because the initial weight distribution is optimized. The steady residual error of GA-FFNNS blind equalization algorithm decreases. However, when the computational complexity increases , the convergence speed slows. For 4PAM input signal, the unequalized and equalized signals by GA-FFNNS blind equalization algorithm in the two channels are shown in Figures 6.30 and 6.31,

6.3 GA optimization neural network structure blind equalization algorithm (GA-FFNNS)

177

0.12 0.10

GA-FFNNS

MSE

0.08 0.06 0.04 GA-FFNNW

BP-FFNN

0.02 0

0

5,000

Figure 6.28: Convergence curves comparison in three algorithms in the common channel.

500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 Iterative number

Figure 6.29: Convergence curves comparison in three algorithms in the typical telephone channel.

1,000

2,000 3,000 Iterative number

4,000

0.10 GA-FFNNS

MSE

0.08 0.06 0.04 GA-FFNNW

0.02

Effect before equalization

0

0

BP-FFNN

5 0

–5 0

2,000

4,000 6,000 Time k

8,000

10,000

Effect after equalization

1

0

–1 0

2,000

4,000 6,000 8,000 Number of samples

10,000

Figure 6.30: The unequalized and equalized signals by GA-FFNNS blind equalization algorithm in the common channel.

178

Effect before equalization

6 Blind Equalization Algorithm Based on Evolutionary Neural Network

5 0 –5 0

2,000

4,000

6,000

8,000

10,000

Time k

Effect after equalization

1

0

–1 0

2,000

4,000 6,000 8,000 Number of samples

10,000

Figure 6.31: The unequalized and equalized signals by GA-FFNNS blind equalization algorithm in the typical telephone channel.

100 FFNN

BER

10–1

10–2 GA-FFNNS

10–3 10–4 10

15

25 20 SNR (dB)

30

35

Figure 6.32: BER-SNR by GA-FFNNS and FFNN blind equalization algorithm in the common channel.

respectively. Simulation conditions are the same as the former. The samples are 40,000 in the simulation. It can be found from the graph that the equalized signals can be effectively judged, and the performance is more stable. The BER-SNR for 4PAM comparisons between FFNN and GA-FFNNS in the two channels are shown in Figures 6.32 and 6.33, respectively. The recovered signals by different algorithms have different BER after 5,000 iterations. As shown in the graph, GA-FFNNS blind equalization algorithm has smaller BER after convergence.

6.4 Summary

179

100 FFNN

BER

10–1

10–2 GA-FFNNS

10–3 10–4

10

15

20 25 SNR (dB)

30

35

Figure 6.33: BER-SNR by GA-FFNNS and FFNN blind equalization algorithm in the typical telephone channel.

6.4 Summary The GA-FFNNW and GA-FFNNS blind equalization algorithms are studied in this chapter. How to choose genetic operators is described and the step of the algorithm is analyzed. Finally, computer simulations are given. The global searching ability of GA and local search speed of BP are adopted by GAFFNNW blind equalization algorithm. The local minimum is avoided to fall into and the convergence speed is accelerated. In the two algorithms, the binary and real encoding are used. The reciprocal of constant modulus cost function is used as the fitness function. Roulette operator is used as selection operator; The single point crossover operator is used as crossover operator. The simple mutation operator used in binary encoding is used as mutation operator. The normal mutation operator is used in real encoding. Computer simulation shows that the convergence speed and steady residual error are improved more significantly by the new algorithm than the traditional neural network blind equalization algorithm. The binary and real encoding can be adopted in GA-FFNNS blind equalization algorithm. The reciprocal of constant modulus cost function is adapted as the fitness function. The expected value operator is used as selection operator; the uniform crossover operator is used as crossover operator; and the uniform mutation operator is used as mutation operator. Computer simulation shows that the convergence speed of the new algorithm is lower than the one of the traditional neural network blind equalization algorithm. However, the steady residual error is reduced. This method is simple, but the genetic optimization computation is great. When neural network optimization is designed to solve complex problems, the number of neurons will increase. The total number of connection weights and the computation complex will increase rapidly. The convergence speed slows down.

7 Blind equalization Algorithm Based on Wavelet Neural Network Abstract: The concept, structure, and characteristics of wavelet neural network are described in this chapter. The basic principle of wavelet neural network blind equalization algorithms is analyzed. Blind equalization algorithm based on three-layer feed-forward wavelet neural network and bilinear feedback wavelet neural network are studied. Iterative formulas are derived for the real and complex system. The effectiveness is verified by computer simulation.

7.1 Basic principle of wavelet neural network 7.1.1 The concept of wavelet neural network 7.1.1.1 The wavelet transform The wavelet was first proposed by Haar in the paper about Hilbert space properties in 1910. The wavelet gets rapid development in 1980s. In 1982, an infinite support, orthogonal piecewise polynomial wavelet (wavelet basis) was proposed by Stromberg. However, it did not attract people’s attention. In 1984, the concept of wavelet was first proposed by French geophysicist J. Morlet in the analysis of geophysical data. The theory of wavelet analysis was established by theoretical physicist A. Grossman. In 1985, the smooth wavelet with the attenuation was first constructed by the French mathematician Y. Meyer. In 1988, the existence of the compactly orthogonal wavelet is proved by Belgian mathematician I. Daubechies, which makes the discrete wavelet analysis possible. The concept of multiresolution analysis is proposed by S. Mallat in 1989, in which the various methods of constructing wavelet are unified. Especially, the fast algorithm of the binary wavelet transform is proposed, which makes the wavelet transform into practice. Time-frequency window methods are adopted in wavelet. The contradiction between time resolution and frequency resolution is solved by wavelet transform. There are good localization properties in both time and frequency domain. The wide time window is used in the low-frequency components to get high-frequency resolution ratio. The narrow time window is used for in the high-frequency components to get low-frequency resolution ratio. The adaptability of wavelet transform leads to a wide use in the engineering technology and signal processing. 7.1.1.2 The concept of wavelet neural network Wavelet neural network is a new type of neural network based on wavelet analysis. It can also be regarded as a new function connection network. In 1992, a multilayer DOI 10.1515/9783110450293-007

7.1 Basic principle of wavelet neural network

181

neural network model based on wavelet analysis was proposed by Zhang Qinghua, the member of the famous French information science IRISA [176]. The wavelet functions instead of Sigmoid are regarded as the transfer function. The connection between the wavelet transform and neural network is established and used in function approximation. The local optimum fundamentally is avoided and the convergence rate speeds up. The discrete affine wavelet network model was proposed by Pati [177] in 1993. The orthogonal wavelet neural network and its learning algorithm are proposed by Baskshi and Stephanopoulous [178]. Orthogonal scaling function and wavelet function are used as the transfer function of neuron. Orthogonal wavelet neural network is proposed by Zhang Jun [179]. The multiwavelet neural network is proposed by Jiao [180]. The interval wavelet neural network model is proposed by Gao Xieping [181], and so on.

7.1.2 The structure of wavelet neural network There are two main ways of the combination between wavelet analysis and neural network. The one is “loose” type structure, as shown in Figure 7.1. The signal is preprocessed by wavelet and then sent into the neural network. The other is the “compact” type, as shown in Figure 7.2. It is also called the wavelet neural network or wavelet network [182].

Wavelet transform

Input

Neural network

Output

Figure 7.1: The “loose” structure of wavelet neural network.

y(n)

w00(n)

ψ

w00(n)

y(n – 1)

~ x1(n) ~ x2(n)

ψ y(n – m + 1)

~ xn(n) ψ

y(n – m) Input layer

wkn(n)

wmk(n) Hidden layer

Output layer

Figure 7.2: The “compact” structure of wavelet neural network.

182

7 Blind equalization Algorithm Based on Wavelet Neural Network

There are three layers which are input layer, hidden layer, and output layer in Figure 7.2. There are m + 1, k + 1, and n neurons in input layer, hidden layer, and output layer respectively. The output layer is linear. The wavelet is used as function transfer function of neurons in the hidden layer. The frequency localization properties of wavelet transform and neural network self-learning function are inherited in the wavelet neural network. The wavelet neural network is widely used in signal processing, data compression, pattern recognition, and fault diagnosis.

7.1.3 The characteristics of wavelet neural network (1) (2) (3)

Expansion and translation are used to do signal multiscale analyzing. The signal is analyzed by the local information of signal is effectively extracted. The neural network has the characteristics of self-learning, adaptive, and fault tolerance. It is a kind of universal function approximation. The advantages of wavelet transform and neural network are obtained. The function approximation and pattern classification make the wavelet neural network to be an asymptotically optimal approximation device.

7.2 Blind equalization algorithm based on feed-forward wavelet neural network 7.2.1 Algorithm principle The basic principle of blind equalization algorithm based on feed-forward wavelet neural network (FFWNN) is that FFWNN is used instead of FFNN (shown in Figure 2.4). The block diagram is shown in Figure 7.3.

n(n) x(n)

Channel h(n)

+

y(n)

feed-forward wavelet neural

~ x (n)

Judgment

x̂ (n)

w(n) Blind equalization Figure 7.3: The block diagram of blind equalization algorithm based on feed-forward wavelet neural network.

7.2 Blind equalization algorithm based on feed-forward wavelet neural network

183

7.2.2 Blind equalization algorithm based on feed-forward wavelet neural network in real number system 7.2.2.1 The structure of FFWNN The principle block diagram of FFWNN is shown in Figure 7.4. Wavelet transform is used as transfer function in the hidden layer neurons of network. The weight coefficient and wavelet transform scaling and translation operators are obtained by training network. The connection weights between input layer and the hidden layer are wij (n) (i = 0, 1, . . . , m; j = 0, 1, . . . , k) while the connection weights between hidden layer and output layer are wj (n). Suppose that the input of input layer is y(n – i); the input of hidden layer is uj (n); the output of hidden layer is Ij (n); the input of output layer is v(n); the ̃ ̃ is as follows: output layer is x(n). The state equation of x(n) m

uj (n) = ∑ wij (n)y(n – i)

(7.1)

i=0

Ij (n) = 8a,b [uj (n)]

(7.2)

k

v(n) = ∑ wj (n)Ij (n)

(7.3)

j=0

̃ x(n) = f (v(n))

(7.4)

where f (⋅) is the transfer function between the input and the output; 8a,b (⋅) is the 1 2

wavelet transform for the input of hidden layer. Morlet 8 (x) = xe– 2 x is adopted as the mother wavelet.

y(n)

w00(n) ψ w0(n)

y(n – 1) ψ

w1(n)

f(·)

~ x(n)

y(n – m + 1) wk(n) ψ y(n – m) Input layer

wmk(n) Hidden layer

Output layer

Figure 7.4: Feed-forward wavelet neural network with a hidden layer.

184

7 Blind equalization Algorithm Based on Wavelet Neural Network

1

x8a,b (x) = |a| 2 8 (

2

1 x – b – (x–b) x–b ) = |a| 2 e 2a2 a a

(7.5)

where a, b are expansion and translation scale factors, respectively. 7.2.2.2 Algorithm derivation When the wavelet neural network is used to solve the problem of blind equalization, the cost function and the transfer function of output layer are respectively shown in formulas (3.6) and (3.42). According to the steepest descent method, the iterative formula of network weights can be obtained. W(n + 1) = W(n) – ,

𝜕J(n) 𝜕W(n)

(7.6)

where , is iterative step size factor. 󵄨 ̃ 󵄨󵄨 󵄨󵄨 𝜕J(n) 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 󵄨 ̃ 󵄨󵄨2 = 2 (󵄨󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 – R2 ) 󵄨󵄨󵄨x(n) 𝜕W(n) 𝜕W(n)

(7.7)

Because there are hidden layer and the output layer in the wavelet neural network, the iterative formula of the weight are different. Iterative formula of output layer weights For the network output layer, the connection weights of the network output layer and the hidden layer are wj (n), 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 󵄨󵄨 = f 󸀠 [v(n)] Ij (n) 𝜕wj (n)

(7.8)

wj (n + 1) = wj (n) + ,1 k(n)Ij (n)

(7.9)

Put eq. (7.7) into eq. (7.8), then

󵄨 ̃ 󵄨󵄨2 󸀠 ̃ [v(n)]. where the iterative step size factor is ,1 , k(n) = –2 [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2] x(n)f Iterative formula of the hidden layer weights In the hidden layer, wavelet transform is performed on the input signal. For the hidden layer unit, the connection weights connected with the input layer are wij (n) 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 󵄨󵄨 𝜕v(n) = f 󸀠 [v(n)] 𝜕wIJ (n) 𝜕wij (n)

(7.10)

𝜕v(n) = wj (n)8󸀠a,b [uj (n)] yi (n – i) 𝜕wij (n)

(7.11)

7.2 Blind equalization algorithm based on feed-forward wavelet neural network

185

So the following function can be obtained by eqs. (7.10) and (7.11): 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 󵄨󵄨 = f 󸀠 [v(n)] 8󸀠a,b [uj (n)] wj (n)yi (n – i) 𝜕wij (n)

(7.12)

Take the derivative of eq. (7.5), 𝜕8󸀠a,b (x) 𝜕x

2 2 1 1 1 – 21 ( x–b x – b 2 – 21 ( x–b a ) – |a| 2 a ) e ( ) e a a a

1

= |a| 2

(7.13)

Plug eqs. (7.12) and (7.13) into eq. (7.7), wij (n + 1) = wij (n) + ,2 kj (n)y(n – i)

(7.14)

where kj (n) = 8󸀠a,b [uj (n)] wj (n)k(n), ,2 is the step factor of hidden layer.

Iterative formula of the expansion factor and translation factor According to the weight iterative formula, we can obtain 𝜕J(n) 𝜕a(n) 𝜕J(n) b(n + 1) = b(n) – ,4 𝜕b(n) a(n + 1) = a(n) – ,3

(7.15) (7.16)

where ,3 and ,4 are expansion factor and translation factor, respectively: 𝜕J(n) 𝜕a(n) 𝜕J(n) 𝜕b(n) 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 󵄨󵄨 𝜕a(n) 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 󵄨󵄨 𝜕b(n) 𝜕8󸀠a,b (x) 𝜕a(n)

󵄨 ̃ 󵄨󵄨 󵄨󵄨 󵄨 ̃ 󵄨󵄨2 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) = 2 [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2] 󵄨󵄨󵄨x(n) 󵄨󵄨 𝜕a(n) 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨 2 󵄨 ̃ 󵄨󵄨 󵄨󵄨󵄨x(n) 󵄨 ̃ 󵄨󵄨 = 2 [󵄨󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 – R2] 󵄨󵄨󵄨x(n) 𝜕b(n) = f 󸀠 [v(n)] = f 󸀠 [v(n)]

𝜕8󸀠a,b [uj (n)] 𝜕a(n) 𝜕8󸀠a,b [uj (n)] 𝜕b(n)

(7.17) (7.18)

wj (n)

(7.19)

wj (n)

(7.20)

2 2 1 1 – 21 ( x–b 1 x – b 2 – 21 ( x–b a ) + |a| 2 ( a ) e ) e 2 a a a2 2 1 x – b – 21 ( x–b a ) )e + ( z 2√|a| 1

= – |a| 2 (x – b)

𝜕8󸀠a,b (x) 𝜕b(n)

1

= – |a| 2

2 2 1 1 – 21 ( x–b 1 x – b 2 – 21 ( x–b a ) + |a| 2 ( a ) e ) e a a a

(7.21)

(7.22)

186

7 Blind equalization Algorithm Based on Wavelet Neural Network

Plug eqs. (7.21), (7.19), (7.17) into eq. (7.15), 󵄨 ̃ 󵄨󵄨2 󵄨 ̃ 󵄨󵄨 󸀠 a(n + 1) = a(n) – 2,3 [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2] 󵄨󵄨󵄨x(n) 󵄨󵄨 f [v(n)] wj (n) 1

× (– |a| 2 (x – b)

2 2 2 1 1 – 21 ( x–b 1 x – b 2 – 21 ( x–b 1 x – b – 21 ( x–b a ) + |a| 2 ( a ) a )) ) ) e e e + ( a z a2 a2 2√|a| (7.23)

Plug eqs. (7.22), (7.20), (7.18) into eq. (7.16), then 󵄨 ̃ 󵄨󵄨2 󵄨 ̃ 󵄨󵄨 󸀠 a(n + 1) = a(n) – 2,3 [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2] 󵄨󵄨󵄨x(n) 󵄨󵄨 f [v(n)] wj (n) 1

× (– |a| 2

2 2 1 1 – 21 ( x–b 1 x – b 2 – 21 ( x–b a ) + |a| 2 ( a ) e ) e ) a a a

(7.24)

According to the iterative formula above, the wavelet neural network blind equalization algorithm is obtained. 7.2.2.3 Computer simulation

0.10 0.08 0.06 0.04 0.02 0

(1) FFNN (2) FFWNN (1)

MSE

MSE

4PAM and 8PAM are used as input sequences. The SNR is 20 dB. The typical telephone channel and common channel are used for simulation. Its expression is shown in eqs. (2.85) and (3.13), respectively. The convergence comparison curves between FFWNN and FFNN by Cheolwoo You [58] in common and typical telephone channels are given in Figures 7.5 and 7.6,

(2)

0

(a)

5,000

10,000

15,000

Interative number

0.10 0.08 0.06 0.04 0.02 0

(1) FFNN (2) FFWNN (1)

0

(b)

5,000

(2)

10,000

15,000

Interative number

0.10 0.08 0.06 0.04 0.02 0 (a)

(1) FFNN (2) FFWNN MSE

MSE

Figure 7.5: Convergence curves of 4PAM: (a) and 8PAM (b) in the typical telephone channel.

(1) (2)

0

5,000

10,000

Iterative number

15,000

0.10 0.08 0.06 0.04 0.02 0 (b)

(1) FFNN (2) FFWNN

(1) (2)

0

5,000

10,000

Iterative number

Figure 7.6: Convergence curves of 4PAM (a) and 8PAM (b) in the common channel.

15,000

187

7.2 Blind equalization algorithm based on feed-forward wavelet neural network

Table 7.1: Bit error statistics of FFWNN and FFNN after the convergence. Modulation Algorithm and channel

Bit error Modulation Algorithm and channel

Bit error

4PAM

41

395

FFWNN Typical telephone channel Common channel FFNN Typical telephone channel Common channel

8PAM

FFWNN Typical telephone channel Common channel FFNN Typical telephone channel Common channel

52 84

88

452 871

888

respectively. In the typical telephone channel, the iterative step-size factors for 4 PAM are ,1 = ,2 = 0.01, ,3 = ,4 = 0.000055; for 8PAM, the iterative step-size factors are ,1 = ,2 = 0.003, ,3 = ,4 = 5.5×10–7 . In common channel, the iterative step-size factors for 4PAM are ,1 = ,2 = 0.018, ,3 = ,4 = 0.000055; for 8PAM, the iterative step-size factors are ,1 = ,2 = 0.003, ,3 = ,4 = 5.5 × 10–7 . The error statistics of the two algorithms (15,000 iterations) are given in Table 7.1.

7.2.3 Blind equalization algorithm based on FFWNN in complex number system 7.2.3.1 Algorithm derivation When the algorithm is extended to the complex value system, a similar method as FFNN in complex value in Chapter 3 can be used in iteration derivation. The cost function and transfer function are shown in the formulas (3.6) and (3.42), respectively. First, the network signal is written in the complex form: y(n – i) = yR (n – i) + jyI (n – i)

(7.25)

wij (n) = wij,R (n) + jwij,I (n)

(7.26)

uj (n) = ∑ wij (n)y(n – i) i

= ∑ [wij,R (n)yR (n – i) – wij,I (n)yI (n – i)] + j ∑ [wij,R (n)yI (n – i) – wij,I (n)yR (n – i)] i

i

(7.27) Ij (n) = >a,b (uj,R (n)) + j>a,b (uj,I (n)) wj (n) = wj,R (n) + jwj,I (n)

(7.28) (7.29)

v(n) = ∑ wj (n)Ij (n) = ∑ [wj,R (n)Ij,R (n) – wj,I (n)Ii,I (n)] i

i

+ j ∑ [wj,R (n)Ii,I (n) + wj,I (n)Ii,R (n)] i

(7.30)

188

7 Blind equalization Algorithm Based on Wavelet Neural Network

̃ x(n) = f (vR (n)) + jf (vI (n))

(7.31)

According to the steepest descent method, an iterative formula of network weights can be obtained W(n + 1) = W(n) – ,

𝜕J(n) 𝜕W(n)

(7.32)

where , is the iteration step-size factor 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 𝜕J(n) 󵄨 ̃ 󵄨󵄨2 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) = [󵄨󵄨󵄨x(n) +j 󵄨 ) 󵄨󵄨 – R2] 󵄨󵄨󵄨x(n) 󵄨󵄨 ( 𝜕W(n) 𝜕WR (n) 𝜕WI (n)

(7.33)

And the real situation is similar; because the three-layer complex-valued feed-forward neural network has the hidden layer unit and the output layer, so the weight of the iterative formula is different.

The weight iterative formula of output layer For the hidden layer of complex-valued FFWNN, the connection weights between the hidden layer and the output layer are wj (n) = wj,R (n) + jwj,I (n): 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕x(n) = = 󵄨󵄨 󵄨 󵄨 ̃ 󵄨󵄨 𝜕wj,R (n) 𝜕wj,R (n) 𝜕wj,R (n) 2 󵄨󵄨x(n) 2 2 1 𝜕 (f (vR (n)) + f (vI (n))) (7.34) = 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 𝜕wj,R (n) 2 󵄨󵄨x(n) 1 󸀠 󸀠 = 󵄨󵄨 󵄨󵄨 [f (vR (n)) f (vR (n)) Ij,R (n) + f (vI (n)) f (vI (n)) Ij,I (n)] ̃ 󵄨󵄨x(n)󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕x(n) = = 󵄨󵄨 󵄨 󵄨 ̃ 󵄨󵄨 𝜕wj,I (n) 𝜕wj,I (n) 𝜕wj,I (n) 2 󵄨󵄨x(n) 2 2 1 𝜕 (f (vR (n)) + f (vI (n))) (7.35) = 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 𝜕wj,I (n) 2 󵄨󵄨x(n) 1 󸀠 󸀠 = 󵄨󵄨 󵄨󵄨 [–f (vR (n)) f (vR (n)) Ij,I (n) + f (vI (n)) f (vI (n)) Ij,R (n)] ̃ x(n) 󵄨󵄨 󵄨󵄨 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 𝜕 󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 1 +j 󵄨 = [f (vR (n)) f 󸀠 (vR (n)) Ij,R (n) + f (vI (n)) f 󸀠 (vI (n)) Ij,R (n)] ̃ 󵄨󵄨󵄨󵄨 𝜕wj,R (n) 𝜕wj,R (n) 󵄨󵄨󵄨󵄨x(n) + j [f (vI (n)) f 󸀠 (vI (n)) Ij,I (n) – f (vR (n)) f 󸀠 (vR (n)) Ij,I (n)] 1 󸀠 󸀠 ∗ = 󵄨󵄨 󵄨󵄨 [f (vR (n)) f (vR (n)) + jf (vI (n)) f (vI (n))] Ij (n) ̃ x(n) 󵄨󵄨 󵄨󵄨

(7.36)

189

7.2 Blind equalization algorithm based on feed-forward wavelet neural network

Plug eq. (7.36) into eq. (7.33) wj (n + 1) = wj (n) + ,1 k(n)Ij∗ (n)

(7.37)

where ,1 is the iterative step-size factor of the output layer; k(n) 󵄨 ̃ 󵄨󵄨2 󸀠 󸀠 –2 (󵄨󵄨󵄨x(n) 󵄨󵄨 – R2 ) [f (vR (n)) f (vR (n)) + jf (vI (n)) f (vI (n))].

=

The weight iterative formula of the hidden layer For the hidden layer of the complex-valued FFWNN, the connection weights between the input layer and the hidden layer are wij (n + 1) = wij,R (n) + jwij,I (n), 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕x(n) = = 󵄨󵄨 󵄨 ̃ 󵄨󵄨󵄨 𝜕wij,R (n) 𝜕wij,R (n) 𝜕wij,R (n) 2 󵄨󵄨x(n) 2 2 1 𝜕 (f (vR (n)) + f (vI (n))) = 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 𝜕wij,R (n) 2 󵄨󵄨x(n) 𝜕vR (n) 𝜕vI (n) 1 + f (vI (n)) f 󸀠 (vI (n)) ] = 󵄨󵄨 [f (vR (n)) f 󸀠 (vR (n)) ̃ 󵄨󵄨󵄨󵄨 𝜕wij,R (n) 𝜕wij,R (n) 󵄨󵄨x(n)

(7.38)

Similarly, 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 󵄨󵄨 𝜕vR (n) 𝜕vI (n) 1 = + f (vI (n)) f 󸀠 (vI (n)) ] [f (vR (n)) f 󸀠 (vR (n)) ̃ 󵄨󵄨󵄨󵄨 𝜕wij,I (n) 󵄨󵄨󵄨󵄨x(n) 𝜕wij,I (n) 𝜕wij,I (n)

(7.39)

where 𝜕 (wj,R (n)Ij,R (n) – wj,I (n)Ij,I (n)) 𝜕vR (n) = 𝜕wij,R (n) 𝜕wij,R (n) = wj,R (n)8󸀠a,b (uj,R (n))

𝜕uj,R (n) 𝜕wij,R (n)

– wj,I (n)8󸀠a,b (uj,I (n))

𝜕uj,I (n) 𝜕wij,R (n)

(7.40)

= wj,R (n)8󸀠a,b (uj,R (n)) yR (n – i) – wj,I (n)8󸀠a,b (uj,I (n)) yI (n – i) 𝜕vI (n) = wj,R (n)8󸀠a,b (uj,I (n)) yI (n – i) + wj,I (n)8󸀠a,b (uj,R (n)) yR (n – i) 𝜕wij,R (n)

(7.41)

𝜕vR (n) = –wj,R (n)8󸀠a,b (uj,R (n)) yI (n – i) – wj,I (n)8󸀠a,b (uj,I (n)) yR (n – i) 𝜕wij,I (n)

(7.42)

𝜕vI (n) = wj,R (n)8󸀠a,b (uj,I (n)) yR (n – i) – wj,I (n)8󸀠a,b (uj,R (n)) yI (n – i) 𝜕wij,I (n)

(7.43)

In the hidden layer, wavelet transform is performed for the input signal. The Morlet is 1 2

8 (x) = xe– 2 x ,

190

7 Blind equalization Algorithm Based on Wavelet Neural Network

2 1 x–b x – b – 21 ( x–b a ) ) = |a| 2 ( )e a a

(7.44)

2 2 1 1 x – b 2 – 21 ( x–b 1 – 21 ( x–b a ) – |a| 2 a ) e ( ) e a a a

(7.45)

1

8a,b (x) = |a| 2 8 ( 𝜕8󸀠a,b (x) 𝜕x

1

= |a| 2

Therefore, from eqs. (7.38)–(7.45) , we can obtain 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨x(n) 𝜕 󵄨󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 1 +j 󵄨 = 󵄨󵄨 {[f (vR (n)) f 󸀠 (vR (n)) 8󸀠a,b (uj,R (n)) wj,R (n)yR (n – i) ̃ 󵄨󵄨󵄨󵄨 𝜕wij,R (n) 𝜕wij,I 󵄨󵄨x(n) + f (vI (n)) f 󸀠 (vI (n)) 8󸀠a,b (uj,R (n)) wj,I (n)yR (n – i) – f (vR (n)) f 󸀠 (vR (n)) 8󸀠a,b (uj,I (n)) wj,I (n)yI (n – i) + f (vI (n)) f 󸀠 (vI (n)) 8󸀠a,b (uj,I (n)) wj,R (n)yI (n – i)] + j [–f (vR (n)) f 󸀠 (vR (n)) 8󸀠a,b (uj,R (n)) wj,R (n)yI (n – i) – f (vI (n)) f 󸀠 (vI (n)) 8󸀠a,b (uj,I ) wj,I (n)yI (n – i) – f (vR (n)) f 󸀠 (vR (n)) 8󸀠a,b (uj,I (n)) wj,I (n)yR (n – i)

(7.46)

+ f (vI (n)) f 󸀠 (vI (n)) 8󸀠a,b (uj,R (n)) wj,R (n)yR (n – i)]} 1 = 󵄨󵄨 8󸀠 (u (n)) Re {[f (vR (n)) f 󸀠 (vR (n)) + jfv (I (n)) f 󸀠 (vI )] wj∗ (n)} ̃ 󵄨󵄨󵄨󵄨 a,b j,R 󵄨󵄨x(n) × [yR (n – i) – jyI (n – i)] 1 󸀠 󸀠 󸀠 ∗ + j 󵄨󵄨 󵄨󵄨 8a,b (uj,I (n)) Im {[f (vR (n)) f (vR (n)) + jfv (I (n)) f (vI )] wj (n)} ̃ 󵄨󵄨x(n)󵄨󵄨 × [yR (n – i) – jyI (n – i)] Plug eq. (7.47) into eq. (7.33), wij (n + 1) = wij (n) + ,2 kj (n)y∗ (n – i)

(7.47)

where ,2 is the hidden layer iterative step-size factor 󵄨 ̃ 󵄨󵄨2 󸀠 󸀠 󸀠 ∗ kj (n) = (󵄨󵄨󵄨x(n) 󵄨󵄨 – R2 ) 8a,b (uj,R (n)) Re {[f (vR (n)) f (vR (n)) + jfv (I (n)) f (vI )] wj (n)} 󵄨 ̃ 󵄨󵄨2 󸀠 󸀠 󸀠 ∗ + j (󵄨󵄨󵄨x(n) 󵄨󵄨 – R2 ) 8a,b (uj,I (n)) Im {[f (vR (n)) f (vR (n)) + jfv (I (n)) f (vI )] wj (n)} Similar to the real number, if the number of hidden layer increases, the iteration formula of the hidden layer above is still available.

The iterative formula of expansion factor and translation factor According to the weight iterative formula,

191

7.2 Blind equalization algorithm based on feed-forward wavelet neural network

𝜕J(n) 𝜕a(n) 𝜕J(n) b(n + 1) = b(n) – ,4 𝜕b(n) a(n + 1) = a(n) – ,3

(7.48) (7.49)

where ,3 and ,4 are the expansion factor and the translation factor, respectively. The iteration step-size factors are 󵄨 ̃ 󵄨󵄨 󵄨󵄨 𝜕J(n) 󵄨 ̃ 󵄨󵄨2 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) = 2 [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2] 󵄨󵄨󵄨x(n) 󵄨󵄨 𝜕a(n) 𝜕a(n) 󵄨 ̃ 󵄨󵄨 󵄨󵄨 𝜕J(n) 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 󵄨 ̃ 󵄨󵄨2 = 2 [󵄨󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 – R2] 󵄨󵄨󵄨x(n) 𝜕b(n) 𝜕b(n)

(7.50) (7.51)

a(n) = aR (n) + jaI (n)

(7.52)

b(n) = bR (n) + jbI (n) 󵄨󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨x(n)󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 𝜕 󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 = +j 󵄨 𝜕a(n) 𝜕aR (n) 𝜕aI (n) 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 𝜕 󵄨󵄨x(n) 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 = +j 󵄨 𝜕b(n) 𝜕bR (n) 𝜕bI (n)

(7.53) (7.54) (7.55)

2 2 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕x(n) 1 𝜕 (f (vR (n)) + f (vI (n))) = = 󵄨󵄨 = 󵄨 ̃ 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 𝜕aR (n) 𝜕aR (n) 𝜕aR (n) 𝜕aR (n) 2 󵄨󵄨x(n) 2 󵄨󵄨󵄨x(n) 󵄨󵄨 (7.56) (n) 𝜕v 𝜕v 1 󸀠 󸀠 R I (n) = 󵄨󵄨 + f (vI (n)) f (vI (n)) ] [f (vR (n)) f (vR (n)) ̃ 󵄨󵄨󵄨󵄨 𝜕aR (n) 𝜕aR (n) 󵄨󵄨x(n) 2 2 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕x(n) 1 𝜕 (f v (R (n)) + f (vI (n))) = = 󵄨󵄨 = 󵄨 󵄨 󵄨 ̃ 󵄨󵄨󵄨 𝜕aI (n) ̃ 󵄨󵄨󵄨 𝜕aI (n) 𝜕aI (n) 𝜕aI (n) 2 󵄨󵄨x(n) 2 󵄨󵄨󵄨x(n) (7.57) (n) 𝜕v 𝜕v 1 󸀠 󸀠 R I (n) = 󵄨󵄨 + f (vI (n)) f (vI (n)) ] [f (vR (n)) f (vR (n)) ̃ 󵄨󵄨󵄨󵄨 𝜕aI (n) 𝜕aI (n) 󵄨󵄨x(n)

𝜕8a,b (uj,R (n)) 𝜕8a,b (uj,I (n)) 𝜕vR (n) 𝜕 (wj,R (n)Ij,R (n) – wj,I (n)Ij,I (n)) = = wj,R (n) –wj,I (n) 𝜕aR (n) 𝜕aR (n) 𝜕aR (n) 𝜕aR (n) (7.58) Similarly, 𝜕8a,b (uj,I (n)) 𝜕8a,b (uj,R (n)) 𝜕vI (n) = wj,R (n) + wj,I (n) 𝜕aR (n) 𝜕aR (n) 𝜕aR (n)

(7.59)

𝜕8a,b (uj,R (n)) 𝜕8a,b (uj,I (n)) 𝜕vR (n) = –wj,R (n) – wj,I (n) 𝜕aI (n) 𝜕aI (n) 𝜕aI (n)

(7.60)

𝜕8a,b (uj,I (n)) 𝜕8a,b (uj,R (n)) 𝜕vI (n) = wj,R (n) – wj,I (n) 𝜕aI (n) 𝜕aI (n) 𝜕aI (n)

(7.61)

192

7 Blind equalization Algorithm Based on Wavelet Neural Network

󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕x(n) = = 󵄨󵄨 󵄨 ̃ 󵄨󵄨󵄨 𝜕bR (n) 𝜕bR (n) 𝜕bR (n) 2 󵄨󵄨x(n) 2 2 1 𝜕 (f (vR (n)) + f (vI (n))) = 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 𝜕bR (n) 2 󵄨󵄨x(n) 𝜕v (n) 𝜕v (n) 1 + f (vI (n)) f 󸀠 (vI (n)) I ] = 󵄨󵄨 [f (vR (n)) f 󸀠 (vR (n)) R ̃ 󵄨󵄨󵄨󵄨 𝜕bR (n) 𝜕bR (n) 󵄨󵄨x(n)

󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕x(n) = = 󵄨󵄨 󵄨 ̃ 󵄨󵄨󵄨 𝜕bi (n) 𝜕bI (n) 𝜕bI (n) 2 󵄨󵄨x(n) 2 2 1 𝜕 (f (vR (n)) + f (vI (n))) = 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 𝜕bi (n) 2 󵄨󵄨x(n) 𝜕v (n) 𝜕v (n) 1 = 󵄨󵄨 + f (vI (n)) f 󸀠 (vI (n)) I ] [f (vR (n)) f 󸀠 (vR (n)) R ̃ 󵄨󵄨󵄨󵄨 𝜕bi (n) 𝜕bi (n) 󵄨󵄨x(n)

(7.62)

(7.63)

𝜕8a,b (uj,R (n)) 𝜕8a,b (uj,I (n)) 𝜕vR (n) 𝜕 (wj,R (n)Ij,R (n) – wj,I (n)Ij,I (n)) = = wj,R (n) –wj,I (n) 𝜕bR (n) 𝜕bR (n) 𝜕bR (n) 𝜕bR (n) (7.64) Similarly, 𝜕8a,b (uj,I (n)) 𝜕8a,b (uj,R (n)) 𝜕vI (n) = wj,R (n) + wj,I (n) 𝜕bR (n) 𝜕bR (n) 𝜕bR (n)

(7.65)

𝜕8a,b (uj,I (n)) 𝜕8a,b (uj,R (n)) 𝜕vR (n) = wj,R (n) + wj,I (n) 𝜕bI (n) 𝜕bI (n) 𝜕bI (n)

(7.66)

𝜕8a,b (uj,I (n)) 𝜕8a,b (uj,R (n)) 𝜕vI (n) = wj,R (n) + wj,I (n) 𝜕bI (n) 𝜕bI (n) 𝜕bI (n) 2 1 x–b 2 1 1 𝜕8a,b (x) 1 1 x – b 2 – 21 ( x–b – ( ) a ) = – |a| 2 (x – b) 2 e 2 a + |a| 2 ( ) e 𝜕a(n) a a a2 2 1 x–b x – b –2( a ) 1 )e ( + z √ 2 |a| 2 1 1 – 1 ( x–b )2 1 𝜕8a,b (x) 1 x – b 2 – 21 ( x–b a ) = – |a| 2 e 2 a + |a| 2 ( ) e 𝜕b(n) a a a

(7.67)

(7.68) (7.69)

The derived functions above are plugged into formulas (7.48) and (7.49), the iterative formula of scale and translation factor can be obtained. According to the weights of iterative formula, FFWNN blind equalization can be obtained. 7.2.3.2 Computer simulation According to the derivation above of complex-valued FFWNN blind equalization algorithm, 16QAM and 32QAM signals are used in computer simulation. The typical telephone channel and common channel are used. The transfer functions are shown in eqs. (2.85) and (3.13), respectively. Gaussian white noise is added in simulation.

193

0.10 0.08 0.06 0.04 0.02 0

(1)

(1) FFNN (2) FFWNN

(2)

0

(a)

5,000

10,000

MSE

MSE

7.2 Blind equalization algorithm based on feed-forward wavelet neural network

15,000

Iterative number

0.10 0.08 0.06 0.04 0.02 0

(1)

0

(b)

5,000

(1) FFNN (2) FFWNN (2)

10,000

15,000

Iterative number

Figure 7.7: Convergence curves of 16QAM in the typical telephone channel (a) and common channel (b).

0.10

MSE

0.06

(1)

0.04

MSE

(1) FFNN (2) FFWNN

0.08 (2)

0.02 0 (a)

0

5,000

10,000

15,000

Iterative number

0.10 0.08 0.06 0.04 0.02 0 (b)

(1) FFNN (2) FFWNN (1)

0

5,000

(2)

10,000

15,000

Iterative number

Figure 7.8: Convergence curves of 32QAM in the typical telephone channel (a) and common channel (b).

Table 7.2: Bit error statistics of FFWNN and FFNN after the convergence. Modulation Algorithm and channel

Bit error Modulation Algorithm and channel

Bit error

16QAM

81

325

FFWNN Typical telephone channel Common channel FFNN Typical telephone channel Common channel

89 121

95

32QAM

FFWNN Typical telephone channel Common channel FFNN Typical telephone channel Common channel

316 501

495

The convergence comparisons curves for 16QAM and 32QAM in the typical telephone channel and the common channel are shown in Figures 7.7 and 7.8, respectively. Compared with FFNN, FFWNN has been improved significantly in both convergence rate and steady-state residual error. The error statistics of the two algorithms (15,000 iterations) are given in Table 7.2

194

7 Blind equalization Algorithm Based on Wavelet Neural Network

7.3 Blind equalization algorithm based on recurrent wavelet neural network 7.3.1 Principle of algorithm The basic principle of recurrent wavelet neural network blind equalization algorithm is that bilinear recurrent wavelet neural network (BLRWNN) is used to substitute the neural network in the neural network blind equalization algorithm. The principle figure is Figure 2.4. The block diagram is as shown in Figure 7.9.

7.3.2 Blind equalization algorithm based on BLRWNN in real number system 7.3.2.1 Algorithm derivation Bilinear recurrent neural network is adopted, as shown in Figure 7.10. T

The input of the network is Y(n) = [ y(n) y(n – 1) ⋅ ⋅ ⋅ y(n – k)] ; the output of out̃ put layer is v(n); the total network output is x(n); bilinear feedback unit weight is ai ; the feed-forward weight is cj ; the bilinear feedback weight is bij , where i = 1, 2, j = 0, 1, . . . , k. The relationship of input and output of BLRWNN can be expressed as 2

2

k

k

̃ – i) + ∑ 8a,b [∑ bij (n)y(n – j)]x(n ̃ – i) + 8a,b [∑ cj (n)y(n – j)] v(n) = ∑ ai (n)x(n i=1 i=1 j=0 ] ] [ [j=0 (7.70) ̃ x(n) = f (v(n))

(7.71)

where 8a,b (⋅) is the wavelet transform of the input signal. Morlet is adopted to wavelet transform, shown in eq. (7.5); f (⋅) is the transfer function, shown in eq. (3.42). The iterative formula of network weights is obtained by the steepest descent method: n(n) x(n)

Channel h(n)

+

y(n) Feedback wavelet ~ x (n) neural network

Judgment

x̂ (n)

Blind equalization Figure 7.9: The block diagram of feedback wavelet neural network blind equalization algorithm.

7.3 Blind equalization algorithm based on recurrent wavelet neural network

y(n)

195

cj

y(n – 1)

ψ

Feedforward unit v(n)

ψ y(n – k)

~ x(n)

f(·)

Σ

ψ ψ ψ bij

ψ

Linear feedback unit

Feedback unit

ai

z–1 z–1 Figure 7.10: The block diagram of BLRWNN.

W(n + 1) = W(n) – ,

𝜕J(n) 𝜕W(n)

(7.72)

where , is the iteration step size factor 󵄨 ̃ 󵄨󵄨 󵄨󵄨 𝜕J(n) 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 󵄨 ̃ 󵄨󵄨2 = 2 [󵄨󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 – R2] 󵄨󵄨󵄨x(n) 𝜕W(n) 𝜕W(n)

(7.73)

There are feed-forward unit, feedback unit, and linear feedback unit in BLRWNN. The following are weight iterative formula derivations. The feed-forward weight iterative formula The connection weight of feed-forward unit and output unit is cj (n) 󵄨 ̃ 󵄨󵄨 k 𝜕 󵄨󵄨󵄨x(n) 󵄨󵄨 = f 󸀠 (v(n)) 8󸀠a,b [∑ cj (n)y(n – j)] y(n – j) 𝜕cj ] [j=0

(7.74)

Plug eq. (7.74) into eqs. (7.73) and (7.72), k

󵄨 ̃ 󵄨󵄨2 󸀠 󸀠 ̃ cj (n + 1) = cj (n) – 2,c [󵄨󵄨󵄨x(n) – j) 󵄨󵄨 – R2] f (v(n)) 8a,b (∑ cj (n)y(n – j)) x(n)y(n j=0

where ,c is feed-forward iterative step factor.

(7.75)

196

7 Blind equalization Algorithm Based on Wavelet Neural Network

The recurrent unit weight iterative formula The connecting weight of recurrent unit and output unit is ai (n): 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 󵄨󵄨 ̃ – i) = f 󸀠 (v(n)) x(n 𝜕ai (n)

(7.76)

Plug eq. (7.76) into eqs. (7.73) and (7.72), 󵄨 ̃ 󵄨󵄨2 󸀠 ̃ x(n ̃ – i) ai (n + 1) = ai (n) – 2,a [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2] f (v(n)) x(n)

(7.77)

where ,a is the iterative step-size factor for the feedback unit.

The weight iterative formula of linear feedback unit The connection weight of linear feedback and output unit is bij (n): 󵄨 ̃ 󵄨󵄨2 k 𝜕 󵄨󵄨󵄨x(n) 󵄨󵄨 ̃ – i)y(n – j) = f 󸀠 (v(n)) 8󸀠a,b [∑ bij (n)y(n – j)] x(n 𝜕bij (n) ] [j=0

(7.78)

Plug eq. (7.78) into eqs. (7.73) and (7.72), k

󵄨 ̃ 󵄨󵄨2 󸀠 󸀠 [ ] ̃ – i)y(n – j) ̃ bij (n + 1) = bij (n) – 2,b [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2] f (v(n)) x(n)8 a,b ∑ eij (n)y(n – j) x(n j=0 ] [ (7.79) where ,b is linear feedback iterative step-size factor.

The iterative formula of expansion and translation factor According to the weight iterative formula, 𝜕J(n) 𝜕a(n) 𝜕J(n) b(n + 1) = b(n) – ,2 𝜕b(n) 󵄨 ̃ 󵄨󵄨 󵄨󵄨 𝜕J(n) 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 󵄨 ̃ 󵄨󵄨2 = 2 [󵄨󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 – R2] 󵄨󵄨󵄨x(n) 𝜕a(n) 𝜕a(n) 󵄨 ̃ 󵄨󵄨 󵄨󵄨 𝜕J(n) 󵄨 ̃ 󵄨󵄨2 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) = 2 [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2] 󵄨󵄨󵄨x(n) 󵄨󵄨 𝜕b(n) 𝜕b(n) a(n + 1) = a(n) – ,1

(7.80) (7.81) (7.82) (7.83)

where ,1 and ,2 are the iteration steps of the expansion and translation factor, respectively:

7.3 Blind equalization algorithm based on recurrent wavelet neural network

k

󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 󵄨󵄨 ̃ – i) = f 󸀠 (v(n)) x(n 𝜕a(n)

k

𝜕8a,b [ ∑ bij (n)y(n – j)] j=0

𝜕8a,b [ ∑ cj (n)y(n – j)] j=0

+ f 󸀠 (v(n))

𝜕a

197

𝜕a (7.84)

k

k

𝜕8a,b [ ∑ bij (n)y(n – j)] 𝜕8a,b [ ∑ cj (n)y(n – j)] 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) j=0 j=0 󵄨󵄨 󸀠 󸀠 ̃ – i) = f (v(n)) x(n + f (v(n)) 𝜕a(n) 𝜕a 𝜕a (7.85) 𝜕8󸀠a,b (x) 𝜕a(n)

𝜕8󸀠a,b (x) 𝜕b(n)

2 2 1 1 – 21 ( x–b 1 x – b 2 – 21 ( x–b a ) + |a| 2 ( a ) ) e e 2 a a a2 2 1 x–b 1 x – b –2( a ) + ( )e z 2√|a| 1

= – |a| 2 (x – b)

1

= – |a| 2

(7.86)

2 2 1 1 – 21 ( x–b 1 x – b 2 – 21 ( x–b a ) + |a| 2 ( a ) e ) e a a a

(7.87)

The iterative formula of scale and translation factors can be obtained by plugging eqs. (7.82)—(7.87) into eqs. (7.80) and (7.81). According to the above iterative weight formula, BLRWNN can be used in blind equalization. 7.3.2.2 Computer simulation 4PAM and 8PAM are the input signals. The Gauss white noise is added in the system. The typical telephone channel and common channel are used. The transmission functions are shown in formulas (2.85) and (3.13). The convergence comparison curves in typical telephone and common channel between BLRWNN and BLRNN blind equalization algorithm are shown in Figures 7.11 and 7.12, respectively. In the typical telephone channel, the iterative steps are ,a = ,b = ,c = 0.015, ,1 = ,2 = 0.00005 for 4PAM, while the iterative step sizes are ,a = ,b = ,c = 0.0009, ,1 = ,2 = 5 × 10–7 for 8PAM. In the common channel, the iterative 0.10 (1)

0.06

(1) BLRNN (2) BLRWNN MSE

MSE

0.08 (2)

0.04 0.02 0 (a)

0

5,000

10,000

Iterative number

15,000

0.10 0.08 0.06 0.04 0.02 0 (b)

(1)

0

5,000

(2)

(1) BLRNN (2) BLRWNN

10,000

15,000

Iterative number

Figure 7.11: Convergence curves of 8PAM in the typical telephone channel (a) and common channel (b).

198

7 Blind equalization Algorithm Based on Wavelet Neural Network

0.10 (1) BLRNN (2) BLRWNN

(1)

0.06

MSE

MSE

0.08

(2)

0.04 0.02 0 (a)

0

5,000

10,000

15,000

Iterative number

0.10 0.08 0.06 0.04 0.02 0

(1) BLRNN (2) BLRWNN

(1) (2)

0

(b)

5,000

10,000

15,000

Iterative number

Figure 7.12: Convergence curves of 8PAM in the typical telephone channel (a) and common channel (b). Table 7.3: Bit error statistics of BLRWNN and BLRNN after the convergence. Modulation Algorithm and channel

Bit error Modulation Algorithm and channel

Bit error

4PAM

66

399

BLRWNN Typical telephone channel Common channel BLRNN Typical telephone channel Common channel

8PAM

50 82

65

BLRWNN Typical telephone channel Common channel BLRNN Typical telephone channel Common channel

350 678

568

steps are ,a = ,b = ,c = 0.015, ,1 = ,2 = 0.00005 for 4PAM and the iterative step sizes are ,a = ,b = ,c = 0.0009, ,1 = ,2 = 5 × 10–7 for 8PAM. From convergence curve, it can be seen that convergence speed and steadystate residual error of the BLRWNN algorithm are not improved in typical telephone channel. However, convergence speed and steady-state residual error of the BLRWNN algorithm are improved for the common channel. The error statistics of the two algorithms (15,000 iterations) are given in Table 7.3.

7.3.3 Blind equalization algorithm based on BLRWNN in complex system 7.3.3.1 Algorithm derivation Similar to the derivation of FFWNN blind equalization algorithm in complex number system, the cost function and transfer function are shown in formulas (3.6) and (3.42), respectively. First, the network signal is written in complex form: cj (n) = cj,R (n) + jcj,I (n)

(7.88)

ai (n) = ai,R (n) + jai,I (n)

(7.89)

199

7.3 Blind equalization algorithm based on recurrent wavelet neural network

bij (n) = bij,R (n) + jbij,I (n)

(7.90)

y(n) = yR (n) + jyI (n)

(7.91)

Suppose that k

p(n) = 8a,b [∑ bij (n)y(n – j)] ] [j=0

(7.92)

k

q(n) = 8a,b [∑ cj (n)y(n – j)] ] [j=0

(7.93)

k

p󸀠 (n) = 8󸀠a,b [∑ bij (n)y(n – j)] ] [j=0

(7.94)

k

q󸀠 (n) = 8󸀠a,b [∑ cj (n)y(n – j)] ] [j=0

(7.95)

The following equations can be obtained: 2

2

k

k

̃ – i) + ∑ 8a,b [∑ bij (n)y(n – j)]x(n ̃ – i) + 8a,b [∑ cj (n)y(n – j)] v(n) = ∑ ai (n)x(n i=1 i=1 ] ] [j=0 [j=0 2

= ∑ {[aiR (n)x̃R (n – i) – aiI (n)x̃I (n – i)] + j [aiR (n)x̃I (n – i) + aiI (n)x̃R (n – i)]} (7.96) i=1 2

+ ∑ {[PR (n)x̃R (n – i) – PI (n)x̃I (n – i)] – j [PR (n)x̃I (n – i) + PI (n)x̃R (n – i)]} i–1

+ qR (n) + qI (n) ̃ x(n) = f (vR (n)) + jf (vI (n))

(7.97)

According to the formula above, the weight iteration equation of the blind equalization algorithm based on BLRWNN in the complex system is as follows: W(n + 1) = W(n) – ,

󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 𝜕J(n) 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 󵄨 ̃ 󵄨󵄨2 = W(n) – 2, [󵄨󵄨󵄨x(n) +j 󵄨 ) (7.98) 󵄨󵄨 ( 󵄨󵄨 – R2] 󵄨󵄨󵄨x(n) 𝜕W(n) 𝜕wR (n) 𝜕wI (n)

The weight iterative formula of feed-forward unit The connection weights of the feed forward unit and the output are cj (n + 1) = cj,R (n) + jcj,I (n).

200

7 Blind equalization Algorithm Based on Wavelet Neural Network

2 2 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕x(n) 1 𝜕 (f (vR (n)) + f (vR (n))) = = 󵄨󵄨 = 󵄨 ̃ 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 𝜕cj,R (n) 𝜕cj,R (n) 𝜕cj,R (n) 𝜕cj,R (n) 2 󵄨󵄨x(n) 2 󵄨󵄨󵄨x(n) 󵄨󵄨 𝜕v (n) 𝜕vI (n) 1 + f (vI (n)) f 󸀠 (vI (n)) ] = 󵄨󵄨 [f (vR (n)) f 󸀠 (vR (n)) R ̃ 󵄨󵄨󵄨󵄨 𝜕cj,R (n) 𝜕cj,R (n) 󵄨󵄨x(n) 1 = 󵄨󵄨 [f (vR (n)) f 󸀠 (vR (n)) q󸀠R (n)yR (n – j) + f (vI (n)) f 󸀠 (vI (n)) q󸀠I (n)yI (n – j)] ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n) (7.99) 2 2 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕x(n) 1 𝜕 [f (vR (n)) + f (vI (n))] = = 󵄨󵄨 = 󵄨 󵄨 󵄨 ̃ 󵄨󵄨󵄨 𝜕cj,I (n) ̃ 󵄨󵄨󵄨 𝜕cj,I (n) 𝜕cj,I (n) 𝜕cj,I (n) 2 󵄨󵄨x(n) 2 󵄨󵄨󵄨x(n) 𝜕vR (n) 𝜕vI (n) 1 󸀠 󸀠 = 󵄨󵄨 󵄨󵄨 [f (vR (n)) f (vR (n)) 𝜕c (n) + f (vI (n)) f (vI (n)) 𝜕c (n)] ̃ j,I j,I 󵄨󵄨x(n)󵄨󵄨 1 󸀠 󸀠 󸀠 = 󵄨󵄨 [–f (vR (n)) f (vR (n)) qR (n)y(n – j) + f (vI (n)) f (vI (n)) q󸀠I (n)yR (n – j)] ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n) (7.100)

The weights iterative formula of the feed-forward units is 󵄨 ̃ 󵄨󵄨2 ∗ ∗ cj (n + 1) = cj (n) – 2,c [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2] Kq (n)y (n – j)

(7.101)

where ,c is the iteration step size factor of the feed-forward unit: K = f (vR (n)) f 󸀠 (vR (n)) + jf (vI (n)) f 󸀠 (vI (n))

(7.102)

The weight iteration formula of recurrent unit The connection weight between recurrent and output unit is ai (n + 1) = ai,R (n) + jdi,I (n). 2 2 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕x(n) 1 𝜕 [f (vR ) + f (vI (n))] = = 󵄨󵄨 = 󵄨 ̃ 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 𝜕ai,R (n) 𝜕ai,R (n) 𝜕ai,R (n) 𝜕ai,R (n) 2 󵄨󵄨x(n) 2 󵄨󵄨󵄨x(n) 󵄨󵄨 𝜕v (n) 𝜕vI (n) 1 + f (vI (n)) f 󸀠 (vI (n)) ] (7.103) = 󵄨󵄨 [f (vR (n)) f 󸀠 (vR (n)) R ̃ 󵄨󵄨󵄨󵄨 𝜕ai,R (n) 𝜕ai,R (n) 󵄨󵄨x(n) 1 󸀠 󸀠 = 󵄨󵄨 󵄨󵄨 [f (vR (n)) f (vR (n)) x̃R (n – i) + f (vI (n)) f (vI (n)) x̃I (n – i)] ̃ 󵄨󵄨x(n)󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 󵄨󵄨 1 = 󵄨󵄨 [–f (vR (n)) f 󸀠 (vR (n)) x̃I (n – i) + f (vI (n)) f 󸀠 (vI (n)) x̃R (n – i)] (7.104) ̃ 𝜕aj,I (n) 󵄨󵄨x(n)󵄨󵄨󵄨󵄨 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 𝜕 󵄨󵄨x(n) 𝜕 󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 󵄨󵄨 1 = 󵄨 +j 󵄨 = f (vR (n)) f 󸀠 (vR (n)) [x̃R (n – i) – jx̃I (n – i)] ̃ 󵄨󵄨󵄨󵄨 𝜕aj (n) 𝜕aj,R (n) 𝜕aj,I (n) 󵄨󵄨󵄨󵄨x(n) 1 󸀠 (7.105) + 󵄨󵄨 󵄨󵄨 f (vI (n)) f (vI (n)) [x̃I (n – i) + jx̃R (n – i)] ̃ x(n) 󵄨󵄨 󵄨󵄨 1 = 󵄨󵄨 [f (vR (n)) f 󸀠 (vR (n)) + jf (vI (n)) f 󸀠 (vI (n))] x̃ ∗ (n – i) ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n)

7.3 Blind equalization algorithm based on recurrent wavelet neural network

201

The weight iteration formula of recurrent unit is 󵄨 ̃ 󵄨󵄨2 ∗ ai (n + 1) = ai (n) – 2,a [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2] Kx (n – i)

(7.106)

where ,a is the iteration step of recurrent unit. The weight iteration formula of bilinear recurrent unit The connection weights between bilinear recurrent unit and output is bij (n) = bij,R (n)+ jbij,I (n): 2 2 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕x(n) 1 𝜕 [f (vR (n)) + f (vI (n))] = = 󵄨󵄨 = 󵄨 󵄨 󵄨 ̃ 󵄨󵄨󵄨 𝜕bij,R (n) ̃ 󵄨󵄨󵄨 𝜕bij,R (n) 𝜕bij,R (n) 𝜕bij,R (n) 2 󵄨󵄨x(n) 2 󵄨󵄨󵄨x(n)

𝜕vR (n) 𝜕vI (n) 1 = 󵄨󵄨 + f (vI (n)) f 󸀠 (vI (n)) ] [f (vR (n)) f 󸀠 (vR (n)) ̃ 󵄨󵄨󵄨󵄨 𝜕bij,R (n) 𝜕bij,R (n) 󵄨󵄨x(n) 1 󸀠 󸀠 󸀠 = 󵄨󵄨 󵄨󵄨 f (vR (n)) f (vR (n)) [x̃R (n – i)yR (n – j) pR (n) – x̃R (n – i)yI (n – j) PI (n)] ̃ 󵄨󵄨x(n)󵄨󵄨 1 + 󵄨󵄨 f (vI (n)) f 󸀠 (vI (n)) [x̃I (n – i)yR (n – j) PR󸀠 (n) + x̃I (n – i)yI (n – j) p󸀠I (n)] ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n) (7.107) 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕x(n) = = 󵄨󵄨 󵄨 ̃ 󵄨󵄨󵄨 𝜕bij,I (n) 𝜕bij,I (n) 𝜕bij,I (n) 2 󵄨󵄨x(n) –1 = 󵄨󵄨 f (vR (n)) f 󸀠 (vR (n)) [x̃R (n – i)yI (n – j) p󸀠I (n) + x̃I (n – i)yR (n – j) p󸀠R (n)] ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n) 1 + 󵄨󵄨 f (vI (n)) f 󸀠 (vI (n)) [x̃R (n – i)yR (n – j) p󸀠R (n) – x̃I (n – i)yI (n – j) p󸀠I (n)] ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n) (7.108) The weight iteration formula of linear recurrent unit is 󵄨 ̃ 󵄨󵄨2 ∗ ∗ ∗ bij (n + 1) = bij (n) – 2,b [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2] Kp (n)y (n – j) x̃ (n – i)

(7.109)

where ,b is the feed-forward unit iteration step. The iterative formula of scale and translation factor

a(n) = aR (n) + jaI (n)

(7.110)

b(n) = bR (n) + jbI (n) 󵄨󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨x(n)󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 𝜕 󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 = +j 󵄨 𝜕a(n) 𝜕aR (n) 𝜕aI (n)

(7.111) (7.112)

202

7 Blind equalization Algorithm Based on Wavelet Neural Network

󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 𝜕 󵄨󵄨x(n) 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 = +j 󵄨 𝜕b(n) 𝜕bR (n) 𝜕bI (n)

(7.113)

2 2 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕x(n) 1 𝜕 [f (vR (n)) + f (vR (n))] = = 󵄨󵄨 = 󵄨 󵄨 󵄨 ̃ 󵄨󵄨󵄨 𝜕aR (n) ̃ 󵄨󵄨󵄨 𝜕aR (n) 𝜕aR (n) 𝜕aR (n) 2 󵄨󵄨x(n) 2 󵄨󵄨󵄨x(n) (n) (n) 𝜕v 𝜕v 1 = 󵄨󵄨 + f (vI (n)) f 󸀠 (vI (n)) I ] [f (vR (n)) f 󸀠 (vR (n)) R ̃ 󵄨󵄨󵄨󵄨 𝜕aR (n) 𝜕aR (n) 󵄨󵄨x(n) 𝜕q (n) 𝜕p (n) 1 ̃ – i) R ) + x(n f (vR (n)) f 󸀠 (vR (n)) ( R = 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 𝜕aR (n) 𝜕aR (n) 󵄨󵄨x(n) 𝜕p (n) 𝜕q (n) 1 ̃ – i) I + x(n ) f (vI (n)) f 󸀠 (vI (n)) ( I + 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 𝜕aR (n) 𝜕aR (n) 󵄨󵄨x(n) (7.114) 2 2 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕x(n) 1 𝜕 [f (vR (n)) + f (vR (n))] = = 󵄨󵄨 = 󵄨 ̃ 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 𝜕aI (n) 𝜕aI (n) 𝜕aI (n) 𝜕aI (n) 2 󵄨󵄨x(n) 2 󵄨󵄨󵄨x(n) 󵄨󵄨 (n) (n) 𝜕v 𝜕v 1 = 󵄨󵄨 + f (vI (n)) f 󸀠 (vI (n)) I ] [f (vR (n)) f 󸀠 (vR (n)) R 󵄨 󵄨 ̃ 󵄨󵄨 𝜕aI (n) 𝜕aI (n) 󵄨󵄨x(n) 𝜕p (n) (n) 𝜕q –1 ̃ – i) I ) + x(n f (vR (n)) f 󸀠 (vR (n)) ( I = 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 𝜕aI (n) 𝜕aI (n) 󵄨󵄨x(n) 𝜕q (n) 𝜕p (n) 1 ̃ – i) R ) + 󵄨󵄨 + x(n f (vI (n)) f 󸀠 (vI (n)) ( R ̃ 󵄨󵄨󵄨󵄨 𝜕aI (n) 𝜕aI (n) 󵄨󵄨x(n) (7.115) 2 2 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕x(n) 1 𝜕 [f (vR (n)) + f (vR (n))] = = 󵄨󵄨 = 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 𝜕bR (n) ̃ 󵄨󵄨󵄨󵄨 𝜕bR (n) 𝜕bR (n) 𝜕bR (n) 2 󵄨󵄨x(n) 2 󵄨󵄨x(n) 𝜕v (n) 𝜕v (n) 1 + f (vI (n)) f 󸀠 (vI (n)) I ] = 󵄨󵄨 [f (vR (n)) f 󸀠 (vR (n)) R ̃ 󵄨󵄨󵄨󵄨 𝜕bR (n) 𝜕bR (n) 󵄨󵄨x(n) 𝜕q (n) 𝜕p (n) 1 ̃ – i) R ) = 󵄨󵄨 + x(n f (vR (n)) f 󸀠 (vR (n)) ( R ̃ 󵄨󵄨󵄨󵄨 𝜕bR (n) 𝜕bR (n) 󵄨󵄨x(n)

𝜕p (n) 𝜕qI (n) 1 󸀠 ̃ – i) I ) + 󵄨󵄨 󵄨󵄨 f (vI (n)) f (vI (n)) ( 𝜕b (n) + x(n ̃ 𝜕bR (n) 󵄨󵄨x(n)󵄨󵄨 R

(7.116)

2 2 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕x(n) 1 𝜕 [f (vR (n)) + f (vR (n))] = = 󵄨󵄨 = 󵄨 󵄨 󵄨 ̃ 󵄨󵄨󵄨 𝜕bI (n) ̃ 󵄨󵄨󵄨 𝜕bI (n) 𝜕bI (n) 𝜕bI (n) 2 󵄨󵄨x(n) 2 󵄨󵄨󵄨x(n) 𝜕v (n) 𝜕v (n) 1 = 󵄨󵄨 + f (vI (n)) f 󸀠 (vI ) I f (vR (n)) f 󸀠 (vR (n)) R ̃ 󵄨󵄨󵄨󵄨 𝜕bI (n) 𝜕bI (n) 󵄨󵄨x(n) (7.117) 𝜕pI (n) 𝜕qI (n) –1 󸀠 ̃ + x(n – i) ) = 󵄨󵄨 (n)) f (v (n)) ( f (v R R ̃ 󵄨󵄨󵄨󵄨 𝜕bI (n) 𝜕bI (n) 󵄨󵄨x(n) 𝜕p (n) 𝜕q (n) 1 ̃ – i) R ) + x(n + 󵄨󵄨 f (vI (n)) f 󸀠 (vI (n)) ( R ̃ 󵄨󵄨󵄨󵄨 𝜕bI (n) 𝜕bI (n) 󵄨󵄨x(n)

203

7.3 Blind equalization algorithm based on recurrent wavelet neural network

𝜕8󸀠a,b (x) 𝜕a(n)

𝜕8󸀠a,b (x) 𝜕b(n)

2 2 1 1 – 21 ( x–b 1 x – b 2 – 21 ( x–b a ) + |a| 2 ( a ) e e ) 2 a a a2 2 1 x – b – 21 ( x–b a ) )e + ( z √ 2 |a| 1

= – |a| 2 (x – b)

1

= – |a| 2

2 2 1 1 – 21 ( x–b 1 x – b 2 – 21 ( x–b a ) + |a| 2 ( a ) e ) e a a a

(7.118) (7.119)

Put eqs. (7.112)–(7.119) into eq. (7.98), the iterative formula of scale and translation factor can be obtained. According to the weight iteration formula above, the blind equalization based on BLRWNN can be obtained. 7.3.3.2 Computer simulation 16QAM and 32QAM signals are used. The typical telephone channel and common channel are used. The transform functions are shown in eqs. (2.85) and (3.13), respectively. In the simulation, the Gauss white noise is used. The convergence comparison curves of BLRWNN and BLRNN for 16QAM and 32QAM are shown in Figures 7.13 and 7.14 in different channels. In typical telephone 0.10 (1) BLRNN (2) BLRWNN

(1)

0.06

MSE

MSE

0.08

(2)

0.04 0.02 0

0

5,000

10,000

15,000

Iterative number

(a)

0.10 0.08 0.06 0.04 0.02 0

(1) BLRNN (2) BLRWNN

(1) (2)

0

5,000

10,000

15,000

Iterative number

(b)

Figure 7.13: Convergence curves of 32QAM in the typical telephone channel (a) and common channel (b).

0.10 (1) BLRNN (2) BLRWNN

(1)

0.06

MSE

MSE

0.08

(2)

0.04 0.02 0 (a)

0

5,000

10,000

Iterative number

15,000

0.10 0.08 0.06 0.04 0.02 0 (b)

(1) BLRNN (2) BLRWNN

(1) (2)

0

5,000

10,000

15,000

Iterative number

Figure 7.14: Convergence curves of 32QAM in the typical telephone channel (a) and common channel (b).

204

7 Blind equalization Algorithm Based on Wavelet Neural Network

Table 7.4: Four bit error statistics of BLRWNN and BLRNN after the convergence. Modulation Algorithm and channel

Bit error Modulation Algorithm and channel

Bit error

16QAM

85

280

BLRWNN Typical telephone channel Common channel BLRNN Typical telephone channel Common channel

65 166

128

32QAM

BLRWNN Typical telephone channel Common channel BLRNN Typical telephone channel Common channel

265 556

530

channel, the iteration steps are ,a = ,b = ,c = 0.0015, ,1 = ,2 = 5 × 10–7 for 16QAM and ,a = ,b = ,c = 0.00015, ,1 = ,2 = 4 × 10–7 for 32QAM. In common channel, the iteration steps are ,a = ,b = ,c = 0.002, ,1 = ,2 = 4 × 10–7 for 16QAM and ,a = ,b = ,c = 0.00008, ,1 = ,2 = 2 × 10–7 for 32QAM. Compared with BLRNN, BLRWNN convergence speed increases while steady residual error is reduced. The error statistics of the two algorithms (15,000 iterations) are given in Table 7.4.

7.4 Summary BLRWNN and BLRNN blind equalization algorithms are studied in this chapter. The algorithm principle and iterative formula are given. Computer simulation is carried out. Three-layer feed-forward neural network structure is adopted by FFWNN. Morlet mother wavelet is used as hidden layer wavelet transform. The real and complex system algorithm iterative formulas are derived. Computer simulation shows that the new algorithm is significantly improved both in convergence speed and in error rate. The bilinear recurrent neural network structure is adopted in recurrent wavelet neural network . Morlet mother wavelet is used as hidden layer wavelet transform. Iterative formulas are derived in real and complex systems. Computer simulation shows that the new algorithm is significantly improved both in convergence speed and in error rate.

8 Application of Neural Network Blind Equalization Algorithm in Medical Image Processing Abstract: The neural network blind equalization algorithm is used in the medical Computed Tomography (CT) image restoration in this chapter. Imaging mechanism of medical CT, degradation process, basic principles of image blind equalization algorithm, and quantitative measures are given. The medical CT images neural network blind equalization algorithm based on zigzag encoding and double zigzag encoding are proposed. The algorithm iterative formulas are deduced. Convergence performance is analyzed and computer simulations are carried out.

8.1 Concept of image blind equalization 8.1.1 Imaging mechanism and degradation process of medical CT image CT (Computed Tomography) image is a topographic image of medical imaging mode by computer reconstruction. Since different human tissues have different reduction to X-ray attenuation, the detection of space distribution characteristics of human tissue to X-ray attenuation can be used as the principal. The human section twodimensional anatomical structure information and three-dimensional anatomical structure information can be provided. In medical CT imaging process, the image degradation and details visibility decline occur because of focal spot size of the X-ray tube, detector and the sampling aperture, voxel size and the filter and smoothing in image reconstruction. The main factors are discussed below. 8.1.1.1 Ray width The ray width and sampling aperture lead CT degradation and smaller visibility. In the process of measurement, all anatomic details in ray width are blurred. If the detector aperture is small and the ray is narrow, image details are clearer. Recently CT makes use of variable aperture adjustment to generate more image details. In order to improve the quality of the image, increasing radiation doses can be used to improve the quality of the image. However, the increase of the radiation doses is harmful to the human body. So improving image quality by algorithm can not only reduce doses but also obtain high-quality CT images. It is an important research direction. DOI 10.1515/9783110450293-008

206

8 Application of Neural Network Blind Equalization Algorithm in Medical Image

8.1.1.2 Ray interval The distance or interval between adjacent rays is another factor for image quality. If rays interval is significantly greater than that of organization size, the anatomical details which are scanned will not be shown. To obtain some anatomical details, the scanning rays need to be concentrated. 8.1.1.3 Image reconstruction Image reconstruction is generally realized through mathematical algorithms. At present, many of the CT system include several filters. Their function is to reduce the artifacts and additive noise, smooth the image and get clear edge. 8.1.1.4 Motion blur In the process of CT imaging, image motion blur occurs when the tested part shifted because of patients’ movement. This phenomenon is called motion artifacts, such as respiratory movement, swallowing movements, heart and gastrointestinal peristalsis, and fidgety patients’ mobility. 8.1.1.5 Defocus blur A large part of X-ray photons that penetrate objects are the scattered photon, which generally deviate from the original incident path. Nowadays, the collimating device of CT scanning system can effectively prevent scattering. However, there is still a portion of scattered photons which can reach the detector. It results in a low-frequency offset and leads to image degradation. Generally speaking, the degraded image is produced by the original degradation image added with noise. The degradation process is known as the point spread function. The CT image degradation process is the result of interaction of many factors. hij is the whole degradation model. The degradation process is shown in Figure 8.1 [15]. Here fij stands for f (i, j) which represents the original image; nij is the noise; gij is the degraded image; hij describe the whole degradation process of collective effect. The mathematical expressions of CT image degradation can be written as gij = hij ∗ fij + nij nij fij

hij

+

gij Figure 8.1: Model of image degradation.

(8.1)

8.1 Concept of image blind equalization

207

8.1.2 The basic principle of medical CT images blind equalization Image blind equalization is to estimate the original image from degradation image when all or part of the unknown information is given. Blind equalization algorithm is extended to solve medical image blind restoration. Image blind equalization can be divided into two-dimensional image and one-dimensional image blind equalization. 8.1.2.1 Two-dimensional image blind equalization The transmission of image can be equivalent to a single-input-single-output linear time-invariant system, as shown in Figure 8.2 [183]. Here fij̃ is the image signal which is restored by image blind equalizer. fijC is the output of the reference system; wij is the two-dimensional impulse response of blind equalizer; cij is the two-dimensional impulse response of reference system. The image size is M × M, then M M

gij = fij ∗hij + nij = ∑ ∑ fkl hi–k,j–l + nij

(8.2)

k=1 l=1 M M

fij̃ = gij ∗wij = ∑ ∑ gi–k,j–l wk,l

(8.3)

k=1 l=1 M M

fijC = gij ∗cij = ∑ ∑ gi–k,j–l ck,l

(8.4)

k=1 l=1

In the case of no noise, fij̃ and fijC can be expressed as fij̃ = fij ∗sij

(8.5)

fijC

(8.6)

=

fij ∗sCij

where sij = hij ∗wij , sCij = hij ∗cij . The two-dimensional blind equalization (blind image restoration) is only to adjust wij when the partial prior information of fij is given. fij̃ is asked to approach to fij as much as possible. So the following function should be satisfied sij = !$ (i – m, j – n)

(8.7)

where ! is nonzero constant. nij fij

hij

gij +

wij

cij

~ fij f cij Figure 8.2: Structure of medical image equalization.

208

8 Application of Neural Network Blind Equalization Algorithm in Medical Image

8.1.2.2 One-dimensional image blind equalization First, the degradation image is converted into one-dimensional signal by zigzag coding or ranks transform. Second, eliminating the effect of point spread function is carried out by the blind equalization algorithm. Finally, the signal is converted into a restored image.

8.1.3 Quantitative measurements Evaluation criterion of medical image blind equalization algorithm contains some parameters, such as MSE (mean square error), PSNR (peak signal-to-noise ratio), and ISNR (improved signal-to-noise ratio), and so on. 8.1.3.1 MSE MSE is the most basic evaluation criterion for image blind equalization. Suppose the size of the image is M ×M, the source is fij , and the estimated goal of blind equalization is fiĵ . MSE is defined as MSE =

2 1 M M ̂ ∑ ∑ [f – f ] M 2 i=1 j=1 ij ij

(8.8)

MSE is a global quantitative measure of the equalization estimation and goal approximation. 8.1.3.2 PSNR For the binary image, the gray value of the image is only 0 or 1, and the peak is 1. PSNR is defined as PSNR = 10 lg

1 = 10 lg MSE

1 1 M2

M ̂ ∑M =1 ∑j=1 [fij

2

– fij ]

(8.9)

For the gray image, the gray value has 256 gray levels, the maximum is 255. PSNR is defined as PSNR = 10 lg

1 = 10 lg MSE

2552 1 M2

2 M ̂ ∑M i=1 ∑j=1 [fij – fij ]

(8.10)

In the course of actual mage equalization, 255 or the maximum gray value is normalized as 1. Then PSNR is defined as PSNR = 10 lg

1 = 10 lg MSE

1 1 M2

M ̂ ∑M i=1 ∑j=1 [fij

2

– fij ]

(8.11)

8.2 Medical CT image neural network blind equalization algorithm based on Zigzag

209

8.1.3.3 ISNR MSE and PSNR only characterize the approximation degree of image equalization. The improvement degree of the restored image is described by ISNR criteria. ISNR is defined as 2

ISNR = 10 lg

M ∑M i=1 ∑j=1 [fij – gij ] M ̂ ∑M i=1 ∑j=1 [fij – fij ]

2

= PSNRf ̂ – PSNRg

(8.12)

where PSNRf ̂ is expressed by eq. (8.11), PSNRg is denoted as PSNRg = 10 lg

1 1 ∑M ∑M [f M 2 i=1 j=1 ij

2

– gij ]

(8.13)

As eq. (8.13) shows, ISNR is the peak SNR difference between restored image and degradation image. ISNR>0 suggests that the restored image is more close to the target than the degraded image. The larger the ISNR is, the larger the improvement is; ISNRnq,1 = ∑ 󵄩󵄩󵄩 ∑ BW(n)󵄩󵄩󵄩 󵄩󵄩 󵄩n=1 k=1 󵄩 󵄩 󵄩 K

(8.25)

N

󵄩 󵄩q >nq,1 = ∑ ∑ 󵄩󵄩󵄩BW(n)󵄩󵄩󵄩

(8.26)

k=1 n=1

Suppose assumptions (A1) and (A2) are established, and then there must be a constant 𝛾 satisfying the following equation [185]:

214

8 Application of Neural Network Blind Equalization Algorithm in Medical Image

1 n > ≥ 𝛾>n2,2 , q,1 erf (W(n + 1)) ≤ erf (W(n)) – =n =

(8.27) 1 n > + 𝛾>n2,2 , q,1

(8.28)

1 n > – 𝛾>n2,2 , q,1

(8.29)

Conclusion (1) was established because of eqs. (8.27) and (8.28). The following equation can be obtained from eq. (8.28), N

0 ≤ erf (W(n + 1)) ≤ erf (W(n)) – =n ≤ erf (W(N)) – ∑ =n

(8.30)

n=1 N

where ∑ =n ≤ erf (W(N)), N → ∞, n=1

∞

∑ =n ≤ erf (W(N)) < ∞

(8.31)

n=1

Suppose that assumptions (A1) and (A2) are established, there must be a constant c1 satisfying eq. (8.32) and a constant c2 satisfying eq. (8.33): 󵄩 󵄩2 ∞ 1 󵄩󵄩󵄩 𝜕erf (W(n)) 󵄩󵄩󵄩 1 󵄩󵄩 󵄩󵄩 ≤ c1 ∑ ( >n2,1 ) < ∞ 󵄩󵄩 𝜕W(n) 󵄩󵄩 n , 󵄩 󵄩 n=1 n=1 ∞

∑ K

(8.32)

N

󵄩 󵄩 ∑ ∑ 󵄩󵄩󵄩BW(n)󵄩󵄩󵄩 ≤ c2

(8.33)

k=1 n=1

The mean value theorem is adopted for p(x) and p󸀠 (x), and there exists a constant c3 satisfying the following equation: 󵄨󵄨󵄩󵄩 𝜕erf (W(n + 1)) 󵄩󵄩 󵄩󵄩 𝜕erf (W(n)) 󵄩󵄩󵄨󵄨 K 󵄩󵄩 󵄩󵄩 󵄩󵄩󵄨󵄨 󵄨󵄨󵄩󵄩 󵄩󵄩 – 󵄩󵄩 󵄩󵄩󵄨󵄨 ≤ c3 ∑ 󵄩󵄩󵄩󵄩dk 󵄩󵄩󵄩󵄩 󵄨󵄨󵄩󵄩 󵄩󵄩󵄨󵄨 𝜕W 𝜕W 󵄩󵄩󵄩 󵄩󵄩󵄩 󵄨󵄨󵄨󵄩󵄩󵄩 󵄩󵄨 k=1

(8.34)

where dk = Wk (n + 1) – Wk (n). From eqs. (8.33) and (8.34), it can be obtained: 󵄩󵄩 𝜕erf (W (n)) 󵄩󵄩 󵄩 󵄩󵄩 k 󵄩󵄩 = 0 lim 󵄩󵄩󵄩󵄩 󵄩󵄩 n→∞ 󵄩 𝜕W (n) k 󵄩 󵄩

(8.35)

Resembling eq. (8.34), there is a constant c4 satisfying the following equation: 󵄩󵄩 𝜕erf (W (n + 1)) 𝜕erf (W (n)) 󵄩󵄩 󵄩󵄩 󵄩󵄩 c4 k+l k 󵄩󵄩 󵄩󵄩 ≤ – 󵄩󵄩 󵄩󵄩 𝜕W 𝜕W l 󵄩 󵄩

(8.36)

8.2 Medical CT image neural network blind equalization algorithm based on Zigzag

215

According to eqs. (8.35) and (8.36), it can be obtained: 󵄩󵄩 𝜕erf (W (n)) 󵄩󵄩 󵄩 󵄩󵄩 k+l 󵄩󵄩 = 0 lim 󵄩󵄩󵄩󵄩 󵄩󵄩 n→∞ 󵄩 𝜕W 󵄩 󵄩

(8.37)

Therefore, conclusion (2) is established. From eq. (8.34), it can be drawn: 󵄩 󵄩 lim 󵄩󵄩Wk (n + 1) – Wk (n)󵄩󵄩󵄩 = 0

n→∞ 󵄩

k = 1, 2, . . . , K

(8.38)

K

󵄩󵄩 󵄩 󵄩 󵄩 󵄩󵄩W(n + 1) – W(n)󵄩󵄩󵄩 ≤ ∑ 󵄩󵄩󵄩Wk (n + 1) – Wk (n)󵄩󵄩󵄩

(8.39)

󵄩 󵄩 lim 󵄩󵄩W(n + 1) – W(n)󵄩󵄩󵄩 = 0

(8.40)

k=1

n→∞ 󵄩

To prove this, a lemma is introduced [186]. Lemma: Supposed that E:ℜMN+M+1 → ℜ is a uniformly differentiable function in set D ⊂ ℜMN+M+1 . There are limited points in D0 = { W ∈ D| 𝜕E(W) = 0}. If the following 𝜕W ∞ conditions are satisfied by a sequence of {Wm }m=1 ⊂ D 󵄩 󵄩 lim 󵄩󵄩Wm+1 – Wm 󵄩󵄩󵄩󵄩 = 0 󵄩󵄩 𝜕E (Wm ) 󵄩󵄩 󵄩󵄩 󵄩 󵄩󵄩 = 0 lim 󵄩󵄩󵄩󵄩 󵄩󵄩 m→∞ 󵄩 𝜕W 󵄩 󵄩 󵄩 m→∞ 󵄩

(8.41) (8.42)

then, there exists a point W∗ ⊂ D0 making the following condition satisfied: lim Wm = W∗

m→∞

(8.43)

where m is iteration number. erf (W(n)) is uniformly differentiable. Because of eq. (8.40) and lemma, there exists W∗ ∈ D0 that satisfies eq. (8.44). lim W(n) = W∗

n→∞

(8.44)

As a result, conclusion (3) is established.

8.2.4 Experimental simulation In order to verify algorithm’s validity, the CT image with 8-bit grayscale is used and the size is 256 * 256. The degradation simulation model is a Gaussian model. The point spread function is 10 * 10 Gauss matrix and its variance is 32 = 0.05. The original CT image is shown in Figure 8.6 (a).The degraded image which convolved by the point spread function and added by Gaussian white noise with mean 0 and variance 0.006

216

8 Application of Neural Network Blind Equalization Algorithm in Medical Image

(a)

(b)

(c)

(d)

(e)

(f)

Figure 8.6: Experiment image and restored images: (a) original CT image; (b) noise blurred image; (c) IBD algorithm; (d) dispersion minimization algorithm; (e) reduction dimension constant module algorithm; and (f) the proposed algorithm.

is shown in Figure 8.6 (b). The 3-layer (or 11-5-1) neural network is performed in the simulation. There is one value “1” in the initial weights of every layer while the others are value “0”. The weight iteration step between the input layer and hidden layer is ,1 = 1 × 10–9 . The weight iteration step between the output layer and hidden layer is ,2 = 2 × 10–8 . R2 = 39, 320.0. The restored image by IBD [187] is shown in Figure 8.6(c) after 100 iterations. The restored image by divergence minimization blind restoration algorithm [188] is shown in Figure 8.6 (d). The restored image by dimensionality reduction constant modulus algorithm proposed in the literature [18] is shown in Figure 8.6 (e). The restored image by medical CT image neural network blind equalization algorithm based on zigzag coding is shown in Figure 8.6 (f). The experimental results show that the computational efficiency of medical CT image neural network blind equalization algorithm based on zigzag coding is not improved compared with divergence minimization blind restoration algorithm and the dimension reduction constant modulus algorithm. However, the image contrast and the PSNR are improved. The restoration effect by IBD algorithm is poor because of its anti-noise effects. Time consumptions (Computer 1.60 GHz, RAM 1 GB) and PSNR by neural network constant modulus blind equalization algorithm based on zigzag coding, minimum

8.2 Medical CT image neural network blind equalization algorithm based on Zigzag

217

Table 8.1: Comparison of different algorithms.

3n2 = 0.05 3n2 = 0.01

PSNR Time (s) PSNR Time (s)

The proposed algorithm

Dispersion minimization algorithm

Reduction dimension constant module algorithm

Iterative blind deconvolution algorithm

25.045 14.376 25.782 13.176

24.788 17.211 24.931 17.123

24.324 8.934 24.716 7.812

23.892 21.435 23.987 22.832

divergence blind restoration algorithm [188], dimension reduction constant modulus algorithm [18], and iterative blind deconvolution algorithm (IBD) [187] are given in Table 8.1. The table shows that PSNR measure of image quality and time cost are improved compared with the reduction dimension constant module algorithm. However, the time is longer for the introduction of multilayer network weight updating. The recovery by IBD is good when there is few noise. However, realization is hard since the alternating big iteration. The computational complexity in divergence minimization blind recovery algorithm increases because of image matrix and inverse matrix iterative calculation. However, the influences by noise is small. The nasal pharynx chronic inflammation from CT image is shown in Figure 8.7(a). An 8-bit grayscale is used and the size is 256 × 256. A 25 × 25 Gauss model with 32 = 0.002 is used in the degradation process. The degraded image which is added by the Gauss white noise with mean 0 and variance 0.05 is shown in Figure 8.7 (b). The 3-layer (or 21-7-1) neural network is performed in the simulation. There is one value “1” in the initial weights in every layer while the others are valued ”0.” The weight iteration step between input layer and the hidden layer is ,1 = 5 × 10–7 . The weight iteration step between hidden layer and output layer is ,2 = 1 × 10–7 , R2 = 39320.0, respectively. The restored image by maximum likelihood method is shown in Figure 8.7 (c); The iteration number of IBD is 100. The restored image is shown in Figure 8.7(d). The restored images by divergence minimization blind restoration algorithm and zigzag coding neural network constant modulus blind equalization algorithm are shown in Figures 8.7 (e) and 8.7 (f), respectively. Figures 8.6 and 8.7 show that the restoration effect by the neural network constant modulus blind equalization algorithm based on zigzag encoding is better compared with the dispersion minimum algorithm, the reduction dimension constant modulus algorithm, and IBD algorithm. The value of PSNR and ISNR is improved by the neural network constant modulus blind equalization algorithm based on zigzag encoding, compared with the dispersion

218

8 Application of Neural Network Blind Equalization Algorithm in Medical Image

(a)

(b)

(c)

(d)

(e)

(f)

Figure 8.7: Experiment image and restored images in the experiment 2: (a) Original CT image (nasopharyngeal chronic inflammation); (b) noise blurred image; (c) IBD algorithm; (d) dispersion minimization algorithm; (e) reduction dimension constant module algorithm; and (f) the proposed algorithm. Table 8.2: Comparison of different algorithms.

PSNR MSE ISNR Time (s)

The proposed algorithm

Dispersion minimization algorithm

Reduction dimension constant module algorithm

Iterative blind deconvolution algorithm

23.7854 37.7453 1.0622 12.736

23.0871 39.6308 0.9736 18.323

23.1543 38.6856 0.9986 7.516

23.0431 40.0848 0.7488 21.322

minimum algorithm [188], the reduction dimension constant modulus algorithm [18], and IBD algorithm [187]. The image is converted into one-dimensional signal sequence by zigzag encoding. The influence of point spread function is eliminated by the neural network blind equalization algorithm. Although the complexity is increased, the value of PSNR, ISNR, and MSE is improved. With the development of hardware computing power, complexity is not considered as the main factor, so the algorithm is practical.

8.3 Medical CT image neural network blind equalization algorithm based on double

219

8.3 Medical CT image neural network blind equalization algorithm based on double zigzag encoding 8.3.1 The basic principle of algorithm According to medical imaging principle, the image features have easily been highlighted by orthogonal transform. In particular, the details of medical images are particularly important. The degradation image is transformed by the orthogonal transform. Image restoration is implemented in the transform domain. In order to restore real and imaginary part simultaneously, the real and imaginary images are converted into a complex signal sequence by zigzag encoding. Blind image restoration is achieved by the idea that the inter-symbol interference eliminated by complex signal. Therefore, the medical image system is equivalent to orthogonal complex signal system in this section. The effect of point spread function on medical image is eliminated by inter-symbol interference eliminated in complex sequence. The medical image is restored by an equivalent complex system. CT can be restored by equalized complex system. The diagram is shown in Figure 8.8. Where gR (n) + jgI (n) is one-dimensional complex degradation signal sequence by double zigzag encoding. fR̂ (n) + jfÎ (n) is the output of neural network blind equalizer. fR̂ (n) + jfÎ (n) is the output of judgment. fiĵ is the estimated image. There exists some nonlinear units in the neural network. The complex function need to be used as neural network transfer function in complex signal processing. The complex function is composed by the real and imaginary part. The complex signal can be divided into two parts before they come into nonlinear transfer function. The first part is used to transmit the real part of the signal, while the other part is used to transmit the imaginary part of the signal. A complex signal is remixed after the signal passes the nonlinear transfer function. The network structure is shown in Figure 8.9.

Degradation image gij

one dimensional sequence

Zigzag row coding Zigzag column coding

× j

~ ~ fR(n) + jfI(n) Neural network

+ gR(n) + jgI(n)

Blind equalization

Restored image

Zigzag row decoding

Re

fˆij

Zigzag column decoding

Im

Image merging

– Judgment + + fˆR(n) + jfˆI(n)

Figure 8.8: The block diagram of neural network blind equalization algorithm based on double zigzag encoding.

220

g(n)

8 Application of Neural Network Blind Equalization Algorithm in Medical Image

qR( fR, fI) + qI( fR, fI)

×

Figure 8.9: Complex value processing unit in the neural network model.

j

The cost function and transfer function of the complex system are shown in formulas (3.6) and (3.31), respectively. Because of the existence of nonlinear transfer function in the neural network, all network weights should be divided into the real part and the imaginary part, namely, W(n) = WR (n) + jWI (n)

󵄨 󵄨 󵄨󵄨 󵄨󵄨 𝜕 󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨󵄨 𝜕J(n) 󵄨 󵄨 󵄨2 󵄨 𝜕 󵄨󵄨 f ̃(n)󵄨󵄨󵄨 = 2 [󵄨󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨󵄨 – R2 ] 󵄨󵄨󵄨󵄨f ̃(n)󵄨󵄨󵄨󵄨 [ 󵄨 +j 󵄨 ] ∇= 𝜕W(n) 𝜕WR (n) 𝜕WI (n)

(8.45) (8.46)

For ease of deducing the formula, the network signal can be written as g (n – l) = gR (n – l) + jgI (n – l)

(8.47)

wlk (n) = wlk,R (n) + jwlk,I (n)

(8.48)

uk (n) = ∑ wlk (n)g (n – l) l

= ∑ [wlk,R (n)gR (n – l) – wlk,I (n)gI (n – l)]

(8.49)

l

+ j ∑ [wlk,R (n)gI (n – l) + wlk,I (n)gR (n – l)] l

Ik (n) = q [uk,R (n)] + jq [uk,I (n)] wk (n) = wk,R (n) + jwk,I (n)

(8.50) (8.51)

v(n) = ∑ wk (n)Ik (n) = ∑ [wk,R (n)Ik,R (n) – wk,I (n)Ik,I (n)] k

k

+ j ∑ [wk,R (n)Ik,I (n) + wk,I (n)Ik,R (n)]

(8.52)

k

f ̃(n) = q [vR (n)] + jq [vI (n)]

(8.53)

8.3.2 Formula derivation of iterative algorithm Simply, a three-layer feed-forward neural network structure is adopted. However, a complex transfer function is needed to deal with the complex value system. Because the network includes input layer, hidden layer and output layer, there are different weights iterative formulas.

221

8.3 Medical CT image neural network blind equalization algorithm based on double

8.3.2.1 Weights iterative formula of the output layer and hidden layer For the complex-valued feed-forward neural network, the weights between output layer and hidden layer are wk (n): 󵄨 󵄨 𝜕 󵄨󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨󵄨

𝜕 [f ̃(n)f ̃∗ (n)] 𝜕 {q2 [vR (n)] + q2 [vI (n)]} 1 1 = 󵄨 󵄨 󵄨 󵄨 𝜕wk,R (n) 2 󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨 𝜕wj,R (n) 𝜕wk,R (n) 2 󵄨󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨󵄨 󵄨 󵄨 1 󸀠 󸀠 = 󵄨󵄨 󵄨󵄨 {q [vR (n)] q [vR (n)] Ik,R (n) + q [vI (n)] q [vI (n)] Ik,I (n)} ̃ 󵄨󵄨󵄨 f (n)󵄨󵄨󵄨 󵄨 󵄨 𝜕 [f ̃(n)f ̃∗ (n)] 𝜕 {q2 [vR (n)] + q2 [vI (n)]} 𝜕 󵄨󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨󵄨 1 1 = 󵄨󵄨 = 󵄨 󵄨 𝜕wk,I (n) 2 󵄨󵄨 f ̃(n)󵄨󵄨󵄨󵄨 𝜕wk,I (n) 𝜕wk,I (n) 2 󵄨󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨󵄨 󵄨 󵄨 1 󸀠 󸀠 = 󵄨󵄨 󵄨 {–q [vR (n)] q [vR (n)] Ik,I (n) + q [vI (n)] q [vI (n)] Ik,R (n)} 󵄨󵄨 f ̃(n)󵄨󵄨󵄨 󵄨 󵄨 =

(8.54)

(8.55)

Equations (8.54) and (8.55) are given as 󵄨 󵄨 𝜕 󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨󵄨 1 +j 󵄨 = {q [vR (n)] q󸀠 [vR (n)] Ik,R (n) + q [vI (n)] q󸀠 [vI (n)] Ik,I (n)} 𝜕wk,R (n) 𝜕wk,I (n) 󵄨󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨󵄨 󵄨 󵄨 1 󸀠 󸀠 + j 󵄨󵄨 󵄨 {–q [vR (n)] q [vR (n)] Ik,I (n) + q [vI (n)] q [vI (n)] Ik,R (n)} 󵄨󵄨 f ̃(n)󵄨󵄨󵄨 󵄨 󵄨 1 󸀠 󸀠 ∗ = 󵄨󵄨 󵄨 {q [vR (n)] q [vR (n)] + jq [vI (n)] q [vI (n)]} Ik (n) 󵄨󵄨 f ̃(n)󵄨󵄨󵄨 󵄨 󵄨 (8.56) 󵄨 󵄨 𝜕 󵄨󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨󵄨

Equation (8.46) is put in eq.(8.56), and the output layer weight iterative formula is 󵄨 󵄨2 wk (n + 1) = wk (n) – 2,k [󵄨󵄨󵄨󵄨f ̃(n)󵄨󵄨󵄨󵄨 – R2 ] {q [vR (n)] q󸀠 [vR (n)] + jq [vI (n)] q󸀠 [vI (n)]} Ik∗ (n) (8.57) where ,k is the step between output layer and hidden layer.

8.3.2.2 Weights iterative formula of the hidden layer The weight between the kth node in the hidden layer and the lth node in the input layer is wlk (n) in the complex feed-forward neural network. The iteration formula is wlk (n + 1) = wlk (n) – ,lk

𝜕J(n) 𝜕wlk (n)

󵄨 󵄨 󵄨󵄨 󵄨󵄨 𝜕 󵄨󵄨󵄨󵄨f ̃(n)󵄨󵄨󵄨󵄨 𝜕J(n) 󵄨󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨 󵄨󵄨 ̃ 󵄨󵄨2 = 2 [󵄨󵄨󵄨 f (n)󵄨󵄨󵄨 – R2 ] 󵄨󵄨󵄨 f (n)󵄨󵄨󵄨 [ +j ] 𝜕wlk (n) 𝜕wlk,R (n) 𝜕wlk,I (n)

(8.58) (8.59)

222

8 Application of Neural Network Blind Equalization Algorithm in Medical Image

According to eqs. (8.47) and (8.53), 󵄨 󵄨 𝜕 󵄨󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨󵄨

𝜕 [f ̃(n)f ̃∗ (n)] 1 󵄨 󵄨 𝜕wlk,R (n) 𝜕wlk,R (n) 2 󵄨󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨󵄨 𝜕wlk,R (n) 𝜕 {q2 [vR (n)] + q2 [vI (n)]} 1 = 󵄨󵄨 󵄨 𝜕wlk,R (n) 2 󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨󵄨 𝜕vR (n) 1 󸀠 = 󵄨󵄨 󵄨 q [vR (n)] q [vR (n)] 𝜕w (n) + 󵄨󵄨 f ̃(n)󵄨󵄨󵄨 lk,R 󵄨 󵄨 =

𝜕√f ̃(n)f ̃∗ (n)

=

𝜕vI (n) 1 󸀠 󵄨󵄨 ̃ 󵄨󵄨 q [vI (n)] q [vI (n)] 𝜕w (n) 󵄨󵄨 f (n)󵄨󵄨 lk,R 󵄨 󵄨 (8.60)

𝜕uk,R (n) 𝜕uk,I (n) 𝜕vR (n) = wk,R (n)q󸀠 [uk,R (n)] – wk,I (n)q󸀠 [uk,I (n)] 𝜕wlk,R (n) 𝜕wlk,R (n) 𝜕wlk,R (n) 󸀠

(8.61)

󸀠

= wk,R (n)q [uk,R (n)] yR (n – l) – wk,I (n)q [uk,I (n)] gI (n – l) 𝜕uk,I (n) 𝜕uk,R (n) 𝜕vI (n) = wk,R (n)q󸀠 [uk,I (n)] + wk,I (n)q󸀠 [uk,R (n)] 𝜕wlk,R (n) 𝜕wlk,R (n) 𝜕wlk,R (n) 󸀠

(8.62)

󸀠

= wk,R (n)q [uk,I (n)] gI (n – l) + wk,I (n)q [uk,R (n)] gR (n – l) Similarly, 󵄨 󵄨 𝜕 󵄨󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨󵄨

𝜕 [f ̃(n)f ̃∗ (n)] 1 󵄨 󵄨 𝜕wlk,I (n) 𝜕wlk,I (n) 2 󵄨󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨󵄨 𝜕wlk,I (n) 𝜕 {q2 [vR (n)] + q2 [vI (n)]} 1 = 󵄨󵄨 󵄨 𝜕wlk,I (n) 2 󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨󵄨 𝜕vR (n) 1 󸀠 = 󵄨󵄨 󵄨 q [vR (n)] q [vR (n)] 𝜕w (n) + 󵄨󵄨 f ̃(n)󵄨󵄨󵄨 lk,I 󵄨 󵄨 =

𝜕√f ̃(n)f ̃∗ (n)

=

𝜕vI (n) 1 󸀠 󵄨󵄨 ̃ 󵄨󵄨 q [vI (n)] q [vI (n)] 𝜕w (n) 󵄨󵄨 f (n)󵄨󵄨 lk,I 󵄨 󵄨 (8.63)

𝜕uk,R (n) 𝜕uk,I (n) 𝜕vR (n) = wk,R (n)q󸀠 [uk,R (n)] – wk,I (n)q󸀠 [uk,I (n)] 𝜕wlk,I (n) 𝜕wlk,I (n) 𝜕wlk,I (n) 󸀠

(8.64)

󸀠

= –wk,R (n)q [uk,R (n)] yI (n – l) – wk,I (n)q [uk,I (n)] gR (n – l) 𝜕uk,I (n) 𝜕uk,R (n) 𝜕vI (n) = wk,R (n)q󸀠 [uk,I (n)] + wk,I (n)q󸀠 [uk,R (n)] 𝜕wlk,R (n) 𝜕wlk,I (n) 𝜕wlk,I (n) 󸀠

󸀠

= wk,R (n)q [uk,I (n)] gR (n – l) + wk,I (n)q [uk,R (n)] gI (n – l)

(8.65)

8.3 Medical CT image neural network blind equalization algorithm based on double

223

Equations (8.61)–(8.65) are put in eq. (8.46), then 󵄨 󵄨 𝜕 󵄨󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨󵄨

𝜕wlk,R (n)

+j

󵄨 󵄨 𝜕 󵄨󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨󵄨

1 = 󵄨󵄨 q [vR (n)] q󸀠 [vR (n)] wk,R (n)q󸀠 [uk,R (n)] gR (n – l) ̃ 𝜕wlk,I (n) 󵄨󵄨 f (n)󵄨󵄨󵄨󵄨 󵄨 󵄨

1 󸀠 󸀠 – 󵄨󵄨 󵄨 q [vR (n)] q [vR (n)] wk,I (n)q [uk,I (n)] gI (n – l) 󵄨󵄨 f ̃(n)󵄨󵄨󵄨 󵄨 󵄨 1 󸀠 󸀠 + 󵄨󵄨 󵄨 q [vI (n)] q [vI (n)] wk,R (n)q [uk,I (n)] gI (n – l) 󵄨󵄨 f ̃(n)󵄨󵄨󵄨 󵄨 󵄨 1 󸀠 󸀠 – 󵄨󵄨 󵄨 q [vI (n)] q [vI (n)] wk,I (n)q [uk,R (n)] gR (n – l) 󵄨󵄨 f ̃(n)󵄨󵄨󵄨 󵄨 󵄨 1 󸀠 󸀠 – j 󵄨󵄨 󵄨 q [vR (n)] q [vR (n)] wk,R (n)q [uk,R (n)] gI (n – l) 󵄨󵄨 f ̃(n)󵄨󵄨󵄨 󵄨 󵄨 1 󸀠 󸀠 – j 󵄨󵄨 󵄨 q [vR (n)] q [vR (n)] wk,I (n)q [uk,I (n)] gR (n – l) 󵄨󵄨 f ̃(n)󵄨󵄨󵄨 󵄨 󵄨 1 󸀠 󸀠 + j 󵄨󵄨 󵄨󵄨 q [vI (n)] q [vI (n)] wk,R (n)q [uk,I (n)] gR (n – l) 󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨 1 󸀠 󸀠 – j 󵄨󵄨 󵄨󵄨 q [vI (n)] q [vI (n)] wk,I (n)q [uk,R (n)] gI (n – l) ̃ 󵄨󵄨󵄨 f (n)󵄨󵄨󵄨 1 󸀠 󸀠 󸀠 ∗ = 󵄨󵄨 󵄨 q [uk,R (n)] Re {[q [vR (n)] q [vR (n)] + jq [vI (n)] q [vI (n)]] wk (n)} 󵄨󵄨 f ̃(n)󵄨󵄨󵄨 󵄨 󵄨 × [gR (n – l) – jgI (n – l)] 1 󸀠 󸀠 󸀠 ∗ + j 󵄨󵄨 󵄨 f [uj,I (n)] Im {[q [vR (n)] q [vR (n)] + jq [vI (n)] q [vI (n)]] wk (n)} 󵄨󵄨 f ̃(n)󵄨󵄨󵄨 󵄨 󵄨 × [gR (n – l) – jgI (n – l)]

(8.66)

Equation (8.58)is put in eq. (8.66), 󵄨 󵄨2 wlk (n + 1) = wlk (n) – 4,lk [󵄨󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨󵄨 – R2 ] q󸀠 [uj,R (n)]

× Re {[q [vR (n)] q󸀠 [vR (n)] + jq [vI (n)] q󸀠 [vI (n)]] wj∗ (n)} × g ∗ (n – l) 󵄨 󵄨2 + j4, [󵄨󵄨󵄨󵄨 f ̃(n)󵄨󵄨󵄨󵄨 – R2 ] q󸀠 [uj,I (n)]

(8.67)

× Im {[q [vR (n)] q󸀠 [vR (n)] + jq [vI (n)] q󸀠 [vI (n)]] wk∗ (n)} × g ∗ (n – l)

where ,lk is the iteration step. According to the formula above, the feed-forward neural network blind equalization algorithm is extended to the complex domain. If the number of the neural network hidden layer increases, the iterative formula of the hidden layer are still available.

224

8 Application of Neural Network Blind Equalization Algorithm in Medical Image

8.3.3 Experimental simulation The CT image is shown in Figure 8.10(a). An 8-bit grayscale is used and the size is 256 × 256. A 25 × 25 Gauss model with 32 = 0.015 is used in the degradation process. The degraded image which is added by the Gauss white noise with mean 0 and variance 0.01 is shown in Figure 8.10(b). The 3-layer (or 11-5-1) neural network is performed in the simulation. There is one value “1” in the initial weights of each layer and the others are value “0”. The weight iteration step between input layer and the hidden layer is ,1 = 5 × 10–8 . The iteration step between hidden layer and output layer is ,2 = 3 × 10–7 , R2 = 39320.0. The proportional coefficient is ! = 0.05. The restored image by maximum likelihood is shown in Figure 8.10(c). The number of iterations is 100, the restored image by IBD is shown in Figure 8.10(d). The restored images by divergence minimum blind image restoration algorithm and neural network blind equalization algorithm based on double Zigzag encoding are shown in Figures 8.10(e) and 8.10(f), respectively. The 8-bit local CT (anterior meditational yolk sac tumor) image is shown in Figure 8.11(a). The size is 256 × 256. The degraded image is added by Gauss white noise

(a)

(b)

(c)

(d)

(e)

(f)

Figure 8.10: Experiment image and restored images: (a) original CT image (ectopic pancreas); (b) noise blurred image; (c) maximum likelihood algorithm; (d) IBD algorithm; (e) dispersion minimization algorithm; and (f) neural network blind equalization algorithm based on double zigzag encoding.

8.3 Medical CT image neural network blind equalization algorithm based on double

225

Table 8.3: Comparison of different algorithms.

PSNR MSE ISNR Time (s)

3n2 3n2 3n2 3n2 3n2 3n2 3n2 3n2

= 0.001 = 0.01 = 0.001 = 0.01 = 0.001 = 0.01 = 0.001 = 0.01

The proposed algorithm

Dispersion minimization algorithm

IBD algorithm

Maximum likelihood algorithm

27.8965 27.7475 34.9856 35.9897 4.9875 4.9586 17.986 16.969

27.3424 27.2700 39.9221 40.9248 4.5632 4.4811 18.126 18.453

26.5664 26.4208 46.1234 46.9867 3.7512 3.6318 22.122 22.232

25.4323 25.1347 50.2312 50.6734 2.3452 2.1243 6.816 6.921

(a)

(b)

(c)

(d)

(e)

(f)

Figure 8.11: Experiment image and restored images in the experiment 2: (a) Original CT image (anterior meditational yolk sac tumor); (b) noise blurred image; (c) maximum likelihood; (d) IBD algorithm; (e) dispersion minimization algorithm; and (f) neural network blind equalization algorithm based on double zigzag encoding.

with mean 0 and variance 0.001. The size of Gauss white noise is 25 × 25. It is shown in Figure 8.11(b). The 3-layer (or 21-7-1) neural network is performed. There is one value “1” in the initial weights of each layer and the others are value “0.” The iteration step between input layer and hidden layer is ,1 = 4 × 10–7 . The iteration step between

226

8 Application of Neural Network Blind Equalization Algorithm in Medical Image

Table 8.4: Comparison of different algorithms.

PSNR MSE ISNR Time (s)

3n2 3n2 3n2 3n2 3n2 3n2 3n2 3n2

= 0.001 = 0.01 = 0.001 = 0.01 = 0.001 = 0.01 = 0.001 = 0.01

The neural network blind equalization algorithm based on double zigzag encoding

Dispersion minimization algorithm

IBD algorithm

Maximum likelihood algorithm

25.9875 25.9500 31.0123 31.8417 2.7899 2.7497 15.762 15.864

25.7342 25.6512 35.4567 36.4294 2.4789 2.4509 17.262 17.323

25.2314 25.0998 42.4563 43.4303 1.9685 1.8995 21.920 21.821

24.9865 24.6570 45.1234 46.4536 1.2344 1.1231 7.112 7.212

hidden layer and output layer is ,2 = 2 × 10–8 , R2 = 39320.0. The proportional coefficient is ! = 0.01. The restored image by maximum likelihood [189] is shown in Figure 8.11 (c). The number of iterations is 100 in IBD algorithm [187]. The restored image by IBD is shown in Figure 8.11(d). The restored image by dispersion minimum [188] blind image restoration algorithms and neural network blind equalization algorithm based on double zigzag encoding are shown in Figures 8.11(e) and 8.10(f), respectively. Figures 8.6 and 8.7 show that the restoration effect by the neural network blind equalization algorithm based on double zigzag encoding is better compared with the dispersion minimum algorithm, the maximum likelihood algorithm, and IBD algorithm. The approximation ability of the neural network is used to obtain the inverse of point spread function, which has the preferable approximation ability. Compared with divergence minimum blind restoration algorithm, the table shows that PSNR and ISNR are improved by neural network blind equalization algorithm based on double zigzag image coding,. The image is transformed into a one-dimensional complex signal sequence by the complex valued transform. As a result, the amount of computation increases. Compared with Dispersion minimization algorithm, IBD algorithm and Maximum likelihood algorithm, the complexity increased while PSNR and ISNR are improved for medical CT image neural network blind equalization algorithm based on double zigzag encoding. The MSE is reduced.

8.4 Summary Medical CT image neural network blind equalization algorithm based on Zigzag encoding is analyzed in this chapter. The transfer function is designed and the iterative formulas are deduced. Experimental simulations are carried out to test the

8.4 Summary

227

convergence performance. The medical CT image neural network blind equalization algorithm based on double zigzag encoding is an extension application of complex value system, which combines the common features in the complex system and neural network blind equalization. Computer simulation results show that the restoration effect, SNR, and PSNR are improved by the proposed algorithm. The MSE is reduced by the proposed algorithm.

Appendix A: Derivation of the Hidden Layer Weight Iterative Formula in the Blind Equalization Algorithm Based on the Complex Three-Layer FFNN In the complex feed-forward neural network (FFNN), the connection weight between the hidden layer and the input layer is wij (n), so the iterative formulas are wij (n + 1) = wij (n) – ,2

𝜕J(n) 𝜕wij (n)

(A.1)

󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕󵄨󵄨󵄨x(n) 󵄨󵄨 󵄨 𝜕J(n) 󵄨 ̃ 󵄨󵄨2 󵄨󵄨 ̃ 󵄨󵄨 𝜕󵄨󵄨󵄨x(n) 󵄨 = 2 [󵄨󵄨󵄨x(n) + j ] – R ] x(n) [ 󵄨󵄨 󵄨󵄨 2 󵄨󵄨 𝜕wij (n) 𝜕wij,R (n) 𝜕wij,I (n)

(A.2)

According to eqs. (3.16)–(3.22), we obtain 󵄨 ̃ 󵄨󵄨 ̃ x̃ ∗ (n)] 𝜕 󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕 [x(n) = = 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 𝜕wij,R (n) 𝜕wij,R (n) 𝜕wij,R (n) 2 󵄨󵄨x(n) =

2 2 1 𝜕 {f [vR (n)] + f [vI (n)]} 󵄨 ̃ 󵄨󵄨 𝜕wij,R 2 󵄨󵄨󵄨x(n) 󵄨󵄨

(A.3)

𝜕vR (n) 𝜕vI (n) 1 1 󸀠 󸀠 = 󵄨󵄨 󵄨󵄨 f [vR (n)] f [vR (n)] 𝜕w (n) + 󵄨󵄨 ̃ 󵄨󵄨 f [vI (n)] f [vI (n)] 𝜕w (n) ̃ 󵄨󵄨x(n)󵄨󵄨 󵄨󵄨x(n)󵄨󵄨 ij,R ij,R 𝜕u 𝜕u (n) (n) 𝜕vR (n) j,R j,I = wj,R (n)f 󸀠[uj,R (n)] – wj,I (n)f 󸀠[uj,I (n)] 𝜕wij,R (n) 𝜕wij,R (n) 𝜕wij,R (n) 󸀠

= wj,R (n)f [uj,R (n)] yR (n – i) – wj,I (n)f [uj,I (n)] yI (n – i)) 𝜕uj,I (n) 𝜕uj,R (n) 𝜕vI (n) = wj,R (n)f 󸀠[uj,I (n)] + wj,I (n)f 󸀠[uj,R (n)] 𝜕wij,R (n) 𝜕wij,R (n) 𝜕wij,R (n) 󸀠

(A.4)

󸀠

(A.5)

󸀠

= wj,R (n)f [uj,I (n)] yI (n – i) + wj,I (n)f [uj,R (n)] yR (n – i) Similarly 󵄨 ̃ 󵄨󵄨 𝜕󵄨󵄨󵄨x(n) 󵄨󵄨 𝜕vR (n) 𝜕vI (n) 1 1 = + f [v (n)] f 󸀠[vR (n)] f [v (n)] f 󸀠[vI (n)] ̃ 󵄨󵄨󵄨󵄨 R ̃ 󵄨󵄨󵄨󵄨 I 𝜕wij,I (n) 󵄨󵄨󵄨󵄨x(n) 𝜕wij,I (n) 󵄨󵄨󵄨󵄨x(n) 𝜕wij,I (n) 𝜕vR (n) = –wj,R (n)f 󸀠[uj,R (n)] yI (n – i) – wj,I (n)f 󸀠[uj,I (n)] yR (n – i) 𝜕wij,I (n) 𝜕vI (n) = wj,R (n)f 󸀠[uj,I (n)] yR (n – i) – wj,I (n)f 󸀠[uj,R (n)] yI (n – i) 𝜕wij,I (n) DOI 10.1515/9783110450293-009

(A.6) (A.7) (A.8)

230

Appendix A: Derivation of the Hidden Layer Weight Iterative Formula

Plugging eqs. (A.3)–(A.8) into eq. (A.2), we obtain 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕 󵄨󵄨x(n) 𝜕 󵄨󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 1 f [v (n)] f 󸀠[vR (n)] wj,R (n)f 󸀠[uj,R (n)] yR (n – i) +j 󵄨 = ̃ 󵄨󵄨󵄨󵄨 R 𝜕wij,R (n) 𝜕wij,I (n) 󵄨󵄨󵄨󵄨x(n) 1 f [v (n)] f 󸀠[vR (n)] wj,I (n)f 󸀠[uj,I (n)] yI (n – i) – 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 R 󵄨󵄨x(n) 1 󸀠 󸀠 + 󵄨󵄨 󵄨󵄨 f [vI (n)] f [vI (n)] wj,R (n)f [uj,I (n)] yI (n – i) ̃ x(n) 󵄨󵄨 󵄨󵄨 1 f [v (n)] f 󸀠[vI (n)] wj,I (n)f 󸀠[uj,R (n)] yR (n – i) – 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 I 󵄨󵄨x(n) 1 f [v (n)] f 󸀠[vR (n)] wj,R (n)f 󸀠[uj,R (n)] yI (n – i) – j 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 R 󵄨󵄨x(n) 1 f [v (n)] f 󸀠[vR (n)] wj,I (n)f 󸀠[uj,I (n)] yR (n – i) – j 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 R 󵄨󵄨x(n) 1 f [v (n)] f 󸀠[vI (n)] wj,R (n)f 󸀠[uj,I (n)] yR (n – i) + j 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 I 󵄨󵄨x(n) 1 󸀠 󸀠 – j 󵄨󵄨 󵄨󵄨 f [vI (n)] f [vI (n)] wj,I (n)f [uj,R (n)] yI (n – i) ̃ x(n) 󵄨󵄨 󵄨󵄨 1 󸀠 f [uj,R (n)] Re {[f [vR (n)] f 󸀠[vR (n)] + jf [vI (n)] f 󸀠[vI (n)]] wj∗ (n)} = 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n)

× [yR (n – i) – jyI (n – i)] 1 󸀠 f [uj,I (n)] Im {[f [vR (n)] f 󸀠[vR (n)] + jf [vI (n)] f 󸀠[vI (n)]] wj∗ (n)} + j 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n)

× [yR (n – i) – jyI (n – i)]

(A.9)

Plugging eq. (A.9) into eq. (A.1), we obtain wij (n + 1) = wij (n) – 2,2 kj (n)y∗ (n – i)

(A.10)

where ,2 is the iterative step size: 󵄨 ̃ 󵄨󵄨2 󸀠 kj (n) = [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2 ] f [uj,R (n)]

× Re {[f [vR (n)] f 󸀠[vR (n)] + jf [vI (n)] f 󸀠[vI (n)]] wj∗ (n)} 󵄨 ̃ 󵄨󵄨2 󸀠 󸀠 + j [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2 ] f [uj,I (n)] × Im {[f [vR (n)] f [vR (n)] +jf [vI (n)] f 󸀠[vI (n)]] wj∗ (n)}

(A.11)

Appendix B: Iterative Formulas Derivation of Complex Blind Equalization Algorithm Based on BRNN (1) Weight iterative formula of the feed-forward unit The connection weight between the feed-forward unit and the output unit is cj (n), so

cj (n + 1) = cj (n) – ,c

𝜕J(n) 𝜕cj (n)

󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 𝜕J(n) 󵄨 ̃ 󵄨󵄨 𝜕󵄨󵄨󵄨x(n) 󵄨 ̃ 󵄨󵄨2 = 2 [󵄨󵄨󵄨x(n) +j 󵄨 ] 󵄨󵄨 [ 󵄨󵄨 – R2 ] 󵄨󵄨󵄨x(n) 𝜕cj (n) 𝜕cj,R (n) 𝜕cj,I (n)

(B.1)

(B.2)

According to eqs. (4.11)–(4.13), we obtain 󵄨 ̃ 󵄨󵄨 ̃ x̃ ∗ (n)] 𝜕󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕 [x(n) = = 󵄨󵄨 󵄨 󵄨 ̃ 󵄨󵄨 𝜕cj,R (n) 𝜕cj,R (n) 𝜕cj,R (n) 2 󵄨󵄨x(n) =

2 2 1 𝜕{f [vR (n)] + f [vI (n)]} 󵄨 ̃ 󵄨󵄨 𝜕cj,R (n) 2 󵄨󵄨󵄨x(n) 󵄨󵄨

(B.3)

𝜕vR (n) 𝜕vI (n) 1 󸀠 󸀠 = 󵄨󵄨 󵄨󵄨 {f [vR (n)] f [vR (n)] 𝜕c (n) + f [vI (n)] f [vI (n)] 𝜕c (n) } ̃ 󵄨󵄨x(n)󵄨󵄨 j,R j,R 1 󸀠 󸀠 = 󵄨󵄨 󵄨 {f [vR (n)] f [vR (n)] yR (n – j) + f [vI (n)] f [vI (n)] yI (n – j)} ̃ 󵄨󵄨x(n)󵄨󵄨󵄨 󵄨 ̃ 󵄨󵄨 ̃ x̃ ∗ (n)] 𝜕󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕 [x(n) = = 󵄨󵄨 󵄨 ̃ 󵄨󵄨󵄨 𝜕cj,I (n) 𝜕cj,I (n) 𝜕cj,I (n) 2 󵄨󵄨x(n) =

2 2 1 𝜕{f [vR (n)] + f [vI (n)]} 󵄨󵄨 ̃ 󵄨󵄨 𝜕cj,I (n) 2 󵄨󵄨x(n)󵄨󵄨

𝜕v (n) 𝜕v (n) 1 + f [vI (n)] f 󸀠 [vI (n)] I } = 󵄨󵄨 {f [vR (n)] f 󸀠 [vR (n)] R ̃ 󵄨󵄨󵄨󵄨 𝜕cj,I (n) 𝜕cj,I (n) 󵄨󵄨x(n) 1 = 󵄨󵄨 {–f [vR (n)] f 󸀠 [vR (n)] yI (n – j) + f [vI (n)] f 󸀠 [vI (n)] yR (n – j)} ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n)

DOI 10.1515/9783110450293-010

(B.4)

232

Appendix B: Iterative Formulas Derivation of Complex Blind Equalization Algorithm

󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕󵄨󵄨x(n) 𝜕󵄨󵄨x(n) 𝜕󵄨󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 󵄨󵄨 = 󵄨 +j 󵄨 𝜕cj (n) 𝜕cj,R (n) 𝜕cj,I (n) 1 = 󵄨󵄨 f [vR (n)] f 󸀠 [vR (n)] [yR (n – j) – jyI (n – j)] ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n) 1 󸀠 + 󵄨󵄨 󵄨󵄨 f [vI (n)] f [vI (n)] [yI (n – j) + jyR (n – j)] ̃ x(n) 󵄨󵄨 󵄨󵄨 1 = 󵄨󵄨 f [vR (n)] f 󸀠 [vR (n)] [yR (n – j) – jyI (n – j)] ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n) 1 󸀠 + j 󵄨󵄨 󵄨󵄨 f [vI (n)] f [vI (n)] [yR (n – j) – jyI (n – j)] ̃ x(n)) 󵄨󵄨 󵄨󵄨 1 = 󵄨󵄨 {f [vR (n)] f 󸀠 [vR (n)] + jf [vI (n)] f 󸀠 [vI (n)]} y∗ (n – j) ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n)

(B.5)

Plugging eq. (B.5) into eq. (B.1), then the weight iterative formula of the feed-forward unit is cj (n + 1) = cj (n) – 2,c Ky∗ (n – j)

(B.6)

where ,c is the iterative step size of the feed-forward unit, and K 󵄨 ̃ 󵄨󵄨2 [󵄨󵄨󵄨x(n) 󵄨󵄨 – R2 ] {f [vR (n)] f 󸀠 [vR (n)] + jf [vI (n)] f 󸀠 [vI (n)]}.

=

(2) Weight iterative formula of the feedback unit The connection weight between the feedback unit and the output unit is ai (n), so ai (n + 1) = ai (n) – ,a

𝜕J(n) 𝜕ai (n)

󵄨󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 󵄨󵄨 𝜕J(n) 󵄨 ̃ 󵄨󵄨2 󵄨 ̃ 󵄨󵄨 𝜕󵄨󵄨󵄨x(n) 󵄨󵄨 + j 𝜕󵄨󵄨x(n) = 2 [󵄨󵄨󵄨x(n) ] 󵄨󵄨 – R2 ] 󵄨󵄨󵄨x(n) 󵄨󵄨 [ 𝜕ai (n) 𝜕ai,R (n) 𝜕ai,I (n)

(B.7) (B.8)

According to eqs. (4.11)–(4.13), we obtain 󵄨 ̃ 󵄨󵄨 ̃ x̃ ∗ (n)] 𝜕󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕 [x(n) = = 󵄨󵄨 󵄨 ̃ 󵄨󵄨󵄨 𝜕ai,R (n) 𝜕ai,R (n) 𝜕ai,R (n) 2 󵄨󵄨x(n) 2 2 1 𝜕{f [vR (n)] + f [vI (n)]} = 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 𝜕ai,R (n) 2 󵄨󵄨x(n) 𝜕v (n) 1 = 󵄨󵄨 f [vR (n)] f 󸀠 [vR (n)] R ̃ 󵄨󵄨󵄨󵄨 𝜕ai,R (n) 󵄨󵄨x(n) 𝜕vI (n) 1 + 󵄨󵄨 f [vI (n)] f 󸀠 [vI (n)] ̃ 󵄨󵄨󵄨󵄨 𝜕ai,R (n) 󵄨󵄨x(n) 1 󸀠 = 󵄨󵄨 󵄨 f [vR (n)] f [vR (n)] x̃R (n – i) ̃ 󵄨󵄨x(n)󵄨󵄨󵄨 1 + 󵄨󵄨 f [vI (n)] f 󸀠 [vI (n)] x̃I (n – i) ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n)

(B.9)

Appendix B: Iterative Formulas Derivation of Complex Blind Equalization Algorithm

233

Similarly, we obtain 󵄨 ̃ 󵄨󵄨 𝜕󵄨󵄨󵄨x(n) 󵄨󵄨 1 1 = – 󵄨󵄨 f [vR (n)] f 󸀠 [vR (n)] x̃I (n – i) + 󵄨󵄨 f [vI (n)] f 󸀠 [vI (n)] xR (n – i) (B.10) ̃ 󵄨󵄨󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 𝜕ai,I (n) 󵄨󵄨x(n) 󵄨󵄨x(n) 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕󵄨󵄨󵄨x(n) 𝜕󵄨󵄨x(n) 𝜕󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 󵄨󵄨 = 󵄨 +j 󵄨 𝜕ai (n) 𝜕ai,R (n) 𝜕ai,I (n) 1 = 󵄨󵄨 f [vR (n)] f [vR (n)] {x̃R (n – i) – jx̃I (n – i)} ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n) 1 + 󵄨󵄨 f [vI (n)] f 󸀠 [vI (n)] {x̃I (n – i) + jx̃R (n = i)} ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n) 1 = 󵄨󵄨 f [vR (n)] f 󸀠 [vR (n)] {x̃R (n – i) – jx̃I (n = i)} ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n) 1 + j 󵄨󵄨 f [vI (n)] f 󸀠 [vI (n)] {x̃R (n – i) – jx̃I (n – i)} ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n) 1 = 󵄨󵄨 {f [vR (n)] f 󸀠 [vR (n)] + jf [vI (n)] f 󸀠 [vI (n)]} x̃ ∗ (n – i) ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n)

(B.11)

Plugging eq. (B.11) into eq. (B.7), the weight iterative formula of the feedback unit is ai (n + 1) = ai (n) – 2,a K x̃ ∗ (n = i)

(B.12)

where ,a is the iterative step size of feedback unit.

(3) Weight iterative formula of the linear feedback unit The connection weight between the linear feedback unit and the output unit is bij (n), so bij (n + 1) = bij (n) – ,

𝜕J(n) 𝜕bij (n)

󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 𝜕J(n) 󵄨 ̃ 󵄨󵄨 𝜕󵄨󵄨󵄨x(n) 󵄨 ̃ 󵄨󵄨2 = 2 [󵄨󵄨󵄨x(n) +j 󵄨 ] 󵄨󵄨 [ 󵄨󵄨 – R2 ] 󵄨󵄨󵄨x(n) 𝜕bij (n) 𝜕bij,R (n) 𝜕bij,I (n) According to eqs. (4.11)–(4.13), we obtain 󵄨 ̃ 󵄨󵄨 ̃ x̃ ∗ (n)] 𝜕󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕 [x(n) = = 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 𝜕bij,R (n) 𝜕bij,R (n) 𝜕bij,R (n) 2 󵄨󵄨x(n) 2 2 1 𝜕{f [vR (n)] + f [vI (n)]} = 󵄨󵄨 ̃ 󵄨󵄨󵄨󵄨 𝜕bij,R (n) 2 󵄨󵄨x(n)

(B.13) (B.14)

234

Appendix B: Iterative Formulas Derivation of Complex Blind Equalization Algorithm

𝜕vR (n) 𝜕vI (n) 1 󸀠 󸀠 = 󵄨󵄨 󵄨󵄨 {f [vR (n)] f [vR (n)] 𝜕b (n) + f [vI (n)] f [vI (n)] 𝜕b (n) } ̃ 󵄨󵄨x(n)󵄨󵄨 ij,R ij,R 1 󸀠 = 󵄨󵄨 f [vR (n)] f [vR (n)] {x̃R (n – i)yR (n – j) – x̃I (n – i)yI (n – j)} ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n) 1 + 󵄨󵄨 f [vI (n)] f 󸀠 [vI (n)] {x̃I (n – i)yR (n – j) + x̃R (n – i)yI (n – j)} ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n)

(B.15)

Similarly, 󵄨 ̃ 󵄨󵄨 ̃ x̃ ∗ (n)] 𝜕󵄨󵄨󵄨x(n) ̃ x̃ ∗ (n) 󵄨󵄨 𝜕√x(n) 1 𝜕 [x(n) = = 󵄨󵄨 󵄨 ̃ 󵄨󵄨󵄨 𝜕bij,I (n) 𝜕bij,I (n) 𝜕bij,I (n) 2 󵄨󵄨x(n) 1 (B.16) = – 󵄨󵄨 f [vR (n)] f 󸀠 [vR (n)] {x̃R (n – i)yI (n – j) + x̃I (n – i)yR (n – j)} ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n) 1 + 󵄨󵄨 f [vI (n)] f 󸀠 [vI (n)] {x̃R (n – i)yR (n – j) – x̃I (n – i)yI (n – j)} ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n) 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 󵄨 ̃ 󵄨󵄨 𝜕󵄨󵄨x(n) 𝜕󵄨󵄨x(n) 𝜕󵄨󵄨󵄨x(n) 󵄨󵄨 󵄨󵄨 󵄨󵄨 = 󵄨 +j 󵄨 𝜕bij (n) 𝜕bij,R (n) 𝜕bij,I (n) 1 = 󵄨󵄨 f [vR (n)] f 󸀠 [vR (n)] {f [vR (n – i)] yR (n – j) – f [vI (n – i)] yI (n – j)} ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n) 1 – j 󵄨󵄨 f [vR (n)] f 󸀠 [vR (n)] {f [vR (n – i)] yI (n – j) + f [vI (n – i)] yR (n – j)} ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n) 1 󸀠 + 󵄨󵄨 󵄨󵄨 f [vI (n)] f [vI (n)] {f [vI (n – i)] yR (n – j) + f [vR (n – i)] yI (n – j)} ̃ x(n) 󵄨󵄨 󵄨󵄨 1 + j 󵄨󵄨 f [vI (n)] f 󸀠 [vI (n)] {f [vR (n – i)] yR (n – j) – f [vI (n – i)] yI (n – j)} ̃ 󵄨󵄨󵄨󵄨 󵄨󵄨x(n) 1 󸀠 󸀠 ∗ ∗ = 󵄨󵄨 󵄨󵄨 {f [vR (n)] f [vR (n)] + jf [vI (n)] f [vI (n)]} x̃ (n – i)y (n – j) ̃ x(n) 󵄨󵄨 󵄨󵄨 (B.17) Plugging eq. (B.17) into eq. (B.13), the weight iterative formula of the linear feedback unit is bij (n + 1) = bij (n) – 2,b Ky∗ (n – j) x̃ ∗ (n – j) where ,b is the iterative step size of the linear feedback unit.

(B.18)

Appendix C: Types of Fuzzy Membership Function There are many kinds of fuzzy membership functions. Commonly used fuzzy membership functions are mainly Z-type, S-type, the intermediate symmetric type, and so on.

1. Z-type Z-type is also known as partial small type, and it is suitable for membership function of small u. Because the membership function distributes in the first quadrant, the values of u are positive. The descending semi-rectangular distribution, descending semi-normal distribution, descending semi-A distribution, descending semi-Cauchy distribution, descending semi-trapezoid distribution, descending semi-mountainshaped distribution, descending semi-concave (convex) distribution, and so on belong to this class.

(1) Descending semi-trapezoid distribution The membership function is 1 { { { {b–u ,(u) = { b – a { { { {0

u≤a a