Matrix Analysis and Applications 9781108277587

This balanced and comprehensive study presents the theory, methods and applications of matrix analysis in a new theoreti

1,474 173 6MB

English Pages 760 Year 2017

Report DMCA / Copyright

DOWNLOAD FILE

Polecaj historie

Matrix Analysis and Applications
 9781108277587

Table of contents :
Contents......Page 8
Preface......Page 18
Notation......Page 22
Abbreviations......Page 32
Algorithms......Page 35
PART I MATRIX ALGEBRA......Page 38
1.1.1 Vectors and Matrices......Page 40
1.1.2 Basic Vector Calculus......Page 43
1.1.3 Basic Matrix Calculus......Page 45
1.1.5 Matrix Functions......Page 48
1.2.1 Elementary Row Operations......Page 50
1.2.2 Gauss Elimination Methods......Page 53
1.3.1 Sets......Page 57
1.3.2 Fields and Vector Spaces......Page 59
1.3.3 Linear Mapping......Page 61
1.4.1 Inner Products of Vectors......Page 64
1.4.2 Norms of Vectors......Page 65
1.4.3 Similarity Comparison Between Vectors......Page 69
1.4.4 Banach Space, Euclidean Space, Hilbert Space......Page 72
1.4.5 Inner Products and Norms of Matrices......Page 73
1.5 Random Vectors......Page 77
1.5.1 Statistical Interpretation of Random Vectors......Page 78
1.5.2 Gaussian Random Vectors......Page 81
1.6.1 Quadratic Forms......Page 84
1.6.2 Determinants......Page 86
1.6.3 Matrix Eigenvalues......Page 89
1.6.4 Matrix Trace......Page 91
1.6.5 Matrix Rank......Page 93
1.7.1 Definition and Properties of Inverse Matrices......Page 96
1.7.2 Matrix Inversion Lemma......Page 97
1.7.3 Inversion of Hermitian Matrices......Page 98
1.7.4 Left and Right Pseudo-Inverse Matrices......Page 100
1.8.1 Definition and Properties......Page 102
1.8.2 Computation of Moore–Penrose Inverse Matrix......Page 106
1.9.1 Direct Sum of Matrices......Page 108
1.9.2 Hadamard Product......Page 109
1.10.1 Kronecker Products......Page 112
1.10.2 Generalized Kronecker Products......Page 114
1.10.3 Khatri–Rao Product......Page 115
1.11.1 Vectorization and Commutation Matrix......Page 116
1.11.2 Matricization of a Vector......Page 119
1.11.3 Properties of Vectorization Operator......Page 120
1.12.1 Sparse Vectors and Sparse Representations......Page 121
1.12.2 Sparse Representation of Face Recognition......Page 123
Exercises......Page 124
2.1 Hermitian Matrices......Page 132
2.2 Idempotent Matrix......Page 133
2.3.1 Permutation Matrix and Exchange Matrix......Page 135
2.3.2 Generalized Permutation Matrix......Page 138
2.4 Orthogonal Matrix and Unitary Matrix......Page 141
2.5.2 Triangular Matrix......Page 144
2.6 Summing Vector and Centering Matrix......Page 146
2.6.1 Summing Vector......Page 147
2.6.2 Centering Matrix......Page 148
2.7.1 Vandermonde Matrix......Page 149
2.7.2 Fourier Matrix......Page 151
2.7.3 Index Vectors......Page 153
2.7.4 FFT Algorithm......Page 155
2.8 Hadamard Matrix......Page 158
2.9.1 Symmetric Toeplitz Matrix......Page 160
2.9.2 Discrete Cosine Transform of Toeplitz Matrix......Page 162
Exercises......Page 163
3.1 Jacobian Matrix and Gradient Matrix......Page 166
3.1.1 Jacobian Matrix......Page 167
3.1.2 Gradient Matrix......Page 169
3.1.3 Calculation of Partial Derivative and Gradient......Page 170
3.2.1 Calculation of Real Matrix Differential......Page 176
3.2.2 Jacobian Matrix Identification......Page 178
3.2.3 Jacobian Matrix of Real Matrix Functions......Page 184
3.3.1 Real Hessian Matrix......Page 187
3.3.2 Real Hessian Matrix Identification......Page 189
3.4.1 Holomorphic Function and Complex Partial Derivative......Page 194
3.4.2 Complex Matrix Differential......Page 198
3.4.3 Complex Gradient Matrix Identification......Page 206
3.5 Complex Hessian Matrices and Identification......Page 211
3.5.1 Complex Hessian Matrices......Page 212
3.5.2 Complex Hessian Matrix Identification......Page 214
Exercises......Page 216
PART II MATRIX ANALYSIS......Page 218
4.1 Real Gradient Analysis......Page 220
4.1.1 Stationary Points and Extreme Points......Page 221
4.1.2 Real Gradient Analysis of f(x)......Page 223
4.1.3 Real Gradient Analysis of f(X)......Page 225
4.2.1 Extreme Point of Complex Variable Function......Page 227
4.2.2 Complex Gradient Analysis......Page 232
4.3.1 Standard Constrained Optimization Problems......Page 235
4.3.2 Convex Sets and Convex Functions......Page 237
4.3.3 Convex Function Identification......Page 240
4.4.1 Gradient Method......Page 242
4.4.2 Conjugate Gradient Method......Page 247
4.4.3 Convergence Rates......Page 252
4.5.1 Lipschitz Continuous Function......Page 254
4.5.2 Nesterov Optimal Gradient Algorithms......Page 257
4.6 Nonsmooth Convex Optimization......Page 260
4.6.1 Subgradient and Subdifferential......Page 261
4.6.2 Proximal Operator......Page 265
4.6.3 Proximal Gradient Method......Page 269
4.7.1 Lagrange Multiplier Method......Page 274
4.7.2 Penalty Function Method......Page 275
4.7.3 Augmented Lagrange Multiplier Method......Page 277
4.7.4 Lagrangian Dual Method......Page 279
4.7.5 Karush–Kuhn–Tucker Conditions......Page 281
4.7.6 Alternating Direction Method of Multipliers......Page 285
4.8.1 Newton Method for Unconstrained Optimization......Page 288
4.8.2 Newton Method for Constrained Optimization......Page 291
4.9.1 Original–Dual Problems......Page 297
4.9.2 First-Order Original–Dual Interior-Point Method......Page 298
4.9.3 Second-Order Original–Dual Interior-Point Method......Page 300
Exercises......Page 305
5.1 Numerical Stability and Condition Number......Page 308
5.2.1 Singular Value Decomposition......Page 311
5.2.2 Properties of Singular Values......Page 314
5.2.3 Rank-Deficient Least Squares Solutions......Page 317
5.3.1 PSVD Problem......Page 320
5.3.2 Accurate Calculation of PSVD......Page 321
5.4 Applications of Singular Value Decomposition......Page 322
5.4.1 Static Systems......Page 323
5.4.2 Image Compression......Page 324
5.5.1 Definition and Properties......Page 326
5.5.2 Algorithms for GSVD......Page 329
5.5.3 Two Application Examples of GSVD......Page 331
5.6 Low-Rank–Sparse Matrix Decomposition......Page 333
5.6.1 Matrix Decomposition Problems......Page 334
5.6.2 Singular Value Thresholding......Page 335
5.6.3 Robust Principal Component Analysis......Page 337
5.7 Matrix Completion......Page 339
5.7.1 Matrix Completion Problems......Page 340
5.7.2 Matrix Completion Model and Incoherence......Page 342
5.7.3 Singular Value Thresholding Algorithm......Page 343
5.7.4 Fast and Accurate Matrix Completion......Page 345
Exercises......Page 349
6 Solving Matrix Equations......Page 352
6.1.1 Ordinary Least Squares Methods......Page 353
6.1.2 Properties of Least Squares Solutions......Page 354
6.1.3 Data Least Squares......Page 357
6.2.1 Tikhonov Regularization......Page 358
6.2.2 Regularized Gauss–Seidel Method......Page 361
6.3.1 TLS Problems......Page 365
6.3.2 TLS Solution......Page 366
6.3.3 Performances of TLS Solution......Page 370
6.3.4 Generalized Total Least Squares......Page 372
6.3.5 Total Least Squares Fitting......Page 374
6.3.6 Total Maximum Likelihood Method......Page 379
6.4 Constrained Total Least Squares......Page 381
6.4.1 Constrained Total Least Squares Method......Page 382
6.4.2 Harmonic Superresolution......Page 384
6.4.3 Image Restoration......Page 385
6.5 Subspace Method for Solving Blind Matrix Equations......Page 387
6.6.1 Nonnegative Matrices......Page 390
6.6.2 Nonnegativity and Sparsity Constraints......Page 392
6.6.3 Nonnegative Matrix Factorization Model......Page 393
6.6.4 Divergences and Deformed Logarithm......Page 398
6.7.1 Multiplication Algorithms......Page 403
6.7.2 Nesterov Optimal Gradient Algorithm......Page 409
6.7.3 Alternating Nonnegative Least Squares......Page 411
6.7.4 Quasi-Newton Method......Page 414
6.7.5 Sparse Nonnegative Matrix Factorization......Page 415
6.8.1 ℓ1-Norm Minimization......Page 418
6.8.2 Lasso and Robust Linear Regression......Page 421
6.8.3 Mutual Coherence and RIP Conditions......Page 424
6.8.4 Relation to Tikhonov Regularization......Page 426
6.8.5 Gradient Analysis of ℓ1-Norm Minimization......Page 427
6.9.1 Basis Pursuit Algorithms......Page 428
6.9.3 Barzilai–Borwein Gradient Projection Algorithm......Page 431
6.9.4 ADMM Algorithms for Lasso Problems......Page 434
6.9.5 LARS Algorithms for Lasso Problems......Page 435
6.9.6 Covariance Graphical Lasso Method......Page 437
6.9.7 Homotopy Algorithm......Page 439
6.9.8 Bregman Iteration Algorithms......Page 440
Exercises......Page 446
7.1.1 Eigenvalue Problem......Page 450
7.1.2 Characteristic Polynomial......Page 452
7.2.1 Eigenvalues......Page 453
7.2.2 Eigenvectors......Page 456
7.3 Similarity Reduction......Page 459
7.3.1 Similarity Transformation of Matrices......Page 460
7.3.2 Similarity Reduction of Matrices......Page 463
7.3.3 Similarity Reduction of Matrix Polynomials......Page 467
7.4.1 Smith Normal Forms......Page 471
7.4.2 Invariant Factor Method......Page 474
7.4.3 Conversion of Jordan Form and Smith Form......Page 478
7.4.4 Finding Smith Blocks from Jordan Blocks......Page 479
7.4.5 Finding Jordan Blocks from Smith Blocks......Page 480
7.5.1 Cayley–Hamilton Theorem......Page 483
7.5.2 Computation of Inverse Matrices......Page 485
7.5.3 Computation of Matrix Powers......Page 487
7.5.4 Calculation of Matrix Exponential Functions......Page 489
7.6.1 Pisarenko Harmonic Decomposition......Page 492
7.6.2 Discrete Karhunen–Loeve Transformation......Page 495
7.6.3 Principal Component Analysis......Page 498
7.7.1 Generalized Eigenvalue Decomposition......Page 500
7.7.2 Total Least Squares Method for GEVD......Page 504
7.7.3 Application of GEVD: ESPRIT......Page 505
7.7.4 Similarity Transformation in GEVD......Page 508
7.8.1 Definition and Properties of Rayleigh Quotient......Page 511
7.8.2 Rayleigh Quotient Iteration......Page 512
7.8.3 Algorithms for Rayleigh Quotient......Page 513
7.9.1 Definition and Properties......Page 515
7.9.2 Effectiveness of Class Discrimination......Page 517
7.9.3 Robust Beamforming......Page 519
7.10.1 Description of Quadratic Eigenvalue Problems......Page 521
7.10.2 Solving Quadratic Eigenvalue Problems......Page 523
7.10.3 Application Examples......Page 527
7.11.1 Joint Diagonalization Problems......Page 532
7.11.2 Orthogonal Approximate Joint Diagonalization......Page 534
7.11.3 Nonorthogonal Approximate Joint Diagonalization......Page 537
Exercises......Page 540
8.1.1 Bases of Subspaces......Page 548
8.1.2 Disjoint Subspaces and Orthogonal Complement......Page 550
8.2.1 Definitions and Properties......Page 553
8.2.2 Subspace Basis Construction......Page 557
8.2.3 SVD-Based Orthonormal Basis Construction......Page 559
8.2.4 Basis Construction of Subspaces Intersection......Page 562
8.3.1 Signal Subspace and Noise Subspace......Page 563
8.3.2 Multiple Signal Classification (MUSIC)......Page 566
8.3.3 Subspace Whitening......Page 568
8.4.1 Equivalent Subspaces......Page 569
8.4.2 Grassmann Manifold......Page 570
8.4.3 Stiefel Manifold......Page 572
8.5 Projection Approximation Subspace Tracking (PAST)......Page 573
8.5.1 Basic PAST Theory......Page 574
8.5.2 PAST Algorithms......Page 577
8.6.1 Rayleigh–Ritz Approximation......Page 579
8.6.2 Fast Subspace Decomposition Algorithm......Page 581
Exercises......Page 583
9.1 Projection and Orthogonal Projection......Page 588
9.1.1 Projection Theorem......Page 589
9.1.2 Mean Square Estimation......Page 591
9.2.1 Projector and Orthogonal Projector......Page 593
9.2.2 Projection Matrices......Page 595
9.2.3 Derivatives of Projection Matrix......Page 598
9.3.1 Updating Formulas for Projection Matrices......Page 599
9.3.2 Prediction Filters......Page 601
9.3.3 Updating of Lattice Adaptive Filter......Page 604
9.4 Oblique Projector of Full Column Rank Matrix......Page 607
9.4.1 Definition and Properties of Oblique Projectors......Page 608
9.4.2 Geometric Interpretation of Oblique Projectors......Page 612
9.4.3 Recursion of Oblique Projectors......Page 615
9.5.1 Definition and Properties......Page 616
9.5.2 Calculation of Oblique Projection......Page 618
9.5.3 Applications of Oblique Projectors......Page 620
Exercises......Page 622
PART III HIGHER-ORDER MATRIX ANALYSIS......Page 624
10.1.1 Tensors......Page 626
10.1.2 Tensor Representation......Page 629
10.2.1 Vectorization and Horizontal Unfolding......Page 634
10.2.2 Longitudinal Unfolding of Tensors......Page 638
10.3.1 Inner Product, Norm and Outer Product......Page 643
10.3.2 Mode-n Product of Tensors......Page 645
10.3.3 Rank of Tensor......Page 649
10.4 Tucker Decomposition of Tensors......Page 651
10.4.1 Tucker Decomposition (Higher-Order SVD)......Page 652
10.4.2 Third-Order SVD......Page 654
10.4.3 Alternating Least Squares Algorithms......Page 658
10.5.1 Bilinear Model......Page 662
10.5.2 Parallel Factor Analysis......Page 664
10.5.3 Uniqueness Condition......Page 672
10.5.4 Alternating Least Squares Algorithm......Page 674
10.6 Applications of Low-Rank Tensor Decomposition......Page 678
10.6.1 Multimodal Data Fusion......Page 679
10.6.2 Fusion of Multimodal Brain Images......Page 681
10.6.3 Process Monitoring......Page 683
10.6.4 Note on Other Applications......Page 685
10.7.1 Tensor–Vector Products......Page 686
10.7.2 Determinants and Eigenvalues of Tensors......Page 688
10.7.3 Generalized Tensor Eigenvalues Problems......Page 693
10.7.4 Orthogonal Decomposition of Symmetric Tensors......Page 695
10.8 Preprocessing and Postprocessing......Page 696
10.8.1 Centering and Scaling of Multi-Way Data......Page 697
10.8.2 Compression of Data Array......Page 698
10.9.1 Multiplication Algorithm......Page 701
10.9.2 ALS Algorithms......Page 704
10.10 Tensor Completion......Page 707
10.10.1 Simultaneous Tensor Decomposition and Completion......Page 708
10.10.2 Smooth PARAFAC Tensor Completion......Page 711
10.11 Software......Page 713
Exercises......Page 715
References......Page 718
Index......Page 745

Citation preview

M AT R I X A NA LY S I S A N D A P P L I C AT I O N S

This balanced and comprehensive study presents the theory, methods and applications of matrix analysis in a new theoretical framework, allowing readers to understand secondorder and higher-order matrix analysis in a completely new light. Alongside the core subjects in matrix analysis, such as singular value analysis, the solution of matrix equations and eigenanalysis, the author introduces new applications and perspectives that are unique to this book. The very topical subjects of gradient analysis and optimization play a central role here. Also included are subspace analysis, projection analysis and tensor analysis, subjects which are often neglected in other books. Having provided a solid foundation to the subject, the author goes on to place particular emphasis on the many applications matrix analysis has in science and engineering, making this book suitable for scientists, engineers and graduate students alike. X I A N - D A Z H A N G is Professor Emeritus in the Department of Automation, at Tsinghua University, Beijing. He was a Distinguished Professor at Xidian University, Xi’an, China – a post awarded by the Ministry of Education of China, and funded by the Ministry of Education of China and the Cheung Kong Scholars Programme – from 1999 to 2002. His areas of research include signal processing, pattern recognition, machine learning and related applied mathematics. He has published over 120 international journal and conference papers, and 7 books in Chinese. He taught the graduate course “Matrix Analysis and Applications” at Tsinghua University from 2004 to 2011.

M AT RI X A NA LYSIS AND A P P L I C ATIONS X I A N - DA Z H A N G Tsinghua University, Beijing

University Printing House, Cambridge CB2 8BS, United Kingdom One Liberty Plaza, 20th Floor, New York, NY 10006, USA 477 Williamstown Road, Port Melbourne, VIC 3207, Australia 4843/24, 2nd Floor, Ansari Road, Daryaganj, Delhi – 110002, India 79 Anson Road, #06–04/06, Singapore 079906 Cambridge University Press is part of the University of Cambridge. It furthers the University’s mission by disseminating knowledge in the pursuit of education, learning, and research at the highest international levels of excellence. www.cambridge.org Information on this title: www.cambridge.org/9781108417419 DOI: 10.1017/9781108277587 c Xian-Da Zhang 2017  This publication is in copyright. Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press. First published 2017 Printed in the United Kingdom by Clays, St Ives plc A catalogue record for this publication is available from the British Library. ISBN 978-1-108-41741-9 Hardback Cambridge University Press has no responsibility for the persistence or accuracy of URLs for external or third-party internet websites referred to in this publication and does not guarantee that any content on such websites is, or will remain, accurate or appropriate.

To John Zhang, Ellen Zhang and Andrew Wei

Contents

Preface Notation Abbreviations Algorithms

PART I 1

MATRIX ALGEBRA

Introduction to Matrix Algebra 1.1 Basic Concepts of Vectors and Matrices 1.1.1 Vectors and Matrices 1.1.2 Basic Vector Calculus 1.1.3 Basic Matrix Calculus 1.1.4 Linear Independence of Vectors 1.1.5 Matrix Functions 1.2 Elementary Row Operations and Applications 1.2.1 Elementary Row Operations 1.2.2 Gauss Elimination Methods 1.3 Sets, Vector Subspaces and Linear Mapping 1.3.1 Sets 1.3.2 Fields and Vector Spaces 1.3.3 Linear Mapping 1.4 Inner Products and Vector Norms 1.4.1 Inner Products of Vectors 1.4.2 Norms of Vectors 1.4.3 Similarity Comparison Between Vectors 1.4.4 Banach Space, Euclidean Space, Hilbert Space 1.4.5 Inner Products and Norms of Matrices 1.5 Random Vectors 1.5.1 Statistical Interpretation of Random Vectors 1.5.2 Gaussian Random Vectors

page xvii xxi xxxi xxxiv

1 3 3 3 6 8 11 11 13 13 16 20 20 22 24 27 27 28 32 35 36 40 41 44

viii

1.6

Performance Indexes of Matrices 1.6.1 Quadratic Forms 1.6.2 Determinants 1.6.3 Matrix Eigenvalues 1.6.4 Matrix Trace 1.6.5 Matrix Rank 1.7 Inverse Matrices and Pseudo-Inverse Matrices 1.7.1 Definition and Properties of Inverse Matrices 1.7.2 Matrix Inversion Lemma 1.7.3 Inversion of Hermitian Matrices 1.7.4 Left and Right Pseudo-Inverse Matrices 1.8 Moore–Penrose Inverse Matrices 1.8.1 Definition and Properties 1.8.2 Computation of Moore–Penrose Inverse Matrix 1.9 Direct Sum and Hadamard Product 1.9.1 Direct Sum of Matrices 1.9.2 Hadamard Product 1.10 Kronecker Products and Khatri–Rao Product 1.10.1 Kronecker Products 1.10.2 Generalized Kronecker Products 1.10.3 Khatri–Rao Product 1.11 Vectorization and Matricization 1.11.1 Vectorization and Commutation Matrix 1.11.2 Matricization of a Vector 1.11.3 Properties of Vectorization Operator 1.12 Sparse Representations 1.12.1 Sparse Vectors and Sparse Representations 1.12.2 Sparse Representation of Face Recognition Exercises 2

Special Matrices 2.1 Hermitian Matrices 2.2 Idempotent Matrix 2.3 Permutation Matrix 2.3.1 Permutation Matrix and Exchange Matrix 2.3.2 Generalized Permutation Matrix 2.4 Orthogonal Matrix and Unitary Matrix 2.5 Band Matrix and Triangular Matrix 2.5.1 Band Matrix 2.5.2 Triangular Matrix 2.6 Summing Vector and Centering Matrix 2.6.1 Summing Vector

47 47 49 52 54 56 59 59 60 61 63 65 65 69 71 71 72 75 75 77 78 79 79 82 83 84 84 86 87 95 95 96 98 98 101 104 107 107 107 109 110

ix

3

4

2.6.2 Centering Matrix 2.7 Vandermonde Matrix and Fourier Matrix 2.7.1 Vandermonde Matrix 2.7.2 Fourier Matrix 2.7.3 Index Vectors 2.7.4 FFT Algorithm 2.8 Hadamard Matrix 2.9 Toeplitz Matrix 2.9.1 Symmetric Toeplitz Matrix 2.9.2 Discrete Cosine Transform of Toeplitz Matrix Exercises

111 112 112 114 116 118 121 123 123 125 126

Matrix Differential 3.1 Jacobian Matrix and Gradient Matrix 3.1.1 Jacobian Matrix 3.1.2 Gradient Matrix 3.1.3 Calculation of Partial Derivative and Gradient 3.2 Real Matrix Differential 3.2.1 Calculation of Real Matrix Differential 3.2.2 Jacobian Matrix Identification 3.2.3 Jacobian Matrix of Real Matrix Functions 3.3 Real Hessian Matrix and Identification 3.3.1 Real Hessian Matrix 3.3.2 Real Hessian Matrix Identification 3.4 Complex Gradient Matrices 3.4.1 Holomorphic Function and Complex Partial Derivative 3.4.2 Complex Matrix Differential 3.4.3 Complex Gradient Matrix Identification 3.5 Complex Hessian Matrices and Identification 3.5.1 Complex Hessian Matrices 3.5.2 Complex Hessian Matrix Identification Exercises

129 129 130 132 133 139 139 141 147 150 150 152 157 157 161 169 174 175 177 179

PART II

181

MATRIX ANALYSIS

Gradient Analysis and Optimization 4.1 Real Gradient Analysis 4.1.1 Stationary Points and Extreme Points 4.1.2 Real Gradient Analysis of f (x) 4.1.3 Real Gradient Analysis of f (X) 4.2 Gradient Analysis of Complex Variable Function 4.2.1 Extreme Point of Complex Variable Function

183 183 184 186 188 190 190

x

5

4.2.2 Complex Gradient Analysis 4.3 Convex Sets and Convex Function Identification 4.3.1 Standard Constrained Optimization Problems 4.3.2 Convex Sets and Convex Functions 4.3.3 Convex Function Identification 4.4 Gradient Methods for Smooth Convex Optimization 4.4.1 Gradient Method 4.4.2 Conjugate Gradient Method 4.4.3 Convergence Rates 4.5 Nesterov Optimal Gradient Method 4.5.1 Lipschitz Continuous Function 4.5.2 Nesterov Optimal Gradient Algorithms 4.6 Nonsmooth Convex Optimization 4.6.1 Subgradient and Subdifferential 4.6.2 Proximal Operator 4.6.3 Proximal Gradient Method 4.7 Constrained Convex Optimization 4.7.1 Lagrange Multiplier Method 4.7.2 Penalty Function Method 4.7.3 Augmented Lagrange Multiplier Method 4.7.4 Lagrangian Dual Method 4.7.5 Karush–Kuhn–Tucker Conditions 4.7.6 Alternating Direction Method of Multipliers 4.8 Newton Methods 4.8.1 Newton Method for Unconstrained Optimization 4.8.2 Newton Method for Constrained Optimization 4.9 Original–Dual Interior-Point Method 4.9.1 Original–Dual Problems 4.9.2 First-Order Original–Dual Interior-Point Method 4.9.3 Second-Order Original–Dual Interior-Point Method Exercises

195 198 198 200 203 205 205 210 215 217 217 220 223 224 228 232 237 237 238 240 242 244 248 251 251 254 260 260 261 263 268

Singular Value Analysis 5.1 Numerical Stability and Condition Number 5.2 Singular Value Decomposition (SVD) 5.2.1 Singular Value Decomposition 5.2.2 Properties of Singular Values 5.2.3 Rank-Deficient Least Squares Solutions 5.3 Product Singular Value Decomposition (PSVD) 5.3.1 PSVD Problem 5.3.2 Accurate Calculation of PSVD 5.4 Applications of Singular Value Decomposition

271 271 274 274 277 280 283 283 284 285

xi

6

5.4.1 Static Systems 5.4.2 Image Compression 5.5 Generalized Singular Value Decomposition (GSVD) 5.5.1 Definition and Properties 5.5.2 Algorithms for GSVD 5.5.3 Two Application Examples of GSVD 5.6 Low-Rank–Sparse Matrix Decomposition 5.6.1 Matrix Decomposition Problems 5.6.2 Singular Value Thresholding 5.6.3 Robust Principal Component Analysis 5.7 Matrix Completion 5.7.1 Matrix Completion Problems 5.7.2 Matrix Completion Model and Incoherence 5.7.3 Singular Value Thresholding Algorithm 5.7.4 Fast and Accurate Matrix Completion Exercises

286 287 289 289 292 294 296 297 298 300 302 303 305 306 308 312

Solving Matrix Equations 6.1 Least Squares Method 6.1.1 Ordinary Least Squares Methods 6.1.2 Properties of Least Squares Solutions 6.1.3 Data Least Squares 6.2 Tikhonov Regularization and Gauss–Seidel Method 6.2.1 Tikhonov Regularization 6.2.2 Regularized Gauss–Seidel Method 6.3 Total Least Squares (TLS) Methods 6.3.1 TLS Problems 6.3.2 TLS Solution 6.3.3 Performances of TLS Solution 6.3.4 Generalized Total Least Squares 6.3.5 Total Least Squares Fitting 6.3.6 Total Maximum Likelihood Method 6.4 Constrained Total Least Squares 6.4.1 Constrained Total Least Squares Method 6.4.2 Harmonic Superresolution 6.4.3 Image Restoration 6.5 Subspace Method for Solving Blind Matrix Equations 6.6 Nonnegative Matrix Factorization: Optimization Theory 6.6.1 Nonnegative Matrices 6.6.2 Nonnegativity and Sparsity Constraints 6.6.3 Nonnegative Matrix Factorization Model 6.6.4 Divergences and Deformed Logarithm

315 316 316 317 320 321 321 324 328 328 329 333 335 337 342 344 345 347 348 350 353 353 355 356 361

xii

7

6.7

Nonnegative Matrix Factorization: Optimization Algorithms 6.7.1 Multiplication Algorithms 6.7.2 Nesterov Optimal Gradient Algorithm 6.7.3 Alternating Nonnegative Least Squares 6.7.4 Quasi-Newton Method 6.7.5 Sparse Nonnegative Matrix Factorization 6.8 Sparse Matrix Equation Solving: Optimization Theory 6.8.1 1 -Norm Minimization 6.8.2 Lasso and Robust Linear Regression 6.8.3 Mutual Coherence and RIP Conditions 6.8.4 Relation to Tikhonov Regularization 6.8.5 Gradient Analysis of 1 -Norm Minimization 6.9 Sparse Matrix Equation Solving: Optimization Algorithms 6.9.1 Basis Pursuit Algorithms 6.9.2 First-Order Augmented Lagrangian Algorithm 6.9.3 Barzilai–Borwein Gradient Projection Algorithm 6.9.4 ADMM Algorithms for Lasso Problems 6.9.5 LARS Algorithms for Lasso Problems 6.9.6 Covariance Graphical Lasso Method 6.9.7 Homotopy Algorithm 6.9.8 Bregman Iteration Algorithms Exercises

366 366 372 374 377 378 381 381 384 387 389 390 391 391 394 394 397 398 400 402 403 409

Eigenanalysis 7.1 Eigenvalue Problem and Characteristic Equation 7.1.1 Eigenvalue Problem 7.1.2 Characteristic Polynomial 7.2 Eigenvalues and Eigenvectors 7.2.1 Eigenvalues 7.2.2 Eigenvectors 7.3 Similarity Reduction 7.3.1 Similarity Transformation of Matrices 7.3.2 Similarity Reduction of Matrices 7.3.3 Similarity Reduction of Matrix Polynomials 7.4 Polynomial Matrices and Balanced Reduction 7.4.1 Smith Normal Forms 7.4.2 Invariant Factor Method 7.4.3 Conversion of Jordan Form and Smith Form 7.4.4 Finding Smith Blocks from Jordan Blocks 7.4.5 Finding Jordan Blocks from Smith Blocks 7.5 Cayley–Hamilton Theorem with Applications 7.5.1 Cayley–Hamilton Theorem

413 413 413 415 416 416 419 422 423 426 430 434 434 437 441 442 443 446 446

xiii

8

7.5.2 Computation of Inverse Matrices 7.5.3 Computation of Matrix Powers 7.5.4 Calculation of Matrix Exponential Functions 7.6 Application Examples of Eigenvalue Decomposition 7.6.1 Pisarenko Harmonic Decomposition 7.6.2 Discrete Karhunen–Loeve Transformation 7.6.3 Principal Component Analysis 7.7 Generalized Eigenvalue Decomposition (GEVD) 7.7.1 Generalized Eigenvalue Decomposition 7.7.2 Total Least Squares Method for GEVD 7.7.3 Application of GEVD: ESPRIT 7.7.4 Similarity Transformation in GEVD 7.8 Rayleigh Quotient 7.8.1 Definition and Properties of Rayleigh Quotient 7.8.2 Rayleigh Quotient Iteration 7.8.3 Algorithms for Rayleigh Quotient 7.9 Generalized Rayleigh Quotient 7.9.1 Definition and Properties 7.9.2 Effectiveness of Class Discrimination 7.9.3 Robust Beamforming 7.10 Quadratic Eigenvalue Problems 7.10.1 Description of Quadratic Eigenvalue Problems 7.10.2 Solving Quadratic Eigenvalue Problems 7.10.3 Application Examples 7.11 Joint Diagonalization 7.11.1 Joint Diagonalization Problems 7.11.2 Orthogonal Approximate Joint Diagonalization 7.11.3 Nonorthogonal Approximate Joint Diagonalization Exercises

448 450 452 455 455 458 461 463 463 467 468 471 474 474 475 476 478 478 480 482 484 484 486 490 495 495 497 500 503

Subspace Analysis and Tracking 8.1 General Theory of Subspaces 8.1.1 Bases of Subspaces 8.1.2 Disjoint Subspaces and Orthogonal Complement 8.2 Column Space, Row Space and Null Space 8.2.1 Definitions and Properties 8.2.2 Subspace Basis Construction 8.2.3 SVD-Based Orthonormal Basis Construction 8.2.4 Basis Construction of Subspaces Intersection 8.3 Subspace Methods 8.3.1 Signal Subspace and Noise Subspace 8.3.2 Multiple Signal Classification (MUSIC)

511 511 511 513 516 516 520 522 525 526 526 529

xiv

9

10

8.3.3 Subspace Whitening 8.4 Grassmann Manifold and Stiefel Manifold 8.4.1 Equivalent Subspaces 8.4.2 Grassmann Manifold 8.4.3 Stiefel Manifold 8.5 Projection Approximation Subspace Tracking (PAST) 8.5.1 Basic PAST Theory 8.5.2 PAST Algorithms 8.6 Fast Subspace Decomposition 8.6.1 Rayleigh–Ritz Approximation 8.6.2 Fast Subspace Decomposition Algorithm Exercises

531 532 532 533 535 536 537 540 542 542 544 546

Projection Analysis 9.1 Projection and Orthogonal Projection 9.1.1 Projection Theorem 9.1.2 Mean Square Estimation 9.2 Projectors and Projection Matrices 9.2.1 Projector and Orthogonal Projector 9.2.2 Projection Matrices 9.2.3 Derivatives of Projection Matrix 9.3 Updating of Projection Matrices 9.3.1 Updating Formulas for Projection Matrices 9.3.2 Prediction Filters 9.3.3 Updating of Lattice Adaptive Filter 9.4 Oblique Projector of Full Column Rank Matrix 9.4.1 Definition and Properties of Oblique Projectors 9.4.2 Geometric Interpretation of Oblique Projectors 9.4.3 Recursion of Oblique Projectors 9.5 Oblique Projector of Full Row Rank Matrices 9.5.1 Definition and Properties 9.5.2 Calculation of Oblique Projection 9.5.3 Applications of Oblique Projectors Exercises

551 551 552 554 556 556 558 561 562 562 564 567 570 571 575 578 579 579 581 583 585

PART III

587

HIGHER-ORDER MATRIX ANALYSIS

Tensor Analysis 10.1 Tensors and their Presentation 10.1.1 Tensors 10.1.2 Tensor Representation 10.2 Vectorization and Matricization of Tensors

589 589 589 592 597

xv

10.2.1 Vectorization and Horizontal Unfolding 10.2.2 Longitudinal Unfolding of Tensors 10.3 Basic Algebraic Operations of Tensors 10.3.1 Inner Product, Norm and Outer Product 10.3.2 Mode-n Product of Tensors 10.3.3 Rank of Tensor 10.4 Tucker Decomposition of Tensors 10.4.1 Tucker Decomposition (Higher-Order SVD) 10.4.2 Third-Order SVD 10.4.3 Alternating Least Squares Algorithms 10.5 Parallel Factor Decomposition of Tensors 10.5.1 Bilinear Model 10.5.2 Parallel Factor Analysis 10.5.3 Uniqueness Condition 10.5.4 Alternating Least Squares Algorithm 10.6 Applications of Low-Rank Tensor Decomposition 10.6.1 Multimodal Data Fusion 10.6.2 Fusion of Multimodal Brain Images 10.6.3 Process Monitoring 10.6.4 Note on Other Applications 10.7 Tensor Eigenvalue Decomposition 10.7.1 Tensor–Vector Products 10.7.2 Determinants and Eigenvalues of Tensors 10.7.3 Generalized Tensor Eigenvalues Problems 10.7.4 Orthogonal Decomposition of Symmetric Tensors 10.8 Preprocessing and Postprocessing 10.8.1 Centering and Scaling of Multi-Way Data 10.8.2 Compression of Data Array 10.9 Nonnegative Tensor Decomposition Algorithms 10.9.1 Multiplication Algorithm 10.9.2 ALS Algorithms 10.10 Tensor Completion 10.10.1 Simultaneous Tensor Decomposition and Completion 10.10.2 Smooth PARAFAC Tensor Completion 10.11 Software Exercises References Index

597 601 606 606 608 612 614 615 617 621 625 625 627 635 637 641 642 644 646 648 649 649 651 656 658 659 660 661 664 664 667 670 671 674 676 678 681 708

Preface

Linear algebra is a vast field of fundamental importance in most areas of pure (and applied) mathematics, while matrices are a key tool for the researchers, scientists, engineers and graduate students majoring in the science and engineering disciplines. From the viewpoint of applications, matrix analysis provides a powerful mathematical modeling and computational framework for posing and solving important scientific and engineering problems. It is no exaggeration to say that matrix analysis is one of the most creative and flexible mathematical tools and that it plays an irreplaceable role in physics, mechanics, signal and information processing, wireless communications, machine learning, computer vision, automatic control, system engineering, aerospace, bioinformatics, medical image processing and many other disciplines, and it effectively supports research in them all. At the same time, novel applications in these disciplines have spawned a number of new results and methods of matrix analysis, such as quadratic eigenvalue problems, joint diagonalization, sparse representation and compressed sensing, matrix completion, nonnegative matrix factorization, tensor analysis and so on. Goal of the Book The main goal of this book is to help the reader develop the skills and background needed to recognize, formulate and solve linear algebraic problems by presenting systematically the theory, methods and applications of matrix analysis. A secondary goal is to help the reader understand some recent applications, perspectives and developments in matrix analysis. Structure of the Book In order to provide a balanced and comprehensive account of the subject, this book covers the core theory and methods in matrix analysis, and places particular emphasis on its typical applications in various science and engineering disciplines. The book consists of ten chapters, spread over three parts. Part I is on matrix algebra: it contains Chapters 1 through 3 and focuses on the necessary background material. Chapter 1 is an introduction to matrix algebra that is devoted to basic matrix operations. This is followed by a description of the vecxvii

xviii

Preface

torization of matrices, the representation of vectors as matrices, i.e. matricization, and the application of sparse matrices to face recognition. Chapter 2 presents some special matrices used commonly in matrix analysis. Chapter 3 presents the matrix differential, which is an important tool in optimization. Part II is on matrix analysis: this is the heart of the book, and deals with the topics that are most frequently needed. It covers both theoretical and practical aspects and consists of six chapters, as follows. Chapter 4 is devoted to the gradient analysis of matrices, with applications in smooth and nonsmooth convex optimization, constrained convex optimization, Newton’s algorithm and the original–dual interior-point method. In Chapter 5 we describe the singular value analysis of matrices, including singular value decomposition, generalized singular value decomposition, low-rank sparse matrix decomposition and matrix completion. Researchers, scientists, engineers and graduate students from a wide variety of disciplines often have to use matrices for modeling purposes and to solve the resulting matrix equations. Chapter 6 focuses on ways to solve such equations and includes the Tikhonov regularization method, the total least squares method, the constrained total least squares method, nonnegative matrix factorization and the solution of sparse matrix equations. Chapter 7 deals with eigenvalue decomposition, matrix reduction, generalized eigenvalue decomposition, the Rayleigh quotient, the generalized Rayleigh quotient, quadratic eigenvalue problems and joint diagonalization. Chapter 8 is devoted to subspace analysis methods and subspace tracking algorithms in adaptive signal processing. Chapter 9 focuses on orthogonal and oblique projections with their applications. Part III is on higher-order matrix analysis and consists simply of Chapter 10. In it, matrix analysis is extended from the second-order case to higher orders via a presentation of the basic algebraic operations, representation as matrices, Tuckey decomposition, parallel factor decomposition, eigenvalue decomposition of tensors, nonnegative tensor decomposition and tensor completion, together with applications. Features of the Book The book introduces a novel theoretical framework for matrix analysis by dividing it into second-order matrix analysis (including gradient analysis, singular value analysis, eigenanalysis, subspace analysis and projection analysis) and higher-order matrix analysis (tensor analysis). Gradient analysis and optimization play an important role in the book. This is a very topical subject and is central to many modern applications (such as communications, signal processing, pattern recognition, machine learning, radar, big data analysis, multimodal brain image fusion etc.) though quite classical in origin. Some more contemporary topics of matrix analysis such as subspace analysis,

Preface

xix

projection analysis and tensor analysis, and which are often missing from other books, are included in our text. Particular emphasis is placed on typical applications of matrix methods in science and engineering. The 80 algorithms for which summaries are given should help readers learn how to conduct computer experiments using related matrix analysis in their studies and research. In order to make these methods easy to understand and master, this book adheres to the principle of both interpreting physics problems in terms of mathematics, and mathematical results in terms of physical ideas. Thus some typical or important matrix analysis problems are introduced by modeling a problem from physics, while some important mathematical results are explained and understood by revealing their physical meaning. Reading the Book The following diagram gives a schematic organization of this book to illustrate the chapter dependences.

Chapter 1

Chapter 2 (optional)

Chapter 3 Part I: Matrix Algebra

Chapter 4

Chapter 5

Chapter 7

Chapter 8

Chapter 9

Chapter 6 Part II: Matrix Analysis

Chapter 10 (optional) Part III: Higher-Order Matrix Analysis

Chapters 2 and 10 are optional. In particular, Chapter 10 is specifically devoted to readers involved in multi-channel or multi-way data analysis and processing. Intended Readership Linear algebra and matrix analysis are used in a very wide range of subjects including physics, statistics, computer science, economics, information science and

xx

Preface

technology (including signal and image processing, communications, automation control, system engineering and pattern recognition), artificial intelligence, bioinformatics, biomedical engineering, to name just a selection. This book is dedicated to providing individuals in those disciplines with a solid foundation of the fundamental skills needed to develop and apply linear algebra and matrix analysis methods in their work. The only background required of the reader is a good knowledge of advanced calculus, so the book will be suitable for graduate students in science and engineering. Acknowledgments The contents of this book reflect the author’s collaboration with his own graduate students Jian Li, Zi-Zhe Ding, Yong-Tao Su, Xi-Lin Li, Heng Yang, Xi-Kai Zhao, Qi Lv, Qiu-Beng Gao, Li Zhang, Jian-Jiang Ding, Lu Wu, Feng Zhu, Ling Zhang, Dong-Xia Chang, De-Guang Xie, Chun-Yu Peng, Dao-Ming Zhang, Kun Wang, Xi-Yuan Wang, Zhong Chen, Tian-Xiang Luan, Liang Zheng, Yong Zhang, Yan-Yi Rao, all at Tsinghua University, and Shun-Tian Lou, Xiao-Long Zhu, Ji-Ming Ye, Fang-Ming Han, Xiao-Jun Li, Jian-Feng Chen at Xidian University. Since 2004 we have taught graduate courses on matrix analysis and applications at Tsinghua University. Over the years I have benefited from keen interest, feedback and suggestions from many people, including my own graduate students, and students in our courses. I wish to thank Dr. Fang-Ming Han for his contribution to co-teaching and then teaching these courses, and Xi-Lin Li, Lu Wu, Dong-Xia Chang, Kun Wang, Zhong Chen, Xi-Yuan Wang and Yan-Yi Rao for their assistance with the teaching. Kun Wang, Zhong Chen, Liang Zheng and Xi-Yuan Wang kindly provided some illustrations in the book. I am grateful to the countless researchers in linear algebra, matrix analysis, information science and technology for their original contributions and to the anonymous reviewers for their critical comments and suggestions, which have greatly improved the text. I am most grateful to the Commissioning Editor, David Liu, the Content Manager, Esther Migu´eliz, and the copyeditor, Susan Parkinson, for their patience, understanding, suggestions and high-quality content management and copyediting in the course of the book’s writing and publication. This book uses some of the contents and materials of my book Matrix Analysis and Applications (Second Edition in Chinese, Tsinghua University Press, 2013). Finally, I am grateful to my wife Xiao-Ying Tang, my son Yuan-Sheng Zhang, my daughter-in-law Lin Yan, my daughter Ye-Wei Zhang, my son-in-law Wei Wei for their support and encouragement in this project.

Notation

Sets R

real numbers

R

n

real n-vectors (n × 1 real matrices)

R

m×n

real m × n matrices

R[x] R[x]

real polynomials m×n

real m × n polynomial matrices

R

I×J×K

real third-order tensors

R

I1 ×···×IN

real N th-order tensor

R+

nonnegative real numbers, nonnegative orthant

R++

positive real numbers

C

complex numbers

C

n

complex n-vectors

C

m×n

complex m × n matrices

C[x] C[x]

complex polynomials m×n

complex m × n polynomial matrices

C

I×J×K

complex third-order tensors

C

I1 ×···×IN

complex N th-order tensors

K

real or complex numbers

K

n

real or complex n-vectors

K

m×n

real or complex m × n matrices

K

I×J×K

real or complex third-order tensors

K

I1 ×···×IN

real or complex N th-order tensors

Z

integers

Z+

nonnegative integers xxi

xxii

Notation

Sets (continued) Sn×n

symmetric n × n matrices

Sn×n + Sn×n ++

symmetric positive semi-definite n × n matrices

[m,n]

S

symmetric mth-order n-dimensional tensors AI1×···×Im, I1= · · · =In

S+

[m,n]

symmetric mth-order n-dimensional nonnegative tensors



for all

x∈A

x belongs to the set A, i.e. x is an element of A

x∈ /A

x is not an element of the set A

U → V

U maps to V

U →W

U transforms to W



such that



exists

A⇒B

A implies B

A⊆B

A is a subset of B

A⊂B

A is a proper subset of B

A∪B

union of sets A and B

A∩B

intersection of sets A and B

A+B

sum set of sets A and B

A−B

set-theoretic difference of sets A and B

X \A

complement of the set A in the set X

X1 × · · · ×Xn

Cartesian product of sets X1 , . . . , Xn

L

linear manifold

Gr(n, r)

Grassmann manifold

St(n, r)

Stiefel manifold

Or

orthogonal group

S



K (A, f ) m

symmetric positive definite n × n matrices

orthogonal complement of the subspace S order-m Krylov subspace generated by A and f

Col(A)

column space of the matrix A

Ker(A)

kernel space of the matrix A

Null(A)

null space of the matrix A

nullity(A)

nullity of the matrix A

Range(A)

range space of the matrix A

Notation

Sets (continued) Row(A)

row space of the matrix A

Span(a1 , . . . , am ) span of vectors a1 , . . . , am Vectors x∗

conjugate of the vector x

x

T

transpose of the vector x

x

H

conjugate transpose (Hermitian conjugate) of the vector x

L(u)

linear transform of the vector u

x 0

0 -norm: the number of nonzero entries in the vector x

x 1

1 -norm of the vector x

x 2

Euclidean norm of the vector x

x p

p -norm or H¨older norm of the vector x

x ∗

nuclear norm of the vector x

x ∞

∞ -norm of the vector x

x, y = x y

inner product of vectors x and y

x ◦ y = xy

outer product of vectors x and y

H

H

x⊥y

orthogonality of vectors x and y

x>0

positive vector, with components xi > 0, ∀ i

x≥0

nonnegative vector, with components xi ≥ 0, ∀ i

x≥y

vector elementwise inequality xi ≥ yi , ∀ i

unvec(x)

matricization of the column vector x

unrvec(x)

row matricization of the column vector x

(m)

θi

(m)

, yi

Rayleigh–Ritz (RR) values, RR vectors

(m) (m) (θi , yi )

Ritz pair

Matrices real m × n matrix A

A ∈ Rm×n A∈C

complex m × n matrix A

m×n

A[x] ∈ R[x]

m×n

A[x] ∈ C[x]

m×n

A A

∗ T

real m × n polynomial matrix A complex m × n polynomial matrix A conjugate of A transpose of A

xxiii

xxiv

Notation

Matrices (continued) AH

conjugate transpose (Hermitian conjugate) of A

(A, B)

matrix pencil

det(A), |A|

determinant of A

tr(A)

trace of A

rank(A)

rank of A

eig(A)

eigenvalues of the Hermitian matrix A

λi (A)

ith eigenvalue of the Hermitian matrix A

λmax (A)

maximum eigenvalue(s) of the Hermitian matrix A

λmin (A)

minimum eigenvalue(s) of the Hermitian matrix A

λ(A, B)

generalized eigenvalue of the matrix pencil (A, B)

σi (A)

ith singular value of A

σmax (A)

maximum singular value(s) of A

σmin (A)

minimum singular value(s) of A

ρ(A)

spectral radius of A

A

−1

inverse of the nonsingular matrix A

A



Moore–Penrose inverse of A

A0

positive definite matrix A

A0

positive semi-definite matrix A

A≺0

negative definite matrix A

A0

negative semi-definite matrix A

A>O

positive (or elementwise positive) matrix A

A≥O

nonnegative (or elementwise nonnegative) matrix A

A≥B

matrix elementwise inequality aij ≥ bij , ∀ i, j

A 1

maximum absolute column-sum norm of A

A ∞

maximum absolute row-sum norm of A

A spec

spectrum norm of A : σmax (A)

A F

Frobenius norm of A

A ∞

max norm of A : the absolute maximum of all entries of A

A G

Mahalanobis norm of A

vec A

column vectorization of A

rvec A

row vectorization of A

off(A)

off function of A = [aij ] :

diag(A)

diagonal function of A = [aij ] :

m

n

i=1,i=j

n

j=1

i=1 |aii |

|aij |2 2

Notation

Matrices (continued) diag(A)

diagonal vector of A = [aij ] : [a11 , . . . , ann ]T

Diag(A)

diagonal matrix of A = [aij ] : Diag(a11 , . . . , ann )

A, B

inner product of A and B : (vec A)H vec B

A⊗B

Kronecker product of matrices A and B

AB

Khatri–Rao product of matrices A and B

A∗B

Hadamard product of matrices A and B

A⊕B

direct sum of matrices A and B

{A}N

matrix group consisting of matrices A1 , . . . , AN

{A}N ⊗ B

generalized Kronecker product of {A}N and B

δx, δX

perturbations of the vector x and the matrix X

cond(A)

condition number of the matrix A

In(A)

inertia of a symmetric matrix A

i+ (A)

number of positive eigenvalues of A

i− (A)

number of negative eigenvalues of A

i0 (A)

number of zero eigenvalues of A

A∼B similarity transformation ∼ A(λ) = B(λ) balanced transformation . A=B essentially equal matrices J = PAP−1

Jordan canonical form of the matrix A

dk (x)

kth determinant divisor of a polynomial matrix A(x)

σk (x)

kth invariant factor of a polynomial matrix A(x)

A(λ)

λ-matrix of the matrix A

S(λ)

Smith normal form of the λ-matrix A(λ)

Special Vectors and Special Matrices PS

projector onto the subspace S

P⊥ S

orthogonal projector onto the subspace S

EH|S

oblique projector onto the subspace H along the subspace S

ES|H

oblique projector onto the subspace S along the subspace H

1

summing vector with all entries 1

0

null or zero vector with all components 0

ei

basic vector with ei = 1 and all other entries 0

π

extracting vector with the last nonzero entry 1

xxv

xxvi

Notation

Special Vectors and Special Matrices (continued) iN

index vector : [0, 1, . . . , N − 1]T

iN,rev

bit-reversed index vector of iN

O

null or zero matrix, with all components zero

I

identity matrix

Kmn

mn × mn commutation matrix

Jn

n × n exchange matrix : Jn = [en , . . . , e1 ]

P

n × n permutation matrix : P = [ei1 , . . . , ein ], i1 , . . . , in ∈ {1, . . . , n}

G

generalized permutation matrix or g-matrix : G = PD

Cn

n × n centering matrix = In − n−1 1n 1Tn

FN

N × N Fourier matrix with entry F (i, k) = (e−j2π/N )(i−1)(k−1)

FN,rev

N × N bit-reversed Fourier matrix

Hn

Hadamard matrix : Hn HTn = HTn Hn = nIn

A

symmetric Toeplitz matrix : [a|i−j| ]ni,j=1

A

complex Toeplitz matrix: Toep[a0 , a1 , . . . , an ] with a−i = a∗i

Qn

n × n real orthogonal matrix : QQT = QT Q = I

Un

n × n unitary matrix : UUH = UH U = I

Qm×n

m × n semi-orthogonal matrix : QT Q = In or QQT = Im

Um×n

m × n para-unitary matrix : UH U = In , m > n, or UUH = Im , m < n

Sb

between-class scatter matrix

Sw

within-class scatter matrix

Tensors A ∈ KI1 ×···×IN

N th-order real or complex tensor

I, E

identity tensor

Ai::

horizontal slice matrix of A ∈ KI×J×K

A:j:

lateral slice matrix of A ∈ KI×J×K

A::k

frontal slice matrix of A ∈ KI×J×K

a:jk , ai:k , aij:

mode-1, model-2, model-3 vectors of A ∈ KI×J×K

vec A

vectorization of tensor A

unvec A A

matricization of tensor A

(JK×I)

,A

(KI×J)

,A

(IJ×K)

matricization of tensor A ∈ KI×J×K

Notation

Tensors (continued) A, B

inner product of tensors : (vec A)H vec B

A F

Frobenius norm of tensor A

A=u◦v◦w

outer product of three vectors u, v, w

X ×n A

Tucker mode-n product of X and A rank of tensor A

rank(A) [[G; U

(1)

,...,U

(N )

Tucker operator of tensor G

]]

G × 1 A ×2 B ×3 C (1)

· · · ×N U

P 

Q 

G ×1 U xijk =

R 

Tucker decomposition (third-order SVD) (N )

higher-order SVD of N th-order tensor

aip bjq ckr

CP decomposition of the third-order tensor X

p=1 q=1 r=1

Axm , Axm−1

tensor–vector product of A ∈ S[m,n] , x ∈ Cn×1

det(A)

determinant of tensor A

λi (A)

ith eigenvalue of tensor A

σ(A)

spectrum of tensor A

Functions and Derivatives def

=

defined to be equal



asymptotically equal (in scaling sense)



approximately equal (in numerical value)

f :R

m

→R

real function f (x), x ∈ Rm , f ∈ R

f : Rm×n → R

real function f (X), X ∈ Rm×n , f ∈ R

f : Cm × Cm → R

real function f (z, z∗ ), z ∈ Cm , f ∈ R

f : Cm×n ×Cm×n → R

real function f (Z, Z∗ ), Z ∈ Cm×n , f ∈ R

dom f, D

definition domain of function f

E

domain of equality constraint function

I

domain of inequality constraint function

F

feasible set

Bc (c; r), Bo (c; r)

closed, open neighborhoods of c with radius r

Bc (C; r), Bo (C; r)

closed, open neighborhoods of C with radius r





f (z, z ), f (Z, Z ) ∗

function of complex variables z, or Z ∗

df (z, z ), df (Z, Z )

complex differentials

D(p g)

distance between vectors p and g

xxvii

xxviii

Notation

Functions and Derivatives (continued) D(x, y)

dissimilarity between vectors x and y

DE (x, y)

Euclidean distance between vectors x and y

DM (x, y)

Mahalanobis distance between vectors x and y

DJg (x, y)

Bregman distance between vectors x and y

Dx , Dvec X

row partial derivative operators

Dx f (x)

row partial derivative vectors of f (x)

Dvec X f (X)

row partial derivative vectors of f (X)

DX

Jacobian operator

DX f (X)

Jacobian matrix of the function f (X)

Dz , Dvec Z

complex cogradient operator

Dz∗ , Dvec Z∗

complex conjugate cogradient operator



Dz f (z, z )

cogradient vector of complex function f (z, z∗ )

Dvec Z f (Z, Z∗ )

cogradient vector of complex function f (Z, Z∗ )

Dz∗ f (z, z∗ )

conjugate cogradient vector of f (z, z∗ )

Dvec Z∗ f (Z, Z∗ )

conjugate cogradient vector of f (Z, Z∗ )

D Z , ∇Z ∗

Jacobian, gradient matrix operator ∗

DZ f (Z, Z )

Jacobian matrices of f (Z, Z∗ )

∇Z f (Z, Z∗ )

gradient matrices of f (Z, Z∗ )

DZ∗ f (Z, Z∗ )

conjugate Jacobian matrices of f (Z, Z∗ )

∇Z∗ f (Z, Z∗ )

conjugate gradient matrices of f (Z, Z∗ )

∇x , ∇vec X

gradient vector operator

∇x f (x)

gradient vector of function f (x)

∇vec X f (X)

gradient vector of function f (X)

∇f (X)

gradient matrix of function f

∇ f

Hessian matrix of function f

2

Hx f (x) ∗

Hf (z, z )

Hessian matrix of function f full Hessian matrix of f (z, z∗ )

Hz,z , Hz,z∗ , Hz∗ ,z∗ part Hessian matrices of function f (z, z∗ ) df, ∂f

differential or subdifferential of function f

g ∈ ∂f

subgradient of function f

Δx

descent direction of function f (x)

Δxnt

Newton step of function f (x)

max f, min f

maximize, minimize function f

Notation

Functions and Derivatives (continued) max{x, y}

maximum of x and y

min{x, y}

minimum of x and y

inf

infimum

sup

supremum

Re, Im

real part, imaginary part of complex number

arg

argument of objective function or complex number

PC (y), PC y

projection operator of the vector y onto the subspace C

x

+

nonnegative vector with entry [x+ ]i = max{xi , 0}

proxh (u)

proximal operator of function h(x) to point u

proxh (U)

proximal operator of function h(X) to point U

soft(x, τ ), Sτ [x] soft thresholding operator of real variable x Probability soft(x, τ ), soft(X, τ )

soft thresholding operator of real variables x, X

Dμ (Σ)

singular value (matrix) thresholding (operation)

IC (x)

indicator function

xLS , XLS

least squares solutions to Ax = b, AX = B

xDLS , XDLS

data least squares solutions to Ax = b, AX = B

xWLS , XWLS

weighted least squares solutions to Ax = b, AX = B

xopt , Xopt

optimal solutions to Ax = b, AX = B

xTik , XTik

Tikhonov solutions to Ax = b, AX = B

xTLS , XTLS

total least squares (TLS) solutions to Ax = b, AX = B

xGTLS , XGTLS

generalized TLS solutions to Ax = b, AX = B

xML , XML

maximum likelihood solutions to Ax = b, AX = B

(α,β) DAB (P G)

alpha–beta (AB) divergence of matrices P and G

Dα (P G)

alpha-divergence of matrices P and G

Dβ (P G)

beta-divergence of matrices P and G

DKL (P G)

Kullback–Leibler divergence of matrices P and G

lnq (x)

Tsallis logarithm

expq (x)

q-exponential

ln1−α (x)

deformed logarithm

exp1−α (x)

deformed exponential

xxix

xxx

Notation

Probability (continued) sign(x)

signum function of real valued variable x

SGN(x)

signum multifunction of real valued variable x

shrink(y, α)

shrink operator

R(x)

Rayleigh quotient, generalized Rayleigh quotient

Moff

off-diagonal matrix corresponding to matrix M

z

−j

x(n)

time-shifting operation on vector x(n)

¯ E{x} = x

expectation (mean) of random vector x

E{xx }

autocorrelation matrix of random vector x

E{xy }

cross-correlation matrix of random vectors x and y

ρxy

correlation coefficient of random vectors x and y

N (c, Σ)

Gaussian random vector with mean (vector) c and

H

H

covariance (matrix) Σ CN (c, Σ)

complex Gaussian random vector with mean (vector) c and covariance (matrix) Σ

f (x1 , . . . , xm ) joint probability density function of random vector x = [x1 , . . . , xm ]T Φx (ω)

characteristic function of random vector x

Abbreviations

AB

alpha–beta

ADMM

alternating direction method of multipliers

ALS

alternating least squares

ANLS

alternating nonnegative least squares

APGL

accelerated proximal gradient line

ARNLS

alternating regularization nonnegative least squares

BBGP

Barzilai–Borwein gradient projection

BCQP

bound-constrained quadratic program

BFGS

Broyden–Fletcher–Goldfarb–Shanno

BP

basis pursuit

BPDN

basis pursuit denoising

BSS

blind source separation

CANDECOMP

canonical factor decomposition

CNMF

constrained nonnegative matrix factorization

CoSaMP

compression sampling matching pursuit

CP

CANDECOMP/PARAFAC

DCT

discrete cosine transform

DFT

discrete Fourier transform

DLS

data least squares

DOA

direction of arrival

EEG

electroencephalography

EM

expectation-maximization

EMML

expectation-maximization maximum likelihood

ESPRIT

estimating signal parameters via rotational invariance technique

xxxi

xxxii

Abbreviations

EVD

eigenvalue decomposition

FAJD

fast approximate joint diagonalization

FAL

first-order augmented Lagrangian

FFT

fast Fourier transform

FISTA

fast iterative soft thresholding algorithm

GEAP

generalized eigenproblem adaptive power

GEVD

generalized eigenvalue decomposition

GSVD

generalized singular value decomposition

GTLS

generalized total least squares

HBM

heavy ball method

HOOI

higher-order orthogonal iteration

HOSVD

higher-order singular value decomposition

ICA

independent component analysis

IDFT

inverse discrete Fourier transform

iid

independent and identically distributed

inf

infimum

KKT

Karush–Kuhn–Tucker

KL

Kullback–Leibler

LARS

least angle regressive

Lasso

least absolute shrinkage and selection operator

LDA

linear discriminant analysis

LMV

Lathauwer–Moor–Vanderwalle

LP

linear programming

LS

least squares

LSI

latent semantic indexing

max

maximize, maximum

MCA

minor component analysis

MIMO

multiple-input–multiple-output

min

minimize, minimum

ML

maximum likelihood

MP

matching pursuit

MPCA

multilinear principal component analysis

MUSIC

multiple signal classification

Abbreviations

NeNMF

Nesterov nonnegative matrix factorization

NMF

nonnegative matrix factorization

NTD

nonnegative tensor decomposition

OGM

optimal gradient method

OMP

orthogonal matching pursuit

PARAFAC parallel factor decomposition PAST

projection approximation subspace tracking

PASTd

projection approximation subspace tracking via deflation

PCA

principal component analysis

PCG

preconditioned conjugate gradient

PCP

principal component pursuit

pdf

positive definite

PMF

positive matrix factorization

psdf

positive semi-definite

PSF

point-spread function

PSVD

product singular value decomposition

RIC

restricted isometry constant

RIP

restricted isometry property

ROMP

regularization orthogonal matching pursuit

RR

Rayleigh–Ritz

QCLP

quadratically constrained linear programming

QEP

quadratic eigenvalue problem

QP

quadratic programming

QV

quadratic variation

sign

signum

SPC

smooth PARAFAC tensor completion

StOMP

stagewise orthogonal matching pursuit

sup

supremum

SVD

singular value decomposition

SVT

singular value thresholding

TLS

total least squares

TV

total variation

UPCA

unfold principal component analysis

VQ

vector quantization

xxxiii

Algorithms

1.1

Reduced row echelon form

1.2

Solving m × n matrix equations Ax = b

1.3

Equation solving method 1

1.4

Equation solving method 2

1.5

Full-rank decomposition

1.6

Column recursive method

1.7

Trace method

2.1

Fast DCT algorithm for Toeplitz matrix

4.1

Gradient descent algorithm and its variants

4.2

Conjugate gradient algorithm

4.3

Biconjugate gradient algorithm

4.4

PCG algorithm via preprocessor

4.5

PCG algorithm without preprocessor

4.6

Nesterov (first) optimal gradient algorithm

4.7

Nesterov algorithm with adaptive convexity parameter

4.8

Nesterov (third) optimal gradient algorithm

4.9

FISTA algorithm with fixed step

4.10

Newton algorithm via backtracking line search

4.11

Feasible start Newton algorithm

4.12

Infeasible start Newton algorithm

4.13

Feasible start complex Newton algorithm

4.14

Feasible start original–dual interior-point algorithm

4.15

Feasible point algorithm

5.1

PSVD(B,C)

5.2

PSVD of BT SC xxxiv

Algorithms

5.3

GSVD algorithm 1

5.4

GSVD algorithm 2

5.5

Tangent algorithm for GSVD

5.6

Robust PCA via accelerated proximal gradient

5.7

Singular value thresholding for matrix completion

5.8

Truncated nuclear norm minimization via APGL search

5.9

Truncated nuclear norm minimization via ADMM

6.1

TLS algorithm for minimum norm solution

6.2

SVD–TLS algorithm

6.3

TLS algorithm for fitting m-dimension syperplane

6.4

Total maximum likelihood algorithm

6.5

Solving blind matrix equation X = Aθ B

6.6

NeNMF algorithm

6.7

Optimal gradient method OGM(Ak , S)

6.8

Orthogonal matching pursuit algorithm

6.9

Subspace pursuit algorithm

6.10 6.11

FAL ({λk , k , τk }k∈Z+ , η)   APG pk , fk , L, Fk , xk−1 , hk , APGSTOP

6.12

GPSR–BB algorithm

6.13

LARS algorithm with Lasso modification

6.14

Coordinate descent algorithm

6.15

Homotopy algorithm

6.16

Linearized Bregman iterative algorithm

7.1

Similarity reduction of matrix A

7.2

Calculation of matrix powers

7.3

Lanczos algorithm for GEVD

7.4

Tangent algorithm for computing the GEVD

7.5

GEVD algorithm for singular matrix B

7.6

Basic ESPRIT algorithm 1

7.7

TLS–ESPRIT algorithm

7.8

Basic ESPRIT algorithm 2

7.9

Rayleigh quotient iteration algorithm

7.10

Rayleigh quotient iteration for general matrix

7.11

Linearized algorithm for QEP

xxxv

xxxvi

Algorithms

7.12

Fast approximate joint diagonalization algorithm

8.1

PAST algorithm

8.2

PAST via deflation (PASTd) algorithm

8.3

Bi-Lanczos iteration

8.4

Fast subspace decomposition algorithm

10.1

ALS algorithm for Tucker decomposition

10.2

HOSVD (X , R1 , . . . , RN )

10.3

HOOI (X , R1 , . . . , RN )

10.4

CP–ALS algorithm via Kiers horizontal unfolding

10.5

CP–ALS algorithm via LMV longitudinal unfolding

10.6

CP–ALS (X , R)

10.7

Regularized ALS algorithm CP-RALS (X , R, N, λ)

10.8

Generalized eigenproblem adaptive power method

10.9

Z-eigenpair adaptive power method

10.10

Tensor power method

10.11

Data array compression algorithm

10.12

Multiplication algorithm for NCP decomposition

10.13

ALS algorithm for nonnegative Tucker decomposition

10.14

CP–NALS (X , R)

10.15

Simultaneous tensor decomposition and completion

10.16

SPC algorithm

PART I MATRIX ALGEBRA

1 Introduction to Matrix Algebra

In science and engineering, we often encounter the problem of solving a system of linear equations. Matrices provide the most basic and useful mathematical tool for describing and solving such systems. Matrices not only have many basic mathematics operations (such as transposition, inner product, outer product, inverse, generalized inverse etc.) but also have a variety of important scalar functions (e.g., a norm, a quadratic form, a determinant, eigenvalues, rank and trace etc.). There are also special matrix operations, such as the direct sum, direct product, Hadamard product, Kronecker product, vectorization, etc. In this chapter, we begin our introduction to matrix algebra by relating matrices to the problem of solving systems of linear equations.

1.1 Basic Concepts of Vectors and Matrices First we introduce the basic concepts of and notation for vectors and matrices.

1.1.1 Vectors and Matrices Let R (or C) denote the set of real (or complex) numbers. An m-dimensional column vector is defined as ⎡ ⎤ x1 ⎢ .. ⎥ x = ⎣ . ⎦. (1.1.1) xm If the ith component xi is a real number, i.e., xi ∈ R, for all i = 1, . . . , m, then x is an m-dimensional real vector and is denoted x ∈ Rm×1 or simply x ∈ Rm . Similarly, if xi ∈ C for some i, then x is known as an m-dimensional complex vector and is denoted x ∈ Cm . Here, Rm and Cm represent the sets of all real and complex m-dimensional column vectors, respectively. An m-dimensional row vector x = [x1 , . . . , xm ] is represented as x ∈ R1×m or 3

4

Introduction to Matrix Algebra

x ∈ C1×m . To save space, an m-dimensional column vector is usually written as the transposed form of a row vector, denoted x = [x1 , . . . , xm ]T . An m × n matrix is expressed as ⎡ ⎤ a11 · · · a1n ⎢ .. ⎥ = [a ]m,n .. A = ⎣ ... (1.1.2) ij i=1,j=1 . . . ⎦ am1

···

amn

The matrix A with (i, j)th real entry aij ∈ R is called an m × n real matrix, and denoted by A ∈ Rm×n . Similarly, A ∈ Cm×n is an m × n complex matrix. An m×n matrix can be represented as A = [a1 , . . . , an ] where its column vectors are aj = [a1j , . . . , amj ]T , j = 1, . . . , n. The system of linear equations ⎫ a11 x1 + a12 x2 + · · · + a1n xn = b1 , ⎪ ⎪ ⎪ ⎪ ⎬ a21 x1 + a22 x2 + · · · + a2n xn = b2 , ⎪ (1.1.3) .. ⎪ ⎪ . ⎪ ⎪ ⎪ ⎭ am1 x1 + am2 x2 + · · · + amn xn = bm can be simply rewritten using vector and matrix symbols as a matrix equation Ax = b, where



a11 ⎢ .. A=⎣ . am1

··· .. . ···

⎤ a1n .. ⎥ , . ⎦ amn

(1.1.4)

⎤ x1 ⎢ ⎥ x = ⎣ ... ⎦ , ⎡

xn

⎤ b1 ⎢ ⎥ b = ⎣ ... ⎦ . ⎡

(1.1.5)

bm

In modeling physical problems, the matrix A is usually the symbolic representation of a physical system (e.g., a linear system, a filter, or a wireless communication channel). There are three different types of vector in science and engineering [232]: (1) Physical vector Its elements are physical quantities with magnitude and direction, such as a displacement vector, a velocity vector, an acceleration vector and so forth. (2) Geometric vector A directed line segment or arrow is usually used to visualize a physical vector. Such a representation is called a geometric vector. For exam−−→ ple, v = AB represents the directed line segment with initial point A and the terminal point B. (3) Algebraic vector A geometric vector can be represented in algebraic form. For −−→ a geometric vector v = AB on a plane, if its initial point is A = (a1 , a2 ) −−→ and its terminal point is B = (b1 , b2 ), then the geometric vector v = AB can

1.1 Basic Concepts of Vectors and Matrices 5   b −a be represented in an algebraic form v = b1 − a1 . Such a geometric vector 2 2

described in algebraic form is known as an algebraic vector. Physical vectors are those often encountered in practical applications, while geometric vectors and algebraic vectors are respectively the visual representation and the algebraic form of physical vectors. Algebraic vectors provide a computational tool for physical vectors. Depending on the different types of element value, algebraic vectors can be divided into the following three types: (1) Constant vector Its entries are real constant numbers or complex constant numbers, e.g., a = [1, 5, 4]T . (2) Function vector It uses functions as entries, e.g., x = [x1 , . . . , xn ]T . (3) Random vector Its entries are random variables or signals, e.g., x(n) = [x1 (n), . . . , xm (n)]T where x1 (n), . . . , xm (n) are m random variables or random signals. Figure 1.1 summarizes the classification of vectors. ⎧ ⎪ Physical vectors ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ Geometric vectors ⎪ ⎪ ⎨ ⎧ Vectors ⎪ Constant vectors ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ Algebraic vectors Function vectors ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ ⎩ Random vectors Figure 1.1 Classification of vectors.

Now we turn to matrices. An m × n matrix A is called a square matrix if m = n, a broad matrix for m < n, and a tall matrix for m > n. The main diagonal of an n × n matrix A = [aij ] is the segment connecting the top left to the bottom right corner. The entries located on the main diagonal, a11 , a22 , . . . , ann , are known as the (main) diagonal elements. An n × n matrix A = [aij ] is called a diagonal matrix if all entries off the main diagonal are zero; it is then denoted by D = Diag(d11 , . . . , dnn ).

(1.1.6)

In particular, a diagonal matrix I = Diag(1, . . . , 1) is called an identity matrix, and O = Diag(0, . . . , 0) is known as a zero matrix. A vector all of whose components are equal to zero is called a zero vector and is denoted as 0 = [0, . . . , 0]T . An n × 1 vector x = [x1 , . . . , xn ]T with only one nonzero entry xi = 1 constitutes

6

Introduction to Matrix Algebra

a basis vector, denoted ei ; e.g., ⎡ ⎤ ⎡ ⎤ 1 0 ⎢0⎥ ⎢1⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ e1 = ⎢0⎥ , e2 = ⎢0⎥ , ⎢.⎥ ⎢.⎥ ⎣ .. ⎦ ⎣ .. ⎦ 0 0

...,

⎡ ⎤ 0 ⎢0⎥ ⎢ ⎥ ⎢ ⎥ en = ⎢0⎥ . ⎢.⎥ ⎣ .. ⎦ 1

(1.1.7)

Clearly, an n × n identity matrix I can be represented as I = [e1 , . . . , en ] using basis vectors. In this book, we use often the following matrix symbols. A(i, :) means the ith row of A. A(:, j) means the jth column of A. A(p : q, r : s) means the (q − p + 1) × (s − r + 1) submatrix consisting of the pth row to the qth row and the rth column to the sth column of A. For example, ⎡ ⎤ a32 a33 a34 ⎢a42 a43 a44 ⎥ ⎥. A(3 : 6, 2 : 4) = ⎢ ⎣a a a ⎦ 52

53

54

a62

a63

a64

A matrix A is an m × n block matrix if it can ⎡ A11 A12 ⎢ A21 A22 ⎢ A = [Aij ] = ⎢ . .. ⎣ .. . Am1 Am2

be represented in the form ⎤ · · · A1n · · · A2n ⎥ ⎥ .. ⎥ , .. . . ⎦ ···

Amn

where the Aij are matrices. The notation [Aij ] refers to a matrix consisting of block matrices.

1.1.2 Basic Vector Calculus Basic vector calculus requires vector addition, vector multiplication by a scalar and vector products. The vector addition of u = [u1 , . . . , un ]T and v = [v1 , . . . , vn ]T is defined as u + v = [u1 + v1 , . . . , un + vn ]T .

(1.1.8)

Vector addition has the following two main properties: • Commutative law u + v = v + u. • Associative law (u + v) ± w = u + (v ± w) = (u ± w) + v. The vector multiplication of an n × 1 vector u by a scalar α is defined as αu = [αu1 , . . . , αun ]T .

(1.1.9)

1.1 Basic Concepts of Vectors and Matrices

7

The basic property of vector multiplication by a scalar is that it obeys the distributive law: α(u + v) = αu + αv.

(1.1.10)

The inner product (or dot product or scalar product) of two real or complex n × 1 vectors u = [u1 , . . . , un ]T and v = [v1 , . . . , vn ]T , denoted u, v, is defined as the real number n  T u i vi , (1.1.11) u, v = u v = u1 v1 + · · · + un vn = i=1

or the complex number u, v = uH v = u∗1 v1 + · · · + u∗n vn =

n 

u∗i vi ,

(1.1.12)

i=1

where uH is the complex conjugate transpose, or Hermitian conjugate, of u. The inner product of two vectors u, v = uH v has several important applications; for example, it may be used to measure the size (or length) of a vector, the distance between vectors, the neighborhood of a vector and so on. We will present these applications later. The outer product (or cross product) of an m × 1 real vector and an n × 1 real vector, denoted u ◦ v, is defined as the m × n real matrix ⎤ ⎡ u1 v1 · · · u1 vn ⎢ .. .. ⎥ ; u ◦ v = uvT = ⎣ ... (1.1.13) . . ⎦ u m v1

···

um v n

if u and v are complex then the outer product is the m × n complex matrix ⎤ ⎡ u1 v1∗ · · · u1 vn∗ ⎢ .. .. ⎥ . u ◦ v = uvH = ⎣ ... (1.1.14) . . ⎦ um v1∗

···

um vn∗

In signal processing, wireless communications, pattern recognition, etc., for two m×1 data vectors x(t) = [x1 (t), . . . , xm (t)]T and y(t) = [y1 (t), . . . , ym (t)]T , the m× m autocorrelation matrix is given by Rxx = E{x(t) ◦ x(t)} = E{x(t)xH (t)} and the m × m cross-correlation matrix is given by Rxy = E{x(t) ◦ y(t)} = E{x(t)yH (t)}, where E is the expectation operator. Given the sample data xi (t), yi (t), i = 1, . . . , m, ˆ xx and the sample cross-correlation t = 1, . . . , N, the sample autocorrelation matrix R ˆ xy can be respectively estimated by matrix R N  ˆ xx = 1 R x(t)xH (t), N t=1

N  ˆ xy = 1 R x(t)yH (t). N t=1

(1.1.15)

8

Introduction to Matrix Algebra

1.1.3 Basic Matrix Calculus Basic matrix calculus requires the matrix transpose, conjugate, conjugate transpose, addition and multiplication. DEFINITION 1.1 If A = [aij ] is an m × n matrix, then its transpose AT is an n × m matrix with the (i, j)th entry [AT ]ij = aji . The conjugate of A is represented as A∗ and is an m × n matrix with (i, j)th entry [A∗ ]ij = a∗ij , while the conjugate or Hermitian transpose of A, denoted AH ∈ Cn×m , is defined as ⎡ ∗ ⎤ a11 a∗21 · · · a∗m1 ⎢ ∗ ⎥ ⎢ a12 a∗22 · · · a∗m2 ⎥ H ∗ T ⎢ ⎥ (1.1.16) A = (A ) = ⎢ . .. .. ⎥ . .. ⎣ .. . . . ⎦ a∗1n a∗2n · · · a∗mn DEFINITION 1.2 An n × n real (complex) matrix satisfying AT = A (AH = A) is called a symmetric matrix (Hermitian matrix). There are the following relationships between the transpose and conjugate transpose of a matrix: AH = (A∗ )T = (AT )∗ . For an m × n block matrix A = [Aij ], its n × m block matrix: ⎡ H A11 AH 21 ⎢ H ⎢ A12 AH 22 AH = ⎢ .. ⎢ .. ⎣ . . H A1n AH 2n

(1.1.17)

conjugate transpose AH = [AH ji ] is an ···

AH m1

··· .. . ···

AH mn



⎥ AH m2 ⎥ ⎥ .. ⎥ . . ⎦

The simplest algebraic operations with matrices are the addition of two matrices and the multiplication of a matrix by a scalar. DEFINITION 1.3 Given two m × n matrices A = [aij ] and B = [bij ], matrix addition A + B is defined by [A + B]ij = aij + bij . Similarly, matrix subtraction A − B is defined as [A − B]ij = aij − bij . By using this definition, it is easy to verify that the addition and subtraction of two matrices obey the following rules: • Commutative law A + B = B + A. • Associative law (A + B) ± C = A + (B ± C) = (A ± C) + B. DEFINITION 1.4 Let A = [aij ] be an m × n matrix, and α be a scalar. The product αA is an m × n matrix and is defined as [αA]ij = αaij .

1.1 Basic Concepts of Vectors and Matrices

9

DEFINITION 1.5 Consider an m × n matrix A = [aij ] and an r × 1 vector x = [x1 , . . . , xr ]T . The product Ax exists only when n = r and is an m × 1 vector whose entries are given by [Ax]i =

n 

aij xj ,

i = 1, . . . , m.

j=1

DEFINITION 1.6 The matrix product of an m × n matrix A = [aij ] and an r × s matrix B = [bij ], denoted AB, exists only when n = r and is an m × s matrix with entries n  [AB]ij = aik bkj , i = 1, . . . , m; j = 1, . . . , s. k=1

THEOREM 1.1

The matrix product obeys the following rules of operation:

(a) Associative law of multiplication If A ∈ Cm×n , B ∈ Cn×p and C ∈ Cp×q , then A(BC) = (AB)C. (b) Left distributive law of multiplication For two m × n matrices A and B, if C is an n × p matrix then (A ± B)C = AC ± BC. (c) Right distributive law of multiplication If A is an m × n matrix, while B and C are two n × p matrices, then A(B ± C) = AB ± AC. (d) If α is a scalar and A and B are two m×n matrices then α(A+B) = αA+αB. Proof We will prove only (a) and (b) here, while the proofs of (c) and (d) are left to the reader as an exercise. (a) Let Am×n = [aij ], Bn×p = [bij ], Cp×q = [cij ], then  p  n n    aik (BC)kj = aik bkl clj [A(BC)]ij = =

k=1 p  n 

k=1

(aik bkl )clj =

l=1 k=1

p 

l=1

[AB]il clj = [(AB)C]ij

l=1

which means that A(BC) = (AB)C. (b) From the rule for matrix multiplication it is known that [AC]ij =

n 

aik ckj ,

[BC]ij =

k=1

n 

bik ckj .

k=1

Then, according to the matrix addition rule, we have [AC + BC]ij = [AC]ij + [BC]ij =

n  k=1

This gives (A + B)C = AC + BC.

(aik + bik )ckj = [(A + B)C]ij .

10

Introduction to Matrix Algebra

Generally speaking, the product of two matrices does not satisfy the commutative law, namely AB = BA. Another important operation on a square matrix is that of finding its inverse. Put x = [x1 , . . . , xn ]T and y = [y1 , . . . , yn ]T . The matrix–vector product Ax = y can be regarded as a linear transform of the vector x, where the n × n matrix A is called the linear transform matrix. Let A−1 denote the linear inverse transform of the vector y onto x. If A−1 exists then one has x = A−1 y.

(1.1.18)

This equation can be viewed as the result of using A−1 to premultiply the original linear transform Ax = y, giving A−1 Ax = A−1 y = x, which means that the linear inverse transform matrix A−1 must satisfy A−1 A = I. Furthermore, x = A−1 y should be invertible as well. In other words, after premultiplying x = A−1 y by A, we get Ax = AA−1 y, which should be consistent with the original linear transform Ax = y. This means that A−1 must also satisfy AA−1 = I. On the basis of the discussion above, the inverse matrix can be defined as follows. DEFINITION 1.7 Let A be an n × n matrix. The matrix A is said to be invertible if there is an n × n matrix A−1 such that AA−1 = A−1 A = I, and A−1 is referred to as the inverse matrix of A. The following are properties of the conjugate, transpose, conjugate transpose and inverse matrices. 1. The matrix conjugate, transpose and conjugate transpose satisfy the distributive law: (A + B)∗ = A∗ + B∗ ,

(A + B)T = AT + BT ,

(A + B)H = AH + BH .

2. The transpose, conjugate transpose and inverse matrix of product of two matrices satisfy the following relationship: (AB)T = BT AT ,

(AB)H = BH AH ,

(AB)−1 = B−1 A−1

in which both A and B are assumed to be invertible. 3. Each of the symbols for the conjugate, transpose and conjugate transpose can be exchanged with the symbol for the inverse: (A∗ )−1 = (A−1 )∗ ,

(AT )−1 = (A−1 )T ,

(AH )−1 = (A−1 )H .

The notations A−∗ = (A−1 )∗ , A−T = (A−1 )T and A−H = (A−1 )H are sometimes used. 4. For any m × n matrix A, the n × n matrix B = AH A and the m × m matrix C = AAH are Hermitian matrices.

1.1 Basic Concepts of Vectors and Matrices

11

1.1.4 Linear Independence of Vectors Consider the system of linear equations (1.1.3). It can be written as the matrix equation Ax = b. Denoting A = [a1 , . . . , an ], the m equations of (1.1.3) can be written as x1 a1 + · · · + xn an = b. This is called a linear combination of the column vectors a1 , . . . , an . DEFINITION 1.8 A set of n vectors, denoted {u1 , . . . , un }, is said to be linearly independent if the matrix equation c 1 u1 + · · · + c n u n = 0

(1.1.19)

has only zero solutions c1 = · · · = cn = 0. If the above equation may hold for a set of coefficients that are not all zero then the n vectors u1 , . . . , un are said to be linearly dependent. An n × n matrix A is nonsingular if and only if the matrix equation Ax = 0 has only the zero solution x = 0. If Ax = 0 exists for any nonzero solution x = 0 then the matrix A is singular. For an n × n matrix A = [a1 , . . . , an ], the matrix equation Ax = 0 is equivalent to a1 x1 + · · · + an xn = 0.

(1.1.20)

From the above definition it follows that the matrix equation Ax = 0 has a zero solution vector only, i.e., the matrix A is nonsingular, if and only if the column vectors a1 , . . . , an of A are linearly independent. Because of importance of this result, it is described in a theorem below. THEOREM 1.2 An n × n matrix A = [a1 , . . . , an ] is nonsingular if and only if its n column vectors a1 , . . . , an are linearly independent. To summarize the above discussions, the nonsingularity of an n × n matrix A can be determined in any of the following three ways: (1) its column vectors are linearly independent; (2) for the matrix equation Ax = b there exists a unique nonzero solution; (3) the matrix equation Ax = 0 has only a zero solution.

1.1.5 Matrix Functions The following are five common matrix functions:

12

Introduction to Matrix Algebra

1. Triangle matrix function sin A =

∞  1 (−1)n A2n+1 1 = A − A3 + A5 − · · · (2n + 1)! 3! 5! n=0

(1.1.21)

cos A =

∞  (−1)n A2n 1 1 = I − A2 + A 4 − · · · . (2n)! 2! 4! n=0

(1.1.22)

2. Logarithm matrix function ∞  (−1)n−1 n 1 1 ln(I + A) = A = A − A2 + A3 − · · · . n 2 3 n=1

(1.1.23)

3. Exponential matrix function [311], [179] eA = e−A =

∞  1 n 1 1 A = I + A + A2 + A 3 + · · · n! 2 3! n=0

(1.1.24)

∞  1 1 1 (−1)n An = I − A + A2 − A3 + · · · n! 2 3! n=0

(1.1.25)

1 1 eAt = I + At + A2 t2 + A3 t3 + · · · . 2 3!

(1.1.26)

4. Matrix derivative If the entries aij of the matrix A are the functions of a parameter t then the derivative of the matrix A is defined as follows: ⎡

da11 ⎢ dt ⎢ ⎢ da21 ⎢ dA ˙ = A=⎢ ⎢ dt dt ⎢ .. ⎢ . ⎣ da m1 dt

da12 dt da22 dt .. . dam2 dt

··· ··· .. . ···

⎤ da1n dt ⎥ ⎥ da2n ⎥ ⎥ . dt ⎥ .. ⎥ ⎥ . ⎥ damn ⎦ dt

(1.1.27)

• The derivative of a matrix exponential function is given by deAt = AeAt = eAt A. dt

(1.1.28)

• The derivative of matrix product is given by dA dB d (AB) = B+A , dt dt dt where A and B are matrix functions of the variable t.

(1.1.29)

1.2 Elementary Row Operations and Applications

5. Matrix integral

⎡ 

⎢ ⎢ Adt = ⎢ ⎢ ⎣ 

a11 dt a21 dt .. . am1 dt

 

a12 dt

···

··· .. .  am2 dt · · · a22 dt .. .

  

a1n dt

13



⎥ a2n dt ⎥ ⎥. ⎥ .. ⎦ .

(1.1.30)

amn dt

1.2 Elementary Row Operations and Applications Simple operations related to the rows of a matrix are referred to as elementary row operations; they can efficiently solve matrix equations, find the inverse of a matrix, construct the basis vectors of a vector space and so on.

1.2.1 Elementary Row Operations When solving an m × n system of linear equations, it is useful to reduce the number of equations. The principle of reduction processes is to keep the solutions of the system of equations unchanged. DEFINITION 1.9 Two systems of linear equations in n unknowns are said to be equivalent systems of equations if they have the same sets of solutions. To transform a given m × n matrix equation Am×n xn = bm into an equivalent matrix equation, a simple and efficient way is to apply successive elementary operations on the given matrix equation. DEFINITION 1.10 The following three types of operation on the rows of a system of linear equations are called elementary row operations. Type I Interchange any two equations, say the pth and qth equations; this is denoted by Rp ↔ Rq . Type II Multiply the pth equation by a nonzero number α; this is denoted by αRp → Rp . Type III Add β times the pth equation to the qth equation; this is denoted by βRp + Rq → Rq . Clearly, any type of elementary row operation does not change the solution of a system of linear equations, so after elementary row operations, the reduced system of linear equations and the original system of linear equations are equivalent. As a matter of fact, any elementary operation on an m × n system of equations Ax = b is equivalent to the same type of elementary operation on the augmented matrix B = [A, b], where the column vector b is written alongside A. Hence, performing elementary row operations on a system of linear equations Ax = b is in practice implemented by using the same elementary row operations on the augmented matrix B = [A, b].

14

Introduction to Matrix Algebra

The discussions above show that if, after a sequence of elementary row operations, the augmented matrix Bm×(n+1) becomes another simpler matrix Cm×(n+1) then two matrices are row equivalent. For the convenience of solving a system of linear equations, the final row equivalent matrix should be of echelon form (see below). The leftmost nonzero entry of a nonzero row is called the leading entry of the row. If the leading entry is equal to 1 then it is said to be the leading-1 entry. A matrix is said to be an echelon matrix if it has the following

DEFINITION 1.11 forms:

(1) all rows with all entries zero are located the bottom of the matrix; (2) the leading entry of each nonzero row appears always to the right of the leading entry of the nonzero row above; (3) all entries below the leading entry of the same column are equal to zero. The following are some examples ⎡ ⎤ ⎡ 2 ∗ ∗ 1 ∗ ⎢0 5 ∗⎥ ⎢0 3 ⎥ ⎢ A=⎢ ⎣0 0 3⎦ , A = ⎣0 0 0

0

0

of echelon matrices: ⎤ ⎡ ⎤ ⎡ ∗ 3 ∗ ∗ 0 ⎢0 0 5⎥ ⎢0 ∗⎥ ⎥, A = ⎢ ⎥ ⎢ ⎣0 0 0⎦ , A = ⎣0 0⎦

0 0 0

0

0

0

0

1 0 0 0

⎤ ∗ 9⎥ ⎥, 0⎦ 0

where ∗ indicates that the entry can be any value. DEFINITION 1.12 [232] An echelon matrix B is said to be a reduced row-echelon form (RREF) matrix if the leading entry of each nonzero row is a leading-1 entry, and each leading-1 entry is the only nonzero entry in the column in which it is located. THEOREM 1.3 Any m × n matrix A is row equivalent to one and only one matrix in reduced row-echelon form. Proof

See [285, Appendix A].

Given an m × n matrix B, Algorithm 1.1 transforms B to a reduced row-echelon form by performing suitable elementary operations. DEFINITION 1.13 [285, p. 15] A pivot position of an m × n matrix A is the position of some leading entry of its echelon form. Each column containing a pivot position is called a pivot column of the matrix A. The following examples show how to perform elementary operations for transforming a matrix to its row-echelon form and reduced row-echelon form, and how to determine the pivot columns of the original matrix.

1.2 Elementary Row Operations and Applications Algorithm 1.1

15

Reduced row-echelon form [229]

input: B ∈ Rm×n . 1. Select the row that has the first nonzero entry, say Ri . If this happens to be the first row, then go to the next step. Otherwise, perform a Type-I elementary operation R1 ↔ Ri such that the first entry of the new first row is nonzero. 2. Make a Type-II elementary operation αR1 → R1 to get a leading-1 entry for the first row. 3. Perform a Type-III elementary operation αRi + αR1 → R1 , i > 1, to make all entries below the leading entry 1 of the first row equal to 0. 4. For the ith row, i = 2, . . . , m, perform suitable elementary operations similar to the above steps to get a leading-1 entry for the ith row, and make all entries below the leading-1 entry of the ith row equal to 0. output: The reduced row-echelon form of B.

EXAMPLE 1.1

Consider the 3 × 5 matrix ⎡ −3 6 −1 1 A = ⎣ 1 −2 2 3 2 −4 5 8

⎤ −7 −1 ⎦ . −4

First, perform the elementary operations (−2)R2 + R3 → R3 and 3R2 + R1 → R1 : ⎡ ⎤ 0 0 5 10 −10 ⎣ 1 −2 2 3 −1 ⎦ . 0 0 1 2 −2 Then, perform (− 25 )R1 + R2 → R2 and (− 15 )R1 + R3 → R3 : ⎡ ⎤ 0 0 5 10 −10 ⎣ 1 −2 0 −1 3 ⎦. 0 0 0 0 0 Finally, perform R1 ↔ R2 : ⎡ ⎤ 1 −2 0 −1 3 ⎣ ¯ 0 0 5 10 −10 ⎦ ¯ 0 0 0 0 0

(row-echelon form).

The underlined positions give the pivot positions of the reworked matrix A. Hence, the first and third columns are the two pivot columns of the original version A. That is, the pivot columns are given by ⎡ ⎤ ⎡ ⎤ −3 −1 ⎣ 1 ⎦, ⎣ 2 ⎦. 2 5

16

Introduction to Matrix Algebra

After performing the row elementary operation (− 15 )R2 → R2 , the above echelon matrix comes to the reduced row-echelon form: ⎡

1 ⎣ 0 0

⎤ −2 0 −1 3 0 1 2 −2 ⎦ . 0 0 0 0

1.2.2 Gauss Elimination Methods Elementary row operations can be used to solve matrix equations and perform matrix inversion. 1. Gauss Elimination Method for Solving Matrix Equations Consider how to solve an n × n matrix equation Ax = b, where the inverse matrix A−1 of the matrix A exists. It is to be hoped that the solution vector x = A−1 b can be obtained by using elementary row operations. Form the n × (n + 1) augmented matrix B = [A, b]. Since the solution x = A−1 b can be written as a new matrix equation Ix = A−1 b, we get a new augmented matrix C = [I, A−1 b] associated with the solution x = A−1 b. Hence, we can write the solution process for the two matrix equations Ax = b and x = A−1 b respectively as follows: Elementary row

Ax = b −−−−−−−−−−→

matrix equations

operations

augmented matrices

[A, b]

Elementary row

−−−−−−−−−−→ operations

x = A−1 b, [I, A−1 b].

This implies that, after suitable elementary row operations on the augmented matrix [A, b], if the left-hand part of the new augmented matrix is an n × n identity matrix I then the (n + 1)th column gives the solution x = A−1 b of the original equation Ax = b directly. This method is called the Gauss or Gauss–Jordan elimination method. EXAMPLE 1.2

Use the Gauss elimination method to solve x1 + x2 + 2x3 = 6, 3x1 + 4x2 − x3 = 5, −x1 + x2 + x3 = 2.

As before, perform elementary row operations on the augmented matrix of the given

1.2 Elementary Row Operations and Applications

17

matrix equation to yield the following results: ⎤ ⎡ 1 1 1 1 2 6 (−3)R +R →R 2 2 1 ⎣ 3 4 −1 5 ⎦ −−−−− −−−−−→ ⎣ 0 1 −1 1 −1 1 1 2 ⎡ ⎤ ⎡ 1 1 2 6 (−1)R +R →R 1 0 2 1 1 ⎣ 0 1 −7 −13 ⎦ −−−−−−−− ⎣ 0 1 −−→ 0 2 3 8 0 2 ⎡ ⎤ ⎡ 1 0 9 19 1 0 9 1 17 R3 →R3 ⎣ 0 1 −7 −13 ⎦ − −−−−−→ ⎣ 0 1 −7 0 0 17 34 0 0 1 ⎡ ⎤ ⎡ 1 0 0 1 1 0 0 3 +R2 →R2 ⎣ 0 1 −7 −13 ⎦ 7R −−− −−−−−→ ⎣ 0 1 0 0 0 1 2 0 0 1 ⎡

⎤ 2 6 R +R →R 3 3 1 −7 −13 ⎦ −−−−−−−→ 1 2 ⎤ 9 19 (−2)R2 +R3 →R3 −7 −13 ⎦ −−−−−−−−−−→ 3 8 ⎤ 19 (−9)R +R →R 3 1 1 −13 ⎦ −−−−−−−−−−→ 2 ⎤ 1 1 ⎦. 2

Hence, the solution to the linear system of equations is given by x1 = 1, x2 = 1 and x3 = 2. Elementary row operations are also applicable for solving the m × n (where m > n) matrix equation Ax = b, as shown in Algorithm 1.2. To do this, it is necessary to transform the augmented matrix B = [A, b] to its reduced row-echelon form. Algorithm 1.2

Solving the m × n matrix equations Ax = b [232]

1. Form the augmented matrix B = [A, b]. 2. Run Algorithm 1.1 to transform the augmented matrix B to its reduced row-echelon form, which is equivalent to the original augmented matrix. 3. From the reduced row-echelon form write down the corresponding system of linear equations equivalent to the original system of linear equations. 4. Solve the new system of linear equations to yield its general solution.

EXAMPLE 1.3

Solve the system of linear equations 2x1 + 2x2 − x3 = 1, −2x1 − 2x2 + 4x3 = 1, 2x1 + 2x2 + 5x3 = 5, −2x1 − 2x2 − 2x3 = −3.

18

Introduction to Matrix Algebra

Perform the Type-II operation ( 12 )R1 → R1 on the augmented matrix: ⎤ ⎡ ⎡ ⎤ 1 1 1 − 12 2 2 −1 1 2 ⎥ ⎢ ⎢ −2 −2 4 1 ⎥ −2 −2 4 1 ⎥. ⎥ → ⎢ B=⎢ ⎥ ⎢ ⎣ 2 2 5 5 ⎦ ⎣ 2 2 5 5 ⎦ −2 −2 −2 −3 −2 −2 −2 −3 Use elementary row operations to make the first entry zero for the ith row, with i = 2, 3, 4: ⎡ ⎤ ⎤ ⎡ 1 1 1 1 − 12 1 1 − 12 2 2 ⎢ ⎥ ⎥ ⎢ ⎢ −2 −2 4 1 ⎥ → ⎢ 0 0 3 2 ⎥. ⎢ ⎥ ⎥ ⎢ ⎣ 0 0 ⎣ 2 2 5 5 ⎦ 6 4 ⎦ −2 −2 −2 −3 0 0 −3 −2 Perform the Type-II ⎡ 1 ⎢ ⎢ 0 ⎢ ⎣ 0 0

operation ( 13 )R2 → R2 : ⎤ ⎡ 1 1 − 12 2 ⎥ ⎢ 0 3 2 ⎥ → ⎢ ⎥ ⎢ ⎣ 0 6 4 ⎦ 0 −3 −2

1 1 0 0 0 0 0 0

− 12

1 2 2 3



⎥ ⎥ 1 ⎥. 6 4 ⎦ −3 −2

Then, using elementary row operations, transform to zero all entries directly above and below the leading-1 entry of the second row: ⎡ ⎡ ⎤ ⎤ 1 1 1 − 12 1 1 0 56 2 ⎢ ⎢ 2 ⎥ 2 ⎥ ⎢ 0 0 ⎢ ⎥ ⎥ 1 3 ⎥ → ⎢ 0 0 1 3 ⎥, ⎢ ⎣ 0 0 ⎣ 0 0 0 0 ⎦ 6 4 ⎦ 0 0 −3 −2 0 0 0 0 from which one has two new linear equations, x1 + x2 = 56 and x3 = 23 . This linear system of equations has infinitely many solutions, its general solution is given by x1 = 56 − x2 , x3 = 23 . If x2 = 1 then a set of particular solutions is given by x1 = − 16 , x2 = 1 and x3 = 23 . Any m × n complex matrix equation Ax = b can be written as (Ar + j Ai )(xr + j xi ) = br + j bi ,

(1.2.1)

where Ar , xr , br and Ai , xi , bi are the real and imaginary parts of A, x, b, respectively. Expand the above equation to yield Ar xr − Ai xi = b r ,

(1.2.2)

Ai xr + Ar xi = b i .

(1.2.3)

1.2 Elementary Row Operations and Applications

19

The above equations can be combined into 

Ar Ai

    xr b = r . xi bi

−Ai Ar

(1.2.4)

Thus, m complex-valued equations with n complex unknowns become 2m realvalued equations with 2n real unknowns. In particular, if m = n then we have complex matrix equation Ax = b   Ar −Ai br augmented matrix Ai Ar b i

Elementary row

x = A−1 b,

−−−−−−−−−−→ operations

Elementary row

−−−−−−−−−−→ operations



In On

On In

 xr . xi

This shows that if we write the n × (n + 1) complex augmented matrix [A, b] as a 2n × (2n + 1) real augmented matrix and perform elementary row operations to make its left-hand side become an 2n × 2n identity matrix then the upper and lower halves of the (2n + 1)th column give respectively the real and imaginary parts of the complex solution vector x of the original complex matrix equation Ax = b. 2. Gauss Elimination Method for Matrix Inversion Consider the inversion operation on an n × n nonsingular matrix A. This problem can be modeled as an n × n matrix equation AX = I whose solution X is the inverse matrix of A. It is easily seen that the augmented matrix of the matrix equation AX = I is [A, I], whereas the augmented matrix of the solution equation IX = A−1 is [I, A−1 ]. Hence, we have the following relations: matrix equation augmented matrix

AX = I

Elementary row

−−−−−−−−−−→ X = A−1 , operations

Elementary row

−−−−−−−−−−→ [I, A−1 ].

[A, I]

operations

This result tells us that if we use elementary row operations on the n×2n augmented matrix [A, I] so that its left-hand part becomes an n × n identity matrix then the right-hand part yields the inverse A−1 of the given n × n matrix A directly. This is called the Gauss elimination method for matrix inversion. Suppose that an n × n complex matrix A is nonsingular. Then, its inversion can be modeled as the complex matrix equation (Ar +jAi )(Xr +jXi ) = I. This complex equation can be rewritten as 

Ar Ai

−Ai Ar



   Xr I = n , Xi On

(1.2.5)

20

Introduction to Matrix Algebra

from which we get the following relation: Elementary row

complex matrix equation AX = I −−−−−−−−−−→ X = A−1 , operations     Ar −Ai In I On Xr Elementary row augmented matrix . −−−−−−−−−−→ n operations Ai Ar O n On In Xi That is to say, if performing elementary row operations on the 2n × 3n augmented matrix to transform its left-hand side to an 2n × 2n identity matrix then the upper and lower halves of the 2n × n matrix on the right give respectively the real and imaginary parts of the inverse matrix A−1 of the complex matrix A.

1.3 Sets, Vector Subspaces and Linear Mapping The set of all n-dimensional vectors with real (complex) components is called a real (complex) n-dimensional vector space, denoted Rn (Cn ). The n-dimensional vectors of real-world problems usually belong to subsets other than the whole set Rn or Cn . These subsets are known as vector subspaces. In this section, we present the sets, the vector subspaces and the linear mapping of one vector subspace onto another.

1.3.1 Sets Before introducing definitions of vector spaces and subspaces, it is necessary to present some set concepts. As the name implies, a set is a collection of elements. A set is usually denoted by S = {·}; inside the braces are the elements of the set S. If there are only a few elements in the set S, we write out these elements within the braces, e.g., S = {a, b, c, d}. To describe the composition of a more complex set mathematically, we use the symbol “|” to mean “such that”. For example, S = {x|P (x) = 0} reads “the element x in set S such that P (x) = 0”. A set with only one element α is called a singleton, denoted {α}. The following are several common notations for set operations: ∀ denotes “for all · · · ”; x ∈ A reads “x belongs to the set A”, i.e., x is an element or member of A; x∈ / A means that x is not an element of the set A;  denotes “such that”; ∃ denotes “there exists”; A ⇒ B reads “condition A results in B” or “A implies B”. As an example, the trivial statement “there is a zero element θ in the set V such that the addition x + θ = x = θ + x holds for all elements x in V ” can be concisely

1.3 Sets, Vector Subspaces and Linear Mapping

21

expressed in the above notation as follows: ∃ θ ∈ V  x + θ = x = θ + x, ∀ x ∈ V. Let A and B be two sets; then the sets have the following basic relations. The notation A ⊆ B reads “the set A is contained in the set B” or “A is a subset of B”, which implies that each element in A is an element in B, namely x ∈ A ⇒ x ∈ B. If A ⊂ B then A is called a proper subset of B. The notation B ⊃ A reads “B contains A” or “B is a superset of A”. The set with no elements is denoted by Ø and is called the null set. The notation A = B reads “the set A equals the set B”, which means that A ⊆ B and B ⊆ A, or x ∈ A ⇔ x ∈ B (any element in A is an element in B, and vice versa). The negation of A = B is written as A = B, implying that A does not belong to B, neither does B belong to A. The union of A and B is denoted as A ∪ B. If X = A ∪ B then X is called the union set of A and B. The union set is defined as follows: X = A ∪ B = {x ∈ X| x ∈ A or x ∈ B}.

(1.3.1)

In other words, the elements of the union set A ∪ B consist of the elements of A and the elements of B. The intersection of both sets A and B is represented by the notation A ∩ B and is defined as follows: X = A ∩ B = {x ∈ X| x ∈ A and x ∈ B}.

(1.3.2)

The set X = A ∩ B is called the intersection set of A and B. Each element of the intersection set A ∩ B consists of elements common to both A and B. The notation Z = A + B means the sum set of sets A and B and is defined as follows: Z = A + B = {z = x + y ∈ Z| x ∈ A, y ∈ B},

(1.3.3)

namely, an element z of the sum set Z = A + B consists of the sum of the element x in A and the element y in B. The set-theoretic difference of the sets A and B, denoted “A − B”, is also termed the difference set and is defined as follows: X = A − B = {x ∈ X| x ∈ A, but x ∈ / B}.

(1.3.4)

That is to say, the difference set A − B is the set of elements of A that are not in B. The difference set A − B is also sometimes denoted by the notation X = A \ B. For example, {Cn \ 0} denotes the set of nonzero vectors in complex n-dimensional vector space. The relative complement of A in X is defined as Ac = X − A = X \ A = {x ∈ X| x ∈ / A}.

(1.3.5)

22

Introduction to Matrix Algebra

EXAMPLE 1.4

For the sets A = {1, 2, 3},

B = {2, 3, 4},

we have A ∪ B = {1, 2, 3, 4},

A ∩ B = {2, 3},

A − B = A \ B = {1},

A + B = {3, 5, 7},

B − A = B \ A = {4}.

If both X and Y are sets, and x ∈ X and y ∈ Y , then the set of all ordered pairs (x, y) is denoted by X × Y and is termed the Cartesian product of the sets X and Y , and is defined as follows:    X × Y = (x, y)x ∈ X, y ∈ Y . (1.3.6) Similarly, X1 × · · · × Xn denotes the Cartesian product of n sets X1 , . . . , Xn , and its elements are the ordered n-ples (x1 , . . . , xn ):    (1.3.7) X1 × · · · × Xn = (x1 , . . . , xn )x1 ∈ X1 , . . . , xn ∈ Xn . If f (X, Y) is a scalar function with real matrices X ∈ Rn×n and Y ∈ Rn×n as variables then in linear mapping notation, the function can be denoted by the Cartesian product form f : Rn×n × Rn×n → R. 1.3.2 Fields and Vector Spaces The previous subsections uses the symbols R, Rm , Rm×n and C, Cm , Cm×n . In this subsection we explain this notation from the viewpoint of vector spaces. DEFINITION 1.14 [229] A field is an algebraic structure F consisting of a nonempty set F together with two operations: for any two elements α, β in the set F , the addition operation α + β and the multiplication operation αβ are uniquely determined such that, for any α, β, γ ∈ F , the following conditions are satisfied: (1) (2) (3) (4) (5) (6) (7)

α + β = β + α and αβ = βα; α + (β + γ) = (α + β) + γ and α(βγ) = (αβ)γ; α(β + γ) = αβ + αγ; there is an element 0 in F such that α + 0 = α for all α ∈ F ; there is an element −α such that α + (−α) = 0 for any α ∈ F ; there exists a nonzero element 1 ∈ F such that α1 = α for all α ∈ F ; there is an element α−1 ∈ F such that αα−1 = 1 for any nonzero element α ∈ F.

The set of rational numbers, the set of real numbers and the set of complex numbers are fields, the rational number field Q, the real number field R and the complex number field C. However, the set of integers is not a field. When considering a nonempty set V with vectors as elements, the concept of the field can be generalized to the concept of vector space.

1.3 Sets, Vector Subspaces and Linear Mapping

23

DEFINITION 1.15 A set V with vectors as elements is called a vector space if the addition operation is defined as the addition of two vectors, the multiplication operation is defined as the product of a vector and a scalar in the scalar field S and, for vectors x, y, w in the set V and scalars a1 , a2 in the scalar field S, the following axioms, also termed postulates or laws, on addition and multiplication hold. 1. Closure properties (1) If x ∈ V and y ∈ V then x + y ∈ V , namely V is closed under addition. This is called the closure for addition. (2) If a1 is a scalar and y ∈ V then a1 y ∈ V , namely V is closed under scalar multiplication. This is called the closure for scalar multiplication. 2. Axioms on addition (1) Commutative law for addition: x + y = y + x, ∀ x, y ∈ V . (2) Associative law for addition: x + (y + w) = (x + y) + w, ∀ x, y, w ∈ V . (3) Existence of the null vector: There exists a null vector 0 in V such that, for any vector y ∈ V , y + 0 = y holds. (4) Existence of negative vectors: Given a vector y ∈ V , there is another vector −y ∈ V such that y + (−y) = 0 = (−y) + y. 3. Axioms on scalar multiplication (1) Associative law for scalar multiplication: a(by) = (ab)y holds for all vectors y ∈ V and all scalars a, b ∈ S. (2) Right distributive law for scalar multiplication: a(x + y) = ax + ay holds for all vectors x, y ∈ V and any scalar a in S. (3) Left distributive law for scalar multiplication: (a + b)y = ay + by holds for all vectors y and all scalars a, b ∈ S. (4) Unity law for scalar multiplication: 1y = y holds for all vectors y ∈ V . As stated in the following theorem, vector spaces have other useful properties. THEOREM 1.4 • • • • • •

If V is a vector space, then

The null vector 0 is unique. The inverse operation −y is unique for each y ∈ V . For every vector y ∈ V , 0y = 0 is true. For every scalar a, a0 = 0 holds. If ay = 0 then a = 0 or y = 0. (−1)y = −y.

Proof

See [232, pp. 365–366].

24

Introduction to Matrix Algebra

If the vectors in a vector space V are real valued, and the scalar field is a real field, then V is referred to as a real vector space. Similarly, if the vectors in V are complex valued, and the scalar field is a complex field, then V is called a complex vector space. The two most typical vector spaces are the m-dimensional real vector space Rm and the m-dimensional complex vector space Cm . In many cases we are interested only in some given subset of a vector space V other than the whole space V . DEFINITION 1.16 Let W be a nonempty subset of the vector space V . Then W is called a vector subspace of V if all elements in W satisfy the following two conditions: (1) x − y ∈ W for all x, y ∈ W ; (2) αx ∈ W for all α ∈ R (or C) and all x ∈ W . Vector subspaces are very useful in engineering applications such as automatic control, signal processing, pattern recognition, wireless communication and so forth. We will devote our attention to the theory and applications of vector subspaces in Chapter 8, “Subspace Analysis and Tracking”.

1.3.3 Linear Mapping In the previous subsections we discussed some simple operations of vectors, vector addition and the multiplication of a vector by a scalar, but we have not yet considered the transformation between vectors in two vector spaces. By Wikipedia, mapping is the creation of maps, a graphic symbolic representation of the significant features of a part of the surface of the Earth. In mathematics, mapping is a synonym for mathematical function or for Morphism. The notation for mappings follows the usual notation for functions. If V is a subspace in Rm and W is a subspace in Rn then the notation T : V → W

(1.3.8)

T : V →W

(1.3.9)

denotes a general mapping, while

represents a linear mapping or linear transformation, which is a mapping such that the following operations hold: 1. T (v1 + v2 ) = T (v1 ) + T (v2 ) for any vectors v1 and v2 in V ; 2. T (αv) = αT (v) for any scalar α. The main example of a linear transformation is given by matrix multiplication.

1.3 Sets, Vector Subspaces and Linear Mapping

25

Thus, a mapping represents a rule for transforming the vectors in V to corresponding vectors in W . The subspace V is said to be the initial set or domain of the mapping T and W its final set or codomain. When v is some vector in the vector space V , T (v) is referred to as the image of the vector v under the mapping T , or the value of the mapping T at the point v, whereas v is called the original image of T (v). Given a subspace A of the vector space V , the mapping T (A) = {T (v) | v ∈ A}

(1.3.10)

represents a collection of the vectors v under the mapping T . In particular, if T (v) represents a collection of transformed outputs of all vectors v in V , i.e., T (V ) = {T (v) | v ∈ V },

(1.3.11)

then we say that T (V ) is the range of the mapping T , denoted by Im(T ) = T (V ) = {T (v) | v ∈ V }.

(1.3.12)

DEFINITION 1.17 A mapping T : V → W is one-to-one if it is either injective or surjective, i.e., T (x) = T (y) implies x = y or distinct elements have distinct images. A one-to-one mapping T : V → W has an inverse mapping T −1 |W → V . The inverse mapping T −1 restores what the mapping T has done. Hence, if T (v) = w then T −1 (w) = v, resulting in T −1 (T (v)) = v, ∀ v ∈ V and T (T −1 (w)) = w, ∀ w ∈ W. DEFINITION 1.18 A mapping T is called a linear mapping or linear transformation in the vector space V if it satisfies the linear relationship T (c1 v1 + c2 v2 ) = c1 T (v1 ) + c2 T (v2 )

(1.3.13)

for all v1 , v2 ∈ V and all scalars c1 , c2 . If u1 , . . . , up are the input vectors of a system in engineering then T (u1 ), . . . , T (up ) can be viewed as the output vectors of the system. The criterion for identifying whether a system is linear is this: if the system input is the linear expression y = c1 u1 + · · · + cp up then we say that the system is linear only if its output satisfies the linear expression T (y) = T (c1 u1 + · · · + cp up ) = c1 T (u1 ) + · · · + cp T (up ). Otherwise, the system is nonlinear. EXAMPLE 1.5

Determine whether the following transformation T1 , T2 : R3 → R2

26

Introduction to Matrix Algebra

are linear:



x1 + x2



, where x = [x1 , x2 , x3 ]T , x21 − x22   x1 − x2 T2 (x) = , where x = [x1 , x2 , x3 ]T . x2 + x3

T1 (x) =

It is easily seen that the transformation T1 | R3 → R2 does not satisfy a linear relationship and thus is not a linear transformation, while T2 | R3 → R2 is a linear transformation since it satisfies a linear relationship. An interesting special application of linear mappings is the electronic amplifier A ∈ Cn×n with high fidelity (Hi-Fi). By Hi-Fi, one means that there is the following linear relationship between any input signal vector u and the corresponding output signal vector Au of the amplifier: Au = λu,

(1.3.14)

where λ > 1 is the amplification factor or gain. The equation above is a typical characteristic equation of a matrix. Another example is that the product of a matrix by a vector Am×n xn×1 can also be viewed as a linear mapping, T : x → Ax or T (x) = Ax, that transforms the vector x in Cn to a vector y = Ax in Cm . In view of this linear mapping or transformation, the product of a matrix A and a vector is usually called the matrix transformation of the vector, where A is a transformation matrix. The following are intrinsic relationships between a linear space and a linear mapping. THEOREM 1.5 [35, p. 29] Let V and W be two vector spaces, and let T : V → W be a linear mapping. Then the following relationships are true: • if M is a linear subspace in V then T (M ) is a linear subspace in W ; • if N is a linear subspace in W then the linear inverse transform T −1 (N ) is a linear subspace in V . A linear mapping has the following basic properties: if T : V → W is a linear mapping then T (0) = 0

and

T (−x) = −T (x).

(1.3.15)

In particular, for a given linear transformation y = Ax where the transformation matrix A is known, if our task is to obtain the output vector y from the input vector x then Ax = y is said to be a forward problem. Conversely, the problem of finding the input vector x from the output vector y is known as an inverse problem. Clearly, the essence of the forward problem is a matrix–vector calculation, while the essence of the inverse problem is solving a matrix equation.

1.4 Inner Products and Vector Norms

27

1.4 Inner Products and Vector Norms In mathematics, physics and engineering, we often need to measure the size (or length) and neighborhood for a given vector, and to calculate the angle, distance and similarity between vectors. In these cases, inner products and vector norms are two essential mathematical tools.

1.4.1 Inner Products of Vectors Let K represent a field that may be either the real field R or the complex field C, and let V be an n-dimensional vector space Rn or Cn . DEFINITION 1.19 The function x, y is called the inner product of two vectors x and y, and V is known as the inner product vector space, if for all vectors x, y, z ∈ V and scalars α, β ∈ K, the mapping function ·, ·|V × V → K satisfies the following three axioms. (1) Conjugate symmetry x, y = y, x∗ . (2) Linearity in the first argument αx + βy, z = αx, z + βy, z. (3) Nonnegativity x, x ≥ 0 and x, x = 0 ⇔ x = 0 (strict positivity). For a real inner product space, the conjugate symmetry becomes the real symmetry x, y = y, x. Linearity in the first argument contains the homogeneity αx, y = αx, y and the additivity x + y, z = x, z + y, z. Conjugate symmetry and linearity in the first argument imply that x, αy = αy, x∗ = α∗ y, x∗ = α∗ x, y, ∗





x, y + z = y + z, x = y, x + z, x = x, y + x, z.

(1.4.1) (1.4.2)

A vector space with an inner product is called an inner product space. It is easy to verify that the function x, y = xH y =

n 

x∗i yi

(1.4.3)

i=1

is an inner product of two vectors, since the three axioms above are satisfied. The function x, y = xH y is called the canonical inner product of n × 1 vectors x = [x1 , . . . , xn ]T and y = [y1 , . . . , yn ]T . Note that in some literature the following canonical inner product formula is used: n  x, y = xT y∗ = xi yi∗ . i=1

However, one sometimes adopts the weighted inner product x, yG = xH Gy,

(1.4.4)

28

Introduction to Matrix Algebra

where G is a weighting matrix such that xH Gx > 0, ∀ x ∈ Cn . −1 EXAMPLE 1.6 Let {e j2πf n }N n=0 be a sinusoidal sequence with frequency f , and  T 2π 2π en (f ) = 1, e j( n+1 )f , . . . , e j( n+1 )nf be an (n+1)×1 complex sinusoid vector. Then the discrete Fourier transform (DFT) of N data samples x(n), n = 0, 1, . . . , N − 1 can be represented by the canonical inner product as follows:

X(f ) =

N −1 

x(n)e−j2πnf /N = eH N −1 (f )x = eN −1 (f ), x,

n=0

where x = [x(0), x(1), . . . , x(N − 1)]T is the data vector.

1.4.2 Norms of Vectors DEFINITION 1.20 Let V be a real or complex vector space. Given a vector x ∈ V , the mapping function p(x)|V → R is called the norm of the vector x ∈ V , if for all vectors x, y ∈ V and any scalar c ∈ K (here K denotes R or C), the following three norm axioms hold: (1) Nonnegativity: p(x) ≥ 0 and p(x) = 0 ⇔ x = 0. (2) Homogeneity: p(cx) = |c| · p(x) is true for all complex constant c. (3) Triangle inequality: p(x + y) ≤ p(x) + p(y). In a real or complex inner product space V , the vector norms have the following properties [35]. 1. 0 = 0 and x > 0, ∀ x = 0. 2. cx = |c| x holds for all vector x ∈ V and any scalar c ∈ K. 3. Polarization identity: For real inner product spaces we have x, y =

 1

x + y 2 − x − y 2 , 4

∀ x, y,

(1.4.5)

and for complex inner product spaces we have x, y =

 1

x + y 2 − x − y 2 − j x + j y 2 + j x − j y 2 , 4

∀ x, y. (1.4.6)

4. Parallelogram law  

x + y 2 + x − y 2 = 2 x 2 + y 2 ,

∀ x, y.

(1.4.7)

5. Triangle inequality

x + y ≤ x + y ,

∀ x, y ∈ V.

(1.4.8)

1.4 Inner Products and Vector Norms

29

6. Cauchy–Schwartz inequality |x, y| ≤ x

y .

(1.4.9)

The equality |x, y| = x

y holds if and only if y = c x, where c is some nonzero complex constant. 1. Common Norms of Constant Vectors The following are several common norms of constant vectors. (1) 0 -norm def

x 0 = number of nonzero entries of x.

(1.4.10)

(2) 1 -norm def

x 1 =

m 

|xi | = |x1 | + · · · + |xm |.

(1.4.11)

i=1

(3) 2 -norm or the Euclidean norm

(4) ∞ -norm

 1/2 def .

x 2 = x E = |x1 |2 + · · · + |xm |2

(1.4.12)

  def

x ∞ = max |x1 |, . . . , |xm | .

(1.4.13)

(5) p -norm or H¨ older norm [275]  def

x p =

m 

1/p |xi |p

,

p ≥ 1.

(1.4.14)

i=1

The 0 -norm does not satisfy the homogeneity cx 0 = |c| x 0 , and thus is a quasi-norm. However, the 0 -norm plays a key role in the representation and analysis of sparse vectors, as will be seen in Section 1.12. The p -norm is a quasi-norm if 0 < p < 1 and a norm if p ≥ 1. Clearly, when p = 1 or p = 2, the p -norm reduces to the 1 -norm or the 2 -norm, respectively. The most commonly used vector norm is the Euclidean norm. The following are several important applications of the Euclidean norm. • Measuring the size or length of a vector: size(x) = x 2 =



x21 + · · · + x2m ,

which is called the Euclidean length. • Defining the -neighborhood of a vector x:    N (x) = y y − x 2 ≤ ,

> 0.

(1.4.15)

(1.4.16)

30

Introduction to Matrix Algebra

• Measuring the distance between vectors x and y:  d(x, y) = x − y 2 = (x1 − y1 )2 + · · · + (xm − ym )2 . This is called the Euclidean distance. • Defining the angle θ (0 ≤ θ ≤ 2π) between vectors x and y:    H  x y x, y def  . θ = arccos  = arccos

x

y x, x y, y

(1.4.17)

(1.4.18)

A vector with unit Euclidean length is referred to as a normalized (or standardized) vector. For any nonzero vector x ∈ Cm , x/x, x1/2 is the normalized version of the vector and has the same direction as x. The norm x is said to be a unitary invariant norm if Ux = x holds for all vectors x ∈ Cm and all unitary matrices U ∈ Cm×m . PROPOSITION 1.1 [214] The Euclidean norm · 2 is unitary invariant. If the inner product x, y = xH y = 0, this implies that the angle between the vectors θ = π/2. In this case, the constant vectors x and y are said to be orthogonal, from which we have the following definition. DEFINITION 1.21 Two constant vectors x and y are said to be orthogonal, and this is denoted by x ⊥ y, if their inner product x, y = xH y = 0. 2. Inner Products and Norms of Function Vectors DEFINITION 1.22 Let x(t), y(t) be two function vectors in the complex vector space Cn , and let the definition field of the function variable t be [a, b] with a < b. Then the inner product of the function vectors x(t) and y(t) is defined as follows: b def x(t), y(t) = xH (t)y(t)dt. (1.4.19) a

It can be verified that the above function satisfies the three axioms of inner products (see Definition 1.19). Note that the real field R is an one-dimensional inner product space, but it is not a Euclidean space as the real field R is not finite dimensional. The angle between function vectors is defined as follows: b xH (t)y(t)dt x(t), y(t) def a  cos θ =  = , (1.4.20)

x(t) · y(t) x(t), x(t) y(t), y(t) where x(t) is the norm of the function vector x(t) and is defined as follows:  1/2 b def H x (t)x(t)dt . (1.4.21)

x(t) = a

1.4 Inner Products and Vector Norms

31

Clearly, if the inner product of two function vectors is equal to zero, i.e., b xH (t)y(t)dt = 0, a

then the angle θ = π/2. Hence, two function vectors are said to be orthogonal in [a, b], denoted x(t) ⊥ y(t), if their inner product is equal to zero. 3. Inner Products and Norms of Random Vectors DEFINITION 1.23 Let x(ξ) and y(ξ) be two n × 1 random vectors of variable ξ. Then the inner product of random vectors is defined as follows:   def x(ξ), y(ξ) = E xH (ξ)y(ξ) , (1.4.22) where E is the expectation operator E{x(ξ)} = [E{x1 (ξ)}, . . . , E{xn (ξ)}]T and the function variable ξ may be time t, circular frequency f , angular frequency ω or space parameter s, and so on. The square of the norm of a random vector x(ξ) is defined as   def

x(ξ) 2 = E xH (ξ)x(ξ) .

(1.4.23)

In contrast with the case for a constant vector or function vector, an m × 1 random vector x(ξ) and an n × 1 random vector y(ξ) are said to be orthogonal if any entry of x(ξ) is orthogonal to each entry of y(ξ). The following is the definition of random vector orthogonality. DEFINITION 1.24 An m×1 random vector x(ξ) and an n×1 random vector y(ξ) are orthogonal, denoted by x(ξ) ⊥ y(ξ), if their cross-correlation matrix is equal to the m × n null matrix O, i.e.,   E x(ξ)yH (ξ) = O. (1.4.24) The following proposition shows that, for any two orthogonal vectors, the square of the norm of their sum is equal to the sum of the squares of the respective vector norms. PROPOSITION 1.2 Proof

If x ⊥ y, then x + y 2 = x 2 + y 2 .

From the axioms of vector norms it is known that

x + y 2 = x + y, x + y = x, x + x, y + y, x + y, y.

(1.4.25)

Since x and y are orthogonal, we have x, y = E{xT y} = 0. Moreover, from the axioms of inner products it is known that y, x = x, y = 0. Substituting this result into Equation (1.4.25), we immediately get

x + y 2 = x, x + y, y = x 2 + y 2 . This completes the proof of the proposition.

32

Introduction to Matrix Algebra

This proposition is also referred to as the Pythagorean theorem. On the orthogonality of two vectors, we have the following conclusions. • Mathematical definitions Two vectors x and y are orthogonal if their inner product is equal to zero, i.e., x, y = 0 (for constant vectors and function vectors), or the mathematical expectation of their outer product is equal to a null matrix O, i.e., E{xyH } = O (for random vectors). • Geometric interpretation If two vectors are orthogonal then their angle is π/2, and the projection of one vector onto the other vector is equal to zero. • Physical significance When two vectors are orthogonal, each vector contains no components of the other, that is, there exist no interactions or interference between these vectors. The main points above are useful for adopting flexibly the orthogonality of two vectors in engineering applications.

1.4.3 Similarity Comparison Between Vectors Clustering and pattern classification are two important techniques in statistical data analysis. By clustering, we mean that a given big data set is divided into several data subsets (categories) in which the data in each subset have common or similar features. Pattern classification involves classifying some unknown stimuli (activations) into a fixed number of categories (classes). The main mathematical tool for clustering and classification is the distance measure. The distance between two vectors p and g, denoted D(p g), is a measure if it has following properties. (1) Nonnegativity and positiveness D(p g) ≥ 0, and equality holds if and only if p = g. (2) Symmetry D(p g) = D(g p). (3) Triangle inequality D(p z) ≤ D(p g) + D(g z). It is easily shown that the Euclidean distance is a measure. In many applications, data vectors must become low-dimensional vectors via some transformation or processing method. These low-dimensional vectors are called the pattern vectors or feature vectors owing to the fact that they extract the features of the original data vectors, and are directly used for pattern clustering and classification. For example, the colors of clouds and the parameters of voice tones are pattern or feature vectors in weather forecasting and voice classification, respectively. The basic rule of clustering and classification is to adopt some distance metric to measure the similarity of two feature vectors. As the name suggests, this similarity is a measure of the degree of similarity between vectors.

1.4 Inner Products and Vector Norms

33

Consider a pattern classification problem. For simplicity, suppose that there are M classes of pattern vectors s1 , . . . , sM . Our problem is: given a feature vector x with an unknown pattern, we hope to recognize to which class it belongs. For this purpose, we need to compare the unknown pattern vector x with M known pattern vectors in order to recognize to which known pattern vector the given vector x is most similar. On the basis of a similarity comparison, we can obtain the pattern or signal classification. A quantity known as the dissimilarity is used to make a reverse measurement of the similarity between vectors: two vectors with a small dissimilarity are similar. Let D(x, s1 ), . . . , D(x, sM ) be the dissimilarities between the unknown pattern vector x and the respective known pattern vectors s1 , . . . , sM . As an example, we will compare the dissimilarities between x and s1 , s2 . If D(x, s1 ) < D(x, s2 ),

(1.4.26)

then we say the unknown pattern vector x is more similar to s1 than s2 , where s1 and s2 are known pattern vectors. The simplest and most intuitive dissimilarity parameter is the Euclidean distance between vectors. The Euclidean distance between the unknown pattern vector x and the ith known pattern vector si , denoted DE (x, si ), is given by  (1.4.27) DE (x, si ) = x − si 2 = (x − si )T (x − si ). The two vectors are exactly the same if their Euclidean distance is equal to zero: DE (x, y) = 0



x = y.

If DE (x, si ) = min DE (x, sk ), k

k = 1, . . . , M,

(1.4.28)

then si ∈ {s1 , . . . , sM } is said to be a nearest neighbour to x. A widely used classification method is nearest neighbor classification, which judges x to belong in the model type corresponding to its nearest neighbour. Another frequently used distance function is the Mahalanobis distance, proposed by Mahalanobis in 1936 [312]. The Mahalanobis distance from the vector x to its mean μ is given by  DM (x, μ) = (x − μ)T C−1 (1.4.29) x (x − μ), where Cx = Cov(x, x) = E{(x − μ)(x − μ)T } is the autocovariance matrix of the vector x. The Mahalanobis distance between vectors x ∈ Rn and y ∈ Rn is denoted by DM (x, y), and is defined as [312]  DM (x, y) = (x − y)T C−1 (1.4.30) xy (x − y),

34

Introduction to Matrix Algebra

where Cxy = Cov(x, y) = E{(x − μx )(y − μy )T } is the cross-covariance matrix of x and y, while μx and μy are the means of x and y, respectively. Clearly, if the covariance matrix is the identity matrix, i.e., C = I, then the Mahalanobis distance reduces to the Euclidean distance. If the covariance matrix takes a diagonal form then the corresponding Mahalanobis distance is called the normalized Euclidean distance and is given by  n  (xi − yi )2 DM (x, y) = ! , (1.4.31) σi2 i=1 in which σi is the standard deviation of xi and yi in the whole sample set. Let M M M   1  μ= si , C = (si − μ)(sj − μ)T (1.4.32) M i=1 i=1 j=1 be the sample mean vector of M known pattern vectors si and the sample crosscovariance matrix. Then the Mahalanobis distance from the unknown pattern vector x to the ith known pattern vector si is defined as  (1.4.33) DM (x, si ) = (x − si )T C−1 (x − si ). By the nearest neighbor classification method, if DM (x, si ) = min DM (x, sk ), k

k = 1, . . . , M,

(1.4.34)

then the unknown pattern vector x is recognized as being in the pattern type to which si belongs. The measure of dissimilarity between vectors is not necessarily limited to distance functions. The cosine function of the acute angle between two vectors, D(x, si ) = cos θi =

xT s i ,

x 2 si 2

(1.4.35)

is an effective measure of dissimilarities as well. If cos θi < cos θj , ∀ j = i, holds then the unknown pattern vector x is said to be most similar to the known pattern vector si . The variant of Equation (1.4.35) D(x, si ) =

x T si xT x + sTi si + xT si

(1.4.36)

is referred to as the Tanimoto measure [469] and is widely used in information retrieval, the classification of diseases, animal and plant classifications etc. The signal to be classified as one of a set of objects is called the target signal. Signal classification is usually based on some physical or geometric attributes or concepts relating to the objects. Let X be a target signal and let Ai represent a classification attribute defining the ith object. Then we can adopt the object–concept

1.4 Inner Products and Vector Norms

35

distance D(X, Ai ) to describe the dissimilarity between the target signal and the ith object [448], and thus we have a relationship similar to Equation (1.4.26) as follows: if D(X, Ai ) < D(X, Aj ),

∀ i, j,

(1.4.37)

then the target signal X is classified as the ith type of object, Ci , with the minimum object–concept distance D(X, Ai ).

1.4.4 Banach Space, Euclidean Space, Hilbert Space A vector space equipped with a vector norm is called a normed vector space. Let X be a vector space. If, for every Cauchy sequence {xn } in X, there exists an element x in X such that lim xn = x

n→∞

then X is known as a complete vector space. The above convergence of a Cauchy sequence of vectors can be equivalently written as a convergence in the norm, lim xn − x = 0.

n→∞

A Banach space is a complete normed vector space. Because the vector norm is a metric that allows the computation of vector length and the distance between vectors, a Banach space is a vector space with a metric that can compute vector length and the distance between vectors and is complete in the sense that a Cauchy sequence of vectors always converges in the norm. A Euclidean space is an affine space with the standard Euclidean structure and encompasses the two-dimensional Euclidean plane, the three-dimensional space of Euclidean geometry, and other higher-dimensional spaces. Euclidean spaces have finite dimension [449]. An n-dimensional Euclidean space is usually denoted by Rn , which is assumed to have the standard Euclidean structure. The standard Euclidean structure includes the following: (1) the standard inner product (also known as the dot product) on Rm , x, y = x · y = xT y = x1 y1 + · · · + xm ym ; (2) the Euclidean length of a vector x on Rm ,  

x = x, y = x21 + · · · + x2m ; (3) the Euclidean distance between x and y on Rm ,  d(x, y) = x − y = (x1 − y1 )2 + · · · + (xm − ym )2 ;

36

Introduction to Matrix Algebra

(4) the (nonreflex) angle θ (0◦ ≤ θ ≤ 180◦ ) between vectors x and y on Rm ,    T  x, y x y θ = arccos = arccos .

x

y

x

y A vector space equipped with an inner product is referred to as an inner product space. Hilbert space, named after David Hilbert, is a generalization of Euclidean space: it is no longer limited to the finite-dimensional case. Similarly to Euclidean space, a Hilbert space is an inner product space that allows the length of a vector and the distance and angle between vectors to be measured, and thus it possesses the definition of orthogonality. Furthermore, a Hilbert space is a complete space, and all Cauchy sequences are equivalent to convergent sequences, so that most concepts in calculus can be extended to Hilbert spaces. Hilbert spaces provide an efficient representation for Fourier series and Fourier transforms based on any orthogonal series.

1.4.5 Inner Products and Norms of Matrices The inner products and norms of vectors can be extended to the inner products and norms of matrices. Consider m × n complex matrices A = [a1 , . . . , an ] and B = [b1 , . . . , bn ]. We stack A and B respectively into the following mn × 1 vectors according to their columns: ⎡ ⎤ ⎡ ⎤ a1 b1 ⎢ .. ⎥ ⎢ .. ⎥ a = vec(A) = ⎣ . ⎦ , b = vec(B) = ⎣ . ⎦ , an

bn

where the elongated vector vec(A) is the vectorization of the matrix A. We will discuss the vectorization of matrices in detail in Section 1.11. DEFINITION 1.25 The inner product of two m × n matrices A and B, denoted A, B| Cm×n ×Cm×n → C, is defined as the inner product of two elongated vectors: A, B = vec(A), vec(B) =

n  i=1

aH i bi

=

n 

ai , bi ,

(1.4.38)

i=1

equivalently written as A, B = (vec A)H vec(B) = tr(AH B),

(1.4.39)

where tr(C) represents the trace function of a square matrix C, defined as the sum of its diagonal entries. Let K represent a real or complex field; thus Km×n denotes either Rm×n or Cm×n .

1.4 Inner Products and Vector Norms

37

DEFINITION 1.26 The norm of the matrix A ∈ Km×n , denoted A , is defined as the real-valued function having the following properties: (a) For any nonzero matrix A = O, its norm is larger than zero, i.e., A > 0, if A = O, and A = 0 if and only if A = O. (b) cA = |c| A for any c ∈ K. (c) A + B ≤ A + B . (d) The norm of the product of two matrices is less than or equal to the product of their norms, that is, AB ≤ A

B . n n EXAMPLE 1.7 For a real-valued function f (A) = i=1 j=1 |aij |, it is easy to verify the following: (a) f (A) ≥ 0; when A = 0, i.e., when aij ≡ 0, f (A) = 0. n n n  n    (b) f (cA) = |caij | = |c| |aij | = |c|f (A). i=1 j=1 n n  

(c) f (A + B) =

i=1 j=1

(|aij + bij |) ≤

i=1 j=1

n n  

(|aij | + |bij |) = f (A) + f (B).

i=1 j=1

(d) For the product of two matrices, we have  n  n  n  n n  n       aik bkj  ≤ |aik ||bkj | f (AB) =    i=1 j=1 k=1 i=1 j=1 k=1  n  n n  n    |aik | |bkl | = f (A)f (B). ≤ i=1 j=1

k=1

l=1

Hence, by Definition 1.26, the real function f (A) = norm.

n i=1

n j=1

|aij | is a matrix

There are three types of common matrix norms: induced norms, entrywise norms and Schatten norms. 1. Induced Norms The induced norm of an m × n matrix A ∈ Km×n is defined by means of the norm

x of the vector x ∈ Kn and the norm Ax of the vector Ax ∈ Km as  # "

Ax  n x ∈ K with x =  0 (1.4.40)

A = sup

x     (1.4.41) = sup Ax x ∈ Kn with x = 1 . The induced norm is also called the operator norm. A common induced norm is the p-norm:

Ax p def , (1.4.42)

A p = sup x=0 x p

38

Introduction to Matrix Algebra

where p = 1, 2, . . . The p-norm is also referred to as the Minkowski p-norm or p -norm. In particular, when p = 1 or ∞ the corresponding induced norms are respectively

A 1 = max

1≤j≤n

A ∞ = max

m 

|aij |,

i=1 n 

1≤i≤m

|aij |.

(1.4.43) (1.4.44)

j=1

That is, A 1 and A ∞ are simply the maximum absolute column sum and maximum absolute row sum of the matrix, respectively. The induced norms A 1 and A ∞ are also called the absolute column sum norm and the absolute row sum norm, respectively. EXAMPLE 1.8

For the matrix



⎤ 1 −2 3 ⎢ −4 5 −6 ⎥ ⎥, A=⎢ ⎣ 7 −8 −9 ⎦ −10 11 12

its absolute column sum norm and row sum norm are respectively given by 

A 1 = max 1 + | − 4| + 7 + | − 10|, | − 2| + 5 + | − 8| + 11,  3 + | − 6| + | − 9| + 12 = max{22, 26, 30} = 30, 

A ∞ = max 1 + | − 2| + 3, | − 4| + 5 + | − 6|, 7 + | − 8| + | − 9|,  | − 10| + 11 + 12 = max{6, 15, 24, 33} = 33. Another common induced matrix norm is the spectral norm, with p = 2, denoted

A 2 = A spec . This is defined as  (1.4.45)

A 2 = A spec = λmax (AH A) = σmax (A), i.e., the spectral norm is the largest singular value of A or the square root of the largest eigenvalue of the positive semi-definite matrix AH A. 2. Entrywise Norm Another type of matrix norm, the entrywise norms, treats an m × n matrix as a column vector of size mn and uses one of the familiar vector norms. Let a = [a11 , . . . , am1 , a12 , . . . , am2 , . . . , a1n , . . . , amn ]T = vec(A) be an mn × 1 elongated vector of the m × n matrix A. If we use the p -norm definition of the elongated vector a then we obtain the p -norm of the matrix A as follows: ⎞1/p ⎛ n m   def |aij |p ⎠ . (1.4.46)

A p = a p = vec(A) p = ⎝ i=1 j=1

1.4 Inner Products and Vector Norms

39

Since this kind of matrix norm is represented by the matrix entries, it is named the entrywise norm. The following are three typical entrywise matrix norms: (1) 1 -norm (p = 1) def

A 1 =

n m  

|aij |;

(1.4.47)

i=1 j=1

(2) Frobenius norm (p = 2) ⎞1/2 ⎛ m  n  def |aij |2 ⎠ ;

A F = ⎝

(1.4.48)

i=1 j=1

(3) max norm or ∞ -norm (p = ∞)

A ∞ =

max

i=1,...,m; j=1,...,n

{|aij |}.

(1.4.49)

The Frobenius norm is an extension of the Euclidean norm of the vector to the elongated vector a = [a11 , . . . , am1 , a12 , . . . , a1n , . . . , amn ]T . The Frobenius norm can be also written in the form of the trace function as follows:  min{m,n}   def ! 1/2 H = tr(A A) = σi2 (A). (1.4.50)

A F = A, A i=1

Given an m × n matrix A, its Frobenius norm weighted by a positive definite matrix Ω, denoted A Ω , is defined by 

A Ω = tr(AH ΩA). (1.4.51) This norm is usually called the Mahalanobis  norm. min{m,n} 2 From A 2 = σmax (A) and A F = σi (A) one gets a relationship i=1 between the induced norm and the entrywise norm with p = 2:

A 2 ≤ A F ,

(1.4.52)

and the equality  holds if and only if the matrix A is a rank-1 matrix or a zero min{m,n} 2 σi (A) = σmax (A) in this case. matrix, since i=1 The Frobenius norm A F is very useful for numerical linear algebra and matrix analysis, since it is more easily calculated than the induced norm A 2 . 3. Schatten Norms The common Schatten norms are the Schatten p-norms defined by the vector of

40

Introduction to Matrix Algebra

the singular values of a matrix. If the singular values of a matrix A ∈ Cm×n are denoted by σi then the Schatten p-norms are defined by ⎞1/p ⎛ min{m,n}  σip ⎠ , p = 1, 2, ∞. (1.4.53)

A p = ⎝ i=1

The Schatten p-norms share the same notation as the induced and entrywise pnorms, but they are different. The case p = 2 yields the Frobenius norm introduced before. The most familiar Schatten p-norm is the Schatten norm with p = 1. In order to avoid confusion, this norm will be denoted by A ∗ or A tr , rather than A1 , and is called the nuclear norm (also known as the trace norm); it is defined as follows: min{m,n}  √

A ∗ = A tr = tr( AH A) = σi , (1.4.54) i=1

where tr(A) is the trace of the square matrix A. If A and B are m × n matrices then their matrix norms have the following properties:

A + B + A − B = 2( A 2 + B 2 ) (parallelogram law),

A + B A − B ≤ A + B . 2

2

(1.4.55) (1.4.56)

The following are the relationships between the inner products and norms [214]. (1) Cauchy–Schwartz inequlity 2

|A, B| ≤ A 2 B 2 .

(1.4.57)

The equals sign holds if and only if A = c B, where c is a complex constant. (2) Pythagoras’ theorem A, B = 0



A + B 2 = A 2 + B 2 .

(1.4.58)

(3) Polarization identity  1 (1.4.59)

A + B 2 − A − B 2 , 4  1 Re (A, B) = (1.4.60)

A + B 2 − A 2 − B 2 , 2 where Re (A, B) represents the real part of the inner product A, B. Re (A, B) =

1.5 Random Vectors In engineering applications, the measured data are usually random variables. A vector with random variables as its entries is called a random vector. In this section, we discuss the statistics and properties of random vectors by focusing on Gaussian random vectors.

1.5 Random Vectors

41

1.5.1 Statistical Interpretation of Random Vectors In the statistical interpretation of random vectors, the first-order and second-order statistics of random vectors are the most important. Given a random vector x(ξ) = [x1 (ξ), . . . , xm (ξ)]T , the mean vector of x(ξ), denoted μx , is defined as follows: ⎡ ⎤ ⎡ ⎤ E{x1 (ξ)} μ1 def ⎢ ⎥ ⎢ .. ⎥ .. μx = E{x(ξ)} = ⎣ (1.5.1) ⎦ = ⎣ . ⎦, . E{xm (ξ)}

μm

where E{xi (ξ)} = μi represents the mean of the random variable xi (ξ). The autocorrelation matrix of the random vector x(ξ) is defined by ⎤ ⎡ r11 · · · r1m   ⎢ def .. ⎥ , .. Rx = E x(ξ)xH (ξ) = ⎣ ... . . ⎦ rm1

···

(1.5.2)

rmm

where rii (i = 1, . . . , m) denotes the autocorrelation function of the random variable xi (ξ):     def (1.5.3) rii = E xi (ξ)x∗i (ξ) = E |xi (ξ)|2 , i = 1, . . . , m, whereas rij represents the cross-correlation function of xi (ξ) and xj (ξ):   def rij = E xi (ξ)x∗j (ξ) , i, j = 1, . . . , m, i = j.

(1.5.4)

Clearly, the autocorrelation matrix is a complex-conjugate-symmetric matrix (i.e., a Hermitian matrix). The autocovariance matrix of the random vector x(ξ), denoted Cx , is defined as follows:   def Cx = Cov(x, x) = E (x(ξ) − μx )(x(ξ) − μx )H ⎤ ⎡ c11 · · · c1m ⎢ .. ⎥ , .. (1.5.5) = ⎣ ... . . ⎦ cm1

···

cmm

where the diagonal entries,   def cii = E |xi (ξ) − μi |2 ,

i = 1, . . . , m,

(1.5.6)

represent the variance σi2 of the random variable xi (ξ), i.e., cii = σi2 , whereas the other entries,     def cij = E [xi (ξ) − μi ][xj (ξ) − μj ]∗ = E xi (ξ)x∗j (ξ) − μi μ∗j = c∗ji , (1.5.7) express the covariance of the random variables xi (ξ) and xj (ξ). The autocovariance matrix is also Hermitian.

42

Introduction to Matrix Algebra

The relationship between the autocorrelation and autocovariance matrices is given by Cx = R x − μ x μ H x .

(1.5.8)

By generalizing the autocorrelation and autocovariance matrices, one obtains the cross-correlation matrix of the random vectors x(ξ) and y(ξ), ⎤ ⎡ rx1 ,y1 · · · rx1 ,ym   def ⎢ .. ⎥ , .. Rxy = E x(ξ)yH (ξ) = ⎣ ... (1.5.9) . . ⎦ rxm ,y1 and the cross-covariance matrix def



Cxy = E [x(ξ) − μx ][y(ξ) − μy ]

···



 H

cx1 ,y1 ⎢ = ⎣ ... cxm ,y1

rxm ,ym ··· .. . ···

⎤ cx1 ,ym .. ⎥ , . ⎦

(1.5.10)

cxm ,ym

  def where rxi ,yj = E xi (ξ)yj∗ (ξ) is the cross-correlation of the random vectors xi (ξ) def

and yj (ξ) and cxi ,yj = E{[xi (ξ)−μxi ][yj (ξ) −μyj ]∗ } is the cross-covariance of xi (ξ) and yj (ξ). It is easily seen that there exists the following relationship between the crosscovariance and cross-correlation matrices: Cxy = Rxy − μx μH y .

(1.5.11)

In real-world applications, a data vector x = [x(0), x(1), . . . , x(N − 1)]T with nonzero mean μx usually needs to undergo a zero-mean normalization: x ← x = [x(0) − μx , x(1) − μx , . . . , x(N − 1) − μx ]T ,

μx =

N −1 1  x(n). N n=0

After zero-mean normalization, the correlation matrices and covariance matrices are equal, i.e., Rx = Cx and Rxy = Cxy . Some properties of these matrices are as follows. 1. The autocorrelation matrix is Hermitian, i.e., RH x = Rx . 2. The autocorrelation matrix of the linear combination vector y = Ax+b satisfies Ry = ARx AH . 3. The cross-correlation matrix is not Hermitian but satisfies RH xy = Ryx . 4. R(x1 +x2 )y = Rx1 y + Rx2 y . 5. If x and y have the same dimension, then Rx+y = Rx + Rxy + Ryx + Ry . 6. RAx,By = ARxy BH .

1.5 Random Vectors

43

The cross-correlation function describes the degree of correlation of two random variables xi (ξ) and xj (ξ). Generally speaking, the larger the autocorrelation function is, the greater is the degree of correlation of the two random vectors. The degree of correlation of two random variables x(ξ) and y(ξ) can be measured by their correlation coefficient, cxy E{(x(ξ) − x ¯)(y(ξ) − y¯)∗ } def = . ρxy =  2 2 σ E{|x(ξ) − x ¯| }E{|y(ξ) − y¯| } x σy

(1.5.12)

Here cxy = E{(x(ξ) − x ¯)(y(ξ) − y¯)∗ } is the cross-covariance of the random variables x(ξ) and y(ξ), while σx2 and σy2 are respectively the variances of x(ξ) and y(ξ). Applying the Cauchy–Schwartz inequality to Equation (1.5.12), we have 0 ≤ |ρxy | ≤ 1.

(1.5.13)

The correlation coefficient ρxy measures the degree of similarity of two random variables x(ξ) and y(ξ). The closer ρxy is to zero, the weaker the degree of similarity of the random variables x(ξ) and y(ξ) is. On the other hand, the closer ρxy is to 1, the more similar x(ξ) and y(ξ) are. In particular, the two extreme values 0 and 1 of correlation coefficients have interesting physical meanings. The case ρxy = 0 means that the cross-covariance cxy = 0, which implies there are no correlated components between the random variables x(ξ) and y(ξ). Thus, if ρxy = 0, the random variables x(ξ) and y(ξ) are said to be uncorrelated. Since this uncorrelation is defined in a statistical sense, it is usually said to be a statistical uncorrelation. It is easy to verify that if x(ξ) = c y(ξ), where c is a complex number, then |ρxy | = 1. Up to a fixed amplitude scaling factor |c| and a phase φ(c), the random variables x(ξ) and y(ξ) are the same, so that x(ξ) = cy(ξ) = |c|e jφ(c) y(ξ). Such a pair of random variables is said to be completely correlated or coherent. DEFINITION 1.27 Two random vectors x(ξ) = [x1 (ξ), . . . , xm (ξ)]T and y(ξ) = [y1 (ξ), . . . , yn (ξ)]T are said to be statistically uncorrelated if their cross-covariance matrix Cxy = Om×n or, equivalently, ρxi ,yj = 0, ∀ i, j. The random variables x(ξ) and y(ξ) are orthogonal if their cross-correlation is equal to zero, namely rxy = E{x(ξ)y ∗ (ξ)} = 0.

(1.5.14)

Similarly, two random vectors x(ξ) = [x1 (ξ), . . . , xm (ξ)]T and y(ξ) = [y1 (ξ), . . . , yn (ξ)]T are said to be orthogonal if any entry xi (ξ) of x(ξ) is orthogonal to any entry yj (ξ) of y(ξ), i.e., rxi ,yj = E{xi (ξ)yj (ξ)} = 0, i = 1, . . . , m, j = 1, . . . , n. Clearly, this implies that the cross-correlation matrix of the two random vectors is equal to the zero matrix, i.e., Rxy = Om×n . DEFINITION 1.28 The m × 1 random vector x(ξ) is said to be orthogonal to the n × 1 random vector y(ξ), if their cross-correlation matrix Rxy = Om×n .

44

Introduction to Matrix Algebra

Note that for the zero-mean normalized m × 1 random vector x(ξ) and n × 1 random vector y(ξ), their statistical uncorrelation and orthogonality are equivalent, as their cross-covariance and cross-correlation matrices are equal, i.e., Cxy = Rxy .

1.5.2 Gaussian Random Vectors DEFINITION 1.29 If each of its entries xi (ξ), i = 1, . . . , m, is a Gaussian random variable then the random vector x = [x1 (ξ), . . . , xm (ξ)]T is called a Gaussian normal random vector. As stated below, the representations of the probability density functions of real and complex Gaussian random vectors are slightly different. Let x ∼ N (¯ x, Γx ) denote a real Gaussian or normal random vector with the ¯ = [¯ ¯ )(x − x ¯ )T }. mean vector x x1 , . . . , x ¯m ]T and covariance matrix Γx = E{(x − x If each entry of the Gaussian random vector is independent identically distributed 2 ¯ )(x − x ¯ )T } = Diag(σ12 , . . . , σm (iid) then its covariance matrix Γx = E{(x − x ), 2 2 where σi = E{(xi − x¯i ) } is the variance of the Gaussian random variable xi . Under the condition that all entries are statistically independent of each other, the probability density function of a Gaussian random vector x ∼ N (¯ x, Γx ) is the joint probability density function of its m random variables, i.e., f (x) = f (x1 , . . . , xm ) = f (x1 ) · · · f (xm )     1 1 ¯ 1 )2 ¯ m )2 (x1 − x (xm − x = ···  exp − exp − 2 2 2σ12 2σm 2πσm 2πσ12   ¯ 1 )2 ¯ m )2 (x1 − x (xm − x 1 exp − − · · · − = 2 2σ12 2σm (2π)m/2 σ1 · · · σm or f (x) =

  1 1 T −1 ¯ ¯ exp − Γ (x − x ) . (x − x ) x 2 (2π)m/2 |Γx |1/2

(1.5.15)

If the entries are not statistically independent of each other, then the probability density function of the Gaussian random vector x ∼ N (¯ x, Γx ) is also given by Equation (1.5.15), but the exponential term becomes [372], [397] m m   ¯) = ¯ )T Γ−1 [Γ−1 (1.5.16) (x − x x (x − x x ]i,j (xi − μi )(xj − μj ), i=1 j=1

[Γ−1 x ]ij

where represents the (i, j)th entry of the inverse matrix Γ−1 x and μi = E{xi } is the mean of the random variable xi . The characteristic function of a real Gaussian random vector is given by   1 T T (1.5.17) Φ x (ω1 , . . . , ωm ) = exp j ω μx − ω Γx ω , 2

1.5 Random Vectors

45

where ω = [ω1 , . . . , ωm ]T . If xi ∼ CN (μi , σi2 ), then x = [x1 , . . . , xm ]T is called a complex Gaussian random vector, denoted x ∼ CN (μx , Γx ), where μx = [μ1 , . . . , μm ]T and Γ are respectively the mean vector and the covariance matrix of the random vector x. If xi = ui + jvi and the random vectors [u1 , v1 ]T , . . . , [um , vm ]T are statistically independent of each other then the probability density function of a complex Gaussian random vector x is given by [397, p. 35-5] −1    m m m ( (  1 m 2 2 f (xi ) = π σi exp − |xi − μi | f (x) = σ2 i=1 i=1 i=1 i * ) 1 = m (1.5.18) exp − (x − μx )H Γ−1 x (x − μx ) , π |Γx | 2 ). The characteristic function of the complex Gaussian where Γx = Diag(σ12 , . . . , σm random vector x is determined by   1 Φx (ω) = exp j Re(ω H μx ) − ω H Γx ω . (1.5.19) 4

A Gaussian random vector x has the following important properties. (1) The probability density function of x is completely described by its mean vector and covariance matrix. (2) If two Gaussian random vectors x and y are statistically uncorrelated then they are also statistically independent. (3) Given a Gaussian random vector x with mean vector μx and covariance matrix Γx , the random vector y obtained by the linear transformation y(ξ) = Ax(ξ) is also a Gaussian random vector, and its probability density function is given by   1 1 T −1 f (y) = exp − ) Γ (y − μ ) (1.5.20) (y − μ y y y 2 (2π)m/2 |Γy |1/2 for real Gaussian random vectors and * ) 1 exp − (y − μy )H Γ−1 f (y) = m y (y − μy ) π |Γy |

(1.5.21)

for complex Gaussian random vectors. In array processing, wireless communications, multiple-channel signal processing etc., one normally uses multiple sensors or array elements to receive multipath signals. In most cases it can be assumed that the additive noise in each sensor is a white Gaussian noise and that these white Gaussian noises are statistically uncorrelated.

46

Introduction to Matrix Algebra

EXAMPLE 1.9 Consider a real Gaussian noise vector x(t) = [x1 (t), . . . , xm (t)]T whose entries are all real Gaussian noise processes that are statistically uncorrelated. If these white Gaussian noises have the same variance σ 2 then + σ 2 , i = j, cxi ,xj = rxi ,xj = (1.5.22) 0, i = j, and thus the autocovariance matrix of x(t) is ⎡ rx ,x   ⎢ 1. 1 T Cx = Rx = E x(t)x (t) = ⎣ .. rxm ,x1

··· .. . ···

⎤ rx1 ,xm .. ⎥ = σ 2 I. . ⎦ rxm ,xm

Hence the statistical expression of a real white Gaussian noise vector is given by   (1.5.23) E{x(t)} = 0 and E x(t)xT (t) = σ 2 I. EXAMPLE 1.10 Consider the complex Gaussian random vector x(t) = [x1 (t), . . . , xm (t)]T whose components are complex white Gaussian noises and are statistically uncorrelated. If xi (t), i = 1, . . . , m, have zero mean and the same variance σ 2 then the real part xR,i (t) and the imaginary part xI,i (t) are two real white Gaussian noises that are statistically independent and have the same variance. This implies that     E xR,i (t) = 0, E xI,i (t) = 0,     1 E x2R,i (t) = E x2I,i (t) = σ 2 , 2   E xR,i (t)xI,i (t) = 0,       E xi (t)x∗i (t) = E x2R,i (t) + E x2I,i (t) = σ 2 . From the above conditions we know that     E x2i (t) = E (xR,i (t) + j xI,i (t))2       = E x2R,i (t) − E x2I,i (t) + j 2E xR,i (t)xI,i (t) 1 1 = σ 2 − σ 2 + 0 = 0. 2 2 Since x1 (t), . . . , xm (t) are statistically uncorrelated, we have     E xi (t)x∗k (t) = 0, i = k. E xi (t)xk (t) = 0, Summarizing the above conditions, we conclude that the statistical expression of the complex white Gaussian noise vector x(t) is given by   E x(t) = 0, (1.5.24)   H 2 (1.5.25) E x(t)x (t) = σ I,   T E x(t)x (t) = O. (1.5.26)

1.6 Performance Indexes of Matrices

47

Note that there is a difference between the statistical representations of the real and complex white Gaussian noise vectors. We will adopt the symbols x(t) ∼ N (0, σ 2 I) and x(t) ∼ CN (0, σ 2 I) to represent respectively the real and complex zero-mean white Gaussian noise vectors x(t).

1.6 Performance Indexes of Matrices An m × n matrix is a multivariate representation having mn components. In mathematics, one often needs a multivariate representation to be described by a scalar. The performance indexes of a matrix are such mathematical tools. In previous sections we have discussed two such performance indexes of matrices: the inner product and the norm. In this section, we present other several important performance indexes: the quadratic form, determinant, eigenvalues, trace and rank of a matrix.

1.6.1 Quadratic Forms The quadratic form of an n × n matrix A n × 1 nonzero vector. For example, ⎡ 1 xT Ax = [x1 , x2 , x3 ] ⎣−1 −1

is defined as xH Ax, where x may be any ⎤⎡ ⎤ 4 2 x1 7 5 ⎦ ⎣ x2 ⎦ x3 6 3

= x21 + 7x22 + 3x23 + 3x1 x2 + x1 x3 + 11x2 x3 . If x = [x1 , . . . , xn ]T , and the aij is the (i, j)th entry of the n × n matrix A, the quadratic form of A can be expressed as xT Ax =

n  n 

xi xj aij =

i=1 j=1

=

n  i=1

From this formula, it ⎡ 1 A = ⎣ −1 −1 ⎡ 1.0 C = ⎣ 1.5 0.5

aii x2i +

n  i=1

n−1 

n 

aii x2i +

n n  

aij xi xj

i=1, i=j j=1

(aij + aji )xi xj .

i=1 j=i+1

is easy to know that the matrices ⎤ ⎡ ⎤ 4 2 1 −1 −1 B = AT = ⎣ 4 7 5 ⎦, 7 6 ⎦, 6 3 2 5 3 ⎤ ⎡ ⎤ 1.5 0.5 1 114 52 7.0 5.5 ⎦ , D = ⎣ −111 7 2 ⎦, ... 5.5 3.0 −51 9 3

48

Introduction to Matrix Algebra

have the same quadratic form, i.e., xT Ax = xT Bx = xT Cx = xT Dx = x21 + 7x22 + 3x23 + 3x1 x2 + x1 x3 + 11x2 x3 . That is to say, for any quadratic form function f (x1 , . . . , xn ) =

n  i=1

αii x2i +

n n  

αij xi xj ,

i=1, i=j j=1

there exist many different matrices A such that their quadratic forms xT Ax = f (x1 , . . . , xn ) are the same. However, there is only one symmetric matrix AT = A satisfying xT Ax = f (x1 , . . . , xn ), whose entries are aij = aji = 12 (αij + αji ), where i = 1, . . . , n, j = 1, . . . , n. Hence, in order to ensure the uniqueness of the definition, it is necessary to assume A to be a real symmetric matrix or a complex Hermitian matrix when discussing the quadratic forms of an n × n matrix A. This assumption ensures that any quadratic form function is real-valued, since (xH Ax)∗ = (xH Ax)H = xH AH x = xH Ax holds for any Hermitian matrix A and any nonzero vector x. One of the basic advantages of a real-valued function is its suitability for comparison with a zero value. If the quadratic form xH Ax > 0 is positive definite then the Hermitian matrix A is also positive definite. Similarly, one can define the positive semi-definiteness, negative definiteness and negative semi-definiteness of a Hermitian matrix. DEFINITION 1.30

A Hermitian matrix A is said to be:

(1) positive definite, denoted A  0, if the quadratic form xH Ax > 0, ∀ x = 0; (2) positive semi-definite, denoted A  0, if the quadratic form xH Ax ≥ 0, ∀ x = 0; (3) negative definite, denoted A ≺ 0, if the quadratic form xH Ax < 0, ∀ x = 0; (4) negative semi-definite, denoted A  0, if the quadratic form xH Ax ≤ 0, ∀ x = 0; (5) indefinite if xH Ax > 0 for some nonzero vectors x and xH Ax < 0 for other nonzero vectors x. ⎡

For example, the real symmetric matrix R = ⎣

3 −1 0

−1 3 −1

⎤ 0 −1 ⎦ 3 2

is positive definite,

as the quadratic form xH Rx = 2x21 + x22 + 2x23 + (x1 − x2 ) + (x2 − x3 )2 > 0 unless x1 = x2 = x3 = 0. One sentence summary: As a performance index, the quadratic form of a Hermitian matrix describes its positive definiteness.

1.6 Performance Indexes of Matrices

49

1.6.2 Determinants The reader will recall that the determinant of a 2 × 2 matrix [aij ] is given by   a a12  def = a11 a22 − a12 a21 . det[aij ] =  11 a21 a22  The determinant of an n × n matrix A, denoted det(A) or |A|, is written as follows:    a11 a12 · · · a1n     a21 a22 · · · a2n    (1.6.1) det(A) = |A| =  . .. ..  . ..  .. . . .   a an2 · · · ann  n1 After removing the ith row and the jth column from the matrix A, the determinant Aij of the remaining matrix is known as the cofactor of the entry aij . In particular, when j = i, Aii is known as the principal minor of A. Letting Aij be the (n − 1) × (n − 1) submatrix obtained by removing the ith row and the jth column from the n × n matrix A, the cofactor Aij is related to the determinant of the submatrix Aij as follows: Aij = (−1)i+j det(Aij ).

(1.6.2)

The determinant of an n × n matrix A is ∀n equal to the sum of the products of each entry of its any row (say the ith row) or any column (say the jth column) by its corresponding cofactor, namely det(A) = ai1 Ai1 + · · · + ain Ain =

n 

aij (−1)i+j det(Aij ),

j=1 n 

det(A) = a1j A1j + · · · + anj Anj =

aij (−1)i+j det(Aij ).

(1.6.3)

(1.6.4)

i=1

Hence the determinant of A can be recursively calculated: an nth-order determinant can be computed from the (n − 1)th-order determinants, while each (n − 1)th-order determinant can be calculated from the (n − 2)th-order determinants, and so forth. For a 3 × 3 matrix A, its determinant is recursively given by ⎤ ⎡ a11 a12 a13 det(A) = det ⎣a21 a22 a23 ⎦ = a11 A11 + a12 A12 + a13 A13 a31 a32 a33       a    a23  a23  1+2 a21 1+3 a21 a22  + a + a = a11 (−1)1+1  22 (−1) (−1) 12 13     a32 a33 a31 a33 a31 a33  = a11 (a22 a33 − a23 a32 ) − a12 (a21 a33 − a23 a31 ) + a13 (a21 a33 − a22 a31 ). This is the diagonal method for a third-order determinant.

50

Introduction to Matrix Algebra

DEFINITION 1.31 matrix.

A matrix with nonzero determinant is known as a nonsingular

1. Determinant Equalities [307] 1. If two rows (or columns) of a matrix A are exchanged then the value of det(A) remains unchanged, but the sign is changed. 2. If some row (or column) of a matrix A is a linear combination of other rows (or columns), then det(A) = 0. In particular, if some row (or column) is proportional or equal to another row (or column), or there is a zero row (or column), then det(A) = 0. 3. The determinant of an identity matrix is equal to 1, i.e., det(I) = 1. 4. Any square matrix A and its transposed matrix AT have the same determinant, i.e., det(A) = det(AT ); however, det(AH ) = (det(AT ))∗ . 5. The determinant of a Hermitian matrix is real-valued, since det(A) = det(AH ) = (det(A))∗ .

(1.6.5)

6. The determinant of the product of two square matrices is equal to the product of their determinants, i.e., det(AB) = det(A) det(B),

A, B ∈ Cn×n .

(1.6.6)

7. For any constant c and any n × n matrix A, det(cA) = cn det(A). 8. If A is nonsingular then det(A−1 ) = 1/ det(A). 9. For matrices Am×m , Bm×n , Cn×m , Dn×n , the determinant of the block matrix for A nonsingular is   A B = det(A) det(D − CA−1 B) (1.6.7) det C D and for D nonsingular is   A B = det(D) det(A − BD−1 C). det C D

(1.6.8)

10. The determinant of a triangular (upper or lower triangular) matrix A is equal to the product of its main diagonal entries: det(A) =

n (

aii .

i=1

The determinant of a diagonal matrix A = Diag(a11 , . . . , ann ) is also equal to the product of its diagonal entries.

1.6 Performance Indexes of Matrices

51

Here we give a proof of Equation (1.6.7):      A B I A−1 B A O det = det I C D C D − CA−1 B O = det(A) det(D − CA−1 B). We can prove Equation (1.6.8) in a similar way. 2. Determinant Inequalities [307] 1. Cauchy–Schwartz inequality: If A, B are m × n matrices, then | det(AH B)|2 ≤ det(AH A) det(BH B). 2. Hadamard inequality: For an m × m matrix A, one has det(A) ≤

m ( i=1

⎛ ⎝

m 

⎞1/2 |aij |2 ⎠

.

j=1

3. Fisher inequality: For Am×m , Bm×n , Cn×n , one has   A B ≤ det(A) det(C). det BH C 4. Minkowski inequality: If Am×m = Om×m and Bm×m = Om×m are positive semi-definite then    m det(A + B) ≥ m det(A) + m det(B). 5. The determinant of a positive definite matrix A is larger than 0, i.e., det(A) > 0. 6. The determinant of a positive semi-definite matrix A is larger than or equal to 0, i.e., det(A) ≥ 0. 7. If the m×m matrix A is positive semi-definite, then (det(A))1/m ≤ m−1 det(A). 8. If the matrices Am×m and Bm×m are positive semi-definite then det(A + B) ≥ det(A) + det(B). 9. If Am×m is positive definite and Bm×m is positive semi-definite then det(A + B) ≥ det(A). 10. If Am×m is positive definite and Bm×m is negative semi-definite then det(A + B) ≤ det(A). One sentence summary: As a performance index, the value of the determinant of a matrix determines whether it is singular.

52

Introduction to Matrix Algebra

1.6.3 Matrix Eigenvalues Consider the output of a linear transformation L whose input is an n × 1 nonzero vector u. If the output is different from the input by a scale factor λ, i.e., Lu = λu,

u = 0,

(1.6.9)

then the scalar λ and the vector u are known as the eigenvalue and the corresponding eigenvector of the linear transformation L. Since Lu = λu implies that the input vector keeps its “direction” unchanged, the vector u must depict an inherent feature of the linear transformation L. This is the reason why the vector u is called the eigenvector of the linear transformation. In this sense, the eigenvalue λ can be regarded as the “gain” of the linear transformation L when u is inputted. When the linear transformation L takes the form of an n × n matrix A, the expression (1.6.9) can be extended to the definition of the eigenvalue and eigenvector of the matrix A: if the linear algebraic equation Au = λu

(1.6.10)

has a nonzero n × 1 solution vector u then the scalar λ is called an eigenvalue of the matrix A, and u is its eigenvector corresponding to λ. The matrix equation (1.6.10) can be written equivalently as (A − λI)u = 0.

(1.6.11)

Since the above equation holds for a nonzero vector, u, the only condition for it have a nonzero solution is that the determinant of the matrix A − λI is equal to zero, i.e., det(A − λI) = 0.

(1.6.12)

This equation is known as the characteristic equation of the matrix A. The characteristic equation (1.6.12) reflects the following facts: • If (1.6.12) holds for λ = 0 then det(A) = 0. This implies that as long as the matrix A has a zero eigenvalue, this matrix must be a singular matrix. • All the eigenvalues of a zero matrix are zero, and for any singular matrix there exists at least one zero eigenvalue. Clearly, if all n diagonal entries of an n × n singular matrix A contains a subtraction of the same scalar x = 0 that is not an eigenvalue of A then the matrix A − xI must be nonsingular, since |A − xI| = 0. Let eig(A) represent the eigenvalues of the matrix A. The basic properties of eigenvalues are listed below: 1. 2. 3. 4.

eig(AB) = eig(BA). An m × n matrix A has at most min{m, n} different eigenvalues. If rank(A) = r, then the matrix A has at most r different eigenvalues. The eigenvalues of the inverse matrix satisfy eig(A−1 ) = 1/eig(A).

1.6 Performance Indexes of Matrices

53

5. Let I be the identity matrix; then

PROPOSITION 1.3 values.

eig(I + cA) = 1 + c eig(A),

(1.6.13)

eig(A − cI) = eig(A) − c.

(1.6.14)

All eigenvalues of a positive definite matrix are positive real

Proof Suppose that A is a positive definite matrix; then the quadratic form xH Ax > 0 holds for any nonzero vector x. If λ is any eigenvalue of A, i.e., Au = λu, then uH Au = uH λu and thus λ = uH Au/(uH u) must be a positive real number, as it is the ratio of two positive real numbers. Since uH u > 0 for any nonzero vector u, from λ = uH Au/(uH u) it is directly known that positive definite and nonpositive definite matrices can be described by their eigenvalues as follows. (1) (2) (3) (4) (5)

Positive definite matrix Its eigenvalues are positive real numbers. Positive semi-definite matrix Its eigenvalues are nonnegative. Negative definite matrix Its eigenvalues are negative. Negative semi-definite matrix Its eigenvalues are nonpositive. Indefinite matrix It has both positive and negative eigenvalues.

If A is a positive definite or positive semi-definite matrix then ( aii . det(A) ≤

(1.6.15)

i

This inequality is called the Hadamard inequality [214, p. 477]. The characteristic equation (1.6.12) suggests two methods for improving the numerical stability and the accuracy of solutions of the matrix equation Ax = b, as described below. 1. Method for Improving the Numerical Stability Consider the matrix equation Ax = b, where A is usually positive definite or nonsingular. However, owing to noise or errors, A may sometimes be close to singular. We can alleviate this difficulty as follows. If λ is a small positive number then −λ cannot be an eigenvalue of A. This implies that the characteristic equation |A − xI| = |A − (−λ)I| = |A + λI| = 0 cannot hold for any λ > 0, and thus the matrix A + λI must be nonsingular. Therefore, if we solve (A + λI)x = b instead of the original matrix equation Ax = b, and λ takes a very small positive value, then one can overcome the singularity of A to improve greatly the numerical stability of solving Ax = b. This method of solving (A + λI)x = b, with λ > 0, instead of Ax = b is the well-known Tikhonov regularization method for solving nearly singular matrix equations.

54

Introduction to Matrix Algebra

2. Method for Improving the Accuracy For a matrix equation Ax = b, with the data matrix A nonsingular but containing additive interference or observation noise, if we choose a very small positive scalar λ to solve (A − λI)x = b instead of Ax = b then the influence of the noise of the data matrix A on the solution vector x will be greatly decreased. This is the basis of the well-known total least squares (TLS) method. We will present the Tikhonov method and the TLS method in detail in Chapter 6. As is seen in the Tikhonov and TLS methods, the diagonal entries of a matrix play a more important role in matrix analysis than its off-diagonal entries. One sentence summary: As a performance index, the eigenvalues of a matrix describe its singularness, positive definiteness and the special structure of its diagonal entries.

1.6.4 Matrix Trace DEFINITION 1.32 The sum of the diagonal entries of an n × n matrix A is known as its trace, denoted tr(A): tr(A) = a11 + · · · + ann =

n 

aii .

(1.6.16)

i=1

The following are some properties of the matrix trace. 1. Trace Equality [307] 1. If both A and B are n × n matrices then tr(A ± B) = tr(A) ± tr(B). 2. If both A and B are n × n matrices and c1 and c2 are constants then tr(c1 A ± c2 B) = c1 tr(A) ± c2 tr(B). In particular, tr(cA) = c tr(A). 3. tr(AT ) = tr(A), tr(A∗ ) = (tr(A))∗ and tr(AH ) = (tr(A))∗ . 4. If A ∈ Cm×n , B ∈ Cn×m then tr(AB) = tr(BA). 5. If A is an m × n matrix then tr(AH A) = 0 implies that A is an m × n zero matrix. 6. xH Ax = tr(AxxH ) and yH x = tr(xyH ). 7. The trace of an n × n matrix is equal to the sum of its eigenvalues, namely tr(A) = λ1 + · · · + λn . 8. The trace of a block matrix satisfies   A B tr = tr(A) + tr(D), C D where A ∈ Cm×m , B ∈ Cm×n , C ∈ Cn×m and D ∈ Cn×n .

1.6 Performance Indexes of Matrices

55

9. For any positive integer k, we have tr(Ak ) =

n 

λki .

(1.6.17)

i=1

By the trace equality tr(UV) = tr(VU), it is easy to see that tr(AH A) = tr(AAH ) =

n n  

aij a∗ij =

i=1 j=1

n n  

|aij |2 .

(1.6.18)

i=1 j=1

However, if we substitute U = A, V = BC and U = AB, V = C into the trace equality tr(UV) = tr(VU), we obtain tr(ABC) = tr(BCA) = tr(CAB).

(1.6.19)

Similarly, if we let U = A, V = BCD or U = AB, V = CD or U = ABC, V = D, respectively, we obtain tr(ABCD) = tr(BCDA) = tr(CDAB) = tr(DABC).

(1.6.20)

Moreover, if A and B are m × m matrices, and B is nonsingular then tr(BAB−1 ) = tr(B−1 AB) = tr(ABB−1 ) = tr(A).

(1.6.21)

2. Trace Inequality [307] 1. If A ∈ Cm×n then tr(AH A) = tr(AAH ) ≥ 0. 2. Schur inequality: tr(A2 ) ≤ tr(AT A). 3. If A, B are two m × n matrices then





tr (AT B)2 ≤ tr(AT A)tr(BT B)



T

2





T

2



tr (A B) tr (A B)



(Cauchy–Schwartz inequality),

≤ tr(A AB B), T

T

≤ tr(AAT BBT ).







4. tr (A + B)(A + B)T ≤ 2 tr(AAT ) + tr(BBT ) . 5. If A and B are two m × m symmetric matrices then tr(AB) ≤ 12 tr(A2 + B2 ). The Frobenius norm of an m × n matrix A can be also defined using the traces of the m × m matrix AH A or that of the n × n matrix AAH , as follows [311, p. 10]:   (1.6.22)

A F = tr(AH A) = tr(AAH ). One sentence summary: As a performance index, the trace of a matrix reflects the sum of its eigenvalues.

56

Introduction to Matrix Algebra

1.6.5 Matrix Rank THEOREM 1.6 [433] Among a set of p-dimensional (row or column) vectors, there are at most p linearly independent (row or column) vectors. THEOREM 1.7 [433] For an m × n matrix A, the number of linearly independent rows and the number of linearly independent columns are the same. From this theorem we have the following definition of the rank of a matrix. DEFINITION 1.33 The rank of an m × n matrix A is defined as the number of its linearly independent rows or columns. It needs to be pointed out that the matrix rank gives only the number of linearly independent rows or columns; it gives no information on the locations of these independent rows or columns. The matrix equation Am×n xn×1 = bm×1 is said to be consistent, if it has at least one exact solution. A matrix equation with no exact solution is said to be inconsistent. Matrix equations can be divided into three types. (1) Well-determined equation If m = n and rank(A) = n, i.e., the matrix A is nonsingular, then the matrix equation Ax = b is said to be well-determined. (2) Under-determined equation The matrix equation Ax = b is said to be underdetermined if the number of linearly independent equations is less than the number of independent unknowns. (3) Over-determined equation The matrix equation Ax = b is said to be overdetermined if the number of linearly independent equations is larger than the number of independent unknowns. The terms “well-determined”, “under-determined” and “over-determined” have the following meanings. • Meaning of well-determined equation The number of independent equations and the number of independent unknowns are the same so that the solution of this system of equations is uniquely determined. The exact solution of a well-determined matrix equation Ax = b is given by x = A−1 b. A well-determined equation is a consistent equation. • Meaning of under-determined equation The number of independent equations is less than the number of independent unknowns, which implies that the number of equations is not enough for determining a unique solution. As a matter of fact, such a system of linear equations has an infinitely many solutions. Hence, any under-determined matrix equation is a consistent equation. • Meaning of over-determined equation Since the number of independent equations is larger than the number of independent unknowns, the number of independent equations appears surplus for determining the unique solution. An

1.6 Performance Indexes of Matrices

57

over-determined matrix equation Ax = b has no exact solution and thus is an inconsistent equation that may in some cases have an approximate solution, for example, a least squares solution. A matrix A with rank(A) = rA has rA linearly independent column vectors. The linear combinations of the rA linearly independent column vectors constitute a vector space, called the column space or the range or the manifold of A. The column space Col(A) or the range Range(A) is rA -dimensional. Hence the rank of a matrix can be defined by using the dimension of its column space or range, as described below. DEFINITION 1.34 The dimension of the column space Col(A) or the range Range(A) of an m × n matrix A is defined as the rank of the matrix, namely rA = dim(Col(A)) = dim(Range(A)).

(1.6.23)

The following statements about the rank of the matrix A are equivalent: (1) rank(A) = k; (2) there are k and not more than k columns of A that combine to give a linearly independent set; (3) there are k and not more than k rows of A that combine to give a linearly independent set; (4) there is a k × k submatrix of A with nonzero determinant, but all the (k + 1) × (k + 1) submatrices of A have zero determinant; (5) the dimension of the column space Col(A) or the range Range(A) equals k; (6) k = n − dim[Null(A)], where Null(A) denotes the null space of the matrix A. THEOREM 1.8 [433]

The rank of the product matrix AB satisfies the inequality rank(AB) ≤ min{rank(A), rank(B)}.

(1.6.24)

LEMMA 1.1 If premultiplying an m×n matrix A by an m×m nonsingular matrix P, or postmultiplying it by an n × n nonsingular matrix Q, then the rank of A is not changed, namely rank(PAQ) = rank(A). LEMMA 1.2

rank[A, B] ≤ rank(A) + rank(B).

LEMMA 1.3

rank(A + B) ≤ rank[A, B] ≤ rank(A) + rank(B).

LEMMA 1.4 For an m × n matrix A and an n × q matrix B, the rank inequality rank(AB) ≥ rank(A) + rank(B) − n is true. 1. Properties of the Rank of a Matrix 1. The rank is a positive integer. 2. The rank is equal to or less than the number of columns or rows of the matrix.

58

Introduction to Matrix Algebra

3. If the rank of an n × n matrix A is equal to n then A is nonsingular, or we say that A is a full rank matrix. 4. If rank(Am×n ) < min{m, n} then A is said to be a rank-deficient matrix. 5. If rank(Am×n ) = m (< n) then the matrix A is a full row rank matrix. 6. If rank(Am×n ) = n (< m) then the matrix A is a full column rank matrix. 7. Premultiplying any matrix A by a full column rank matrix or postmultiplying it by a full row rank matrix leaves the rank of the matrix A unchanged. 2. Rank Equalities (1) If A ∈ Cm×n , then rank(AH ) = rank(AT ) = rank(A∗ ) = rank(A). (2) If A ∈ Cm×n and c = 0, then rank(cA) = rank(A). (3) If A ∈ Cm×m and C ∈ Cn×n are nonsingular then rank(AB) = rank(B) = rank(BC) = rank(ABC) for B ∈ Cm×n . That is, after premultiplying and/or postmultiplying by a nonsingular matrix, the rank of B remains unchanged. (4) For A, B ∈ Cm×n , rank(A) = rank(B) if and only if there exist nonsingular matrices X ∈ Cm×m and Y ∈ Cn×n such that B = XAY. (5) rank(AAH ) = rank(AH A) = rank(A). (6) If A ∈ Cm×m then rank(A) = m ⇔ det(A) = 0 ⇔ A is nonsingular. 3. Rank Inequalities • rank(A) ≤ min{m, n} for any m × n matrix A. • If A, B ∈ Cm×n then rank(A + B) ≤ rank(A) + rank(B). • If A ∈ Cm×k and B ∈ Ck×n then rank(A) + rank(B) − k ≤ rank(AB) ≤ min{rank(A), rank(B)}. One sentence summary: As a performance index, the matrix rank describes the linear independence of the rows (or columns) of a matrix, which reflects the full rank or rank deficiency of the matrix. Table 1.1 summarizes the five important performance indexes discussed above and how they describe matrix performance. Table 1.1 Performance indexes of matrices Performance index

Matrix property determined by the performance index

quadratic form

positive definiteness and non-negative definiteness

determinant

singularity

eigenvalues

singularity and positive definiteness

trace

sum of diagonal entries, sum of eigenvalues

rank

linear independence of rows (or columns)

1.7 Inverse Matrices and Pseudo-Inverse Matrices

59

1.7 Inverse Matrices and Pseudo-Inverse Matrices Matrix inversion is an important aspect of matrix calculus. In particular, the matrix inversion lemma is often used in signal processing, system sciences, automatic control, neural networks and so on. In this section we discuss the inverse of a full-rank square matrix and the pseudo-inverse of a non-square matrix with full row (or full column) rank. Regarding the inversion of a nonsquare or rank-deficient matrix, we will discuss this in the next section.

1.7.1 Definition and Properties of Inverse Matrices An n×n matrix is called nonsingular if it has n linearly independent column vectors and n linearly independent row vectors. A nonsingular matrix can also be defined from the viewpoint of a linear system: a linear transformation or a square matrix A is said to be nonsingular if it produces a zero output only for zero input; otherwise it is singular. If a matrix is nonsingular then its inverse must exist. Conversely, a singular matrix has no inverse. The n × n matrix B such that BA = AB = I is called the inverse matrix of A, denoted B = A−1 . If A−1 exists then the matrix A is said to be nonsingular or invertible. On the nonsingularity or invertibility of an n × n matrix A, the following statements are equivalent [214]. (1) A is nonsingular. (2) A−1 exists. (3) rank(A) = n. (4) All rows of A are linearly independent. (5) All columns of A are linearly independent. (6) det(A) = 0. (7) The dimension of the range of A is n. (8) The dimension of the null space of A is equal to zero. (9) Ax = b is a consistent equation for every b ∈ Cn . (10) Ax = b has a unique solution for every b. (11) Ax = 0 has only the trivial solution x = 0. 1. Properties of the Inverse Matrix A−1 [25], [214] 1. A−1 A = AA−1 = I. 2. A−1 is unique. 3. The determinant of the inverse matrix is equal to the reciprocal of the determinant of the original matrix, i.e., |A−1 | = 1/|A|. 4. The inverse matrix A−1 is nonsingular.

60

Introduction to Matrix Algebra

5. The inverse matrix of an inverse matrix is the original matrix, i.e., (A−1 )−1 = A. 6. The inverse matrix of a Hermitian matrix A = AH satisfies (AH )−1 = (A−1 )H = A−H . 7. If AH = A then (A−1 )H = A−1 . That is to say, the inverse matrix of any Hermitian matrix is a Hermitian matrix as well. 8. (A∗ )−1 = (A−1 )∗ . 9. If A and B are invertible then (AB)−1 = B−1 A−1 . 10. If A = Diag(a1 , . . . , am ) is a diagonal matrix then its inverse matrix −1 A−1 = Diag(a−1 1 , . . . , am ).

11. Let A be nonsingular. If A is an orthogonal matrix then A−1 = AT , and if A is a unitary matrix then A−1 = AH .

1.7.2 Matrix Inversion Lemma LEMMA 1.5 Let A be an n×n invertible matrix, and x and y be two n×1 vectors such that (A + xyH ) is invertible; then (A + xyH )−1 = A−1 −

A−1 xyH A−1 . 1 + yH A−1 x

(1.7.1)

Lemma 1.5 is called the matrix inversion lemma, and was presented by Sherman and Morrison [438], [439] in 1949 and 1950. The matrix inversion lemma can be extended to an inversion formula for a sum of matrices: (A + UBV)−1 = A−1 − A−1 UB(B + BVA−1 UB)−1 BVA−1 = A−1 − A−1 U(I + BVA−1 U)−1 BVA−1

(1.7.2)

(A − UV)−1 = A−1 + A−1 U(I − VA−1 U)−1 VA−1 .

(1.7.3)

or

The above formula was obtained by Woodbury in 1950 [513] and is called the Woodbury formula. Taking U = u, B = β and V = vH , the Woodbury formula gives the result (A + βuvH )−1 = A−1 −

β A−1 uvH A−1 . 1 + βvH A−1 u

(1.7.4)

In particular, if we let β = 1 then Equation (1.7.4) reduces to formula (1.7.1), the matrix inversion lemma of Sherman and Morrison. As a matter of fact, before Woodbury obtained the inversion formula (1.7.2),

1.7 Inverse Matrices and Pseudo-Inverse Matrices

61

Duncan [139] in 1944 and Guttman [192] in 1946 had obtained the following inversion formula: (A − UD−1 V)−1 = A−1 + A−1 U(D − VA−1 U)−1 VA−1 .

(1.7.5)

This formula is called the Duncan–Guttman inversion formula [391], [392]. In addition to the Woodbury formula, the inverse matrix of a sum of matrices also has the following forms [206]: (A + UBV)−1 = A−1 − A−1 (I + UBVA−1 )−1 UBVA−1 =A

−1

=A

−1

=A

−1

−A

−1

−A

−1

−A

−1

−1

UB(I + VA

UBV(I + A UBVA

−1

UB)

−1

−1

UBV)

(1.7.6)

−1

(1.7.7)

VA −1

A

−1

−1 −1

(I + UBVA

)

(1.7.8)

.

(1.7.9)

The following are inversion formulas for block matrices. When the matrix A is invertible, one has [22]: −1 A U V D  −1 A + A−1 U(D − VA−1 U)−1 VA−1 = −(D − VA−1 U)−1 VA−1



 −A−1 U(D − VA−1 U)−1 . (D − VA−1 U)−1 (1.7.10)

If the matrices A and D are invertible, then [216], [217] 

A V

U D

−1



(A − UD−1 V)−1 = −1 −D V(A − UD−1 V)−1

−A−1 U(D − VA−1 U)−1 (D − VA−1 U)−1

 (1.7.11)

or [139] 

A V

U D

−1

 =

(A − UD−1 V)−1 −(D − VA−1 U)−1 VA−1

 −(A − UD−1 V)−1 UD−1 . (1.7.12) (D − VA−1 U)−1

1.7.3 Inversion of Hermitian Matrices Consider the inversion of a (m + 1) × (m + 1) nonsingular Hermitian matrix Rm+1 . First, an (m + 1) × (m + 1) Hermitian matrix can always be written in a block form:   Rm r m Rm+1 = H , (1.7.13) rm ρm where ρm is the (m + 1, m + 1)th entry of Rm+1 , and Rm is an m × m Hermitian matrix.

62

Introduction to Matrix Algebra

Now, we consider how to compute the inverse matrix R−1 m+1 using the inverse −1 matrix Rm . For this purpose, let   Qm qm (1.7.14) Qm+1 = qH αm m be the inverse matrix of Rm+1 . Then we have     I R m rm Q m q m = m Rm+1 Qm+1 = H H rm ρm qH α 0 m m m

0m 1

 (1.7.15)

which gives the following four equations: Rm Qm + r m qH m = Im , rH m Qm

(1.7.16)

0H m,

(1.7.17)

R m q m + r m α m = 0m ,

(1.7.18)

rH m qm

+

ρm qH m

=

+ ρm αm = 1.

(1.7.19)

If Rm is invertible, then from Equation (1.7.18), it follows that qm = −αm R−1 m rm .

(1.7.20)

Substitute this result into Equation (1.7.19) to yield αm =

1 . −1 ρm − rH m Rm r m

(1.7.21)

After substituting Equation (1.7.21) into Equation (1.7.20), we can obtain qm =

−R−1 m rm . H ρm − rb R−1 m rm

(1.7.22)

Then, substitute Equation (1.7.22) into Equation (1.7.16): −1 H −1 Qm = R−1 m − Rm r m q m = Rm +

−1 H R−1 m rm (Rm rm ) . −1 ρm − rH m R m rm

(1.7.23)

In order to simplify (1.7.21)–(1.7.23), put bm = [b0 , b1 , . . . , bm−1 ]T = −R−1 m rm , def

(m)

def

βm = ρm −

(m)

−1 rH m Rm r m

(m)

= ρm +

rH m bm .

(1.7.24) (1.7.25)

Then (1.7.21)–(1.7.23) can be respectively simplified to αm =

1 , βm

qm =

1 bm , βm

Qm = R−1 m +

1 bm bH m. βm

Substituting the these results into (1.7.15), it is immediately seen that     −1 1 b m bH Rm 0m bm −1 m . + Rm+1 = Qm+1 = bH 1 0H 0 βm m m

(1.7.26)

1.7 Inverse Matrices and Pseudo-Inverse Matrices

63

This formula for calculating the (m + 1) × (m + 1) inverse Rm+1 from the m × m inverse Rm is called the block inversion lemma for Hermitian matrices [352]. Given an M × M nonsingular Hermitian matrix H, using the block inversion lemma for m = 1, . . . , M − 1 we can find the inverse matrix H−1 in a recursive way.

1.7.4 Left and Right Pseudo-Inverse Matrices From a broader perspective, any n × m matrix G may be called the inverse of a given m × n matrix A, if the product of G and A is equal to the identity matrix I. According to the different possible structures of the m × n matrix A, for the matrix G to be such that the product of it and A is equal to the identity matrix there exist three possibilities as shown in the next example. EXAMPLE 1.11 Consider the inverses of the following three matrices, ⎡ ⎤ ⎡ ⎤   2 −2 −1 4 8 1 3 1 . A1 = ⎣ 1 1 −2 ⎦ , A2 = ⎣ 5 −7 ⎦ , A3 = 2 5 1 1 0 −1 −2 3 For the matrix A1 , there is a unique matrix ⎡ ⎤ −1 −2 5 G = ⎣ −1 −1 3 ⎦ −1 −2 4 such that GA1 = A1 G = I3×3 . In this case, the matrix G is actually the inverse matrix of A1 , i.e., G = A−1 1 . In the case of the matrix A2 , there exist an infinite number of 2 × 3 matrices L such that LA2 = I2×2 , for example  7    2 0 0 3 7 68 17 , ... L= , L= 0 2 5 0 2 5 For the matrix A3 , there are no 3 × 2 matrices G3 such that G3 A3 = I3×3 but there exist more than one 3 × 2 matrix R such that A3 R = I2×2 , for example ⎡ ⎤ ⎡ ⎤ 1 1 −1 1 R = ⎣ −1 0 ⎦, R = ⎣ 0 0 ⎦, ... 3 −1 2 −1 Summarizing the above discussion, we see that, in addition to the inverse matrix A satisfying AA−1 = A−1 A = I, there are two other forms of inverse matrix that satisfy only LA = I or AR = I. −1

DEFINITION 1.35 [433] The matrix L satisfying LA = I but not AL = I is called the left inverse of the matrix A. Similarly, the matrix R satisfying AR = I but not RA = I is said to be the right inverse of A.

64

Introduction to Matrix Algebra

Remark A matrix A ∈ Cm×n has a left inverse only when m ≥ n and a right inverse only when m ≤ n, respectively. As shown in Example 1.11, for a given m × n matrix A, when m > n it is possible that other n × m matrices L to satisfy LA = In , while when m < n it is possible that other n × m matrices R to satisfy AR = Im . That is to say, the left or right inverse of a given matrix A is usually not unique. Let us consider the conditions for a unique solution of the left and right inverse matrices. Let m > n and let the m × n matrix A have full column rank, i.e., rank(A) = n. In this case, the n × n matrix AH A is invertible. It is easy to verify that L = (AH A)−1 AH

(1.7.27)

satisfies LA = I but not AL = I. This type of left inverse matrix is uniquely determined, and is usually called the left pseudo-inverse matrix of A. On the other hand, if m < n and A has the full row rank, i.e., rank(A) = m, then the m × m matrix AAH is invertible. Define R = AH (AAH )−1 .

(1.7.28)

It is easy to see that R satisfies AR = I rather than RA = I. The matrix R is also uniquely determined and is usually called the right pseudo-inverse matrix of A. The left pseudo-inverse matrix is closely related to the least squares solution of overdetermined equations, while the right pseudo-inverse matrix is closely related to the least squares minimum-norm solution of under-determined equations. A detailed discussion is given in Chapter 6. The following are the dimensional recursions for the left and right pseudo-inverse matrices [542]. Consider an n × m matrix Fm (where n > m) and its left pseudo-inverse matrix −1 H F†m = (FH Fm . Let Fm = [Fm−1 , fm ], where fm is the mth column of the m Fm ) matrix Fm and rank(Fm ) = m; then a recursive computation formula for F†m is given by ⎤ ⎡ † −1 Fm−1 − F†m−1 fm eH m Δm ⎦, F†m = ⎣ (1.7.29) H −1 em Δm H −1 ; the initial recursion value where em = (In − Fm−1 F†m−1 )fm and Δ−1 m = (fm em ) † H H is F1 = f1 /(f1 f1 ). Given a matrix Fm ∈ Cn×m , where n < m, and again writing Fm = [Fm−1 , fm ], H −1 the right pseudo-inverse matrix F†m = FH has the following recursive m (Fm Fm ) formula: ⎡ † ⎤ Fm−1 − Δm F†m−1 fm cm ⎦. F†m = ⎣ (1.7.30) Δm cH m

1.8 Moore–Penrose Inverse Matrices

65

† H H Here cH m = fm (In − Fm−1 Fm−1 ) and Δm = cm fm . The initial recursion value is F†1 = f1H /(f1H f1 ).

1.8 Moore–Penrose Inverse Matrices In the above section we discussed the inverse matrix A−1 of a nonsingular square H −1 H matrix A, the left pseudo-inverse matrix A−1 A of an m × n (m > L = (A A) n) matrix A with full column rank, and the right pseudo-inverse matrix A−1 R = AH (AAH )−1 of an m × n (m < n) matrix A with full row rank. A natural question to ask is: does there exist an inverse matrix for an m × n rank-deficient matrix? If such a inverse matrix exists, what conditions should it meet?

1.8.1 Definition and Properties Consider an m × n rank-deficient matrix A, regardless of the size of m and n but with rank(A) = k < min{m, n}. The inverse of an m × n rank-deficient matrix is said to be its generalized inverse matrix and is an n × m matrix. Let A† denote the generalized inverse of the matrix A. From the matrix rank property rank(AB) ≤ min{rank(A), rank(B)}, it follows that neither AA† = Im×m nor A† A = In×n hold, because either the m × m matrix AA† or the n × n matrix A† A is rank-deficient, the maximum of their ranks is rank(A) = k and is less than min{m, n}. Since AA† = Im×m and A† A = In×n , it is necessary to consider using a product of three matrices to define the generalized inverse of a rank-deficient matrix A. For this purpose, let us consider solving the linear matrix equation Ax = y. If A† is the generalized inverse matrix of A then Ax = y ⇒ x = A† y. Substituting x = A† y into Ax = y, we have AA† y = y, and thus AA† Ax = Ax. Since this equation should hold for any nonzero vector x, the following condition should essentially be satisfied: AA† A = A.

(1.8.1)

Unfortunately, the matrix A† meeting the condition (1.8.1) is not unique. To overcome this difficulty we must add other conditions. The condition AA† A = A in (1.8.1) can only guarantee that A† is a generalized inverse matrix of A; it cannot conversely guarantee that A is also a generalized inverse matrix of A† , whereas A and A† should essentially be mutually generalized inverses. This is one of the main reasons why the definition AA† A = A does not uniquely determine A† . Now consider the solution equation of the original matrix equation Ax = y, i.e., x = A† y. Our problem is: given any y = 0, find x. Clearly, x = A† y can be written as x = A† Ax, yielding A† y = A† AA† y. Since A† y = A† AA† y should hold for

66

Introduction to Matrix Algebra

any nonzero vector y, the condition A† AA† = A†

(1.8.2)

must be satisfied as well. This shows that if A† is the generalized inverse of the rank-deficient matrix A, then the two conditions in Equations (1.8.1) and (1.8.2) must be satisfied at the same time. However, these two conditions are still not enough for a unique definition of A† . If an m × n matrix A is of full column rank or full row rank, we certainly hope that the generalized inverse matrix A† will include the left and right pseudoinverse matrices as two special cases. Although the left pseudo-inverse matrix L = (AH A)−1 AH of the m × n full column rank matrix A is such that LA = In×n rather than AL = Im×m , it is clear that AL = A(AH A)−1 AH = (AL)H is an m × m Hermitian matrix. Coincidentally, for any right pseudo-inverse matrix R = AH (AAH )−1 , the matrix product RA = AH (AAH )−1 A = (RA)H is an n×n Hermitian matrix. In other words, in order to guarantee that A† exists uniquely for any m × n matrix A, the following two conditions must be added: AA† = (AA† )H , †



H

A A = (A A) .

(1.8.3) (1.8.4)

On the basis of the conditions (1.8.1)–(1.8.4), one has the following definition of A† . DEFINITION 1.36 [384] Let A be any m × n matrix; an n × m matrix A† is said to be the Moore–Penrose inverse of A if A† meets the following four conditions (usually called the Moore–Penrose conditions): (a) AA† A = A; (b) A† AA† = A† ; (c) AA† is an m × m Hermitian matrix, i.e., AA† = (AA† )H ; (d) A† A is an n × n Hermitian matrix, i.e., A† A = (A† A)H . Remark 1 From the projection viewpoint, Moore [330] showed in 1935 that the generalized inverse matrix A† of an m × n matrix A must meet two conditions, but these conditions are not convenient for practical use. After two decades, Penrose [384] in 1955 presented the four conditions (a)–(d) stated above. In 1956, Rado [407] showed that the four conditions of Penrose are equivalent to the two conditions of Moore. Therefore the conditions (a)–(d) are called the Moore–Penrose conditions, and the generalized inverse matrix satisfying the Moore–Penrose conditions is referred to as the Moore–Penrose inverse of A. Remark 2

In particular, the Moore–Penrose condition (a) is the condition that

1.8 Moore–Penrose Inverse Matrices

67

the generalized inverse matrix A† of A must meet, while (b) is the condition that the generalized inverse matrix A = (A† )† of the matrix A† must meet. Depending on to what extent the Moore–Penrose conditions are met, the generalized inverse matrix can be classified as follows [174]. (1) The generalized inverse matrix A† satisfying all four conditions is the Moore– Penrose inverse of A. (2) The matrix G = A† satisfying conditions (a) and (b) is said to be the selfreflexive generalized inverse of A. (3) The matrix A† satisfying conditions (a), (b) and (c) is referred to as the regularized generalized inverse of A. (4) The matrix A† satisfying conditions (a), (b) and (d) is called the weak generalized inverse of A. It is easy to verify that the inverse matrices and the various generalized inverse matrices described early are special examples of the Moore–Penrose inverse matrices. • The inverse matrix A−1 of an n × n nonsingular matrix A meets all four Moore– Penrose conditions. • The left pseudo-inverse matrix (AH A)−1 AH of an m × n (m > n) matrix A meets all four Moore–Penrose conditions. • The right pseudo-inverse matrix AH (AAH )−1 of an m × n (m < n) matrix A meets all four Moore–Penrose conditions. • The left inverse matrix Ln×m such that LAm×n = In is a weak generalized inverse matrix satisfying the Moore–Penrose conditions (a), (b) and (d). • The right inverse matrix Rn×m such that Am×n R = Im is a regularized generalized inverse matrix satisfying the Moore–Penrose conditions (a), (b) and (c). The inverse matrix A−1 , the left pseudo-inverse matrix (AH A)−1 AH and the right pseudo-inverse matrix AH (AAH )−1 are uniquely determined, respectively. Similarly, any Moore–Penrose inverse matrix is uniquely determined as well. The Moore–Penrose inverse of any m × n matrix A can be uniquely determined by [51] A† = (AH A)† AH

(if m ≥ n)

(1.8.5)

A† = AH (AAH )†

(if m ≤ n).

(1.8.6)

or [187]

From Definition 1.36 it is easily seen that both A† = (AH A)† AH and A† = AH (AAH )† meet the four Moore–Penrose conditions. From [307], [311], [400], [401] and [410], the Moore–Penrose inverse matrices A† have the following properties.

68

Introduction to Matrix Algebra

1. For an m × n matrix A, its Moore–Penrose inverse A† is uniquely determined. 2. The Moore–Penrose inverse of the complex conjugate transpose matrix AH is given by (AH )† = (A† )H = A†H = AH† . 3. The generalized inverse of a Moore–Penrose inverse matrix is equal to the original matrix, namely (A† )† = A. 4. If c = 0 then (cA)† = c−1 A† . 5. If D = Diag(d11 , . . . , dnn ), then D† = Diag(d†11 , . . . , d†nn ), where d†ii = d−1 ii (if † dii = 0) or dii = 0 (if dii = 0). 6. The Moore–Penrose inverse of an m × n zero matrix Om×n is an n × m zero matrix, i.e., O†m×n = On×m . 7. The Moore–Penrose inverse of the n × 1 vector x is an 1 × n vector and is given by x† = (xH x)−1 xH . 8. If AH = A and A2 = A then A† = A. 9. If A = BC, B is of full column rank, and C is of full row rank then A† = C† B† = CH (CCH )−1 (BH B)−1 BH . 10. (AAH )† = (A† )H A† and (AAH )† (AAH ) = AA† . 11. If the matrices Ai are mutually orthogonal, i.e., AH i Aj = O, i = j, then (A1 + · · · + Am )† = A†1 + · · · + A†m . 12. Regarding the ranks of generalized inverse matrices, one has rank(A† ) = rank(A) = rank(AH ) = rank(A† A) = rank(AA† ) = rank(AA† A) = rank(A† AA† ). 13. The Moore–Penrose inverse of any matrix Am×n can be determined by A† = (AH A)† AH or A† = AH (AAH )† . In particular, the inverse matrices of the full rank matrices are as follows: • If A is of full column rank then A† = (AH A)−1 AH , i.e., the Moore–Penrose inverse of a matrix A with full column rank reduces to the left pseudo-inverse matrix of A. • If A is of full row rank then A† = AH (AAH )−1 , i.e., the Moore–Penrose inverse of a matrix A with full row rank reduces to the right pseudo-inverse matrix of A. • If A is nonsingular then A† = A−1 , that is, the Moore–Penrose inverse of a nonsingular matrix A reduces to the inverse A−1 of A. 14. For a matrix Am×n , even if AA† = Im , A† A = In , AH (AH )† = In and (AH )† AH = Im , the following results are true: • • • •

A† AAH = AH and AH AA† = AH ; AA† (A† )H = (A† )H and (AH )† A† A = (A† )H ; (AH )† AA = A and AAH (AH )† = A; AH (A† )H A† = A† and A† (A† )H AH = A† .

1.8 Moore–Penrose Inverse Matrices

69

1.8.2 Computation of Moore–Penrose Inverse Matrix Consider an m × n matrix A with rank r, where r ≤ min(m, n). The following are four methods for computing the Moore–Penrose inverse matrix A† . 1. Equation-Solving Method [384] Step 1 Solve the matrix equations AAH XH = A and AH AY = AH to yield the solutions XH and Y, respectively. Step 2

Compute the generalized inverse matrix A† = XAY.

The following are two equation-solving algorithms for Moore–Penrose inverse matrices. Algorithm 1.3

Equation-solving method 1 [187]

1. Compute the matrix B = AAH . 2. Solve the matrix equation B2 XH = B to get the matrix XH . 3. Calculate the Moore–Penrose inverse matrix B† = (AAH )† = XBXH . 4. Compute the Moore–Penrose inverse matrix A† = AH B† .

Algorithm 1.4

Equation-solving method 2 [187]

1. Calculate the matrix B = AH A. 2. Solve the matrix equation B2 XH = X to obtain the matrix XH . 3. Compute the Moore–Penrose inverse matrix B† = (AH A)† = XBXH . 4. Calculate the Moore–Penrose inverse matrix A† = B† AH .

Algorithm 1.3 computes A† = AH (AAH )† and Algorithm 1.4 computes A† = (AH A)† AH . If the number of columns of the matrix Am×n is larger than the number of its rows then the dimension of the matrix product AAH is less than the dimension of AH A. In this case Algorithm 1.3 needs less computation. If, on the contrary, the number of rows is larger than the number of columns then one should select Algorithm 1.4. 2. Full-Rank Decomposition Method DEFINITION 1.37 Let a rank-deficient matrix Am×n have rank r < min{m, n}. If A = FG, where Fm×r is of full column rank and Gr×n is of full row rank, then A = FG is called the full-rank decomposition of the matrix A. LEMMA 1.6 [433] A matrix A ∈ Cm×n with rank(A) = r can be decomposed into A = FG, where F ∈ Cm×r and G ∈ Cr×n have full-column rank and full-row rank, respectively.

70

Introduction to Matrix Algebra

If A = FG is a full-rank decomposition of the m × n matrix A, then A† = G† F† = GH (GGH )−1 (FH F)−1 FH meets the four Moore–Penrose conditions, so the n × m matrix A† must be the Moore–Penrose inverse matrix of A. Elementary row operations easily implement the full-rank decomposition of a rank-deficient matrix A ∈ Cm×n , as shown in Algorithm 1.5. Algorithm 1.5

Full-rank decomposition

1. Use elementary row operations to get the reduced row-echelon form of A. 2. Use the pivot columns of A as the column vectors of a matrix F. 3. Use the nonzero rows in the reduced row-echelon to get the row vectors of a matrix G. 4. The full-rank decomposition is given by A = FG.

EXAMPLE 1.12

In Example 1.1, via elementary ⎡ −3 6 −1 1 A = ⎣ 1 −2 2 3 2 −4 5 8

row operations on the matrix ⎤ −7 −1 ⎦ , −4

we obtained its reduced row-echelon form, ⎡ ⎤ 1 −2 0 −1 3 ⎣ 0 0 1 2 −2 ⎦ . 0 0 0 0 0 The pivot columns of A are the first and third columns, so we have ⎡ ⎤   −3 −1 1 −2 0 −1 3 ⎣ ⎦ . F= 1 2 , G= 0 0 1 2 −2 2 5 Hence we get the full-rank ⎡ −3 6 −1 1 ⎣ 1 −2 2 3 2 −4 5 8

decomposition A = FG: ⎤ ⎤ ⎡  −7 −3 −1  1 −2 0 −1 3 . −1 ⎦ = ⎣ 1 2 ⎦ 0 0 1 2 −2 2 5 −4 3. Recursive Methods

Block the matrix Am×n into Ak = [Ak−1 , ak ], where Ak−1 consists of the first k − 1 columns of A and ak is the kth column of A. Then, the Moore–Penrose inverse A†k of the block matrix Ak can be recursively calculated from A†k−1 . When k = n, we get the Moore–Penrose inverse matrix A† . Such a recursive algorithm was presented by Greville in 1960 [188]. Algorithm 1.6 below is available for all matrices. However, when the number of

1.9 Direct Sum and Hadamard Product Algorithm 1.6

71

Column recursive method

−1 H initialize: A†1 = a†1 = (aH a1 . Put k = 2. 1 a1 )

repeat 1. Compute dk = A†k−1 ak .  −1 H † (1 + dH dk Ak−1 , k dk ) 2. Compute bk = (a − Ak−1 dk )† , k

A†k−1 − dk bk † 3. Compute Ak = . bk

if dk − Ak−1 dk = 0; if ak − Ak−1 dk = 0.

4. exit if k = n. return k ← k + 1. output: A† .

rows of A is small, in order to reduce the number of recursions it is recommended that one should first use the column recursive algorithm 1.6 for finding the Moore– Penrose matrix (AH )† = AH† , and then compute A† = (AH† )H . 4. Trace Method The trace method for computing the Moore–Penrose inverse matrix is shown in Algorithm 1.7. Algorithm 1.7

Trace method [397]

input: A ∈ Rm×n and rank(A) = r. initialize: B = AT A, and set C1 = I, k = 1. repeat 1. Compute Ck+1 = k−1 tr(Ck B)I − Ck B. 2. exit if k = r − 1. return k ← k + 1. output: A† =

r Ck AT . tr(Ck B)

1.9 Direct Sum and Hadamard Product 1.9.1 Direct Sum of Matrices DEFINITION 1.38 [186] The direct sum of an m × m matrix A and an n × n matrix B, denoted A ⊕ B, is an (m + n) × (m + n) matrix and is defined as follows:   A Om×n . (1.9.1) A⊕B= On×m B

72

Introduction to Matrix Algebra

From the above definition, it is easily shown that the direct sum of matrices has the following properties [214], [390]: 1. If c is a constant then c (A ⊕ B) = cA ⊕ cB. 2. The direct sum does not satisfy exchangeability, i.e., A ⊕ B = B ⊕ A unless A = B. 3. If A, B are two m × m matrices and C and D are two n × n matrices then (A ± B) ⊕ (C ± D) = (A ⊕ C) ± (B ⊕ D), (A ⊕ C)(B ⊕ D) = AB ⊕ CD. 4. If A, B, C are m × m, n × n, p × p matrices, respectively, then A ⊕ (B ⊕ C) = (A ⊕ B) ⊕ C = A ⊕ B ⊕ C. 5. If Am×m and Bn×n are respectively orthogonal matrices then A ⊕ B is an (m + n) × (m + n) orthogonal matrix. 6. The complex conjugate, transpose, complex conjugate transpose and inverse matrices of the direct sum of two matrices are given by (A ⊕ B)∗ = A∗ ⊕ B∗ , (A ⊕ B)T = AT ⊕ BT , (A ⊕ B)H = AH ⊕ BH , (A ⊕ B)−1 = A−1 ⊕ B−1

(if A−1 and B−1 exist).

7. The trace, rank and determinant of the direct sum of N matrices are as follows: N −1  N −1 ,  tr Ai = tr(Ai ), i=0

rank

N −1 ,

det

i=0

=

Ai

i=0

N −1 ,

i=0



rank(Ai ),

i=0

 Ai

N −1 

=

N −1 (

det(Ai ).

i=0

1.9.2 Hadamard Product DEFINITION 1.39 The Hadamard product of two m × n matrices A = [aij ] and B = [bij ] is denoted A ∗ B and is also an m × n matrix, each entry of which is defined as the product of the corresponding entries of the two matrices: [A ∗ B]ij = aij bij . That is, the Hadamard product is a mapping Rm×n × Rm×n → Rm×n .

(1.9.2)

1.9 Direct Sum and Hadamard Product

73

The Hadamard product is also known as the Schur product or the elementwise product. The following theorem describes the positive definiteness of the Hadamard product and is usually known as the Hadamard product theorem [214]. THEOREM 1.9 If two m × m matrices A and B are positive definite (positive semi-definite) then their Hadamard product A∗B is positive definite (positive semidefinite) as well. COROLLARY 1.1 (Fejer theorem) [214] An m × m matrix A = [aij ] is positive semi-definite if and only if m m   aij bij ≥ 0 i=1 j=1

holds for all m × m positive semi-definite matrices B = [bij ]. The following theorems describe the relationship between the Hadamard product and the matrix trace. THEOREM 1.10 [311, p. 46] Let A, B, C be m × n matrices, 1 = [1, . . . , 1]T be n an n × 1 summing vector and D = Diag(d1 , . . . , dm ), where di = j=1 aij ; then     tr AT (B ∗ C) = tr (AT ∗ BT )C , (1.9.3) 1T AT (B ∗ C)1 = tr(BT DC).

(1.9.4)

THEOREM 1.11 [311, p. 46] Let A, B be two n × n positive definite square matrices, and 1 = [1, . . . , 1]T be an n × 1 summing vector. Suppose that M is an n × n diagonal matrix, i.e., M = Diag(μ1 , . . . , μn ), while m = M1 is an n × 1 vector. Then one has tr(AMBT M) = mT (A ∗ B)m,

(1.9.5)

tr(AB ) = 1 (A ∗ B)1,

(1.9.6)

MA ∗ B M = M(A ∗ B )M.

(1.9.7)

T

T

T

T

From the above definition, it is known that Hadamard products obey the exchange law, the associative law and the distributive law of the addition: A ∗ B = B ∗ A,

(1.9.8)

A ∗ (B ∗ C) = (A ∗ B) ∗ C,

(1.9.9)

A ∗ (B ± C) = A ∗ B ± A ∗ C.

(1.9.10)

The properties of Hadamard products are summarized below [311]. 1. If A, B are m × n matrices then (A ∗ B)T = AT ∗ BT ,

(A ∗ B)H = AH ∗ BH ,

(A ∗ B)∗ = A∗ ∗ B∗ .

74

Introduction to Matrix Algebra

2. The Hadamard product of a matrix Am×n and a zero matrix Om×n is given by A ∗ Om×n = Om×n ∗ A = Om×n . 3. If c is a constant then c (A ∗ B) = (cA) ∗ B = A ∗ (c B). 4. The Hadamard product of two positive definite (positive semi-definite) matrices A, B is positive definite (positive semi-definite) as well. 5. The Hadamard product of the matrix Am×m = [aij ] and the identity matrix Im is an m × m diagonal matrix, i.e., A ∗ Im = Im ∗ A = Diag(A) = Diag(a11 , . . . , amm ). 6. If A, B, D are three m × m matrices and D is a diagonal matrix then (DA) ∗ (BD) = D(A ∗ B)D. 7. If A, C are two m × m matrices and B, D are two n × n matrices then (A ⊕ B) ∗ (C ⊕ D) = (A ∗ C) ⊕ (B ∗ D). 8. If A, B, C, D are all m × n matrices then (A + B) ∗ (C + D) = A ∗ C + A ∗ D + B ∗ C + B ∗ D. 9. If A, B, C are m × n matrices then     tr AT (B ∗ C) = tr (AT ∗ BT )C . The Hadamard product of n×n matrices A and B obeys the following inequality: (1) Oppenheim inequality [23, p. 144]

If A and B are positive semi-definite, then

|A ∗ B| ≥ a11 · · · ann |B|.

(1.9.11)

(2) If A and B are positive semi-definite, then [327] |A ∗ B| ≥ |AB|.

(1.9.12)

(3) Eigenvalue inequality [23, p.144] If A and B are positive semi-definite and ˆ ,...,λ ˆn λ1 , . . . , λn are the eigenvalues of the Hadamard product A ∗ B, while λ 1 are the eigenvalues of the matrix product AB, then n ( i=k

λi ≥

n (

ˆi, λ

k = 1, . . . , n.

(1.9.13)

i=k

(4) Rank inequality [327] rank(A ∗ B) ≤ rank(A) rank(B).

(1.9.14)

The Hadamard products of matrices are useful in lossy compression algorithms (such as JPEG).

1.10 Kronecker Products and Khatri–Rao Product

75

1.10 Kronecker Products and Khatri–Rao Product The Hadamard product described in the previous section is a special product of two matrices. In this section, we discuss two other special products of two matrices: the Kronecker products and the Khatri–Rao product.

1.10.1 Kronecker Products Kronecker products are divided into right and left Kronecker products. DEFINITION 1.40 (Right Kronecker product) [32] Given an m × n matrix A = [a1 , . . . , an ] and another p × q matrix B, their right Kronecker product A ⊗ B is an mp × nq matrix defined by ⎡ ⎤ a11 B a12 B · · · a1n B ⎢ a21 B a22 B · · · a2n B ⎥ ⎢ ⎥ (1.10.1) A ⊗ B = [aij B]m,n . .. .. ⎥ . .. i=1,j=1 = ⎢ ⎣ .. . . . ⎦ am1 B am2 B · · · amn B DEFINITION 1.41 (Left Kronecker product) [186], [413] For an m × n matrix A and a p × q matrix B = [b1 , . . . , bq ], their left Kronecker product A ⊗ B is an mp × nq matrix defined by ⎡ ⎤ Ab11 Ab12 · · · Ab1q ⎢Ab21 Ab22 · · · Ab2q ⎥ ⎢ ⎥ (1.10.2) [A ⊗ B]left = [Abij ]p,q = ⎢ . .. .. ⎥ . .. i=1,j=1 ⎣ .. . . . ⎦ Abp1

Abp2

···

Abpq

Clearly, the left or right Kronecker product is a mapping Rm×n ×Rp×q → Rmp×nq . It is easily seen that if we adopt the right Kronecker product form then the left Kronecker product can be written as [A ⊗ B]left = B ⊗ A. Since the right Kronecker product form is the one generally adopted, this book uses the right Kronecker product hereafter unless otherwise stated. In particular, when n = 1 and q = 1, the Kronecker product of two matrices reduces to the Kronecker product of two column vectors a ∈ Rm and b ∈ Rp : ⎤ ⎡ a1 b ⎢ . ⎥ a ⊗ b = [ai b]m (1.10.3) i=1 = ⎣ .. ⎦ . am b The result is an mp × 1 vector. Evidently, the outer product of two vectors x ◦ y = xyT can be also represented using the Kronecker product as x ◦ y = x ⊗ yT . The Kronecker product is also known as the direct product or tensor product [307].

76

Introduction to Matrix Algebra

Summarizing results from [32], [60] and other literature, Kronecker products have the following properties. 1. The Kronecker product of any matrix and a zero matrix is equal to the zero matrix, i.e., A ⊗ O = O ⊗ A = O. 2. If α and β are constants then αA ⊗ βB = αβ(A ⊗ B). 3. The Kronecker product of an m×m identity matrix and an n×n identity matrix is equal to an mn × mn identity matrix, i.e., Im ⊗ In = Imn . 4. For matrices Am×n , Bn×k , Cl×p , Dp×q , we have (AB) ⊗ (CD) = (A ⊗ C)(B ⊗ D).

(1.10.4)

5. For matrices Am×n , Bp×q , Cp×q , we have A ⊗ (B ± C) = A ⊗ B ± A ⊗ C,

(1.10.5)

(B ± C) ⊗ A = B ⊗ A ± C ⊗ A.

(1.10.6)

6. The inverse and generalized inverse matrix of Kronecker products satisfy (A ⊗ B)−1 = A−1 ⊗ B−1 ,

(A ⊗ B)† = A† ⊗ B† .

(1.10.7)

7. The transpose and the complex conjugate transpose of Kronecker products are given by (A ⊗ B)T = AT ⊗ BT ,

(A ⊗ B)H = AH ⊗ BH .

(1.10.8)

8. The rank of the Kronecker product is rank(A ⊗ B) = rank(A)rank(B).

(1.10.9)

9. The determinant of the Kronecker product det(An×n ⊗ Bm×m ) = (det A)m (det B)n .

(1.10.10)

10. The trace of the Kronecker product is tr(A ⊗ B) = tr(A)tr(B).

(1.10.11)

11. For matrices Am×n , Bm×n , Cp×q , Dp×q , we have (A + B) ⊗ (C + D) = A ⊗ C + A ⊗ D + B ⊗ C + B ⊗ D.

(1.10.12)

12. For matrices Am×n , Bp×q , Ck×l , it is true that (A ⊗ B) ⊗ C = A ⊗ (B ⊗ C).

(1.10.13)

13. For matrices Am×n , Bk×l , Cp×q , Dr×s , there is the following relationship: (A ⊗ B) ⊗ (C ⊗ D) = A ⊗ B ⊗ C ⊗ D.

(1.10.14)

14. For matrices Am×n , Bp×q , Cn×r , Dq×s , Er×k , Fs×l , we have (A ⊗ B)(C ⊗ D)(E ⊗ F) = (ACE) ⊗ (BDF).

(1.10.15)

1.10 Kronecker Products and Khatri–Rao Product

77

15. The following is a special example of Equation (1.10.15) (see [32], [186]): A ⊗ D = (AIp ) ⊗ (Iq D) = (A ⊗ Iq )(Ip ⊗ D),

(1.10.16)

where Ip ⊗ D is a block diagonal matrix (for the right Kronecker product) or a sparse matrix (for the left Kronecker product), while A ⊗ Iq is a sparse matrix (for the right Kronecker product) or a block diagonal matrix (for the left Kronecker product). 16. For matrices Am×n , Bp×q , we have exp(A ⊗ B) = exp(A) ⊗ exp(B). 17. Let A ∈ Cm×n and B ∈ Cp×q then [311, p. 47] Kpm (A ⊗ B) = (B ⊗ A)Kqn ,

(1.10.17)

Kpm (A ⊗ B)Knq = B ⊗ A,

(1.10.18)

Kpm (A ⊗ B) = B ⊗ A,

(1.10.19)

Kmp (B ⊗ A) = A ⊗ B,

(1.10.20)

where K is a commutation matrix (see Subsection 1.11.1).

1.10.2 Generalized Kronecker Products The generalized Kronecker product is the Kronecker product of a matrix group consisting of more matrices and another matrix. DEFINITION 1.42 [390] Given N matrices Ai ∈ Cm×n , i = 1, . . . , N that constitute a matrix group {A}N , the Kronecker product of the matrix group {A}N and an N × l matrix B, known as the generalized Kronecker product, is defined by ⎤ ⎡ A1 ⊗ b 1 ⎥ ⎢ .. {A}N ⊗ B = ⎣ (1.10.21) ⎦, . A N ⊗ bN where bi is the ith row vector of the matrix B. EXAMPLE 1.13

Put

⎧ ⎫ 1 ⎪ ⎪ ⎪ ⎪ 1 ⎪ ⎬ ⎨ 2 −1 ⎪ {A}2 =   , ⎪ ⎪ 2 −j ⎪ ⎪ ⎪ ⎪ ⎭ ⎩ 1 j

 B=

1 1

2 −1

 .

Then the generalized Kronecker product {A}2 ⊗ B is given by ⎤ ⎡ ⎡   ⎤ 1 1 1 1 2 2 ⊗ [1, 2] ⎥ ⎢ ⎢ 2 −1 4 −2 ⎥ ⎥ ⎢ 2 −1 ⎢ ⎥. {A}2 ⊗ B = ⎢   ⎥=⎣ 2 −j −2 j ⎦ ⎦ ⎣ 2 −j ⊗ [1, −1] 1 j −1 −j 1 j

78

Introduction to Matrix Algebra

It should be noted that the generalized Kronecker product of two matrix groups {A} and {B} is still a matrix group rather than a single matrix. EXAMPLE 1.14

Let

⎧ ⎫ 1 1 ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ 1 −1 ⎬ {A}2 =   , ⎪ ⎪ 1 −j ⎪ ⎪ ⎪ ⎪ ⎭ ⎩ 1 j

# [1, 1] . [1, −1]

" {B} =

The generalized Kronecker product is given by ⎫ ⎧ ⎧  1 1 ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⊗ [1, 1] ⎪ ⎪ ⎬ ⎪ ⎨ ⎨ 1 −1 {A}2 ⊗ {B} =  =   ⎪ ⎪ 1 −j ⎪ ⎪ ⎪ ⎪ ⎪ ⊗ [1, −1]⎪ ⎭ ⎪ ⎩ ⎩ 1 j

1 1

1 1 −1 1

1 1

−j −1 j −1

⎫ ⎪ ⎪ ⎪ ⎬  . j ⎪ ⎪ ⎪ ⎭ −j

1 −1

Generalized Kronecker products have important applications in filter bank analysis and the derivation of fast algorithms in the Haar transform and the Hadamard transform [390]. Based on the generalized Kronecker product, we can derive the fast Fourier transform (FFT) algorithm, which we will discuss in Subsection 2.7.4 in Chapter 2.

1.10.3 Khatri–Rao Product DEFINITION 1.43 The Khatri–Rao product of two matrices with the same number of columns, G ∈ Rp×n and F ∈ Rq×n , is denoted by F  G and is defined as [244], [409] F  G = [f1 ⊗ g1 , f2 ⊗ g2 , . . . , fn ⊗ gn ] ∈ Rpq×n .

(1.10.22)

Thus the Khatri–Rao product consists of the Kronecker product of the corresponding column vectors of two matrices. Hence the Khatri–Rao product is called the columnwise Kronecker product. More generally, if A = [A1 , . . . , Au ] and B = [B1 , . . . , Bu ] are two block matrices, and the submatrices Ai and Bi have the same number of columns, then A  B = [A1 ⊗ B1 , A2 ⊗ B2 , . . . , Au ⊗ Bu ].

(1.10.23)

Khatri–Rao products have a number of properties [32], [301]. 1. The basic properties of the Khatri–Rao product itself are follows. • Distributive law (A + B)  D = A  D + B  D. • Associative law A  B  C = (A  B)  C = A  (B  C). • Commutative law A  B = Knn (B  A), where Knn is a commutation matrix; see the next section.

1.11 Vectorization and Matricization

79

2. The relationship between the Khatri–Rao product and the Kronecker product is (A ⊗ B)(F  G) = AF  BG.

(1.10.24)

3. The relationships between the Khatri–Rao product and the Hadamard product are (A  B) ∗ (C  D) = (A ∗ C)  (B ∗ D),

(1.10.25)

(A  B) (A  B) = (A A) ∗ (B B),  † (A  B)† = (AT A) ∗ (BT B) (A  B)T .

(1.10.26)

T

T

T

(1.10.27)

More generally, (A  B  C)T (A  B  C) = (AT A) ∗ (BT B) ∗ (CT C),  † (A  B  C)† = (AT A) ∗ (BT B) ∗ (CT C) (A  B  C)T .

(1.10.28) (1.10.29)

1.11 Vectorization and Matricization There exist functions or operators that transform a matrix into a vector or vice versa. These functions or operators are the vectorization of a matrix and the matricization of a vector.

1.11.1 Vectorization and Commutation Matrix The vectorization of a matrix A ∈ Rm×n , denoted vec(A), is a linear transformation that arranges the entries of A = [aij ] as an mn × 1 vector via column stacking: vec(A) = [a11 , . . . , am1 , . . . , a1n , . . . , amn ]T .

(1.11.1)

A matrix A can be also arranged as a row vector by stacking the rows; this is known as the row vectorization of the matrix, denoted rvec(A), and is defined as (1.11.2) rvec(A) = [a11 , . . . , a1n , . . . , am1 , . . . , amn ].   a11 a12 , vec(A) = [a11 , a21 , a12 , a22 ]T and For instance, given a matrix A = a21 a22 rvec(A) = [a11 , a12 , a21 , a22 ]. Clearly, there exist the following relationships between the vectorization and the row vectorization of a matrix: rvec(A) = (vec(AT ))T ,

vec(AT ) = (rvec A)T .

(1.11.3)

One obvious fact is that, for a given m × n matrix A, the two vectors vec(A) and vec(AT ) contain the same entries but the orders of their entries are different. Interestingly, there is a unique mn × mn permutation matrix that can transform

80

Introduction to Matrix Algebra

vec(A) into vec(AT ). This permutation matrix is known as the commutation matrix; it is denoted Kmn and is defined by Kmn vec(A) = vec(AT ).

(1.11.4)

Similarly, there is an nm × nm permutation matrix transforming vec(AT ) into vec(A). Such a commutation matrix, denoted Knm , is defined by Knm vec(AT ) = vec(A).

(1.11.5)

From (1.11.4) and (1.11.5) it can be seen that Knm Kmn vec(A) = Knm vec(AT ) = vec(A). Since this formula holds for any m × n matrix A, we have Knm Kmn = Imn or K−1 mn = Knm . The mn × mn commutation matrix Kmn has the following properties [310]. 1. Kmn vec(A) = vec(AT ) and Knm vec(AT ) = vec(A), where A is an m × n matrix. 2. KTmn Kmn = Kmn KTmn = Imn , or K−1 mn = Knm . 3. KTmn = Knm . 4. Kmn can be represented as a Kronecker product of the essential vectors: Kmn =

n 

(eTj ⊗ Im ⊗ ej ).

j=1

5. K1n = Kn1 = In . 6. Knm Kmn vec(A) = Knm vec(AT ) = vec(A). 7. The eigenvalues of the commutation matrix Knn are 1 and −1 and their multiplicities are respectively 12 n(n + 1) and 12 n(n − 1). 8. The rank of the commutation matrix is given by rank(Kmn ) = 1+d(m−1, n−1), where d(m, n) is the greatest common divisor of m and n and d(n, 0) = d(0, n) = n. 9. Kmn (A ⊗ B)Kpq = B ⊗ A, and thus can be equivalently written as Kmn (A ⊗ B) = (B ⊗ A)Kqp , where A is an n × p matrix and B is an m × q matrix. In particular, Kmn (An×n ⊗ Bm×m ) = (B ⊗ A)Kmn .  T 10. tr(Kmn (Am×n ⊗ Bm×n )) = tr(AT B) = vec(BT ) Kmn vec(A). The construction of an mn × mn commutation matrix is as follows. First let ⎤ ⎡ K1 ⎥ ⎢ (1.11.6) Kmn = ⎣ ... ⎦ , Ki ∈ Rn×mn , i = 1, . . . , m; Km

1.11 Vectorization and Matricization

then the (i, j)th entry of the first submatrix K1 is given by + 1, j = (i − 1)m + 1, i = 1, . . . , n, K1 (i, j) = 0, otherwise.

81

(1.11.7)

Next, the ith submatrix Ki (i = 2, . . . , m) is constructed from the (i − 1)th submatrix Ki−1 as follows: Ki = [0, Ki−1 (1 : mn − 1)],

i = 2, . . . , m,

(1.11.8)

where Ki−1 (1 : mn−1) denotes a submatrix consisting of the first (mn−1) columns of the n × m submatrix Ki−1 . EXAMPLE 1.15

For m = 2, n = 4, ⎡ 1 0 ⎢0 0 ⎢ ⎢0 0 ⎢ ⎢ ⎢0 0 K24 = ⎢ ⎢0 1 ⎢ ⎢0 0 ⎢ ⎣0 0 0 0

we have 0 1 0 0 0 0 0 0

0 0 0 0 0 1 0 0

0 0 1 0 0 0 0 0

0 0 0 0 0 0 1 0

0 0 0 1 0 0 0 0

⎤ 0 0⎥ ⎥ 0⎥ ⎥ ⎥ 0⎥ ⎥. 0⎥ ⎥ 0⎥ ⎥ 0⎦ 1

0 0 1 0 0 0 0 0

0 0 0 0 1 0 0 0

0 0 0 0 0 0 1 0

0 1 0 0 0 0 0 0

0 0 0 1 0 0 0 0

0 0 0 0 0 1 0 0

⎤ 0 0⎥ ⎥ 0⎥ ⎥ ⎥ 0⎥ ⎥. 0⎥ ⎥ 0⎥ ⎥ 0⎦ 1

0 1 0 0 0 0 0 0

0 0 0 1 0 0 0 0

0 0 0 0 0 1 0 0

⎤⎡ ⎤ ⎡ ⎤ a11 a11 0 ⎢ ⎥ ⎢ ⎥ 0⎥ ⎢a21 ⎥ ⎥ ⎢a12 ⎥ ⎢ ⎥ ⎢ ⎥ 0⎥ ⎥ ⎢a31 ⎥ ⎢a21 ⎥ ⎥⎢ ⎥ ⎢ ⎥ 0⎥ ⎢a41 ⎥ ⎢a22 ⎥ ⎥ ⎢ ⎥ = ⎢ ⎥ = vec(AT ). 0⎥ ⎢a12 ⎥ ⎢a31 ⎥ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ 0⎥ ⎥ ⎢a22 ⎥ ⎢a32 ⎥ 0⎦ ⎣a32 ⎦ ⎣a41 ⎦ 1 a42 a42

If m = 4, n = 2 then ⎡

K42

1 ⎢0 ⎢ ⎢0 ⎢ ⎢ ⎢0 =⎢ ⎢0 ⎢ ⎢0 ⎢ ⎣0 0

Hence we get ⎡

1 ⎢0 ⎢ ⎢0 ⎢ ⎢ ⎢0 K42 vec(A) = ⎢ ⎢0 ⎢ ⎢0 ⎢ ⎣0 0

0 0 1 0 0 0 0 0

0 0 0 0 1 0 0 0

0 0 0 0 0 0 1 0

82

Introduction to Matrix Algebra

1.11.2 Matricization of a Vector The operation for transforming an mn × 1 vector a = [a1 , . . . , amn ]T into an m × n matrix A is known as the matricization of the column vector a, denoted unvecm,n (a), and is defined as ⎡ ⎤ a1 am+1 · · · am(n−1)+1 ⎢ a2 am+2 · · · am(n−1)+2 ⎥ ⎢ ⎥ (1.11.9) Am×n = unvecm,n (a) = ⎢ . ⎥. .. .. .. ⎣ .. ⎦ . . . a2m

am

···

amn

Clearly, the (i, j)th entry Aij of the matrix A is given by the kth entry ak of the vector a as follows: Aij = ai+(j−1)m ,

i = 1, . . . , m, j = 1, . . . , n.

(1.11.10)

Similarly, the operation for transforming a 1 × mn row vector b = [b1 , . . . , bmn ] into an m×n matrix B is called the matricization of the row vector b. This matricization is denoted unrvecm,n (b), and is defined as follows: ⎡ ⎤ b1 b2 ··· bn ⎢ bn+1 bn+2 · · · b2n ⎥ ⎢ ⎥ (1.11.11) Bm×n = unrvecm,n (b) = ⎢ .. .. .. ⎥ . .. ⎣ . . . . ⎦ b(m−1)n+1 b(m−1)n+2 · · · bmn This is equivalently represented in element form as Bij = bj+(i−1)n ,

i = 1, . . . , m, j = 1, . . . , n.

(1.11.12)

It can be seen from the above definitions that there are the following relationships between matricization (unvec) and column vectorization (vec) or row vectorization (rvec): ⎡ ⎤ A11 · · · A1n ⎢ .. .. ⎥ −vec T .. −→ ⎣ . . . ⎦← −−− [A11 , . . . , Am1 , . . . , A1n , . . . , Amn ] , unvec Am1 · · · Amn ⎤ ⎡ A11 · · · A1n rvec ⎢ .. .. ⎥ − .. −→ ⎣ . . . ⎦← −−− [A11 , . . . , A1n , . . . , Am1 , . . . , Amn ], unvec Am1 · · · Amn which can be written as unvecm,n (a) = Am×n



vec(Am×n ) = amn×1 ,

(1.11.13)

unrvecm,n (b) = Bm×n



rvec(Bm×n ) = b1×mn .

(1.11.14)

1.11 Vectorization and Matricization

83

1.11.3 Properties of Vectorization Operator The vectorization operator has the following properties [68], [207], [311]. 1. The vectorization of a transposed matrix is given by vec(AT ) = Kmn vec(A) for A ∈ Cm×n . 2. The vectorization of a matrix sum is given by vec(A + B) = vec(A) + vec(B). 3. The vectorization of a Kronecker product [311, p. 184] is given by vec(X ⊗ Y) = (Im ⊗ Kqp ⊗ In )(vec(X) ⊗ vec(Y)).

(1.11.15)

4. The trace of a matrix product is given by tr(AT B) = (vec(A))T vec(B), H

H

(1.11.16)

tr(A B) = (vec(A)) vec(B),

(1.11.17)

tr(ABC) = (vec(A)) (Ip ⊗ B) vec(C),

(1.11.18)

T

while the trace of the product of four matrices is determined by [311, p. 31] tr(ABCD) = (vec(DT ))T (CT ⊗ A) vec(B) = (vec(D))T (A ⊗ CT ) vec(BT ). 5. The Kronecker product of two vectors a and b can be represented as the vectorization of their outer product baT as follows: a ⊗ b = vec(baT ) = vec(b ◦ a).

(1.11.19)

6. The vectorization of the Hadamard product is given by vec(A ∗ B) = vec(A) ∗ vec(B) = Diag(vec(A)) vec(B),

(1.11.20)

where Diag(vec(A)) is a diagonal matrix whose entries are the vectorization function vec(A). 7. The relation of the vectorization function to the Khatri–Rao product [60] is as follows: vec(Um×p Vp×p Wp×n ) = (WT  U)diag(V),

(1.11.21)

where diag(V) = [v11 , . . . , vpp ]T and vii are the diagonal entries of V. 8. The relation of the vectorization of the matrix product Am×p Bp×q Cq×n to the Kronecker product [428, p. 263] is given by vec(ABC) = (CT ⊗ A) vec(B),

(1.11.22)

vec(ABC) = (Iq ⊗ AB) vec(C) = (C B ⊗ Im ) vec(A), T

T

vec(AC) = (Ip ⊗ A) vec(C) = (C ⊗ Im ) vec(A). T

(1.11.23) (1.11.24)

84

Introduction to Matrix Algebra

EXAMPLE 1.16 Consider the matrix equation AXB = C, where A ∈ Rm×n , X ∈ Rn×p , B ∈ Rp×q and C ∈ Rm×q . By using the vectorization function property vec(AXB) = (BT ⊗ A) vec(X), the vectorization vec(AXB) = vec(C) of the original matrix equation can be rewritten as the Kronecker product form [410] (BT ⊗ A)vec(X) = vec(C), and thus vec(X) = (BT ⊗ A)† vec(C). Then by matricizing vec(X), we get the solution matrix X of the original matrix equation AXB = C. EXAMPLE 1.17 Consider solving the matrix equation AX + XB = Y, where all the matrices are n × n. By the vectorization operator property vec(ADB) = (BT ⊗A) vec(D), we have vec(AX) = vec(AXI) = (In ⊗A)vec(X) and vec(XB) = vec(IXB) = (BT ⊗In )vec(X), and thus the original matrix equation AX+XB = Y can be rewritten as (In ⊗ A + BT ⊗ In ) vec(X) = vec(Y), from which we get vec(X) = (In ⊗ A + BT ⊗ In )† vec(Y). Then, by matricizing vec(X), we get the solution X.

1.12 Sparse Representations A sparse linear combination of prototype signal-atoms (see below) representing a target signal is known as a sparse representation.

1.12.1 Sparse Vectors and Sparse Representations A vector or matrix most of whose elements are zero is known as a sparse vector or a sparse matrix. A signal vector y ∈ Rm can be decomposed into at most m orthogonal basis (vectors) gk ∈ Rm , k = 1, . . . , m. Such an orthogonal basis is called a complete orthogonal basis. In the signal decomposition y = Gc =

m 

c i gi ,

(1.12.1)

i=1

the coefficient vector c must be nonsparse. If the signal vector y ∈ Rm is decomposed into a linear combination of n vectors ai ∈ Rm , i = 1, . . . , n (where n > m), i.e., y = Ax =

n 

x i ai

(n > m),

(1.12.2)

i=1

then the n (> m) vectors ai ∈ Rm , i = 1, . . . , n cannot be an orthogonal basis set. In order to distinguish them from true basis vectors, nonorthogonal column vectors are called atoms or frames. Because the number n of atoms is larger than the dimension m of the vector space Rm , such a set of atoms is said to be overcomplete.

1.12 Sparse Representations

85

A matrix consisting of overcomplete atoms, A = [a1 , . . . , an ] ∈ Rm×n (n > m), is referred to as a dictionary. For a dictionary (matrix) A ∈ Rm×n , it is usually assumed that (1) the row number m of A is less than its column number n; (2) the dictionary A has full-row rank, i.e., rank(A) = m; (3) all columns of A have unit Euclidean norms, i.e., aj 2 = 1, j = 1, . . . , n. The overcomplete signal decomposition formula (1.12.2) is a under-determined equation that has infinitely many solution vectors x. There are two common methods for solving this type of under-determined equations. 1. Classical method (finding the minimum 2 -norm solution) min x 2

subject to Ax = y.

(1.12.3)

The advantage of this method is that the solution is unique and is the minimum norm solution, also called the minimum energy (sum of squared amplitudes of each component) solution or the shortest distance (from the origin) solution. However, because each entry of this solution vector usually takes a nonzero value, it does not meet the requirements of sparse representation in many practical applications. 2. Modern method (finding the minimum 0 -norm solution) min x 0

subject to Ax = y,

(1.12.4)

where the 0 -norm x 0 is defined as the number of nonzero elements of the vector x. The advantage of this method is that it selects a sparse solution vector, which makes it available for many practical applications. The disadvantage of this method is that its computation is more complex. If in the observation data there exist errors or background noise then the minimum 0 -norm solution becomes min x 0

subject to Ax − y 2 ≤ ,

(1.12.5)

where > 0 is very small. When the coefficient vector x is sparse, the signal decomposition y = Ax is known as a sparse decomposition of signals. The columns of the dictionary matrix A are called explanatory variables; the signal vector y is known as the response variable or target signal; Ax is said to be the linear response predictor, while x can be regarded as the sparse representation of the target signal y corresponding to the dictionary A. Equation (1.12.4) is a sparse representation problem of the target signal y corresponding to the dictionary A, while Equation (1.12.5) is a sparse approximation problem of the target signal. Given a positive integer K, if the 0 -norm of the vector x is less than or equal to

86

Introduction to Matrix Algebra

K, i.e., x 0 ≤ K, then x is said to be K-sparse. For a given signal vector y and a dictionary A, if the coefficient vector x satisfying Ax = y has the minimum 0 -norm then x is said to be the sparsest representation of the target signal y corresponding to the dictionary A. Sparse representation is a type of linear inverse problem. In communications and information theory, the matrix A ∈ Rm×N and the vector x ∈ RN represent respectively the coding matrix and the plaintext to be sent, and the observation vector y ∈ Rm is the ciphertext. The linear inverse problem becomes a decoding problem: how to recover the plaintext x from the ciphertext y.

1.12.2 Sparse Representation of Face Recognition As a typical application of sparse representation, let us consider a face recognition problem. Suppose that there are c classes of target faces that the R1 × R2 matrix representation results for each image of a given face have been vectorized to an m × 1 vector d (where m = R1 × R2 is the number of samples creating each facial image, e.g., m = 512 × 512), and suppose that each column vector is normalized to the unit Euclidean norm. Hence Ni training images of the ith known face under different types of illumination can be represented as an m × Ni matrix Di = [di,1 , di,2 , . . . , di,Ni ] ∈ Rm×Ni . Given a large enough training set Di , then a new image y of the ith known face, shot under another illumination, can be represented as a linear combination of the known training images, i.e., y ≈ Di αi , where αi ∈ Rm is a coefficient vector. The problem is that in practical applications we do not usually know to which target collection of known faces the new experimental samples belong, so we need to make face recognition: determine to which target class these samples belong. If we know or make a rough guess that the new testing sample is a certain unknown target face in the c classes of known target faces, a dictionary consisting of training samples for c classes can be written as a training data matrix as follows: D = [D1 , . . . , Dc ] = [D1,1 , . . . , D1,N1 , . . . , Dc,1 , . . . , Dc,Nc ] ∈ Rm×N , (1.12.6) c where N = i=1 Ni denotes the total number of the training images of the c target faces. Hence the face image y to be recognized can be represented as the linear combination ⎤ ⎡ 0N 1 ⎢ . ⎥ ⎢ .. ⎥ ⎥ ⎢ ⎥ ⎢0 ⎢ Ni−1 ⎥ ⎥ ⎢ (1.12.7) y = Dα0 = [D1,1 . . . D1,N1 . . . Dc,1 . . . Dc,Nc ] ⎢ αi ⎥ , ⎥ ⎢ ⎢0Ni+1 ⎥ ⎥ ⎢ ⎢ .. ⎥ ⎣ . ⎦ 0N c

Exercises

87

where 0Nk , k = 1, . . . , i − 1, i + 1, . . . , c is an Nk × 1 zero vector. The face recognition becomes a problem for solving a matrix equation or a linear inverse problem: given a data vector y and a data matrix D, solve for the solution vector α0 of the matrix equation y = Dα0 . It should be noted that, since m < N in the general case, the matrix equation y = Dα0 is under-determined and has infinitely many solutions, among which the sparsest solution is the solution of interest. Since the solution vector must be a sparse vector, the face recognition problem can be described as an optimization problem: min α0 0

subject to y = Dα0 .

(1.12.8)

This is a typical 0 -norm minimization problem. We will discuss the solution of this problem in Chapter 6.

Exercises 1.1 1.2 1.3

1.4

1.5

T

Let x = [x1 , . . . , xm ] , y = [y1 , . . . , yn ]T , z = [z1 , . . . , zk ]T be complex vectors. Use the matrix form to represent the outer products x ◦ y and x ◦ y ◦ z. Show the associative law of matrix addition (A + B) + C = A + (B + C) and the right distributive law of matrix product (A + B)C = AC + BC. Given the system of linear equations ⎧ + ⎪ ⎨ 2y1 − y2 = x1 , 3z1 − z2 = y1 , and , y1 + 2y2 = x2 , ⎪ 5z1 + 2z2 = y2 , ⎩ −2y1 + 3y2 = x3 , use z1 , z2 to represent x1 , x2 , x3 . By using elementary row operations, simplify the matrix ⎡ 0 0 0 0 2 8 4 ⎢ 0 0 0 1 4 9 7 A=⎢ ⎣ 0 3 −11 −3 −8 −15 −32 0 −2 −8 1 6 13 21

⎤ ⎥ ⎥ ⎦

into its reduced-echelon form. Use elementary row operations to solve the system of linear equations 2x1 − 4x2 + 3x3 − 4x4 − 11x5 = 28, −x1 + 2x2 − x3 + 2x4 + 5x5 = −13, −3x3 + 2x4 + 5x5 = −10, 3x1 − 5x2 + 10x3 − 7x4 + 12x5 = 31.

88

1.6

Introduction to Matrix Algebra

Set 13 + 23 + · · · + n3 = a1 n + a2 n2 + a3 n3 + a4 n4 .

1.7

Find the constants a1 , a2 , a3 , a4 . (Hint: Let n = 1, 2, 3, 4, to get a system of linear equations.) Suppose that F : R3 → R2 is a transformation defined as ⎡ ⎤   x1 2x1 − x2 ⎣ , x = x2 ⎦ . F (x) = x2 + 5x3 x3

Determine whether F is a linear transformation. 1.8 Show that xT Ax = tr(xT Ax) and xT Ax = tr(AxxT ). 1.9 Given A ∈ Rn×n , demonstrate the Schur inequality tr(A2 ) ≤ tr(AT A), where equality holds if and only if A is a symmetric matrix. 1.10 The roots λ satisfying |A − λI| = 0 are known as the eigenvalues of the matrix A. Show that if λ is a single eigenvalue of A then there is at least one determinant |Ak − λI| = 0, where Ak is the matrix remaining after the kth row and kth column of A have been removed. 1.11 The equation of a straight line can be represented as ax + by = −1. Show that the equation of the line through two points (x1 , y1 ) and (x2 , y2 ) is given by   1 x y     1 x y  = 0. 1 1  1 x y  2

2

1.12 The equation of a plane can be represented as ax + by + cz = −1. Show that the equation of the plane through the three points (xi , yi , zi ), i = 1, 2, 3, is determined by   1 x y z     1 x1 y1 z1     1 x2 y2 z2  = 0.   1 x y z  3 3 3 1.13 Given a vector set {x1 , . . . , xp } with xp = 0, let a1 , a2 , . . . , ap−1 be any constants, and let yi = xi +ai xp , i = 1, . . . , p−1. Show that the vectors in the set {y1 , . . . , yp−1 } are linearly independent, if and only if the vectors x1 , . . . , xp are linearly independent. 1.14 Without expanding the determinants, prove the following results:     a b c   a + b b + c c + a      2 d e f  =  d + e e + f f + d x y z  x + y y + z z + x 

Exercises

and

 0  2 a b

a 0 c

  c b  b + a a+c c  =  b a 0  a

89

 c  b  . c + b

1.15 Suppose that An×n is positive definite and Bn×n is positive semi-definite; show that det(A + B) ≥ det(A). 1.16 Let A12×12 satisfy A5 = 3A. Find all the possible values of |A|. 1.17 Given a block matrix X = [A, B], show that  T   A A A T B . |x|2 = |AAT + BBT | =  T B A BT B  1.18 Let A2 = A. Use two methods to show that rank(I−A) = n−rank(A): (a) use the property that the matrix trace and the matrix rank are equal; (b) consider linearly independent solutions of the matrix equation (I − A)x = 0. 1.19 Given the matrix ⎤ ⎡ 1 1 1 2 A = ⎣ −1 0 2 −3 ⎦ , 2 4 8 5 find the rank of A. (Hint: Transform the matrix A into its echelon form.) 1.20 Let A be an m × n matrix. Show that rank(A) ≤ m and rank(A) ≤ n. 1.21 Consider the system of linear equations x1 + 3x2 − x3 = a1 , x1 + 2x2 = a2 , 3x1 + 7x2 − x3 = a3 . (a) Determine the necessary and sufficient condition for the above system of linear equations to be consistent. (b) Consider three cases: (i) a1 = 2, a2 = 2, a3 = 6; (ii) a1 = 1, a2 = 0, a3 = −2; (iii) a1 = 0, a2 = 1, a3 = 2. Determine whether the above linear system of equations is consistent. If it is, then give the corresponding solution. 1.22 Determine that for what values of α, the system of linear equations (α + 3)x1 + x2 + 2x3 = α, 3(α + 1)x1 + αx2 + (α + 3)x3 = 3, αx1 + (α − 1)x2 + x3 = α, has a unique solution, no solution or infinitely many solutions. In the case

90

Introduction to Matrix Algebra

where the system of linear equations has infinitely many solutions, find a general solution. 1.23 Set a1 = [1, 1, 1, 3]T , a2 = [−1, −3, 5, 1]T , a3 = [3, 2, −1, p + 2]T and a4 = [−2, −6, 10, p]T . (a) For what values of p are the above four vectors linearly independent? Use a linear combination of a1 , . . . , a4 to express a = [4, 1, 6, 10]T . (b) For what values of p are the above four vectors linearly dependent? Find the rank of the matrix [a1 , a2 , a3 , a4 ] in these cases. 1.24 Given



⎤ a a1 = ⎣ 2 ⎦ , 10



⎤ −2 a2 = ⎣ 1 ⎦ , 5



⎤ −1 a3 = ⎣ 1 ⎦ , 4

⎡ ⎤ 1 b = ⎣b⎦ , c

find respectively the values of a, b, c such that the following conditions are satisfied: (a) b can be linearly expressed by a1 , a2 , a3 and this expression is unique; (b) b cannot be linearly expressed by a1 , a2 , a3 ; (c) b can be linearly expressed by a1 , a2 , a3 , but not uniquely. Find a general expression for b in this case. 1.25 Show that A nonsingular





 A B det = det(A) det(D − CA−1 B). C D

1.26 Verify that the vector group ⎧⎡ 1 ⎪ ⎪ ⎨⎢ ⎢ 0 ⎣ 1 ⎪ ⎪ ⎩ 2

⎤⎫ ⎤ ⎡ −1 ⎪ −1 ⎪ ⎥ ⎢ 1 ⎥ ⎢ −2 ⎥⎬ ⎥ ⎥,⎢ ⎥,⎢ ⎦ ⎣ 1 ⎦ ⎣ 1 ⎦⎪ ⎪ ⎭ 0 0 ⎤ ⎡

is a set of orthogonal vectors. 1.27 Suppose that A = A2 is an idempotent matrix. Show that all its eigenvalues are either 1 or 0. 1.28 If A is an idempotent matrix, show that each of AH , IA and I − AH is a idempotent matrix. 1.29 Given the 3 × 5 matrix ⎡ ⎤ 1 3 2 5 7 A = ⎣ 2 1 0 6 1⎦ , 1 1 2 5 4 use the MATLAB functions orth(A) and null(A) to find an orthogonal basis for the column space Span(A) and for the null space Null(A).

Exercises

91

1.30 Let x and y be any two vectors in the Euclidean space Rn . Show the Cauchy– Schwartz inequality |xT y| ≤ x

y . (Hint: Observe that x − cy 2 ≥ 0 holds for all scalars c.) 1.31 Define a transformation H : R2 → R2 as     x x1 + x 2 − 1 , x= 1 . H(x) = 3x1 x2 Determine whether H is a linear transformation. 1.32 Put ⎡ ⎤ ⎡ ⎤ 1 0.5 0.2 0.3 P = ⎣ 0.3 0.8 0.3 ⎦ , x0 = ⎣0⎦ . 0 0.2 0 0.4 Assume that the state vector of a particular system can be described by the Markov chain xk+1 = Pxk , k = 0, 1, . . . Compute the state vectors x1 , . . . , x15 and analyze the change in the system over time. 1.33 Given a matrix function ⎡ ⎤ 2x −1 x 2 ⎢ 4 x 1 −1 ⎥ ⎥, A(x) = ⎢ ⎣ 3 2 x 5 ⎦ 1

−2

3

x

find d3 |A(x)|/dx3 . (Hint: Expand the determinant |A(x)| according to any row or column; only the terms x4 and x3 are involved.) 1.34 Show that if A1 is nonsingular then   A 1 A2  −1   A3 A4  = |A1 ||A4 − A3 A1 A2 |. 1.35 Determine the positive definite status of the following quadratics: (a) f = −2x21 − 8x22 − 6x23 + 2x1 x2 + 2x1 x3 , (b) f = x21 + 4x22 + 9x23 + 15x24 − 2x1 x2 + 4x1 x3 + 2x1 x4 − 6x2 x4 − 12x3 x4 . 1.36 Show that (a) if B is a real-valued nonsingular matrix then A = BBT is positive definite. (b) if |C| = 0 then A = CCH is positive definite. 1.37 Suppose that A and B are n × n real matrices. Show the following Cauchy– Schwartz inequalities for the trace function: (a) (tr(AT B))2 ≤ tr(AT ABT B), where the equality holds if and only if AB is a symmetric matrix; (b) (tr(AT B))2 ≤ tr(AAT BBT ), where the equality holds if and only if AT B is a symmetric matrix. 1.38 Show that tr(ABC) = tr(BCA) = tr(CAB).

92

Introduction to Matrix Algebra

1.39 Assuming that the inverse matrices appearing below exist, show the following results: (a) (b) (c) (d)

(A−1 + I)−1 = A(A + I)−1 . (A−1 + B−1 )−1 = A(A + B)−1 B = B(A + B)−1 A. (I + AB)−1 A = A(I + BA)−1 . A − A(A + B)−1 A = B − B(A + B)−1 B.

1.40 Show that an eigenvalue of the inverse of a matrix A is equal to the reciprocal of the corresponding eigenvalue of the original matrix, i.e., eig(A−1 ) = 1/eig(A). 1.41 Verify that A† = (AH A)† AH and A† = AH (AAH )† meet the four Moore– Penrose conditions. 1.42 Find the Moore–Penrose inverse of the matrix A = [1, 5, 7]T . 1.43 Prove that A(AT A)−2 AT is the Moore–Penrose inverse of the matrix AAT . 1.44 Given the matrix ⎡ ⎤ 1 0 −2 ⎢ 0 1 −1 ⎥ ⎥, X=⎢ ⎣ −1 1 1 ⎦ 2

1.45

1.46

1.47 1.48

−1

2

use respectively the recursive method (Algorithm 1.6) and the trace method (Algorithm 1.7) find the Moore–Penrose inverse matrix X† . Consider the mapping UV = W, where U ∈ Cm×n , V ∈ Cn×p , W ∈ Cm×p and U is a rank-deficient matrix. Show that V = U† W, where U† ∈ Cn×m is the Moore–Penrose inverse of U. Show that if Ax = b is consistent then its general solution is x = A† b + (I − A† A)z, where A† is the Moore–Penrose inverse of A and z is any constant vector. Show that the linear mapping T : V → W has the properties T (0) = 0 and T (−x) = −T (x). Show that, for any m × n matrix A, its vectorization function is given by vec(A) = (In ⊗ A)vec(In ) = (AT ⊗ In )vec(Im ).

1.49 Show that the necessary and sufficient condition for A ⊗ B to be nonsingular is that both the matrices A, B are nonsingular. Furthermore, show that (A ⊗ B)−1 = A−1 ⊗ B−1 . 1.50 Show that (A ⊗ B)† = A† ⊗ B† . 1.51 If A, B, C are m × m matrices and CT = C, show that (vec(C))T (A ⊗ B) vec(C) = (vec(C))T (B ⊗ A) vec(C). 1.52 Show that tr(ABCD) = (vec(DT ))T (CT ⊗ A) vec(B) = (vec(D)T )(A ⊗ CT ) vec(BT ).

Exercises

93

 1.53 Let A, B ∈ Rm×n , show tr(AT B) = (vec(A))T vec(B) = vec(A∗B), where  vec(·) represents the sum of all elements of the column vector functions. 1.54 Let xi and xj be two column vectors of the matrix X, their covariance matrix is Cov xi , xTj = Mij . Hence the variance–covariance matrix of the vectorization function vec(X), denoted Var(vec(X)) is a block matrix with the submatrix Mij , i.e., Var(vec(X)) = {Mij }. For the special cases of Mij = mij V, if mij is the entry of the matrix M, show that: (a) Var(vec(X)) = M ⊗ V; (b) Var(vec(TX)) = M ⊗ TVTT ; (c) Var(vec(XT )) = V ⊗ M. 1.55 Consider two n × n matrices A and B. (a) Let d = [d1 , . . . , dn ]T and D = Diag(d1 , . . . , dn ). Show that dT (A∗B)d = tr(ADBT D). (b) If both A and B are the positive definite matrices, show that the Hadamard product A ∗ B is a positive definite matrix. This property is known as the positive definiteness of the Hadamard product. 1.56 Show that A ⊗ B = vec(BAT ). 1.57 Show that vec(PQ) = (QT ⊗ P) vec(I) = (QT ⊗ I) vec(P) = (I ⊗ P) vec(Q). 1.58 Verify the following result: tr(XA) = (vec(AT ))T vec(X) = (vec(XT ))T vec(A). 1.59 Let A be an m × n matrix, and B be a p × q matrix. Show (A ⊗ B)Knq = Kmp (B ⊗ A),

Kpm (A ⊗ B)Knq = B ⊗ A,

where Kij is the commutation matrix. 1.60 For any m × n matrix A and p × 1 vector b, show the following results. (a) (b) (c) (d)

Kpm (A ⊗ b) = b ⊗ A. Kmp (b ⊗ A) = A ⊗ b. (A ⊗ b)Knp = bT ⊗ A. (bT ⊗ A)Kpn = A ⊗ b.

1.61 Show that the Kronecker product of the m × m identity matrix and the n × n identity matrix yields the mn × mn identity matrix, namely Im ⊗ In = Imn . 1.62 Set x ∈ RI×JK , G ∈ RP ×QR , A ∈ RI×P , B ∈ RJ×Q , C ∈ RK×R and AT A = IP , BT B = IQ , CT C = IR . Show that if X = AG(C ⊗ B)T and X and A, B, C are given then the matrix G can be recovered or reconstructed from G = AT X(C ⊗ B). 1.63 Use vec(UVW) = (WT ⊗ U) vec(V) to find the vectorization representation of X = AG(C ⊗ B)T . 1.64 Apply the Kronecker product to solve for the unknown matrix X in the matrix equation AXB = C.

94

Introduction to Matrix Algebra

1.65 Apply the Kronecker product to solve for the unknown matrix X in the generalized continuous-time Lyapunov equation LX + XN = Y, where the dimensions of all matrices are n × n and the matrices L and N are invertible. 1.66 Show the following results: (A  B) ∗ (C  D) = (A ∗ C)  (B ∗ D), (A  B  C)T (A  B  C) = (AT A) ∗ (BT B) ∗ (CT C),  † (A  B)† = AT A) ∗ (BT B) (A  B)T . 1.67 Try extending the 0 -norm of a vector to the 0 -norm of an m × n matrix. Use this to define whether a matrix is K-sparse.

2 Special Matrices

In real-world applications, we often meet matrices with special structures. Such matrices are collectively known as special matrices. Understanding the internal structure of these special matrices is helpful for using them and for simplifying their representation. This chapter will focus on some of the more common special matrices with applications.

2.1 Hermitian Matrices In linear algebra, the adjoint of a matrix A refers to its conjugate transpose and is commonly denoted by AH . If AH = A, then A is said to be self-adjoint. A self-adjoint matrix A = AH ∈ Cn×n is customarily known as a Hermitian matrix. A matrix A is known as anti-Hermitian if A = −AH . The centro-Hermitian matrix R is an n × n matrix whose entries have the sym∗ . metry rij = rn−j+1,n−i+1 Hermitian matrices have the following properties. 1. A is a Hermitian matrix, if and only if xH Ax is a real number for all complex vectors x. 2. For all A ∈ Cn×n , the matrices A + AH , AAH and AH A are Hermitian. 3. If A is a Hermitian matrix, then Ak are Hermitian matrices for all k = 1, 2, 3, . . . If A is a nonsingular Hermitian matrix then its inverse, A−1 , is Hermitian as well. 4. If A and B are Hermitian matrices then αA + βB is Hermitian for all real numbers α and β. 5. If A and B are Hermitian then AB + BA and j(AB − BA) are also Hermitian. 6. If A and B are anti-Hermitian matrices then αA + βB is anti-Hermitian for all real numbers α and β. 7. For all A ∈ Cn×n , the matrix A − AH is anti-Hermitian. √ 8. If A is a Hermitian matrix then j A (j = −1) is anti-Hermitian. 9. If A is an anti-Hermitian matrix, then j A is Hermitian. 95

96

Special Matrices

Positive definition criterion for a Hermitian matrix An n × n Hermitian matrix A is positive definite, if and only if any of the following conditions is satisfied. • • • •

The quadratic form xH Ax > 0, ∀ x = 0. The eigenvalues of A are all larger than zero. There is an n × n nonsingular matrix R such that A = RH R. There exists an n × n nonsingular matrix P such that PH AP is positive definite.

Let z be a Gaussian random vector with zero mean. Then its correlation matrix Rzz = E{zzH } is Hermitian, and is always positive definite, because the quadratic form xH Rzz x = E{|xH z|2 } > 0. Positive definite and positive semi-definite matrices obey the following inequalities [214, Section 8.7]: (1) Hadamard inequality If an m × m matrix A = [aij ] is positive definite then det(A) 

m (

aii

i=1

and equality holds if and only if A is a diagonal matrix.   A B is position definite, where (2) Fischer inequality If a block matrix P = BH C submatrices A and C are the square nonzero matrices, then det(P)  det(A) det(C). (3) Oppenheim inequality For two m × m positive semi-definite matrices A and B, m ( det(A) bii  det(A  B), i=1

where A  B is the Hadamard product of A and B. (4) Minkowski inequality For two m × m positive definite matrices A and B, one has    n det(A + B) ≥ n det(A) + n det(B).

2.2 Idempotent Matrix In some applications one uses a multiple product of an n×n matrix A, resulting into three special matrices: an idempotent matrix, a unipotent matrix and a tripotent matrix. DEFINITION 2.1

A matrix An×n is idempotent, if A2 = AA = A.

2.2 Idempotent Matrix

97

The eigenvalues of any idempotent matrix except for the identity matrix I take only the values 1 and 0, but a matrix whose eigenvalues take only 1 and 0 is not necessarily an idempotent matrix. For example, the matrix ⎡ ⎤ 11 3 3 1⎣ B= 1 1 1 ⎦ 8 −12 −4 −4 has three eigenvalues 1, 0, 0, but it is not an idempotent matrix, because ⎡ ⎤ 11 3 3 1 B2 = ⎣ 0 0 0 ⎦ = B. 8 −11 −3 −3 An idempotent matrix has the following useful properties [433], [386]: 1. 2. 3. 4. 5. 6. 7. 8. 9.

The eigenvalues of an idempotent matrix take only the values 1 and 0. All idempotent matrices A = I are singular. The rank and trace of any idempotent matrix are equal, i.e., rank(A) = tr(A). If A is an idempotent matrix then AH is also an idempotent matrix, i.e., A H AH = A H . If A is an idempotent n × n matrix then In − A is also an idempotent matrix, and rank(In − A) = n − rank(A). All symmetric idempotent matrices are positive semi-definite. Let the n × n idempotent matrix A have the rank rA ; then A has rA eigenvalues equal to 1 and n − rA eigenvalues equal to zero. A symmetric idempotent matrix A can be expressed as A = LLT , where L satisfies LT L = IrA and rA is the rank of A. All the idempotent matrices A are diagonalizable, i.e.,   IrA O −1 , (2.2.1) U AU = Σ = O O where rA = rank(A) and U is a unitary matrix.

DEFINITION 2.2 I.

A matrix An×n is called unipotent or involutory, if A2 = AA =

If an n×n matrix A is unipotent, then the function f (·) has the following property [386]: 1 f (sI + tA) = [(I + A)f (s + t) + (I − A)f (s − t)]. (2.2.2) 2 There exists a relationship between an idempotent matrix and an unipotent matrix: the matrix A is an unipotent matrix if and only if 12 (A + I) is an idempotent matrix. DEFINITION 2.3

A matrix An×n is called a nilpotent matrix, if A2 = AA = O.

98

Special Matrices

If A is a nilpotent matrix, then the function f (·) has the following property [386]: f (sI + tA) = If (s) + tAf (s),

(2.2.3)

where f (s) is the first-order derivative of f (s). DEFINITION 2.4

A matrix An×n is called tripotent, if A3 = A.

It is easily seen that if A is a tripotent matrix then −A is also a tripotent matrix. It should be noted that a tripotent matrix is not necessarily an idempotent matrix, although an idempotent matrix must be a triponent matrix (because if A2 = A then A3 = A2 A = AA = A). In order to show this point, consider the eigenvalues of a tripotent matrix. Let λ be an eigenvalue of the tripotent matrix A, and u the eigenvector corresponding to λ, i.e., Au = λu. Premultiplying both sides of Au = λu by A, we have A2 u = λAu = λ2 u. Premultiplying A2 u = λ2 u by A, we immediately have A3 u = λ2 Au = λ3 u. Because A3 = A is a tripotent matrix, the above equation can be written as Au = λ3 u, and thus the eigenvalues of a tripotent matrix satisfies the relation λ = λ3 . That is to say, the eigenvalues of a tripotent matrix have three possible values −1, 0, +1; which is different from the eigenvalues of an idempotent matrix, which take only the values 0 and +1. In this sense, an idempotent matrix is a special tripotent matrix with no eigenvalue equal to −1.

2.3 Permutation Matrix The n × n identity matrix I = [e1 , . . . , en ] consists of the basis vectors in an order such that the matrix is diagonal. On changing the order of the basis vectors, we obtain four special matrices: the permutation matrix, the commutation matrix, the exchange matrix and the selection matrix.

2.3.1 Permutation Matrix and Exchange Matrix DEFINITION 2.5 A square matrix is known as a permutation matrix, if each of its rows and columns have one and only one nonzero entry equal to 1. A permutation matrix P has the following properties [60]. (1) PT P = PPT = I, i.e., a permutation matrix is orthogonal. (2) PT = P−1 . (3) PT AP and A have the same diagonal entries but their order may be different.

2.3 Permutation Matrix

EXAMPLE 2.1

Given the 5 × 4 matrix ⎡ a11 a12 ⎢a21 a22 ⎢ A=⎢ ⎢a31 a32 ⎣a 41 a42 a51 a52

For the permutation matrices ⎡ 0 0 ⎢0 1 P4 = ⎢ ⎣1 0 0 0 we have



a51 ⎢a31 ⎢ P5 A = ⎢ ⎢a21 ⎣a 41 a11

a52 a32 a22 a42 a12

0 0 0 1

a53 a33 a23 a43 a13



1 0⎥ ⎥, 0⎦ 0 ⎤ a54 a34 ⎥ ⎥ a24 ⎥ ⎥, a ⎦ 44

a14

a13 a23 a33 a43 a53

99

⎤ a14 a24 ⎥ ⎥ a34 ⎥ ⎥. a ⎦ 44

a54



0 ⎢0 ⎢ P5 = ⎢ ⎢0 ⎣0 1

0 0 1 0 0 ⎡

a13 ⎢a23 ⎢ AP4 = ⎢ ⎢a33 ⎣a 43 a53

0 1 0 0 0

0 0 0 1 0 a12 a22 a32 a42 a52

⎤ 1 0⎥ ⎥ 0⎥ ⎥, 0⎦ 0 a14 a24 a34 a44 a54

⎤ a11 a21 ⎥ ⎥ a31 ⎥ ⎥. a ⎦ 41

a51

That is, premultiplying an m × n matrix A by an m × m permutation matrix is equivalent to rearranging the rows of A, and postmultiplying the matrix A by an n × n permutation matrix is equivalent to rearranging the columns of A. The new sequences of rows or columns depend on the structures of the permutation matrices. A p × q permutation matrix can be a random arrangement of q basis vectors e1 , . . . , eq ∈ Rp×1 . If the basis vectors are arranged according to some rule, the permutation matrix may become one of three special permutation matrices: a commutation matrix, an exchange matrix or a shift matrix. 1. Commutation Matrix As stated in Section 1.11, the commutation matrix Kmn is defined as the special permutation matrix such that Kmn vec Am×n = vec(AT ). The role of this matrix is to commute the entry positions of an mn × 1 vector in such a way that the new vector Kmn vec(A) becomes vec(AT ), so this special permutation matrix is known as the commutation matrix. 2. Exchange Matrix The exchange matrix is usually represented using the notation J, and is defined as ⎡ ⎤ 0 1 ⎢ 1 ⎥ ⎢ ⎥ (2.3.1) J=⎢ ⎥ · ⎣ ⎦ ·· 1 0

100

Special Matrices

which has every entry on the cross-diagonal line equal to 1, whereas all other entries in the matrix are equal to zero. The exchange matrix is called the reflection matrix or backward identity matrix, since it can be regarded as a backward arrangement of the basis vectors, namely J = [en , en−1 , . . . , e1 ]. Premultiplying an m × n matrix A by the exchange matrix Jm , we can invert the order of the rows of A, i.e., we exchange all row positions with respect to its central horizontal axis, obtaining ⎡ ⎤ am1 am2 · · · amn ⎢ .. .. .. .. ⎥ ⎢ . . . ⎥ (2.3.2) Jm A = ⎢ . ⎥. ⎣a a ··· a ⎦ 21

22

a11

a12

···

2n

a1n

Similarly, postmultiplying A by the exchange matrix Jn , we invert the order of the columns of A, i.e., we exchange all column positions with respect to its central vertical axis: ⎡ ⎤ a1n · · · a12 a11 ⎢ a2n · · · a22 a21 ⎥ ⎢ ⎥ AJn = ⎢ . (2.3.3) .. .. .. ⎥ . . ⎣ . . . . ⎦ amn

···

am2

am1

It is easily seen that J2 = JJ = I,

JT = J.

(2.3.4)

Here J2 = I is the involutory property of the exchange matrix, and JT = J expresses its symmetry. That is, an exchange matrix is both an involutory matrix (J2 = I) and a symmetric matrix (JT = J). As stated in Chapter 1, the commutation matrix has the property KTmn = K−1 mn = Knm . Clearly, if m = n then KTnn = K−1 nn = Knn . This implies that K2nn = Knn Knn = Inn ,

(2.3.5)

KTnn

(2.3.6)

= Knn .

That is to say, like the exchange matrix, a square commutation matrix is both an involutory matrix and a symmetric matrix. In particular, when premultiplying an m × n matrix A by an m × m exchange matrix Jm and postmultiplying an n × n exchange matrix Jn , we have ⎡ ⎤ am,n am,n−1 ··· am,1 ⎢am−1,n am−1,n−1 · · · am−1,1 ⎥ ⎢ ⎥ (2.3.7) Jm AJn = ⎢ . .. .. .. ⎥ . ⎣ .. . . . ⎦ a1,n−1 ··· a1,1 a1,n

2.3 Permutation Matrix

101

Similarly, the operation Jc inverts the element order of the column vector c and c J inverts the element order of the row vector cT . T

3. Shift Matrix The n × n shift matrix is defined as ⎡ 0 ⎢0 ⎢ ⎢ P = ⎢ ... ⎢ ⎣0 1

1 0 .. .

0 1 .. .

··· ··· .. .

0 0

0 0

··· ···

⎤ 0 0⎥ ⎥ .. ⎥ . .⎥ ⎥ 1⎦

(2.3.8)

0

In other words, the entries of the shift matrix are pi,i+1 = 1 (1  i  n − 1) and pn1 = 1; all other entries are equal to zero. Evidently, a shift matrix can be represented using basis vectors as P = [en , e1 , . . . , en−1 ]. Given an m × n matrix A, the operation Pm A will make A’s first row move to its lowermost row, namely ⎡ ⎤ a21 a22 · · · a2n ⎢ .. .. .. .. ⎥ ⎢ . . . ⎥ Pm A = ⎢ . ⎥. ⎣a a ··· a ⎦ m1

m2

a11

a12

···

mn

a1n

Similarly, postmultiplying A by an n × n shift matrix Pn , we have ⎡ ⎤ a1n a11 · · · a1,n−1 ⎢ a2n a21 · · · a2,n−1 ⎥ ⎢ ⎥ APn = ⎢ . .. .. .. ⎥ . ⎣ .. . . . ⎦ amn

am1

···

am,n−1

The role of Pn is to move the rightmost column of A to the leftmost position. It is easy to see that, unlike a square exchange matrix Jn or a square commutation matrix Knn , a shift matrix is neither involutory nor symmetric.

2.3.2 Generalized Permutation Matrix 1. Generalized Permutation Matrix Consider the observation-data model x(t) = As(t) =

n 

ai si (t),

(2.3.9)

i=1

where s(t) = [s1 (t), . . . , sn (t)]T represents a source vector and A = [a1 , . . . , an ], with ai ∈ Rm×1 , i = 1, · · · , n, is an m × n constant-coefficient matrix (m ≥ n)

102

Special Matrices

that represents the linear mixing process of n sources and is known as the mixture matrix. The mixture matrix A is assumed to be of full-column rank. The question is how to recover the source vector s(t) = [s1 (t), . . . , sn (t)]T using only the m-dimensional observation-data vector x(t). This is the well-known blind source separation (BSS) problem. Here, the terminology “blind” has two meanings: (a) the sources s1 (t), . . . , sn (t) are unobservable; (b) how the n sources are mixed is unknown (i.e., the mixture matrix A is unknown). The core issue of the BSS problem is to identify the generalized inverse of the mixture matrix, A† = (AT A)−1 AT , because the source vector s(t) is then easily recovered using s(t) = A† x(t). Unfortunately, in mixture-matrix identification there exist two uncertainties or ambiguities. (1) If the ith and jth source signals are exchanged in order, and the ith and the jth columns of the mixture matrix A are exchanged at the same time, then the observation data vector x(t) is unchanged. This shows that the source signal ordering cannot be identified from only the observation-data vector x(t). Such an ambiguity is known as the ordering uncertainty of separated signals. (2) Using only the observation data vector x(t), it is impossible to identify accurately the amplitudes, si (t), i = 1, . . . , n, of the source signals, because x(t) =

n  ai αi si (t), α i=1 i

(2.3.10)

where the αi are unknown scalars. This ambiguity is called the amplitude uncertainty of the separate signals. The two uncertainties above can be described by a generalized permutation matrix. DEFINITION 2.6 An m × m matrix G is known as a generalized permutation matrix (or a g-matrix), if each row and each column has one and only one nonzero entry. It is easy to show that a square matrix is a g-matrix if and only if it can be decomposed into the product G of a permutation matrix P and a nonsingular diagonal matrix D: G = PD.

(2.3.11)

For instance, ⎡

0 0 ⎢0 0 ⎢ G=⎢ ⎢0 γ ⎣0 0 ρ 0

0 β 0 0 0

⎤ ⎡ 0 0 α ⎢0 0 0⎥ ⎥ ⎢ ⎢ 0 0⎥ ⎥ = ⎢0 ⎣0 ⎦ λ 0 1 0 0

0 0 1 0 0

0 1 0 0 0

0 0 0 1 0

⎤⎡ 1 ρ ⎢ 0⎥ ⎥⎢ ⎥ 0⎥ ⎢ ⎢ 0⎦ ⎣ 0 0

0

⎤ ⎥ ⎥ ⎥. ⎥ ⎦

γ β λ α

2.3 Permutation Matrix

103

By the above definition, if premultiplying (or postmultiplying) a matrix A by a g-matrix, then the rows (or columns) of A will be rearranged, and all entries of each row (or column) of A will be multiplied by a scalar factor. For example, ⎡

0 0 ⎢0 0 ⎢ ⎢0 γ ⎢ ⎣0 0 ρ 0

0 β 0 0 0

0 0 0 λ 0

⎤⎡ α a11 ⎢a21 0⎥ ⎥⎢ ⎢ 0⎥ ⎥ ⎢a31 ⎦ 0 ⎣a41 0 a51

a12 a22 a32 a42 a52

a13 a23 a33 a43 a53

⎤ ⎡ αa51 a14 ⎢βa31 a24 ⎥ ⎥ ⎢ ⎢ a34 ⎥ ⎥ = ⎢ γa21 ⎣ λa ⎦ a44 41 a54 ρa11

αa52 βa32 γa22 λa42 ρa12

αa53 βa33 γa23 λa43 ρa13

⎤ αa54 βa34 ⎥ ⎥ γa24 ⎥ ⎥. λa44 ⎦ ρa14

Using the g-matrix, Equation (2.3.10) can be equivalently written as x(t) = AG−1 (Gs)(t) = AG−1˜s(t),

(2.3.12)

where ˜s(t) = Gs(t). Therefore, the task of BSS is now to identify the product AG−1 of the matrix A and the generalized inverse matrix G−1 rather than the generalized inverse A† , which greatly eases the solution of the BSS problem. That is to say, the solution of the BSS is now given by ˜s(t) = (AG−1 )† x(t) = GA† x(t). If we let W = GA† be a de-mixing or separation matrix, then adaptive BSS is used to update adaptively both the de-mixing matrix W(t) and the reconstructed source signal vector ˜s(t) = W(t)x(t). 2. Selection Matrix As the name implies, the selection matrix is a type of matrix that can select certain rows or columns of a matrix. Let xi = [xi (1), . . . , xi (N )]T and, given an m × N matrix ⎡ ⎤ x1 (1) x1 (2) · · · x1 (N ) ⎢ x2 (1) x2 (2) · · · x2 (N ) ⎥ ⎢ ⎥ X=⎢ . .. .. .. ⎥ , ⎣ .. . . . ⎦ xm (1)

xm (2)

···

xm (N )

we have ⎡ ⎢ ⎢ J1 X = ⎢ ⎣ ⎡

x1 (1) x2 (1) .. .

x1 (2) x2 (2) .. .

··· ··· .. .

xm−1 (1)

xm−1 (2)

···

x2 (1) ⎢ x3 (1) ⎢ J2 X = ⎢ . ⎣ .. xm (1)

x2 (2) x3 (2) .. .

··· ··· .. .

xm (2)

···

x1 (N ) x2 (N ) .. .

⎤ ⎥ ⎥ ⎥ ⎦

if

J1 = [Im−1 , 0m−1 ],

xm−1 (N ) ⎤ x2 (N ) x3 (N ) ⎥ ⎥ .. ⎥ if J2 = [0m−1 , Im−1 ]. . ⎦ xm (N )

104

Special Matrices

That is, the matrix J1 X selects the uppermost m − 1 rows of the original matrix X, and the matrix J2 X selects the lowermost m − 1 rows of X. Similarly, we have ⎡ ⎤ x1 (1) x1 (2) · · · x1 (N − 1)   ⎢ x2 (1) x2 (2) · · · x2 (N − 1) ⎥ IN −1 ⎢ ⎥ if J , = XJ1 = ⎢ . ⎥ .. .. .. 1 0N −1 ⎣ .. ⎦ . . . xm (1)

xm (2)

···

x1 (2) ⎢ x2 (2) ⎢ XJ2 = ⎢ . ⎣ .. xm (2)

x1 (3) x2 (3) .. .

··· ··· .. .

xm (3)

···



xm (N − 1) ⎤ x1 (N ) x2 (N ) ⎥ ⎥ .. ⎥ if . ⎦ xm (N )

 0N −1 J2 = . IN −1 

In other words, the matrix XJ1 selects the leftmost N − 1 columns of the matrix X, and the matrix XJ2 selects the rightmost N − 1 columns of X.

2.4 Orthogonal Matrix and Unitary Matrix The vectors x1 , . . . , xk ∈ Cn constitute an orthogonal set if xH i xj = 0, 1 ≤ i < j ≤ k. Moreover, if the vectors are normalized, i.e., x 22 = xH x = 1, i = 1, . . . , k, then i i the orthogonal set is known as an orthonormal set. THEOREM 2.1

A set of orthogonal nonzero vectors is linearly independent.

Proof Suppose that {x1 , . . . , xk } is an orthogonal set, and 0 = α1 x1 + · · · + αk xk . Then we have 0 = 0H 0 =

k  k  i=1 j=1

αi∗ αj xH i xj =

k 

|αi |2 xH i xi .

i=1

k

2 H Since the vectors are orthogonal, and xH i xi > 0, i=1 |αi | xi xi = 0 implies that 2 all |αi | = 0 and so all αi = 0, and thus x1 , . . . , xk are linearly independent.

DEFINITION 2.7

A real square matrix Q ∈ Rn×n is orthogonal, if QQT = QT Q = I.

(2.4.1)

A complex square matrix U ∈ Cn×n is unitary matrix if UUH = UH U = I.

(2.4.2)

DEFINITION 2.8 A real m×n matrix Qm×n is called semi-orthogonal if it satisfies only QQT = Im or QT Q = In . Similarly, a complex matrix Um×n is known as para-unitary if it satisfies only UUH = Im or UH U = In . THEOREM 2.2 [214]

If U ∈ Cn×n then the following statements are equivalent:

2.4 Orthogonal Matrix and Unitary Matrix

• • • • •

105

U is a unitary matrix; U is nonsingular and UH = U−1 ; UUH = UH U = I; UH is a unitary matrix; the columns of U = [u1 , . . . , un ] constitute an orthonormal set, namely + 1, i = j, H ui uj = δ(i − j) = 0, i = j;

• the rows of U constitute an orthonormal set; • for all x ∈ Cn , the Euclidean lengths of y = Ux and x are the same, namely

y 2 = x 2 . If a linear transformation matrix A is unitary, then the linear transformation Ax is known as a unitary transformation. A unitary transformation has the following properties. (1) The vector inner product is invariant under unitary transformation: x, y = Ax, Ay, since Ax, Ay = (Ax)H Ay = xH AH Ay = xH y = x, y. (2) The vector norm is invariant under unitary transformation, i.e., Ax 2 = x 2 , since Ax 2 = Ax, Ax = x, x = x 2 . (3) The angle between the vectors Ax and Ay is also invariant under unitary transformation, namely cos θ =

x, y Ax, Ay = .

Ax

Ay

x

y

(2.4.3)

This is the result of combining the former two properties. The determinant of a unitary matrix is equal to 1, i.e., | det(A)| = 1,

if A is unitary.

(2.4.4)

DEFINITION 2.9 The matrix B ∈ Cn×n such that B = UH AU is said to be unitarily equivalent to A ∈ Cn×n . If U is a real orthogonal matrix then we say that B is orthogonally equivalent to A. DEFINITION 2.10 AAH .

The matrix A ∈ Cn×n is known as a normal matrix if AH A =

The following list summarizes the properties of a unitary matrix [307]. 1. Am×m is a unitary matrix ⇔ the columns of A are orthonormal vectors. 2. Am×m is a unitary matrix ⇔ the rows of A are orthonormal vectors. 3. Am×m is real that A is a unitary matrix ⇔ A is orthogonal.

106

Special Matrices

4. Am×m is a unitary matrix ⇔ AAH = AH A = Im ⇔ AT is a unitary matrix ⇔ AH is a unitary matrix ⇔ A∗ is a unitary matrix ⇔ A−1 is a unitary matrix ⇔ Ai is a unitary matrix, i = 1, 2, . . . 5. Am×m , Bm×m are unitary matrices ⇒ AB is a unitary matrix. 6. If Am×m is a unitary matrix then • • • • • • •

| det(A)| = 1. rank(A) = m. A is normal, i.e., AAH = AH A. λ is an eigenvalue of A ⇒ |λ| = 1. For a matrix Bm×n , AB F = B F . For a matrix Bn×m , BA F = B F . For a vector xm×1 , Ax 2 = x 2 .

7. If Am×m and Bn×n are two unitary matrices then • A ⊕ B is a unitary matrix. • A ⊗ B is a unitary matrix. DEFINITION 2.11 An N × N diagonal matrix whose diagonal entries take the values either +1 or −1 is known as a signature matrix. DEFINITION 2.12 Let J be an N ×N signature matrix. Then the N ×N matrix Q such that QJQT = J is called the J-orthogonal matrix or the hypernormal matrix. From the above definition we know that when a signature matrix equals the identity matrix, i.e., J = I, the J-orthogonal matrix Q reduces to an orthogonal matrix. Clearly, a J-orthogonal matrix Q is nonsingular, and the absolute value of its determinant is equal to 1. Any N × N J-orthogonal matrix Q can be equivalently defined by QT JQ = J.

(2.4.5)

Substituting the definition formula QJQT = J into the right-hand side of Equation (2.4.5), it immediately follows that QT JQ = QJQT .

(2.4.6)

This symmetry is known as the hyperbolic symmetry of the signature matrix J. The matrix vvT Q=J−2 T (2.4.7) v Jv is called the hyperbolic Householder matrix [71]. In particular, if J = I then the hyperbolic Householder matrix reduces to the Householder matrix.

2.5 Band Matrix and Triangular Matrix

107

2.5 Band Matrix and Triangular Matrix A triangular matrix is one of the standard forms of matrix decomposition; the triangular matrix itself is a special example of the band matrix.

2.5.1 Band Matrix A matrix A ∈ Cm×n such that aij = 0, |i − j| > k, is known as a band matrix. In particular, if aij = 0, ∀ i > j + p, then A is said to have under bandwidth p, and if aij = 0, ∀ j > i + q, then A is said to have upper bandwidth q. The following is an example of a 7 × 5 band matrix with under bandwidth 1 and upper bandwidth 2: ⎤ ⎡ × × × 0 0 ⎢× × × × 0 ⎥ ⎥ ⎢ ⎢ 0 × × × ×⎥ ⎥ ⎢ ⎥ ⎢ ⎢ 0 0 × × ×⎥ , ⎥ ⎢ ⎢ 0 0 0 × ×⎥ ⎥ ⎢ ⎣ 0 0 0 0 ×⎦ 0 0 0 0 0 where × denotes any nonzero entry. A special form of band matrix is of particular interest. This is the tridiagonal matrix. A matrix A ∈ Cn×n is tridiagonal, if its entries aij = 0 when |i − j| > 1. Clearly, the tridiagonal matrix is a square band matrix with upper bandwidth 1 and under bandwidth 1. However, the tridiagonal matrix is a special example of the Hessenberg matrix, which is defined below. An n × n matrix A is known as an upper Hessenberg matrix if it has the form ⎤ ⎡ a11 a12 a13 · · · a1n ⎥ ⎢a ⎢ 21 a22 a23 · · · a2n ⎥ ⎥ ⎢ ⎢ 0 a32 a33 · · · a3n ⎥ ⎥ ⎢ A=⎢ 0 0 a43 · · · a4n ⎥ . ⎥ ⎢ .. .. .. ⎥ ⎢ .. .. . ⎣ . . . . ⎦ 0 0 0 · · · ann A matrix A is known as a lower Hessenberg matrix if AT is a upper Hessenberg matrix.

2.5.2 Triangular Matrix Two common special band matrices are the upper triangular matrix and the lower triangular matrix. The triangular matrix is a canonical form in matrix decomposition.

108

Special Matrices

A square matrix U = [uij ] with uij = 0, i > j, is called an upper triangular matrix, and its general form is ⎤ ⎡ u11 u12 · · · u1n ⎢ 0 u22 · · · u2n ⎥ ⎥ ⎢ U=⎢ . .. .. ⎥ ⇒ |U| = u11 u22 · · · unn . .. ⎣ .. . . . ⎦ 0

···

0

unn

A square matrix L = [lij ] such that lij = 0, i < j, is known as the lower triangular matrix, and its general form is ⎡ ⎤ l11 0 ⎢ l12 l22 ⎥ ⎢ ⎥ L=⎢ . ⎥ ⇒ |L| = l11 l22 · · · lnn . .. .. ⎣ .. ⎦ . . ln1

ln2

···

lnn

In fact, a triangular matrix is not only an upper Hessenberg matrix but also a lower Hessenberg matrix. Summarizing the definitions on triangular matrixes, a square matrix A = [aij ] is said to be (1) (2) (3) (4) (5) (6)

lower triangular if aij = 0 (i < j); strict lower triangular if aij = 0 (i ≤ j); unit triangular if aij = 0 (i < j), aii = 1 (∀ i); upper triangular if aij = 0 (i > j); strict upper triangular if aij = 0 (i ≥ j); unit upper triangular if aij = 0 (i > j), aii = 1 (∀ i). 1. Properties of Upper Triangular Matrices

1. The product of upper triangular matrices is also an upper triangular matrix; i.e., if U1 , . . . , Uk are upper triangular matrices then U = U1 · · · Uk is also an upper triangular matrix. 2. The determinant of an upper triangular matrix U = [uij ] is equal to the product of its diagonal entries: det(U) = u11 · · · unn =

n (

uii .

i=1

3. The inverse of an upper triangular matrix is also an upper triangular matrix. 4. The kth power Uk of an upper triangular matrix Un×n is also an upper triangular matrix, and its ith diagonal entry is equal to ukii . 5. The eigenvalues of an upper triangular matrix Un×n are u11 , . . . , unn . 6. A positive definite Hermitian matrix A can be decomposed into A = TH DT, where T is a unit upper triangular complex matrix, and D is a real diagonal matrix.

2.6 Summing Vector and Centering Matrix

109

2. Properties of Lower Triangular Matrices 1. A product of lower triangular matrices is also a lower triangular matrix, i.e., if L1 , . . . , Lk are lower triangular matrices then L = L1 · · · Lk is also a lower triangular matrix. 2. The determinant of a lower triangular matrix is equal to the product of its diagonal entries, i.e., det(L) = l11 · · · lnn =

n (

lii .

i=1

3. The inverse of a lower triangular matrix is also a lower triangular matrix. 4. The kth power Lk of a lower triangular matrix Ln×n is also a lower triangular k matrix, and its ith diagonal entry is lii . 5. The eigenvalues of a lower triangular matrix Ln×n are l11 , . . . , lnn . 6. A positive definite matrix An×n can be decomposed into the product of a lower triangular matrix Ln×n and its transpose, i.e., A = LLT . This decomposition is called the Cholesky decomposition of A. The lower triangular matrix L such that A = LLT is sometimes known as the square root of A. More generally, any matrix B satisfying B2 = A

(2.5.1)

is known as the square root matrix of A, denoted A1/2 . It is should be noted that the square root matrix of a square matrix A is not necessarily unique. 3. Block Triangular Matrices If, for a block triangular matrix, the matrix blocks on its diagonal line or crossdiagonal line are invertible, then its inverse is given respectively by 

A O B C

−1

−1 A B C O  −1 A B O C



 A−1 O , = −C−1 BA−1 C−1   O C−1 , = B−1 −B−1 AC−1   −1 −A−1 BC−1 A . = O C−1 

(2.5.2) (2.5.3) (2.5.4)

2.6 Summing Vector and Centering Matrix A special vector and a special matrix used frequently in statistics and data processing are the summing vector and centering matrix.

110

Special Matrices

2.6.1 Summing Vector A vector with all entries 1 is known as a summing vector and is denoted 1 = [1, . . . , 1]T . It is called a summing vector because the sum of n scalars can be expressed as the inner product of a summing vector and another vector: given an m-vector x = [x1 , . . . , xm ]T , the sum of its components can be expressed as m m T T i=1 xi = 1 x or i=1 xi = x 1. EXAMPLE 2.2 expressed as

If we let x = [a, b, −c, d]T then the sum a + b − c + d can be ⎡

⎤ a ⎢b⎥ T ⎥ a + b − c + d = [1 1 1 1] ⎢ ⎣−c⎦ = 1 x = 1, x. d

In some calculations we may meet summing vectors with different dimensions. In this case, we usually indicate by a subscript the dimension of each summing vector as a subscript in order to avoid confusion. For example, 13 = [1, 1, 1]T . Consider this product of the summing vector and a matrix: ⎡ ⎤ 4 −1 1T3 X3×2 = [1 1 1] ⎣ −4 3 ⎦ = [1 1] = 1T2 . 1 −1 The inner product of a summing vector and itself is an integer equal to its dimension, namely 1, 1 = 1Tn 1n = n.

(2.6.1)

The outer product of two summing vectors is a matrix all of whose entries are equal to 1; for example,     1 1 1 1 T = J2×3 . [1 1 1] = 12 13 = 1 1 1 1 More generally, 1p 1Tq = Jp×q

(a matrix with all entries equal to 1).

(2.6.2)

It is easy to verify that Jm×p Jp×n = pJm×n ,

Jp×q 1q = q1p ,

1Tp Jp×q = p1Tq .

(2.6.3)

In particular, for an n × n matrix Jn , we have Jn = 1n 1Tn ,

J2n = nJn .

(2.6.4)

Hence, selecting ¯ = 1J , J n n n

(2.6.5)

2.6 Summing Vector and Centering Matrix

111

¯2 = J ¯ . That is, J ¯ is idempotent. we have J n n n

2.6.2 Centering Matrix The matrix ¯ = I − 1J C n = In − J n n n n

(2.6.6)

is known as a centering matrix. It is easily verified that a centering matrix is not only symmetric but also idempotent, namely Cn = CTn = C2n . In addition, a centering matrix has the following properties: 0 Cn 1 = 0, Cn Jn = Jn Cn = 0.

(2.6.7)

(2.6.8)

The summing vector 1 and the centering matrix C are very useful in mathematical statistics [433, p. 67]. First, the mean of a set of data x1 , . . . , xn can be expressed using a summing vector as follows: 1 1 1 1 x = (x1 + · · · + xn ) = xT 1 = 1T x, n i=1 i n n n n

x ¯=

(2.6.9)

where x = [x1 , . . . , xn ]T is the data vector. Next, using the definition formula (2.6.6) of the centering matrix and its property shown in (2.6.8), it follows that ¯ = x − 1 11T x = x − x Cx = x − Jx ¯1 n ¯ , . . . , xn − x ¯ ]T . = [x1 − x

(2.6.10)

In other words, the role of the linear transformation matrix C is to subtract the mean of n data from the data vector x. This is the mathematical meaning of the centering matrix. Moreover, for the inner product of the vector Cx, we have ¯ , . . . , xn − x ¯][x1 − x ¯ , . . . , xn − x ¯ ]T Cx, Cx = (Cx)T Cx = [x1 − x n  = (xi − x ¯ )2 . i=1

From Equation (2.6.8) it is known that CT C = CC = C, and thus the above

112

Special Matrices

equation can be simplified to xT Cx =

n 

(xi − x ¯ )2 .

(2.6.11)

i=1

The right-hand side of (2.6.11) is the well-known covariance of the data x1 , . . . , xn . That is to say, the covariance of a set of data can be reexpressed in the quadratic form of the centering matrix as the kernel xT Cx.

2.7 Vandermonde Matrix and Fourier Matrix Consider two special matrices whose entries in each row (or column) constitute a geometric series. These two special matrices are the Vandermonde matrix and the Fourier matrix, and they have wide applications in engineering. In fact, the Fourier matrix is a special example of the Vandermonde matrix.

2.7.1 Vandermonde Matrix The n × n Vandermonde matrix is a matrix taking the special form ⎡

1

x1

⎢ ⎢ 1 x2 A=⎢ ⎢ .. .. ⎣. . 1 xn or



1 ⎢ x1 ⎢ ⎢ 2 A=⎢ ⎢ x1 ⎢ .. ⎣ . xn−1 1



x21

···

x22 .. .

··· .. .

⎥ ⎥ xn−1 2 ⎥ .. ⎥ . ⎦

x2n

···

xn−1 n

1 x2

··· ···

x22 .. .

··· .. .

xn−1 2

···

xn−1 1

⎤ 1 xn ⎥ ⎥ ⎥ x2n ⎥ ⎥. .. ⎥ . ⎦ n−1 xn

(2.7.1)

(2.7.2)

That is to say, the entries of each row (or column) constitute a geometric series. The Vandermonde matrix has a prominent property: if the n parameters x1 , . . . , xn are different then the Vandermonde matrix is nonsingular, because its determinant is given by [32, p. 193] det(A) =

n ( i,j=1, i>j

(xi − xj ).

(2.7.3)

2.7 Vandermonde Matrix and Fourier Matrix

113

Given an n × n complex Vandermonde matrix ⎡

1 a1 .. .

1 a2 .. .

··· ··· .. .

1 an .. .

an−1 1

an−1 2

···

an−1 n

⎢ ⎢ A=⎢ ⎣

⎤ ⎥ ⎥ ⎥, ⎦

ak ∈ C,

(2.7.4)

its inverse is given by [339] ⎡

⎤T

⎢ σ (a , . . . , aj−1 , aj+1 , . . . , an ) ⎥ i+j n−j 1 ⎥ A−1 = ⎢ j−1 n ⎣(−1) ⎦   (aj − ak ) (ak − aj ) k=1

k=j+1

,

(2.7.5)

i=1,...,n j=1,...,n

where σ0 (a1 , . . . , aj−1 , aj+1 , . . . , an ) = 1, σ1 (a1 , . . . , aj−1 , aj+1 , . . . , an ) = σi (a1 , . . . , aj−1 , aj+1 , . . . , an ) =

n  k=1,k=j n 

ak , ak · · · ak+i−1 , i = 2, . . . , n,

k=1,k=j

with ai = 0, i > n. EXAMPLE 2.3 In the extended Prony method, the signal model is assumed to be the superposition of p exponential functions, namely x ˆn =

p 

bi zin ,

n = 0, 1, . . . , N − 1

(2.7.6)

i=1

is used as the mathematical model for fitting the data x0 , x1 , . . . , xN −1 . In general, bi and zi are assumed to be complex numbers, and bi = Ai exp(j θi ),

zi = exp[(αi + j 2πfi )Δt],

where Ai is the amplitude of the ith exponential function, θi is its phase (in radians), αi its damping factor, fi its oscillation frequency (in Hz), and Δt denotes the sampling time interval (in seconds). The matrix form of Equation (2.7.6) is ˆ Φb = x ˆ = [ˆ in which b = [b0 , b1 , . . . , bp ]T , x x0 , x ˆ1 , . . . , x ˆN −1 ]T , and Φ is a complex Vander-

114

Special Matrices

monde matrix given by ⎡

1 ⎢ z1 ⎢ ⎢ 2 Φ=⎢ ⎢ z1 ⎢ .. ⎣ . z1N −1

1 z2

1 z3

··· ···

z22 .. .

z32 .. .

··· .. .

z2N −1

z3N −1

···

⎤ 1 zp ⎥ ⎥ ⎥ zp2 ⎥ ⎥. .. ⎥ . ⎦

(2.7.7)

zpN −1

N −1 By minimizing the square error = ˆn |2 we get a least squares n=1 |xn − x ˆ , as follows: solution of the matrix equation Φb = x b = [ΦH Φ]−1 ΦH x.

(2.7.8)

It is easily shown that the computation of ΦH Φ in Equation (2.7.8) can be greatly simplified so that, without doing the multiplication operation on the Vandermonde matrix, we can get directly ⎡ ⎤ γ11 γ12 · · · γ1p ⎢γ21 γ22 · · · γ2p ⎥ ⎢ ⎥ . .. .. .. ⎥ , (2.7.9) ΦH Φ = ⎢ ⎢ .. . . . ⎥ ⎣ ⎦ γp1

γp2

···

γpp

where γij =

(zi∗ zj )N − 1 . (zi∗ zj ) − 1

(2.7.10)

The N × p matrix Φ shown in Equation (2.7.7) is one of two Vandermonde matrices widely applied in signal processing; another common complex Vandermonde matrix is given by ⎡ ⎤ 1 1 ··· 1 ⎢ eλ1 eλ 2 ··· eλ d ⎥ ⎢ ⎥ ⎢ ⎥. .. .. .. .. (2.7.11) Φ=⎢ ⎥ . . . . ⎣ ⎦ eλ1 (N −1) eλ2 (N −1) · · · eλd (N −1)

2.7.2 Fourier Matrix The Fourier transform of the discrete-time signals x0 , x1 , . . . , xN −1 is known as the discrete Fourier transform (DFT) or spectrum of these signals, and is defined as Xk =

N −1  n=0

xn e−j 2πnk/N =

N −1  n=0

xn wnk ,

k = 0, 1, . . . , N − 1.

(2.7.12)

2.7 Vandermonde Matrix and Fourier Matrix

This equation can be expressed in ⎤ ⎡ ⎡ 1 1 X0 ⎢ X1 ⎥ ⎢ 1 w ⎥ ⎢ ⎢ ⎢ . ⎥ = ⎢. .. ⎣ .. ⎦ ⎣ .. . XN −1 1 wN −1

115

matrix form as ⎤⎡

··· ··· .. .

wN −1 .. .

···

w(N −1)(N −1)

1

⎥⎢ ⎥⎢ ⎥⎢ ⎦⎣

x0 x1 .. .

⎤ ⎥ ⎥ ⎥ ⎦

(2.7.13)

xN −1

or simply as ˆ = Fx, x

(2.7.14)

ˆ = [X0 , X1 , . . . , XN −1 ]T are respectively the where x = [x0 , x1 , . . . , xN −1 ]T and x discrete-time signal vector and the spectrum vector, whereas ⎤ ⎡ 1 1 ··· 1 ⎥ ⎢1 w ··· wN −1 ⎥ ⎢ ⎥ , w = e−j 2π/N ⎢ .. .. .. (2.7.15) F = ⎢ .. ⎥ . . . ⎦ ⎣. 1 wN −1 · · · w(N −1)(N −1) is known as the Fourier matrix with (i, k)th entry F (i, k) = w(i−1)(k−1) . Clearly, each row and each column of the Fourier matrix constitute a respective geometric series, and thus the Fourier matrix can be regarded as an N × N special Vandermonde matrix. From the definition it can be seen that the Fourier matrix is symmetric, i.e., FT = F. Equation (2.7.14) shows that the DFT of a discrete-time signal vector can be expressed by the matrix F, which is the reason why F is termed the Fourier matrix. Moreover, from the definition of the Fourier matrix it is easy to verify that FH F = FFH = N I.

(2.7.16)

Since the Fourier matrix is a special Vandermonde matrix that is nonsingular, it follows from FH F = N I that the inverse of the Fourier matrix is given by F−1 =

1 H 1 F = F∗ . N N

(2.7.17)

Hence, from Equation (2.7.2) it can be seen immediately that ˆ= x = F−1 x which can ⎡ x0 ⎢ x1 ⎢ ⎢ . ⎣ ..

be written as ⎡ ⎤ 1 ⎢1 ⎥ 1 ⎢ ⎥ ⎢. ⎥= . ⎦ N⎢ ⎣.

xN −1

1 w∗ .. .

1 (wN −1 )∗

··· ··· .. . ···

1 ∗ ˆ, F x N ⎤⎡

1 (w

(2.7.18)

N −1 ∗

)

.. . (w(N −1)(N −1) )∗

⎤ X0 ⎥⎢ ⎥ ⎢ X1 ⎥ ⎥⎢ . ⎥ , ⎥⎣ . ⎥ . ⎦ ⎦ XN −1

(2.7.19)

116

Special Matrices

thus xn =

N −1 1  Xk e j2πnk/N , N

n = 0, 1, . . . , N − 1.

(2.7.20)

k=0

This is the formula for the inverse discrete Fourier transform (IDFT). From the definition it is easy to see that the n × n Fourier matrix F has the following properties [307]. (1) The Fourier matrix is symmetric, i.e., FT = F. (2) The inverse of the Fourier matrix is given by F−1 = N −1 F∗ . (3) F2 = P = [e1 , eN , eN −1 , . . . , e2 ] (the permutation matrix), where ek is the basis vector whose kth entry equals 1 and whose other entries are zero. (4) F4 = I.

2.7.3 Index Vectors Given an N × 1 vector x = [x0 , x1 , . . . , xN −1 ]T , we will consider the subscript representation of its elements in binary code. Define the index vector [390] ⎡ ⎤ 0 ⎢ 1 ⎥ ⎢ ⎥ (2.7.21) iN = ⎢ ⎥ , N = 2n , .. ⎣ ⎦ . N − 1 in which i denotes the binary representation of the integer i, with i = 0, 1, . . . , N − 1. EXAMPLE 2.4

For N = 22 = 4 and N = 23 = 8, we have respectively ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 0 00 0 ⎢1⎥ ⎢01⎥ ⎢1⎥ ⎢ ⎥ ⇔ i4 = ⎢ ⎥ = ⎢ ⎥ , ⎣2⎦ ⎣10⎦ ⎣2⎦ 3 11 3 ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ 000 0 0 ⎢1⎥ ⎢1⎥ ⎢001⎥ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎢2⎥ ⎢2⎥ ⎢010⎥ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎢3⎥ ⎢3⎥ ⎢011⎥ ⎥. ⎢ ⎥ ⇔ i8 = ⎢ ⎥ = ⎢ ⎢4⎥ ⎢4⎥ ⎢100⎥ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎢5⎥ ⎢5⎥ ⎢101⎥ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎣6⎦ ⎣6⎦ ⎣110⎦ 111 7 7

2.7 Vandermonde Matrix and Fourier Matrix

117

Importantly, an index vector can be repressed as the Kronecker product of binary codes 0 and 1. Let a, b, c, d be binary codes; then the binary Kronecker product of binary codes is defined as ⎡ ⎤ ac     ⎢ad⎥ a c ⊗ =⎢ ⎥ , b 2 d 2 ⎣ bc ⎦ bd 2

(2.7.22)

where xy denotes the arrangement in order of the binary code x and y rather than their product. EXAMPLE 2.5 The binary Kronecker product representations of the index vectors i4 and i8 are as follows: ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 01 00 00 0     ⎢01 10 ⎥ ⎢01⎥ ⎢1⎥ 00 01 ⎥ ⎢ ⎥ ⎢ ⎥ ⊗ =⎢ i4 = ⎣1 0 ⎦ = ⎣10⎦ = ⎣2⎦ 11 10 1 0 11 3 11 10 and ⎤ ⎡ ⎤ ⎤ ⎡ 0 000 02 01 00 ⎢0 0 1 ⎥ ⎢001⎥ ⎢1⎥ 2 1 0 ⎥ ⎢ ⎥ ⎥ ⎢ ⎡ ⎤ ⎢ ⎢0 1 0 ⎥ ⎢010⎥ ⎢2⎥ 01 00 ⎥ ⎢ ⎥ ⎢ 2 1 0⎥ ⎢         ⎥ ⎢ ⎥ ⎥ ⎢ ⎢01 10 ⎥ ⎢ 01 00 02 02 ⎢02 11 10 ⎥ ⎢011⎥ ⎢3⎥ ⎢ ⎥ = ⊗ ⊗ = ⊗⎣ i8 = = ⎥ = ⎢ ⎥. ⎢ ⎢ ⎥ 12 11 10 12 11 00 ⎦ ⎢12 01 00 ⎥ ⎢100⎥ ⎢4⎥ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎢12 01 10 ⎥ ⎢101⎥ ⎢5⎥ 11 10 ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎣12 11 00 ⎦ ⎣110⎦ ⎣6⎦ 7 111 12 11 10 ⎡

More generally, the binary Kronecker product representation of the index vector iN is given by        0N −2 0 0 0N −1 ⊗ ⊗ ··· ⊗ 1 ⊗ 0 = 1N −1 1N −2 11 10 ⎤ ⎡ ⎤ ⎡ 0 0N −1 · · · 01 00 ⎢0N −1 · · · 01 10 ⎥ ⎢ 1 ⎥ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ .. .. = =⎢ ⎥. ⎢ ⎥ . . ⎥ ⎢ ⎥ ⎢ ⎣ ⎣1 ⎦ N − 2⎦ ···1 0 

iN

N −1

1 0

1N −1 · · · 11 10

(2.7.23)

N − 1

On the other hand, the bit-reversed index vector of iN is denoted iN,rev and is

118

Special Matrices

defined as follows:

⎡ iN,rev

0rev 1rev .. .



⎢ ⎥ ⎢ ⎥ ⎢ ⎥ =⎢ ⎥, ⎢ ⎥ ⎣N − 2 ⎦ rev N − 1rev

(2.7.24)

where irev represents the bit-reversed result of the binary code of the integer i, i = 0, 1, . . . , N − 1. For instance, if N = 8, then the reversed results of the binary code 001 and 011 are respectively 100 and 110. Interestingly, on reversing the order of the binary Kronecker product of the index vector iN , we obtain directly the bit-reversed index vector       01 0N −1 00 iN,rev = ⊗ ⊗ ··· ⊗ . (2.7.25) 10 11 1N −1 EXAMPLE 2.6 by

The bit-reversed index vectors of i4 and i8 are respectively given

i4,rev

and

i8,rev

⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 00 01 00 0     ⎢ ⎥ ⎢ ⎥ ⎢ 0 0 00 11 ⎥ ⎢10⎥ ⎢2⎥ ⎥ = 0 ⊗ 1 =⎢ = = ⎣ 10 11 10 01 ⎦ ⎣01⎦ ⎣1⎦ 11 3 10 11

⎤ ⎡ ⎤ ⎤ ⎡ 0 000 00 01 02 ⎢00 01 12 ⎥ ⎢100⎥ ⎢4⎥ ⎥ ⎢ ⎥ ⎥ ⎢ ⎢ ⎡ ⎤ ⎢0 1 0 ⎥ ⎢010⎥ ⎢2⎥ 00 01 0 1 2⎥ ⎥ ⎢ ⎥ ⎢ ⎢   ⎢       ⎥ ⎢ ⎥ ⎥ ⎢ ⎢00 11 ⎥ 110 0 01 02 00 0 1 1 ⎥ ⎢6⎥ ⎢ ⎥ ⎢ 0 1 2 2 ⎥⊗ = ⊗ ⊗ =⎢ = = ⎥ = ⎢ ⎥. ⎢ ⎥ ⎢ ⎣1 0 ⎦ ⎢10 01 02 ⎥ ⎢001⎥ ⎢1⎥ 10 11 12 12 0 1 ⎥ ⎢ ⎥ ⎥ ⎢ ⎢ ⎢10 01 12 ⎥ ⎢101⎥ ⎢5⎥ 10 11 ⎥ ⎢ ⎥ ⎥ ⎢ ⎢ ⎣10 11 02 ⎦ ⎣011⎦ ⎣3⎦ 7 111 10 11 12 ⎡

Notice that the intermediate result of a binary Kronecker product should be written in conventional binary code sequence; for example, 00 01 12 should be written as 100.

2.7.4 FFT Algorithm ˆ = Fx shown in Equation (2.7.14). We now consider a fast algorithm for the DFT x To this end, perform Type-I elementary row operation on the augmented matrix ˆ ]. Clearly, during the elementary row operations, the subscript indices of the [F, x

2.7 Vandermonde Matrix and Fourier Matrix

119

row vectors of the Fourier matrix F and the subscript indices of the elements of the ˆ are the same. Hence, we have output vector x ˆ rev = FN,rev x. x

(2.7.26)

Therefore, the key step in deriving the N -point FFT algorithm is to construct the N × N bit-reversed Fourier matrix FN,rev . Letting ⎧ ⎫ 1 1 ⎪ ⎪ ⎪ ⎪ ⎪   ⎨ 1 −1 ⎪ ⎬ 1 1 , (2.7.27) {A}2 =   , B= ⎪ 1 −1 1 −j ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ ⎭ 1 j the N × N bit-reversed Fourier matrix can be recursively constructed by FN,rev = {A}N/2 ⊗ {A}N/4 ⊗ · · · ⊗ {A}2 ⊗ B, where + {A}N/2 =

{A}N/4 {R}N/4

0 ,

{R}N/4

⎧ ⎫ ⎪ ⎨ R1 ⎪ ⎬ .. = . ⎪ ⎪ ⎩ ⎭ RN/4

(2.7.28)

(2.7.29)

together with  Rk = EXAMPLE 2.7

 1 e−j(2k−1)π/N . 1 −e−j(2k−1)π/N

k = 1, . . . ,

N . 2

(2.7.30)

When N = 4, we have the 4 × 4 bit-reversed Fourier matrix ⎡

⎤ ⎡ 1 1 ⊗ [1, 1] ⎥ ⎢ 1 −1 ⎥ ⎢  ⎥=⎣ 1 −j ⎦ −j ⊗ [1, −1] 1 j j

1 1 1 −1

⎢ ⎢ F4,rev = {A}2 ⊗ B = ⎢ ⎣ 1 1



⎤ 1 1 1 −1 ⎥ ⎥ . (2.7.31) −1 j ⎦ −1 −j

From Equation (2.7.26), we have ⎡

⎤ ⎡ X0 ⎢ X2 ⎥ ⎢ ⎢ ⎥=⎢ ⎣X ⎦ ⎣ 1 X3

1 1 1 −1 1 −j 1 j

This is just the four-point FFT algorithm.

⎤⎡ ⎤ 1 1 x0 ⎢x 1 ⎥ 1 −1 ⎥ ⎥⎢ ⎥. −1 j ⎦ ⎣x 2 ⎦ −1 −j x3

(2.7.32)

120

Special Matrices

EXAMPLE 2.8

When N = 8, from (2.7.29) we have ⎧   ⎫ ⎪ ⎪ 1 1 ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ 1 −1 ⎪ ⎪ ⎪ ⎪ ⎪  ⎪  ⎪ ⎪ ⎪ ⎪ 1 −j ⎪ ⎪ ⎪ ⎪ ⎪ # ⎪ " ⎨ ⎬ 1 j {A}2 , {A}4 = =   ⎪ {R}2 1 e−jπ/4 ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ 1 −e−jπ/4 ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪   ⎪ ⎪ ⎪ ⎪ −j3π/4 ⎪ ⎪ 1 e ⎪ ⎪ ⎪ ⎩ 1 −e−j3π/4 ⎪ ⎭

(2.7.33)

and thus the 8 × 8 bit-reversed Fourier matrix is given by F8,rev = {A}4 ⊗ ({A}2 ⊗ B) = {A}4 ⊗ F4,rev ⎧   ⎫ ⎪ ⎪ 1 1 ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ 1 −1 ⎪ ⎪ ⎪ ⎪ ⎪ ⎪   ⎪ ⎪ ⎡ ⎤ ⎪ ⎪ 1 −j ⎪ ⎪ 1 1 1 1 ⎪ ⎪ ⎪ ⎪ ⎨ ⎬ ⎢ 1 j 1 −1 ⎥ ⎢ 1 −1 ⎥ ⊗ =   ⎣ 1 −j −1 −jπ/4 ⎪ ⎪ j ⎦ 1 e ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ −jπ/4 ⎪ ⎪ 1 j −1 −j ⎪ ⎪ ⎪ 1 −e ⎪ ⎪ ⎪  ⎪ ⎪ ⎪ ⎪ 1 ⎪ e−j3π/4 ⎪ ⎪ ⎪ ⎪ ⎩ 1 −e−j3π/4 ⎪ ⎭ ⎡ 1 1 1 1 1 1 ⎢1 −1 1 −1 1 −1 ⎢ ⎢1 −j −1 j 1 −j ⎢ ⎢ j −1 −j 1 j ⎢1 =⎢ ⎢1 e−jπ/4 −j e−j3π/4 −1 −e−jπ/4 ⎢ ⎢1 −e−jπ/4 −j −e−j3π/4 −1 e−jπ/4 ⎢ −jπ/4 ⎣1 e−j3π/4 j e −1 −e−j3π/4 −j3π/4 −jπ/4 1 −e j −e −1 e−j3π/4 From Equation (2.7.26) it ⎡ ⎤ ⎡ X0 1 1 ⎢X ⎥ ⎢1 −1 ⎢ 4⎥ ⎢ ⎢X ⎥ ⎢ 1 −j ⎢ 2⎥ ⎢ ⎢ ⎥ ⎢ j ⎢ X6 ⎥ ⎢ 1 ⎢ ⎥=⎢ ⎢ X1 ⎥ ⎢ 1 e−jπ/4 ⎢ ⎥ ⎢ ⎢X5 ⎥ ⎢1 −e−jπ/4 ⎢ ⎥ ⎢ ⎣X3 ⎦ ⎣1 e−j3π/4 1 −e−j3π/4 X7

⎤ 1 1 1 −1 ⎥ ⎥ ⎥ −1 j ⎥ ⎥ −1 −j ⎥ −j3π/4 ⎥. ⎥ j −e −j3π/4 ⎥ ⎥ j e ⎥ −jπ/4 ⎦ −j −e −j e−jπ/4 (2.7.34)

immediately follows that 1 1 1 −1 −1 j −1 −j −j e−j3π/4 −j −e−j3π/4 j e−jπ/4 j −e−jπ/4

1 1 1 −1 1 −j 1 j −1 −e−jπ/4 −1 e−jπ/4 −1 −e−j3π/4 −1 e−j3π/4

⎤⎡ ⎤ x0 1 1 ⎢x ⎥ 1 −1 ⎥ ⎥⎢ 1 ⎥ ⎥⎢x ⎥ −1 j ⎥⎢ 2 ⎥ ⎥⎢ ⎥ −1 −j ⎥⎢x3 ⎥ −j3π/4 ⎥⎢ ⎥ . ⎥⎢x4 ⎥ j −e ⎥⎢ ⎥ ⎢ ⎥ j e−j3π/4 ⎥ ⎥⎢x5 ⎥ −jπ/4 ⎦⎣ ⎦ −j −e x6 −j e−jπ/4 x7

2.8 Hadamard Matrix

121

(2.7.35) This is the eight-point FFT algorithm. Similarly, we can derive other N (= 16, 32, . . .)-point FFT algorithms.

2.8 Hadamard Matrix The Hadamard matrix is an important special matrix in communication, information theory, signal processing and so on. DEFINITION 2.13 Hn ∈ Rn×n is known as a Hadamard matrix if its entries take the values either +1 or −1 such that Hn HTn = HTn Hn = nIn .

(2.8.1)

The properties of the Hadamard matrix are as follows. (1) After multiplying the entries of any row or column of the Hadamard matrix by −1, the result is still a Hadamard matrix. Hence, we can obtain a special Hadamard matrix, with all entries of the first row and of the first column equal to +1. Such a special Hadamard matrix is known as a standardized Hadamard matrix. (2) An n × n Hadamard matrix exists only when n = 2k , where k is an integer. √ (3) (1/ n)Hn is an orthonormal matrix. (4) The determinant of the n × n Hadamard matrix Hn is det(Hn ) = nn/2 . Standardizing a Hadamard matrix will greatly help to construct a Hadamard matrix with higher-dimension. The following theorem gives a general construction approach for the standardized orthonomal Hadamard matrix. THEOREM 2.3 Putting n = 2k , where k = 1, 2, . . ., the standardized orthonormal Hadamard matrix has the following general construction formula: ¯  ¯ 1 Hn/2 Hn/2 ¯ Hn = √ , (2.8.2) ¯ ¯ 2 H −H n/2

where ¯ = √1 H 2 2



1 1 1 −1

n/2

 .

(2.8.3)

¯ is clearly a standardizing orthonormal Hadamard matrix, since Proof First, H 2 T ¯ ¯ ¯ ¯ ¯ k be the standardizing orthonormal Hadamard H2 H2 = H2 HT2 = I2 . Next, let H 2 k T ¯ ¯ Tk = I k k . Hence, for n = 2k+1 , it is ¯ ¯ kH matrix for n = 2 , namely H2k H2k = H 2 2 ×2 2 easily seen that   ¯ 2k H ¯ k 1 H 2 ¯ H2k+1 = √ ¯ k −H ¯ k 2 H 2

2

122

Special Matrices

satisfies the orthogonality condition, i.e.,   T ¯ Tk ¯ 2k ¯ k H H H 1 2 2 T ¯ k+1 = ¯ k+1 H H 2 2 ¯ Tk H ¯ Tk −H ¯ k 2 H 2

2

2

 ¯ k H 2 = I2k+1 ×2k+1 . ¯ k −H 2

¯ Tk+1 = I k+1 k+1 . Moreover, since H ¯ k ¯ k+1 H Similarly, it is easy to show that H 2 2 ×2 2 2 ¯ is standardized, H2k+1 is standardized as well. Hence, the theorem holds also for n = 2k+1 . By mathematical induction, the theorem is proved. A nonstandardizing Hadamard matrix can be written as Hn = Hn/2 ⊗ H2 = H2 ⊗ · · · ⊗ H2 (n = 2k ), (2.8.4)   1 1 . where H2 = 1 −1 Evidently, there is the following relationship between the standardizing and nonstandardizing Hadamard matrices: ¯ = √1 H . H n n n EXAMPLE 2.9

(2.8.5)

When n = 23 = 8, we have the Hadamard matrix       1 1 1 1 1 1 ⊗ ⊗ H8 = 1 −1 1 −1 1 −1 ⎤ ⎡ 1 1 1 1 1 1 1 1 ⎢ 1 −1 1 −1 1 −1 1 −1 ⎥ ⎥ ⎢ ⎢ 1 1 −1 −1 1 1 −1 −1 ⎥ ⎥ ⎢ ⎥ ⎢ 1 1 −1 −1 1 ⎥ ⎢ 1 −1 −1 =⎢ ⎥. ⎢ 1 1 1 1 −1 −1 −1 −1 ⎥ ⎢ ⎥ ⎢ 1 −1 1 −1 −1 1 −1 1 ⎥ ⎢ ⎥ ⎣ 1 1 −1 −1 −1 −1 1 1 ⎦ 1 −1 −1 1 −1 1 1 −1

Figure 2.1 shows the waveforms of the eight rows of the Hadamard matrix φ0 (t), φ1 (t), . . . , φ7 (t). It is easily seen that the Hadamard waveforms φ0 (t), φ1 (t), . . . , φ7 (t) are orthog1 onal to each other, namely 0 φi (t)φj (t)dt = δ(i − j). If H is a Hadamard matrix, the linear transformation Y = HX is known as the Hadamard transform of the matrix X. Since the Hadamard matrix is a standardizing orthonormal matrix, and its entries are only +1 or −1, the Hadamard matrix is the only orthonormal transformation using addition and subtraction alone. The Hadamard matrix may be used for mobile communication coding. The resulting code is called the Hadamard (or Walsh–Hadamard) code. In addition, owing to the orthogonality between its row vectors, they can be used to simulate the spread waveform vector of each user in a code division multiple access (CDMA) system.

2.9 Toeplitz Matrix φ0 (t) 1

φ1 (t) 1

1 t

123

φ2 (t) 1

φ3 (t) 1

1 t

1 t

1 t

−1

−1

−1

−1

φ4 (t) 1

φ5 (t) 1

φ6 (t) 1

φ7 (t) 1

1 t

1 t

1 t

−1

−1

−1

1 t −1

Figure 2.1 Waveform of each row of the Hadamard matrix H8 .

2.9 Toeplitz Matrix In the early twentieth century, Toeplitz [467] presented a special matrix in his paper on the bilinear function relating to the Laurent series. All entries on any diagonal of this special matrix take the same values, namely ⎡

a0 ⎢ a1 ⎢ ⎢a 2 A=⎢ ⎢ ⎢. ⎣ ..

a−1 a0

a−2 a−1

a1

a0

··· ··· .. .

.. .

..

..

an

an−1

. ···

. a1

⎤ a−n a−n+1 ⎥ ⎥ .. ⎥ n . ⎥ ⎥ = [ai−j ]i,j=0 . ⎥ a−1 ⎦

(2.9.1)

a0

This form of matrix with A = [ai−j ]ni,j=0 is known as the Toeplitz matrix. Clearly, an (n + 1) × (n + 1) Toeplitz matrix is determined completely by the entries a0 , a−1 , . . . , a−n of its first row and the entries a0 , a1 , . . . , an of its first column.

2.9.1 Symmetric Toeplitz Matrix The entries of a symmetric Toeplitz matrix A = [a|i−j| ]ni,j=0 satisfy the symmetry relations a−i = ai , i = 1, 2, . . . , n. Thus, a symmetric Toeplitz matrix is completely described by the entries of its first row. An (n + 1) × (n + 1) symmetric Toeplitz matrix is usually denoted Toep[a0 , a1 , . . . , an ].

124

Special Matrices

If the entries of a complex Toeplitz matrix ⎡ a0 a∗1 a∗2 ⎢ ⎢ a1 a0 a∗1 ⎢ ⎢ A = ⎢ a2 a1 a0 ⎢ ⎢ .. .. . .. ⎣. . an

an−1

···

satisfy a−i = a∗i , i.e., ⎤ ··· a∗n ⎥ · · · a∗n−1 ⎥ ⎥ .. ⎥ .. , . . ⎥ ⎥ ⎥ .. . a∗ ⎦

(2.9.2)

1

a1

a0

then it is known as a Hermitian Toeplitz matrix. Furthermore, an (n + 1) × (n + 1) Toeplitz matrix with the special structure ⎤ ⎡ 0 −a∗1 −a∗2 · · · −a∗n ⎥ ⎢ ⎢ a1 0 −a∗1 · · · −a∗n−1 ⎥ ⎥ ⎢ .. ⎥ ⎢ .. (2.9.3) AS = ⎢ a 2 . a1 0 . ⎥ ⎥ ⎢ ⎥ ⎢ .. .. . . .. .. ⎣. . −a∗ ⎦ an

an−1

···

1

a1

0

is called a skew-Hermitian Toeplitz matrix, whereas ⎡ ⎤ a0 −a∗1 −a∗2 · · · −a∗n ⎢ ⎥ ⎢ a1 a0 −a∗1 · · · −a∗n−1 ⎥ ⎢ ⎥ .. ⎥ ⎢ .. A = ⎢ a2 ⎥ . a a . 1 0 ⎢ ⎥ ⎢ .. ⎥ .. . . . . ∗ ⎣. . . . −a1 ⎦ an an−1 · · · a1 a0

(2.9.4)

is known as a skew-Hermitian-type Toeplitz matrix. Toeplitz matrices have the following properties [397]. (1) The linear combination of several Toeplitz matrices is also a Toeplitz matrix. (2) If the entries of a Toeplitz matrix A satisfy aij = a|i−j| , then A is a symmetric Toeplitz matrix. (3) The transpose AT of the Toeplitz matrix A is also a Toeplitz matrix. (4) The entries of a Toeplitz matrix are symmetric relative to its cross-diagonal lines. In statistical signal processing and other related fields, it is necessary to solve the matrix equation Ax = b with symmetric Toeplitz matrix A. Such a system of linear equations is called a Toeplitz system. By using the special structure of the Toeplitz matrix, a class of Levinson recursive algorithms has been proposed for solving a Toeplitz system of linear equations. For a real positive-definite Toeplitz matrix, its classical Levinson recursion [291] involved a large redundancy in the calculation. In order to reduce the computational

2.9 Toeplitz Matrix

125

redundancy, Delsarte and Genin presented successively a split Levinson algorithm [120] and a split Schur algorithm [121]. Later, Krishna and Morgera [265] extended the split Levinson recursion to a Toeplitz system of complex linear equations. Although these algorithms are recursive, their computational complexity is O(n2 ). In addition to the Levinson recursive algorithms, one can apply an FFT algorithm to solve the Toeplitz system of linear equations, and this class of algorithms only requires a computational complexity O(n log2 n), which is faster than the Levinson recursive algorithms. In view of this, FFT algorithms for solving the Toeplitz system of linear equations are called fast Toeplitz algorithms in most of the literature. These fast Toeplitz algorithms include the Kumar algorithm [269], the Davis algorithm [117] and so on, while some publications refer to this class of FFT algorithms as superfast algorithms. The computation complexity of the Kumar algorithm is O(n log2 n). In particular, the conjugate gradient algorithm in [89] requires only a computational complexity O(n log n) for solving the Toeplitz system of complex linear equations.

2.9.2 Discrete Cosine Transform of Toeplitz Matrix An N ×N Toeplitz matrix can be computed fast using an N th-order discrete cosine transform (DCT). The N th-order DCT can be expressed as an N × N matrix T, where the (m, l)th −1 entry of T = [tm,l ]N m,l=0 is defined as ) π * tm,l = τm cos m(2l + 1) for m, l = 0, 1, . . . , N − 1, (2.9.5) 2N + 1/N , m = 0, (2.9.6) τm =  2/N , m = 1, 2, . . . , N − 1. From the above definition, it is easy to see that T−1 = TT , namely T is an ˆ = TATT orthogonal matrix. Hence, given any N × N matrix A, its DCT matrix A and the original matrix A have the same eigenvalues. Algorithm 2.1 shows the fast DCT algorithm for a Toeplitz matrix. The common way of computing Toeplitz matrix eigenvalues is to adopt some procedure (such as the Given rotation) to transform the nondiagonal entries of A to zero, namely to diagonalize A. As mentioned above, since T is an orthogonal ˆ = TATT and A have the same eigenvalues. Hence, we can diagonalize the matrix, A ˆ DCT A instead of A itself to find the eigenvalues of A. In some cases, the diagonal ˆ are already exact estimates of the eigenvalues of A; entries of the DCT matrix A see e.g. [185]. ˆ can be used as an approxSome examples show [357] that the fast DCT matrix A ˆ are focused imation of the Toeplitz matrix A because the main components of A on a much smaller matrix partition that corresponds to the steady part of A. From

126

Special Matrices

Algorithm 2.1

Fast DCT algorithm for a Toeplitz matrix [357]

−1 N −1 input: Toeplitz matrix A = [al,k ]N l,k=0 = [ak−l ]l,k=0 .

1. For m = 0, 1, . . . , N − 1, compute N −1 π xm,0 = . wm(2l+1) al,0 , w = exp −j 2N l=0

(Complexity: one 2N -point DFT.) 2. For n = −(N − 1), . . . , (N − 1), compute N −1 N −1 v1 (n) = w2nk ak+1 , v2 (n) = w2nk a1−N +k . k=0

k=0

(Complexity: two 2N -point DFTs.) 3. For m = 0, 1, . . . , N − 1, compute xm,N = (−1)m xm,0 + w−m ((−1)m v1 (n) − v2 (n)). (Complexity: N -times multiplication.) 4. For n = −(N − 1), . . . , (N − 1), compute N −1 N −1 kw2nk ak+1 , u2 (n) = kw2nk a1−N +k . u1 (n) = k=0

k=0

(Complexity: two 2N -point DFTs.) 5. For m = 0, 1, . . . , N − 1; n = −(N − 1), . . . , (N − 1), compute   1 ym,n = −2n xm,0 w−2n − xm,N w−2n (−1)n + wm (v1 (n)−(−1)m v2 (n)) , w − w2m ym,−m = xm,0 + (N − 1)(−1)m xm,N − w−m (u1 (−m) − (−1)m u2 (−m)). (Complexity: several multiplications are needed for each value.) 6. For m = 0, 1, . . . , N − 1; n = −(N − 1), . . . , (N − 1), compute   ∗ ym,n + ym,−n . a ˆm,n = τm τn Re wn 2 (Complexity: several multiplications are required for each value.)

ˆ is easily found the viewpoint of a decision about rank, that of the DCT matrix A seen but the rank of A is difficult to ascertain. Notice that, since A and its DCT ˆ have the same eigenvalues, their ranks are the same. A

Exercises 2.1 2.2 2.3 2.4

Let A be an idempotent and symmetric matrix. Show that A is semi-definite. Let A be an n × n antisymmetric matrix such that A = −AT . Show that if n is an odd number then the determinant of A must be equal to zero. Show that if A = −AT then A + I is nonsingular. Assume that A = −AT . Show that the Cayley transform T = (I−A)(I+A)−1 is an orthogonal matrix.

Exercises

2.5

Prove the following properties of an idempotent matrix. (a) (b) (c) (d)

2.6

127

The eigenvalues of any idempotent matrix take only the values 1 and 0. All idempotent matrices (except for the identity matrix) A are singular. If A is an idempotent matrix then rank(A) = tr(A). If A is an idempotent matrix then AH is also an idempotent matrix.

If A is an idempotent matrix, show that (a) the matrices Ak and A have the same eigenvalues. (b) Ak and A have the same rank.

2.7

2.8 2.9 2.10 2.11 2.12 2.13

Let A be an orthogonal matrix, and let A + I be nonsingular. Show that the matrix A can be expressed as the Cayley transform A = (I − S)(I + S)−1 , where S is a real antisymmetric matrix such that A = −AT . Let P be an n×n permutation matrix. Show that there exists a positive integer k such that Pk = I. (Hint: Consider the matrix sequence P, P2 , P3 , . . .) Assume that P and Q are two n × n permutation matrices. Show that PQ is also an n × n permutation matrix. Show that, for any matrix A, there is a triangular matrix T such that TA is unitary. Show that, for a given matrix A, there is a matrix J, whose diagonal entries take the values +1 or −1, such that JA + I is nonsingular. Show that if H = A + j B is a Hermitian matrix and A is nonsingular then | det(H)|2 = |A|2 |I + A−1 BA−1 B|. Let A, S be n × n matrices and let S be nonsingular. (a) Show that (S−1 AS)2 = S−1 A2 S and (S−1 AS)3 = S−1 A3 S. (b) Use mathematical induction to show that (S−1 AS)k = S−1 Ak S, where k is a positive integer.

2.14 Let A be an n × n real matrix. Show that B = (A + AT )/2 is a symmetric matrix, while C = (A − AT )/2 is an antisymmetric matrix. 2.15 Let yi = yi (x1 , . . . , xn ), i = 1, . . . , n be n functions of x1 , . . . , xn . The matrix J = J(y, x) = [∂yi /∂xj ] is called the Jacobian matrix of the function yi (x1 , . . . , xn ), i = 1, . . . , n, and its determinant is the Jacobian determinant; y = [y1 , . . . , yn ]T and x = [x1 , . . . , xn ]T . Show that J(z, y)J(y, x) = J(z, x). 2.16 Let n × n matrices X and Y be symmetric, and let Y = AXAT . Show that the Jacobian determinant |J(Y, X)| = |A|n+1 . 2.17 [32, p. 265] An n × n matrix M is known as a Markov matrix if its entries n satisfy the condition mij ≥ 0, i=1 mij = 1, j = 1, 2, . . . , n. Assuming that P and Q are two Markov matrices, show that (a) for a constant 0 ≤ λ ≤ 1, the matrix λP + (1 − λ)Q is a Markov matrix. (b) the matrix product PQ is a Markov matrix as well. 2.18 Show the following properties of the Fourier matrix F:

128

Special Matrices

(a) F−1 = N −1 F∗ . (b) F2 = [e1 , eN , eN −1 , . . . , e2 ]. (c) F4 = I. 2.19 The square matrix A satisfying AAH = AH A is known as a normal matrix. Show that if A is a normal matrix then the matrix A − λI is also a normal matrix. 2.20 Two matrices A and B such that their product satisfies the commutative law AB = BA are called exchangeable matrices. Show that if A and B are exchangeable then the condition for AH and B to be exchangeable is that A is a normal matrix. 2.21 Let A be an n × n complex matrix with eigenvalues λ1 , . . . , λn . Show that A is a normal matrix if and only if one of the following conditions holds: (a) (b) (c) (d)

A = B + jC where B and C are Hermitian and exchangeable matrices. The eigenvalues of AAH are |λ1 |2 , . . . , |λn |2 . The eigenvalues of A + AH are λ1 + λ∗1 , . . . , λn + λ∗n . A = UΣUH , where U is a unitary matrix and Σ is a diagonal matrix.

3 Matrix Differential

The matrix differential is a generalization of the multivariate function differential. The matrix differential (including the matrix partial derivative and gradient) is an important operation tool in matrix analysis and calculation and has wide applications in statistics, optimization, manifold computation, geometric physics, differential geometry, econometrics and many engineering disciplines.

3.1 Jacobian Matrix and Gradient Matrix In the first half of this chapter we discuss the partial derivatives of a real scalar function, real vector function and real matrix function with respect to a real vector or matrix variable. In order to facilitate understanding, the symbols for variables and functions are first introduced below. • x = [x1 , . . . , xm ]T ∈ Rm denotes a real vector variable. • X = [x1 , . . . , xn ] ∈ Rm×n denotes a real matrix variable. • f (x) ∈ R is a real scalar function with an m × 1 real vector x as variable, denoted f : Rm → R, such as f (x) = tr(xxT ). • f (X) ∈ R is a real scalar function whose variable is an m × n real matrix X, denoted f : Rm×n → R; e.g., f (X) = det(XT X). • f (x) ∈ Rp is a p-dimensional column vector function with an m × 1 real vector x as variable, denoted f : Rm → Rp ; e.g., f (x) = bx + c. • f (X) ∈ Rp is a p-dimensional real column vector function whose variable is an m × n real matrix X, denoted f : Rm×n → Rp , such as f (X) = vec(XT X). • F(x) ∈ Rp×q is a p × q real matrix function whose variable is an m × 1 real vector x, denoted F : Rm → Rp×q ; e.g., F(x) = xxT . • F(X) ∈ Rp×q is a p × q real matrix function whose variable is an m × n real matrix X, denoted F : Rm×n → Rp×q ; e.g., F(X) = XT X. Table 3.1 summarizes the classification of the real functions above. 129

130

Matrix Differential

Table 3.1 Classification of real functions Function type

Variable x ∈ Rm

Variable X ∈ Rm×n

scalar function f ∈ R

f (x) f : Rm → R f (x) f : Rm → Rp F(x) F : Rm → Rp×q

f (X) f : Rm×n → R f (X) f : Rm×n → Rp F(X) F : Rm×n → Rp×q

vector function f ∈ Rp matrix function F ∈ Rp×q

In this section we discuss the partial derivatives of a real scalar function and a real matrix function.

3.1.1 Jacobian Matrix DEFINITION 3.1 The row partial derivative operator with respect to an m × 1 vector or an m × n matrix X are respectively defined as   ∂ ∂ ∂ def (3.1.1) = ,..., Dx = ∂ xT ∂x1 ∂xm and Dvec X

  ∂ ∂ ∂ ∂ ∂ . = = ,..., ,..., ,..., ∂(vec X)T ∂x11 ∂xm1 ∂x1n ∂xmn

def

The Jacobian operator with respect to an m × n matrix X is defined as ⎡ ⎤ ∂ ∂ ··· ⎢ ∂x11 ∂xm1 ⎥ ⎢ . ∂ def .. ⎥ . ⎢ ⎥ .. DX = = ⎢ .. . ⎥. T ∂X ⎣ ∂ ∂ ⎦ ··· ∂x1n

(3.1.2)

(3.1.3)

∂xmn

DEFINITION 3.2 The partial derivative vector resulting from the operation of Dx on a real scalar function f (x) with m × 1 vector variable x is an 1 × m row vector and is given by   ∂f (x) ∂f (x) ∂f (x) . (3.1.4) = ,..., Dx f (x) = ∂ xT ∂x1 ∂xm The partial derivative matrix ⎡ ∂f (X) ··· ⎢ ∂x11 ∂f (X) ⎢ . .. =⎢ DX f (X) = . ⎢ .. ∂ XT ⎣ ∂f (X) ···

DEFINITION 3.3

∂x1n



∂f (X) ∂xm1 ⎥

⎥ ⎥ ∈ Rn×m ⎥ ∂f (X) ⎦ .. .

∂xmn

(3.1.5)

3.1 Jacobian Matrix and Gradient Matrix

131

is known as the Jacobian matrix of the real scalar function f (X), and the partial derivative vector ∂f (X) ∂(vec X)T   ∂f (X) ∂f (X) ∂f (X) ∂f (X) = ,..., ,..., ,..., ∂x11 ∂xm1 ∂x1n ∂xmn

Dvec X f (X) =

(3.1.6)

is the row partial derivative vector of the real scalar function f (X) with matrix variable X. There is the following relationship between the Jacobian matrix and the row partial derivative vector:  T Dvec X f (X) = rvecDX f (X) = vecDTX f (X) . (3.1.7) That is to say, the row vector partial derivative Dvec X f (X) is equal to the transpose of the column vectorization vec(DTX f (X)) of the Jacobian matrix DTX f (X). This important relation is the basis of the Jacobian matrix identification. As a matter of fact, the Jacobian matrix is more useful than the row partial derivative vector. The following theorem provides a specific expression for the Jacobian matrix of a p × q real-valued matrix function F(X) with m × n matrix variable X. DEFINITION 3.4 [311] Let the vectorization of a p × q matrix function F(X) be given by def

vecF(X) = [f11 (X), . . . , fp1 (X), . . . , f1q (X), . . . , fpq (X)]T ∈ Rpq .

(3.1.8)

Then the pq × mn Jacobian matrix of F(X) is defined as def

DX F(X) = whose specific expression DX F(X) ⎡ ⎤ ⎡ ∂f11 ∂f11 ⎢ ∂(vec X)T ⎥ ⎢ ∂x11 · · · ⎢ ⎥ ⎢ . .. .. ⎢ ⎥ ⎢ . . ⎢ ⎥ . . ⎢ ∂fp1 ⎥ ⎢ ∂fp1 ⎢ ⎥ ⎢ ⎢ ··· ⎢ ∂(vec X)T ⎥ ⎢ ∂x ⎢ ⎥ ⎢ 11 ⎢ ⎥ ⎢ .. .. .. ⎢ ⎥=⎢ . . . ⎢ ⎥ ⎢ ⎢ ∂f1q ⎥ ⎢ ∂f1q ⎢ ⎥ ··· ⎢ ∂(vec X)T ⎥ ⎢ ⎢ ∂x11 ⎢ ⎥ .. .. .. ⎢ ⎥ ⎢ . . . ⎢ ⎥ ⎢ ⎣ ⎣ ∂fpq ⎦ ∂fpq ··· T ∂(vec X)

∂x11

∂ vecF(X) ∈ Rpq×mn ∂(vecX)T

(3.1.9)

is given by ∂f11 ∂xm1

··· .. .

∂f11 ∂x1n

··· .. .

∂fp1 ∂xm1

··· .. .

∂fp1 ∂x1n

··· .. .

∂f1q ∂xm1

··· .. .

∂f1q ∂x1n

··· .. .

∂fpq ∂xm1

···

∂fpq ∂x1n

···

.. . .. . .. .

.. . .. . .. .



∂f11 ∂xmn ⎥

⎥ ⎥ ⎥ ∂fp1 ⎥ ⎥ ∂xmn ⎥ .. ⎥ ⎥ . ⎥. ∂f1q ⎥ ⎥ ∂xmn ⎥ ⎥ .. ⎥ . ⎥ ⎦ .. .

∂fpq ∂xmn

(3.1.10)

132

Matrix Differential

3.1.2 Gradient Matrix The partial derivative operator in column form is known as the gradient vector operator. DEFINITION 3.5 The gradient vector operators with respect to an m × 1 vector x and to an m × n matrix X are respectively defined as T  ∂ ∂ def ∂ ∇x = ,..., , (3.1.11) = ∂x ∂x1 ∂xm and ∇vec X

T  ∂ ∂ ∂ ∂ ∂ = ,..., ,..., ,..., . = ∂ vec X ∂x11 ∂x1n ∂xm1 ∂xmn

def

DEFINITION 3.6 X is defined as

(3.1.12)

The gradient matrix operator with respect to an m × n matrix ⎡ def

∇X =

∂ ⎢ ∂x11

⎢ . ∂ .. =⎢ ∂X ⎢ ⎣ ∂

∂xm1

··· ..

.

···



∂ ∂x1n ⎥

.. .

∂ ∂xmn

⎥ ⎥. ⎥ ⎦

(3.1.13)

DEFINITION 3.7 The gradient vectors of functions f (x) and f (X) are respectively defined as T  ∂f (x) ∂f (x) def ∂f (x) ∇x f (x) = ,..., = , (3.1.14) ∂x1 ∂xm ∂x T  ∂f (X) ∂f (X) ∂f (X) def ∂f (X) ,..., ,..., ,..., . (3.1.15) ∇vec X f (X) = ∂x11 ∂xm1 ∂x1n ∂xmn DEFINITION 3.8

The gradient matrix of the function f (X) is defined as ⎡ ⎤ ∂f (X) ∂f (X) ··· ⎢ ∂x11 ∂x1n ⎥ ⎢ . .. ⎥ . ⎢ ⎥ ∂f (X) .. ∇X f (X) = ⎢ .. (3.1.16) . ⎥= ∂X . ⎣ ∂f (X) ∂f (X) ⎦ ··· ∂xm1

∂xmn

Clearly, there is the following relationship between the gradient matrix and the gradient vector of a function f (X): ∇X f (X) = unvec∇vec X f (X).

(3.1.17)

Comparing Equation (3.1.16) with Equation (3.1.5), we have ∇X f (X) = DTX f (X).

(3.1.18)

That is to say, the gradient matrix of a real scalar function f (X) is equal to the transpose of its Jacobian matrix.

3.1 Jacobian Matrix and Gradient Matrix

133

As pointed out by Kreutz-Delgado [262], in manifold computation [3], geometric physics [430], [162], differential geometry [452] and so forth, the row partial derivative vector and the Jacobian matrix are the “most natural” choices. As we will see later, the row partial derivative vector and the Jacobian matrix are also the most natural choices for matrix derivatives. However, in optimization and many engineering applications, the gradient vector and the gradient matrix are a more natural choice than the row partial derivative vector and the Jacobian matrix. An obvious fact is that, given a real scalar function f (x), its gradient vector is directly equal to the transpose of the partial derivative vector. In this sense, the partial derivative in row vector form is a covariant form of the gradient vector, so the row partial derivative vector is also known as the cogradient vector. Similarly, the Jacobian matrix is sometimes called the cogradient matrix. The cogradient is a covariant operator [162] that itself is not the gradient, but is related to the gradient. For this reason, the partial derivative operator ∂/∂xT and the Jacobian operator ∂/∂ XT are known as the (row) partial derivative operator, the covariant form of the gradient operator or the cogradient operator. The direction of the negative gradient −∇x f (x) is known as the gradient flow direction of the function f (x) at the point x, and is expressed as x˙ = −∇x f (x) or

˙ = −∇ X vec X f (X).

(3.1.19)

From the definition formula of the gradient vector the following can be deduced. (1) In the gradient flow direction, the function f (x) decreases at the maximum descent rate. On the contrary, in the oppositive direction (i.e., the positive gradient direction), the function increases at the maximum ascent rate. (2) Each component of the gradient vector gives the rate of change of the scalar function f (x) in the component direction. DEFINITION 3.9 For a real matrix function F(X) ∈ Rp×q with matrix variable X ∈ Rm×n , its gradient matrix is defined as T  ∂ (vec F(X))T ∂ vecF(X) ∇X F(X) = . (3.1.20) = ∂ vecX ∂(vec X)T Obviously, one has ∇X F(X) = (DX F(X))T .

(3.1.21)

In other words, the gradient matrix of a real matrix function is a transpose of its Jacobian matrix.

3.1.3 Calculation of Partial Derivative and Gradient The gradient computation of a real function with respect to its matrix variable has the following properties and rules [307].

134

Matrix Differential

1. If f (X) = c, where c is a real constant and X is an m × n real matrix, then the gradient ∂c/∂X = Om×n . 2. Linear rule If f (X) and g(X) are respectively real-valued functions of the matrix variable X, and c1 and c2 are two real constants, then ∂f (X) ∂g(X) ∂[c1 f (X) + c2 g(X)] . = c1 + c2 ∂X ∂X ∂X

(3.1.22)

3. Product rule If f (X), g(X) and h(X) are real-valued functions of the matrix variable X then ∂(f (X)g(X)) ∂f (X) ∂g(X) = g(X) + f (X) ∂X ∂X ∂X

(3.1.23)

and ∂(f (X)g(X)h(X)) ∂f (X) ∂g(X) = g(X)h(X) + f (X)h(X) ∂X ∂X ∂X ∂h(X) . + f (X)g(X) ∂X 4. Quotient rule

If g(X) = 0 then

1 ∂(f (X)/g(X)) = 2 ∂X g (X)



 ∂f (X) ∂g(X) g(X) . − f (X) ∂X ∂X

(3.1.24)

(3.1.25)

5. Chain rule If X is an m × n matrix and y = f (X) and g(y) are respectively the real-valued functions of the matrix variable X and of the scalar variable y then ∂g(f (X)) dg(y) ∂f (X) . = ∂X dy ∂X

(3.1.26)

As an extension, if g(F(X)) = g(F), where F = [fkl ] ∈ Rp×q and X = [xij ] ∈ R then the chain rule is given by [386]   p q ∂g(F)   ∂g(F) ∂fkl ∂g(F) = = . (3.1.27) ∂ X ij ∂xij ∂fkl ∂xij m×n

k=1 l=1

When computing the partial derivative of the functions f (x) and f (X), we have to make the following basic assumption. Independence assumption Given a real-valued function f , we assume that the m m×n and the matrix variable X = [xij ]m,n vector variable x = [xi ]m i=1 ∈ R i=1,j=1 ∈ R do not themselves have any special structure; namely, the entries of x and X are independent. The independence assumption can be expressed in mathematical form as follows: + 1, i = j, ∂xi = δij = (3.1.28) ∂xj 0, otherwise,

3.1 Jacobian Matrix and Gradient Matrix

and ∂xkl = δki δlj = ∂xij

+

1,

k = i and l = j,

0,

otherwise.

135

(3.1.29)

Equations (3.1.28) and (3.1.29) are the basic formulas for partial derivative computation. Below we will give a few examples. EXAMPLE 3.1 xT Ax.

Find the Jacobian matrix of the real-valued function f (x) =

Solution Because xT Ax =

n n

akl xk xl , by using Equation (3.1.28) we can find

k=1 l=1

the ith component of the row partial derivative vector ∂ xT Ax/∂xT , given by   T n n n n   ∂  ∂x Ax = a x x = x a + xl ail , kl k l k ki ∂xT ∂xi i k=1 l=1

k=1

l=1

from which we immediately get the row partial derivative vector Df (x) = xT A + xT AT = xT (A + AT ) and the gradient vector ∇X f (x) = (Df (x))T= (A + AT )x. EXAMPLE 3.2 Find the Jacobian matrix of the real function f (x) = aT XXT b, where X ∈ Rm×n and a, b ∈ Rn×1 . Solution Noticing that T

T

a XX b =

m  m 

 ak

k=1 l=1

n 

 xkp xlp

bl

p=1

and using Equation (3.1.29), we can see easily that   m m n ∂f (X)    ∂ak xkp xlp bl ∂f (X) = = ∂ XT ij ∂xji ∂xji k=1 l=1 p=1  m  m  n   ∂xkp ∂xlp ak xlp bl = + ak xkp bl ∂xji ∂xji p=1 =

=

k=1 l=1 m  m  n 

aj xli bl +

i=1 l=1 j=1 n m    T

n m  m  

ak xki bj

k=1 i=1 j=1

   X b i aj + XT a i bj ,

i=1 j=1

which yields respectively the Jacobian matrix and the gradient matrix, DX f (X) = XT (baT + abT ) and EXAMPLE 3.3

∇X f (X) = (baT + abT )X.

Consider the objective function f (X) = tr(XB), where X and B

136

Matrix Differential

are m × n and n × m real matrices, respectively. Find the Jacobian matrix and the gradient matrix of f (X) = tr(XB). Solution First, the (k, l)th entry of the matrix product XB is given by [XB]kl = n m n p=1 xkp bpl , so the trace of the matrix product is tr(XB) = p=1 l=1 xlp bpl . Hence, using Equation (3.1.29), it is easily found that m n    n m    ∂xlp ∂ tr(XB) ∂ = = x b bpl = bij , lp pl T ∂X ∂xji p=1 ∂xji ij p=1 l=1

l=1

that is, ∂ tr(XB)/∂ XT = B. Moreover, since tr(BX) = tr(XB), we see that the n × m Jacobian matrix and the m × n gradient matrix are respectively given by DX tr(XB) = DX tr(BX) = B and

∇X tr(XB) = ∇X tr(BX) = BT .

The following are examples of computing the Jacobian matrix and the gradient matrix of a real matrix function. EXAMPLE 3.4 Setting F(X) = X ∈ Rm×n and then computing the partial derivative directly yields ∂xkl ∂fkl = = δlj δki , ∂xij ∂xij which yields the Jacobian matrix DX X = In ⊗ Im = Imn ∈ Rmn×mn . EXAMPLE 3.5 Let F(X) = AXB, where A ∈ Rp×m , X ∈ Rm×n , B ∈ Rn×q . From the partial derivative m n ∂ ( u=1 v=1 aku xuv bvl ) ∂fkl ∂[AXB]kl = = = bjl aki , ∂xij ∂xij ∂xij we get respectively the pq × mn Jacobian matrix and the mn × pq gradient matrix: DX (AXB) = BT ⊗ A and

∇X (AXB) = B ⊗ AT .

EXAMPLE 3.6 Let F(X) = AXT B, where A ∈ Rp×n , X ∈ Rm×n , B ∈ Rm×q . Compute the partial derivative m n ∂ ( u=1 v=1 aku xvu bvl ) ∂fkl ∂[AXT B]kl = = = bil akj ∂xij ∂xij ∂xij to obtain respectively the pq×mn Jacobian matrix and the mn×pq gradient matrix: DX (AXT B) = (BT ⊗ A)Kmn

and

∇X (AXT B) = Knm (B ⊗ AT ),

where Kmn and Knm are commutation matrices.

3.1 Jacobian Matrix and Gradient Matrix

137

Let X ∈ Rm×n , then n ∂ ( u=1 xku xlu ) ∂fkl ∂[XXT ]kl = = = δli xkj + xlj δki , ∂xij ∂xij ∂xij n ∂ ( u=1 xuk xul ) ∂[XT X]kl ∂fkl = = = xil δkj + δlj xik . ∂xij ∂xij ∂xij

EXAMPLE 3.7

Hence the Jacobian matrices are given by DX (XXT ) = (Im ⊗ X)Kmn + (X ⊗ Im ) = (Kmm + Im2 )(X ⊗ Im ) ∈ Rmm×mn , DX (XT X) = (XT ⊗ In )Kmn + (In ⊗ XT ) = (Knn + In2 )(In ⊗ XT ) ∈ Rnn×mn and the gradient matrices are given by ∇X (XXT ) = (XT ⊗ Im )(Kmm + Im2 ) ∈ Rmn×mm , ∇X (XT X) = (In ⊗ X)(Knn + In2 ) ∈ Rmn×nn in which we have used (Ap×m ⊗Bq×n )Kmn = Kpq (Bq×n ⊗Ap×m ) and KTmn = Knm . EXAMPLE 3.8 by

The partial derivatives of the products of three matrices are given    ∂ p xpk [BX]pl + [XT B]kp xpl ∂[XT BX]kl = ∂xij ∂xij   = δpi δkj [BX]pl + δpi δlj [XT B]kp p

∂[XBXT ]kl ∂xij

= [BX]il δkj + δlj [XT B]ki    ∂ p xkp [BXT ]pl + [XB]kp xlp = ∂xij * ) δki δpj [BXT ]pl + δpj δli [XB]kp = p

= [BXT ]jl δki + δli [XB]kj , and thus we obtain respectively the Jacobian matrix and the gradient matrix:     DX (XT BX) = (BX)T ⊗ In Kmn + In ⊗ (XT B) ∈ Rnn×mn , DX (XBXT ) = (XBT ) ⊗ Im + (Im ⊗ (XB)) Kmn ∈ Rmm×mn ;   ∇X (XT BX) = Knm ((BX) ⊗ In ) + In ⊗ (BT X) ∈ Rmn×nn ,   ∇X (XBXT ) = (BXT ) ⊗ Im + Knm Im ⊗ (XB)T ∈ Rmn×mm . Table 3.2 summarizes the partial derivative, the Jacobian matrix and the gradient matrix for several typical matrix functions.

138

Matrix Differential

Table 3.2 Partial derivative, Jacobian matrix and gradient matrix of F(X) F(X)

∂fkl /∂xij

DX F(X)

∇X F(X)

AXB

bjl aki

BT ⊗ A

B ⊗ AT

AT XB

bjl aik

B T ⊗ AT

B⊗A

T

blj aki

B⊗A

B T ⊗ AT

AT XBT

blj aik

B ⊗ AT

BT ⊗ A

AXT B

bil akj

(BT ⊗ A)Kmn

Knm (B ⊗ AT )

AT XT B

bil ajk

(BT ⊗ AT )Kmn

Knm (B ⊗ A)

AXT BT

bli akj

(B ⊗ A)Kmn

Knm (BT ⊗ AT )

AT XT B T

bli ajk

(B ⊗ AT )Kmn

Knm (BT ⊗ A)

XXT

δli xkj + xlj δki

(Kmm + Im2 )(X ⊗ Im )

(XT ⊗ Im )(Kmm + Im2 )

XT X

xil δkj + δlj xik

(Knn + In2 )(In ⊗ XT )   (BX)T ⊗ In Kmn   + In ⊗ (XT B)

(In ⊗ X)(Knn + In2 )

(XBT ) ⊗ Im

(BXT ) ⊗ Im   +Knm Im ⊗ (XB)T

AXB

XT BX

XBXT

[BX]il δkj + δlj [XT B]ki [BXT ]jl δki + δli [XB]kj

+ (Im ⊗ (XB)) Kmn

Knm ((BX) ⊗ In )   + In ⊗ (BT X)

If X = x ∈ Rm×1 then from Table 3.2 we obtain Dx (xxT ) = Im2 (x ⊗ Im ) + Kmm (x ⊗ Im ) = (x ⊗ Im ) + (Im ⊗ x),

(3.1.30)

since Kmm (x ⊗ Im ) = (Im ⊗ x)Km1 and Km1 = Im . Similarly, we can obtain Dx (xxT ) = (K11 + I1 )(I1 ⊗ xT ) = 2xT .

(3.1.31)

It should be noted that direct computation of partial derivatives ∂fkl /∂xij can be used to find the Jacobian matrices and the gradient matrices of many matrix functions, but, for more complex functions (such as the inverse matrix, the Moore– Penrose inverse matrix and the exponential functions of a matrix), direct computation of their partial derivatives is more complicated and difficult. Hence, naturally, we want to have an easily remembered and effective mathematical tool for computing the Jacobian matrices and the gradient matrices of real scalar functions and real matrix functions. Such a mathematical tool is the matrix differential, which is the topic of the next section.

3.2 Real Matrix Differential

139

3.2 Real Matrix Differential The matrix differential is an effective mathematical tool for computing the partial derivatives of a function with respect to its variables. This section introduces the first-order real matrix differential together with its theory, calculation methods and applications.

3.2.1 Calculation of Real Matrix Differential The differential of an m × n matrix X = [xij ] is known as the matrix differential, denoted dX, and is defined as dX = [dxij ]m,n i=1,j=1 . EXAMPLE 3.9

Consider the differential of the scalar function tr(U). We have   n n   d(tr U) = d uii = duii = tr(dU), i=1

i=1

namely d(tr U) = tr(dU). EXAMPLE 3.10 given by

An entry of the matrix differential of the matrix product UV is

      [d(UV)]ij = d [UV]ij = d uik vkj = d(uik vkj ) =



k



(duik )vkj + uik dvkj =

k

 k

k

(duik )vkj +



uik dvkj

k

= [(dU)V]ij + [U(dV)]ij . Hence, we have for the matrix differential d(UV) = (dU)V + U(dV). The real matrix differential has the following two basic properties. (1) Transpose The differential of a matrix transpose is equal to the transpose of the matrix differential, i.e., d(XT ) = (dX)T = dXT . (2) Linearity d(αX + βY) = α dX + β dY. The following summarizes common computation formulas for the matrix differential [311, pp. 148–154]. 1. The differential of a constant matrix is a zero matrix, namely dA = O. 2. The matrix differential of the product αX is given by d(αX) = α dX. 3. The matrix differential of a transposed matrix is equal to the transpose of the original matrix differential, namely d(XT ) = (dX)T = dXT . 4. The matrix differential of the sum (or difference) of two matrices is given by d(U ± V) = dU ± dV. 5. The matrix differential of the matrix product is d(AXB) = A(dX)B.

140

Matrix Differential

6. The matrix differentials of the functions UV and UVW, where U = F(X), V = G(X), W = H(X), are respectively given by d(UV) = (dU)V + U(dV) d(UVW) = (dU)VW + U(dV)W + UV(dW).

(3.2.1) (3.2.2)

7. The differential of the matrix trace d(tr X) is equal to the trace of the matrix differential dX, namely d(tr X) = tr(dX).

(3.2.3)

In particular, the differential of the trace of the matrix function F(X) is given by d(tr F(X)) = tr(dF(X)). 8. The differential of the determinant of X is d|X| = |X|tr(X−1 dX).

(3.2.4)

In particular, the differential of the determinant of the matrix function F(X) is given by d|F(X)| = |F(X)|tr(F−1 (X)dF(X)). 9. The matrix differential of the Kronecker product is given by d(U ⊗ V) = (dU) ⊗ V + U ⊗ dV.

(3.2.5)

10. The matrix differential of the Hadamard product is given by d(U ∗ V) = (dU) ∗ V + U ∗ dV.

(3.2.6)

11. The matrix differential of the inverse matrix is d(X−1 ) = −X−1 (dX)X−1 .

(3.2.7)

12. The differential of the vectorization function vec X is equal to the vectorization of the matrix differential, i.e., d vec X = vec(dX).

(3.2.8)

13. The differential of the matrix logarithm is d log X = X−1 dX.

(3.2.9)

In particular, d log F(X) = F−1 (X) dF(X). 14. The matrix differentials of X† , X† X and XX† , where X† is the Moore–Penrose inverses of X, are d(X† ) = − X† (dX)X† + X† (X† )T (dXT )(I − XX† ) + (I − X† X)(dXT )(X† )T X† ,  T d(X† X) = X† (dX)(I − X† X) + X† (dX)(I − X† X) , *T ) d(XX† ) = (I − XX† )(dX)X† + (I − XX† )(dX)X† .

(3.2.10) (3.2.11) (3.2.12)

3.2 Real Matrix Differential

141

3.2.2 Jacobian Matrix Identification In multivariate calculus, the multivariate function f (x1 , . . . , xm ) is said to be differentiable at the point (x1 , . . . , xm ), if a change in f (x1 , . . . , xm ) can be expressed as Δf (x1 , . . . , xm ) = f (x1 + Δx1 , . . . , xm + Δxm ) − f (x1 , . . . , xm ) = A1 Δx1 + · · · + Am Δxm + O(Δx1 , . . . , Δxm ), where A1 , . . . , Am are independent of Δx1 , . . . , Δxm , respectively, and O(Δx1 , . . . , Δxm ) denotes the second-order and the higher-order terms in Δx1 , . . . , Δxm . In this case, the partial derivative ∂f /∂x1 , . . . , ∂f /∂xm must exist, and ∂f = A1 , ∂x1

...

,

∂f = Am . ∂xm

The linear part of the change Δf (x1 , . . . , xm ), A1 Δx1 + · · · + Am Δxm =

∂f ∂f dx1 + · · · + dx , ∂x1 ∂xm m

is said to be the differential or first-order differential of the multivariate function f (x1 , . . . , xm ) and is denoted by df (x1 , . . . , xm ) =

∂f ∂f dx + · · · + dx . ∂x1 1 ∂xm m

(3.2.13)

The sufficient condition for a multivariate function f (x1 , . . . , xm ) to be differentiable at the point (x1 , . . . , xm ) is that the partial derivatives ∂f /∂x1 , . . . , ∂f /∂xm exist and are continuous. 1. Jacobian Matrix Identification for a Real Function f (x) Consider a scalar function f (x) with variable x = [x1 , . . . , xm ]T ∈ Rm . By regarding the elements x1 , . . . , xm as m variables, and using Equation (3.2.13), we can obtain the differential of the scalar function f (x) directly, as follows: ⎤ ⎡  dx1  ∂f (x) ∂f (x) ∂f (x) ∂f (x) ⎢ . ⎥ df (x) = dx1 + · · · + dxm = ... ⎣ .. ⎦ ∂x1 ∂xm ∂x1 ∂xm dxm or df (x) = where

∂f (x) ∂f (x) dx = (dx)T , ∂ xT ∂x

  ∂f (x) ∂f (x) ∂f (x) = , . . . , and dx = [dx1 , . . . , dxm ]T . ∂ xT ∂x1 ∂xm

(3.2.14)

Equation (3.2.14) is a vector form of the differential rule that implies an important application. If we let A = ∂f (x)/∂ xT then the first-order differential can be denoted as a trace: df (x) = (∂f (x)/∂ xT )dx = tr(A dx). This shows that there

142

Matrix Differential

is an equivalence relationship between the Jacobian matrix of the scalar function f (x) and its matrix differential, namely df (x) = tr(A dx)



Dx f (x) =

∂f (x) = A. ∂ xT

(3.2.15)

In other words, if the differential of the function f (x) is denoted as df (x) = tr(A dx) then the matrix A is just the Jacobian matrix of the function f (x). 2. Jacobian Matrix Identification for a Real Function f (X) Consider a scalar function f (X) with variable X = [x1 , . . . , xn ] ∈ Rm×n . Denoting xj = [x1j , . . . , xmj ]T , j = 1, . . . , n, then from Equation (3.2.2) it is easily known that df (X) =

∂f (X) ∂f (X) dx1 + · · · + dxn ∂ x1 ∂xn

⎤ dx11 ⎢ . ⎥ ⎢ .. ⎥ ⎥ ⎢ ⎥ ⎢  ⎢ dxm1 ⎥  ⎥ ∂f (X) ∂f (X) ∂f (X) ⎢ ∂f (X) ⎢ .. ⎥ = ... ... ... . ⎥ ⎢ ∂x11 ∂xm1 ∂x1n ∂xmn ⎢ ⎥ ⎢ dx1n ⎥ ⎥ ⎢ ⎢ .. ⎥ ⎣ . ⎦ dxmn =



∂f (X) d vecX = Dvec X f (X) dvecX. ∂ (vecX)T

(3.2.16)

By the relationship between the row  T partial derivative vector and the Jacobian matrix, Dvec X f (X) = vec DTX f (X) , Equation (3.2.16) can be written as df (X) = (vec AT )T d(vec X), where



∂f (X) ⎢ ∂x11

A = DX f (X) =

∂f (X) ⎢ . =⎢ ⎢ .. ∂ XT ⎣ ∂f (X) ∂x1n

··· ..

.

···

(3.2.17) ⎤

∂f (X) ∂xm1 ⎥

⎥ ⎥ ⎥ ∂f (X) ⎦ .. .

(3.2.18)

∂xmn

is the Jacobian matrix of the scalar function f (X). Using the relationship between the vectorization operator vec and the trace function tr(BT C) = (vec B)T vec C, and letting B = AT and C = dX, then Equation (3.2.17) can be expressed in the trace form as df (X) = tr(AdX).

(3.2.19)

3.2 Real Matrix Differential

143

This can be regarded as the canonical form of the differential of a scalar function f (X). The above discussion shows that once the matrix differential of a scalar function df (X) is expressed in its canonical form, we can identify the Jacobian matrix and/or the gradient matrix of the scalar function f (X), as shown below. PROPOSITION 3.1 If a scalar function f (X) is differentiable at the point X then the Jacobian matrix A can be directly identified as follows [311]: df (x) = tr(Adx)



Dx f (x) = A,

(3.2.20)

df (X) = tr(AdX)



DX f (X) = A.

(3.2.21)

Proposition 3.1 motivates the following effective approach to directly identifying the Jacobian matrix DX f (X) of the scalar function f (X): Step 1 Find the differential df (X) of the real function f (X), and denote it in the canonical form as df (X) = tr(A dX). Step 2

The Jacobian matrix is directly given by A.

It has been shown [311] that the Jacobian matrix A is uniquely determined: if there are A1 and A2 such that df (X) = Ai dX, i = 1, 2, then A1 = A2 . Since the gradient matrix is the transpose of the Jacobian matrix for a given real function f (X), Proposition 3.1 implies in addition that df (X) = tr(AdX) ⇔ ∇X f (X) = AT .

(3.2.22)

Because the Jacobian matrix A is uniquely determined, the gradient matrix is unique determined as well. 3. Jacobian Matrices of Trace Functions EXAMPLE 3.11 Consider the quadratic form f (x) = xT Ax, where A is a square constant matrix. Writing f (x) = tr(f (x)) and then using the expression for the differential of a matrix product, it is easily obtained that   df (x) = d tr(xT Ax) = tr (dx)T Ax + xT Adx     = tr (dxT Ax)T + xT Adx = tr xT AT dx + xT Adx   = tr xT (A + AT )dx . By Proposition 3.1, we can obtain the gradient vector directly: ∇x (xT Ax) =

T ∂ xT Ax  T = x (A + AT ) = (AT + A)x. ∂x

(3.2.23)

Clearly, if A is a symmetric matrix, then ∇x (xT Ax) = ∂xT Ax/∂x = 2Ax.

144

Matrix Differential

For tr(XT X), since tr(AT B) = tr(BT A), we have     d(tr(XT X)) = tr d(XT X) = tr (dX)T X + XT dX     = tr (dX)T X + tr(XT dX) = tr 2XT dX .

EXAMPLE 3.12

On the one hand, from Proposition 3.1 it is known that the gradient matrix is given by ∂ tr(XT X) = (2XT )T = 2X. ∂X

(3.2.24)

On the other hand, the differential of the trace of three-matrix product tr(XT AX) is given by     d tr(XT AX) = tr d(XT AX) = tr (dX)T AX + XT AdX   = tr (dX)T AX + tr(XT AdX)   = tr (AX)T dX + tr(XT AdX)   = tr XT (AT + A)dX , which yields the gradient matrix T ∂ tr(XT AX)  T T = X (A + A) = (A + AT )X. ∂X

(3.2.25)

Given a trace including an inverse matrix, tr(AX−1 ), since     d tr(AX−1 ) = tr d(AX−1 ) = tr AdX−1     = −tr AX−1 (dX)X−1 = −tr X−1 AX−1 dX ,

EXAMPLE 3.13

the gradient matrix is given by ∂ tr(AX−1 )/∂ X = −(X−1 AX−1 )T . The following are the main points in applying Proposition 3.1: (1) Any scalar function f (X) can always be written in the form of a trace function, because f (X) = tr(f (X)). (2) No matter where dX appears initially in the trace function, we can place it in the rightmost position via the trace property tr(C(dX)B) = tr(BC dX), giving the canonical form df (X) = tr(A dX). Table 3.3 summarizes the differential matrices and Jacobian matrices of several typical trace functions [311]; note that A−2 = A−1 A−1 . 4. Jacobian Matrices of Determinant Functions From the matrix differential d|X| = |X| tr(X−1 dX) and Proposition 3.1, it can immediately be seen that the gradient matrix of the determinant |X| is ∂|X| = |X|(X−1 )T = |X| X−T . ∂X

(3.2.26)

3.2 Real Matrix Differential

145

Table 3.3 Differential matrices and Jacobian matrices of trace functions ∂f (X)/∂XT

Trace function f (X) df (X) tr(X) tr(X

tr(IdX)

−1

−tr(X

)

tr(AX)

−2

I dX)

−X−2

tr(AdX)

A

tr(X )

2tr(XdX)

2X

tr(XT X)

2tr(XT dX)   tr XT (A + AT )dX   tr (A + AT )XT dX

2XT

tr ((AX + XA)dX)   −tr X−1 AX−1 dX   −tr X−1 BAX−1 dX   −tr (X + A)−2 dX

AX + XA

tr((AXB + BXA)dX)   tr (AXT B + AT XT BT )dX   tr XT (BA + AT BT )dX   tr (BA + AT BT )XT dX

AXB + BXA

2

tr(XT AX) tr(XAXT ) tr(XAX) tr(AX

−1

)

tr(AX−1 B)   tr (X + A)−1 tr(XAXB) T

tr(XAX B) tr(AXXT B) tr(AXT XB)

XT (A + AT ) (A + AT )XT −X−1 AX−1 −X−1 BAX−1 −(X + A)−2 AXT B + AT XT BT XT (BA + AT BT ) (BA + AT BT )XT

For the logarithm of the determinant, log |X|, its matrix differential is d log |X| = |X|−1 d|X| = |X|−1 tr(|X|X−1 dX) = tr(X−1 dX),

(3.2.27)

hence the gradient matrix of the determinant logarithm function log |X| is determined by ∂ log |X| (3.2.28) = X−T . ∂X d|X| = |X|tr(X−1 dX) it is known that Consider the determinant of X2 . From  d|X2 | = d|X|2 = 2|X| d|X| = 2|X|2 tr X−1 dX . By applying Proposition 3.1, we have ∂|X|2 (3.2.29) = 2|X|2 (X−1 )T = 2|X|2 X−T . ∂X More generally, the matrix differential of |Xk | is as follows: d|Xk | = |Xk | tr(X−k dXk ) = |Xk |tr(X−k (kXk−1 )dX) = k|Xk | tr(X−1 dX). Hence ∂|Xk | = k|Xk | X−T . ∂X

(3.2.30)

146

Matrix Differential

Letting X ∈ Rm×n and rank(X) = m, i.e., XXT is invertible, then we have ) * d|XXT | = |XXT | tr (XXT )−1 d(XXT ) * ) ** ) ) = |XXT | tr (XXT )−1 (dX)XT + tr (XXT )−1 X(dX)T ) ) * ) ** = |XXT | tr XT (XXT )−1 dX + tr XT (XXT )−1 dX ) * = tr 2|XXT | XT (XXT )−1 dX . By Proposition 3.1, we get the gradient matrix ∂|XXT | = 2|XXT | (XXT )−1 X. ∂X Similarly, set X ∈ Rm×n . If rank(X) = n, i.e., XT X is invertible, then   d|XT X| = tr 2|XT X|(XT X)−1 XT dX , T

T

T

(3.2.31)

(3.2.32)

−1

and hence ∂|X X|/∂ X = 2|X X| X(X X) . Table 3.4 summarizes the real matrix differentials and the Jacobian matrices of several typical determinant functions. Table 3.4 Differentials and Jacobian matrices of determinant functions f (X)

df (X)

∂f (X)/∂X

|X|

|X| tr(X−1 dX)

|X|X−1

log |X|

tr(X−1 dX)

X−1

|X−1 |

−|X−1 | tr(X−1 dX)   2|X|2 tr X−1 dX

−|X−1 |X−1

|X2 |

k|X|k tr(X−1 dX)   |XXT | 2|XXT | tr XT (XXT )−1 dX   |XT X| 2|XT X| tr (XT X)−1 XT dX   log |XT X| 2tr (XT X)−1 XT dX   |AXB| |AXB| tr B(AXB)−1 AdX  |XAXT | tr AXT (XAXT )−1

|XAXT |  +(XA)T (XAT XT )−1 dX  |XT AX|tr (XT AX)−T (AX)T

|XT AX|  +(XT AX)−1 XT A dX |Xk |

2|X|2 X−1 k|X|k X−1 2|XXT |XT (XXT )−1 2|XT X|(XT X)−1 XT 2(XT X)−1 XT |AXB|B(AXB)−1 A |XAXT | AXT (XAXT )−1 +(XA)T (XAT XT )−1 |XT AX| (XT AX)−T (AX)T

+(XT AX)−1 XT A

3.2 Real Matrix Differential

147

3.2.3 Jacobian Matrix of Real Matrix Functions Let fkl = fkl (X) be the entry of the kth row and lth column of the real matrix function F(X); then dfkl (X) = [dF(X)]kl represents the differential of the scalar function fkl (X). From Equation (3.2.16) we have ⎤ ⎡ dx11 ⎢ . ⎥ ⎢ .. ⎥ ⎥ ⎢ ⎥ ⎢  ⎢ dxm1 ⎥  ⎢ ∂fkl (X) ∂fkl (X) ∂fkl (X) ∂fkl (X) ⎢ . ⎥ .. ⎥ dfkl (X) = ... ... ... ⎥. ∂x11 ∂xm1 ∂x1n ∂xmn ⎢ ⎥ ⎢ ⎢ dx1n ⎥ ⎥ ⎢ ⎢ .. ⎥ ⎣ . ⎦ dxmn The above result can be rewritten in vectorization form as follows: dvecF(X) = AdvecX,

(3.2.33)

where dvecF(X) = [df11 (X), . . . , dfp1 (X), . . . , df1q (X), . . . , dfpq (X)]T , d vecX = [dx11 , . . . , dxm1 , . . . , dx1n , . . . , dxmn ] and

⎡ ∂f (X) 11 ⎢ ∂x11 ⎢ .. ⎢ . ⎢ ⎢ ∂fp1 (X) ⎢ ⎢ ∂x 11 ⎢ ⎢ .. A=⎢ . ⎢ ⎢ ∂f1q (X) ⎢ ⎢ ∂x11 ⎢ .. ⎢ ⎢ . ⎣ ∂fpq (X) ∂x11

=

··· .. . ··· .. . ··· .. . ···

∂f11 (X) ∂xm1

···

.. .

.. .

∂fp1 (X) ∂xm1

···

.. .

.. .

∂f1q (X) ∂xm1

···

.. .

.. .

∂fpq (X) ∂xm1

···

T

(3.2.34) (3.2.35)

∂f11 (X) ∂x1n

···

.. .

.. .

∂fp1 (X) ∂x1n

···

.. .

.. .

∂f1q (X) ∂x1n

···

.. .

.. .

∂fpq (X) ∂x1n

···

∂f11 (X) ⎤ ∂xmn ⎥

⎥ ⎥ ⎥ ∂fp1 (X) ⎥ ⎥ ∂xmn ⎥ ⎥ ⎥ .. ⎥ . ⎥ ∂f1q (X) ⎥ ⎥ ∂xmn ⎥ ⎥ .. ⎥ ⎥ . ⎦ .. .

∂fpq (X) ∂xmn

∂ vecF(X) . ∂(vec X)T

(3.2.36)

In other words, the matrix A is the Jacobian matrix DX F(X) of the matrix function F(X). Let F(X) ∈ Rp×q be a matrix function including X and XT as variables, where X ∈ Rm×n . Then the first-order matrix differential is given by d vec F(X) = A d(vecX) + Bd(vec XT ),

A, B ∈ Rpq×mn .

148

Matrix Differential

Since d(vecXT ) = Kmn dvecX, the above equation can be rewritten as d vec F(X) = (A + BKmn )d vec X.

(3.2.37)

The above discussion can be summarized in the following proposition. PROPOSITION 3.2 Given a matrix function F(X) : Rm×n → Rp×q , then its pq × mn Jacobian matrix can be determined from d vecF(X) = A dvecX + Bd(vecX)T ⇔

DX F(X) =

∂ vecF(X) = A + BKmn ∂ (vecX)T

(3.2.38)

and its mn × pq gradient matrix can be identified using ∇X F(X) = (DX F(X))T = AT + Knm BT .

(3.2.39)

Importantly, because dF(X) = A(dX)B



d vecF(X) = (BT ⊗ A)dvecX,

dF(X) = C(dXT )D



dvecF(X) = (DT ⊗ C)Kmn dvecX,

identification given in Proposition 3.2 can be simplified to the differential of F(X). THEOREM 3.1 Given a matrix function F(X) : Rm×n → Rp×q , then its pq × mn Jacobian matrix can be identified as follows: [311] dF(X) = A(dX)B + C(dXT )D, ⇔

DX F(X) =

∂ vec F(X) = (BT ⊗ A) + (DT ⊗ C)Kmn , ∂(vec X)T

(3.2.40)

the mn × pq gradient matrix can be determined from ∂(vec F(X))T (3.2.41) = (B ⊗ AT ) + Knm (D ⊗ CT ). ∂ vec X Table 3.5 summarizes the matrix differentials and Jacobian matrices of some real functions. ∇X F(X) =

EXAMPLE 3.14 Given a matrix function AXT B whose matrix differential is d(AXT B) = A(dXT )B, the Jacobian matrix of AXT B is DX (AXT B) = (BT ⊗ A)Kmn ,

(3.2.42)

whose transpose yields the gradient matrix of AXT B. EXAMPLE 3.15 For the function XT BX, its matrix differential is d(XT BX) = XT BdX + d(XT )BX, which yields the Jacobian matrix of XT BX as follows:   (3.2.43) DX (XT BX) = I ⊗ (XT B) + (BX)T ⊗ I Kmn ; the transpose gives the gradient matrix of XT BX.

3.2 Real Matrix Differential

149

Table 3.5 Matrix differentials and Jacobian matrices of real functions Functions

Matrix differential

Jacobian matrix

f (x) : R → R

df (x) = Adx

A∈R

f (x) : R

df (x) = Adx

A ∈ R1×m

f (X) : Rm×n → R

df (X) = tr(AdX)

A ∈ Rn×m

f (x) : Rm → Rp

df (x) = Adx

A ∈ Rp×m

f (X) : Rm×n → Rp

df (X) = Ad(vecX)

A ∈ Rp×mn

F(x) : Rm → Rp×q

d vecF(x) = Adx

A ∈ Rpq×m

F(X) : Rm×n → Rp×q

dF(X) = A(dX)B

(BT ⊗ A) ∈ Rpq×mn

F(X) : Rm×n → Rp×q

dF(X) = C(dXT )D

(DT ⊗ C)Kmn ∈ Rpq×mn

m

→R

Table 3.6 Jacobian matrices of some matrix functions F(X)

dF(X)

Jacobian matrix

X X

X dX + (dX )X

(In ⊗ XT ) + (XT ⊗ In )Kmn

XXT

X(dXT ) + (dX)XT

(Im ⊗ X)Kmn + (X ⊗ Im )

AXT B

A(dXT )B

(BT ⊗ A)Kmn

XT BX

XT B dX + (dXT )BX

I ⊗ (XT B) + ((BX)T ⊗ I)Kmn

AXT BXC

A(dXT )BXC + AXT B(dX)C

((BXC)T ⊗ A)Kmn +CT ⊗ (AXT B)

AXBXT C

A(dX)BXT C + AXB(dXT )C

(BXT C)T ⊗ A +(CT ⊗ (AXB))Kmn

X−1

−X−1 (dX)X−1

−(X−T ⊗ X−1 )

k

k

T

Xk log X exp(X)

T

T

Xj−1 (dX)Xk−j

(XT )k−j ⊗ Xj−1

j=1

j=1

X−1 dX

I ⊗ X−1





k=0

k 1 Xj (dX)Xk−j (k + 1)! j=0

k=0

k 1 (XT )k−j ⊗ Xj (k + 1)! j=0

Table 3.6 lists some matrix functions and their Jacobian matrices. It should be noted that the matrix differentials of some matrix functions may not be representable in the canonical form required by Theorem 3.1, but nevertheless

150

Matrix Differential

can be expressed in the canonical form in Proposition 3.2. In such cases, we must use Proposition 3.2 to identify the Jacobian matrix. EXAMPLE 3.16 Let F(X, Y) = X ⊗ Y be the Kronecker product of two matrices X ∈ Rp×m and Y ∈ Rn×q . Consider the matrix differential dF(X, Y) = (dX) ⊗ Y + X ⊗ (dY). By the vectorization formula vec(X ⊗ Y) = (Im ⊗ Kqp ⊗ In )(vec X ⊗ vec Y) we have vec(dX ⊗ Y) = (Im ⊗ Kqp ⊗ In )(d vecX ⊗ vec Y) = (Im ⊗ Kqp ⊗ In )(Ipm ⊗ vec Y)d vecX,

(3.2.44)

vec(X ⊗ dY) = (Im ⊗ Kqp ⊗ In )(vec X ⊗ d vecY) = (Im ⊗ Kqp ⊗ In )(vec X ⊗ Inq )d vecY.

(3.2.45)

Hence, the Jacobian matrices are respectively as follows: DX (X ⊗ Y) = (Im ⊗ Kqp ⊗ In )(Ipm ⊗ vec Y),

(3.2.46)

DY (X ⊗ Y) = (Im ⊗ Kqp ⊗ In )(vec X ⊗ Inq ).

(3.2.47)

The analysis and examples in this section show that the first-order real matrix differential is indeed an effective mathematical tool for identifying the Jacobian matrix and the gradient matrix of a real function. The operation of this tool is simple and easy to master.

3.3 Real Hessian Matrix and Identification In real-world problems, we have to employ not only the first-order derivative but also the second-order derivative of a given real function in order to obtain more information about it. In the above section we presented the Jacobian matrix and the gradient matrix, which are two useful representations of the first-order derivative of a real-valued function. Here we discuss the second-order derivative of a real function. The main problem with which we are concerned is how to obtain the Hessian matrix of a real function.

3.3.1 Real Hessian Matrix Consider a real vector x ∈ Rm×1 , the second-order derivative of a real function f (x) is known as the Hessian matrix, denoted H[f (x)], and is defined as   ∂f (x) ∂ ∂ 2 f (x) ∈ Rm×m , = (3.3.1) H[f (x)] = ∂ x∂ xT ∂ x ∂ xT which is simply represented as H[f (x)] = ∇2x f (x) = ∇x (Dx f (x)),

(3.3.2)

3.3 Real Hessian Matrix and Identification

151

where Dx is the cogradient operator. Hence, the (i, j)th entry of the Hessian matrix is defined as    2  ∂f (x) ∂ f (x) ∂ , (3.3.3) = [Hf (x)]ij = ∂ x ∂ xT ij ∂xi ∂xj thus for the matrix itself we have ⎡

∂2f ⎢ ∂x1 ∂x1

H[f (x)] =

···

⎢ ∂ 2 f (x) ⎢ .. = ⎢ . ⎢ ∂ x ∂ xT ⎣ ∂2f

..

.. .

.

···

∂xm ∂x1



∂2f ∂x1 ∂xm ⎥

∂2f ∂xm ∂xm

⎥ ⎥ ⎥ ∈ Rm×m . ⎥ ⎦

(3.3.4)

This shows that the Hessian matrix of a real scalar function f (x) is an m × m matrix that consists of m2 second-order partial derivatives of f (x) with respect to the entries xi of the vector variable x. By the definition, it can be seen that the Hessian matrix of a real scalar function f (x) is a real symmetric matrix, i.e, T

(H[f (x)]) = H[f (x)].

(3.3.5)

The reason is that, for a second-differentiable continuous function f (x), its second∂2f ∂2f order derivative is independent of the order of differentiation, i.e., = . ∂xi ∂xj

∂xj ∂xi

By mimicking the definition of the Hessian matrix of a real scalar function f (x), we can define the Hessian matrix of a real scalar function f (X): H[f (X)] =

∂ 2 f (X) = ∇X (DX f (X)) ∈ Rmn×mn , ∂ vecX∂(vecX)T

which can be written in element form as ⎡ ∂2f ∂2f ∂2f ⎢ ∂x11 ∂x11 · · · ∂x11 ∂xm1 · · · ∂x11 ∂x1n ⎢ .. .. .. .. .. ⎢ ⎢ . . . . . ⎢ ⎢ ∂2f ∂2f ∂2f ⎢ ··· ··· ⎢ ∂xm1 ∂x11 ∂xm1 ∂xm1 ∂xm1 ∂x1n ⎢ .. .. .. .. .. ⎢ . . . . . ⎢ ⎢ ∂2f ∂2f ⎢ ∂2f ··· ··· ⎢ ∂x1n ∂xm1 ∂x1n ∂x1n ⎢ ∂x1n ∂x11 ⎢ . . . .. .. ⎢ .. .. .. . . ⎢ ⎣ ∂2f ∂2f ∂2f ··· ··· ∂xmn ∂x11

From

∂2f ∂xij ∂xkl

=

∂xmn ∂xm1

∂2f ∂xkl ∂xij

∂xmn ∂x1n

···



∂2f ∂x11 ∂xmn ⎥

⎥ ⎥ ⎥ ⎥ ⎥ ∂2f ⎥ ··· ∂xm1 ∂xmn ⎥ ⎥ .. .. ⎥. . . ⎥ ⎥ 2 ∂ f ⎥ ··· ⎥ ∂x1n ∂xmn ⎥ ⎥ .. .. ⎥ . . ⎥ 2 ⎦ ∂ f ··· .. .

(3.3.6)

.. .

(3.3.7)

∂xmn ∂xmn

it is easily seen that the Hessian matrix of a real scalar

152

Matrix Differential

function f (X) is a real symmetric matrix: (H[f (X)])T = H[f (X)].

(3.3.8)

3.3.2 Real Hessian Matrix Identification Let us discuss how to identify the Hessian matrices for real scalar functions f (x) and f (X). First, consider the case of a real scalar function f (x). In many cases, it may involve considerable trouble to compute a Hessian matrix from its definition formula. A simpler method is based on the relationship between the second-order differential matrix and the Hessian matrix. Noting that the differential dx is not a function of the vector x, we have d2 x = d(dx) = 0.

(3.3.9)

Keeping this point in mind, from Equation (3.2.14) it is easy to find the secondorder differential d2 f (x) = d(df (x)):   ∂ df (x) ∂ ∂f 2 (x) ∂f (x) d2 f (x) = dxT dx = (dx)T dx = dxT T ∂x ∂x ∂x ∂ x ∂ xT which can simply be written as d2 f (x) = (dx)T H[f (x)]dx,

(3.3.10)

where H[f (x)] =

∂f 2 (x) ∂ x ∂ xT

(3.3.11)

is the Hessian matrix of the function f (x) and the (i, j)th entry of H[f (x)] is given by   ∂f (x) ∂f 2 (x) ∂ = . (3.3.12) hij = ∂xi ∂xj ∂xi ∂xj Noting that the matrix A ∈ R1×m in the first-order differential df (x) = Adx is usually a real row vector and that its differential is still a real row vector, we have dA = (dx)T B ∈ R1×m , where B ∈ Rm×m . Hence, the second-order differential of the function f (x) is a quadratic function, d2 f (x) = d(Adx) = (dx)T Bdx.

(3.3.13)

By comparing Equation (3.3.13) with Equation (3.3.10), it can be seen that the Hessian matrix Hx [f (x)] = B. In order to ensure that the Hessian matrix is real symmetric, we take 1 (3.3.14) H[f (x)] = (BT + B). 2

3.3 Real Hessian Matrix and Identification

153

Now consider the case of a real scalar function f (X). From Equation (3.2.16) it follows that the second-order differential of the scalar function f (X) is given by ∂ df (X) ∂ vec X   ∂f (X) ∂ T = d(vec X) d vec X ∂ vec X ∂(vec X)T ∂f 2 (X) = (d vecX)T d vecX, ∂ vecX ∂(vec X)T

d2 f (X) = d(vec X)T

namely d2 f (X) = d(vecX)T H[f (X)]dvecX.

(3.3.15)

Here H[f (X)] =

∂ 2 f (X) ∂ vecX ∂(vecX)T

(3.3.16)

is the Hessian matrix of f (X). Formula (3.3.15) is called the second-order (matrix) differential rule for f (X). For the real scalar function f (X), the matrix A in the first-order differential df (X) = Ad vecX is usually a real row vector of the variable matrix X and the differential of A is still a real row vector; thus we have dA = d(vec X)T B

∈ R1×mn ,

where B ∈ Rmn×mn . Hence, the second-order differential of f (X) is a quadratic function d2 f (X) = d(vec X)T BdvecX.

(3.3.17)

By comparing Equation (3.3.17) with Equation (3.3.15), it follows that the Hessian matrix of the function f (X) is H[f (X)] =

1 T (B + B), 2

(3.3.18)

because a real Hessian matrix must be symmetric. The results above can be summarized in the following proposition. PROPOSITION 3.3 For a scalar function f (·) with vector x or matrix X as function variable, there is the following second-order identification relation between the second-order differential and the Hessian matrix: d2 f (x) = (dx)T Bdx



d2 f (X) = d(vecX)T Bdvec X



1 T (B + B), 2 1 H[f (X)] = (BT + B). 2 H[f (x)] =

(3.3.19) (3.3.20)

154

Matrix Differential

More generally, we have the second-order identification theorem for matrix function as follows. THEOREM 3.2 [311, p. 115] Let F : S → Rm×p be a matrix function defined on a set S ∈ Rn×q , and twice differentiable at an interior point C of S. Then d2 vec(F(C; X)) = (Imp ⊗ d vecX)T B(C)dvecX

(3.3.21)

for all X ∈ Rn×q , if and only if H[F(C)] = 12 (B(C) + (B(C)) v ), where B, (B )v ∈ Rpmqn×mn are given by ⎡ T⎤ ⎡ ⎤ B11 B11 ⎢ . ⎥ ⎢ . ⎥ ⎢ .. ⎥ ⎢ .. ⎥ ⎢ ⎢ ⎥ ⎥ ⎢BT ⎥ ⎢B ⎥ ⎢ p1 ⎥ ⎢ p1 ⎥ ⎢ . ⎥ ⎢ . ⎥

⎢ ⎥ ⎥ (3.3.22) B=⎢ ⎢ .. ⎥ , (B )v = ⎢ .. ⎥ ⎢ T⎥ ⎢ ⎥ ⎢B1q ⎥ ⎢B1q ⎥ ⎢ ⎢ ⎥ ⎥ ⎢ .. ⎥ ⎢ .. ⎥ ⎣ . ⎦ ⎣ . ⎦ Bpq BTpq with Bij ∈ Rmn×mn , ∀ i = 1, . . . , p, j = 1, . . . , q. Theorem 3.2 gives the following second-order identification formula for a matrix function: d2 vecF(X) = (Imp ⊗ d vecX)T Bd vecX 1 ⇔ H[F(X)] = [B + (B )v ]. 2

(3.3.23)

In particular, if F ∈ Rm×p takes the form of a scalar function f or a vector function f ∈ Rm×1 , and the variable matrix X ∈ Rn×q a scalar x or a vector x ∈ Rn×1 , then the above second-order identification formula yields the following identification formulas: f (x): d2 [f (x)] = β(dx)2 ⇔ H[f (x)] = β, f (x): d2 [f (x)] = (dx)T Bdx ⇔ H[f (x)] = 12 (B + BT ), f (X): d2 [f (X)] = d(vec X)T BdvecX ⇔ H[f (X)] = 12 (B + BT ); f (x): d2 [f (x)] = b(dx)2 ⇔ H[f (x)] = b, f (x): d2 [f (x)] = (Im ⊗ dx)T Bdx ⇔ H[f (x)] = 12 [B + (B )v ], f (X): d2 [f (X)] = (Im ⊗ d(vec X))T BdvecX ⇔ H[f (X)] = 12 [B + (B )v ]; F(x): d2 [F(x)] = B(dx)2 ⇔ H[F(x)] = vec B, F(x): d2 [vecF] = (Imp ⊗ dx)T Bdx ⇔ H[F(x)] = 12 [B + (B )v ]. The Hessian matrix identification based on Theorem 3.2 requires a row vectorization operation on the second-order matrix differential. An interesting question is

3.3 Real Hessian Matrix and Identification

155

whether we can directly identify the Hessian matrix from the second-order matrix differential without such a row vectorization operation. As mentioned earlier, the first-order differential of a scalar function f (X) can be written in the canonical form of trace function as df (X) = tr(AdX), where A = A(X) ∈ Rn×m is generally a matrix function of the variable matrix X ∈ Rm×n . Without loss of generality, it can be assumed that the differential matrix of the matrix function A = A(X) takes one of two forms: dA = B(dX)C,

or

dA = U(dX)T V,

B, C ∈ Rn×m

U ∈ Rn×n , V ∈ Rm×m .

(3.3.24) (3.3.25)

Notice that the above two differential matrices include (dX)C, BdX and (dX)T V, U(dX)T as two special examples. Substituting (3.3.24) and (3.3.25) into df (X) = tr(AdX), respectively, it is known that the second-order differential d2 f (X) = tr(dAdX) of the real scalar function f (X) takes one of two forms d2 f (X) = tr(B(dX)CdX)   d2 f (X) = tr V(dX)U(dX)T .

or

(3.3.26) (3.3.27)

By the trace property tr(ABCD) = (vec DT )T (CT ⊗ A)vec B, we have tr(B(dX)CdX) = (vec(dX)T )T (CT ⊗ B)vec(dX) = (d(Kmn vec X))T (CT ⊗ B)dvecX 

tr VdXU(dX)

 T

= (d(vec X))T Knm (CT ⊗ B)dvecX,

(3.3.28)

= (vec(dX)) (U ⊗ V)vec(dX) T

T

= (d(vec X))T (UT ⊗ V)dvecX,

(3.3.29)

KTmn = Knm. in which we have used the results Kmn vec Am×n = vec(ATm×n ) and  The expressions d2 f (X) = tr(B(dX)CdX) and d2 f (X) = tr V(dX)U(dX)T can be viewed as two canonical forms for the second-order matrix differentials of the real function f (X).   Each of the second-order differentials tr V(dX)U(dX)T and tr(B(dX)CdX) can be equivalently expressed as the the canonical quadratic form required by Proposition 3.3, so this can be equivalently stated as the following theorem for Hessian matrix identification. THEOREM 3.3 [311, p. 192] Let the real function f (X) with the m×n real matrix X as variable be second-order differentiable. Then there is the following relation between the second-order differential matrix d2 f (X) and the Hessian matrix H[f (X)]:   d2 f (X) = tr V(dX)U(dX)T 1 ⇔ H[f (X)] = (UT ⊗ V + U ⊗ VT ), (3.3.30) 2

156

Matrix Differential

where U ∈ Rn×n , V ∈ Rm×m , or



d2 f (X) = tr(B(dX)CdX) 1 H[f (X)] = Knm (CT ⊗ B + BT ⊗ C), 2

(3.3.31)

where Knm is the commutation matrix. Theorem 3.3 shows that the basic problem in the Hessian matrix identification is how to express the second-order matrix differential of a given real scalar function as one of the two canonical forms required by Theorem 3.3. The following three examples show how to find the Hessian matrix of a given real function using Theorem 3.3. EXAMPLE 3.17 Consider the real function f (X) = tr(X−1 ), where X is an n × n matrix. First, computing the first-order differential   df (X) = −tr X−1 (dX)X−1 and then using d(trU) = tr(dU), we get the second-order differential     d2 f (X) = −tr (dX−1 )(dX)X−1 − tr X−1 (dX)(dX−1 )   = 2 tr X−1 (dX)X−1 (dX)X−1   = 2 tr X−2 (dX)X−1 dX . Now, by Theorem 3.3, we get the Hessian matrix H[f (X)] = EXAMPLE 3.18 tial is

  ∂ 2 tr(X−1 ) = Knn X−T ⊗ X−2 + (X−2 )T ⊗ X−1 . ∂ vecX∂(vecX)T For the real function f (X) = tr(XT AX), its first-order differen  df (X) = tr XT (A + AT )dX .

Thus, the second-order differential is   d2 f (X) = tr (A + AT )(dX)(dX)T . By Theorem 3.3, the Hessian matrix is given by H[f (X)] =

∂ 2 tr(XT AX) = I ⊗ (A + AT ). ∂ vecX∂(vecX)T

EXAMPLE 3.19 The first-order differential of the function log |Xn×n | is d log |X| = tr(X−1 dx), and the second-order differential is −tr(X−1 (dx)X−1 dx). From Theorem 3.3, the Hessian matrix is obtained as follows: H[f (X)] =

∂ 2 log |X| = −Knn (X−T ⊗ X−1 ). ∂ vecX∂ (vecX)T

3.4 Complex Gradient Matrices

157

3.4 Complex Gradient Matrices In array processing and wireless communications, a narrowband signal is usually expressed as a complex equivalent baseband signal, and thus the transmit and receive signals with system parameters are expressed as complex vectors. In these applications, the objective function of an optimization problem is a real-valued function of a complex vector or matrix. Hence, the gradient of the objective function is a complex vector or matrix. Obviously, this complex gradient has the following two forms: (1) complex gradient the gradient of the objective function with respect to the complex vector or matrix variable itself; (2) conjugate gradient the gradient of the objective function with respect to the complex conjugate vector or matrix variable.

3.4.1 Holomorphic Function and Complex Partial Derivative Before discussing the complex gradient and conjugate gradient, it is necessary to recall the relevant facts about complex functions. For convenience, we first list the standard symbols for complex variables and complex functions. • z = [z1 , . . . , zm ]T ∈ Cm is a complex variable vector whose complex conjugate is z∗ . • Z = [z1 , . . . , zn ] ∈ Cm×n is a complex variable matrix with complex conjugate Z∗ . • f (z, z∗ ) ∈ C is a complex scalar function of m × 1 complex variable vectors z and z∗ , denoted f : Cm × Cm → C. • f (Z, Z∗ ) ∈ C is a complex scalar function of m × n complex variable matrices Z and Z∗ , denoted f : Cm×n × Cm×n → C. • f (z, z∗ ) ∈ Cp is a p × 1 complex vector function of m × 1 complex variable vectors z and z∗ , denoted f : Cm × Cm → Cp . • f (Z, Z∗ ) ∈ Cp is a p × 1 complex vector function of m × n complex variable matrices Z and Z∗ , denoted f : Cm×n × Cm×n → Cp . • F(z, z∗ ) ∈ Cp×q is a p × q complex matrix function of m × 1 complex variable vectors z and z∗ , denoted F : Cm × Cm → Cp×q . • F(Z, Z∗ ) ∈ Cp×q is a p × q complex matrix function of m × n complex variable matrices Z and Z∗ , denoted F : Cm×n × Cm×n → Cp×q . Table 3.7 summarizes the classification of complex-valued functions.

158

Matrix Differential

Table 3.7 Classification of complex-valued functions Function

Variables z, z ∗ ∈ C

Variables z, z∗ ∈ Cm

Variables Z, Z∗ ∈ Cm×n

f ∈C

f (z, z ∗ ) f :C×C→C

f (z, z∗ ) Cm × Cm → C

f (Z, Z∗ ) Cm×n × Cm×n → C

f ∈ Cp

f (z, z ∗ ) f : C × C → Cp

f (z, z∗ ) Cm × Cm → Cp

f (Z, Z∗ ) Cm×n × Cm×n → Cp

F ∈ Cp×q

F(z, z ∗ ) F : C × C → Cp×q

F(z, z∗ ) Cm × Cm → Cp×q

F(Z, Z∗ ) Cm×n × Cm×n → Cp×q

DEFINITION 3.10 [264] Let D ⊆ C be the definition domain of the function f : D → C. The function f (z) with complex variable z is said to be a complex analytic function in the domain D if f (z) is complex differentiable, namely lim

Δz→0

f (z + Δz) − f (z) exists for all z ∈ D. Δz

The terminology “complex analytic” is commonly replaced by the completely synonymous terminology “holomorphic”. Hence, a complex analytic function is usually called a holomorphic function. It is noted that a complex function is (real) analytic in the real-variable x-domain and y-domain, but it is not necessarily holomorphic in the complex variable domain z = x + jy, i.e., it may be complex nonanalytic. It is assumed that the complex function f (z) can be expressed in terms of its real part u(x, y) and imaginary part v(x, y) as f (z) = u(x, y) + jv(x, y), where z = x + jy and both u(x, y) and v(x, y) are real functions. For a holomorphic scalar function, the following four statements are equivalent [155]. (1) The complex function f (z) is a holomorphic (i.e., complex analytic) function. (2) The derivative f (z) of the complex function exists and is continuous. (3) The complex function f (z) satisfies the Cauchy–Riemann condition ∂v ∂u = ∂x ∂y

and

∂v ∂u =− . ∂x ∂y

(3.4.1)

(4) All derivatives of the complex function f (z) exist, and f (z) has a convergent power series. The Cauchy–Riemann condition is also known as the Cauchy–Riemann equations, and its direct result is that the function f (z) = u(x, y) + jv(x, y) is a holomorphic function only when both the real functions u(x, y) and v(x, y) satisfy the Laplace

3.4 Complex Gradient Matrices

159

equation at the same time: ∂ 2 u(x, y) ∂ 2 u(x, y) + =0 ∂x2 ∂y 2

and

∂ 2 v(x, y) ∂ 2 v(x, y) + = 0. ∂x2 ∂y 2

(3.4.2)

A real function g(x, y) is called a harmonic function, if it satisfies the Laplace equation ∂ 2 g(x, y) ∂ 2 g(x, y) + = 0. (3.4.3) ∂x2 ∂y 2 A complex function f (z) = u(x, y) + jv(x, y) is not a holomorphic function if either of the real functions u(x, y) and v(x, y) does not meet the Cauchy–Riemann condition or the Laplace equation. Although the power function z n , the exponential function ez , the logarithmic function ln z, the sine function sin z and the cosine function cos z are holomorphic functions, i.e., analytic functions in the complex plane, many commonly used functions are not holomorphic, as shown in the following examples. • For the complex function f (z) = z ∗ = x − jy = u(x, y) + jv(x, y), its real part u(x, y) = x and imaginary part v(x, y) = −y obviously do not meet the Cauchy– Riemann condition

∂u ∂v . = ∂x ∂y

• Any nonconstant real function f (z) ∈ R does not satisfy the Cauchy–Riemann conditions

∂u ∂u ∂v ∂v and = = − , because the imaginary part v(x, y) of f (z) = ∂x ∂y ∂y ∂x 

u(x, y) + jv(x, y) is zero. In particular, the real function f (z) = |z| = x2 + y 2 is nondifferentiable, while for f (z) = |z|2 = x2 + y 2 = u(x, y) + jv(x, y), its real part u(x, y) = x2 + y 2 is not a harmonic function because the Laplace condition is not satisfied, namely,

∂ 2 u(x, y) ∂ 2 u(x, y) + = 0. 2 ∂x ∂y 2

• The complex functions f (z) = Re z = x and f (z) = Im z = y do not meet the Cauchy–Riemann condition. Since many common complex functions f (z) are not holomorphic, a natural question to ask is whether there is a general representation form that can ensure that any complex function is holomorphic. To answer this question, it is necessary to recall the definition on the derivative of a complex number z and its conjugate z ∗ in complex function theory. The formal partial derivatives of complex numbers are defined as   ∂ 1 ∂ ∂ , (3.4.4) = −j ∂z 2 ∂x ∂y   ∂ 1 ∂ ∂ . (3.4.5) = +j ∗ ∂z 2 ∂x ∂y The formal partial derivatives above were presented by Wirtinger [511] in 1927, so they are sometimes called Wirtinger partial derivatives.

160

Matrix Differential

Regarding the partial derivatives of the complex variable z = x + jy, there is a basic assumption on the independence of the real and imaginary parts: ∂x =0 ∂y

and

∂y = 0. ∂x

(3.4.6)

By the definition of the partial derivative and by the above independence assumption, it is easy to find that     1 ∂y ∂z ∂x ∂y 1 ∂x ∂x ∂y + j = + j = + j + j ∂z ∗ ∂z ∗ ∂z ∗ 2 ∂x ∂y 2 ∂x ∂y 1 1 = (1 + 0) + j (0 + j), 2 2     ∂x ∂y 1 ∂x ∂x 1 ∂y ∂y ∂z ∗ = −j = −j −j −j ∂z ∂z ∂z 2 ∂x ∂y 2 ∂x ∂y 1 1 = (1 − 0) − j (0 − j). 2 2 This implies that ∂z ∗ ∂z =0 and = 0. (3.4.7) ∗ ∂z ∂z Equation (3.4.7) reveals a basic result in the theory of complex variables: the complex variable z and the complex conjugate variable z ∗ are independent variables. In the standard framework of complex functions, a complex function f (z) (where def

z = x + jy) is written in the real polar coordinates r = (x, y) as f (r) = f (x, y). However, in the framework of complex derivatives, under the independence assumption of the real and imaginary parts, the complex function f (z) is written as def

f (c) = f (z, z ∗ ) in the conjugate coordinates c = (z, z ∗ ) instead of the polar coordinates r = (x, y). Hence, when finding the complex partial derivative ∇z f (z, z ∗ ) and the complex conjugate partial derivative ∇z∗ f (z, z ∗ ), the complex variable z and the complex conjugate variable z ∗ are regarded as two independent variables:  ∂f (z, z ∗ )  ∗ ∇z f (z, z ) = ,  ∗ ∂z z =const  (3.4.8) ∂f (z, z ∗ )  ∗ ∇z ∗ f (z, z ) = . ∂z ∗ z=const This implies that when any nonholomorphic function f (z) is written as f (z, z ∗ ), it becomes holomorphic, because, for a fixed z ∗ value, the complex function f (z, z ∗ ) is analytic on the whole complex plane z = x + jy; and, for a fixed z value, the complex function f (z, z ∗ ) is analytic on the whole complex plane z ∗ = x − jy, see, e.g., [155], [263]. EXAMPLE 3.20 Given a real function f (z, z ∗ ) = |z|2 = zz ∗ , its first-order partial derivatives ∂|z|2 /∂z = z ∗ and ∂|z|2 /∂z ∗ = z exist and are continuous. This is to say, although f (z) = |z|2 itself is not a holomorphic function with respect to z, the

3.4 Complex Gradient Matrices

161

function f (z, z ∗ ) = |z|2 = zz ∗ is analytic on the whole complex plane z = x + jy (when z ∗ is fixed as a constant) and is analytic on the whole complex plane z ∗ = x − jy (when z is fixed as a constant). Table 3.8 gives for comparison the nonholomorphic and holomorphic representation forms of complex functions. Table 3.8 Nonholomorphic and holomorphic functions Functions Coordinates Representation

Nonholomorphic  def r = (x, y) ∈ R × R z = x + jy

Holomorphic  def c = (z, z ∗ ) ∈ C × C z = x + jy, z ∗ = x − jy

f (r) = f (x, y)

f (c) = f (z, z ∗ )

The following are common formulas and rules for the complex partial derivatives [263]: (1) The conjugate partial derivative of the complex conjugate function ∂f ∗ (z, z ∗ ) = ∂z ∗



∂f (z, z ∗ ) ∂z

∗ .

(2) The partial derivative of the conjugate of the complex function ∂f ∗ (z, z ∗ ) = ∂z



∂f (z, z ∗ ) ∂z ∗

∗

∂f ∗ (z, z ∗ ) : ∂z ∗

(3.4.9) ∂f ∗ (z, z ∗ ) : ∂z

.

(3.4.10)

(3) Complex differential rule df (z, z ∗ ) =

∂f (z, z ∗ ) ∂f (z, z ∗ ) ∗ dz . dz + ∂z ∂z ∗

(3.4.11)

(4) Complex chain rule ∂h(g(z, z ∗ )) ∂h(g(z, z ∗ )) ∂g(z, z ∗ ) ∂h(g(z, z ∗ )) ∂g ∗ (z, z ∗ ) = + , ∂z ∂g(z, z ∗ ) ∂z ∂g ∗ (z, z ∗ ) ∂z ∂h(g(z, z ∗ )) ∂g(z, z ∗ ) ∂h(g(z, z ∗ )) ∂g ∗ (z, z ∗ ) ∂h(g(z, z ∗ )) = + . ∂z ∗ ∂g(z, z ∗ ) ∂z ∗ ∂g ∗ (z, z ∗ ) ∂z ∗

(3.4.12) (3.4.13)

3.4.2 Complex Matrix Differential The concepts of the complex function f (z) and the holomorphic function f (z, z ∗ ) of a complex scalar z can be easily extended to the complex matrix function F(Z) and the holomorphic complex matrix function F(Z, Z∗ ).

162

Matrix Differential

On the holomorphic complex matrix functions, the following statements are equivalent [68]. (1) The matrix function F(Z) is a holomorphic function of the complex matrix variable Z. ∂ vecF(Z) (2) The complex matrix differential d vecF(Z) = d vecZ. T

∂(vecZ) ∂ vecF(Z) = O (the zero matrix) holds. (3) For all Z, ∂(vec Z∗ )T ∂ vecF(Z) ∂ vecF(Z) +j = O holds. (4) For all Z, ∂(vecReZ)T ∂(vecImZ)T

The complex matrix function F(Z, Z∗ ) is obviously a holomorphic function, and its matrix differential is ∂ vecF(Z, Z∗ ) ∂ vecF(Z, Z∗ ) d vecZ + d vecZ∗ . ∂(vecZ)T ∂(vecZ∗ )T

d vecF(Z, Z∗ ) =

(3.4.14)

The partial derivative of the holomorphic function F(Z, Z∗ ) with respect to the real part ReZ of the matrix variable Z is given by ∂ vecF(Z, Z∗ ) ∂ vecF(Z, Z∗ ) ∂ vecF(Z, Z∗ ) = + , T ∂(vecReZ) ∂(vecZ)T ∂(vecZ∗ )T and the partial derivative of F(Z, Z∗ ) with respect to the imaginary part ImZ of the matrix variable Z is as follows:   ∂ vecF(Z, Z∗ ) ∂ vecF(Z, Z∗ ) ∂ vecF(Z, Z∗ ) . = j − ∂(vecImZ)T ∂(vecZ)T ∂(vecZ∗ )T The complex matrix differential d Z = [dZij ]m,n i=1,j=1 has the following properties [68]. 1. 2. 3. 4. 5.

Transpose dZT = d(ZT ) = (dZ)T . Hermitian transpose dZH = d(ZH ) = (dZ)H . Conjugate dZ∗ = d(Z∗ ) = (dZ)∗ . Linearity (additive rule) d(Y + Z) = dY + dZ. Chain rule If F is a function of Y, while Y is a function of Z, then d vecF = where

∂ vecF ∂ vecY ∂ vecF d vecY = d vecZ, ∂(vecY)T ∂(vecY)T ∂(vecZ)T

∂ vecF ∂ vecY and are the normal complex partial derivative and ∂(vecY)T ∂(vec Z)T

the generalized complex partial derivative, respectively. 6. Multiplication rule d(UV) = (dU)V + U(dV) dvec(UV) = (VT ⊗ I)dvecU + (I ⊗ U)dvecV. 7. Kronecker product d(Y ⊗ Z) = dY ⊗ Z + Y ⊗ dZ.

3.4 Complex Gradient Matrices

163

8. Hadamard product d(Y ∗ Z) = dY ∗ Z + Y ∗ dZ. In the following we derive the relationship between the complex matrix differential and the complex partial derivative. First, the complex differential rule for scalar variables, df (z, z ∗ ) =

∂f (z, z ∗ ) ∂f (z, z ∗ ) ∗ dz dz + ∂z ∂z ∗

(3.4.15)

is easily extended to a complex differential rule for the multivariate real scalar ∗ )): function f (·) = f ((z1 , z1∗ ), . . . , (zm , zm ∂f (·) ∂f (·) ∗ ∂f (·) ∗ ∂f (·) dz1 + · · · + dzm + dz1 + · · · + dzm ∗ ∂z1 ∂zm ∂z1∗ ∂zm ∂f (·) ∂f (·) ∗ = dz + dz . (3.4.16) ∂ zT ∂ zH

df (·) =

This complex differential rule is the basis of the complex matrix differential. In particular, if f (·) = f (z, z∗ ), then df (z, z∗ ) =

∂f (z, z∗ ) ∂f (z, z∗ ) ∗ dz + dz T ∂z ∂ zH

or, simply denoted, df (z, z∗ ) = Dz f (z, z∗ ) dz + Dz∗ f (z, z∗ ) dz∗ . ∗ T ] and Here dz = [dz1 , . . . , dzm ]T , dz∗ = [dz1∗ , . . . , dzm    ∂f (z, z∗ )  ∂f (z, z∗ ) ∂f (z, z∗ ) Dz f (z, z∗ ) = , = , . . . , ∂ zT z∗ =const ∂z1 ∂zm    ∂f (z, z∗ )  ∂f (z, z∗ ) ∂f (z, z∗ ) ∗ Dz∗ f (z, z ) = = ,..., ∗ ∂ zH z=const ∂z1∗ ∂zm

(3.4.17)

(3.4.18) (3.4.19)

are respectively the cogradient vector and the conjugate cogradient vector of the real scalar function f (z, z∗ ), while   ∂ def ∂ ∂ Dz = , (3.4.20) = , . . . , ∂ zT ∂z1 ∂zm   ∂ def ∂ ∂ (3.4.21) = , . . . , D z∗ = ∗ ∂ zH ∂z1∗ ∂zm are termed the cogradient operator and the conjugate cogradient operator of complex vector variable z ∈ Cm , respectively. Let z = x + jy = [z1 , . . . , zm ]T ∈ Cm with x = [x1 , . . . , xm ]T ∈ Rm , y = [y1 , . . . , ym ]T ∈ Rm , namely zi = xi + jyi , i = 1, . . . , m; the real part xi and the imaginary part yi are two independent variables.

164

Matrix Differential

Applying the complex partial derivative operators   ∂ 1 ∂ ∂ = −j Dzi = , ∂zi 2 ∂xi ∂yi   ∂ ∂ 1 ∂ Dzi∗ = ∗ = +j ∂zi 2 ∂xi ∂yi

(3.4.22) (3.4.23)

to each element of the row vector zT = [z1 , . . . , zm ], we obtain the following complex cogradient operator   ∂ ∂ 1 ∂ Dz = (3.4.24) = − j ∂ zT 2 ∂ xT ∂ yT and the complex conjugate cogradient operator   ∂ ∂ 1 ∂ Dz∗ = = +j T . ∂ zH 2 ∂ xT ∂y

(3.4.25)

Similarly, the complex gradient operator and the complex conjugate gradient operator are respectively defined as T  ∂ def ∂ ∂ ∇z = = ,..., , ∂z ∂z1 ∂zm T  ∂ def ∂ ∂ ∇z ∗ = = , . . . , . ∗ ∂ z∗ ∂z1∗ ∂zm

(3.4.26) (3.4.27)

Hence, the complex gradient vector and the complex conjugate gradient vector of the real scalar function f (z, z∗ ) are respectively defined as  ∂f (z, z∗ )  ∗ ∇z f (z, z ) = = (Dz f (z, z∗ ))T (3.4.28) ∂ z z∗ = const vector  ∂f (z, z∗ )  ∇z∗ f (z, z∗ ) = = (Dz∗ f (z, z∗ ))T . (3.4.29) ∂ z∗ z = const vector Applying the complex partial derivative operator to each element of the complex vector z = [z1 , . . . , zm ]T , we get the complex gradient operator   ∂ ∂ 1 ∂ (3.4.30) ∇z = = −j ∂z 2 ∂x ∂y and the complex conjugate gradient operator   ∂ ∂ 1 ∂ ∇z∗ = . = + j ∂ z∗ 2 ∂x ∂y

(3.4.31)

By the definitions of the complex gradient operator and the complex conjugate

3.4 Complex Gradient Matrices

165

gradient operator, it is not difficult to obtain     ∂ zT ∂ xT ∂ yT 1 ∂ xT ∂ xT 1 ∂ yT ∂ yT = +j = −j +j −j = Im×m , ∂z ∂z ∂z 2 ∂x ∂y 2 ∂x ∂y     1 ∂ yT ∂ zT ∂ xT ∂ yT 1 ∂ xT ∂ xT ∂ yT + j = Om×m . = + j = + j + j ∂ z∗ ∂z∗ ∂ z∗ 2 ∂x ∂y 2 ∂x ∂y When finding the above two equations, we used

∂ xT ∂ xT = Om×m and = Im×m , ∂x ∂y

∂ yT ∂ yT = Om×m , because the real part x and the imaginary part y of = Im×m , ∂y ∂x

the complex vector variable z are independent. Summarizing the above results and their conjugate, transpose and complex conjugate transpose, we have the following important results: ∂ zT = I, ∂z ∂ zT = O, ∂ z∗

∂ zH = I, ∂ z∗ ∂ zH = O, ∂z

∂z = I, ∂ zT ∂z = O, ∂ zH

∂ z∗ = I, ∂ zH ∂ z∗ = O. ∂ zT

(3.4.32) (3.4.33)

The above results reveal an important fact of the complex matrix differential: under the basic assumption that the real part and imaginary part of a complex vector are independent, the complex vector variable z and its complex conjugate vector variable z∗ can be viewed as two independent variables. This important fact is not surprising because the angle between z and z∗ is π/2, i.e., they are orthogonal to each other. Hence, we can summarize the rules for using the cogradient operator and gradient operator as follows. (1) When using the complex cogradient operator ∂/∂ zT or the complex gradient operator ∂/∂ z, the complex conjugate vector variable z∗ is handled as a constant vector. (2) When using the complex conjugate cogradient operator ∂/∂ zH or the complex conjugate gradient operator ∂/∂ z∗ , the vector variable z is handled as a constant vector. We might regard the above rules as embodying the independent rule of complex partial derivative operators: when applying the complex partial derivative operator, the complex cogradient operator, the complex conjugate cogradient operator, the complex gradient operator or the complex conjugate gradient operator, the complex vector variables z and z∗ can be handled as independent vector variables, namely, when one vector is the variable of a given function, the other can be viewed as a constant vector. At this point, let us consider the real scalar function f (Z, Z∗ ) with variables Z, Z∗ ∈ Cm×n . Performing the vectorization of Z and Z∗ , respectively, from Equation (3.4.17) we get the first-order complex differential rule for the real scalar

166

Matrix Differential

function f (Z, Z∗ ): ∂f (Z, Z∗ ) dvecZ + ∂(vecZ)T ∂f (Z, Z∗ ) = dvecZ + ∂(vecZ)T

df (Z, Z∗ ) =

∂f (Z, Z∗ ) dvecZ∗ ∂(vecZ∗ )T ∂f (Z, Z∗ ) dvecZ∗ , ∂(vecZ∗ )T

(3.4.34)

where   ∂f (Z, Z∗ ) ∂f (Z, Z∗ ) ∂f (Z, Z∗ ) ∂f (Z, Z∗ ) ∂f (Z, Z∗ ) , = ,..., ,..., ,..., ∂(vec Z)T ∂Z11 ∂Zm1 ∂Z1n ∂Zmn   ∂f (Z, Z∗ ) ∂f (Z, Z∗ ) ∂f (Z, Z∗ ) ∂f (Z, Z∗ ) ∂f (Z, Z∗ ) . = , . . . , , . . . , , . . . , ∗ ∗ ∗ ∗ ∂(vec Z∗ )T ∂Z11 ∂Zm1 ∂Z1n ∂Zmn Now define the complex cogradient vector and the complex conjugate cogradient vector: ∂f (Z, Z∗ ) , ∂(vec Z)T ∂f (Z, Z∗ ) Dvec Z∗ f (Z, Z∗ ) = . ∂(vec Z∗ )T Dvec Z f (Z, Z∗ ) =

(3.4.35) (3.4.36)

The complex gradient vector and the complex conjugate gradient vector of the function f (Z, Z∗ ) are respectively defined as ∂f (Z, Z∗ ) , ∂ vec Z ∂f (Z, Z∗ ) . ∇vec Z∗ f (Z, Z∗ ) = ∂ vec Z∗ ∇vec Z f (Z, Z∗ ) =

(3.4.37) (3.4.38)

The conjugate gradient vector ∇vec Z∗ f (Z, Z∗ ) has the following properties [68]: (1) The conjugate gradient vector of the function f (Z, Z∗ ) at an extreme point is equal to the zero vector, i.e., ∇vec Z∗ f (Z, Z∗ ) = 0. (2) The conjugate gradient vector ∇vec Z∗ f (Z, Z∗ ) and the negative conjugate gradient vector −∇vec Z∗ f (Z, Z∗ ) point in the direction of the steepest ascent and steepest descent of the function f (Z, Z∗ ), respectively. (3) The step length of the steepest increase slope is ∇vec Z∗ f (Z, Z∗ ) 2 . (4) The conjugate gradient vector ∇vec Z∗ f (Z, Z∗ ) is the normal to the surface f (Z, Z∗ ) = const. Hence, the conjugate gradient vector ∇vec Z∗ f (Z, Z∗ ) and the negative conjugate gradient vector −∇vec Z∗ f (Z, Z∗ ) can be used separately in gradient ascent algorithms and gradient descent algorithms. Furthermore, the complex Jacobian matrix and the complex conjugate Jacobian

3.4 Complex Gradient Matrices

167

matrix of the real scalar function f (Z, Z∗ ) are respectively as follows:  ∂f (Z, Z∗ )  DZ f (Z, Z ) = ∂ ZT Z∗ = const matrix ⎡ ∂f (Z, Z∗ ) ∂f (Z, Z∗ ) ⎤ ··· ∂Zm1 ⎥ ⎢ ∂Z11 ⎢ ⎥ .. .. .. =⎢ ⎥, . . . ⎣ ⎦ ∗ ∗ ∂f (Z, Z ) ∂f (Z, Z ) ··· ∂Z1n ∂Zmn  ∗  ) ∂f (Z, Z def  DZ∗ f (Z, Z∗ ) = ∂ ZH Z = const matrix ⎡ ⎤ ∂f (Z, Z∗ ) ∂f (Z, Z∗ ) · · · ∗ ∗ ⎢ ∂Z11 ⎥ ∂Zm1 ⎢ ⎥ . . . ⎥. . . . =⎢ . . . ⎢ ⎥ ⎣ ∂f (Z, Z∗ ) ∗ ⎦ ∂f (Z, Z ) ··· ∗ ∗ ∗

def

∂Z1n

(3.4.39)

(3.4.40)

∂Zmn

Similarly, the complex gradient matrix and the complex conjugate gradient matrix of the real scalar function f (Z, Z∗ ) are respectively given by  ∂f (Z, Z∗ )  ∇Z f (Z, Z ) =  ∗ ∂Z Z = const matrix ⎡ ∂f (Z, Z∗ ) ∂f (Z, Z∗ ) ⎤ ··· ∂Z1n ⎥ ⎢ ∂Z11 ⎢ ⎥ .. .. .. =⎢ ⎥, . . . ⎣ ⎦ ∗ ∗ ∂f (Z, Z ) ∂f (Z, Z ) ··· ∂Zm1 ∂Zmn  ∗  ) ∂f (Z, Z def  ∇Z∗ f (Z, Z∗ ) = ∂ Z∗ Z = const matrix ⎡ ⎤ ∂f (Z, Z∗ ) ∂f (Z, Z∗ ) · · · ∗ ∗ ⎥ ⎢ ∂Z11 ∂Z1n ⎥ ⎢ . . . ⎥. . . . =⎢ . . . ⎥ ⎢ ⎣ ∂f (Z, Z∗ ) ∗ ⎦ ∂f (Z, Z ) ··· ∗ ∗ ∗

def

∂Zm1

(3.4.41)

(3.4.42)

∂Zmn

Summarizing the definitions above, there are the following relations among the various complex partial derivatives of the real scalar function f (Z, Z): (1) The conjugate gradient (cogradient) vector is equal to the complex conjugate of the gradient (cogradient) vector; and the conjugate Jacobian (gradient) matrix is equal to the complex conjugate of the Jacobian (gradient) matrix. (2) The gradient (conjugate gradient) vector is equal to the transpose of the co-

168

Matrix Differential

gradient (conjugate cogradient) vector, namely ∇vec Z f (Z, Z∗ ) = DTvec Z f (Z, Z∗ ), ∗

∇vec Z∗ f (Z, Z ) =

(3.4.43)

DTvec Z∗ f (Z, Z∗ ).

(3.4.44)

(3) The cogradient (conjugate cogradient) vector is equal to the transpose of the vectorization of Jacobian (conjugate Jacobian) matrix: Dvec Z f (Z, Z) = (vecDZ f (Z, Z∗ )) , T



(3.4.45)

T

Dvec Z∗ f (Z, Z) = (vecDZ∗ f (Z, Z )) .

(3.4.46)

(4) The gradient (conjugate gradient) matrix is equal to the transpose of the Jacobian (conjugate Jacobian) matrix: ∇Z f (Z, Z∗ ) = DTZ f (Z, Z∗ ), ∗

∇Z∗ f (Z, Z ) =

(3.4.47)

DTZ∗ f (Z, Z∗ ).

(3.4.48)

The following are the rules of operation for the complex gradient. 1. If f (Z, Z∗ ) = c (a constant), then its gradient matrix and conjugate gradient matrix are equal to the zero matrix, namely ∂c/∂ Z = O and ∂c/∂ Z∗ = O. 2. Linear rule If f (Z, Z∗ ) and g(Z, Z∗ ) are scalar functions, and c1 and c2 are complex numbers, then ∂(c1 f (Z, Z∗ ) + c2 g(Z, Z∗ )) ∂f (Z, Z∗ ) ∂g(Z, Z∗ ) = c1 + c2 . ∗ ∗ ∂Z ∂Z ∂ Z∗ 3. Multiplication rule ∗ ∗ ∂f (Z, Z∗ )g(Z, Z∗ ) ∗ ∂f (Z, Z ) ∗ ∂g(Z, Z ) = g(Z, Z ) + f (Z, Z ) . ∂Z∗ ∂ Z∗ ∂ Z∗

If g(Z, Z∗ ) = 0 then   ∗ ∗ ∂f /g 1 ∗ ∂f (Z, Z ) ∗ ∂g(Z, Z ) g(Z, Z ) . = 2 − f (Z, Z ) ∂ Z∗ g (Z, Z∗ ) ∂ Z∗ ∂ Z∗

4. Quotient rule

If h(Z, Z∗ ) = g(F(Z, Z∗ ), F∗ (Z, Z∗ )) then the quotient rule becomes ∂h(Z, Z∗ ) ∂g(F(Z, Z∗ ), F∗ (Z, Z∗ )) ∂ (vec F(Z, Z∗ )) = T ∂ vec Z ∂ vec Z ∂ (vec F(Z, Z∗ ))

T

∂g(F(Z, Z∗ ), F∗ (Z, Z∗ )) ∂ (vec F∗ (Z, Z∗ )) , T ∂ vec Z ∂ (vec F∗ (Z, Z∗ )) T

+

(3.4.49)

∂h(Z, Z∗ ) ∂g(F(Z, Z∗ ), F∗ (Z, Z∗ )) ∂ (vecF(Z, Z∗ )) = T ∂ vec Z∗ ∂ vec Z∗ ∂ (vec F(Z, Z∗ ))

T

∂g(F(Z, Z∗ ), F∗ (Z, Z∗ )) ∂ (vec F∗ (Z, Z∗ )) . T ∂ vec Z∗ ∂ (vecF∗ (Z, Z∗ )) T

+

(3.4.50)

3.4 Complex Gradient Matrices

169

3.4.3 Complex Gradient Matrix Identification If we let A = DZ f (Z, Z∗ ) and

B = DZ∗ f (Z, Z∗ ),

(3.4.51)

then ∂f (Z, Z∗ ) = rvecDZ f (Z, Z∗ ) = rvecA = (vec(AT ))T , ∂(vec Z)T ∂f (Z, Z∗ ) = rvecDZ∗ f (Z, Z∗ ) = rvecB = (vec(BT ))T . ∂(vec Z)H

(3.4.52) (3.4.53)

Hence, the first-order complex matrix differential formula (3.4.34) can be rewritten as df (Z, Z∗ ) = (vec(AT ))T dvec Z + (vec(BT ))T dvecZ∗ .

(3.4.54)

Using tr(CT D) = (vec C)T vecD, Equation (3.4.54) can be written as df (Z, Z∗ ) = tr(AdZ + BdZ∗ ).

(3.4.55)

From Equations (3.4.51)–(3.4.55) we can obtain the following proposition for identifying the complex Jacobian matrix and the complex gradient matrix. PROPOSITION 3.4 Given a scalar function f (Z, Z∗ ) : Cm×n × Cm×n → C, its complex Jacobian and gradient matrices can be respectively identified by + DZ f (Z, Z∗ ) = A, ∗ ∗ df (Z, Z ) = tr(AdZ + BdZ ) ⇔ (3.4.56) DZ∗ f (Z, Z∗ ) = B + ∇Z f (Z, Z∗ ) = AT , ∗ ∗ df (Z, Z ) = tr(AdZ + BdZ ) ⇔ (3.4.57) ∇Z∗ f (Z, Z∗ ) = BT . That is to say, the complex gradient matrix and the complex conjugate gradient matrix are respectively identified as the transposes of the matrices A and B. This proposition shows that the key to identifying a complex Jacobian or gradient matrix is to write the matrix differential of the function f (Z, Z∗ ) in the canonical form df (Z, Z∗ ) = tr(AdZ+BdZ∗ ). In particular, if f (Z, Z∗ ) is a real function then B = A∗ . EXAMPLE 3.21

The matrix differential of tr(ZAZ∗ B) is given by

d(tr(ZAZ∗ B)) = tr((dZ)AZ∗ B) + tr(ZA(dZ∗ )B) = tr(AZ∗ BdZ) + tr(BZAdZ∗ ) from which it follows that the gradient matrix and the complex conjugate gradient

170

Matrix Differential

matrix of tr(ZAZ∗ B) are respectively given by ∇Z tr(ZAZ∗ B) = (AZ∗ B)T = BT ZH AT , ∇Z∗ tr(ZAZ∗ B) = (BZA)T = AT ZT BT . EXAMPLE 3.22

From the complex matrix differentials



d|ZZ | = |ZZ∗ |tr((ZZ∗ )−1 d(ZZ∗ )) = |ZZ∗ |tr(Z∗ (ZZ∗ )−1 dZ) + |ZZ∗ |tr((ZZ∗ )−1 ZdZ∗ ), d|ZZH | = |ZZH |tr((ZZH )−1 d(ZZH )) = |ZZH |tr(ZH (ZZH )−1 dZ) + |ZZH |tr(ZT (Z∗ ZT )−1 dZ∗ ), we get the complex gradient matrix and the complex conjugate gradient matrix as follows: ∇Z |ZZ∗ | = |ZZ∗ |(ZH ZT )−1 ZH ,

∇Z∗ |ZZ∗ | = |ZZ∗ |ZT (ZH ZT )−1 ,

∇Z |ZZH | = |ZZH |(Z∗ ZT )−1 Z∗ ,

∇Z∗ |ZZH | = |ZZH |(ZZH )−1 Z.

Table 3.9 lists the complex gradient matrices of several trace functions. Table 3.9 Complex gradient matrices of trace functions f (Z, Z∗ )

df

∂f /∂Z

∂f /∂ Z∗

tr(AZ)

tr(AdZ)

AT

O

H



T

tr(AZ )

tr(A dZ )

T

tr(ZAZ B) tr(ZAZB) ∗

tr(ZAZ B)

O

T

T

T

T

tr((AZ B + A Z B )dZ) tr((AZB + BZA)dZ) ∗

A

T

T

B ZA + BZA

O

T

O

(AZB + BZA) ∗

tr(AZ BddZ + BZAddZ )

T

H

B Z A

T

AT ZT BT

tr(ZAZH B) tr(AZH BdZ + AT ZT BT dZ∗ ) BT Z∗ AT tr(AZ k

−1

tr(Z )

)

−1

−tr(Z k tr(Z

AZ

k−1

−1

dZ)

dZ)

−T

−Z

T

BZA −T

A Z

T k−1

k(Z )

O O

Table 3.10 lists the complex gradient matrices of several determinant functions. If f (z, z∗ ) = [f1 (z, z∗ ), . . . , fn (z, z∗ )]T is an n × 1 complex vector function with m × 1 complex vector variable then ⎤ ⎡ ⎤ ⎤ ⎡ ⎡ df1 (z, z∗ ) Dz f1 (z, z∗ ) Dz∗ f1 (z, z∗ ) ⎥ ⎢ ⎥ ⎥ ∗ ⎢ ⎢ .. .. .. ⎦=⎣ ⎦ dz + ⎣ ⎦ dz , ⎣ . . . dfn (z, z∗ )

Dz fn (z, z∗ )

Dz∗ fn (z, z∗ )

which can simply be written as df (z, z∗ ) = Dz f (z, z∗ )dz + Dz∗ f (z, z∗ )dz∗ ,

(3.4.58)

3.4 Complex Gradient Matrices

171

Table 3.10 Complex gradient matrices of determinant functions f (Z, Z∗ ) df

∂f /∂ Z∗

∂f /∂Z

|Z|tr(Z−1 dZ) |Z|Z−T T T T −1 2|ZZ |tr(Z (ZZ ) dZ) 2|ZZT |(ZZT )−1 Z 2|ZT Z|tr((ZT Z)−1 ZT dZ) 2|ZT Z|Z(ZT Z)−1 ZZ∗ |tr(Z∗ (ZZ∗ )−1 dZ |ZZ∗ |(ZH ZT )−1 ZH + (ZZ∗ )−1 ZdZ∗ )

|Z| |ZZT | |ZT Z| |ZZ∗ |

|Z∗ Z|tr((Z∗ Z)−1 Z∗ dZ

|Z∗ Z|

|ZZH |tr(ZH (ZZH )−1 dZ + ZT (Z∗ ZT )−1 dZ∗ ) |ZH Z|tr((ZH Z)−1 ZH dZ

|ZH Z|

+ (ZT Z∗ )−1 ZT dZ∗ ) k|Z|k tr(Z−1 dZ)

|Zk |

|ZZ∗ |ZT (ZH ZT )−1

|Z∗ Z|ZH (ZT ZH )−1 |Z∗ Z|(ZT ZH )−1 ZT

+ Z(Z∗ Z)−1 dZ∗ )

|ZZH |

O O O

|ZZH |(Z∗ ZT )−1 Z∗

|ZZH |(ZZH )−1 Z

|ZH Z|Z∗ (ZT Z∗ )−1

|ZH Z|Z(ZH Z)−1

k|Z|k Z−T

O

where df (z, z∗ ) = [df1 (z, z∗ ), . . . , dfn (z, z∗ )]T , while ⎡ Dz f (z, z∗ ) =

∂f1 (z, z∗ ) ⎢ ∂z1

∂ f (z, z∗ ) ⎢ =⎢ ⎢ ∂ zT ⎣ ∂f

.. .

n (z, z



) ∂z1 ⎡ ∂f1 (z, z∗ ) ⎢ ∂z1∗

∂ f (z, z∗ ) ⎢ .. Dz∗ f (z, z∗ ) = =⎢ . ⎢ ∂ zH ⎣ ∂f (z, z∗ ) n ∂z1∗

··· ..

.

··· ··· ..

.

···



∂f1 (z, z∗ ) ∂zm ⎥

⎥ ⎥, ⎥ ∗ ⎦ ∂fn (z, z ) .. .

(3.4.59)

∂zm ⎤ ∂f1 (z, z∗ ) ∗ ∂zm ⎥

⎥ ⎥ ⎥ ∗ ⎦ ∂fn (z, z ) .. .

(3.4.60)

∗ ∂zm

are respectively the complex Jacobian matrix and the complex conjugate Jacobian matrix of the vector function f (z, z∗ ). The above result is easily extended to the case of complex matrix functions as follows. For a p×q matrix function F(Z, Z∗ ) with m×n complex matrix variable Z, if F(Z, Z∗ ) = [f1 (Z, Z∗ ), . . . , fq (Z, Z∗ )], then dF(Z, Z∗ ) = [df1 (Z, Z∗ ), . . . , dfq (Z, Z∗ )], and (3.4.58) holds for the vector functions fi (Z, Z∗ ), i = 1, . . . , q. This implies that ⎤ ⎡ ⎤ ⎤ ⎡ df1 (Z, Z∗ ) Dvec Z f1 (Z, Z∗ ) Dvec Z∗ f1 (Z, Z∗ ) ⎥ ⎢ ⎥ ⎥ ⎢ ⎢ .. .. .. ∗ ⎦=⎣ ⎦ dvecZ + ⎣ ⎦ dvecZ , (3.4.61) ⎣ . . . ⎡

dfq (Z, Z∗ )

Dvec Z fq (Z, Z∗ )

Dvec Z∗ fq (Z, Z∗ )

172

Matrix Differential

where Dvec Z fi (Z, Z∗ ) =

∂ fi (Z, Z∗ ) ∈ Cp×mn , ∂(vecZ)T

Dvec Z∗ fi (Z, Z∗ ) =

∂ fi (Z, Z∗ ) ∈ Cp×mn . ∂(vecZ∗ )T

Equation (3.4.61) can be simply rewritten as dvecF(Z, Z∗ ) = AdvecZ + BdvecZ∗ ∈ Cpq ,

(3.4.62)

where dvecF(Z, Z∗ )) = [df11 (Z, Z∗ ), . . . , dfp1 (Z, Z∗ ), . . . , df1q (Z, Z∗ ), . . . , dfpq (Z, Z∗ )]T , dvecZ = [dZ11 , . . . , dZm1 , . . . , dZ1n , . . . , dZmn ]T , ∗ ∗ ∗ ∗ dvecZ∗ = [dZ11 , . . . , dZm1 , . . . , dZ1n , . . . , dZmn ]T ,

while

⎡ ∂f (Z, Z∗ ) 11 ⎢ ∂Z11 ⎢ .. ⎢ . ⎢ ⎢ ∂fp1 (Z, Z∗ ) ⎢ ⎢ ∂Z11 ⎢ .. A=⎢ . ⎢ ⎢ ∂f (Z, Z∗ ) ⎢ 1q ⎢ ∂Z 11 ⎢ ⎢ .. ⎢ . ⎣ ∗ ∂fpq (Z, Z ) ∂Z11

··· .. . ··· .. . ··· .. . ···

∂f11 (Z, Z∗ ) ∂Zm1

.. .

··· .. .

.. .

··· .. .

.. .

··· .. .

∂fp1 (Z, Z∗ ) ∂Zm1 ∂f1q (Z, Z∗ ) ∂Zm1 ∂fpq (Z, Z∗ ) ∂Zm1

∂ vec F(Z, Z∗ ) = Dvec Z F(Z, Z∗ ), ∂(vec Z)T ⎡ ∂f (Z, Z∗ ) ∂f11 (Z, Z∗ ) 11 ··· ∗ ∗ ∂Zm1 ⎢ ∂Z11 ⎢ . . . ⎢ .. .. .. ⎢ ⎢ ∂fp1 (Z, Z∗ ) ∂fp1 (Z, Z∗ ) ⎢ ··· ⎢ ∂Z ∗ ∗ ∂Zm1 11 ⎢ ⎢ . . . .. .. .. B=⎢ ⎢ ⎢ ∂f1q (Z, Z∗ ) ∂f1q (Z, Z∗ ) ⎢ ··· ∗ ∗ ⎢ ∂Z11 ∂Zm1 ⎢ .. .. .. ⎢ ⎢ . . . ⎣ ∂f (Z, Z∗ ) ∂fpq (Z, Z∗ ) pq ··· ∗ ∗

···

∂f11 (Z, Z∗ ) ∂Z1n

.. .

··· .. .

.. .

··· .. .

.. .

··· .. .

∂fp1 (Z, Z∗ ) ∂Z1n ∂f1q (Z, Z∗ ) ∂Z1n ∂fpq (Z, Z∗ ) ∂Z1n

···



∂f11 (Z, Z∗ ) ∂Zmn ⎥

⎥ ⎥ ⎥ ∗ ⎥ ∂fp1 (Z, Z ) ⎥ ∂Zmn ⎥ ⎥ .. ⎥ . ⎥ ∂f1q (Z, Z∗ ) ⎥ ⎥ ∂Zmn ⎥ ⎥ ⎥ .. ⎥ . ∗ ⎦ .. .

∂fpq (Z, Z ) ∂Zmn

=

∂Z11

=



∂Zm1

(3.4.63) ··· .. . ··· .. . ··· .. . ···

∂ vec F(Z, Z ) = Dvec Z∗ F(Z, Z∗ ). ∂(vec Z∗ )T

∂f11 (Z, Z∗ ) ∗ ∂Z1n

···

∂fp1 (Z, Z∗ ) ∗ ∂Z1n

···

∂f1q (Z, Z∗ ) ∗ ∂Z1n

···

∂fpq (Z, Z∗ ) ∗ ∂Z1n

···

.. . .. . .. .

.. . .. . .. .



∂f11 (Z, Z∗ ) ∗ ∂Zmn ⎥

⎥ ⎥ ⎥ ∗ ⎥ ∂fp1 (Z, Z ) ⎥ ⎥ ∗ ∂Zmn ⎥ ⎥ .. ⎥ . ⎥ ∗ ⎥ ∂f1q (Z, Z ) ⎥ ∗ ⎥ ∂Zmn ⎥ .. ⎥ ⎥ . ⎦ ∂f (Z, Z∗ ) .. .

pq

∗ ∂Zmn

(3.4.64)

3.4 Complex Gradient Matrices

173

Obviously the matrices A and B are the complex Jacobian matrix and the complex conjugate Jacobian matrix, respectively. The complex gradient matrix and the complex conjugate gradient matrix of the matrix function F(Z, Z∗ ) are respectively defined as ∂(vec F(Z, Z∗ ))T = (Dvec Z F(Z, Z∗ ))T , ∂ vec Z ∂(vec F(Z, Z∗ ))T ∇vec Z∗ F(Z, Z∗ ) = = (Dvec Z∗ F(Z, Z∗ ))T . ∂ vec Z∗ ∇vec Z F(Z, Z∗ ) =

(3.4.65) (3.4.66)

In particular, for a real scalar function f (Z, Z∗ ), Equation (3.4.62) reduces to Equation (3.4.34). Summarizing the above discussions, from Equation (3.4.62) we get the following proposition. PROPOSITION 3.5 For a complex matrix function F(Z, Z∗ ) ∈ Cp×q with Z, Z∗ ∈ Cm×n , its complex Jacobian matrix and the conjugate Jacobian matrix can be identified as follows: dvecF(Z, Z∗ ) = Advec Z + BdvecZ∗ + Dvec Z F(Z, Z∗ ) = A, ⇔ Dvec Z∗ F(Z, Z∗ ) = B,

(3.4.67)

and the complex gradient matrix and the conjugate gradient matrix can be identified by dvecF(Z, Z∗ ) = AdvecZ + BdvecZ∗ + ∇vec Z F(Z, Z∗ ) = AT , ⇔ ∇vec Z∗ F(Z, Z∗ ) = BT .

(3.4.68)

If d(F(Z, Z∗ )) = A(dZ)B + C(dZ∗ )D, then the vectorization result is given by dvecF(Z, Z∗ ) = (BT ⊗ A)dvecZ + (DT ⊗ C)dvecZ∗ . By Proposition 3.5 we have the following identification formula: dF(Z, Z∗ ) = A(dZ)B + C(dZ∗ )D + Dvec Z F(Z, Z∗ ) = BT ⊗ A, ⇔ Dvec Z∗ F(Z, Z∗ ) = DT ⊗ C.

(3.4.69)

Similarly, if dF(Z, Z∗ ) = A(dZ)T B + C(dZ∗ )T D then we have the result dvecF(Z, Z∗ ) = (BT ⊗ A)dvecZT + (DT ⊗ C)dvecZH = (BT ⊗ A)Kmn dvecZ + (DT ⊗ C)Kmn dvecZ∗ , where we have used the vectorization property vecXTm×n = Kmn vecX. By Prop-

174

Matrix Differential

osition 3.5, the following identification formula is obtained: dF(Z, Z∗ ) = A(dZ)T B + C(dZ∗ )T D + Dvec Z F(Z, Z∗ ) = (BT ⊗ A)Kmn , ⇔ Dvec Z∗ F(Z, Z∗ ) = (DT ⊗ C)Kmn .

(3.4.70)

The above equation shows that, as in the vector case, the key to identifying the gradient matrix and conjugate gradient matrix of a matrix function F(Z, Z∗ ) is to write its matrix differential into the canonical form dF(Z, Z∗ ) = A(dZ)T B + C(dZ∗ )T D. Table 3.11 lists the corresponding relationships between the first-order complex matrix differential and the complex Jacobian matrix, where z ∈ Cm , Z ∈ Cm×n , F ∈ Cp×q . Table 3.11 Complex matrix differential and complex Jacobian matrix Function First-order matrix differential

Jacobian matrix

∂f ∂f =b = a, ∂z ∂z ∗ ∂f ∂f f (z, z∗ ) df (z, z∗ ) = aT dz + bT dz∗ = aT , = bT ∂zT ∂ zH ∂f ∂f f (Z, Z∗ ) df (Z, Z∗ ) = tr(AdZ + BdZ∗ ) = A, =B ∂ ZT ∂ ZH ∂ vec F ∂ vec F F(Z, Z∗ ) d(vec F) = Ad(vec Z) + Bd(vec Z∗ ) = A, =B ∂(vec Z)T ∂(vec Z∗ )T ⎧ ∂ vec F T ⎪ ⎪ ⎨ ∂(vec Z)T = B ⊗ A ∗ dF = A(dZ)B + C(dZ )D, ∂ vec F ⎪ ⎪ ⎩ = DT ⊗ C ∂(vec Z∗ )T ⎧ ∂vec F T ⎪ ⎪ ⎨ ∂(vec Z)T = (B ⊗ A)Kmn T ∗ T dF = A(dZ) B + C(dZ ) D ⎪ ∂ vecF ⎪ = (DT ⊗ C)Kmn ⎩ ∂ (vec Z∗ )T f (z, z ∗ )

df (z, z ∗ ) = adz + bdz ∗

3.5 Complex Hessian Matrices and Identification The previous section analyzed the first-order complex derivative matrices (the complex Jacobian matrix and complex gradient matrix) and their identification. In this section we discuss the second-order complex derivative matrices of the function f (Z, Z∗ ) and their identification. These second-order complex matrices are a full Hessian matrix and four part Hessian matrices.

3.5 Complex Hessian Matrices and Identification

175

3.5.1 Complex Hessian Matrices The differential of the real function f = f (Z, Z∗ ) can be equivalently written as df = (DZ f )dvecZ + (DZ∗ f )dvecZ∗ ,

(3.5.1)

where DZ f =

∂f , ∂(vec Z)T

DZ∗ f =

∂f . ∂(vec Z∗ )T

(3.5.2)

Noting that the differentials of both DZ f and DZ∗ f are row vectors, we have T ∂ DZ f ∂ DZ f ∗ d(DZ f ) = dvecZ dvecZ + ∂ vecZ ∂ vecZ∗ ∂2f ∂2f ∗ T = (dvecZ)T + (dvecZ ) ∂ vecZ∂(vecZ)T ∂vecZ∗ ∂ (vecZ)T T  ∂ DZ∗ f ∂ DZ∗ f d(DZ∗ f ) = dvecZ∗ dvecZ + ∂ vecZ ∂(vecZ)T ∂2f ∂2f = (dvecZ)T + (d(vec Z∗ )T ) . ∗ T ∗ ∂ vecZ∂(vec Z ) ∂ vecZ ∂(vecZ∗ )T 

Since dvecZ is not a function of vecZ, and dvecZ∗ is not a function of vecZ∗ , it follows that d2 (vec Z) = d(dvecZ) = 0,

d2 (vecZ∗ ) = d(dvecZ∗ ) = 0.

(3.5.3)

Hence, the second-order differential of f = f (Z, Z∗ ) is given by d2 f = d(DZ f )d(vecZ) + d(DZ∗ f )dvecZ∗ ∂2f dvecZ ∂ vecZ∂(vecZ)T ∂2f dvecZ + (dvecZ∗ )T ∗ ∂ vecZ ∂(vecZ)T ∂2f dvecZ∗ + (dvecZ)T ∂ vecZ∂(vec Z∗ )T ∂2f + (dvecZ∗ )T dvecZ∗ , ∗ ∂ vecZ ∂(vec Z∗ )T

= (dvecZ)T

which can be written as    HZ∗ ,Z d2 f = (dvecZ∗ )T , (dvecZ)T HZ,Z H    dvecZ dvecZ , H = dvecZ∗ dvecZ∗

HZ∗ ,Z∗ HZ,Z∗



dvecZ dvecZ∗



(3.5.4)

176

Matrix Differential

where

 HZ∗ ,Z H= HZ,Z

HZ∗ ,Z∗ HZ,Z∗

 (3.5.5)

is known as the full complex Hessian matrix of the function f (Z, Z∗ ), and ⎫ ∂ 2 f (Z, Z∗ ) ⎪ ⎪ , HZ∗ ,Z = ⎪ ⎪ ∂ vecZ∗ ∂(vec Z)T ⎪ ⎪ ⎪ ⎪ 2 ∗ ⎪ ∂ f (Z, Z ) ⎪ ⎪ ⎪ , HZ∗ ,Z∗ = ∗ ∗ T ∂ vecZ ∂(vec Z ) ⎬ (3.5.6) ⎪ ∂ 2 f (Z, Z∗ ) ⎪ ⎪ , HZ,Z = ⎪ ∂ vecZ∂(vec Z)T ⎪ ⎪ ⎪ ⎪ ⎪ 2 ∗ ⎪ ∂ f (Z, Z ) ⎪ ⎪ ⎭ , HZ,Z∗ = ∗ T ∂ vecZ∂(vec Z ) are respectively the part complex Hessian matrices of the function f (Z, Z∗ ); of these, HZ∗ ,Z is the main complex Hessian matrix of the function f (Z, Z∗ ). By the above definition formulas, it is easy to show that the Hessian matrices of a scalar function f (Z, Z∗ ) have the following properties. 1. The part complex Hessian matrices HZ∗ ,Z and HZ,Z∗ are Hermitian, and are complex conjugates: HZ∗ ,Z = HH Z∗ ,Z ,

HZ,Z∗ = HH Z,Z∗ ,

HZ∗ ,Z = H∗Z,Z∗ .

(3.5.7)

2. The other two part complex Hessian matrices are symmetric and complex conjugate each other: HZ,Z = HTZ,Z ,

HZ∗ ,Z∗ = HTZ∗ ,Z∗

HZ,Z = H∗Z∗ ,Z∗ .

(3.5.8)

3. The complex full Hessian matrix is Hermitian, i.e., H = HH . From Equation (3.5.4) it can be seen that, because the second-order differential of the real function f (Z, Z∗ ), d2 f , is a quadratic function, the positive definiteness of the full Hessian matrix is determined by d2 f : (1) (2) (3) (4)

The The The The

full full full full

Hessian Hessian Hessian Hessian

matrix matrix matrix matrix

is is is is

positive definite if d2 f > 0 holds for all vec Z. positive semi-definite if d2 f ≥ 0 holds for all vecZ. negative definite if d2 f < 0 holds for all vecZ. negative semi-definite if d2 f ≤ 0 holds for all vecZ.

For a real function f = f (z, z∗ ), its second-order differential is given by 

dz d f= dz∗ 2

H

 dz , H dz∗ 

(3.5.9)

3.5 Complex Hessian Matrices and Identification

177

in which H is the full complex Hessian matrix of the function f (z, z∗ ), and is defined as   Hz∗ ,z Hz∗ ,z∗ H= . (3.5.10) Hz,z Hz,z∗ The four part complex Hessian matrices are given by ⎫ ∂ 2 f (z, z∗ ) ⎪ Hz∗ ,z = ,⎪ ⎪ ⎪ ∂ z∗ ∂ zT ⎪ ⎪ 2 ∗ ⎪ ⎪ ∂ f (z, z ) ⎪ ⎬ Hz∗ ,z∗ = ,⎪ ∗ H ∂z ∂z ∂ 2 f (z, z∗ ) ⎪ ⎪ Hz,z = ,⎪ ⎪ ∂ z∂ zT ⎪ ⎪ ⎪ 2 ∗ ⎪ ∂ f (z, z ) ⎪ ⎪ Hz,z∗ = .⎭ ∂ z∂ zH

(3.5.11)

Clearly, the full complex Hessian matrix is Hermitian, i.e., H = HH ; and the part complex Hessian matrices have the following relationships: Hz∗ ,z = HH z∗ ,z , Hz,z =

HTz,z ,

Hz,z∗ = HH z,z∗ ,

Hz∗ ,z = H∗z,z∗ ,

(3.5.12)

HTz∗ ,z∗

H∗z∗ ,z∗ .

(3.5.13)

Hz∗ ,z∗ =

Hz,z =

The above analysis show that, in complex Hessian matrix identification, we just need to identify two of the part Hessian matrices HZ,Z and HZ,Z∗ , since HZ∗ ,Z∗ = H∗Z,Z and HZ∗ ,Z = H∗Z,Z∗ . 3.5.2 Complex Hessian Matrix Identification By Proposition 3.5 it is known that the complex differential of a real scalar function f (Z, Z∗ ) can be written in the canonical form df (Z, Z∗ ) = tr(AdZ+A∗ dZ∗ ), where A = A(Z, Z∗ ) is usually a complex matrix function of the matrix variables Z and Z∗ . Hence, the complex matrix differential of A can be expressed as dA = C(dZ)D + E(dZ∗ )F

or dA = D(dZ)T C + F(dZ∗ )T E.

(3.5.14)

Substituting (3.5.14) into the second-order complex differential d2 f (Z, Z∗ ) = d(df (Z, Z∗ )) = tr(dAdZ + dA∗ dZ∗ ),

(3.5.15)

we get d2 f (Z, Z∗ ) = tr(C(dZ)DdZ) + tr(E(dZ∗ )FdZ) + tr(C∗ (dZ∗ )D∗ dZ∗ ) + tr(E∗ (dZ)F∗ dZ∗ ),

(3.5.16)

or d2 f (Z, Z∗ ) = tr(D(dZ)T CdZ) + tr(F(dZ∗ )T EdZ) + tr(D∗ (dZ∗ )T C∗ dZ∗ ) + tr(F∗ (dZ)T E∗ dZ∗ ).

(3.5.17)

178

Matrix Differential

Case 1 Hessian matrix identification based on (3.5.16) By the trace property tr(XYUV) = (vec(VT ))T (UT ⊗ X)vecY, the vectorization property vecZT = Kmn vecZ and the commutation matrix property KTmn = Knm , it is easy to see that tr(C(dZ)DdZ) = (Kmn vecdZ)T (DT ⊗ C)vecdZ = (dvecZ)T Knm (DT ⊗ C)dvecZ. Similarly, we have tr(E(dZ∗ )FdZ) = (dvecZ)T Knm (FT ⊗ E)dvecZ∗ , tr(E∗ (dZ)F∗ dZ∗ ) = (dvecZ∗ )T Knm (FH ⊗ E∗ )dvecZ, tr(C∗ (dZ∗ )D∗ dZ∗ ) = (dvecZ∗ )T Knm (DH ⊗ C∗ )dvecZ∗ . Then, on the one hand, Equation (3.5.16) can be written as follows: d2 f (Z, Z∗ ) = (dvecZ)T Knm (DT ⊗ C)dvecZ + (dvecZ)T Knm (FT ⊗ E)dvecZ∗ + (dvecZ∗ )T Knm (FH ⊗ E∗ )dvecZ + (dvecZ∗ )T Knm (DH ⊗ C∗ )dvecZ.

(3.5.18)

On the other hand, Equation (3.5.4) can be rewritten as follows: d2 f = (dvecZ)T HZ,Z dvecZ + (dvecZ)T HZ,Z∗ dvecZ∗ + (dvecZ∗ )T HZ∗ ,Z dvecZ) + (dvecZ∗ )T HZ∗ ,Z∗ dvecZ∗ . By comparing (3.5.19) with (3.5.18), it immediately follows that  1 HZ,Z = H∗Z∗ ,Z∗ = Knm (DT ⊗ C) + (Knm (DT ⊗ C))H , 2  1 ∗ HZ,Z∗ = HZ∗ ,Z = Knm (FT ⊗ E) + (Knm (FT ⊗ E))H . 2 Case 2 Hessian matrix identification based on (3.5.17)

(3.5.19)

(3.5.20) (3.5.21)

By the trace property tr(XYUV) = (vecVT )T (UT ⊗ X)vec(Y), it is easy to see that the second-order differential formula (3.5.17) can be rewritten as d2 f (Z, Z∗ ) = (dvecZ)T (DT ⊗ C)dvecZ + (dvecZ∗ )T (FT ⊗ E)dvecZ + (dvecZ)T (FH ⊗ E∗ )dvecZ∗ + (dvecZ∗ )T (DH ⊗ C∗ )dvecZ. Comparing (3.5.22) with (3.5.19), we have  1 T HZ,Z = H∗Z∗ ,Z∗ = D ⊗ C + (DT ⊗ C)H , 2  1 T ∗ HZ,Z∗ = HZ∗ ,Z = (F ⊗ E) + (FT ⊗ E)H . 2

(3.5.22)

(3.5.23) (3.5.24)

Exercises

179

The results above can be summarized in the following theorem on complex Hessian matrix identification. THEOREM 3.4 Let f (Z, Z∗ ) be a differentiable real function of m × n matrix variables Z, Z∗ . If the second-order differential d2 f (Z, Z∗ ) has the canonical form (3.5.16) or (3.5.17) then the complex Hessian matrix can be identified by  1 Knm (DT ⊗ C) + (Knm (DT ⊗ C))T tr(C(dZ)DdZ) ⇔ HZ,Z = 2  1 ∗ tr(E(dZ)FdZ ) ⇔ HZ,Z∗ = Knm (FT ⊗ E) + (Knm (FT ⊗ E))H 2 or    1 T tr C(dZ)D(dZ)T ⇔ HZ,Z = D ⊗ C + (DT ⊗ C)T 2    1 T ∗ T tr E(dZ)F(dZ ) ⇔ HZ,Z∗ = (F ⊗ E) + (FT ⊗ E)H 2 together with HZ∗ ,Z∗ = H∗Z,Z and HZ∗ ,Z = H∗Z,Z∗ . The complex conjugate gradient matrix and the full complex Hessian matrix play an important role in the optimization of an objective function with a complex matrix as variable.

Exercises Show that the matrix differential of the Kronecker product is given by d(X ⊗ Y) = (dX)Y + X ⊗ dY. 3.2 Prove that d(X∗Y) = (dX)Y+X∗dY, where X∗Y represents the Hadamard product of X and Y. 3.3 Show that d(UVW) = (dU)VW + U(dV)W + UV(dW). More generally, show that d(U1 U2 · · · Uk ) = (dU1 )U2 · · · Uk + · · · + U1 · · · Uk−1 (dUk ). 3.4 Compute d(tr(Ak )), where k is an integer. 3.5 Let the eigenvalues of the matrix X ∈ Rn×n be λ1 , . . . , λn . Use |X| = λ1 · · · λn to show that d|X| = |X|tr(X−1 dX). (Hint: Rewrite the determinant as |X| = exp(log λ1 + · · · + log λn ).) 3.6 Find the Jacobian matrices and the gradient matrices of the determinant logarithm functions log |XT AX|, log |XAXT | and log |XAX|. 3.7 Show the result dtr(XT X) = 2tr(XT dX). 3.8 Show that the matrix differential of a Kronecker product d(Y ⊗ Z) is dY ⊗ Z + Y ⊗ dZ.   3.9 Given the trace function tr (XT CX)−1 A , find its differential and gradient matrix. 3.10 Show that  dA−1 −1 il =− A−1 ij Akl . dAjk j 3.1

k

180

3.11 3.12 3.13 3.14 3.15 3.16 3.17 3.18

Matrix Differential

 (Hint: Write A−1 A = I as j A−1 ij Ajk = δik .) Given the determinant logarithms log |XT AX|, log |XAXT | and log |XAX|, find their Hessian matrices. Identify the Jacobian matrices of the matrix functions AXB and AX−1 B. Find the gradient matrices of the functions tr(AX−1 B) and tr(AXT BXC). Given the determinant functions |XT AX|, |XAXT |, |XAX| and |XT AXT |, determine their gradient matrices. Given a matrix X ∈ Cn×n with eigenvalues λ1 , . . . , λn , use |X| = λ1 · · · λn to prove that d|X| = |X|tr(X−1 dX). Compute d|A−k | and dtr(A−k ), where k is an integer. Identify the Jacobian matrices and the gradient matrices of the determinant logarithms log |XT AX|, log |XAXT | and log |XAX|, respectively. Show that if F is a matrix function and second differentiable then d2 log |F| = −tr(F−1 dF)2 + tr(F−1 )d2 F.

3.19 Use d(X−1 X) to find the differential of the inverse matrix, dX−1 . 3.20 Show that, for a nonsingular matrix X ∈ Rn×n , its inverse matrix X−1 is infinitely differentiable and its rth-order differential is given by d(r) (X−1 ) = (−1)r r!(X−1 dX)r X−1 ,

r = 1, 2, . . .

3.21 Let X ∈ Rm×n , and let its Moore–Penrose inverse matrix be X† . Show that  T d(X† X) = X† (dX)(In − X† X) + X† (dX)(In − X† X) ,  T d(XX† ) = (Im − XX† )(dX)X† + (Im − XX† )(dX)X† . 3.22 3.23 3.24 3.25

(Hint: The matrices X† X and XX† are idempotent and symmetric.) Find the Hessian matrices of the real scalar functions f (x) = aT x and f (x) = xT Ax. Given a real scalar function f (X) = tr(AXBXT ), show that its Hessian matrix is given by H[f (X)] = BT ⊗ A + B ⊗ AT . Compute the complex matrix differential dtr(XAXH B). Given a complex-valued function f (X, X∗ ) = |XAXH B|, find its gradient matrix ∇X f (X, X∗ ) = =

∂f (X, X∗ ) . ∂ X∗

∂f (X, X∗ ) and conjugate gradient matrix ∇X∗ f (X, X∗ ) ∂X

3.26 Given a function f (x) = (|aH x|2 − 1)2 , find its conjugate gradient vector ∂f (x) . ∂ x∗

3.27 Let f (X) = log |det(X)|, find its conjugate gradient vector

∂f (X) . ∂ X∗

3.28 Let f (Z, Z∗ ) = |ZZH |, find its gradient matrix and its four part Hessian matrices.

PART II MATRIX ANALYSIS

4 Gradient Analysis and Optimization

Optimization theory mainly studies the extreme values (the maximal or minimal values) of an objective function, which is usually some real-valued function of a real vector or matrix variable. In many engineering applications, however, the variable of an objective function is a complex vector or matrix. Optimization theory mainly considers (1) the existence conditions for an extremum value (gradient analysis); (2) the design of optimization algorithms and convergence analysis.

4.1 Real Gradient Analysis Consider an unconstrained minimization problem for f (x) : Rn → R described by min f (x),

(4.1.1)

x ∈S

where S ∈ Rn is a subset of the n-dimensional vector space Rn ; the vector variable x ∈ S is called the optimization vector and is chosen so as to fulfil Equation (4.1.1). The function f : Rn → R is known as the objective function; it represents the cost or price paid by selecting the optimization vector x, and thus is also called the cost function. Conversely, the negative cost function −f (x) can be understood as the value function or utility function of x. Hence, solving the optimization problem (4.1.1) corresponds to minimizing the cost function or maximizing the value function. That is to say, the minimization problem min f (x) and the maximization x ∈S

problem max{−f (x)} of the negative objective function are equivalent. x ∈S

The above optimization problem has no constraint conditions and thus is called an unconstrained optimization problem. Most nonlinear programming methods for solving unconstrained optimization problems are based on the idea of relaxation and approximation [344]. • Relaxation The sequence {ak }∞ k=0 is known as a relaxation sequence, if ak+1 ≤ ak , ∀ k ≥ 0. Hence, in the process of solving iteratively the optimization problem (4.1.1) it is necessary to generate a relaxation sequence of the cost function f (xk+1 ) ≤ f (xk ), k = 0, 1, . . . 183

184

Gradient Analysis and Optimization

• Approximation Approximating an objective function means using a simpler objective function instead of the original objective function. Therefore, by relaxation and approximation, we can achieve the following. (1) If the objective function f (x) is lower bounded in the definition domain S ∈ Rn then the sequence {f (xk )}∞ k=0 must converge. (2) In any case, we can improve upon the initial value of the objective function f (x). (3) The minimization of a nonlinear objective function f (x) can be implemented by numerical methods to a sufficiently high approximation accuracy.

4.1.1 Stationary Points and Extreme Points The stationary points and extreme points of an objective function play a key role in optimization. The analysis of stationary points depends on the gradient (i.e., the first-order gradient) vector of the objective function, and the analysis of the extreme points depends on the Hessian matrix (i.e., the second-order gradient) of the objective function. Hence the gradient analysis of an objective function is divided into first-order gradient analysis (stationary-point analysis) and secondorder gradient analysis (extreme-point analysis). In optimization, a stationary point or critical point of a differentiable function of one variable is a point in the domain of the function where its derivative is zero. Informally, a stationary point is a point where the function stops increasing or decreasing (hence the name); it could be a saddle point, however, is usually expected to be a global minimum or maximum point of the objective function f (x). DEFINITION 4.1 A point x in the vector subspace S ∈ Rn is known as a global minimum point of the function f (x) if f (x ) ≤ f (x),

∀ x ∈ S, x = x .

(4.1.2)

A global minimum point is also called an absolute minimum point. The value of the function at this point, f (x ), is called the global minimum or absolute minimum of the function f (x) in the vector subspace S. Let D denote the definition domain of a function f (x), where D may be the whole vector space Rn or some subset of it. If f (x ) < f (x),

∀ x ∈ D,

(4.1.3)

then x is said to be a strict global minimum point or a strict absolute minimum point of the function f (x). Needless to say, the ideal goal of minimization is to find the global minimum of

4.1 Real Gradient Analysis

185

a given objective function. However, this desired object is often difficult to achieve. The reasons are as follows. (1) It is usually difficult to know the global or complete information about a function f (x) in the definition domain D. (2) It is usually impractical to design an algorithm for identifying a global extreme point, because it is almost impossible to compare the value f (x ) with all values of the function f (x) in the definition domain D. In contrast, it is much easier to obtain local information about an objective function f (x) in the vicinity of some point c. At the same time, it is much simpler to design an algorithm for comparing the function value at some point c with other function values at the points near c. Hence, most minimization algorithms can find only a local minimum point c; the value of the objective function at c is the minimum of its values in the neighborhood of c. DEFINITION 4.2 Given a point c ∈ D and a positive number r, the set of all points x satisfying x − c 2 < r is said to be an open neighborhood with radius r of the point c, denoted Bo (c; r) = {x| x ∈ D, x − c 2 < r}.

(4.1.4)

Bc (c; r) = {x| x ∈ D, x − c 2 ≤ r}, then we say that Bc (c; r) is a closed neighborhood of c.

(4.1.5)

If

DEFINITION 4.3 Consider a scalar r > 0, and let x = c + Δx be a point in the definition domain D. If f (c) ≤ f (c + Δx),

∀ 0 < Δx 2 ≤ r,

(4.1.6)

then the point c and the function value f (c) are called a local minimum point and a local minimum (value) of the function f (x), respectively. If ∀ 0 < Δx 2 ≤ r,

f (c) < f (c + Δx),

(4.1.7)

then the point c and the function value f (c) are known as a strict local minimum point and a strict local minimum of the function f (x), respectively. If f (c) ≤ f (x),

∀ x ∈ D,

(4.1.8)

then the point c and the function value f (c) are said to be a global minimum point and a global minimum of the function f (x) in the definition domain D, respectively. If f (c) < f (x),

∀ x ∈ D, x = c,

(4.1.9)

then the point c and the function value f (c) are referred to as a strict global minimum point and a strict global minimum of the function f (x) in the definition domain D, respectively.

186

Gradient Analysis and Optimization

DEFINITION 4.4 Consider a scalar r > 0, and let x = c + Δx be a point in the definition domain D. If f (c) ≥ f (c + Δx),

∀ 0 < Δx 2 ≤ r,

(4.1.10)

then the point c and the function value f (c) are called a local maximum point and a local maximum (value) of the function f (x), respectively. If ∀ 0 < Δx 2 ≤ r,

f (c) > f (c + Δx),

(4.1.11)

then the point c and the function value f (c) are known as a strict local maximum point and a strict local maximum of the function f (x), respectively. If f (c) ≥ f (x),

∀ x ∈ D,

(4.1.12)

then the point c and the function value f (c) are said to be a global maximum point and a global maximum of the function f (x) in the definition domain D, respectively. If f (c) > f (x),

∀ x ∈ D, x = c,

(4.1.13)

then the point c and the function value f (c) are referred to as a strict global maximum point and a strict global maximum of the function f (x) in the definition domain D, respectively. It should be emphasized that the minimum points and the maximum points are collectively called the extreme points of the function f (x), and the minimal value and the maximal value are known as the extrema or extreme values of f (x). A local minimum (maximum) point and a strict local minimum (maximum) point are sometimes called a weak local minimum (maximum) point and a strong local minimum (maximum) point, respectively. In particular, if some point x0 is a unique local extreme point of a function f (x) in the neighborhood B(c; r) then it is said to be an isolated local extreme point. In order to facilitate the understanding of the stationary points and other extreme points, Figure 4.1 shows the curve of a one-variable function f (x) with definition domain D = (0, 6). The points x = 1 and x = 4 are a local minimum point and a local maximum point, respectively; x = 2 and x = 3 are a strict local maximum point and a strict local minimum point, respectively; all these points are extrema, while x = 5 is only a saddle point.

4.1.2 Real Gradient Analysis of f (x) In practical applications, it is still very troublesome to compare directly the value of an objective function f (x) at some point with all possible values in its neighborhood. Fortunately, the Taylor series expansion provides a simple method for coping with this difficulty.

4.1 Real Gradient Analysis

187

Figure 4.1 Extreme points of a one-variable function defined over the interval (0, 6).

The second-order Taylor series approximation of f (x) at the point c is given by 1 T f (c + Δx) = f (c) + (∇f (c)) Δx + (Δx)T H[f (c)]Δx, 2 where

(4.1.14)

 ∂f (c) ∂f (x)  , ∇f (c) = = ∂c ∂ x x=c  ∂ 2 f (x)  ∂ 2 f (c) = H[f (c)] = ∂c∂cT ∂ x∂ xT 

(4.1.15) (4.1.16)

x=c

are respectively the gradient vector and the Hessian matrix of f (x) at the point c. From (4.1.14) we get the following two necessary conditions of a local minimizer of f (x). THEOREM 4.1 (First-order necessary conditions) [353] If x∗ is a local extreme point of a function f (x) and f (x) is continuously differentiable in the neighborhood B(x∗ ; r) of the point x∗ , then  ∂f (x)  ∇x∗ f (x∗ ) = = 0. (4.1.17) ∂ x x=x ∗

THEOREM 4.2 (Second-order necessary conditions) [311], [353] If x∗ is a local minimum point of f (x) and the second-order gradient ∇2x f (x) is continuous in an open neighborhood Bo (x∗ ; r) of x∗ , then   ∂f (x)  ∂ 2 f (x)  2 = 0 and ∇x∗ f (x∗ ) =  0, (4.1.18) ∇x∗ f (x∗ ) = ∂ x x=x ∂ x∂ xT x=x ∗



i.e., the gradient vector of f (x) at the point x∗ is a zero vector and the Hessian matrix ∇2x f (x) at the point x∗ is a positive semi-definite matrix. If Equation (4.1.18) in Theorem 4.2 is replaced by   ∂f (x)  ∂ 2 f (x)  2 = 0 and ∇ f (x ) = 0 ∇x∗ f (x∗ ) = ∗ x∗ ∂ x x=x ∂ x∂ xT x=x ∗



(4.1.19)

188

Gradient Analysis and Optimization

then Theorem 4.2 yields the second-order necessary condition for x∗ to be a local maximum point of the function f (x). In Equation (4.1.19), ∇2x∗ f (x∗ )  0 means that the Hessian matrix at the point x∗ , ∇2x∗ f (x∗ ), is negative semi-definite. It should be emphasized that Theorem 4.2 provides only a necessary condition, not a sufficient condition, for there to be a local minimum point of the function f (x). However, for an unconstrained optimization algorithm, we usually hope to determine directly whether its convergence point x∗ is in fact an extreme point of the given objective function f (x). The following theorem gives the answer to this question. THEOREM 4.3 (Second-order sufficient conditions) [311], [353] Let ∇2x f (x) be continuous in an open neighborhood of x∗ . If the conditions   ∂f (x)  ∂ 2 f (x)  2 = 0 and ∇x∗ f (x∗ ) =  0 (4.1.20) ∇x∗ f (x∗ ) = ∂ x x=x ∂ x∂ xT x=x ∗



are satisfied then x∗ is a strict local minimum point of the objection function f (x). Here ∇2x f (x∗ )  0 means that the Hessian matrix ∇2x∗ f (x∗ ) at the point x∗ is a positive definite matrix. Note 1 Second-order necessary conditions for a local maximum point: If x∗ is a local maximum point of f (x), and the second-order gradient ∇2x f (x) is continuous in an open neighborhood Bo (x∗ ; r) of x∗ , then   ∂f (x)  ∂ 2 f (x)  2 = 0 and ∇x∗ f (x∗ ) =  0. (4.1.21) ∇x∗ f (x∗ ) = ∂ x x=x ∂ x∂ xT x=x ∗



Note 2 Second-order sufficient conditions for a strict local maximum point: suppose that ∇2x f (x) is continuous in an open neighborhood of x∗ ; if the conditions   ∂f (x)  ∂ 2 f (x)  2 = 0 and ∇x∗ f (x∗ ) = ≺ 0 (4.1.22) ∇x∗ f (x∗ ) = ∂ x x=x ∂ x∂ xT x=x ∗



are satisfied then x∗ is a strict local maximum point of the objection function f (x). If ∇f (x∗ ) = 0 but ∇2x∗ f (x∗ ) is an indefinite matrix, then x∗ is a saddle point, not an extreme point, of f (x) in the neighborhood B(x∗ ; r).

4.1.3 Real Gradient Analysis of f (X) Now consider a real function f (X) : Rm×n → R with matrix variable X ∈ Rm×n . First, perform the vectorization of the matrix variable to get an mn × 1 vector vecX. Let D be the subset of the matrix space Rm×n which is the definition domain of the m × n matrix variable X, i.e., X ∈ D.

4.1 Real Gradient Analysis

189

DEFINITION 4.5 A neighborhood B(X∗ ; r) with center point vecX∗ and the radius r is defined as B(X∗ ; r) = {X|X ∈ Rm×n , vecX − vecX∗ 2 < r}.

(4.1.23)

From Equation (4.1.16), the second-order Taylor series approximation formula of the function f (X) at the point X∗ is given by T  ∂f (X∗ ) f (X∗ + ΔX) = f (X∗ ) + vec(ΔX) ∂ vecX∗ 1 ∂ 2 f (X∗ ) + (vec(ΔX))T vec(ΔX) 2 ∂ vecX∗ ∂ (vecX∗ )T  T = f (X∗ ) + ∇vec X∗ f (X∗ ) vec(ΔX) 1 + (vec(ΔX))T H[f (X∗ )]vec(ΔX), (4.1.24) 2 where  ∂f (X)  ∈ Rmn ∇vec X∗ f (X∗ ) = ∂ vecX X=X ∗   ∂ 2 f (X)  H[f (X∗ )] = ∈ Rmn×mn T ∂ vecX∂(vec X) X=X ∗

are the gradient vector and the Hessian matrix of f (X) at the point X∗ , respectively. Comparing Equation (4.1.24) with Equation (4.1.14), we have the following conditions similar to Theorems 4.1–4.3. (1) First-order necessary condition for a minimizer If X∗ is a local extreme point of a function f (X), and f (X) is continuously differentiable in the neighborhood B(X∗ ; r) of the point X∗ , then  ∂f (X)  = 0. (4.1.25) ∇vec X∗ f (X∗ ) = ∂ vecX X=X ∗

(2) Second-order necessary conditions for a local minimum point If X∗ is a local minimum point of f (X), and the Hessian matrix H[f (X)] is continuous in an open neighborhood Bo (X∗ ; r) then   ∂ 2 f (X)  ∇vec X∗ f (X∗ ) = 0 and  0. (4.1.26) T ∂ vecX∂ (vecX) X=X ∗

(3) Second-order sufficient conditions for a strict local minimum point It is assumed that H[f (x)] is continuous in an open neighborhood of X∗ . If the conditions   ∂ 2 f (X)  ∇vec X∗ f (X∗ ) = 0 and 0 (4.1.27) ∂ vecX∂ (vecX)T X=X ∗

are satisfied, then X∗ is a strict local minimum point of f (X).

190

Gradient Analysis and Optimization

(4) Second-order necessary conditions for a local maximum point If X∗ is a local maximum point of f (X), and the Hessian matrix H[f (X)] is continuous in an open neighborhood Bo (X∗ ; r), then   ∂ 2 f (X)  ∇vec X∗ f (X∗ ) = 0 and  0. (4.1.28) T ∂ vecX∂ (vecX) X=X ∗

(5) Second-order sufficient conditions for a strict local maximum point It is assumed that H[f (X)] is continuous in an open neighborhood of X∗ . If the conditions   ∂ 2 f (X)  ∇vec X∗ f (X∗ ) = 0 and ≺0 (4.1.29) T ∂ vecX∂ (vecX) X=X ∗

are satisfied then X∗ is a strict local maximum point of f (X). The point X∗ is only a saddle point of f (X), if ∇vec X∗ f (X∗ ) = 0

and

  ∂ 2 f (X)  is indefinite. T ∂ vecX∂(vecX) X=X ∗

4.2 Gradient Analysis of Complex Variable Function In many engineering applications (such as wireless communications, radar and sonar), the signals encountered in practice are real-valued functions of complex vector variables. In this section we discuss the extreme point of a complex variable function and gradient analysis in unconstrained optimization.

4.2.1 Extreme Point of Complex Variable Function Consider the unconstrained optimization of a real-valued function f (z, z∗ ) : Cn × Cn → R. From the first-order differential ∂f (z, z∗ ) ∂f (z, z∗ ) ∗ dz + dz ∂ zT ∂zH     ∂f (z, z∗ ) ∂f (z, z∗ ) dz = , dz∗ ∂ zT ∂zH

df (z, z∗ ) =

(4.2.1)

and the second-order differential ⎡

∂ 2 f (z, z∗ ) ⎢ ∂ z∗ ∂ zT

d2 f (z, z∗ ) = [dzH , dzT ] ⎣

∂ 2 f (z, z∗ ) ∂ z∂ zT



∂ 2 f (z, z∗ )   ∂ z∗ ∂ zH ⎥ dz ∂ 2 f (z, z∗ ) ∂ z∂ zH



dz∗

,

(4.2.2)

4.2 Gradient Analysis of Complex Variable Function

191

it can be easily seen that the Taylor series approximation of the function f (z, z∗ ) at the point c is given by    ∂f (c, c∗ ) ∂f (c, c∗ ) Δc ∗ ∗ f (z, z ) ≈ f (c, c ) + , Δc∗ ∂c ∂c∗ ⎡ 2 ⎤ ∂ f (c, c∗ ) ∂ 2 f (c, c∗ )   ∗ T 1 ∂ c∗ ∂ cH ⎥ Δc H T ⎢ ∂c ∂c + [Δc , Δc ] ⎣ 2 ⎦ Δc∗ 2 ∂ f (c, c∗ ) ∂ 2 f (c, c∗ ) ∂ c∂ cT

∂ c∂ cH

1 = f (c, c∗ ) + (∇f (c, c∗ ))T Δ˜ c + (Δ˜ c, (4.2.3) c)H H(f (c, c∗ ))Δ˜ 2   Δc with Δc = z − c and Δc∗ = z∗ − c∗ ; while the gradient vector where Δ˜ c= Δc∗ is ⎡ ∗ ⎤ ⎢ ∇f (c, c∗ ) = ⎣ and the Hessian matrix is

∂f (z, z ) ⎥ ∂z ∈ C2n , ⎦ ∂f (z, z∗ ) ∂ z∗ x=c



∂ 2 f (z, z∗ ) ⎢ ∂ z∗ ∂ zT

H[f (c, c∗ )] = ⎣

∂ 2 f (z, z∗ ) ∂ z∂ zT

(4.2.4)



∂ 2 f (z, z∗ ) ∂ z∗ ∂ zH ⎥



∂ 2 f (z, z∗ ) ∂ z∂ zH x=c

∈ C2n×2n .

(4.2.5)

Consider the neighborhood of the point c B(c; r) = {x|0 < Δc 2 = x − c 2 ≤ r}, where Δc 2 is small enough so that the second-order term in Equation (4.2.3) can be ignored. This yields the first-order approximation to the function, as follows: f (z, z∗ ) ≈ f (c, c∗ ) + (∇f (c, c∗ ))T Δ˜ c

(for Δc 2 small enough).

(4.2.6)



Clearly, in order that f (c, c ) takes a minimal value or a maximal value, it is required that ∇f (c, c∗ )) = 02n×1 . This condition is equivalent to ∂f (c, c∗ )/∂ c = 0n×1 and ∂f (c, c∗ )/∂ c∗ = 0n×1 , or simply the condition that the conjugate derivative vector of the function f (z, z∗ ) at the point c is a zero vector, namely  ∂f (z, z∗ )  ∂f (c, c∗ ) = = 0n×1 . (4.2.7) ∂ c∗ ∂ z∗ z=c The Taylor series approximation to the real-valued function f (z, z∗ ) at its stationary point c is given to second-order by 1 f (z, z∗ ) ≈ f (c, c∗ ) + (Δ˜ c. (4.2.8) c)H H[f (c, c∗ )]Δ˜ 2 From Equations (4.2.7) and (4.2.8) we have the following conditions on extreme points, similar to Theorems 4.1–4.3.

192

Gradient Analysis and Optimization

(1) First-order necessary condition for a local extreme point If z∗ is a local extreme point of a function f (z, z∗ ), and f (z, z∗ ) is continuously differentiable in the neighborhood B(c; r) of the point c, then  ∂f (z, z∗ )  = 0. (4.2.9) ∇c f (c, c∗ ) =  ∂z z=c (2) Second-order necessary conditions for a local minimum point If c is a local minimum point of f (z, z∗ ), and the Hessian matrix H[f (z, z∗ )] is continuous in an open neighborhood Bo (c; r) of the point c, then ⎤ ⎡ 2 ∂ f (z,z∗ ) ∂ 2 f (z,z∗ ) ∗ T ∗ H ∂f (c, c∗ ) ∂z ∂z ∂z ∂z ⎦ = 0 and H[f (c, c∗ )] = ⎣ 2  0. (4.2.10) ∂ f (z,z∗ ) ∂ 2 f (z,z∗ ) ∂c∗ ∂z∂ zT

∂z∂zH

z=c

(3) Second-order sufficient conditions for a strict local minimum point Suppose that H[f (z, z∗ )] is continuous in an open neighborhood of c. If the conditions ∂f (c, c∗ ) = 0 and ∂c∗

H[f (c, c∗ )]  0

(4.2.11)

are satisfied then c is a strict local minimum point of f (z, z∗ ). (4) Second-order necessary conditions for a local maximum point If c is a local maximum point of f (z, z∗ ), and the Hessian matrix H[f (z, z∗ )] is continuous in an open neighborhood Bo (c; r) of the point c, then ∂f (c, c∗ ) = 0 and ∂c∗

H[f (c, c∗ )]  0.

(4.2.12)

(5) Second-order sufficient conditions for a strict local maximum point Suppose that H[f (z, z∗ )] is continuous in an open neighborhood of c. If the conditions ∂f (c, c∗ ) = 0 and ∂c∗

H[f (c, c∗ )] ≺ 0.

(4.2.13)

are satisfied then c is a strict local maximum point of f (z, z∗ ). ∂f (c, c∗ )

If = 0, but the Hessian matrix H[f (c, c∗ )] is indefinite, then c is only ∂ c∗ a saddle point of f (z, z∗ ), not an extremum. For a real-valued function f (Z, Z∗ ) : Cm×n × Cm×n → R with complex matrix variable Z, it is necessary to perform the vectorization of the matrix variables Z and Z∗ respectively, to get vecZ and vecZ∗ . From the first-order differential T T   ∂f (Z, Z∗ ) ∂f (Z, Z∗ ) df (Z, Z∗ ) = vec(dZ) + vec(dZ∗ ) ∂vec Z ∂vec Z∗    ∂f (Z, Z∗ ) ∂f (Z, Z∗ ) vec(dZ) = (4.2.14) , ∂(vecZ)T ∂(vec Z∗ )T ) vec(dZ)∗

4.2 Gradient Analysis of Complex Variable Function

and the second-order differential ⎡ ∂ 2 f (Z,Z∗ )  H ∂(vec Z∗ )∂(vec Z)T vec(dZ) ⎣ d2 f (Z, Z∗ ) = ∗ 2 vec(dZ ) ∂ f (Z,Z∗ )

193

∂ 2 f (Z,Z∗ ) ∂(vec Z∗ )∂(vec Z∗ )T ∂ 2 f (Z,Z∗ ) ∂(vec Z)∂(vec Z∗ )T )

∂(vec Z)∂(vec Z)T



  ⎦ vec(dZ) vec(dZ∗ ) (4.2.15)

it follows that the second-order series approximation of f (Z, Z∗ ) at the point C is given by ˜ f (Z, Z∗ ) = f (C, C∗ ) + (∇f (C, C∗ ))T vec(ΔC) 1 ˜ ˜ H H[f (C, C∗ )]vec(ΔC), + (vec(ΔC)) 2 where

 ˜ = ΔC

(4.2.16)

   Z−C ΔC = ∈ C2m×n Z ∗ − C∗ ΔC∗ ⎡ ∂f (Z,Z∗ ) ⎤

∇f (C, C∗ ) = ⎣

∂(vec Z) ∂f (Z,Z∗ ) ∂(vec Z∗ )



(4.2.17)

∈ C2mn×1 ,

(4.2.18)

Z=C

and H[f (C, C∗ )] = H[f (Z, Z∗ )]|Z=C ⎡ ∂ 2 f (Z,Z∗ ) =⎣  =

∂(vec Z∗ )∂(vec Z)T

∂ 2 f (Z,Z∗ ) ∂(vec Z∗ )∂(vec Z∗ )T

∂ 2 f (Z,Z∗ ) ∂(vec Z)∂(vec Z)T

∂ 2 f (Z,Z∗ ) ∂(vec Z)∂(vec Z∗ )T

H HZZ

Z∗ Z

H HZZ∗

Z∗ Z∗



∈ C2mn×2mn .

⎤ ⎦ Z=C

(4.2.19)

Z=C

Notice that ∂f (Z, Z∗ ) = 0mn×1 , ∂vec Z ∗ ∂f (Z, Z ) ⇔ = Om×n . ∂Z∗

∇f (C, C∗ ) = 02mn×1 ⇔ ∂f (Z, Z∗ ) = 0mn×1 ∂vec Z∗

On comparing Equation (4.2.16) with Equation (4.2.3), we have the following conditions for the extreme points of f (Z, Z∗ ). (1) First-order necessary condition for a local extreme point If C is a local extreme point of a function f (Z, Z∗ ), and f (Z, Z∗ ) is continuously differentiable in the neighborhood B(C; r) of the point C, then  ∂f (Z, Z∗ )  ∗ = O. (4.2.20) ∇f (C, C ) = 0 or ∂Z∗ Z=C

194

Gradient Analysis and Optimization

(2) Second-order necessary conditions for a local minimum point If C is a local minimum point of f (Z, Z∗ ), and the Hessian matrix H[f (Z, Z∗ )] is continuous in an open neighborhood Bo (C; r) of the point C, then  ∂f (Z, Z∗ )  = O and H[f (C, C∗ )]  0. (4.2.21) ∂Z∗ Z=C (3) Second-order sufficient conditions for a strict local minimum point Suppose that H[f (Z, Z∗ )] is continuous in an open neighborhood of C. If the conditions  ∂f (Z, Z∗ )  = O and H[f (C, C∗ )]  0 (4.2.22) ∂Z∗ Z=C are satisfied, then C is a strict local minimum point of f (Z, Z∗ ). (4) Second-order necessary conditions for a local maximum point If C is a local maximum point of f (Z, Z∗ ), and the Hessian matrix H[f (Z, Z∗ )] is continuous in an open neighborhood Bo (C; r) of the point C, then  ∂f (Z, Z∗ )  = O and H[f (C, C∗ )]  0. (4.2.23) ∂Z∗ Z=C (5) Second-order sufficient conditions for a strict local maximum point Suppose that H[f (Z, Z∗ )] is continuous in an open neighborhood of C, if the conditions  ∂f (Z, Z∗ )  = O and H[f (C, C∗ )] ≺ 0 (4.2.24) ∂Z∗ Z=C are satisfied, then C is a strict local maximum point of f (Z, Z∗ ).  ∂f (Z, Z∗ )  If = O, but the Hessian matrix H[f (C, C∗ )] is indefinite, then C  ∗ ∂Z

Z=C

is only a saddle point of f (Z, Z∗ ). Table 4.1 summarizes the extreme-point conditions for three complex variable functions. The Hessian matrices in Table 4.1 are defined as follows: ⎡ 2 ∗ 2 ∗ ⎤ ∂ f (z, z )

⎢ ∂ z ∗ ∂z H[f (c, c )] = ⎣ 2 ∗ ∗

∂ f (z, z ) ∂z∂z ⎡ 2 ∂ f (z, z∗ ) ⎢ ∂ z∗ ∂ zT

H[f (c, c∗ )] = ⎣

∂ f (z, z ) ∂ z ∗ ∂z ∗ ⎥



∈ C2×2 ,



∈ C2n×2n ,

∂ 2 f (z, z ∗ ) ∂ z ∂z ∗ z=c ⎤ ∂ 2 f (z, z∗ ) ∂ z∗ ∂ zH ⎥

∂ 2 f (z, z∗ ) ∂ 2 f (z, z∗ ) ∂ z∂ zT ∂ z∂ zH z=c ⎡ ⎤ 2 ∗ ∂ f (Z, Z ) ∂ 2 f (Z, Z∗ ) ⎢ ∂(vecZ∗ )∂ (vecZ)T ∂(vecZ∗ )∂(vecZ∗ )T ⎥

H[f (C, C∗ )] = ⎢ ⎣

∂ 2 f (Z, Z∗ ) ∂(vecZ)∂(vecZ)T

⎥ ∈ C2mn×2mn . ⎦ ∂ 2 f (Z, Z∗ ) ∂(vecZ)∂(vecZ∗ )T Z = C

4.2 Gradient Analysis of Complex Variable Function

195

Table 4.1 Extreme point conditions for the complex variable functions f (z, z ∗ ) : C → R f (z, z∗ ) : Cn → R f (Z, Z∗ ) : Cm×n → R    ∂f (z, z ∗ )  ∂f (z, z∗ )  ∂f (Z, Z∗ )  = 0 = 0 =O ∂z ∗ z=c ∂ z∗ z=c ∂Z∗ Z=C

Stationary point Local minimum

H[f (c, c∗ )] 0

H[f (c, c∗ )] 0

H[f (C, C∗ )] 0

Strict local minimum

H[f (c, c∗ )] 0

H[f (c, c∗ )] 0

H[f (C, C∗ )] 0

Local maximum

H[f (c, c∗ )] 0

H[f (c, c∗ )] 0

H[f (C, C∗ )] 0

Strict local maximum H[f (c, c∗ )] ≺ 0

H[f (c, c∗ )] ≺ 0

H[f (C, C∗ )] ≺ 0

H[f (c, c∗ )] indef.

H[f (C, C∗ )] indef.

Saddle point

H[f (c, c∗ )] indef.

4.2.2 Complex Gradient Analysis Given a real-valued objective function f (w, w∗ ) or f (W, W∗ ), the gradient analysis of its unconstrained minimization problem can be summarized as follows. (1) The conjugate gradient matrix determines a closed solution of the minimization problem. (2) The sufficient conditions of local minimum points are determined by the conjugate gradient vector and the Hessian matrix of the objective function. (3) The negative direction of the conjugate gradient vector determines the steepestdescent iterative algorithm for solving the minimization problem. (4) The Hessian matrix gives the Newton algorithm for solving the minimization problem. In the following, we discuss the above gradient analysis. 1. Closed Solution of the Unconstrained Minimization Problem By letting the conjugate gradient vector (or matrix) be equal to a zero vector (or matrix), we can find a closed solution of the given unconstrained minimization problem. EXAMPLE 4.1 For an over-determined matrix equation Az = b, define the error sum of squares, J(z) = Az − b 22 = (Az − b)H (Az − b) = zH AH Az − zH AH b − bH Az + bH b, as the objective function. Let its conjugate gradient vector ∇z∗ J(z) = AH Az −

196

Gradient Analysis and Optimization

AH b be equal to a zero vector. Clearly, if AH A is nonsingular then z = (AH A)−1 AH b.

(4.2.25)

This is the LS solution of the over-determined matrix equation Az = b. EXAMPLE 4.2 For an over-determined matrix equation Az = b, define the loglikelihood function 1 1 l(ˆ z) = C − 2 eH e = C − 2 (b − Aˆ z)H (b − Aˆ z), (4.2.26) σ σ where C is a real constant. Then, the conjugate gradient of the log-likelihood function is given by ∂l(ˆ z) 1 1 z) = = 2 AH b − 2 AH Aˆ z. ∇zˆ∗ l(ˆ ∗ ∂z σ σ By setting ∇zˆ∗ l(ˆ z) = 0, we obtain AH Azopt = AH b, where zopt is the solution for maximizing the log-likelihood function l(ˆ z). Hence, if AH A is nonsingular then zopt = (AH A)−1 AH b.

(4.2.27)

This is the ML solution of the matrix equation Az = b. Clearly, the ML solution and the LS solution are the same for the matrix equation Az = b. 2. Sufficient Condition for Local Minimum Point The sufficient conditions for a strict local minimum point are given by Equation (4.2.11). In particular, for a convex function f (z, z∗ ), any local minimum point z0 is a global minimum point of the function. If the convex function f (z, z∗ ) is ∂f (z, z∗ )  = 0 is a global differentiable then the stationary point z0 such that  ∗ minimum point of f (z, z∗ ).

∂z

z=z0

3. Steepest Descent Direction of Real Objective Function There are two choices in determining a stationary point C of an objective function f (Z, Z∗ ) with a complex matrix as variable:   ∂f (Z, Z∗ )  ∂f (Z, Z∗ )  = O or = Om×n . (4.2.28) m×n  ∂Z ∂Z∗ Z=C Z=C The question is which gradient to select when designing a learning algorithm for an optimization problem. To answer this question, it is necessary to introduce the definition of the curvature direction. DEFINITION 4.6 [158] If H is the Hessian operator acting on a nonlinear function f (x), a vector p is said to be: (1) the direction of positive curvature of the function f , if pH Hp > 0;

4.2 Gradient Analysis of Complex Variable Function

197

(2) the direction of zero curvature of the function f , if pH Hp = 0; (3) the direction of negative curvature of the function f , if pH Hp < 0. The scalar pH Hp is called the curvature of the function f along the direction p. The curvature direction is the direction of the maximum rate of change of the objective function. THEOREM 4.4 [57] Let f (z) be a real-valued function of the complex vector z. By regarding z and z∗ as independent variables, the curvature direction of the real objective function f (z, z∗ ) is given by the conjugate gradient vector ∇z∗ f (z, z∗ ). Theorem 4.4 shows that each component of ∇z∗ f (z, z∗ ), or ∇vec Z∗ f (Z, Z∗ ), gives the rate of change of the objective function f (z, z∗ ), or f (Z, Z∗ ), in this direction: • the conjugate gradient vector ∇z∗ f (z, z∗ ), or ∇vec Z∗ f (Z, Z∗ ), gives the fastest growing direction of the objective function; • the negative gradient vector −∇z∗ f (z, z∗ ), or −∇vec Z∗ f (Z, Z∗ ), provides the steepest decreasing direction of the objective function. Hence, along the negative conjugate gradient direction, the objective function reaches its minimum point at the fastest rate. This optimization algorithm is called the gradient descent algorithm or steepest descent algorithm. As a geometric interpretation of Theorem 4.4, Figure 4.2 shows the gradient and the conjugate gradient of the function f (z) = |z|2 at the points c1 and c2 , where ∂f (z)  , i = 1, 2. ∇c∗i f =  ∗ ∂z

z=ci

jy −∇c1 f −∇c∗ f 1

∇c∗ f 1

c1 ∇c1 f

−∇c∗ f

0

x

2

c2

∇c∗ f 2

Figure 4.2 The gradient and the conjugate gradient of f (z) = |z|2 .

Clearly, only the negative direction of the conjugate gradient −∇z∗ f (z) points to the global minimum z = 0 of the objective function. Hence, in minimization

198

Gradient Analysis and Optimization

problems, the negative direction of the conjugate gradient is used as the updating direction: zk = zk−1 − μ∇z∗ f (z),

μ > 0.

(4.2.29)

That is to say, the correction amount −μ∇z∗ f (z) of alternate solutions in the iterative process is proportional to the negative conjugate gradient of the objective function. Equation (4.2.29) is called the learning algorithm for the alternate solution of the optimization problem. Because the direction of the negative conjugate gradient vector always points in the decreasing direction of an objection function, this type of learning algorithm is called a steepest descent algorithm. The constant μ in a steepest descent algorithm is referred to as the learning step and determines the rate at which the alternate solution converges to the optimization solution. 4. Newton Algorithm for Solving Minimization Problems The conjugate gradient vector contains only first-order information about an objective function. If, further, we adopt the second-order derivative information provided by the Hessian matrix of the objective function, then this promises an optimization algorithm with better performance. An optimization algorithm based on the Hessian matrix is the well-known Newton method. The Newton method is a simple and efficient constrained optimization algorithm, and is widely applied. 4.3 Convex Sets and Convex Function Identification In the above section we discussed unconstrained optimization problems. This section presents constrained optimization theory. The basic idea of solving a constrained optimization problem is to transform it into a unconstrained optimization problem. 4.3.1 Standard Constrained Optimization Problems Consider the standard form of a constrained optimization problem min f0 (x) x

fi (x) ≤ 0, i = 1, . . . , m, Ax = b

(4.3.1)

fi (x) ≤ 0, i = 1, . . . , m, hi (x) = 0, i = 1, . . . , q.

(4.3.2)

subject to

which can be written as min f0 (x) x

subject to

The variable x in constrained optimization problems is called the optimization variable or decision variable and the function f0 (x) is the objective function or cost function, while fi (x) ≤ 0, x ∈ I,

and

hi (x) = 0, x ∈ E,

(4.3.3)

4.3 Convex Sets and Convex Function Identification

199

are known as the inequality constraint and the equality constraint, respectively; I and E are respectively the definition domains of the inequality constraint function and the equality constraint function, i.e., I=

m 1

dom fi

and

E=

i=1

q 1

dom hi .

(4.3.4)

i=1

The inequality constraint and the equality constraint are collectively called explicit constraints. An optimization problem without explicit constraints (i.e., m = q = 0) reduces to an unconstrained optimization problem. The inequality constraint fi (x) ≤ 0, i = 1, . . . , m and the equality constraint hi (x) = 0, i = 1, . . . , q, denote m + q strict requirements or stipulations restricting the possible choices of x. An objective function f0 (x) represents the costs paid by selecting x. Conversely, the negative objective function −f0 (x) can be understood as the values or benefits got by selecting x. Hence, solving the constrained optimization problem (4.3.2) amounts to choosing x given m + q strict requirements such that the cost is minimized or the value is maximized. The optimal solution of the constrained optimization problem (4.3.2) is denoted p , and is defined as the infimum of the objective function f0 (x): p = inf{f0 (x)|fi (x) ≤ 0, i = 1, . . . , m, hi (x) = 0, i = 1, . . . , q}.

(4.3.5)

If p = ∞, then the constrained optimization problem (4.3.2) is said to be infeasible, i.e., no point x can meet the m + q constraint conditions. If p = −∞ then the constrained optimization problem (4.3.2) is lower unbounded. The following are the key steps in solving the constrained optimization (4.3.2). (1) Search for the feasible points of a given constrained optimization problem. (2) Search for the point such that the constrained optimization problem reaches the optimal value. (3) Avoid turning the original constrained optimization into a lower unbounded problem. The point x meeting all the inequality constraints and equality constraints is called a feasible point. The set of all feasible points is called the feasible domain or feasible set, denoted F, and is defined as def

F = I ∩ E = {x|fi (x) ≤ 0, i = 1, . . . , m, hi (x) = 0, i = 1, . . . , q}.

(4.3.6)

The points not in the feasible set are referred to as infeasible points. The intersection set of the definition domain domf0 and the feasible domain F of an objective function, D = domf0 ∩

m 1 i=1

domfi ∩

p 1 i=1

domhi = domf0 ∩ F,

(4.3.7)

200

Gradient Analysis and Optimization

is called the definition domain of a constrained optimization problem. A feasible point x is optimal if f0 (x) = p . It is usually difficult to solve a constrained optimization problem. When the number of decision variables in x is large, it is particularly difficult. This is due to the following reasons [211]. (1) A constrained optimization problem may be occupied by a local optimal solution in its definition domain. (2) It is very difficult to find a feasible point. (3) The stopping criteria in general unconstrained optimization algorithms usually fail in a constrained optimization problem. (4) The convergence rate of a constrained optimization algorithm is usually poor. (5) The size of numerical problem can make a constrained minimization algorithm either completely stop or wander, so that these algorithm cannot converge in the normal way. The above difficulties in solving constrained optimization problems can be overcome by using the convex optimization technique. In essence, convex optimization is the minimization (or maximization) of a convex (or concave) function under the constraint of a convex set. Convex optimization is a fusion of optimization, convex analysis and numerical computation.

4.3.2 Convex Sets and Convex Functions First, we introduce two basic concepts in convex optimization. DEFINITION 4.7 A set S ∈ Rn is convex, if the line connecting any two points x, y ∈ S is also in the set S, namely x, y ∈ S, θ ∈ [0, 1]



θx + (1 − θ)y ∈ S.

(4.3.8)

Figure 4.3 gives a schematic diagram of a convex set and a nonconvex set. b

b a

(a) Convex set

a (b) Nonconvex set

Figure 4.3 Convex set and nonconvex set.

    Many familiar setsare  convex,e.g., the unit ball S = x x 2 ≤ 1 . However, the unit sphere S = x x 2 = 1 is not a convex set because the line connecting two points on the sphere is obviously not itself on the sphere.

4.3 Convex Sets and Convex Function Identification

201

A convex set has the following important properties [344, Theorem 2.2.4]. Letting S1 ⊆ Rn and S2 ⊆ Rm be two convex sets, and A(x) : Rn → Rm be the linear operator such that A(x) = Ax + b, then:    1. The intersection S1 ∩ S2 = x ∈ Rn x ∈ S1 , x ∈ S2 (m = n) is a convex set.    2. The sum of sets S1 + S2 = z = x + yx ∈ S1 , y ∈ S2 (m = n) is a convex set.    3. The direct sum S1 ⊕ S2 = (x, y) ∈ Rn+m x ∈ S1 , y ∈ S2 is a convex set.    4. The conic hull K(S1 ) = z ∈ Rn z = βx, x ∈ S1 , β ≥ 0 is a convex set.    5. The affine image A(S1 ) = y ∈ Rm y = A(x), x ∈ S1 is a convex set.    6. The inverse affine image A−1 (S2 ) = x ∈ Rn x = A−1 (y), y ∈ S2 is a convex set. 7. The following convex hull is a convex set:    conv(S1 , S2 ) = z ∈ Rn z = αx + (1 − α)y, x ∈ S1 , y ∈ S2 ; α ∈ [0, 1] . The most important property of a convex set is an extension of Property 1: the intersection of any number (or even an uncountable number) of convex sets is still a convex set. For instance, the intersection of the unit ball and nonnegative orthant Rn+ , S = {x| x 2 ≤ 1, xi ≥ 0}, retains the convexity. However, the union of two convex sets is not usually convex. For example, S1 = {x| x 2 ≤ 1} and S2 = {x| x − 3 × 1 2 ≤ 1} (where 1 is a vector with all entries equal to 1) are convex sets, but their union S1 ∪ S2 is not convex, because the segment connecting any two points in the two balls is not in S1 ∪ S2 . Given a vector x ∈ Rn and a constant ρ > 0, then    (4.3.9) Bo (x, ρ) = y ∈ Rn  y − x 2 < ρ ,    n Bc (x, ρ) = y ∈ R y − x 2 ≤ ρ , (4.3.10) are called the open ball and the closed ball with center x and radius ρ. A convex set S ⊆ Rn is known as a convex cone if all rays starting at the origin and all segments connecting any two points of these rays are still within the convex set, i.e., x, y ∈ S, λ, μ ≥ 0



λx + μy ∈ S.

(4.3.11)

The nonnegative orthant Rn+ = {x ∈ Rn |x  0} is a convex cone. The set of n positive semi-definite matrices X  0, S+ = {X ∈ Rn×n |X  0}, is a convex cone as well, since the positive combination of any number of positive semi-definite n matrices is still positive semi-definite. Hence, S+ is called the positive semi-definite cone. DEFINITION 4.8 The vector function f (x) : Rn → Rm is known as an affine function if it has the form f (x) = Ax + b.

(4.3.12)

202

Gradient Analysis and Optimization

Similarly, the matrix function F(x) : Rn → Rp×q is called an affine function, if it has the form F(x) = A0 + x1 A1 + · · · + xn An ,

(4.3.13)

where Ai ∈ Rp×q . An affine function sometimes is roughly referred to as a linear function (see below). DEFINITION 4.9 [211] Given n × 1 vectors xi ∈ Rn and real numbers θi ∈ R, then y = θ1 x1 + · · · + θk xk is called: (a) a linear combination for any real numbers θi ;  (b) an affine combination if i θi = 1;  (c) a convex combination if i θi = 1 and all θi ≥ 0; (d) a conic combination if θi ≥ 0, i = 1, . . . , k. An intersection of sets has the following properties [211]. 1. 2. 3. 4.

The The The The

intersection intersection intersection intersection

of of of of

subspaces is also a subspace. affine functions is also an affine function. convex sets is also a convex set. convex cones is also a convex cone.

DEFINITION 4.10 [344] Given a convex set S ∈ Rn and a function f : S → R, then we have the following. • The function f : Rn → R is convex if and only if S = domf is a convex set, and for all vectors x, y ∈ S and each scalar α ∈ (0, 1), the function satisfies the Jensen inequality f (αx + (1 − α)y) ≤ αf (x) + (1 − α)f (y).

(4.3.14)

• The function f (x) is known as a strictly convex function if and only if S = domf is a convex set and, for all vectors x, y ∈ S and each scalar α ∈ (0, 1), the function satisfies the inequality f (αx + (1 − α)y) < αf (x) + (1 − α)f (y).

(4.3.15)

In optimization, it is often required that an objective function is strongly convex. A strongly convex function has the following three equivalent definitions: (1) The function f (x) is strongly convex if [344] f (αx + (1 − α)y) ≤ αf (x) + (1 − α)f (y) −

μ α(1 − α) x − y 22 2

holds for all vectors x, y ∈ S and scalars α ∈ [0, 1] and μ > 0.

(4.3.16)

4.3 Convex Sets and Convex Function Identification

203

(2) The function f (x) is strongly convex if [43] (∇f (x) − ∇f (y))T (x − y) ≥ μ x − y 22 holds for all vectors x, y ∈ S and some scalar μ > 0. (3) The function f (x) is strongly convex if [344] μ f (y) ≥ f (x) + [∇f (x)]T (y − x) + y − x 22 2 for some scalar μ > 0.

(4.3.17)

(4.3.18)

In the above three definitions, the constant μ (> 0) is called the convexity parameter of the strongly convex function f (x). The following is the relationship between convex functions, strictly convex functions and strongly convex functions: strongly convex function ⇒ strictly convex function ⇒ convex function. (4.3.19) DEFINITION 4.11 [460] A function f (x) is quasi-convex if, for all vectors x, y ∈ S and a scalar α ∈ [0, 1], the inequality f (αx + (1 − α)y) ≤ max{f (x), f (y)}

(4.3.20)

holds. A function f (x) is strongly quasi-convex if, for all vectors x, y ∈ S, x = y and the scalar α ∈ (0, 1), the strict inequality f (αx + (1 − α)y) < max{f (x), f (y)}

(4.3.21)

holds. A function f (x) is referred to as strictly quasi-convex if the strict inequality (4.3.21) holds for all vectors x, y ∈ S, f (x) = f (y) and a scalar α ∈ (0, 1). 4.3.3 Convex Function Identification Consider an objective function defined in the convex set S, f (x) : S → R, a natural question to ask is how to determine whether the function is convex. Convexfunction identification methods are divided into first-order gradient and secondorder gradient identification methods. DEFINITION 4.12 [432] Given a convex set S ∈ Rn , then the mapping function F(x) : S → Rn is called: (a) a monotone function in the convex set S if F(x) − F(y), x − y ≥ 0,

∀ x, y ∈ S;

(4.3.22)

(b) a strictly monotone function in the convex set S if F(x) − F(y), x − y > 0,

∀ x, y ∈ S and x = y;

(4.3.23)

(c) a strongly monotone function in the convex set S if F(x) − F(y), x − y ≥ μ x − y 22 ,

∀ x, y ∈ S.

(4.3.24)

204

Gradient Analysis and Optimization

If the gradient vector of a function f (x) is taken as the mapping function, i.e., F(x) = ∇x f (x), then we have the following first-order necessary and sufficient conditions for identifying the convex function. 1. First-Order Necessary and Sufficient Conditions THEOREM 4.5 [432] Let f : S → R be a first differentiable function defined in the convex set S in the n-dimensional vector space Rn , then for all vectors x, y ∈ S, we have f (x) convex



∇x f (x) − ∇x f (y), x − y ≥ 0,

f (x) strictly convex



∇x f (x) − ∇x f (y), x − y > 0, x = y,

f (x) strongly convex



∇x f (x) − ∇x f (y), x − y ≥ μ x − y 22 .

THEOREM 4.6 [53] If the function f : S → R is differentiable in a convex definition domain then f is convex if and only if f (y) ≥ f (x) + ∇x f (x), y − x.

(4.3.25)

2. Second-Order Necessary and Sufficient Conditions THEOREM 4.7 [311] Let f : S → R be a function defined in the convex set S ∈ Rn , and second differentiable, then f (x) is a convex function if and only if the Hessian matrix is positive semi-definite, namely Hx [f (x)] =

∂ 2 f (x)  0, ∂ x∂ xT

∀ x ∈ S.

(4.3.26)

Remark Let f : S → R be a function defined in the convex set S ∈ Rn , and second differentiable; then f (x) is strictly convex if and only if the Hessian matrix is positive definite, namely Hx [f (x)] =

∂ 2 f (x)  0, ∂ x∂ xT

∀ x ∈ S,

(4.3.27)

whereas the sufficient condition for a strict minimum point requires that the Hessian matrix is positive definite at one point c, Equation (4.3.27) requires that the Hessian matrix is positive definite at all points in the convex set S. The following basic properties are useful for identifying the convexity of a given function [211]. 1. The function f : Rn → R is convex, if and only if f˜(t) = f (x0 + th) is convex for t ∈ R and all vectors x0 , h ∈ Rn . 2. If α1 , α2 ≥ 0, and f1 (x) and f2 (x) are convex functions, then f (x) = α1 f1 (x) + α2 f2 (x) is also convex.  3. If p(y) ≥ 0 and q(x, y) is a convex function at the point x ∈ S then p(y)q(x, y)d y is convex at all points x ∈ S.

4.4 Gradient Methods for Smooth Convex Optimization

205

4. If f (x) is a convex function then its affine transformation f (Ax + b) is also convex. It should be pointed out that, apart from the 0 -norm, the vector norms for all other p (p ≥ 1), 1/p  n  p

x p = | xi | , p ≥ 1; x ∞ = max | xi | (4.3.28) i

i=1

are convex functions.

4.4 Gradient Methods for Smooth Convex Optimization Convex optimization is divided into first-order algorithms and second-order algorithms. Focusing upon smoothing functions, this section presents first-order algorithms for smooth convex optimization: the gradient method, the gradient-projection method, the conjugate gradient method and the Nesterov optimal gradient method.

4.4.1 Gradient Method Consider the smooth convex optimization problem of an objective function f : Q → R, described by min f (x),

(4.4.1)

x∈Q

where x ∈ Q ⊂ Rn , and f (x) is smooth and convex. As shown in Figure 4.4, the vector x can be regarded as the input of a black box, the outputs of which are the function f (x) and the gradient function ∇f (x).  x

black box

f (x) ∇f (x)

Figure 4.4 The first-order black-box method.

Let xopt denote the optimal solution of min f (x). First-order black-box optimization uses only f (x) and its gradient vector ∇f (x) for finding the vector y ∈ Q such that f (y) − f (xopt ) ≤ , where is a given accuracy error. The solution y satisfying the above condition is known as the -suboptimal solution of the objective function f (x). The first-order black-box optimization method includes two basic tasks: (1) the design of a first-order iterative optimization algorithm;

206

Gradient Analysis and Optimization

(2) the convergence-rate or complexity analysis of the optimization algorithm. First, we introduce optimization-algorithm design. The descent method is a simplest first-order optimization method whose basic idea for solving the unconstrained minimization problem of a convex function min f (x) is as follows: when Q = Rn , use the optimization sequence xk+1 = xk + μk Δxk ,

k = 1, 2, . . .

(4.4.2)

to search for the optimal point xopt . In the above equation, k = 1, 2, . . . represents the iterative number and μk ≥ 0 is the step size or step length of the kth iteration; the concatenated symbol of Δ and x, Δx, denotes a vector in Rn known as the step direction or search direction of the objective function f (x), while Δxk = xk+1 − xk represents the search direction at the kth iteration. Because the minimization-algorithm design requires the objective function to decrease during the iteration process, i.e., f (xk+1 ) < f (xk ),

∀ xk ∈ dom f,

(4.4.3)

this method is called the descent method. From the first-order approximation expression to the objective function at the point xk f (xk+1 ) ≈ f (xk ) + (∇f (xk ))T Δxk ,

(4.4.4)

(∇f (xk ))T Δxk < 0

(4.4.5)

it follows that if

then f (xk+1 ) < f (xk ). Hence the search direction Δxk such that (∇f (xk ))T Δxk < 0 is called the descent step or descent direction of the objective function f (x) at the kth iteration. Obviously, in order to make (∇f (xk ))T Δxk < 0, we should take Δxk = −∇f (xk ) cos θ,

(4.4.6)

where 0 ≤ θ < π/2 is the acute angle between the descent direction and the negative gradient direction −∇f (xk ). The case θ = 0 implies Δxk = −∇f (xk ). That is to say, the negative gradient direction of the objective function f at the point x is directly taken as the search direction. In this case, the length of the descent step Δxk 2 = ∇f (xk ) 2 takes the maximum value, and thus the descent direction Δxk has the maximal descent step or rate and the corresponding descent algorithm, xk+1 = xk − μk ∇f (xk ), is called the steepest descent method.

k = 1, 2, . . . ,

(4.4.7)

4.4 Gradient Methods for Smooth Convex Optimization

207

The steepest descent method can be explained using the quadratic approximation to the function. The second-order Taylor series expansion is given by 1 f (y) ≈ f (x) + (∇f (x))T (y − x) + (y − x)T ∇2 f (x)(y − x). 2

(4.4.8)

If t−1 I is used to replace the Hessian matrix ∇2 f (x) then f (y) ≈ f (x) + (∇f (x))T (y − x) +

1

y − x 22 . 2t

(4.4.9)

This is the quadratic approximation to the function f (x) at the point y. ∂f (y) It is easily found that the gradient vector ∇f (y) = = ∇f (x) + t−1 (y − x). ∂y

Letting ∇f (y) = 0, the solution is given by y = x − t∇f (x). Then letting y = xk+1 and x = xk , we immediately get the updating formula of the steepest descent algorithm, xk+1 = xk − t∇f (xk ). The steepest descent direction Δx = −∇f (x) uses only first-order gradient information about the objective function f (x). If second-order information, i.e., the Hessian matrix ∇2 f (xk ) of the objective function is used then we may find a better search direction. In this case, the optimal descent direction Δx is the solution minimizing the second-order Taylor approximation function of f (x): " # 1 T T 2 min f (x + Δx) = f (x) + (∇f (x)) Δx + (Δx) ∇ f (x)Δx . (4.4.10) Δx 2 At the optimal point, the gradient vector of the objective function with respect to the parameter vector Δx has to equal zero, i.e., ∂f (x + Δx) = ∇f (x) + ∇2 f (x)Δx = 0 ∂Δx  −1 ∇f (x), ⇔ Δxnt = − ∇2 f (x)

(4.4.11)

where Δxnt is called the Newton step or the Newton descent direction and the corresponding optimization method is the Newton or Newton–Raphson method. Algorithm 4.1 summarizes the gradient descent algorithm and its variants. Algorithm 4.1

Gradient descent algorithm and its variants

given: An initial point x1 ∈ dom f and the allowed error  > 0. Put k = 1. repeat 1. Compute the gradient ∇f (xk ) (and the Hessian matrix ∇2 f (xk )). 2. Choose the descent direction  − ∇f (xk ) (steepest descent method), Δxk = −1  2 ∇f (xk ) (Newton method). − ∇ f (xk ) 3. Choose a step μk > 0, and update xk+1 = xk + μk Δxk . 4. exit if |f (xk+1 ) − f (xk )| ≤ . return k ← k + 1. output: x ← xk .

208

Gradient Analysis and Optimization

According to different choices of the step μk , gradient descent algorithms have the following common variants [344]. (1) Before the gradient algorithm is run, a step sequence {μk }∞ k=0 is chosen, e.g., √ μk = μ (fixed step) or μk = h/ k + 1. (2) Full relaxation Take μk = arg min f (xk − μ∇f (xk )). μ≥0

(3) Goldstein–Armijo rule Find xk+1 = xk − μ∇f (xk ) such that α∇f (xk ), xk − xk+1  ≤ f (xk ) − f (xk+1 ), β∇f (xk ), xk − xk+1  ≥ f (xk ) − f (xk+1 ), where 0 < α < β < 1 are two fixed parameters. Note that the variable vector x is unconstrained in the gradient algorithms, i.e., x ∈ Rn . When choosing a constrained variable vector x ∈ C, where C ⊂ Rn , then the update formula in the gradient algorithm should be replaced by a formula containing the projected gradient: xk+1 = PC (xk − μk ∇f (xk )).

(4.4.12)

This algorithm is called the projected gradient method or the gradient-projection method. In (4.4.12), PC (y) is known as the projection operator and is defined as 1

x − y 22 . 2 x∈C The projection operator can be equivalently expressed as PC (y) = arg min

PC (y) = PC y,

(4.4.13)

(4.4.14)

where PC is the projection matrix onto the subspace C. If C is the column space of the matrix A, then PA = A(AT A)−1 AT .

(4.4.15)

We will discuss the projection matrix in detail in Chapter 9 (on projection analysis). In particular, if C = Rn , i.e., the variable vector x is unconstrained, then the projection operator is equal to the identity matrix, namely PC = I, and thus PRn (y) = Py = y,

∀ y ∈ Rn .

In this case, the projected gradient algorithm simplifies to the actual gradient algorithm. The following are projections of the vector x onto several typical sets [488]. 1. The projection onto the affine set C = {x|Ax = b} with A ∈ Rp×n , rank(A) = p is PC (x) = x + AT (AAT )−1 (b − Ax).

(4.4.16)

If p ' n or AAT = I, then the projection PC (x) gives a low cost result.

4.4 Gradient Methods for Smooth Convex Optimization

209

 2. The projection onto the hyperplane C = {xaT x = b} (where a = 0) is PC (x) = x +

b − aT x a.

a 22

(4.4.17)

3. The projection onto the nonnegative orthant C = Rn+ is PC (x) = (x)+



[(x)+ ]i = max{xi , 0}.

4. The projection onto the half-space C = {x|aT x ≤ b} (where a = 0): + b − aT x a, if aT x > b, x+ a22 PC (x) = x, if aT x ≤ b.

(4.4.18)

(4.4.19)

5. The projection onto the rectangular set C = [a, b] (where ai ≤ xi ≤ bi ) is + ai , PC (x) =

if xi ≤ ai ,

xi ,

if ai ≤ xi ≤ bi ,

bi ,

if xi ≥ bi .

(4.4.20)

6. The projection onto the second-order cones C = {(x, t)| x 2 ≤ t, x ∈ Rn } is ⎧ ⎪ (x, t), if x 2 ≤ t, ⎪ ⎪ ⎪   ⎨ t + x2 x PC (x) = (4.4.21) , if − t < x 2 < t, 2x2 t ⎪ ⎪ ⎪ ⎪ ⎩ (0, 0), if x 2 ≤ −t, x = 0. 7. The projection onto the Euclidean ball C = {x| x 2 ≤ 1} is + 1 x, if x 2 > 1, PC (x) = x2 x, if x 2 ≤ 1.

(4.4.22)

8. The projection onto the 1 -norm ball C = {x| x 1 ≤ 1} is + xi − λ, if xi > λ, PC (x)i =

0,

if − λ ≤ xi ≤ λ,

xi + λ,

if xi < −λ,

(4.4.23)

where λ = 0 if x 1 ≤ 1; otherwise, λ is the solution of the equation n  max{| xi | − λ, 0} = 1. i=1

9. The projection onto the positive semi-definite cones C = Sn+ is n  PC (X) = max{0, λi }qi qTi , where X =

n i=1

i=1

λi qi qTi is the eigenvalue decomposition of X.

(4.4.24)

210

Gradient Analysis and Optimization

4.4.2 Conjugate Gradient Method Consider the iteration solution of the matrix equation Ax = b, where A ∈ Rn×n is a nonsingular matrix. This matrix equation can be equivalently written as x = (I − A)x + b.

(4.4.25)

This inspires the following iteration algorithm: xk+1 = (I − A)xk + b.

(4.4.26)

This algorithm is known as the Richardson iteration and can be rewritten in the more general form xk+1 = Mxk + c,

(4.4.27)

where M is an n × n matrix, called the iteration matrix. An iteration with the form of Equation (4.4.27) is known as a stationary iterative method and is not as effective as a nonstationary iterative method. Nonstationary iteration methods comprise a class of iteration methods in which xk+1 is related to the former iteration solutions xk , xk−1 , . . . , x0 . The most typical nonstationary iteration method is the Krylov subspace method, described by xk+1 = x0 + Kk ,

(4.4.28)

Kk = Span(r0 , Ar0 , . . . , Ak−1 r0 )

(4.4.29)

where

is called the kth iteration Krylov subspace; x0 is the initial iteration value and r0 denotes the initial residual vector. Krylov subspace methods have various forms, of which the three most common are the conjugate gradient method, the biconjugate gradient method and the preconditioned conjugate gradient method. 1. Conjugate Gradient Algorithm The conjugate gradient method uses r0 = Ax0 − b as the initial residual vector. The applicable object of the conjugate gradient method is limited to the symmetric positive definite equation, Ax = b, where A is an n × n symmetric positive definite matrix. The nonzero vector combination {p0 , p1 , . . . , pk } is called A-orthogonal or Aconjugate if pTi Apj = 0,

∀ i = j.

(4.4.30)

This property is known as A-orthogonality or A-conjugacy. Obviously, if A = I then the A-conjugacy reduces to general orthogonality. All algorithms adopting the conjugate vector as the update direction are known as conjugate direction algorithms. If the conjugate vectors p0 , p1 , . . . , pn−1 are not

4.4 Gradient Methods for Smooth Convex Optimization

211

predetermined, but are updated by the gradient descent method in the updating process, then we say that the minimization algorithm for the objective function f (x) is a conjugate gradient algorithm. Algorithm 4.2 gives a conjugate gradient algorithm. Algorithm 4.2

Conjugate gradient algorithm [179], [243]

input: A = AT ∈ Rn×n , b ∈ Rn , the largest iteration step number kmax and the allowed error . initialization: Choose x0 ∈ Rn , and let r = Ax0 − b and ρ0 = r22 . repeat 1. If k = 1 then p = r. Otherwise, let β = ρk−1 /ρk−2 and p = r + βp. 2. w = Ap. 3. α = ρk−1 /(pT w). 4. x = x + αp. 5. r = r − αw. 6. ρk = r22 . √ 7. exit if ρk < b2 or k = kmax . return k ← k + 1. output: x ← xk .

From Algorithm 4.2 it can be seen that in the iteration process of the conjugate gradient algorithm, the solution of the matrix equation Ax = b is given by xk =

k  i=1

αi pi =

k  ri−1 , ri−1  i=1

pi , Api 

pi ,

(4.4.31)

that is, xk belongs to the kth Krylov subspace xk ∈ Span{p1 , p2 , . . . , pk } = Span{r0 , Ar0 , . . . , Ak−1 r0 }. The fixed-iteration method requires the updating of the iteration matrix M, but no matrix needs to be updated in Algorithm 4.2. Hence, the Krylov subspace method is also called the matrix-free method [243]. 2. Biconjugate Gradient Algorithm If the matrix A is not a real symmetric matrix, then we can use the biconjugate gradient method of Fletcher [156] for solving the matrix equation Ax = b. As the ¯ , that are A-conjugate in name implies, there are two search directions, p and p this method: ¯ Ti Apj = pTi A¯ p pj = 0,

i = j,

¯rTi rj = rTi ¯rTj ¯ Tj ¯rTi pj = rTi p

= 0,

i = j,

= 0,

j < i.

(4.4.32)

212

Gradient Analysis and Optimization

Algorithm 4.3 shows a biconjugate gradient algorithm. Algorithm 4.3

Biconjugate gradient algorithm [156], [230]

¯1 = ¯ initialization: p1 = r1 , p r1 . repeat 1. αk = ¯ rTk rk /(¯ pTk Apk ). 2. rk+1 = rk − αk Apk . ¯k. 3. ¯rk+1 = ¯rk − αk AT p rTk+1 rk+1 /(¯rTk rk ). 4. βk = ¯ 5. pk+1 = rk+1 + βk pk . ¯ k+1 = ¯rk+1 + βk p ¯k. 6. p 7. exit if k = kmax . return k ← k + 1. ¯ k+1 ← x ¯ k + αk p ¯k. output: x ← xk + αk pk , x

3. Preconditioned Conjugate Gradient Algorithm Consider the symmetric indefinite saddle point problem      x f A BT = , B O q g where A is an n × n real symmetric positive definite matrix, B is an m × n real matrix with full row rank m (≤ n), and O is an m × m zero matrix. Bramble and Pasciak [56] developed a preconditioned conjugate gradient iteration method for solving the above problems. The basic idea of the preconditioned conjugate gradient iteration is as follows: through a clever choice of the scalar product form, the preconditioned saddle matrix becomes a symmetric positive definite matrix. To simplify the discussion, it is assumed that the matrix equation with large condition number Ax = b needs to be converted into a new symmetric positive definite equation. To this end, let M be a symmetric positive definite matrix that can approximate the matrix A and for which it is easier to find the inverse matrix. Hence, the original matrix equation Ax = b is converted into M−1 Ax = M−1 b such that two matrix equations have the same solution. However, in the new matrix equation M−1 Ax = M−1 b there exists a hidden danger: M−1 A is generally not either symmetric or positive definite even if both M and A are symmetric positive definite. Therefore, it is unreliable to use the matrix M−1 directly as the preprocessor of the matrix equation Ax = b. Let S be the square root of a symmetric matrix M, i.e., M = SST , where S is symmetric positive definite. Now, use S−1 instead of M−1 as the preprocessor to

4.4 Gradient Methods for Smooth Convex Optimization

213

ˆ, convert the original matrix equation Ax = b to S−1 Ax = S−1 b. If x = S−T x then the preconditioned matrix equation is given by ˆ = S−1 b. S−1 AS−T x

(4.4.33)

Compared with the matrix M−1 A, which is not symmetric positive definite, S AS−T must be symmetric positive definite if A is symmetric positive definite. The symmetry of S−1 AS−T is easily seen, its positive definiteness can be verified as follows: by checking the quadratic function it easily follows that yT (S−1 AS−T )y = zT Az, where z = S−T y. Because A is symmetric positive definite, we have zT Az > 0, ∀ z = 0, and thus yT (S−1 AS−T )y > 0, ∀ y = 0. That is to say, S−1 AS−T must be positive definite. At this point, the conjugate gradient method can be applied to solve the matrix ˆ , and then we can recover x via x = S−T x ˆ. equation (4.4.33) in order to get x Algorithm 4.4 shows a preconditioned conjugate gradient (PCG) algorithm with a preprocessor, developed in [440]. −1

Algorithm 4.4

PCG algorithm with preprocessor

input: A, b, preprocessor S−1 , maximal number of iterations kmax , and allowed error  < 1. initialization: k = 0, r0 = Ax − b, d0 = S−1 r0 , δnew = rT0 d0 , δ0 = δnew . repeat 1. qk+1 = Adk . 2. α = δnew /(dTk qk+1 ). 3. xk+1 = xk + αdk . 4. If k can be divided exactly by 50, then rk+1 = b − Axk . Otherwise, update rk+1 = rk − αqk+1 . 5. sk+1 = S−1 rk+1 . 6. δold = δnew . 7. δnew = rTk+1 sk+1 . 8. exit if k = kmax or δnew < 2 δ0 . 9. β = δnew /δold . 10. dk+1 = sk+1 + βdk . return k ← k + 1. output: x ← xk .

The use of a preprocessor can be avoided, because there are correspondences ˆ = S−1 b, as between the variables of the matrix equations Ax = b and S−1 AS−T x follows [243]: ˆk, xk = S−1 x

rk = Sˆrk ,

ˆk, pk = S−1 p

zk = S−1 ˆrk .

On the basis of these correspondences, it easy to develop a PCG algorithm without preprocessor [243], as shown in Algorithm 4.5.

214 Algorithm 4.5

Gradient Analysis and Optimization PCG algorithm without preprocessor

input: A = AT ∈ Rn×n , b ∈ Rn , maximal iteration number kmax , and allowed error . initialization: x0 ∈ Rn , r0 = Ax0 − b, ρ0 = r22 and M = SST . Put k = 1. repeat 1. zk = Mrk−1 . 2. τk−1 = zTk rk−1 . 3. If k = 1, then β = 0, p1 = z1 . Otherwise, β = τk−1 /τk−2 , pk ← zk + βpk . 4. wk = Apk . 5. α = τk−1 /(pTk wk ). 6. xk+1 = xk + αpk . 7. rk = rk−1 − αwk . 8. ρk = rTk rk . √ 9. exit if ρk < b2 or k = kmax . return k ← k + 1. output: x ← xk .

A wonderful introduction to the conjugate gradient method is presented in [440]. Regarding the complex matrix equation Ax = b, where A ∈ Cn×n , x ∈ Cn , b ∈ n C , we can write it as the following real matrix equation:      b AR −AI xR = R . (4.4.34) A I AR xI bI If A = AR + jAI is a Hermitian positive definite matrix, then Equation (4.4.34) is a symmetric positive definite matrix equation. Hence, (xR , xI ) can be solved by adopting the conjugate gradient algorithm or the preconditioned conjugate gradient algorithm. The preconditioned conjugate gradient method is a widely available technique for solving partial differential equations, and it has important applications in optimal control [208]. As a matter of fact, it is usually necessary to apply the preconditioned conjugate gradient algorithm to solve the KKT equation or the Newton equation in optimization problems in order to improve the numerical stability of solving the optimization search direction procedure. In addition to the three conjugate gradient algorithms described above, there is a projected conjugate gradient algorithm [179]. This algorithm requires the use of a seed space Kk (A, r0 ) to solve the matrix equation Axq = bq , q = 1, 2, . . . The main advantage of gradient methods is that the computation of each iteration is very simple; however, their convergence is slow. Hence, an important question is how to accelerate a gradient method so as to get an optimal gradient method. To this end, it is necessary first to analyze the convergence rate of the gradient methods.

4.4 Gradient Methods for Smooth Convex Optimization

215

4.4.3 Convergence Rates By the convergence rate is meant the number of iterations needed for an optimization algorithm to make the estimated error of the objective function achieve the required accuracy, or given an iteration number K, the accuracy that an optimization algorithm reaches. The inverse of the convergence rate is known as the complexity of an optimization algorithm. Let x be a local or global minimum point. The estimation error of an optimization algorithm is defined as the difference between the value of an objective function at iteration point xk and its value at the global minimum point, i.e., δk = f (xk ) − f (x ). We are naturally interested in the convergence problems of an optimization algorithm: (1) Given an iteration number K, what is the designed accuracy

lim δk ?

1≤k≤K

(2) Given an allowed accuracy , how many iterations does the algorithm need to achieve the designed accuracy min δk ≤ ? k

When analyzing the convergence problems of optimization algorithms, we often focus on the speed of the updating sequence {xk } at which the objective function argument converges to its ideal minimum point x . In numerical analysis, the speed at which a sequence reaches its limit is called the convergence rate. 1. Q-Convergence Rates Assume that a sequence {xk } converges to x . If there are a real number α ≥ 1 and a positive constant μ independent of the iterations k such that

xk+1 − x 2 k→∞ xk − x α 2

μ = lim

(4.4.35)

then we say that {xk } has an α-order Q-convergence rate. The Q-convergence rate means the quotient convergence rate. It has the following typical values [360]: 1. When α = 1, the Q-convergence rate is called the limit-convergence rate of {xk }:

xk+1 − x 2 . k→∞ xk − x 2

μ = lim

(4.4.36)

According to the value of μ, the limit-convergence rate of the sequence {xk } can be divided into three types: (1) sublinear convergence rate, α = 1, μ = 1; (2) linear convergence rate, α = 1, μ ∈ (0, 1); (3) superlinear convergence rate, α = 1, μ = 0 or 1 < α < 2, μ = 0. 2. When α = 2, we say that the sequence {xk } has a quadratic convergence rate.

216

Gradient Analysis and Optimization

3. When α = 3, we say that the sequence {xk } has a cubic convergence rate. If {xk } is sublinearly convergent, and lim

k→∞

xk+2 − xk+1 2 = 1,

xk+1 − xk 2

then the sequence {xk } is said to have a logarithmic convergence rate. Sublinear convergence comprises a class of slow convergence rates; the linear convergence comprises a class of fast convergence; and the superlinear convergence and quadratic convergence comprise classes of respectively very fast and extremely fast convergence. When designing an optimization algorithm, we often require it at least to be linearly convergent, preferably to be quadratically convergent. The ultra-fast cubic convergence rate is in general difficult to achieve. 2. Local Convergence Rates The Q-convergence rate is a limited convergence rate. When we evaluate an optimization algorithm, a practical question is: how many iterations does it need to achieve the desired accuracy? The answer depends on the local convergence rate of the output objective function sequence of a given optimization algorithm. The local convergence rate of the sequence {xk } is denoted rk and is defined by 2 2 2 xk+1 − x 2 2 2 . rk = 2 (4.4.37) x k − x 2 2 The complexity of an optimization algorithm is defined as the inverse of the local convergence rate of the updating variable. The following is a classification of local convergence rates [344]. (1) Sublinear rate This convergence rate is given by a power function of the iteration number k and is usually denoted as   1 . (4.4.38) f (xk ) − f (x ) ≤ = O √ k (2) Linear rate This convergence rate is expressed as an exponential function of the iteration number k, and is usually defined as   1 f (xk ) − f (x ) ≤ = O . (4.4.39) k (3) Quadratic rate This convergence rate is a bi-exponential function of the iteration number k and is usually measured by   1 f (xk ) − f (x ) ≤ = O . (4.4.40) k2

4.5 Nesterov Optimal Gradient Method

217

For example, in order to achieve the approximation accuracy f (xk ) − f (x ) ≤ = 0.0001, optimization algorithms with sublinear, linear and quadratic rates need to run about 108 , 104 and 100 iterations, respectively. THEOREM 4.8 [344] Let = f (xk ) − f (x ) be the estimation error of the objective function given by the updating sequence of the gradient method xk+1 = xk − α∇f (xk ). For a convex function f (x), the upper bound of the estimation error = f (xk ) − f (x ) is given by f (xk ) − f (x ) ≤

2L x0 − x 22 . k+4

(4.4.41)

This theorem shows that the local convergence rate of the gradient method xk+1 = xk − α∇f (xk ) is the linear rate O(1/k). Although this is a fast convergence rate, it is not by far optimal, as will be seen in the next section.

4.5 Nesterov Optimal Gradient Method Let Q ⊂ R be a convex set in the vector space Rn . Consider the unconstrained optimization problem min f (x). n

x∈Q

4.5.1 Lipschitz Continuous Function DEFINITION 4.13 [344] We say that an objective function f (x) is Lipschitz continuous in the definition domain Q if |f (x) − f (y)| ≤ L x − y 2 ,

∀ x, y ∈ Q

(4.5.1)

holds for some Lipschitz constant L > 0. Similarly, we say that the gradient vector ∇f (x) of a differentiable function f (x) is Lipschitz continuous in the definition domain Q if

∇f (x) − ∇f (y) 2 ≤ L x − y 2 ,

∀ x, y ∈ Q

(4.5.2)

holds for some Lipschitz constant L > 0. Hereafter a Lipschitz continuous function with the Lipschitz constant L will be denoted as an L-Lipschitz continuous function. The function f (x) is said to be continuous at the point x0 , if limx→x0 f (x) = f (x0 ). When we say that f (x) is a continuous function, this means the f (x) is continuous at every point x ∈ Q. A Lipschitz continuous function f (x) must be a continuous function, but a continuous function is not necessarily a Lipschitz continuous function.

218

Gradient Analysis and Optimization

√ EXAMPLE 4.3 The function f (x) = 1/ x is not a Lipschitz continuous function in the open interval (0, 1). If it is assumed func  continuous   to be a Lipschitz 1  1 1  1 tion, then it must satisfy |f (x1 ) − f (x2 )| =  √ − √  ≤ L − , namely x1 x2 x1 x2   1   1 2  √ + √  ≤ L. However, when the point x → 0, and x1 = 1/n , x2 = 9/n2 , we x1 x2 √ have L ≥ n/4 → ∞. Hence, the continuous function f (x) = 1/ x is not a Lipschitz continuous function in the open interval (0, 1). An everywhere differentiable function is called a smooth function. A smooth function must be continuous, but a continuous function is not necessarily smooth. A typical example occurs when the continuous function has “sharp” point at which it is nondifferentiable. Hence, a Lipschitz continuous function is not necessarily differentiable, but a function f (x) with Lipschitz continuous gradient in the definition domain Q must be smooth in Q, because the definition of a Lipschitz continuous gradient stipulates that f (x) is differentiable in the definition domain. In convex optimization, the notation CLk,p (Q) (where Q ⊆ Rn ) denotes the Lipschitz continuous function class with the following properties [344]: • The function f ∈ CLk,p (Q) is k times continuously differentiable in Q. • The pth-order derivative of the function f ∈ CLk,p (Q) is L-Lipschitz continuous:

f (p) (x) − f (p) (y) ≤ L x − y 2 ,

∀ x, y ∈ Q.

CLk,p (Q)

is said to be a differentiable function. Obviously, it If k = 0, then f ∈ always has p ≤ k. If q > k then CLq,p (Q) ⊆ CLk,p (Q). For example, CL2,1 (Q) ⊆ CL1,1 (Q). The following are three common function classes CLk,p (Q): (1) f (x) ∈ CL0,0 (Q) is L-Lipschitz continuous but nondifferentiable in Q; (2) f (x) ∈ CL1,0 (Q) is L-Lipschitz continuously differentiable in Q, but its gradient is not; (3) f (x) ∈ CL1,1 (Q) is L-Lipschitz continuously differentiable in Q, and its gradient ∇f (x) is L-Lipschitz continuous in Q. The basic property of the CLk,p (Q) function class is that if f1 ∈ CLk,p (Q), f2 ∈ 1 k,p CL2 (Q) and α, β ∈ R then (Q), αf1 + βf2 ∈ CLk,p 3

where L3 = |α| L1 + |β| L2 . Among all Lipschitz continuous functions, CL1,1 (Q), with the Lipschitz continuous gradient, is the most important function class, and is widely applied in convex optimization. On the CL1,1 (Q) function class, one has the following two lemmas [344]. LEMMA 4.1

The function f (x) belongs to CL2,1 (Rn ) if and only if

f

(x) F ≤ L,

∀ x ∈ Rn .

(4.5.3)

4.5 Nesterov Optimal Gradient Method

LEMMA 4.2

219

If f (x) ∈ CL1,1 (Q) then

|f (y) − f (x) − ∇f (x), y − x| ≤

L

y − x 22 , 2

∀ x, y ∈ Q.

(4.5.4)

Since the function CL2,1 must also be a CL1,1 function, Lemma 4.1 directly provides a simple way to determine whether a function belongs to the CL1,1 function class. Below we give some examples of the applications of Lemma 4.1 to determine the CL1,1 (Q) function class. EXAMPLE 4.4 The linear function f (x) = a, x + b belongs to the C01,1 function class; namely, the gradient of any linear function is not Lipschitz continuous because f (x) = a,

f

(x) = O



f

(x) 2 = 0.

EXAMPLE 4.5 The quadratic function f (x) = 12 xT Ax + aT x + b belongs to the 1,1 C A (Rn ) function class, because F

∇f (x) = Ax + a,

∇2 f (x) = A



f

(x) 2 . = A F .

1,1 EXAMPLE 4.6 The logarithmic function f (x) = ln(1 + ex ) belongs to the C1/2 (R) function class, because   1 + e2x  1 ex ex 1 



≤ , 1 − f (x) = , f (x) = ⇒ |f (x)| = 1 + ex (1 + ex )2 2 (1 + ex )2  2

where, in order to find f

(x), we let y = f (x), yielding y = f

(x) = y(ln y) . √ EXAMPLE 4.7 The function f (x) = 1 + x2 belongs to the C11,1 (R) function class, because f (x) = √

x , 1 + x2

f

(x) =

1 (1 + x2 )3/2



|f

(x)| ≤ 1.

Lemma 4.2 is a key inequality for analyzing the convergence rate of a CL1,1 function f (x) in gradient algorithms. By Lemma 4.2, it follows [536] that L − xk 22

x 2 k+1 L L ≤ f (xk ) + ∇f (xk ), x − xk  + x − xk 22 − x − xk+1 22 2 2 L L 2 2 ≤ f (x) + x − xk 2 − x − xk+1 2 . 2 2

f (xk+1 ) ≤ f (xk ) + ∇f (xk ), xk+1 − xk  +

Put x = x and δk = f (xk ) − f (x ); then L L

x − xk+1 22 ≤ −δk+1 + x − xk 22 2 2 k+1  L ≤ ··· ≤ − δi + x − x0 22 . 2 i=1

0≤

220

Gradient Analysis and Optimization

From the estimation errors of the projected gradient algorithm δ1 ≥ δ2 ≥ · · · ≥ δk+1 , it follows that −(δ1 +· · ·+δk+1 ) ≤ −(k+1)δk+1 , and thus the above inequality can be simplified to 0≤

L L

x − xk+1 22 ≤ −(k + 1)δk + x − x0 22 , 2 2

from which the upper bound of the convergence rate of the projected gradient algorithm is given by [536] δk = f (xk ) − f (x ) ≤

L x − x0 22 . 2(k + 1)

(4.5.5)

This shows that, as for the (basic) gradient method, the local convergence rate of the projected gradient method is O(1/k).

4.5.2 Nesterov Optimal Gradient Algorithms THEOREM 4.9 [343] Let f (x) be a convex function with L-Lipschitz gradient, if the updating sequence {xk } meets the condition xk ∈ x0 + Span{x0 , . . . , xk−1 } then the lower bound of the estimation error = f (xk ) − f (x ) achieved by any first-order optimization method is given by f (xk ) − f (x ) ≥

3L x0 − x 22 , 32(k + 1)2

(4.5.6)

where Span{u0 , . . . , uk−1 } denotes the linear subspace spanned by u0 , . . . , uk−1 , x0 is the initial value of the gradient method and f (x ) denotes the minimal value of the function f . Theorem 4.9 shows that the optimal convergence rate of any first-order optimization method is the quadratic rate O(1/k 2 ). Since the convergence rate of gradient methods is the linear rate O(1/k), and the convergence rate of the optimal first-order optimization methods is the quadratic rate O(1/k 2 ), the gradient methods are far from optimal. The heavy ball method (HBM) can efficiently improve the convergence rate of the gradient methods. The HBM is a two-step method: let y0 and x0 be two initial vectors and let αk and βk be two positive valued sequences; then the first-order method for solving an unconstrained minimization problem minn f (x) can use two-step updates [396]: x∈R

yk = βk yk−1 − ∇f (xk ), xk+1 = xk + αk yk .

0 (4.5.7)

4.5 Nesterov Optimal Gradient Method

221

In particular, letting y0 = 0, the above two-step updates can be rewritten as the one-step update xk+1 = xk − αk ∇f (xk ) + βk (xk − xk−1 ),

(4.5.8)

where xk − xk−1 is called the momentum. As can be seen in (4.5.7), the HBM treats the iterations as a point mass with momentum xk − xk−1 , which thus tends to continue moving in the direction xk − xk−1 . Let yk = xk + βk (xk − xk−1 ), and use ∇f (yk ) instead of ∇f (xk ). Then the update equation (4.5.7) becomes 0 yk = xk + βk (xk − xk−1 ), xk+1 = yk − αk ∇f (yk ). This is the basic form of the optimal gradient method proposed by Nesterov in 1983 [343], usually called the Nesterov (first) optimal gradient algorithm and shown in Algorithm 4.6. Algorithm 4.6

Nesterov (first) optimal gradient algorithm [343]

initialization: Choose x−1 = 0, x0 ∈ Rn and α0 ∈ (0, 1). Set k = 0, q = μ/L. repeat 1. Compute αk+1 ∈ (0, 1) from the equation α2k+1 = (1 − αk+1 )α2k + qαk+1 . α (1 − αk ) 2. Set βk = k2 . αk + αk+1 3. Compute yk = xk + βk (xk − xk−1 ). 4. Compute xk+1 = yk − αk+1 ∇f (yk ). 5. exit if xk converges. return k ← k + 1. output: x ← xk .

It is easily seen that the Nesterov optimal gradient algorithm is intuitively like the heavy ball formula (4.5.7) but it is not identical; i.e., it uses ∇f (y) instead of ∇f (x). In the Nesterov optimal gradient algorithm, the estimation sequence {xk } is an approximation solution sequence, and {yk } is a searching point sequence. THEOREM 4.10 [343] Let f be a convex function with L-Lipschitz gradient. The Nesterov optimal gradient algorithm achieves f (xk ) − f (x∗ ) ≤

CL xk − x∗ 22 . (k + 1)2

(4.5.9)

222

Gradient Analysis and Optimization

Clearly, the convergence rate of the Nesterov optimal gradient method is the optimal rate for the first-order gradient methods up to constants. Hence, the Nesterov gradient method is indeed an optimal first-order minimization method. It has been shown [180] that the use of a strong convexity constant μ∗ is very effective in reducing the number of iterations of the Nesterov algorithm. A Nesterov algorithm with adaptive convexity parameter, using a decreasing sequence μk → μ∗ , was proposed in [180]. It is shown in Algorithm 4.7. Algorithm 4.7

Nesterov algorithm with adaptive convexity parameter

input: Lipschitz constant L, and the convexity parameter guess μ∗ ≤ L. given: x0 , v0 = x0 , γ0 > 0, β > 1, μ0 ∈ [μ∗ , γ0 ). Set k = 0, θ ∈ [0, 1], μ+ = μ0 . repeat 1. dk = vk − xk . 2. yk = xk + θk dk . 3. exit if ∇f (yk ) = 0. 4. Steepest descent step: xk+1 = yk − ν∇f (yk ). If L is known, then ν ≥ 1/L. 5. If (γk − μ∗ ) < β(μ+ − μ∗ ), then choose μ+ ∈ [μ∗ , γ/β]. ∇f (yk )2 . 6. Compute μ ˜= 2[f (yk ) − f (xk+1 )]

7. If μ+ ≥ μ ˜, then μ+ = max{μ∗ , μ ˜/10}.

8. Compute αk as the largest root of Aα2 + Bα + C = 0. 9. γk+1 = (1 − αk )γk + αk μ+ . 1 ((1 − αk )γk vk + αk (μ+ yk − ∇f (yk ))). 10. vk+1 = γk+1 11. return k ← k + 1. output: yk as an optimal solution.

The coefficients A, B, C in step 8 are given by )μ * G = γk

vk − yk 2 + (∇f (yk ))T (vk − yk ) , 2 1 A = G + ∇f (yk ) 2 + (μ − γk )(f (xk ) − f (yk )), 2   B = (μ − γk ) f (xk+1 ) − f (xk ) − γk (f (yk ) − f (xk )) − G,   C = γk f (xk+1 ) − f (xk ) , with μ = μ+ It is suggested that γ0 = L if L is known, μ0 = max{μ∗ , γ0 /100} and β = 1.02. The Nesterov (first) optimal gradient algorithm has two limitations: (1) yk may be out of the definition domain Q, and thus the objective function f (x) is required to be well-defined at every point in Q;

4.6 Nonsmooth Convex Optimization

223

(2) this algorithm is available only for the minimization of the Euclidean norm

x 2 . To cope with these limitations, Nesterov developed two other optimal gradient algorithms [346], [345]. Denote by d(x) a prox-function of the domain Q. It is assumed that d(x) is continuous and strongly convex on Q. Depending on whether

x 1 or x 2 is being minimized, there are different choices for the prox-function [345]: n n   |xi |, d(x) = ln n + xi ln |xi |, (4.5.10) using x 1 = i=1

using x 2 =

 n

i=1

1/2 x2i

,

d(x) =

i=1

2 n  1 1 xi − . 2 i=1 n

Define the ith element of the n × 1 mapping vector VQ (u, g) as [345]  −1 n (i) −gi −gi uj e , i = 1, . . . , n. VQ (u, g) = ui e

(4.5.11)

(4.5.12)

j=1

Algorithm 4.8 shows the Nesterov third optimal gradient algorithm. Algorithm 4.8

Nesterov (third) optimal gradient algorithm [345]

given: Choose x0 ∈ Rn and α ∈ (0, 1). Set k = 0.   initialization: y0 = argminx L d(x) + 12 (f (x0 ) + f  (x0 ), x − x0 ) : x ∈ Q . α repeat    i+1 L d(x) + α 2 i=0 k

1. Find zk = argminx 2 k+3

(f (xi ) + ∇f (xi ), x − xi ) : x ∈ Q .

and xk+1 = τk zk + (1 − τk )yk .

α ˆ k+1 (i) = VQ(i) zk , τk ∇f (xk+1 ) , i = 1, . . . , n. 3. Use (4.5.12) to compute x

2. Set τk =

L

ˆ k+1 + (1 − τk )yk . 4. Set yk+1 = τk x 5. exit if xk converges. return k ← k + 1. output: x ← xk .

4.6 Nonsmooth Convex Optimization The gradient methods require that for the objective function f (x), the gradient ∇f (x) exists at the point x; and the Nesterov optimal gradient method requires, further, that the objective function has L-Lipschitz continuous gradient. Hence, the gradient methods in general and the Nesterov optimal gradient method in particular are available only for smooth convex optimization. As an important extension

224

Gradient Analysis and Optimization

of gradient methods, this section focuses upon the proximal gradient method which is available for nonsmooth convex optimization. The following are two typical examples of nonsmooth convex minimization. (1) Basis pursuit de-noising seeks a sparse near-solution to an under-determined matrix equation [95] " # λ 2 min x 1 + Ax − b 2 . (4.6.1) x∈Rn 2 This problem is essentially equivalent to the Lasso (sparse least squares) problem [462] λ minn Ax − b 22 subject to x 1 ≤ K. (4.6.2) x∈R 2 (2) Robust principal component analysis approximates an input data matrix D to a sum of a low-rank matrix L and a sparse matrix S: 3 4 γ min L ∗ + λ S 1 + L + S − D 2F . (4.6.3) 2 The key challenge of the above minimization problems is the nonsmoothness induced by the nondifferentiable 1 norm · 1 and nuclear norm · ∗ . Consider the following combinatorial optimization problem min {F (x) = f (x) + h(x)} , x∈E

(4.6.4)

where E ⊂ Rn is a finite-dimensional real vector space and h : E → R is a convex function, but is nondifferentiable or nonsmooth in E. f : Rn → R is a continuous smooth convex function, and its gradient is LLipschitz continuous:

∇f (x) − ∇f (y) 2 ≤ L x − y 2 ,

∀ x, y ∈ Rn .

To address the challenge of nonsmoothness, there are two problems to be solved: • how to cope with the nonsmoothness; • how to design a Nesterov-like optimal gradient method for nonsmooth convex optimization.

4.6.1 Subgradient and Subdifferential Because the gradient vector of a nonsmooth function h(x) does not exist everywhere, neither a gradient method nor the Nesterov optimal gradient method is available. A natural question to ask is whether a nonsmooth function has some class of “generalized gradient” similar to a gradient vector. For a twice continuous differentiable function f (x), its second-order approximation is given by f (x + Δx) ≈ f (x) + (∇f (x))T Δx + (Δx)T HΔx.

4.6 Nonsmooth Convex Optimization

225

If the Hessian matrix H is positive semi-definite or positive definite then we have the inequality f (x + Δx) ≥ f (x) + (∇f (x))T Δx, or f (y) ≥ f (x) + (∇f (x))T (y − x),

∀ x, y ∈ dom f (x).

(4.6.5)

Although a nonsmooth function h(x) does not have a gradient vector ∇h(x), it is possible to find another vector g instead of the gradient vector ∇f (x) such that the inequality (4.6.5) still holds. DEFINITION 4.14 A vector g ∈ Rn is said to be a subgradient vector of the function h : Rn → R at the point x ∈ Rn if h(y) ≥ h(x) + gT (y − x),

∀ x, y ∈ dom h.

(4.6.6)

The set of all subgradient vectors of the function h at the point x is known as the subdifferential of the function h at the point x, denoted ∂h(x), and is defined as  def   ∂h(x) = gh(y) ≥ h(x) + gT (y − x), ∀ y ∈ dom h . (4.6.7) When h(x) is differentiable, we have ∂h(x) = {∇h(x)}, as the gradient vector of a smooth function is unique, so we can view the gradient operator ∇h of a convex and differentiable function as a point-to-point mapping, i.e., ∇h maps each point x ∈ dom h to the point ∇h(x). In contrast, the subdifferential operator ∂h, defined in Equation (4.6.7), of a closed proper convex function h can be viewed as a pointto-set mapping, i.e., ∂h maps each point x ∈ dom h to the set ∂h(x).1 Any point g ∈ ∂h(x) is called a subgradient of h at x. Generally speaking, a function h(x) may have one or several subgradient vectors at some point x. The function h(x) is said to be subdifferentiable at the point x if it at least has one subgradient vector. More generally, the function h(x) is said to be subdifferentiable in the definition domain dom h, if it is subdifferentiable at all points x ∈ dom h. The basic properties of the subdifferential are as follows [344], [54]. 1. Convexity ∂h(x) is always a closed convex set, even if h(x) is not convex. 2. Nonempty and boundedness If x ∈ int(dom h) then the subdifferential ∂h(x) is nonempty and bounded. 3. Nonnegative factor If α > 0, then ∂(αh(x)) = α∂h(x). 4. Subdifferential If h is convex and differentiable at the point x then the subdifferential is a singleton ∂h(x) = {∇h(x)}, namely its gradient is its unique subgradient. Conversely, if h is a convex function and ∂h(x) = {g} then h is differentiable at the point x and g = ∇h(x). 1

A proper convex function f (x) takes values in the extended real number line such that f (x) < +∞ for at least one x and f (x) > −∞ for every x.

226

Gradient Analysis and Optimization

5. Minimum point of nondifferentiable function The point x is a minimum point of the convex function h if and only if h is subdifferentiable at x and 0 ∈ ∂h(x ).

(4.6.8)

This condition is known as the first-order optimality condition of the nonsmooth convex function h(x). If h is differentiable, then the first-order optimality condition 0 ∈ ∂h(x) simplifies to ∇h(x) = 0. 6. Subdifferential of a sum of functions If h1 , . . . , hm are convex functions then the subdifferential of h(x) = h1 (x) + · · · + hm (x) is given by ∂h(x) = ∂h1 (x) + · · · + ∂hm (x). 7. Subdifferential of affine transform If φ(x) = h(Ax + b) then the subdifferential ∂φ(x) = AT ∂h(Ax + b). 8. Subdifferential of pointwise maximal function Let h be a pointwise maximal function of the convex functions h1 , . . . , hm , i.e., h(x) = max hi (x); then i=1,...,m

∂h(x) = conv (∪ {∂hi (x)|hi (x) = h(x)}) . That is to say, the subdifferential of a pointwise maximal function h(x) is the convex hull of the union set of subdifferentials of the “active function” hi (x) at the point x. ˜ The subgradient vector g of the function h(x) is denoted as g = ∇h(x). EXAMPLE 4.8 The function h(x) = |x| is nonsmooth at x = 0 as its gradient ∇|x| does not exist there. To find the subgradient vectors of h(x) = |x|, rewrite the function as h(s, x) = |x| = sx from which it follows that the gradient of the function f (s, x) is given by ∂f (s, x)/∂x = s; s = −1 if x < 0 and s = +1 if x > 0. Moreover, if x = 0 then the subgradient vector should satisfy Equation (4.6.6), i.e., |y| ≥ gy or g ∈ [−1, +1]. Hence, the subdifferential of the function |x| is given by ⎧ ⎪ x < 0, ⎪ ⎨{−1}, ∂|x| = {+1}, (4.6.9) x > 0, ⎪ ⎪ ⎩[−1, 1], x = 0. EXAMPLE 4.9

Consider the function h(x) =

n i=1

|aTi x − bi |. Write

I− (x) = {i|ai , x − bi < 0}, I+ (x) = {i|ai , x − bi > 0}, I0 (x) = {i|ai , x − bi = 0}.

4.6 Nonsmooth Convex Optimization

Let h(x) =

n i=1

|aTi x − bi | =

n

T i=1 si (ai x

⎧ ⎪ ⎪ ⎨−1, si = 1, ⎪ ⎪ ⎩[−1, 1],

Hence the subdifferential of h(x) = ∂h(x) =

n  i=1

n i=1



s i ai =

− bi ); then x ∈ I− (x), x ∈ I+ (x), x ∈ I0 (x).

|aTi x − bi | is given by 

ai −

i∈I+ (x)

227



ai +

i∈I− (x)

α i ai ,

(4.6.10)

i∈I0 (x)

where αi = [−1, 1]. EXAMPLE 4.10 The 1 -norm h(x) = x 1 = |x1 | + · · · + |xm | is a special example n of the function h(x) = i=1 |aTi x − bi | with ai = ei and bi = 0, where ei is the basis vector with nonzero entry ei = 1. Hence, by Equation (4.6.10), we have    ei − ei + αi ei , x = 0, ∂h(x) = i∈I+ (x)

i∈I− (x)

i∈I0 (x)

where I− (x) = {i|xi < 0}, I+ (x) = {i|xi > 0} and I0 (x) = {i|xi = 0}. For x = 0, by Definition 4.14 the subgradient should satisfy y 1 ≥ 0 1 + gT (y − 0) with h(0) = 0, i.e., y 1 ≥ gT y, yielding gi = max1≤i≤n |yi | ≤ 1, y = 0. Therefore, the subdifferential of the 1 -norm h(x) = x 1 is given by ⎧   ⎨ ei − ei + αi ei , x = 0, xi 0 ⎩ {y ∈ Rn | max1≤i≤n |yi | ≤ 1}, x = 0, y = 0. EXAMPLE 4.11 x = 0, and thus

The Euclidean norm h(x) = x 2 is differentiable away from

∂ x 2 = {∇ x 2 } = {∇(xT x)1/2 } = {x/ x 2 },

∀ x = 0.

At the point x = 0, by Equation (4.6.6) we have g ∈ ∂h(0)



y 2 ≥ 0 + gT (y − 0)



g, y ≤ 1,

y 2

Hence, the subdifferential of the function x 2 is given by + x = 0, {x/ x 2 }, ∂ x 2 = n {y ∈ R | y 2 ≤ 1}, x = 0, y = 0.

∀ y = 0.

(4.6.12)

Let X ∈ Rm×n be any matrix with singular value decomposition X = UΣVT . Then the subdifferential of the nuclear norm of the matrix X (i.e., the sum of all singular values) is given by [79], [292], [503]    ∂ X ∗ = UVT +WW ∈ Rm×n , UT W = O, WV = O, W spec ≤ 1 . (4.6.13)

228

Gradient Analysis and Optimization

4.6.2 Proximal Operator Let Ci = dom fi (x), i = 1, . . . , P be a closed convex set of the m-dimensional 5P Euclidean space Rm and C = i=1 Ci be the intersection of these closed convex sets. Consider the combinatorial optimization problem min x∈C

P 

fi (x),

(4.6.14)

i=1

where the closed convex sets Ci , i = 1, . . . , P express the constraints imposed on the combinatorial optimization solution x. The types of intersection C can be divided into the following three cases [73]: (1) The intersection C is nonempty and “small” (all members of C are quite similar). (2) The intersection C is nonempty and “large” (the differences between the members of C are large). (3) The intersection C is empty, which means that the imposed constraints of intersecting sets are mutually contradictory. It is difficult to solve the combinatorial optimization problem (4.6.14) directly. But, if + 0, x ∈ Ci , f1 (x) = x − x0 , fi (x) = ICi (x) = +∞, x ∈ / Ci , then the original combinatorial optimization problem can be divided into separate problems x∈

min  P

i=2

Ci

x − x0 .

(4.6.15)

Differently from the combinatorial optimization problem (4.6.14), the separated optimization problems (4.6.15) can be solved using the projection method. In particular, when the Ci are convex sets, the projection of a convex objective function onto the intersection of these convex sets is closely related to the proximal operator of the objective function. DEFINITION 4.15 [331]

or

The proximal operator of a convex function h(x) is defined as " # 1 2 proxh (u) = arg min h(x) + x − u 2 2 x

(4.6.16)

" # 1 2 proxμh (u) = arg min h(x) +

x − u 2 2μ x

(4.6.17)

with scale parameter μ > 0.

4.6 Nonsmooth Convex Optimization

229

In particular, for a convex function h(X), its proximal operator is defined as " # 1 2 (4.6.18) proxμh (U) = arg min h(X) +

X − U F . 2μ X The proximal operator has the following important properties [488]: 1. Existence and uniqueness The proximal operator proxh (u) always exists, and is unique for all x. 2. Subgradient characterization There is the following correspondence between the proximal mapping proxh (u) and the subgradient ∂h(x): x = proxh (u)



x − u ∈ ∂h(x).

(4.6.19)

3. Nonexpansive mapping The proximal operator proxh (u) is a nonexpansive ˆ = proxh (ˆ mapping with constant 1: if x = proxh (u) and x u) then ˆ ) ≥ x − x ˆ 22 . ˆ )T (u − u (x − x 4. Proximal operator of a separable sum function If h : Rn1 × Rn2 → R is a separable sum function, i.e., h(x1 , x2 ) = h1 (x1 ) + h2 (x2 ), then proxh (x1 , x2 ) = (proxh1 (x1 ), proxh2 (x2 )). 5. Scaling and translation of argument proxh (x) =

If h(x) = f (αx + b), where α = 0, then

 1 proxα2 f (αx + b) − b . α

6. Proximal operator of the conjugate function If h∗ (x) is the conjugate function of the function h(x) then, for all μ > 0, the proximal operator of the conjugate function is given by proxμh∗ (x) = x − μproxh/μ (x/μ). If μ = 1 then the above equation simplifies to x = proxh (x) + proxh∗ (x).

(4.6.20)

This decomposition is called the Moreau decomposition. An operator closely related to the proximal operator is the soft thresholding operator of a real variable. DEFINITION 4.16 The action of the soft thresholding operator on a real variable x ∈ R is denoted Sτ [x] or soft(x, τ ) and is defined as ⎧ ⎪ ⎪ ⎨x − τ, x > τ, soft(x, τ ) = Sτ [x] = 0, (4.6.21) |x| ≤ τ, ⎪ ⎪ ⎩x + τ, x < −τ.

230

Gradient Analysis and Optimization

Here τ > 0 is called the soft threshold value of the real variable x. The action of the soft thresholding operator can be equivalently written as soft(x, τ ) = (x − τ )+ − (−x − τ )+ = max{x − τ, 0} − max{−x − τ, 0} = (x − τ )+ + (x + τ )− = max{x − τ, 0} + min{x + τ, 0}. This operator is also known as the shrinkage operator, because it can reduce the variable x, the elements of the vector x and the matrix X to zero, thereby shrinking the range of elements. Hence, the action of the soft thresholding operator is sometimes written as [29], [52] soft(x, τ ) = (|x| − τ )+ sign(x) = (1 − τ /|x|)+ x.

(4.6.22)

The soft thresholding operation on a real vector x ∈ Rn , denoted soft(x, τ ), is defined as a vector with entries soft(x, τ )i = max{xi − τ, 0} + min{xi + τ, 0} + x − τ, x > τ, i

=

i

0, xi + τ,

|xi | ≤ τ, xi < −τ.

(4.6.23)

The soft thresholding operation on a real matrix X ∈ Rm×n , denoted soft(X), is defined as an m × n real matrix with entries soft(X, τ )ij = max{xij − τ, 0} + min{xij + τ, 0} + x − τ, x > τ, ij

=

ij

0, xij + τ,

|xij | ≤ τ, xij < −τ.

(4.6.24)

Given a function h(x), our goal is to find an explicit expression for " # 1 2

x − u 2 . proxμh (u) = arg min h(x) + 2μ x THEOREM 4.11 [344, p. 129] Let x denote an optimal solution of the minimization problem min φ(x). If the function φ(x) is subdifferentiable then x = x∈dom

arg min φ(x) or φ(x ) = min φ(x) if and only if 0 ∈ ∂φ(x ). x∈domφ

x∈domφ

By the above theorem, the first-order optimality condition of the function φ(x) = h(x) +

1

x − u 22 2μ

(4.6.25)

0 ∈ ∂h(x ) +

1 (x − u), μ

(4.6.26)

is given by

4.6 Nonsmooth Convex Optimization

because ∂ we have

231

1 1

x − u 22 = (x − u). Hence, if and only if 0 ∈ ∂h(x ) + μ−1 (x − u), 2μ μ

" # 1 2 x = proxμh (u) = arg min h(x) +

x − u 2 . 2μ x

(4.6.27)

From Equation (4.6.26) we have 0 ∈ μ∂h(x ) + (x − u)



u ∈ (I + μ∂h)x



(I + μ∂h)−1 u ∈ x .

Since x is only one point, (I + μ∂h)−1 u ∈ x should read as (I + μ∂h)−1 u = x and thus 0 ∈ μ∂h(x ) + (x − u)



x = (I + μ∂h)−1 u



proxμh (u) = (I + μ∂h)−1 u.

This shows that the proximal operator proxμh and the subdifferential operator ∂h are related as follows: proxμh = (I + μ∂h)−1 .

(4.6.28)

The (point-to-point) mapping (I + μ∂h)−1 is known as the resolvent of the subdifferential operator ∂h with parameter μ > 0, so the proximal operator proxμh is the resolvent of the subdifferential operator ∂h. Notice that the subdifferential ∂h(x) is a point-to-set mapping, for which neither direction is unique, whereas the proximal operation proxμh (u) is a point-to-point mapping: proxμh (u) maps any point u to a unique point x. EXAMPLE 4.12 Consider a linear function h(x) = aT x + b, where a ∈ Rn , b ∈ R. By the first-order optimality condition   ∂ 1 1 aT x + b +

x − u 22 = a + (x − u) = 0, ∂x 2μ μ we get proxμh (u) = x = u − μa. EXAMPLE 4.13 For the quadratic function h(x) = 12 xT Ax − bT x + c, where A ∈ Rn×n is positive definite. The first-order optimality condition of proxμh (u) is   1 T ∂ 1 1 x Ax − bT x + c +

x − u 22 = Ax − b + (x − u) = 0, ∂x 2 2μ μ with x = x = proxμh (u). Hence, we have proxμh (u) = (A + μ−1 I)−1 (μ−1 u + b) = u + (A + μ−1 I)−1 (b − Au). EXAMPLE 4.14

If h is the indicator function " 0, h(x) = IC (x) = +∞,

x ∈ C, x∈ / C,

232

Gradient Analysis and Optimization

where C is a closed nonempty convex set, then the proximal operator of h(x) reduces to Euclidean projection onto C, i.e., PC (u) = arg minx∈C x − u 2 . Table 4.2 lists the proximal operators of several typical functions [109], [373].

Table 4.2 Proximal operators of several typical functions Functions

Proximal operators

h(x) = φ(x − z)

proxμh (u) = z + proxμφ (u − z)

h(x) = φ(x/ρ)

proxh (u) = ρproxφ/ρ2 (u/ρ)

h(x) = φ(−x)

proxμh (u) = −proxμφ (−u)



proxμh (u) = u − proxμφ (u)

h(x) = φ (x)

proxμh (u) = PC (u) = arg minx∈C x − u2

h(x) = IC (x)

proxμh (u) = u − μPC (u/μ)  ui − μ, ui > μ, |ui | ≤ μ, (proxμh (u))i = 0, ui + μ, ui < −μ.  (1 − μ/u2 )u, u2 ≥ μ, proxμh (u) = 0, u2 < μ.

T

h(x) = supy∈C y x h(x) = x1 h(x) = x2

proxμh (u) = u − μ a

h(x) = aT x + b h(x) = h(x) =

proxμh (x) = u + (A + μ−1 I)−1 (b − Au)    (proxμh (u))i = 12 ui + u2i + 4μ

1 T x Ax + bT x 2  − n i=1 log xi

In the following, we present the proximal gradient method for solving nonsmooth convex optimization problems.

4.6.3 Proximal Gradient Method Consider a typical form of nonsmooth convex optimization, min J(x) = f (x) + h(x), x

(4.6.29)

where f (x) is a convex, smooth (i.e., differentiable) and L-Lipschitz function, and h(x) is a convex but nonsmooth function (such as x 1 , X ∗ and so on). 1. Quadratic Approximation Consider the quadratic approximation of an L-Lipschitz smooth function f (x)

4.6 Nonsmooth Convex Optimization

233

around the point xk : 1 f (x) = f (xk ) + (x − xk )T ∇f (xk ) + (x − xk )T ∇2 f (xk )(x − xk ) 2 L T ≈ f (xk ) + (x − xk ) ∇f (xk ) + x − xk 22 , 2 where ∇2 f (xk ) is approximated by a diagonal matrix LI. Minimize J(x) = f (x) + h(x) via iteration to yield xk+1 = arg min {f (x) + h(x)} x " # L ≈ arg min f (xk ) + (x − xk )T ∇f (xk ) + x − xk 22 + h(x) 2 x + 2  22 0 2 L2 1 2 = arg min h(x) + 2x − xk − ∇f (xk ) 2 2 2 L x 2   1 = proxL−1 h xk − ∇f (xk ) . L In practical applications, the Lipschitz constant L of f (x) is usually unknown. Hence, a question of how to choose L in order to accelerate the convergence of xk arises. To this end, let μ = 1/L and consider the fixed point iteration xk+1 = proxμh (xk − μ∇f (xk )) .

(4.6.30)

This iteration is called the proximal gradient method for solving the nonsmooth convex optimization problem, where the step μ is chosen to equal some constant or is determined by linear searching. 2. Forward–Backward Splitting To derive a proximal gradient algorithm for nonsmooth convex optimization, let A = ∂h and B = ∇f denote the subdifferential operator and the gradient operator, respectively. Then, the first-order optimality condition for the objective function h(x) + f (x), 0 ∈ (∂h + ∇f )x , can be written in operator form as 0 ∈ (A + B)x , and thus we have 0 ∈ (A + B)x



(I − μB)x ∈ (I + μA)x



(I + μA)−1 (I − μB)x = x .

(4.6.31)

Here (I − μB) is a forward operator and (I + μA)−1 is a backward operator. The backward operator (I + μ∂h)−1 is sometimes known as the resolvent of ∂h with parameter μ. By proxμh = (I + μA)−1 and (I − μB)xk = xk − μ∇f (xk ), Equation (4.6.31) can be written as the fixed-point iteration xk+1 = (I + μA)−1 (I − μB)xk = proxμh (xk − μ∇f (xk )) .

(4.6.32)

234

Gradient Analysis and Optimization

If we let yk = (I − μB)xk = xk − μ∇f (xk )

(4.6.33)

then Equation (4.6.32) reduces to xk+1 = (I + μA)−1 yk = proxμh (yk ).

(4.6.34)

In other words, the proximal gradient algorithm can be split two iterations: (1) forward iteration yk = (I − μB)xk = xk − μ∇f (xk ); (2) backward iteration xk+1 = (I + μA)−1 yk = proxμh (yk ). The forward iteration is an explicit iteration which is easily computed, whereas the backward iteration is an implicit iteration. EXAMPLE 4.15

For the 1 -norm function h(x) = x 1 , the backward iteration is xk+1 = proxμ x 1 (yk ) = soft · 1 (yk , μ) = [soft · 1 (yk (1), μ), . . . , soft · 1 (yk (n), μ)]T

(4.6.35)

can be obtained by the soft thresholding operation [163]: xk+1 (i) = (soft · 1 (yk , μ))i = sign(yk (i)) max{|yk (i)| − μ, 0},

(4.6.36)

for i = 1, . . . , n. Here xk+1 (i) and yk (i) are the ith entries of the vectors xk+1 and yk , respectively. EXAMPLE 4.16

For the 2 -norm function h(x) = x 2 , the backward iteration is xk+1 = proxμ x 2 (yk ) = soft · 2 (yk , μ) = [soft · 2 (yk (1), μ), . . . , soft · 2 (yk (n), μ)]T ,

(4.6.37)

where soft · 2 (yk (i), μ) = max{|yk (i)| − μ, 0}

yk (i) ,

yk 2

i = 1, . . . , n.

(4.6.38)

min{m,n} EXAMPLE 4.17 For the nuclear norm of matrix X ∗ = i=1 σi (X), the corresponding proximal gradient method becomes   Xk = proxμ · ∗ Xk−1 − μ∇f (Xk−1 ) . (4.6.39) If W = UΣVT is the SVD of the matrix W = Xk−1 − μ∇f (Xk−1 ) then proxμ · ∗ (W) = UDμ (Σ)VT ,

(4.6.40)

where Dμ (Σ) is called the singular value thresholding operation, defined as " σi (X) − μ, if σi (X) > μ, [Dμ (Σ)]i = (4.6.41) 0, otherwise.

4.6 Nonsmooth Convex Optimization

235

EXAMPLE 4.18 If the nonsmooth function h(x) = IC (x) is an indicator function, then the proximal gradient iteration xk+1 = proxμh (xk − μ∇f (xk )) reduces to the gradient projection iteration xk+1 = PC (xk − μ∇f (xk )) .

(4.6.42)

A comparison between other gradient methods and the proximal gradient method is summarized below. • The update formulas are as follows: general gradient method, Newton method,

xk+1 = xk − μ∇f (xk );

xk+1 = xk − μH−1 (xk )∇f (xk );

proximal gradient method,

xk+1 = proxμh (xk − μ∇f (xk )) .

General gradient methods and the Newton method use a low-level (explicit) update, whereas the proximal gradient method uses a high-level (implicit) operation proxμh . • General gradient methods and the Newton method are available only for smooth and unconstrained optimization problems, while the proximal gradient method is available for smooth or nonsmooth and/or constrained or unconstrained optimization problems. • The Newton method allows modest-sized problems to be addressed; the general gradient methods are applicable for large-size problems, and, sometimes, distributed implementations, and the proximal gradient method is available for all large-size problems and distributed implementations. In particular, the proximal gradient method includes the general gradient method, the projected gradient method and the iterative soft thresholding method as special cases. (1) Gradient method If h(x) = 0, as proxh (x) = x the proximal gradient algorithm (4.6.30) reduces to the general gradient algorithm xk = xk−1 − μk ∇f (xk−1 ). That is to say, the gradient algorithm is a special case of the proximal gradient algorithm when the nonsmooth convex function h(x) = 0. (2) Projected gradient method For h(x) = IC (x), an indictor function, the minimization problem (4.6.4) becomes the unconstrained minimization min f (x). x∈C

Because proxh (x) = PC (x), the proximal gradient algorithm reduces to   (4.6.43) xk = PC xk−1 − μk ∇f (xk−1 ) 2 22 = arg min 2u − xk−1 + μk ∇f (xk−1 )22 . (4.6.44) u∈C

This is just the projected gradient method.

236

Gradient Analysis and Optimization

(3) Iterative soft thresholding method When h(x) = x 1 , the minimization problem (4.6.4) becomes the unconstrained minimization problem min (f (x) + x 1 ). In this case, the proximal gradient algorithm becomes   (4.6.45) xk = proxμk h xk−1 − μk ∇f (xk−1 ) . This is called the iterative soft thresholding ⎧ ⎪ ⎨ ui − μ, proxμh (u)i = 0, ⎪ ⎩ ui + μ,

method, for which ui > μ, − μ ≤ ui ≤ μ, ui < −μ.

From the viewpoint of convergence, the proximal gradient method is suboptimal. A Nesterov-like proximal gradient algorithm was developed by Beck and Teboulle [29], who called it the fast iterative soft thresholding algorithm (FISTA), as shown in Algorithm 4.9. Algorithm 4.9

FISTA algorithm with fixed step [29]

input: The Lipschitz constant L of ∇f (x). Initialize: y1 = x0 ∈ Rn , t1 = 1. repeat

  1 1. Compute xk = proxL−1 h yk − ∇f (yk ) . L  2 1 + 1 + 4tk . 2. Compute tk+1 = 2   t −1 (xk − xk−1 ). 3. Compute yk+1 = xk + k tk+1 4. exit if xk is converged.

return k ← k + 1. output: x ← xk .

THEOREM 4.12 [29] Let {xk } and {yk } be two sequences generated by the FISTA algorithm; then, for any iteration k ≥ 1, F (xk ) − F (x ) ≤

2L xk − x 22 , (k + 1)2

∀ x ∈ X ,

where x and X represent respectively the optimal solution and the optimal solution set of min (F (x) = f (x) + h(x)). Theorem 4.12 shows that to achieve the -optimal solution F (¯ x)  − F (x ) ≤ , the √ FISTA algorithm needs at most (C/ − 1) iterations, where C = 2L x0 − x 22 . Since it has the same fast convergence rate as the optimal first-order algorithm, the FISTA is indeed an optimal algorithm.

4.7 Constrained Convex Optimization

237

4.7 Constrained Convex Optimization Consider the constrained minimization problem min f0 (x) x

subject to

fi (x) ≤ 0, 1 ≤ i ≤ m, hi (x) = 0, 1 ≤ i ≤ q.

(4.7.1)

If the objective function f0 (x) and the inequality constraint functions fi (x), i = 1, . . . , m are convex, and the equality constraint functions hi (x) have the affine form h(x) = Ax − b, then Equation (4.7.1) is called a constrained convex optimization problem. The basic idea in solving a constrained optimization problem is to transform it into an unconstrained optimization problem. The transformation methods are of three types: (1) the Lagrange multiplier method; (2) the penalty function method; (3) the augmented Lagrange multiplier method. 4.7.1 Lagrange Multiplier Method Consider a simpler form of the constrained optimization problem (4.7.1) min f (x) subject to

Ax = b,

(4.7.2)

where x ∈ Rn , A ∈ Rm×n and the objective function f : Rn → R is convex. The Lagrange multiplier method transforms the constrained optimization problem (4.7.2) into an unconstrained optimization problem with objective function L(x, λ) = f (x) + λT (Ax − b),

(4.7.3)

where λ ≥ 0 is known as the dual variable or the Lagrange multiplier vector, all of whose entries must be larger than or equal to zero. The dual objective function of the original optimization problem (4.7.2) is given by g(λ) = inf L(x, λ) = −f ∗ (−AT λ) − bT λ, x

(4.7.4)

in which f ∗ is the convex conjugate of f . By the Lagrange multiplier method, the original equality constrained minimization problem (4.7.2) becomes the dual maximization problem   maxm g(λ) = −f ∗ (−AT λ) − bT λ . (4.7.5) λ∈R

The dual ascent method uses gradient ascent to solve the maximization problem (4.7.5). The dual ascent method consists of two steps: xk+1 = arg min L(x, λk ),

(4.7.6)

λk+1 = λk + μk (Axk+1 − b),

(4.7.7)

x

238

Gradient Analysis and Optimization

where (4.7.6) is the minimization step of the original variable x whereas (4.7.7) is the updating step of the dual variable λ, with step size μk . Because the dual variable λ ≥ 0 can be viewed as a “price” vector, the updating of the dual variable is also called the price-ascent or price-adjustment step. The object of price-ascent is to maximize the revenue function λk . The dual ascent method contains two aspects. (1) The update of the dual variable λ adopts the gradient ascent method. (2) The choice of step size μk should ensure the ascent of the dual objective function, namely g(λk+1 ) > g(λk ).

4.7.2 Penalty Function Method The penalty function method is a widely used constrained optimization method, and its basic idea is this: by using the penalty function and/or the barrier function, a constrained optimization becomes an unconstrained optimization of the composite function consisting of the original objective function and the constraint conditions. For the standard constrained optimization problem (4.7.1), the feasible set is defined as the set of points satisfying all the inequality and equality constraints, namely    F = xfi (x) ≤ 0, i = 1, . . . , m, hi (x) = 0, i = 1, . . . , q . (4.7.8) The set of points satisfying only the strict inequality constraints fi (x) < 0 and the equality constraints hi (x) = 0 is denoted by    relint(F) = xfi (x) < 0, i = 1, . . . , m, hi (x) = 0, i = 1, . . . , q (4.7.9) and is called the relative feasible interior set or the relative strictly feasible set. The points in the feasible interior set are called relative interior points. The penalty function method transforms the original constrained optimization problem into the unconstrained optimization problem   (4.7.10) min Lρ (x) = f0 (x) + ρp(x) , x∈S

where the coefficient ρ is a penalty parameter that reflects the intensity of “punishment” via the weighting of the penalty function p(x); the greater ρ is, the greater the value of the penalty term. The transformed optimization problem (4.7.10) is often called the auxiliary optimization problem of the original constrained optimization problem. The main property of the penalty function is as follows: if p1 (x) is the penalty for the closed set F1 , and p2 (x) is the penalty for the closed set F2 , then p1 (x) + p2 (x) is the penalty for the intersection F1 ∩ F2 . The following are two common penalty functions.

4.7 Constrained Convex Optimization

239

(1) Exterior penalty function p(x) = ρ1

m  

q  r |hi (x)|2 , max{0, fi (x)} + ρ2

i=1

(4.7.11)

i=1

where r is usually 1 or 2. (2) Interior penalty function p(x) = ρ1

m  i=1



q  1 |hi (x)|2 . log(−fi (x)) + ρ2 fi (x) i=1

(4.7.12)

In the exterior penalty function, if fi (x) ≤ 0, ∀ i = 1, . . . , m and hi (x) = 0, ∀ i = 1, . . . , q, then p(x) = 0, i.e., the penalty function has no effect on the interior points in the feasible set int(F). On the contrary, if, for some iteration point xk , the inequality constraint fi (xk ) > 0, i ∈ {1, . . . , m}, and/or h(xk ) = 0, i ∈ {1, . . . , q}, then the penalty term p(xk ) = 0, that is, any point outside the feasible set F is “punished”. Hence, the penalty function defined in (4.7.11) is known as an exterior penalty function. The role of (4.7.12) is equivalent to establishing a fence on the feasible set boundary bnd(F) to block any point in the feasible interior set int(F) from crossing the boundary of the feasible set bnd(F). Because the points in the relative feasible interior set relint(F) are slightly punished, the penalty function given in (4.7.12) is called the interior penalty function; it is also known as the barrier function. The following are three typical barrier functions for a closed set F [344]: (1) the power-function barrier function φ(x) =

m i=1



1 , p ≥ 1; (fi (x))p

m 1 log(−fi (x)); fi (x) i=1   m 1 exp − . (3) the exponential barrier function φ(x) = fi (x) i=1

(2) the logarithmic barrier function φ(x) = −

In other words, the logarithmic barrier function in (4.7.12) can be replaced by the power-function barrier function or the exponential barrier function. m  1 When p = 1, the power-function barrier function φ(x) = − is called the i=1

fi (x)

inverse barrier function and was presented by Carroll in 1961 [86]; whereas φ(x) = μ

m  i=1

1 log(−fi (x))

(4.7.13)

is known as the classical Fiacco–McCormick logarithmic barrier function [152], where μ is the barrier parameter. A summary of the features of the external penalty function follows.

240

Gradient Analysis and Optimization

(1) The exterior penalty function method is usually known as the penalty function method, and the interior penalty function method is customarily called the barrier method. (2) The exterior penalty function method punishes all points outside of the feasible set, and its solution satisfies all the inequality constraints fi (x) ≤ 0, i = 1, . . . , m, and all the equality constraints hi (x) = 0, i = 1, . . . , q, and is thus an exact solution to the constrained optimization problem. In other words, the exterior penalty function method is an optimal design scheme. In contrast, the interior penalty function method or barrier method blocks all points on the boundary of the feasible set, and the solution that is found satisfies only the strict inequality fi (x) < 0, i = 1, . . . , m, and the equality constraints hi (x) = 0, i = 1, . . . , q, and hence is only approximate. That is to say, the interior penalty function method is a suboptimal design scheme. (3) The exterior penalty function method can be started using an unfeasible point, and its convergence is slow, whereas the interior penalty function method requires the initial point to be a feasible interior point, so its selection is difficult; but it has a good convergence and approximation performance. In evolutionary computations the exterior penalty function method is normally used, while the interior function method is NP-hard owing to the feasible initial point search. Engineering designers, especially process controllers, prefer to use the interior penalty function method, because this method allows the designers to observe the changes in the objective function value corresponding to the design points in the feasible set in the optimization process. However, this facility cannot be provided by any exterior penalty function method. In the strict sense of the penalty function classification, all the above kinds of penalty function belong to the “death penalty”: the infeasible solution points x ∈ S \ F (the difference set of the search space S and the feasible set F ) are completely ruled out by the penalty function p(x) = +∞. If the feasible search space is convex, or it is the rational part of the whole search space, then this death penalty works very well [325]. However, for genetic algorithms and evolutionary computations, the boundary between the feasible set and the infeasible set is unknown and thus it is difficult to determine the precise position of the feasible set. In these cases, other penalty functions should be used [529]: static, dynamic, annealing, adaptive or co-evolutionary penalties.

4.7.3 Augmented Lagrange Multiplier Method According to [278], [41], the main deficiencies of the Lagrange multiplier method are as follows.

4.7 Constrained Convex Optimization

241

(1) Only when a constrained optimization problem has a locally convex structure, is the dual unconstrained optimization problem well-defined, so that the updating of the Lagrange multiplier λk+1 = λk + αk h(xk ) can be implemented. (2) The convergence of the Lagrangian objective function is more time-consuming, as the updating of the Lagrange multiplier is an ascent iteration and its convergence is only moderately fast. According to [41], the deficiency in the penalty function method is that its convergence is slow, and a large penalty parameter easily results in an ill-conditioned unconstrained optimization problem and thus causes numerical instability of the optimization algorithm. A simple and efficient way to overcome the disadvantages of the Lagrange multiplier method and the penalty function method is to combine them into an augmented Lagrange multiplier method. Consider the augmented Lagrangian function L(x, λ, ν) = f0 (x) +

m 

λi fi (x) +

i=1 T

q 

νi hi (x)

i=1

= f0 (x) + λ f (x) + ν T h(x),

(4.7.14)

where λ = [λ1 , . . . , λm ]T and ν = [ν1 , . . . , νq ]T are the Lagrange multiplier vector and the penalty parameter vector, respectively, whereas f (x) = [f1 (x), . . . , fm (x)]T and h(x) = [h1 (x), . . . , hq (x)]T are the inequality constraint vector and the equality constraint vector, respectively. If on the one hand ν = 0 then the augmented Lagrangian function simplifies to the Lagrangian function: L(x, λ, 0) = L(x, λ) = f0 (x) +

m 

λi fi (x).

i=1

This implies that the augmented Lagrange multiplier method reduces to the Lagrange multiplier method for an inequality constrained-optimization problem. On the other hand, if λ = 0 and ν = ρh(x) with ρ > 0 then the augmented Lagrangian function reduces to the penalty function: L(x, 0, ν) = L(x, ν) = f0 (x) + ρ

q 

|hi (x)|2 .

i=1

This implies that the augmented Lagrange multiplier method includes the penalty function method for a equality constrained optimization problem as a special case. The above two facts show that the augmented Lagrange multiplier method combines the Lagrange multiplier method and the penalty function method.

242

Gradient Analysis and Optimization

4.7.4 Lagrangian Dual Method As in the Lagrange multiplier method, the Lagrange multiplier vector λ of the augmented Lagrange multiplier method has the nonnegative constraint λ ≥ 0. Under this constraint, let us consider how to solve the unconstrained minimization problem + 0 q m   λi fi (x) + νi hi (x) . (4.7.15) min L(x, λ, ν) = min f0 (x) + λ≥0, ν

i=1

i=1

The vector x is known as the optimization variable or decision variable or primal variable, while λ is the dual variable. The original constrained optimization problem (4.7.1) is simply called the original problem, and the unconstrained optimization problem (4.7.15) is simply known as the dual problem. Owing to the nonnegativity of the Lagrange multiplier vector λ, the augmented Lagrangian function L(x, λ, ν) may tend to negative infinity when some λi equals a very large positive number. Therefore, we first need to maximize the original augmented Lagrangian function, to get + 0 q m   J1 (x) = max f0 (x) + λi fi (x) + νi hi (x) . (4.7.16) λ≥0, ν

i=1

i=1

The problem with using the unconstrained maximization (4.7.16) is that it is impossible to avoid violation of the constraint fi (x) > 0. This may result in J1 (x) equalling positive infinity; namely, " f0 (x), if x meets all original constraints, J1 (x) = (4.7.17) (f0 (x), +∞), otherwise. From the above equation it follows that in order to minimize f0 (x) subject to all inequality and equality constraints, we should minimize J1 (x) to get the primal cost function JP (x) = min J1 (x) = min max L(x, λ, ν). x

x

λ≥0, ν

(4.7.18)

This is a minimax problem, whose solution is the supremum of the Lagrangian function L(x, λ, ν), namely   q m   λi fi (x) + νi hi (x) . (4.7.19) JP (x) = sup f0 (x) + i=1

i=1

From (4.7.19) and (4.7.17) it follows that the optimal value of the original constrained minimization problem is given by p = JP (x ) = min f0 (x) = f0 (x ), x

which is simply known as the optimal primal value.

(4.7.20)

4.7 Constrained Convex Optimization

243

However, the minimization of a nonconvex objective function cannot be converted into the minimization of another convex function. Hence, if f0 (x) is a convex function then, even if we designed an optimization algorithm that can find a local ˜ of the original cost function, there is no guarantee that x ˜ is a global extremum x extremum point. Fortunately, the minimization of a convex function f (x) and the maximization of the concave function −f (x) are equivalent. On the basis of this dual relation, it is easy to obtain a dual method for solving the optimization problem of a nonconvex objective function: convert the minimization of a nonconvex objective function into the maximization of a concave objective function. For this purpose, construct another objective function from the Lagrangian function L(x, λ, ν): J2 (λ, ν) = min L(x, λ, ν) x + = min f0 (x) + x

m  i=1

λi fi (x) +

q 

0 νi hi (x) .

(4.7.21)

i=1

From the above equation it is known that + if x meets all the original constraints, min f0 (x), x min L(x, λ, ν) = x (−∞, min f0 (x)), otherwise. x

(4.7.22) Its maximization function JD (λ, ν) = max J2 (λ, ν) = max min L(x, λ, ν) λ≥0, ν

λ≥0, ν

x

(4.7.23)

is called the dual objective function for the original problem. This is a maximin problem for the Lagrangian function L(x, λ, ν). Since the maximin of the Lagrangian function L(x, λ, ν) is its infimum, we have   q m   λi fi (x) + νi hi (x) . (4.7.24) JD (λ, ν) = inf f0 (x) + i=1

i=1

The dual objective function defined by Equation (4.7.24) has the following characteristics. (1) The function JD (λ, ν) is an infimum of the augmented Lagrangian function L(x, λ, ν). (2) The function JD (λ, ν) is a maximizing objective function, and thus it is a value or utility function rather than a cost function. (3) The function JD (λ, ν) is lower unbounded: its lower bound is −∞. Hence, JD (λ, ν) is a concave function of the variable x even if f0 (x) is not a convex function.

244

Gradient Analysis and Optimization

THEOREM 4.13 [353, p. 16] Any local minimum point x of an unconstrained convex optimization function f (x) is a global minimum point. If the convex function f (x) is differentiable then the stationary point x such that ∂f (x)/∂x = 0 is a global minimum point of f (x). Theorem 4.13 shows that any extreme point of the concave function is a global extreme point. Therefore, the algorithm design of the standard constrained minimization problem (4.7.1) becomes the design of an unconstrained maximization algorithm for the dual objective function JD (λ, ν). Such a method is called the Lagrangian dual method.

4.7.5 Karush–Kuhn–Tucker Conditions The optimal value of the dual objective function is simply called the optimal dual value, denoted d = JD (λ , ν ).

(4.7.25)

From (4.7.22) and (4.7.23) it is immediately known that d ≤ min f0 (x) = p . x

(4.7.26)

The difference between the optimal primal value p and the optimal dual value d , denoted p − d , is known as the duality gap between the original minimization problem and the dual maximization problem. Equation (4.7.26) gives the relationship between the maximin and the minimax of the augmented Lagrangian function L(x, λ, ν). In fact, for any nonnegative realvalued function f (x, y), there is the following inequality relation between the maximin and minimax: max min f (x, y) ≤ min max f (x, y). x

y

y

x

(4.7.27)

If d ≤ p then the Lagrangian dual method is said to have weak duality, while when d = p , we say that the Lagrangian dual method satisfies strong duality. Given an allowed dual gap , the points x and (λ, ν) satisfying p − f0 (x) ≤

(4.7.28)

are respectively called the -suboptimal original point and the -suboptimal dual points of the dual-concave-function maximization problem. Let x and (λ , ν ) represent any original optimal point and dual optimal points with zero dual gap = 0. Since x among all original feasible points x minimizes the augmented Lagrangian objective function L(x, λ , ν ), the gradient vector of L(x, λ , ν ) at the point x must be equal to the zero vector, namely ∇f0 (x ) +

m  i=1

λ i ∇fi (x ) +

q  i=1

νi ∇hi (x ) = 0.

4.7 Constrained Convex Optimization

245

Therefore, the Karush–Kuhn–Tucker (KKT) conditions (i.e., the first-order necessary conditions) of the Lagrangian dual unconstrained optimization problem are given by [353] ⎫ fi (x ) ≤ 0, i = 1, . . . , m (original inequality constraints), ⎪ ⎪ ⎪ ⎪ ⎪ hi (x ) = 0, i = 1, . . . , q (original equality constraints), ⎪ ⎪ ⎪ ⎪ ⎬ λi ≥ 0, i = 1, . . . , m (nonnegativity), (4.7.29) ⎪ λ i fi (x ) = 0, i = 1, . . . , m (complementary slackness), ⎪ ⎪ ⎪ ⎪ q m ⎪   ⎪ ⎪ ∇f0 (x )+ λi ∇fi (x ) + νi ∇hi (x ) = 0 (stationary point).⎪ ⎭ i=1

i=1

A point x satisfying the KKT conditions is called a KKT point. Remark 1 The first KKT condition and the second KKT condition are the original inequality and equality constraint conditions, respectively. Remark 2 The third KKT condition is the nonnegative condition of the Lagrange multiplier λi , which is a key constraint of the Lagrangian dual method. Remark 3 The fourth KKT condition (complementary slackness) is also called dual complementary and is another key constraint of the Lagrangian dual method. This condition implies that, for a violated constraint fi (x) > 0, the corresponding Lagrange multiplier λi must equal zero, and thus we can avoid completely any violated constraint. Thus the role of this condition is to establish a barrier fi (x) = 0, i = 1, . . . , m on the boundary of inequality constraints, that prevents the occurrence of constraint violation fi (x) > 0. Remark 4 The fifth KKT condition is the condition for there to be a stationary point at minx L(x, λ, ν). Remark 5 If the inequality constraint fi (x) ≤ 0, i = 1, . . . , m, in the constrained optimization (4.7.1) becomes ci (x) ≥ 0, i = 1, . . . , m, then the Lagrangian function should be modified to L(x, λ, ν) = f0 (x) −

m  i=1

λi ci (x) +

q 

νi hi (x),

i=1

and all inequality constrained functions fi (x) in the KKT condition formula (4.7.29) should be replaced by −ci (x). In the following, we discuss the necessary modifications of the KKT conditions under some assumptions. DEFINITION 4.17 For inequality constraints fi (x) ≤ 0, i = 1, . . . , m, if fi (¯ x) = 0 ¯ then the ith constraint is said to be an active constraint at the at the point x ¯ . If fi (¯ point x x) < 0 then the ith constraint is called an inactive constraint at ¯ . If fi (¯ the point x x) > 0 then the ith constraint is known as a violated constraint

246

Gradient Analysis and Optimization

¯ . The index set of all active constraints at the point x ¯ is denoted at the point x ¯. A(¯ x) = {i|fi (¯ x) = 0} and is referred to as the active set of the point x Let m inequality constraints fi (x), i = 1, . . . , m, have k active constraints fA1 (x ), . . . , fAk (x ) and m − k inactive constraints at some KKT point x . In order to satisfy the complementarity in the KKT conditions λi fi (x ) = 0, the Lagrange multipliers λ i corresponding to the inactive constraints fi (x ) < 0 must be equal to zero. This implies that the last KKT condition in (4.7.29) becomes ∇f0 (x ) +



λ i ∇fi (x ) +







∂f0 (x ) ⎢ ∂x1 ⎥

⎢ ⎢ ⎢ ⎣ ∂f

∂h1 (x ) ⎢ ∂x1

⎥ ⎢ ⎥+⎢ ⎥ ⎢ ⎣ ∂h  ⎦ (x ) .. .

0

∂xn



νi ∇hi (x ) = 0

i=1

i∈A

or

q 

.. .

1 (x ∂xn



∂fA1 (x )

⎢ ∂x1 ⎢ .. = −⎢ . ⎢ ⎣ ∂f (x ) A1

∂xn



)

··· ..

.

···



∂hq (x ) ⎡ ⎤ ∂x1 ⎥ ν1

··· ..

⎥⎢ . ⎥ ⎥⎣ . ⎦ ⎥ .  ⎦ ∂hq (x ) νq .. .

.

···

∂xn ⎤ ∂fAk (x ) ⎡ ⎤ ∂x1 ⎥ λA1 

⎥⎢ . ⎥ ⎥⎣ . ⎦ . ⎥ ∂fAk (x ) ⎦ λ Ak .. .

∂xn

namely ∇f0 (x ) + (Jh (x ))T ν = −(JA (x ))T λ A ,

(4.7.30)

where Jh (x ) is the Jacobian matrix of the equality constraint function hi (x) = 0, i = 1, . . . , q at the point x and ⎡ ⎤ ∂fA1 (x ) ∂fA1 (x ) ···   ⎢ ∂x1 ∂xn ⎥ ⎢ ⎥ . .. . ⎢ ⎥ ∈ Rk×n , . . JA (x ) = ⎢ (4.7.31) . . . ⎥ ⎣ ∂fAk (x )  ⎦ ∂fAk (x ) ···   ∂x1

λ A

=

[λ A1 , . . . , λ Ak ]

∂xn

∈R , k

(4.7.32)

are the Jacobian matrix and Lagrange multiplier vector respectively of the active constraint function. x) of the active constraint Equation (4.7.30) shows that if the Jacobian matrix JA (¯ ¯ is of full row rank, then the actively constrained function at the feasible point x Lagrange multiplier vector can be uniquely determined:   x)JA (¯ x)T )−1 JA (¯ x) ∇f0 (¯ x) + (Jh (¯ x))T ν . (4.7.33) λ A = −(JA (¯ In optimization-algorithm design we always want strong duality to be set up. A simple method for determining whether strong duality holds is Slater’s theorem.

4.7 Constrained Convex Optimization

247

Define the relative interior of the feasible domain of the original inequality constraint function F as follows: relint(F) = {x|fi (x) < 0, i = 1, . . . , m, hi (x) = 0, i = 1, . . . , p}.

(4.7.34)

¯ ∈ relint(F) is called a A point in the relative interior of the feasible domain x relative interior point. In an optimization process, the constraint restriction that iterative points should be in the interior of the feasible domain is known as the Slater condition. Slater’s theorem says that if the Slater condition is satisfied, and the original inequality optimization problem (4.7.1) is a convex optimization problem, then the optimal value d of the dual unconstrained optimization problem (4.7.23) is equal to the the optimal value p of the original optimization problem, i.e., strong duality holds. The following summarizes the relationships between the original constrained optimization problem and the Lagrangian dual unconstrained convex optimization problem. (1) Only when the inequality constraint functions fi (x), i = 1, . . . , m, are convex, and the equality constraint functions hi (x), i = 1, . . . , q, are affine, can an original constrained optimization problem be converted into a dual unconstrained maximization problem by the Lagrangian dual method. (2) The maximization of a concave function is equivalent to the minimization of the corresponding convex function. (3) If the original constrained optimization is a convex problem then the points ˜ ν) ˜ and (λ, ˜ satisfying the KKT condiof the Lagrangian objective function x tions are the original and dual optimal points, respectively. In other words, the optimal solution d of the Lagrangian dual unconstrained optimization is the optimal solution p of the original constrained convex optimization problem. (4) In general, the optimal solution of the Lagrangian dual unconstrained optimization problem is not the optimal solution of the original constrained optimization problem but only a -suboptimal solution, where = f0 (x ) − JD (λ , ν ). Consider the constrained optimization problem with equality and inequality constraints min f (x) subject to x

Ax ≤ h, Bx = b.

(4.7.35)

Let the nonnegative vector s ≥ 0 be a slack variable vector such that Ax + s = h. Hence, the inequality constraint Ax ≤ h becomes the equality constraint Ax + s − h = 0. Taking the penalty function φ(g(x)) = 12 g(x) 22 , the augmented Lagrangian objective function is given by Lρ (x, s, λ, ν) = f (x) + λT (Ax + s − h) + ν T (Bx − b)  ρ + Bx − b 22 + Ax + s − h 22 , 2

(4.7.36)

248

Gradient Analysis and Optimization

where the two Lagrange multiplier vectors λ ≥ 0 and ν ≥ 0 are nonnegative vectors, and the penalty parameter ρ > 0. The dual gradient ascent method for solving the original optimization problem (4.7.35) is given by xk+1 = arg min Lρ (x, sk , λk , νk ),

(4.7.37)

sk+1 = arg min Lρ (xk+1 , s, λk , νk ),

(4.7.38)

λk+1 = λk + ρk (Axk+1 + sk+1 − h),

(4.7.39)

νk+1 = νk + ρk (Bxk+1 − b),

(4.7.40)

x

s≥0

where the gradient vectors are ∂Lρ (xk+1 , sk+1 , λk , νk ) = Axk+1 + sk+1 − h, ∂λk ∂Lρ (xk+1 , sk+1 , λk , νk ) = Bxk+1 − b. ∂νk Equations (4.7.37) and (4.7.38) are respectively the updates of the original variable x and the intermediate variable s, whereas Equations (4.7.39) and (4.7.40) are the dual gradient ascent iterations of the Lagrange multiplier vectors λ and ν, taking into account the inequality constraint Ax ≥ h and the equality constraint Bx = b, respectively.

4.7.6 Alternating Direction Method of Multipliers In applied statistics and machine learning we often encounter large-scale equality constrained optimization problems, where the dimension of x ∈ Rn is very large. If the vector x may be decomposed into several subvectors, i.e., x = (x1 , . . . , xr ), and the objective function may also be decomposed into f (x) =

r  i=1

fi (xi ),

r where xi ∈ Rni and i=1 ni = n, then a large-scale optimization problem can be transformed into a few distributed optimization problems. The alternating direction multiplier method (ADMM) is a simple and effective method for solving distributed optimization problems. The ADMM method decomposes an optimization problem into smaller subproblems; then, their local solutions are restored or reconstructed into a large-scale optimization solution of the original problem. The ADMM was proposed by Gabay and Mercier [167] and Glowinski and Marrocco [176] independently in the mid-1970s.

4.7 Constrained Convex Optimization

249

Corresponding to the composition of the objective function f (x), the equality constraint matrix is blocked as follows: A = [A1 , . . . , Ar ],

Ax =

r 

Ai x i .

i=1

Hence the augmented Lagrangian objective function can be written as [52] Lρ (x, λ) =

r 

Li (xi , λ)

i=1

=

r  

fi (xi ) + λ Ai xi T



i=1

22 2 r 2 ρ2 2 2 − λ b + 2 (Ai xi ) − b2 . 2 2 2 i=1 T

2

Applying the dual ascent method to the augmented Lagrangian objective function yields a decentralized algorithm for parallel computing [52]: = arg min Li (xi , λk ), xk+1 i xi ∈Rni

λk+1 = λk + ρk

 r 

i = 1, . . . , r,

(4.7.41)

 Ai xk+1 i

−b .

(4.7.42)

i=1

Here the updates xi (i = 1, . . . , r) can be run independently in parallel. Because xi , i = 1, . . . , r are updated in an alternating or sequential manner, this augmented multiplier method is known as the “alternating direction” method of multipliers. In practical applications, the simplest decomposition is the objective function decomposition of r = 2: min {f (x) + h(z)}

subject to Ax + Bz = b,

(4.7.43)

where x ∈ Rn , z ∈ Rm , A ∈ Rp×n , B ∈ Rp×m , b ∈ Rp . The augmented Lagrangian cost function of the optimization problem (4.7.43) is given by Lρ (x, z, λ) = f (x) + h(z) + λT (Ax + Bz − b) ρ + Ax + Bz − b 22 . 2

(4.7.44)

It is easily seen that the optimality conditions are divided into the original feasibility condition Ax + Bz − b = 0

(4.7.45)

and two dual feasibility conditions, 0 ∈ ∂f (x) + AT λ + ρAT (Ax + Bz − b) = ∂f (x) + AT λ,

(4.7.46)

0 ∈ ∂h(z) + B λ + ρB (Ax + Bz − b) = ∂h(z) + B λ,

(4.7.47)

T

T

T

250

Gradient Analysis and Optimization

where ∂f (x) and ∂h(z) are the subdifferentials of the subobjective functions f (x) and h(z), respectively. The updates of the ADMM for the optimization problem min Lρ (x, z, λ) are as follows: xk+1 = arg min Lρ (x, zk , λk ),

(4.7.48)

zk+1 = arg min Lρ (xk+1 , z, λk ),

(4.7.49)

λk+1 = λk + ρk (Axk+1 + Bzk+1 − b).

(4.7.50)

x∈Rn

z∈Rm

The original feasibility cannot be strictly satisfied; its error rk = Axk + Bzk − b

(4.7.51)

is known as the original residual (vector) in the kth iteration. Hence the update of the Lagrange multiplier vector can be simply written as λk+1 = λk + ρk rk+1 .

(4.7.52)

Likewise, neither can dual feasibility be strictly satisfied. Because xk+1 is the minimization variable of Lρ (x, zk , λk ), we have 0 ∈ ∂f (xk+1 ) + AT λk + ρAT (Axk+1 + Bzk − b) = ∂f (xk+1 ) + AT (λk + ρrk+1 + ρB(zk − zk+1 )) = ∂f (xk+1 ) + AT λk+1 + ρAT B(zk − zk+1 ). Comparing this result with the dual feasibility formula (4.7.46), it is easy to see that sk+1 = ρAT B(zk − zk+1 )

(4.7.53)

is the error vector of the dual feasibility, and hence is known as the dual residual vector in the (k + 1)th iteration. The stopping criterion of the ADMM is as follows: the original residual and the dual residual in the (k + 1)th iteration should be very small, namely [52]

rk+1 2 ≤ pri ,

sk+1 2 ≤ dual ,

(4.7.54)

where pri and dual are respectively the allowed disturbances of the primal feasibility and the dual feasibility. If we let ν = (1/ρ)λ be the Lagrange multiplier vector scaled by the ratio 1/ρ, called the scaled dual vector, then Equations (4.7.48)–(4.7.50) become [52]   xk+1 = arg min f (x) + (ρ/2) Ax + Bzk − b + νk 22 , (4.7.55) n x∈R   zk+1 = arg min h(z) + (ρ/2) Axk+1 + Bz − b + νk 22 , (4.7.56) z∈Rm

νk+1 = νk + Axk+1 + Bzk+1 − b = νk + rk+1 .

(4.7.57)

4.8 Newton Methods

251

The scaled dual vector has an interesting interpretation [52]: from the residual at the kth iteration, rk = Axk + Bzk − b, it is easily seen that νk = ν0 +

k 

ri .

(4.7.58)

i=1

That is, the scaled dual vector is the running sum of the original residuals of all k iterations. Equations (4.7.55)–(4.7.57) are referred to as the scaled ADMM, while Equations (4.7.48)–(4.7.50) are the ADMM with no scaling.

4.8 Newton Methods The first-order optimization algorithms use only the zero-order information f (x) and the first-order information ∇f (x) about an objective function. It is known [305] that if the objective function is twice differentiable then the Newton method based on the Hessian matrix is of quadratic or more rapid convergence.

4.8.1 Newton Method for Unconstrained Optimization For an unconstrained optimization minn f (x), if the Hessian matrix H = ∇2 f (x) x∈R

is positive definite then, from the Newton equation ∇2 f (x)Δx = −∇f (x), we get the Newton step Δx = −(∇2 f (x))−1 ∇f (x), which results in the gradient descent algorithm xk+1 = xk − μk (∇2 f (xk ))−1 ∇f (xk ).

(4.8.1)

This is the well-known Newton method. The Newton method may encounter the following two thorny issues in applications. (1) The Hessian matrix H = ∇2 f (x) is difficult to find. (2) Even if H = ∇2 f (x) can be found, its inverse H−1 = (∇2 f (x))−1 may be numerically unstable. There are the following three methods for resolving the above two thorny issues. 1. Truncated Newton Method Instead of using the inverse of the Hessian matrix directly, an iteration method for solving the Newton matrix equation ∇2 f (x)Δxnt = −∇f (x) is used to find an approximate solution for the Newton step Δxnt . The iteration method for solving an Newton matrix equation approximately is known as the truncated Newton method [122], where the conjugate gradient algo-

252

Gradient Analysis and Optimization

rithm and the preconditioned conjugate gradient algorithm are two popular algorithms for such a case. The truncated Newton method is especially useful for large-scale unconstrained and constrained optimization problems and interior-point methods. 2. Modified Newton Method When the Hessian matrix is not positive definite, the Newton matrix equation can be modified to [158] (∇2 f (x) + E)Δxnt = −∇f (x), (4.8.2) where E is a positive semi-definite matrix that is usually taken as a diagonal matrix such that ∇2 f (x) + E is symmetric positive definite. Such a method is called the modified Newton method. The typical modified Newton method takes E = δI, where δ > 0 is small. 3. Quasi-Newton Method Using a symmetric positive definite matrix Bk to approximate the inverse of the Hessian matrix H−1 k in the Newton method, we have the following recursion: xk+1 = xk − Bk ∇f (xk ).

(4.8.3)

The Newton method using a symmetric matrix to approximate the Hessian matrix is called the quasi-Newton method. Usually a symmetric matrix ΔHk = Hk+1 − Hk is used to approximate the Hessian matrix, and we denote ρk = f (xk+1 ) − f (xk ) and δk = xk+1 − xk . According to different approximations for the Hessian matrix, the quasi-Newton procedure employs the following three common methods [344]. (1) Rank-1 updating method ΔHk =

(δk − Hk ρk )(δk − Hk ρk )T . δk − Hk ρk , ρk 

(2) DFP (Davidon–Fletcher–Powell) method ΔHk =

δk δkT H ρ ρT H − k k k k. ρk , δk  Hk ρk , ρk 

(3) BFGS (Broyden–Fletcher–Goldfarb–Shanno) method ΔHk =

H ρ ρT H Hk ρk δkT + δk ρTk Hk − βk k k k k , Hk ρk , ρk  Hk ρk , ρk 

where βk = 1 + ρk , δk /Hk ρk , ρk . An important property of the DFP and BFGS methods is that they preserve the positive definiteness of the matrices.

4.8 Newton Methods

253

In the iterative process of the various Newton methods, it is usually required that a search is made of the optimal point along the direction of the straight line {x+μΔx | μ ≥ 0}. This step is called linear search. However, this choice of procedure can only minimize the objective function approximately, i.e., make it sufficiently small. Such a search for approximate minimization is called an inexact line search. In this search the step size μk on the search line {x + μΔx | μ ≥ 0} must decrease the objective function f (xk ) sufficiently to ensure that f (xk + μΔxk ) < f (xk ) + αμ(∇f (xk ))T Δxk ,

α ∈ (0, 1).

(4.8.4)

The inequality condition (4.8.4) is sometimes called the Armijo condition [42], [157], [305], [353]. In general, for a larger step size μ, the Armijo condition is usually not met. Hence, Newton algorithms can start the search from μ = 1; if the Armijo condition is not met then the step size μ needs to be reduced to β μ, with the correction factor β ∈ (0, 1). If for step size β μ the Armijo condition is still not met, then the step size needs to be reduced again until the Aemijo condition is met. Such a search method is known as backtracking line search. Algorithm 4.10 shows the Newton algorithm via backtracking line search. Algorithm 4.10

Newton algorithm via backtracking line search

initialization: x1 ∈ dom f (x), and parameters α ∈ (0, 0.5), β ∈ (0, 1). Put k = 1. repeat 1. Compute bk = ∇f (xk ) and Hk = ∇2 f (xk ). 2. Solve the Newton equation Hk Δxk = −bk . 3. Update xk+1 = xk + μΔxk . 4. exit if |f (xk+1 ) − f (xk )| < . return k ← k + 1. output: x ← xk .

The backtracking line search method can ensure that the objective function satisfies f (xk+1 ) < f (xk ) and the step size μ may not need to be too small. If the Newton equation is solved by other methods in step 2, then Algorithm 4.10 becomes the truncated Newton method, the modified Newton method or the quasi-Newton method, respectively. Now consider the minimization min f (z), where z ∈ Cn and f : Cn → R. By the second-order Taylor series approximation of a binary complex function, ∂f (x, y) ∂f (x, y) f (x + h1 , y + h2 ) = f (x, y) + h1 + h2 ∂x ∂y   2 2 1 ∂ f (x, y) ∂ f (x, y) ∂ 2 f (x, y) h21 , + + 2h1 h2 + h22 2! ∂x∂x ∂x∂y ∂y∂y it is easily seen that the second-order Taylor series approximation of the holomor-

254

Gradient Analysis and Optimization

phic function f (z, z∗ ) is given by f (z + Δz, z∗ + Δz∗ )     Δz ∗ ∗ T ∗ T = f (z, z ) + (∇z f (z, z )) , (∇z∗ f (z, z )) Δz∗ ⎡ 2 ⎤ ∂ f (z, z∗ ) ∂ 2 f (z, z∗ )    ∗ T 1 ∂ z∗ ∂ zH ⎥ Δz H T ⎢ ∂z ∂z . (Δz) , (Δz) ⎣ 2 + ⎦ Δz∗ 2 ∂ f (z, z∗ ) ∂ 2 f (z, z∗ ) ∂ z∂ zT

From the first-order optimality condition

∂ z∂ zH

(4.8.5)  

0 ∂f (z + Δz, z∗ + Δz∗ )   = , we obtain 0 Δz ∂ ∗ Δz

immediately the equation for the complex Newton step:      ∇zk f (zk , z∗k ) Hz∗k , zk Hz∗k , z∗k Δznt,k , = − ∇z∗k f (zk , z∗k ) Hzk , zk Hzk , z∗k Δz∗nt,k

(4.8.6)

where Hz∗k , zk =

∂ 2 f (zk , z∗k ) , ∂ z∗k ∂ zTk

Hz∗k , z∗k (k) =

Hzk , zk =

∂ 2 f (zk , z∗k ) , ∂ zk ∂ zTk

Hzk , z∗k =

∂ 2 f (zk , z∗k ) , ∂ z∗k ∂ zH k

∂ 2 f (zk , z∗k ) , ∂ zk ∂ zH k

(4.8.7)

are the part-Hessian matrices of the holomorphic function f (zk , z∗k ), respectively. Hence, the updating formula of the complex Newton method is described by       Δznt,k zk+1 zk = ∗ +μ . (4.8.8) Δz∗nt,k z∗k+1 zk 4.8.2 Newton Method for Constrained Optimization First, consider the equality constrained optimization problem with a real variable: min f (x) x

subject to

Ax = b,

(4.8.9)

where f : Rn → R is a convex function and is twice differentiable, while A ∈ Rp×n and rank(A) = p with p < n. Let Δxnt denote the Newton search direction. Then the second-order Taylor approximation of the objective function f (x) is given by 1 f (x + Δxnt ) = f (x) + (∇f (x))T Δxnt + (Δxnt )T ∇2 f (x)Δxnt , 2 subject to A(x + Δxnt ) = b and AΔxnt = 0. In other words, the Newton search direction can be determined by the equality constrained optimization problem: " # 1 T T 2 min f (x) + (∇f (x)) Δxnt + (Δxnt ) ∇ f (x)Δxnt subject to AΔxnt = 0. Δxnt 2

4.8 Newton Methods

255

Letting λ be the Lagrange multiplier vector multiplying the equality constraint AΔxnt = 0, we have the Lagrangian objective function 1 L(Δxnt , λ) = f (x)+(∇f (x))T Δxnt + (Δxnt )T ∇2 f (x)Δxnt +λT AΔxnt . (4.8.10) 2 By the first-order optimal condition

∂L(Δxnt , λ) = 0 and the constraint condition ∂Δxnt

AΔxnt = 0, it is easily obtained that

∇f (x) + ∇2 f (x)Δxnt + AT λ = 0 and which can be merged into  2 ∇ f (x) A

AT O



AΔxnt = 0

   Δxnt −∇f (x) . = O λ

(4.8.11)

Let x be the optimal solution to the equality constrained optimization problem (4.8.9), and let the corresponding optimal value of objective function f (x) be    p = inf f (x)Ax = b = f (x ). (4.8.12) As a stopping criterion for the Newton algorithm, one can use [53] λ2 (x) = (Δxnt )T ∇2 f (x)Δxnt .

(4.8.13)

Algorithm 4.11 takes a feasible point as the initial point, and thus is called the feasible-start Newton algorithm. This algorithm has the following features: (1) It is a descent method, because backtracking line search can ensure objective function descent at each iteration, and that the equality constraint is satisfied at each iteration point xk . (2) It requires a feasible point as an initial point. However, in many cases it is not easy to find a feasible point as an initial point. In these cases, it is necessary to consider an infeasible start Newton method. If xk is an infeasible point, consider the equality constrained optimization problem " # 1 min f (xk + Δxk ) = f (xk ) + (∇f (xk ))T Δxk + (Δxk )T ∇2 f (xk )Δxk Δxk 2 subject to A(xk + Δxk ) = b. Let λk+1 (= λk + Δλk ) be the Lagrange multiplier vector corresponding to the equality constraint A(xk + Δxk ) = b. Then the Lagrangian objective function is 1 L(Δxk , λk+1 ) = f (xk ) + (∇f (xk ))T Δxk + (Δxk )T ∇2 f (xk )Δxk 2 + λTk+1 (A(xk + Δxk ) − b).

256

Gradient Analysis and Optimization

Algorithm 4.11

Feasible-start Newton algorithm [53]

given: A feasible initial point x1 ∈ dom f with Ax1 = b, tolerance  > 0, parameters α ∈ (0, 0.5), β ∈ (0, 1). Put k = 1. repeat 1. Compute the gradient vector ∇f (xk ) and the Hessian matrix ∇2 f (xk ). 2. Use the preconditioned conjugate gradient algorithm to solve ⎤ ⎡ ⎤ ⎤⎡ ⎡ −∇f (xk ) Δxnt,k ∇2 f (xk ) AT ⎦=⎣ ⎦. ⎦⎣ ⎣ O A O λnt 3. Compute λ2 (xk ) = (Δxnt,k )T ∇2 f (xk )Δxnt,k . 4. exit if λ2 (xk ) < . 5. Let μ = 1. 6. while not converged do 7. Update μ ← βμ. 8. break if f (xk + μΔxnt,k ) < f (xk ) + αμ(∇f (xk ))T Δxnt,k . 9. end while 10. Update xk+1 ← xk + μΔxnt,k . return k ← k + 1. output: x ← xk .

From the optimal condition known that

∂L(Δxk , λk+1 ) ∂L(Δxk , λk+1 ) = 0 and = 0, it is easily ∂Δxk ∂λk+1

∇f (xk ) + ∇2 f (xk )Δxk + AT λk+1 = 0, AΔxk = −(Axk − b), which can be merged into  2 ∇ f (xk ) A

AT O



   Δxk ∇f (xk ) . =− Axk − b λk+1

Substituting λk+1 = λk + Δλk into the above equation, we have      2 ∇f (xk ) + AT λk ∇ f (xk ) AT Δxk . =− A O Δλk Axk − b

(4.8.14)

(4.8.15)

Algorithm 4.12 shows an infeasible-start Newton algorithm based on (4.8.15).

4.8 Newton Methods Algorithm 4.12

257

Infeasible-start Newton algorithm [53]

given: An initial point x1 ∈ Rn , any initial Lagrange multiplier λ1 ∈ Rp , an allowed tolerance  > 0, α ∈ (0, 0.5), and β ∈ (0, 1). Put k = 1. repeat 1. Compute the gradient vector ∇f (xk ) and the Hessian matrix ∇2 f (xk ). 2. Compute r(xk , λk ) via Equation (4.8.16). 3. exit if Axk = b and r(xk , λk )2 < . 4. Adopt the preconditioned conjugate gradient algorithm to solve the KKT equation (4.8.15), yielding the Newton step (Δxk , Δλk ). 5. Let μ = 1. 6. while not converged do 7. Update μ ← βμ. 8. break if f (xk + μΔxnt,k ) < f (xk ) + αμ(∇f (xk ))T Δxnt,k . 9. end while 10. Newton update





xk+1 xk Δxk = +μ . λk+1 λk Δλk

return k ← k + 1. output: x ← xk , λ ← λk .

Equation (4.8.15) reveals one of the stopping criterions of the Newton algorithms. Define the residual vector     rdual (xk , λk ) ∇f (xk ) + AT λk , (4.8.16) r(xk , λk ) = = rpri (xk , λk ) Axk − b where rdual (xk , λk ) = ∇f (xk ) + AT λk and rpri (xk , λk ) = Axk − b represent the dual and original residual vectors, respectively. Obviously, one stopping criterion of the infeasible-start Newton algorithm can be summarized as follows:         rdual (xk , λk ) Δxk 0 0 ≈ ⇔ ≈ ⇔ r(xk , λk ) 2 < (4.8.17) rpri (xk , λk ) Δλk 0 0 holds for a very small disturbance error > 0. Another stopping criterion of the infeasible-start Newton algorithm is that the equality constraint must be met. This stopping criterion can ensure that the convergence point of the Newton algorithm is feasible, although its initial point and most iteration points are allowed to be infeasible. Now consider the equality constrained minimization problem with complex variable: min f (z) subject to z

Az = b,

(4.8.18)

258

Gradient Analysis and Optimization

where z ∈ Cn , f : Cn → R is the convex function and twice differentiable and A ∈ Cp×n with rank(A) = p and p < n. Let (Δzk , Δz∗k ) denote the complex search direction at the kth iteration. Then the second-order Taylor expansion of f (zk , z∗k ) is given by f (zk + Δzk , z∗k + Δz∗k ) = f (zk , z∗k ) + (∇zk f (zk , z∗k ))T Δzk + (∇z∗k f (zk , z∗k ))T Δz∗k ⎡ 2 ⎤ ∂ f (zk , z∗k ) ∂ 2 f (zk , z∗k )   ⎥ Δzk  ⎢ ∂ z∗k ∂ zTk ∂ z∗k ∂ zH 1 k H T ⎢ ⎥ + (Δzk ) , (Δzk ) ⎣ 2 ∗ 2 ∂ f (zk , z∗k ) ∂ 2 f (zk , z∗k ) ⎦ Δzk ∂ zk ∂ zTk

∂ z k ∂ zH k

subject to A(zk + Δzk ) = b or AΔzk = 0 if zk is a feasible point. In other words, the complex Newton search direction can be determined by the equality constrained optimization problem: min

Δzk ,Δz∗ k

f (zk + Δzk , z∗k + Δz∗k ) subject to

AΔzk = 0.

(4.8.19)

Let λk+1 = λk + Δλk ∈ Rp be a Lagrange multiplier vector. Then the equality constrained minimization problem (4.8.19) can be transformed into the unconstrained minimization problem   min∗ f (zk + Δzk , z∗k + Δz∗k ) + (λk + Δλk )T AΔzk . Δzk ,Δzk ;Δλk

It is easy to obtain the complex Newton equation ⎡

∂ 2 f (zk , z∗k ) ⎢ ∂ z∗k ∂ zTk

⎢ ⎢ ∂ 2 f (z , z∗ ) ⎢ k k ⎢ ∂ z ∂ zT ⎣ k k A

∂ 2 f (zk , z∗k ) ∂ z∗k ∂ zH k ∂ 2 f (zk , z∗k ) ∂ z k ∂ zH k

O

⎤ AT ⎡ ⎤ ⎡ ⎥ Δz ⎤ ∇zk f (zk , z∗k ) + AT λk ⎥ k ⎥⎣ ∗⎦ ⎦ . (4.8.20) Δzk = − ⎣ ∇z∗k f (zk , z∗k ) O⎥ ⎥ ⎦ Δλk 0 O

Define the residual vector r(zk , z∗k , λk ) =



 ∇zk f (zk , z∗k ) + AT λk . ∇z∗k f (zk , z∗k )

(4.8.21)

The feasible-start complex Newton algorithm is shown in Algorithm 4.13. This algorithm is easily extended into the infeasible start complex Newton algorithm. For this end, the equality constrained minimization problem corresponding to Equation (4.8.19) becomes min

Δzk ,Δz∗ k

f (zk + Δzk , z∗k + Δz∗k ) subject to

A(zk + Δzk ) = b.

(4.8.22)

4.8 Newton Methods Algorithm 4.13

259

Feasible-start complex Newton algorithm

given: A feasible initial point z1 ∈ dom f with Az1 = b, tolerance  > 0, parameters α ∈ (0, 0.5), β ∈ (0, 1). Put k = 1. repeat 1. Compute the residual vector r(zk , z∗k , λk ) via Equation (4.8.21). 2. exit if r(zk , z∗k , λk )2 < . 3. Compute ∇z f (zk , z∗k ), ∇z∗ f (zk , z∗k ) and the full Hessian matrix ⎡ 2 ⎤ ∂ f (zk , z∗k ) ∂ 2 f (zk , z∗k ) ⎢ ∂ z ∗ ∂ zT ⎥ ∂ z∗k ∂ zH ⎢ ⎥ k Hk = ⎢ 2 k k∗ ⎥. 2 ∗ ⎣ ∂ f (zk , zk ) ∂ f (zk , zk ) ⎦ ∂ zk ∂ zTk ∂zk ∂ zH k 4. Use the preconditioned conjugate gradient algorithm to solve the Newton equation (4.8.20), yielding the Newton step (Δznt,k , Δz∗nt,k , Δλnt,k ). 5. Let μ = 1. 6. while not converged do 7. Update μ ← βμ. 8. break if f (zk +



μΔznt,k , z∗k

+

μΔz∗nt,k )


0. The relationship xi zi = μ, ∀ i = 1, . . . , m, is

262

Gradient Analysis and Optimization

also called the perturbed complementarity condition or complementary slackness. Obviously, if μ → 0 then xi zi → 0. Replacing the complementarity condition with perturbed complementarity, the KKT equations can be equivalently written as Ax = b,

x ≥ 0,

A y + z = c,

z ≥ 0,

T

(4.9.11)

T

x z = nμ. DEFINITION 4.18 [417] If the original problem has a feasible solution x > 0 and the dual problem has also a solution (y, z) with z > 0 then the original–dual optimization problem is said to meet the interior-point condition (IPC). If the interior-point condition is satisfied then the solution of Equation (4.9.11) is denoted (x(μ), y(μ), z(μ)) and is called the μ-center of the original problem (P ) and the dual problem (D). The set of all μ-centers is known as the center path of the original–dual problem. Substituting xk = xk−1 + Δxk , yk = yk−1 + Δyk and zk = zk−1 + Δzk into Equation (4.9.11), we have [416] AΔxk = b − Axk−1 , A Δyk + Δzk = c − AT yk−1 − zk−1 , T

zTk−1 Δxk + xTk Δzk−1 = nμ − xTk−1 zk−1 , equivalently written as ⎡

A ⎣ O zTk−1

O AT 0T

⎤⎡ ⎤ ⎡ ⎤ O Δxk b − Axk−1 I ⎦ ⎣Δyk ⎦ = ⎣c − AT yk−1 − zk−1 ⎦ . T xk−1 Δzk nμ − xTk−1 zk−1

(4.9.12)

The interior-point method based on the first-order optimal condition is called the first-order interior-point method, whose key step is to solve the KKT equation (4.9.12) to obtain the Newton step (Δxk , Δyk , Δzk ). This Newton step can be implemented by finding the inverse matrix directly, but an approach with better numerical performance is to use an iteration method for solving the KKT equation (4.9.12). Notice that since the leftmost matrix is not real symmetric, the conjugate gradient method and the preconditioned conjugate gradient method are unable to be used. Algorithm 4.14 gives a feasible-start original–dual interior-point algorithm. The above algorithm requires (x0 , y0 , z0 ) to be a feasible point. This feasible point can be determined by using Algorithm 4.15.

4.9 Original–Dual Interior-Point Method Algorithm 4.14

263

Feasible-start original–dual interior-point algorithm [416]

input: Accuracy parameter  > 0, updating parameter θ, 0 < θ < 1, feasible point (x0 , y0 , z0 ). initialization: Take μ0 = xT0 z0 /n, and set k = 1. repeat 1. Solve the KKT equation (4.9.12) to get the solution (Δxk , Δyk , Δzk ). 2. Update μk = (1 − θ)μk−1 . 3. Update the original variable, dual variable and Lagrange multiplier: xk = xk−1 + μk Δxk , yk = yk−1 + μk Δyk , zk = zk−1 + μk Δzk . exit if xTk zk < . return k ← k + 1. output: (x, y, z) ← (xk , yk , zk )

Algorithm 4.15

Feasible-point algorithm [416]

input: Accuracy parameter  > 0, penalty update parameter θ (0 < θ < 1) and threshold parameter τ > 0. initialization: x0 > 0, z0 > 0, y0 , xT0 z0 = nμ0 . Put k = 1. repeat 1. Compute the residuals rk−1 = b − AT xk−1 , rk−1 = c − AT yk−1 − zk−1 . c b 2. μ-update νk−1 = μk−1 /μ0 . 3. Solve the ⎤ ⎡ ⎤ ⎡ KKT equation ⎤ ⎡ Δf xk θνk−1 r0b A O O ⎥⎢ ⎥ ⎢ ⎥ ⎢ AT I ⎦ ⎣Δf yk ⎦ = ⎣ θνk−1 r0c ⎦. ⎣ O T T T f T zk−1 0 xk−1 Δ zk nμ − xk−1 zk−1 4. Update (xk , yk , zk ) = (xk−1 , yk−1 , zk−1 ) + (Δf xk , Δf yk , Δf zk ). 5. exit if max{xTk zk , b − AT xk , c − AT yk − zk } ≤ . return k ← k + 1. output: (x, y, z) ← (xk , yk , zk ).

4.9.3 Second-Order Original–Dual Interior-Point Method In the first-order original–dual interior-point method there exists the following shortcomings. • The matrix in the KKT equation is not symmetric, so it is difficult to apply the conjugate gradient method or the preconditioned conjugate gradient method for solving the KKT equation. • The KKT equation uses only the first-order optimal condition; it does not use the second-order information provided by the Hessian matrix. • It is difficult to ensure that iteration points are interior points. • It is inconvenient to extend this method to the general nonlinear optimization problem.

264

Gradient Analysis and Optimization

In order to overcome the above shortcomings, the original–dual interior-point method for nonlinear optimization consists of the following three basic elements. (1) Barrier function Limit the variable x to interior-points. (2) Newton method Solve the KKT equation efficiently. (3) Backtracking line search Determine a suitable step size. This interior-point method, combining the barrier function with the Newton method, is known as the second-order original–dual interior-point method. Define a slack variable z ∈ Rm which is a nonnegative vector z ≥ 0 such that h(x)−z = 0. Then the inequality constrained original problem (P ) can be expressed as the equality constrained optimization problem min f (x) subject to

h(x) = z, z ≥ 0.

(4.9.13)

To remove the inequality z ≥ 0, we introduce the Fiacco–McCormick logarithmic barrier m  log zi , (4.9.14) bμ (x, z) = f (x) − μ i=1

where μ > 0 is the barrier parameter. For very small μ, the barrier function bμ (x, z) and the original objective function f (x) have the same role at points for which the constraints are not close to zero. The equality constrained optimization problem (4.9.13) can be expressed as the unconstrained optimization problem + 0 m  min Lμ (x, z, λ) = f (x) − μ log zi − λT (h(x) − z) , (4.9.15) x, z, λ

i=1

where Lμ (x, z, λ) is the Lagrangian objective function and λ ∈ Rm is the Lagrange multiplier or the dual variable. Denote Z = Diag(z1 , . . . , zm ), Let ∇x Lμ =

Λ = Diag(λ1 , . . . , λm ).

(4.9.16)

∂Lμ ∂L ∂Lμ = 0, ∇z Lμ = Tμ = 0 and ∇λ Lμ = = 0. Then, the first∂ xT ∂z ∂λT

order optimal condition is given by ∇f (x) − (∇h(x))T λ = 0, −μ1 + ZΛ1 = 0, h(x) − z = 0. Here 1 is an m-dimensional vector with all entries equal to 1.

(4.9.17)

4.9 Original–Dual Interior-Point Method

265

In order to derive the Newton equation for the unconstrained optimization problem (4.9.15), consider the Lagrangian objective function Lμ (x + Δx, z + Δz, λ + Δλ) = f (x + Δx) − μ

m 

log(zi + Δzi )

i=1

  − (λ + Δλ)T h(x + Δx) − z − Δz , where 1 f (x + Δx) = f (x) + (∇f (x))T Δx + (Δx)T ∇2 f (x)Δx, 2 1 hi (x + Δx) = hi (x) + (∇hi (x))T Δx + (Δx)T ∇2 hi (x)Δx, 2 for i = 1, . . . , m. Letting ∇x Lμ =

∂Lμ ∂Lμ ∂Lμ = 0, ∇z Lμ = = 0 and ∇λ Lμ = = 0, ∂(Δx)T ∂(Δz)T ∂(Δλ)T

the Newton equation is given by [490] ⎡ ⎤ ⎤⎡ ⎤ ⎡ H(x, λ) O −(A(x))T Δx −∇f (x) + (∇h(x))T λ ⎣ O ⎦, ⎦ ⎣ Δz ⎦ = ⎣ μ1 − ZΛ1 Λ Z z − h(x) A(x) −I O Δλ

(4.9.18)

where H(x, λ) = ∇ f (x) − 2

m 

λi ∇2 hi (x),

A(x) = ∇h(x).

(4.9.19)

i=1

In (4.9.18), Δx is called the optimality direction and Δz the centrality direction, while Δλ is known as the feasibility direction [490]. The triple (Δx, Δz, Δλ) gives the Newton step, i.e. the update search direction of the interior-point method. From (4.9.18) it is easy to get [490] ⎡ ⎤⎡ ⎤ ⎡ ⎤ −H(x, λ) O (A(x))T Δx α ⎣ (4.9.20) O −Z−1 Λ −I ⎦ ⎣ Δz ⎦ = ⎣−β ⎦ , A(x) −I O Δλ γ where α = ∇f (x) − (∇h(x))T λ, β = μZ

−1

1 − λ,

γ = z − h(x).

(4.9.21) (4.9.22) (4.9.23)

The above three variables have the following meanings. (1) The vector γ measures the original infeasibility: γ = 0 means that x is a feasible point satisfying the equality constraint h(x) − z = 0; otherwise, x is an infeasible point.

266

Gradient Analysis and Optimization

(2) The vector α measures the dual infeasibility: α = 0 means that the dual variable λ satisfies the first-order optimal condition, and thus is feasible; otherwise, λ violates the first-order optimal condition and hence is infeasible. (3) The vector β measures the complementary slackness: β = 0, i.e., μZ−1 1 = λ, means that the complementary slackness zi λi = μ, ∀ i = 1, . . . , m and complementarity is completely satisfied when μ = 0. The smaller is the deviation of μ from zero, the smaller is the complementary slackness; the larger μ is, the larger is the complementary slackness. Equation (4.9.20) can be decomposed into the following two parts: Δz = Λ−1 Z(β − Δλ) and



−H(x, λ) A(x)

(A(x))T ZΛ−1



   α Δx = . γ + ZΛ−1 β Δλ

(4.9.24)

(4.9.25)

Importantly, the original Newton equation (4.9.20) becomes a sub-Newton equation (4.9.25) with smaller dimension. The Newton equation (4.9.25) is easy to solve iteratively via the preconditioned conjugate gradient algorithm. Once the Newton step (Δx, Δz, Δλ) is found from Equation (4.9.24) and the solution of Equation (4.9.25), we can perform the following updating formulae xk+1 = xk + ηΔxk , zk+1 = zk + ηΔzk ,

(4.9.26)

λk+1 = λk + ηΔλk . Here η is the common step size of the updates. 1. Modification of Convex Optimization Problems The key of the interior-point method is to ensure that each iteration point xk is an interior point such that h(xk ) > 0. However, for nonquadratic convex optimization, to simply select step size is insufficient to ensure the positiveness of the nonnegative variables. For this reason, it is necessary to modify the convex optimization problem appropriately. A simple modification of the convex optimization problem is achieved by introducing the merit function. Unlike the general cost function minimization, the minimization of the merit function has two purposes: as is necessary, to promote the appearance of iterative points close to the local minimum point of the objective function, but also to ensure that the iteration points are feasible points. Hence the main procedure using the merit function is twofold: (1) transform the constrained optimization problem into an unconstrained optimization problem;

4.9 Original–Dual Interior-Point Method

267

(2) reduce the merit function in order to move the iterative point closer to optimal solution of the original constrained optimization problem. Consider the classical Fiacco–McCormick merit function [152] ψρ,μ (x, z) = f (x) − μ

m 

ρ log zi + h(x) − z 22 . 2 i=1

(4.9.27)

Define the dual normal matrix (4.9.28) N(x, λ, z) = H(x, λ) + (A(x))T Z−1 ΛA(x), m let b(x, λ) = f (x) − i=1 λi log zi represent the barrier function and let the dual normal matrix N(x, λ, z) be positive definite. Then it can be shown [490] that the search direction (Δx, Δλ) determined by Equation (4.9.20) has the following properties. • If γ = 0 then



∇x b(x, λ) ∇λ b(x, λ)

 Δx ≤ 0. Δλ

T 

• There is a ρmin ≥ 0 such that for each ρ > ρmin , the following inequality holds:  T   Δx ∇x ψρ,μ (x, z) ≤ 0. Δλ ∇z ψρ,μ (x, z) In the above two cases, the equality holds if and only if (x, z) satisfies (4.9.17) for some λ. 2. Modification of Nonconvex Optimization Problems For a nonconvex optimization problem, the Hessian matrix H(x, λ) may not be positive semi-definite, hence the dual normal matrix N(x, λ, z) may not be positive definite. In this case, the Hessian matrix needs to be added a very small perturba˜ tion. That is to say, using H(x, λ) = H(x, λ) + δI to replace the Hessian matrix H(x, λ) in the dual normal matrix, we obtain ˜ N(x, λ, z) = H(x, λ) + δI + (A(x))T Z−1 ΛA(x). Although interior-point methods usually take only a few iterations to converge, they are difficult to handle for a higher-dimensional matrix because the complexity of computing the step direction is O(m6 ), where m is the dimension of the matrix. As a result, on a typical personal computer (PC), generic interior-point solvers cannot handle matrices with dimensions larger than m = 102 . However, image and video processing often involve matrices of dimension m = 104 to 105 and web search and bioinformatics can easily involve matrices of dimension m = 106 and beyond. In these real applications, generic interior point solvers are too limited, and only first-order gradient methods are practical [298].

268

Gradient Analysis and Optimization

Exercises 4.1

4.2

Let y be a real-valued measurement data vector given by y = αx+v, where α is a real scalar, x represents a real-valued deterministic process, the additive noise vector v has zero-mean, and covariance matrix Rv = E{vvT }. Find an optimal filter vector w such that α ˆ = wT y is an unbiased estimator with minimum variance. Let f (t) be a given function. Consider the minimization of the quadratic function: 1 2 min Q(x) = min (f (t) − x0 − x1 t − · · · − xn tn ) dt. 0

4.3

4.4

Determine whether the matrix associated with the set of linear equations is ill-conditioned. Given a function f (X) = log | det(X)|, where X ∈ Cn×n and | det(X)| is the absolute value of the determinant det(X). Find the conjugate gradient matrix df /dX∗ . Consider the equation y = Aθ + e, where e is an error vector. Define the def weighted sum of squared errors Ew = eH We, where W is a Hermitian positive definite matrix which weights the errors. (a) Find the solution for the parameter vector θ such that Ew is minimized. This solution is called the weighted least squares estimate of θ. (b) Use the LDLH decomposition W = LDLH to prove that the weighted least squares criterion is equivalent to the prewhitening of the error or data vector.

4.5 4.6

Let a cost function be f (w) = wH Re w and let the constraint on the filter be Re(wH x) = b, where b is a constant. Find the optimal filter vector w. Explain whether the following constrained optimization problems have solutions: (a) min (x1 + x2 ) subject to x21 + x22 = 2, 0 ≤ x1 ≤ 1, 0 ≤ x2 ≤ 1; (b) min (x1 + x2 ) subject to x21 + x22 ≤ 1, x1 + x2 = 4; (c) min (x1 x2 ) subject to x1 + x2 = 3.

4.7

4.8 4.9

Consider the constrained optimization problem min{(x − 1)(y + 1)} subject to x − y = 0. Use the Lagrange multiplier method to show that the minimum point is (1, 1) and the Lagrange multiplier λ = 1. For a Lagrangian function ψ(x, y) = (x − 1)(y + 1) − 1(x − y), show that ψ(x, y) has a saddle point at (0, 0), i.e., the point (0, 0) cannot minimize ψ(x, y). Solve the constrained optimization problem min{J(x, y, z) = x2 + y 2 + z 2 } subject to 3x + 4y − z = 25. The steepest descent direction of an objective function J(M) is given by dJ(M) − . Given a positive definite matrix W, determine which of the gradient dM

matrices −W

dJ(M) dM

and −

dJ(M) W dM

is the steepest descent direction of J(M),

Exercises

269

and write down the corresponding gradient descent algorithm. 4.10 Let an observation data vector be produced by the linear regression model y = Xβ + , E{ } = 0, E{ T } = σ 2 I. We want to design a filter matrix A whose output vector e = Ay satisfies E{e− } = 0, and minimizes E{(e− )T (e− )}. Show that this optimization problem is equivalent to   min tr(AT A) − 2tr(A) subject to AX = O, where O is a zero matrix. 4.11 Show that the solution matrix of the optimization problem   min tr(AT A) − 2tr(A) ˆ = I − XX† . subject to AX = O (zero matrix) is given by A 4.12 Show that the unconstrained minimization problem 3 4 min (y − Xβ)T (V + XXT )† (y − Xβ) and the constrained minimization problem   min (y − Xβ)V† (y − Xβ) subject to

(I − VV† )Xβ = (I − VV† )y

have the same solution vector β. 4.13 Solve the constrained optimization problem   min tr(AT A) − 2tr(A) subject to Ax = 0. A

4.14 Show that the constrained optimization problem 1 min xT x x 2

subject to Cx = b

has the unique solution x∗ = C† b. 4.15 Consider matrices Y ∈ Rn×m , Z ∈ Rn×(n−m) and let their columns constitute a linearly independent set. If a solution vector subject to Ax = b is expressed as x = YxY + ZxZ , where xY and xZ are some m × 1 and (n − m) × 1 vectors, respectively, show that x = Y(AY)−1 b + ZxZ .   4.16 If a constrained optimization problem is min tr(AVAT ) subject to AX = W, show that A = W(XT V0† X)† XT V†0 +Q(I−V0 V0† ), where V0 = V+XXT and Q is any matrix. 4.17 Assume for a matrix X that rank(X) ≤ r. Show that, for all semi-orthogonal matrices A such that AT A = Ir and all matrices Z ∈ Rn×r , the following result is true: 4 3 min tr (X − ZAT )(X − ZAT )T = 0. X

270

Gradient Analysis and Optimization

4.18 For p > 1 and ai ≥ 0, i = 1, . . . , n, show that for each set of nonnegative n real numbers {x1 , . . . , xn } such that i=1 xqi = 1, where q = p/(p − 1), the relationship 1/p  n n   p ai xi ≤ ai i=1

i=1

is satisfied, and equality holds if and only if a1 = · · · = an = 0 or xqi = p 1/p api ( k=1 apk ) , i = 1, . . . , n. This relationship is called the representation p 1/p theorem of ( i=1 api ) . 4.19 Show that the solution matrix of the constrained optimization problem   min tr(AT A) − 2tr(A) subject to AX = O (the zero matrix) ˆ = I − XX† . (Hint: The matrix constraint condition AX = O is given by A is equivalent to the scalar constraint condition tr(LAX) = 0, ∀ L.)

5 Singular Value Analysis

Beltrami (1835–1899) and Jordan (1838–1921) are recognized as the founders of singular value decomposition (SVD): Beltrami in 1873 published the first paper on SVD [34]. One year later, Jordan published his own independent derivation of SVD [236]. Now, the SVD and its generalizations are one of the most useful and efficient tools for numerical linear algebra, and are widely applied in statistical analysis, and physics and in applied sciences, such as signal and image processing, system theory, control, communication, computer vision and so on. This chapter presents first the concept of the stability of numerical computations and the condition number of matrices in order to introduce the necessity of the SVD of matrices; then we discuss SVD and generalized SVD together with their numerical computation and applications.

5.1 Numerical Stability and Condition Number In many applications such as information science and engineering, it is often necessary to consider an important problem: in the actual observation data there exist some uncertainties or errors, and, furthermore, numerical calculation of the data is always accompanied by error. What is the impact of these errors? Is a particular algorithm numerically stable for data processing? In order to answer these questions, the following two concepts are extremely important: (1) the numerical stability of various kinds of algorithm; (2) the condition number or perturbation analysis of the problem of interest. Let f be some application problem, let d ∈ D be the data without noise or disturbance, where D denotes a data group, and let f (d ) ∈ F represent the solution of f , where F is a solution set. Given observed data d ∈ D, we want to evaluate f (d). Owing to background noise and/or observation error, f (d) is usually different from f (d ). If f (d) is “close” to f (d ) then the problem f is “well-conditioned”. On the contrary, if f (d) is obviously different from f (d ) even when d is very close to d then we say that the problem f is “ill-conditioned”. If there is no further 271

272

Singular Value Analysis

information about the problem f , the term “approximation” cannot describe the situation accurately. In perturbation theory, a method or algorithm for solving a problem f is said to be numerically stable if its sensitivity to a perturbation is not larger than the sensitivity inherent in the original problem. A stable algorithm can guarantee that the solution f (d) with a slight perturbation Δd of d approximates closely the solution without the perturbation. More precisely, f is stable if the approximate solution f (d) is close to the solution f (d ) without perturbation for all d ∈ D close to d . A mathematical description of numerical stability is discussed below. First, consider the well-determined linear equation Ax = b, where the n × n coefficient matrix A has known entries and the n × 1 data vector b is also known, while the n × 1 vector x is an unknown parameter vector to be solved. Naturally, we will be interested in the stability of this solution: if the coefficient matrix A and/or the data vector b are perturbed, how will the solution vector x be changed? Does it maintain a certain stability? By studying the influence of perturbations of the coefficient matrix A and/or the data vector b, we can obtain a numerical value, called the condition number, describing an important characteristic of the coefficient matrix A. For the convenience of analysis, we first assume that there is only a perturbation δb of the data vector b, while the matrix A is stable. In this case, the exact solution vector x will be perturbed to x + δ x, namely A(x + δ x) = b + δ b.

(5.1.1)

δ x = A−1 δ b,

(5.1.2)

This implies that

since Ax = b. By the property of the Frobenius norm, Equation (5.1.2) gives

δ x 2 ≤ A−1 F δ b 2 .

(5.1.3)

Similarly, from the linear equation Ax = b we get

b 2 ≤ A F x 2 .

(5.1.4)

From Equations (5.1.3) and (5.1.4) it is immediately clear that   δ b 2

δ x 2 ≤ A F A−1 F .

x 2

b 2

(5.1.5)

Next, consider the influence of simultaneous perturbations δx and δA. In this case, the linear equation becomes (A + δA)(x + δ x) = b.

5.1 Numerical Stability and Condition Number

273

From the above equation it can be derived that   δx = (A + δA)−1 − A−1 b = {A−1 (A − (A + δA)) (A + δA)−1 }b = −A−1 δA(A + δA)−1 b = −A−1 δA(x + δx)

(5.1.6)

from which we have

δx 2 ≤ A−1 F δA F x + δx 2 , namely   δA F

δx 2 ≤ A F A−1 F .

x + δx 2

A F

(5.1.7)

Equations (5.1.5) and (5.1.7) show that the relative error of the solution vector x is proportional to the numerical value cond(A) = A F A−1 F ,

(5.1.8)

where cond(A) is called the condition number of the matrix A and is sometimes denoted κ(A). If a small perturbation of a coefficient matrix A causes only a small perturbation of the solution vector x, then the matrix A is said to be a well-conditioned. If, however, a small perturbation of A causes a large perturbation of x then the matrix A is ill-conditioned. The condition number characterizes the influence of errors in the entries of A on the solution vector; hence it is an important index measuring the numerical stability of a set of linear equations. Further, consider an over-determined linear equation Ax = b, where A is an m × n matrix with m > n. An over-determined equation has a unique linear least squares solution given by AH Ax = AH b, i.e., x = (AH A)−1 AH b. It is easy to show (see Subsection 5.2.2) that   2  cond AH A = cond(A) .

(5.1.9)

(5.1.10)

From Equations (5.1.5) and (5.1.7) it can be seen that the effects of the datavector error δ b and the coefficient-matrix error δA on the solution-vector error for the over-determined equation (5.1.9) are each proportional to the square of the condition number of A. EXAMPLE 5.1

Consider the matrix



⎤ 1 1 A = ⎣δ 0⎦ , 0 δ

274

Singular Value Analysis

where δ is small. The condition number of A is of the same order of magnitude as δ −1 . Since   1 + δ2 1 H B=A A= , 1 1 + δ2 the condition number of AH A is of order δ −2 . In addition, if we use the QR decomposition A = QR to solve an over-determined equation Ax = b then cond(Q) = 1,

cond(A) = cond(QH A) = cond(R),

(5.1.11)

since QH Q = I. In this case the effects of the errors of b and A will be proportional to the condition number of A, as shown in Equations (5.1.5) and (5.1.7), respectively. The above fact tells us that, when solving an over-determined matrix equation, the QR decomposition method has better numerical stability (i.e., a smaller condition number) than the least squares method. If the condition number of A is large then the matrix equation Ax = b is illconditioned. In this case, for a data vector b close to the ideal vector b without perturbation, the solution associated with b will be far from the ideal solution associated with b . A more effective approach than QR decomposition for solving this class of illconditioned linear equations is the total least squares method (which will be presented in Chapter 6), whose basis is the singular value decomposition of matrices. This is the topic of the next section.

5.2 Singular Value Decomposition (SVD) Singular value decomposition is one of the most basic and most important tools in modern numerical analysis (especially in numerical computations). This section presents the definition and geometric interpretation of SVD together with the properties of singular values.

5.2.1 Singular Value Decomposition The SVD was proposed first by Beltrami in 1873 for a real square matrix A [34]. Consider the bilinear function f (x, y) = xT Ay,

A ∈ Rn×n ;

by introducing the linear transformation x = Uξ and y = Vη Beltrami transformed the bilinear function into f (x, y) = ξ T Sη, where S = UT AV.

(5.2.1)

5.2 Singular Value Decomposition (SVD)

275

Beltrami observed that if both U and V are constrained to be orthogonal matrices then in their choice of entries there exist n2 − n degrees of freedom, respectively. Beltrami proposed using these degrees of freedom to make all off-diagonal entries of S zero, so that the matrix S = Σ = Diag(σ1 , σ2 , . . . , σn ) becomes diagonal. Then premultiplying (5.2.1) by U, postmultiplying it by VT and making use of the orthonormality of U and V, we immediately get A = UΣVT .

(5.2.2)

This is the singular value decomposition (SVD) of a real square matrix A obtained by Beltrami in 1873 [34]. In 1874, Jordan independently derived the SVD of real square matrices [236]. The history of the invention of SVD can be found in MacDuffee’s book [309, p. 78] or in the paper of Stewart [454], which reviews the early development of SVD in detail. Later, Autonne in 1902 [19] extended the SVD to complex square matrices, while Eckart and Young [141] in 1939 extended it further to general complex rectangular matrices; therefore, the SVD theorem for a complex rectangular matrix is usually called the Autonee–Eckart–Young theorem, stated below. THEOREM 5.1 (SVD) Let A ∈ Rm×n (or Cm×n ); then there exist orthogonal (or unitary) matrices U ∈ Rm×m (or U ∈ Cm×m ) and V ∈ Rn×n (or V ∈ Cn×n ) such that   Σ1 O T H , (5.2.3) A = UΣV (or UΣV ), Σ = O O where Σ1 = Diag(σ1 , . . . , σr ) with diagonal entries σ1 ≥ · · · ≥ σr > 0,

r = rank(A).

(5.2.4)

The above theorem was first shown by Eckart and Young [141] in 1939, but the proof by Klema and Laub [250] is simpler. The numerical values σ1 , . . . , σr together with σr+1 = · · · = σn = 0 are called the singular values of the matrix A. EXAMPLE 5.2 A singular value σi of an m×n matrix A is called a single singular value of A if σi = σj , ∀ j = i. Here are several explanations and remarks on singular values and SVD. 1. The SVD of a matrix A can be rewritten as the vector form A=

r 

σi ui viH .

(5.2.5)

i=1

This expression is sometimes said to be the dyadic decomposition of A [179].

276

Singular Value Analysis

2. Suppose that the n × n matrix V is unitary. Postmultiply (5.2.3) by V to get AV = UΣ, whose column vectors are given by " σi ui , i = 1, 2, . . . , r, Avi = (5.2.6) 0, i = r + 1, r + 2, . . . , n. Therefore the column vectors vi of V are known as the right singular vectors of the matrix A, and V is called the right singular-vector matrix of A. 3. Suppose that the m × m matrix U is unitary. Premultiply (5.2.3) by UH to yield UH A = ΣV, whose column vectors are given by " σi viT , i = 1, 2, . . . , r, H ui A = (5.2.7) 0, i = r + 1, r + 2, . . . , n. The column vectors ui are called the left singular vectors of the matrix A, and U is the left singular-vector matrix of A. 4. When the matrix rank r = rank(A) < min{m, n}, because σr+1 = · · · = σh = 0 with h = min{m, n}, the SVD formula (5.2.3) can be simplified to A = Ur Σr VrH ,

(5.2.8)

where Ur = [u1 , . . . , ur ], Vr = [v1 , . . . , vr ] and Σr = Diag(σ1 , . . . , σr ). Equation (5.2.8) is called the truncated singular value decomposition or the thin singular value decomposition of the matrix A. In contrast, Equation (5.2.3) is known as the full singular value decomposition. H 5. Premultiplying (5.2.6) by uH i , and noting that ui ui = 1, it is easy to obtain uH i Avi = σi ,

i = 1, 2, . . . , min{m, n},

which can be written in matrix form as   Σ1 O , Σ1 = Diag(σ1 , . . . , σr ). UH AV = Σ = O O

(5.2.9)

(5.2.10)

Equations (5.2.3) and (5.2.10) are two definitive forms of SVD. 6. From (5.2.3) it follows that AAH = UΣ2 UH . This shows that the singular value σi of an m × n matrix A is the positive square root of the corresponding nonnegative eigenvalue of the matrix product AAH . 7. If the matrix Am×n has rank r then • the leftmost r columns of the m × m unitary matrix U constitute an orthonormal basis of the column space of the matrix A, i.e., Col(A) = Span{u1 , . . . , ur }; • the leftmost r columns of the n×n unitary matrix V constitute an orthonormal basis of the row space of A or the column space of AH , i.e., Row(A) = Span{v1 , . . . , vr }; • the rightmost n − r columns of V constitute an orthonormal basis of the null space of the matrix A, i.e., Null(A) = Span{vr+1 , . . . , vn };

5.2 Singular Value Decomposition (SVD)

277

• the rightmost m − r columns of U constitute an orthonormal basis of the null space of the matrix AH , i.e., Null(AH )=Span{ur+1 , . . . , um }.

5.2.2 Properties of Singular Values THEOREM 5.2 (Eckart–Young theorem) [141] Cm×n are given by σ1 ≥ σ2 ≥ · · · ≥ σr ≥ 0,

If the singular values of A ∈

r = rank(A),

then σk =

min ( E spec |rank(A + E) ≤ k − 1) ,

E∈Cm×n

k = 1, . . . , r,

(5.2.11)

and there is an error matrix Ek with Ek spec = σk such that rank(A + Ek ) = k − 1,

k = 1, . . . , r.

(5.2.12)

The Eckart–Young theorem shows that the singular value σk is equal to the minimum spectral norm of the error matrix Ek such that the rank of A + Ek is k − 1. An important application of the Eckart–Young theorem is to the best rank-k approximation of the matrix A, where k < r = rank(A). Define k  Ak = σi ui viH , k < r; (5.2.13) i=1

then Ak is the solution of the optimization problem Ak = arg min A − X 2F ,

k < r,

(5.2.14)

rank(X)=k

and the squared approximation error is given by 2 + · · · + σr2 .

A − Ak 2F = σk+1

(5.2.15)

THEOREM 5.3 [215], [226] Let A be an m × n matrix with singular values σ1 ≥ · · · ≥ σr , where r = min{m, n}. If the p × q matrix B is a submatrix of A with singular values γ1 ≥ · · · ≥ γmin{p,q} then σi ≥ γ i ,

i = 1, . . . , min{p, q}

(5.2.16)

and γi ≥ σi+(m−p)+(n−q) ,

i ≤ min{p + q − m, p + q − n}.

(5.2.17)

This is the interlacing theorem for singular values. The singular values of a matrix are closely related to its norm, determinant and condition number.

278

Singular Value Analysis

1. Relationship between Singular Values and Norms The spectral norm of a matrix A is equal to its maximum singular value, namely

A spec = σ1 .

(5.2.18)

Since the Frobenius norm A F of a matrix A is of unitary invariance, i.e.,

UH AV F = A F , we have 

A F =

n m  

1/2 |aij |

2

= UH AV F = Σ F =

 σ12 + · · · + σr2 . (5.2.19)

i=1 j=1

That is to say, the Frobenius norm of any matrix is equal to the positive square root of the sum of all nonzero squared singular values of the matrix. 2. Relationship between Singular Values and Determinant Let A be an n × n matrix. Since the absolute value of the determinant of any unitary matrix is equal to 1, from Theorem 5.1 it follows that | det(A)| = | det(UΣVH )| = | det Σ| = σ1 · · · σn .

(5.2.20)

If all the σi are nonzero then | det(A)| =  0, which shows that A is nonsingular. If there is at least one singular value σi = 0 (i > r) then det(A) = 0, namely A is singular. This is the reason why the σi , i = 1, . . . , min{m, n} are known as the singular values. 3. Relationship between Singular Values and Condition Number For a given m × n matrix A, its condition number can be defined by its singular values as cond(A) = σ1 /σp ,

p = min{m, n}.

(5.2.21)

From the definition formula (5.2.21) it can be seen that the condition number is a positive number larger than or equal to 1 since σ1 ≥ σp . Obviously, because there is at least one singular value σp = 0, the condition number of a singular matrix is infinitely large. If the condition number is not infinite but is large then the matrix A is said to be nearly singular. This implies that when the condition number is large, the linear dependence between the row (or column) vectors is strong. On the other hand, the condition number of an orthogonal or unitary matrix is equal to 1 by Equation (5.1.8). In this sense, the orthogonal or unitary matrix is of “ideal condition”. For an over-determined linear equation Ax = b, the SVD of AH A is AH A = VΣ2 VH ,

(5.2.22)

i.e., the maximum and minimum singular values of the matrix AH A are respectively

5.2 Singular Value Decomposition (SVD)

279

the squares of the maximum and minimum singular values of A, and thus cond(AH A) =

σ12 = (cond(A))2 . σn2

(5.2.23)

In other words, the condition number of the matrix AH A is the square of the condition number of A. This implies that, for a given data matrix X, its SVD has better numerical stability than the EVD of the covariance matrix XXH . The following are equality relationships for singular values [307]. 1. Am×n and its Hermitian matrix AH have the same nonzero singular values. 2. The nonzero singular values of a matrix Am×n are the positive square root of the nonzero eigenvalues of AAH or AH A. 3. There is a single singular value σ > 0 of a matrix Am×n if and only if σ 2 is a single eigenvalue of AAH or AH A. 4. If p = min{m, and σ1 , . . . , σp are the singular values of an m × n matrix A,  H n},  p then tr A A = i=1 σi2 . 5. The absolute value of the determinant of an n × n matrix is equal to the product of its singular values, namely | det(A)| = σ1 · · · σn . 6. The spectral norm of a matrix A is equal to its maximum singular value, namely

A spec = σmax . 7. If m ≥ n then, for a matrix Am×n , one has * )  σmin (A) = min (xH AH Ax)1/2  xH x = 1, x ∈ Cn , * )  σmax (A) = max (xH AH Ax)1/2  xH x = 1, x ∈ Cn . 8. If an m × m matrix A is nonsingular, then   1/2  1 xH (A−1 )H A−1 x  n . = max  x = 0, x ∈ C  σmin (A) xH x 9. If σ1 , . . . , σp are the nonzero singular values of A ∈ Cm×n , with p = min{m, n},

then the matrix

O AH

A O

has 2p nonzero singular values σ1 , . . . , σp , −σ1 , . . . ,

−σp and |m − n| zero singular values.

10. If A = U ΣO1 O VH is the SVD of the m×n matrix A then the Moore–Penrose O inverse matrix of A is given by   −1 Σ1 O † UH . A =V O O 11. When P and Q are respectively an m × m and an n × n unitary matrix, the SVD of PAQH is given by ˜ V ˜H PAQH = UΣ

(5.2.24)

˜ = PU and V ˜ = QV. That is to say, the two matrices PAQH and A have with U

280

Singular Value Analysis

the same singular values, i.e., the singular values have the unitary invariance, but their singular vectors are different. The following summarizes the inequality relations of singular values [214], [215], [275], [93], [307]. (1) If A and B are m × n matrices then, for p = min{m, n}, one has σi+j−1 (A + B) ≤ σi (A) + σj (B),

1 ≤ i, j ≤ p, i + j ≤ p + 1.

In particular, when j = 1, σi (A + B) ≤ σi (A) + σ1 (B) holds for i = 1, . . . , p. (2) Given Am×n and Bm×n , one has σmax (A + B) ≤ σmax (A) + σmax (B). (3) If A and B are the m × n matrices then p 

[σj (A + B) − σj (A)]2 ≤ B 2F ,

p = min{m, n}.

j=1

(4) If the singular values of A satisfies σ1 (A) ≥ · · · ≥ σm (A) then k  j=1

[σm−k+j (A)]2 ≤

k 

aH j aj ≤

j=1

k 

[σj (A)]2 ,

k = 1, 2, . . . , m.

j=1

(5) If p = min{m, n}, and the singular values of Am×n and Bm×n are arranged as σ1 (A) ≥ · · · ≥ σp (A), σ1 (B) ≥ · · · ≥ σp (B) and σ1 (A + B) ≥ · · · ≥ σp (A + B), then σi+j−1 (ABH ) ≤ σi (A)σj (B),

1 ≤ i, j ≤ p, i + j ≤ p + 1.

(6) If B is an m × (n − 1) matrix obtained by deleting any column of an m × n matrix A, and their singular values are arranged in descending order, then σ1 (A) ≥ σ1 (B) ≥ σ2 (A) ≥ σ2 (B) ≥ · · · ≥ σh (A) ≥ σh (B) ≥ 0, where h = min{m, n − 1}. (7) The maximum singular value of a matrix Am×n satisfies the inequality  σmax (A) ≥

1 tr(AH A) n

1/2 .

5.2.3 Rank-Deficient Least Squares Solutions In applications of singular value analysis, it is usually necessary to use a low-rank matrix to approximate a noisy or disturbed matrix. The following theorem gives an evaluation of the quality of approximation.

5.2 Singular Value Decomposition (SVD)

281

p THEOREM 5.4 Let A = σi ui viT be the SVD of A ∈ Rm×n , where p = i=1 rank(A). If k < p and Ak = ki=1 σi ui viT is the rank-k approximation of A, then the approximation quality can be respectively measured by the spectral norm or the Frobenius norm, min rank(B)=k

A − B spec = A − Ak spec = σk+1 ,

or

 min rank(B)=k

A − B F = A − Ak F =

q 

(5.2.25)

1/2 σi2

(5.2.26)

i=k+1

with q = min{m, n}. Proof

See, e.g., [140], [328], [226].

In signal processing and system theory, the most common system of linear equations Ax = b is over-determined and rank-deficient; namely, the row number m of the matrix A ∈ Cm×n is larger than its column number n and r = rank(A) < n. Let the SVD of A be given by A = UΣVH , where Σ = Diag(σ1 , . . . , σr , 0, . . . , 0). Consider (5.2.27) G = VΣ† UH , where Σ† = Diag(1/σ1 , . . . , 1/σr , 0, . . . , 0). By the property of singular values it is known that G is the Moore–Penrose inverse matrix of A. Hence ˆ = Gb = VΣ† UH b x

(5.2.28)

which can be represented as xLS =

r 

(uH i b/σi )vi .

(5.2.29)

i=1

The corresponding minimum residual is given by 2 2 ρLS = AxLS − b 2 = 2[ur+1 , . . . , um ]H b22 .

(5.2.30)

The approach to solving the least squares problem via SVD is simply known as the SVD method. Although, in theory, when i > r the singular values σi = 0, the computed singular values σˆi , i > r, are not usually equal to zero and sometimes even have quite a large perturbation. In these cases, an estimate of the matrix rank r is required. In signal processing and system theory, the rank estimate rˆ is usually called the effective rank. Effective-rank determination is carried out by one of the following two common methods. 1. Normalized Singular Value Method Compute the normalized singular values σ ¯i =

σ ˆi , σ ˆ1

(5.2.31)

282

Singular Value Analysis

and select the largest integer i satisfying the criterion σ ¯i ≥ as an estimate of the effective rank rˆ. Obviously, this criterion is equivalent to choosing the maximum integer i satisfying σ ˆi ≥ σ ˆ1

(5.2.32)

as rˆ; here is a very small positive number, e.g., = 0.1 or = 0.05. 2. Norm Ratio Method Let an m × n matrix Ak be the rank-k approximation to the original m × n matrix A. Define the Frobenius norm ratio as 

Ak F σ12 + · · · + σk2 ν(k) = = 2 , h = min{m, n}, (5.2.33)

A F σ1 + · · · + σh2 and choose the minimum integer k satisfying ν(k) ≥ α

(5.2.34)

as the effective rank estimate rˆ, where α is at the threshold of 1, e.g., α = 0.997 or 0.998. After the effective rank rˆ has been determined via the above two criteria, ˆ LS = x

rˆ 

(ˆ uH σi )ˆ vi i b/ˆ

(5.2.35)

i=1

can be regarded as a reasonable approximation to the LS solution xLS . Clearly, this solution is the LS solution of the linear equation Arˆx = b, where Arˆ =

rˆ 

σi ui viH .

(5.2.36)

i=1

In LS problems, using Arˆ instead of A corresponds to filtering out the smaller singular values. This filtering is effective for a noisy matrix A. It is easily seen ˆ LS given by Equation (5.2.35) still contains n parameters. that the LS solution x However, by the rank-deficiency of the matrix A we know that the unknown parameter vector x contains only r independent parameters; the other parameters are linearly dependent on r independent parameters. In many engineering applications, we naturally want to find r independent parameters other than the n parameters containing redundancy components. In other words, our objective is to estimate only the principal parameters and to eliminate minor components. This problem can be solved via the low-rank total least squares (TLS) method, which will be presented in Chapter 6.

5.3 Product Singular Value Decomposition (PSVD)

283

5.3 Product Singular Value Decomposition (PSVD) In the previous section we discussed the SVD of a single matrix. This section presents the SVD of the product of two matrices.

5.3.1 PSVD Problem By the product singular value decomposition (PSVD) is meant the SVD of a matrix product BT C. Consider the matrix product A = BT C,

B ∈ Cp×m ,

C ∈ Cp×n ,

rank(B) = rank(C) = p.

(5.3.1)

In principle, the PSVD is equivalent to the direct SVD of the matrix product. However, if we first compute the product of two matrices, and then calculate the PSVD then the smaller singular values are often considerably disturbed. To illustrate this, look at the following example [135]. EXAMPLE 5.3

Let



BT =

1 ξ −1 ξ

1 B C= √ 2 T

 , 

1 C= √ 2 1−ξ −1 − ξ



1 1 −1 1  1+ξ . −1 + ξ

 , (5.3.2)

Clearly, C is an orthogonal matrix and the two columns [1, −1]T and [ξ, ξ]T of BT are orthogonal to each √ other. The real singular values of the matrix product BT C √ are σ1 = 2 and σ2 = 2|ξ|. However, if |ξ| is smaller than the cutoff error , then

1 1 1 the floating point calculation result of Equation (5.3.2) is BT C = √ −1 −1 2 √ whose singular values are σ ˆ1 = 2 and σ ˆ2 = 0. In this case, σ ˆ1 = σ1 and σ ˆ 2 ≈ σ2 . If |ξ| > 1/ , then the floating point calculation result of the matrix product is

√ 1 −ξ ξ T B C = √ −ξ ξ , whose computed singular values are σ ˆ1 = 2 |ξ| and σ ˆ2 = 0. 2

ˆ2 is obviously different from σ2 . In this case σ ˆ1 is clearly larger than σ1 , and σ Laub et al. [283] pointed out that when a linear system is close to uncontrollable and unobservable an accurate calculation for the small singular values is very important because, if a nonzero small singular value is calculated as zero, it may lead to a wrong conclusion: a minimum-phase system could be determined as a nonminimum phase system. The above examples show that to use the direct SVD of the matrix product BT C is not numerically desirable. Hence, it is necessary to consider how to compute the SVD of A = BT C such that the PSVD has an accuracy close to that of B and C. This is the so-called product singular value decomposition problem. Such a PSVD was first proposed by Fernando and Hammarling in 1988 [151], and is described in the following theorem.

284

Singular Value Analysis

THEOREM 5.5 (PSVD) [151] Let BT ∈ Cm×p , C ∈ Cp×n . Then there are two unitary matrices U ∈ Cm×m , V ∈ Cn×n and a nonsingular matrix Q ∈ Cp×p such that ⎤ ⎡ I ⎦, UBH Q = ⎣ OB (5.3.3) ΣB ⎤ ⎡ OC ⎦, Q−1 CVH = ⎣ (5.3.4) I ΣC where ΣB = Diag(s1 , . . . , sr ), 1 > s1 ≥ · · · ≥ sr > 0, ΣC = Diag(t1 , . . . , tr ), 1 > t1 ≥ · · · ≥ tr > 0, and s2i + t2i = 1, i = 1, . . . , r. By Theorem 5.5, we can deduce that UBH CVH = Diag(OC , OB , ΣB ΣC ). Hence the singular values of the matrix product BH C consist of both zero and nonzero singular values given by si ti , i = 1, . . . , r.

5.3.2 Accurate Calculation of PSVD The basic idea of the accurate calculation algorithm for PSVD in [135] is as follows: after any matrix A is multiplied by an orthogonal matrix, the singular values of A remain unchanged. Hence, for matrices B and C, if we let B = TBU,

C = (TT )−1 CV,

(5.3.5)

where T is nonsingular and U and V are orthogonal matrices, then B T C = UT BT TT (TT )−1 CV = UT (BT C)V and BT C have the exactly same singular values (including zero singular values). Given B ∈ Rp×m and C ∈ Rp×n , p ≤ min{m, n}, denote the row vectors of B as τ bi , i = 1, 2, . . . , p. Drmac’s PSVD algorithm is shown in Algorithm 5.1.   Σ⊕O , so the SVD of BT C After performing Algorithm 5.1, we get V, QF and O is given by U(BT C)VT = Σ, where U=



 T V I

 Σ⊕O . Σ= O 

QTF ,

(5.3.6)

The above algorithm for computing the PSVD of two matrices has been extended to the accurate calculation of the PSVD of three matrices [137]; see Algorithm 5.2.

5.4 Applications of Singular Value Decomposition Algorithm 5.1

285

PSVD(B, C) [135]

input: B ∈ Rp×m , C ∈ Rp×n , p ≤ min{m, n}. Denote Bτ = [bτ1 , . . . , bτp ]. 1. Compute Bτ = Diag(bτ1 2 , . . . , bτp 2 ). Let B1 = B†τ B, C1 = Bτ C. 2. Calculate the QR decomposition of CT1 ⎡ ⎤ R CT1 Π = Q ⎣ ⎦ , O where R ∈ Rr×p , rank(R) = r and Q is a (n − r) × p orthogonal matrix. 3. Compute F = BT1 ΠRT . 4. Compute the QR decomposition of F ⎤ ⎡ RF ⎦. FΠF = QF ⎣ O 5. Compute the SVD Σ = VT RF W. output: The SVD of BT C is ⎡ ⎤ ⎡ Σ⊕O VT ⎣ ⎦=⎣ O

⎤ I

   ⎦ QTF BT C Q(W ⊕ In−p ) ,

where A ⊕ D represents the direct sum of A and D.

Algorithm 5.2

PSVD of BT SC [137]

input: B ∈ Rp×m , S ∈ Rp×q , C ∈ Rq×n , p ≤ m, q ≤ n. 1. Compute Br = Diag(bτ1 2 , . . . , bτp 2 ) and Cr = Diag(cτ1 2 , . . . , cτq 2 ). where bτi and cτj are the ith row vectors of B and C, respectively. 2. Set B1 = B†τ B, C1 = C†τ C, S1 = Bτ SCτ . 3. Compute LU decomposition Π1 S1 Π2 = LU, where L ∈ Rp×p , U ∈ Rp×q . 4. Compute M = LT Π1 B1 and N = UΠT2 C1 . 5. Compute the SVD of MT N via Algorithm 5.1 to get Q, QF , V and W. output:

The SVD of BT SC is ⎤ ⎡ ⎤ ⎡    Σ⊕O VT ⎦ QTF BT SC Q(W ⊕ In−p ) . ⎣ ⎦=⎣ I O

5.4 Applications of Singular Value Decomposition The SVD has been widely applied for solving many engineering problems. This section presents two typical application examples of SVD, in static system modeling and in image compression.

286

Singular Value Analysis

5.4.1 Static Systems Taking an electronic device as an example, we consider the SVD of a static system of voltages v1 and v2 and currents i1 and i2 : ⎡ ⎤ v  1    ⎥ 1 −1 0 0 ⎢ v ⎢ 2⎥ = 0 . (5.4.1) 0 0 1 1 ⎣ i1 ⎦ 0 6 78 9 i F

2

In this model of a static system, the entries of the matrix F limits the allowable values of v1 , v2 , i1 , i2 . If the measurement devices for voltages and currents have the same accuracy (for example 1%) then we can easily detect whether any set of measurements is a solution of Equation (5.4.1) within the desired range of accuracy. Let us assume that another method gives a static system model different from Equation (5.4.1): ⎡ ⎤ v1     6 6 ⎥ ⎢ 1 −1 10 10 ⎢v 2 ⎥ = 0 . (5.4.2) ⎣i ⎦ 0 0 0 1 1 1 i2 Obviously, only when the current measurements are very accurate, will a set of v1 , v2 , i1 , i2 satisfy Equation (5.4.2) with suitable accuracy; for the general case where current measurements have perhaps 1% measurement errors, Equation (5.4.2) is quite different from the static system model (5.4.1). In this case, the voltage relationships given by Equations (5.4.1) and (5.4.2) are v1 −v2 = 0 and v1 −v2 +104 = 0, respectively. However, from the algebraic viewpoint, Equations (5.4.1) and (5.4.2) are completely equivalent. Therefore, we hope to have some means to compare several algebraically equivalent model representations in order to determine which we want, i.e., one that can be used for the common static system model available in general rather than special circumstances. The basic mathematical tool for solving this problem is the SVD. More generally, consider a matrix equation for a static system including n resistors [100]:   v = 0. (5.4.3) F i Here F is an m × n matrix. In order to simplify the representation, some constant compensation terms are removed. Such a representation is very versatile, and can come from physical devices (e.g., linearized physical equations) and network equations. The action of the matrix F on the accurate and nonaccurate data parts can be analyzed using SVD. Let the SVD of F be F = UT ΣV.

(5.4.4)

5.4 Applications of Singular Value Decomposition

287

Using a truncated SVD, the nonprecision parts of F can be removed, and thus the SVD of the matrix F will provide a design equation that is algebraically equivalent to but numerically more reliable than the original static system model. Since U is an orthogonal matrix, from Equations (5.4.3) and (5.4.4) we have   v = 0. (5.4.5) ΣV i If the diagonal matrix Σ is blocked,  Σ=

Σ1 O

 O , O

then making a corresponding blocking of the orthogonal matrix V,   A B , V= C D where [A, B] includes the top r rows of V, Equation (5.4.5) can be rewritten as     Σ1 O A B v = 0. O O C D i Hence we obtain the new system model   v [A, B] = 0, i

(5.4.6)

which is algebraically equivalent to, but numerically much more reliable than, the original system model.

5.4.2 Image Compression The rapid development of science and technology and the popularity of network applications have produced a very large amount of digital information that needs to be stored, processed and transmitted. Image information, as an important multimedia resource, involves a large amount of data. For example, a 1024 × 768 24-bit BMP image uses about 2.25 MB. Large amounts of image information bring much compressive stress to the capacity of memorizers, the channel bandwidth of communications trunks and the processing speed of computers. Clearly, to solve this problem it is not realistic simply to increase the memory capacity and improve the channel bandwidth and processing speed of a computer. Hence, image compression is very necessary. Image compression is feasible because the image data are highly correlated. There is a large correlation between adjacent pixels in most images, and thus an image has a lot of spatial redundancy. Also, there is a large correlation between the front and back frames of the image sequence (i.e., there is time redundancy). Furthermore, if

288

Singular Value Analysis

the same code length is used to represent symbols with different probabilities of occurrence then a lot of bit numbers will be wasted, that is, the image representations will have symbol redundancy. Spatial redundancy, time redundancy and symbol redundancy constitute the three main factors in image compression. It is also important that a certain amount of distortion is allowed in image compression. Because an image has a matrix structure, the SVD can be applied to image compression. An important function of SVD is to reduce greatly the dimension of a matrix or image. In image and video processing the frames of an image are arranged into the columns of an image matrix A ∈ Rm×n . If the rank of A is r = rank(A), and its truncated SVD is given by A=

r 

σi ui viT ,

i=1

then the original image matrix A can be represented by an m×r orthogonal matrix Ur = [ui , . . . , ur ], an n × r orthogonal matrix Vr = [v1 , . . . , vr ]T and a diagonal vector composed of the singular values of A, σ = [σ1 , . . . , σr ]T . In other words, the orthogonal matrices Ur and Vr contain mr and nr data entries, respectively. So, the m × n original image matrix A (total data mn) is compressed into Ur , Vr and the diagonal vector σ (total data (m + n + 1)r). Then, the image compression ratio is given by mn ρ= . (5.4.7) r(m + n + 1) Hence, the basic idea of image compression based on SVD is to find the SVD of the image matrix and to use the k singular values and the corresponding left- and right-singular vectors to reconstruct the original image. If k ≥ r then the corresponding image compression is known as the lossless compression. Conversely, if k < r then the image compression is said to be lossy compression. In the general case, the number of selected singular values should meet the condition k(m + n + 1) ' mn.

(5.4.8)

Thus, during the process of transferring images, we need only transfer k(m + n + 1) data entries on selected singular values and the corresponding left- and rightsingular vectors rather than mn data entries. At the receiving end, after receiving k(m + n + 1) data entries, we can reconstruct the original image matrix through Ak =

k 

σi ui viT .

(5.4.9)

i=1

The error between the reconstructed image Ak and the original image A is measured

5.5 Generalized Singular Value Decomposition (GSVD)

289

by 2

2 2 + σk+2 + · · · + σr2 .

A − Ak F = σk+1

(5.4.10)

The contribution of some singular value σi to the image can be defined as i =

σ12

σi2 . + · · · + σr2

(5.4.11)

For a given image, the larger a singular value is, the larger is its contribution to the image information. Conversely, the smaller a singular value is, the smaller is its k contribution to the image information. For example, if i=1 i is close to 1 then the main information about the image is contained in Ak . On the basis of satisfying the visual requirements, if a suitable number of larger singular values k (< r) is selected then the original image A can be recovered by using these Ak . The smaller k is, the smaller is the data amount needed to approximate A, and thus the larger is the compression ratio; as k → r, the more similar to A is the compressed image matrix Ak . In some applications, if the compression ratio is specified in advance then the number k of singular values to be used for image compression can be found, and thus the contribution of image compression k can be computed from i=1 i .

5.5 Generalized Singular Value Decomposition (GSVD) Previous sections presented the SVD of a single matrix and the SVD of the product of two matrices BT C. In this section we discuss the SVD of the matrix pencil or pair (A, B). This decomposition is called generalized SVD.

5.5.1 Definition and Properties The generalized singular value decomposition (GSVD) method was proposed by Van Loan in 1976 [483]. THEOREM 5.6 [483] If A ∈ Cm×n , m ≥ n, and B ∈ Cp×n , there exist two unitary matrices U ∈ Cm×m and V ∈ Cp×p together with a nonsingular matrix Q ∈ Cn×n such that ⎤ ⎡ Ir ⎦ ∈ Rm×k , (5.5.1) ΣA = ⎣ UAQ = [ΣA , O] , SA k n−k

VBQ = [ΣB , O] , k n−k

⎡ ΣB = ⎣

OA OB

⎤ ⎦ ∈ Rp×k ,

SB Ik−r−s

(5.5.2)

290

Singular Value Analysis

where SA = Diag(αr+1 , . . . , αr+s ), SB = Diag(βr+1 , . . . , βr+s ), 1 > αr+1 ≥ · · · ≥ αr+s > 0, 0 < βr+1 ≤ · · · ≤ βr+s < 1, αi2

+ βi2 = 1,

i = r + 1, . . . , r + s,

with integers   A − rank(B), B   A . s = rank(A) + rank(B) − rank B

k = rank

  A , B

r = rank

(5.5.3) (5.5.4)

There is a variety of methods to prove this theorem, these can be found in Van Loan [483], Paige and Saunders [370], Golub and Van Loan [179] and Zha [540]. Using the method of [483], the diagonal entries of the diagonal matrices ΣA in Equation (5.5.1) and ΣB in Equation (5.5.2) constitute singular value pairs (αi , βi ). Since OA is a (m − r − s) × (k − r − s) zero matrix and OB is a (p − k + r) × r zero matrix, the first k singular value pairs are respectively given by αi = 1, αi , β i

βi = 0,

i = 1, . . . , r,

(the entries of SA and SB ), αi = 0,

βi = 1,

i = r + 1, . . . , r + s,

i = r + s + 1, . . . , k.

Hence the first k generalized singular values are given by ⎧ ⎪ ⎨ αi /βi = 1/0 = ∞, i = 1, . . . , r, γi = αi /βi , i = r + 1, . . . , r + s, ⎪ ⎩ i = r + s + 1, . . . , k. αi /βi = 0/1 = 0, These k singular value pairs (αi , βi ) are called the nontrivial generalized singular value pairs of the matrix pencil (A, B); αi /βi i = 1, 2, . . . , k, are known as the nontrivial generalized singular values of (A, B) and include the infinite, finite and zero values. In the contrary, the other n − k pairs of generalized singular values, corresponding to the zero vectors in Equations (5.5.1) and (5.5.2), are called the trivial generalized singular value pairs of the matrix pencil (A, B). The number of columns of the matrix A is restricted to be not greater than its number of rows in Theorem 5.6. Paige and Saunders [370] generalized Theorem 5.6 to propose the GSVD of any matrix pencil (A, B) with the same column number for A and B.

5.5 Generalized Singular Value Decomposition (GSVD)

291

Let A ∈ Cm×n and B ∈ Cp×n . Then for the block

THEOREM 5.7 (GSVD2) [370] matrix

  A K= , B

t = rank(K),

there are unitary matrices U ∈ Cm×m ,

V ∈ Cp×p ,

W ∈ Ct×t ,

Q ∈ Cn×n

such that t

n−t

H R9 , 6789 O ], UH AQ = ΣA [ 6W78 t

n−t

H VH BQ = ΣB [ 6W78 R9 , 6789 O ],

where

⎡ ΣA m×t

=⎣



IA

⎦,

DA OA

⎡ ΣB = ⎣ p×t



IB



DB

(5.5.5)

OB

and R ∈ Ct×t is nonsingular; its singular values are equal to the nonzero singular values of the matrix K. The matrix IA is an r × r identity matrix and IB is a (t − r − s) × (t − r − s) identity matrix, where the values of r and s are dependent on the given data, and OA and OB are respectively the (m − r − s) × (t − r − s) and (p − t + r) × r zero matrices that may have not any row or column (i.e., m − r − s = 0, t − r − s = 0 or p − t + r = 0 may occur) whereas DA = Diag(αr+1 , αr+2 , . . . , αr+s ), DB = Diag(βr+1 , βr+2 , . . . , βr+s ) satisfy 1 > αr+1 ≥ αr+2 ≥ · · · ≥ αr+s > 0,

0 < βr+1 ≤ βr+2 ≤ · · · ≤ βr+s < 1

and αi2 + βi2 = 1, Proof

i = r + 1, r + 2, . . . , r + s.

See [218].

The following are a few remarks on the GSVD. Remark 1

From the above definition it can be deduced that H AB−1 = UΣA [WH R, O]QH · Q[WH R, O]−1 Σ−1 B V H = UΣA Σ−1 B V .

This shows the equivalence between the GSVD of the matrix pencil (A, B) and the SVD of the matrix product AB−1 , because the generalized left and right singularvector matrices U and V of (A, B) are equal to the left and right singular-vector matrices U and V of AB−1 , respectively.

292

Singular Value Analysis

Remark 2 From the equivalence between the GSVD of (A, B) and the SVD of AB−1 it is obvious that if B = I is an identity matrix, then the GSVD simplifies to the general SVD. This result can be directly obtained from the definition of the GSVD, because all singular values of the identity matrix are equal to 1, and thus the generalized singular values of the matrix pencil (A, I) are equivalent to the singular values of A. Remark 3 Because AB−1 has the form of a quotient, and the generalized singular values are the quotients of the singular values of the matrices A and B, the GSVD is sometimes called the quotient singular value decomposition (QSVD).

5.5.2 Algorithms for GSVD If either A or B is an ill-conditioned matrix, then the calculation of AB−1 will lead usually to large numerical error, so the SVD of AB−1 is not recommended for computing the GSVD of the matrix pencil (A, B). A natural question to ask is whether we can get the GSVD of (A, B) directly without computing AB−1 . It is entirely possible, because we have the following theorem. THEOREM 5.8 [179] If A ∈ Cm1 ×n (m1 ≥ n) and B ∈ Cm2 ×n (m2 ≥ n) then there exists a nonsingular matrix X ∈ Cn×n such that XH (AH A)X = DA = Diag(α1 , α2 , . . . , αn ) ,

αk ≥ 0,

X (B B)X = DB = Diag(β1 , β2 , . . . , βn ) , βk ≥ 0.  The quantities σk = αk /βk are called the generalized singular values of the matrix pencil (A, B), and the columns xk of X are known as the generalized singular vectors associated with the generalized singular values σk . H

H

The above theorem provides various algorithms for computing the GSVD of the matrix pencil (A, B). In particular, we are interested in seeking the generalized singular-vector matrix X with DB = I, because in this case the generalized singular √ values σk are given directly by αk . Algorithms 5.3 and 5.4 give two GSVD algorithms. The main difference between Algorithms 5.3 and 5.4 is that the former requires computation of the matrix product AH A and BH B, and the latter avoids this computation. Because computation of the matrix product may make the condition number become large, Algorithm 5.4 has a better numerical performance than Algorithm 5.3. In 1998, Drmac [136] developed a tangent algorithm for computing the GSVD, as shown in Algorithm 5.5. This algorithm is divided into two phases: in the first phase the matrix pencil (A, B) is simplified into a matrix F; in the second phase the SVD of F is computed. The theoretical basis of the tangent algorithm is as follows: the GSVD is invariant

5.5 Generalized Singular Value Decomposition (GSVD) Algorithm 5.3

293

GSVD algorithm 1 [484]

input: A ∈ Cm1 ×n , B ∈ Cm2 ×n . 1. Compute S1 = AH A and S2 = BH B. 2. Calculate the eigenvalue decomposition UH 2 S2 U2 = D = Diag(γ1 , . . . , γn ). 3. Compute Y = U2 D−1/2 and C = Y H S1 Y. 4. Compute the EVD QH CQ = Diag(α1 , . . . , αn ), where QH Q = I. output: Generalized singular-vector matrix X = YQ and generalized singular values √ αk , k = 1, . . . , n. Algorithm 5.4

GSVD algorithm 2 [484]

input: A ∈ Cm1 ×n , B ∈ Cm2 ×n . 1. Compute the SVD UH 2 BV2 = D = Diag(γ1 , . . . , γn ). 2. Calculate Y = V2 D−1 V2 = Diag(1/γ1 , . . . , 1/γn ). 3. Compute C = AY. 4. Compute the SVD UH 1 CV1 = DA = Diag(α1 , . . . , αn ). output: Generalized singular-vector matrix X = YV1 , generalized singular values αk , k = 1, . . . , n. Algorithm 5.5

Tangent algorithm for GSVD [136]

input: A = [a1 , . . . , an ] ∈ Rm×n , B ∈ Rp×n , m ≥ n, rank(B) = n. 1. Compute ΔA = Diag(a1 2 , . . . , an 2 ), Ac = AΔ−1 A ,

B1 = BΔ−1 A .

 R = QT B1 Π. O 3. Solve the matrix equation FR = Ac Π to get F = Ac ΠR−1 .   Σ 4. Compute the SVD = VT FU. O   U O −1 . 5. Compute X = Δ−1 U and W = Q A ΠR O Ip−n output: The GSVD of (A,

B) reads as



2. Compute the Householder QR decomposition

VT AX =

Σ , O

WT BX =

I . O

That is, the generalized singular values of (A, B) are given by the diagonal entries of Σ, and the generalized left and right singular-vector matrices of (A, B) are given by V and W, respectively.

under a equivalence transformation, namely (A, B) → (A , B ) = (UT AS, VT BS),

(5.5.6)

294

Singular Value Analysis

where U, V are any orthogonal matrices and S is any nonsingular matrix. Hence, by the definition, two matrix pencils (A, B) and (A , B ) have the same GSVD.

5.5.3 Two Application Examples of GSVD The following are two application examples of GSVD. 1. Multi-Microphone Speech Enhancement A noisy speech signal sampled by multi-microphone at a discrete time k can be described by the observation model y(k) = x(k) + v(k), where x(k) and v(k) are the speech signal vector and the additive noise vector, respectively. Letting Ryy = E{y(k)yT (k)} and Rvv = E{v(k)vT (k)} represent the autocorrelation matrices of the observation data vector y and the additive noise vector v, respectively, one can make their joint diagonalization 0 2 Ryy = QDiag(σ12 , . . . , σm )QT , (5.5.7) 2 Rvv = QDiag(η12 , . . . , ηm )QT . Doclo and Moonen [127] showed in 2002 that, in order to achieve multi-microphone speech enhancement, the M × M optimal filter matrix W(k) with minimum meansquare error (MMSE) is given by −1 W(k) = R−1 yy (k)Rxx (k) = Ryy (k)(Ryy (k) − Rvv (k))   2 η12 η22 ηM −T = Q Diag 1 − 2 , 1 − 2 , . . . , 1 − 2 Q. σ1 σ2 σM

(5.5.8)

Let Y(k) be p×M speech-data matrices containing p speech-data vectors recorded during speech-and-noise periods, and let V(k ) be the q×M additive noise matrix containing q noise data vectors recorded during noise-only periods: ⎤ ⎤ ⎡ T

⎡ T y (k − p + 1) v (k − q + 1) ⎥ ⎥ ⎢ ⎢ .. .. ⎥ ⎥ ⎢ ⎢ . .

⎥, ⎥ ⎢ V(k (5.5.9) ) = Y(k) = ⎢ ⎢ yT (k − 1) ⎥ ⎢ vT (k − 1) ⎥ , ⎦ ⎦ ⎣ ⎣ yT (k) vT (k ) where p and q are typically larger than M . The GSVD of the matrix pencil (Y(k), V(k )) is defined as Y(k) = UY ΣY QT ,

V(k ) = UV ΣV QT ,

(5.5.10)

where ΣY = Diag(σ1 , . . . , σM ), ΣV = Diag(η1 , . . . , ηM ), UY and UV are orthogonal matrices, Q is an invertible but not necessarily orthogonal matrix containing the

5.5 Generalized Singular Value Decomposition (GSVD)

295

generalized singular vectors and σi /ηi are the generalized singular values. Substituting these results into Equation (5.5.8), the estimate of the optimal filter matrix is given by [127]   2 2 ˆ = Q−T Diag 1 − p η1 , . . . , p ηM QT . W (5.5.11) 2 q σ12 q σM 2. Dimension Reduction of Clustered Text Data In signal processing, pattern recognition, image processing, information retrieval and so on, dimension reduction is imperative for efficiently manipulating the massive quantity of data. Suppose that a given m × n term-document matrix A ∈ Rm×n is already partitioned into k clusters: A = [A1 , . . . , Ak ],

where

Ai ∈ R

m×ni

,

k 

ni = n.

i=1

Our goal is to find a lower-dimensional representation of the term-document matrix A such that the cluster structure existing in m-dimensional space is preserved in the lower-dimensional representation. The centroid ci of each cluster matrix Ai is defined as the average of the columns in Ai , i.e., 1 ci = Ai 1i , where 1i = [1, . . . , 1]T ∈ Rni ×1 , ni and the global centroid is given by c=

1 A1, n

where

1 = [1, . . . , 1]T ∈ Rn×1 .

Denote the ith cluster as Ai = [ai1 , . . . , aini ], and let Ni = {i1, . . . , ini } denote the set of column indices that belong to the cluster i. Then the within-class scatter matrix Sw is defined as [218] Sw =

k  

(aj − ci )(aj − ci )T

i=1 j∈Ni

and the between-class scatter matrix Sb is defined as Sb =

k 

ni (ci − c)(ci − c)T .

i=1

Defining the matrices Hw = [A1 − c1 1T1 , . . . , Ak − ck 1Tk ] ∈ Rm×n , √ √ Hb = [ n1 (c1 − c), . . . , nk (c − ck )] ∈ Rm×k ,

(5.5.12)

296

Singular Value Analysis

the corresponding scatter matrices can be represented as Sw = Hw HTw ∈ Rm×m ,

Sb = Hb HTb ∈ Rm×m .

It is assumed that the lower-dimensional representation of the term-document matrix A ∈ Rm×n consists of l vectors ui ∈ Rm×1 with ui 2 = 1, where i = 1, . . . , l and l < min{m, n, k}. Consider the generalized Rayleigh quotient λi =

HTb ui 22 uTi Sb ui = ;

HTw ui 22 uTi Sw ui

(5.5.13)

its numerator and denominator denote the between-class scatter and the withinclass scatter of the vector ui , respectively. As the measure of cluster quality, the between-class scatter should be as large as possible, while the within-class scatter should be as small as possible. Hence, for each ui , the generalized Rayleigh quotient, namely uTi Sb ui = λi uTi Sw ui

or

uTi Hb HTb ui = λi uTi Hw HTw ui

(5.5.14)

should be maximized. This is the generalized eigenvalue decomposition of the cluster matrix pencil (Sb , Sw ). Equation (5.5.14) can be equivalently written as  HTb ui = σi HTw ui , σi = λi . (5.5.15) This is just the GSVD of the matrix pencil (HTb , HTw ). From the above analysis, the dimension reduction method for a given termdocument matrix A ∈ Rm×n can be summarized as follows. (1) Compute the matrices Hb and Hw using Equation (5.5.12). (2) Perform the GSVD of the matrix pencil (Hb , Hw ) to get the nonzero generalized singular values σi , i = 1, . . . , l and the corresponding left-generalized singular vectors u1 , . . . , ul . (3) The lower-dimensional representation of A is given by [u1 , . . . , ul ]T . The GSVD has also been applied to the comparative analysis of genome-scale expression data sets of two different organisms [13], to discriminant analysis [219] etc.

5.6 Low-Rank–Sparse Matrix Decomposition In the fields of applied science and engineering (such as image, voice and video processing, bioinformatics, network search and e-commerce), a data set is often high-dimensional, and its dimension can be even up to a million. It is important to discover and utilize the lower-dimensional structures in a high-dimensional data set.

5.6 Low-Rank–Sparse Matrix Decomposition

297

5.6.1 Matrix Decomposition Problems In many areas of engineering and applied sciences (such as machine learning, control, system engineering, signal processing, pattern recognition and computer vision), a data (image) matrix A ∈ Rm×n usually has the following special constructions. (1) High dimension m, n are usually very large. (2) Low rank r = rank(A) ' min{m, n}. (3) Sparsity Some observations are grossly corrupted, so that A becomes A + E; some entries of the error matrix E may be very large but most are equal to zero. (4) Incomplete data Some observations may be missing too. The following are several typical application areas with the above special constructions [90], [77]. 1. Graphical modeling In many applications, because a small number of characteristic factors can explain most of the statistics of the observation data, a high-dimensional covariance matrix is commonly approximated by a low-rank matrix. In the graphical model [284] the inverse of the covariance matrix (also called the information matrix) is assumed to be sparse with respect to some graphic. Thus, in the statistical model setting, the data matrix is often decomposed into the sum of a low-rank matrix and a sparse matrix, to describe the roles of unobserved hidden variables and of the graphical model, respectively. 2. Composite system identification In system identification a composite system is often represented by the sum of a low-rank Hankel matrix and a sparse Hankel matrix, where the sparse Hankel matrix corresponds to a linear time-invariant system with sparse impulse response, and the low-rank Hankel matrix corresponds to the minimum realization system with a small order of model. 3. Face recognition Since images of a convex, Lambertian surface under different types of lighting span a lower-dimensional subspace [27], the lower-dimensional models are the most effective for image data. In particular, human face images can be well-approximated by a lower-dimensional subspace. Therefore, to retrieve this subspace correctly is crucial in many applications such as face recognition and alignment. However, actual face images often suffer from shadows and highlighting effects (specularities, or saturations in brightness), and hence it is necessary to remove such defects in face images by using low-rank matrix decomposition. 4. Video surveillance Given a sequence of surveillance video frames, it is often necessary to identify activities standing out from the background. If the video frames are stacked as columns of a data matrix D and a low-rank and sparse

298

Singular Value Analysis

decomposition D = A + E is made then the low-rank matrix A naturally corresponds to the stationary background and the sparse matrix E captures the moving objects in the foreground. 5. Latent semantic indexing Web search engines often need to index and retrieve the contents of a huge set of documents. A popular scheme is latent semantic indexing (LSI) [124]; its basic idea is to encode the relevance of a term or word in a document (for example, by its frequency) and to use these relevance indicators as elements of a document-versus-term matrix D. If D can be decomposed into a sum of a low-rank matrix A and a sparse matrix E then A could capture the common words used in all the documents while E captures the few key words that best distinguish each document from others. The above examples show that for many applications it is not enough to decompose a data matrix into the sum of a low-rank matrix and an error (or perturbation) matrix. A better approach is to decompose a data matrix D into the sum of a low-rank matrix A and a sparse matrix E, i.e., D = A + E, in order to recover its low-rank component matrix. Such a decomposition is called a low-rank–sparse matrix decomposition.

5.6.2 Singular Value Thresholding Singular value thresholding is a key operation in matrix recovery and matrix completion. Before discussing in detail how to solve matrix recovery (and matrix completion) problems, it is necessary to introduce singular value thresholding. Consider the truncated SVD of a low-rank matrix W ∈ Rm×n : W = UΣVT ,

Σ = Diag(σ1 , . . . , σr ),

(5.6.1)

where r = rank(W) ' min{m, n}, U ∈ Rm×r , V ∈ Rn×r . Let the threshold value τ ≥ 0, then Dτ (W) = UDτ (Σ)VT

(5.6.2)

is called the singular value thresholding (SVT) of the matrix W, where Dτ (Σ) = soft(Σ, τ ) = Diag ((σ1 − τ )+ , . . . , (σr − τ )+ ) is known as the soft thresholding and " σi − τ, (σi − τ )+ = 0,

(5.6.3)

if σi > τ, otherwise,

is the soft thresholding operation. The relationship between the SVT and the SVD is as follows. • If the soft threshold value τ = 0 then the SVT reduces to the truncated SVD (5.6.1).

5.6 Low-Rank–Sparse Matrix Decomposition

299

• All singular values are soft thresholded by the soft threshold value τ > 0, which just changes the magnitudes of singular values; it does not change the left and right singular-vector matrices U and V. The proper choice of soft threshold value τ can effectively set some singular values to zero. In this sense the SVT transform is also called a singular value shrinkage operator. It is noted that if the soft threshold value τ is larger than most of the singular values, then the rank of the SVT operator D(W) will be much smaller than the rank of the original matrix W. The following are two key points for applying the SVT: (1) How should an optimization problem be written in a standard form suitable for the SVT? (2) How should the soft thresholding value τ be chosen? The answers to the above two questions depend on the following proposition and theorem, respectively. PROPOSITION 5.1 Let X, Y = tr(XT Y) denote the inner product of matrices X ∈ Rm×n and Y ∈ Rm×n . Then we have

X ± Y 2F = X 2F ± 2X, Y + Y 2F .

(5.6.4)

Proof By the relationship between the Frobenius norm and the trace of a matrix,

A F = tr(AH A), we have

X ± Y 2F = tr[(X ± Y)T (X ± Y)] = tr(XT X ± XT Y ± YT X + YT Y) = X 2F ± 2tr(XT Y) + Y 2F = X 2F ± 2X, Y + Y 2F , where we have used the trace property tr(YT X) = tr(XT Y). THEOREM 5.9 For each soft threshold value μ > 0 and the matrix W ∈ Rm×n , the SVT operator obeys [75] " # 1 Usoft(Σ, μ)VT = arg min μ X ∗ + X − W 2F = proxμ · ∗ (W), (5.6.5) 2 X and [75]

" # 1 soft(W, μ) = arg min μ X 1 + X − W 2F = proxμ · 1 (W), 2 X

(5.6.6)

where UΣVT is the SVD of W, and the soft thresholding (shrinkage) operator + wij − μ, wij > μ, [soft(W, μ)]ij =

wij + μ, wij < −μ, 0,

otherwise,

300

Singular Value Analysis

with wij ∈ R the (i, j)th entry of W ∈ Rm×n . Theorem 5.9 tells us that when solving a matrix recovery or matrix completion problem, the underlying question is how to transform the corresponding optimization problem into the normalized form shown in Equations (5.6.5) and/or (5.6.6).

5.6.3 Robust Principal Component Analysis Given an observation or data matrix D = A + E, where A and E are unknown, but A is known to be low rank and E is known to be sparse, the aim is to recover A. Because the low-rank matrix A can be regarded as the principal component of the data matrix D, and the sparse matrix E may have a few gross errors or outlying observations, the SVD based principal component analysis (PCA) usually breaks down. Hence, this problem is a robust principal component analysis problem [298], [514]. So-called robust PCA is able to correctly recover underlying low-rank structure in the data matrix, even in the presence of gross errors or outlying observations. This problem is also called the principal component pursuit (PCP) problem [90] since it tracks down the principal components of the data matrix D by minimizing a combination of the nuclear norm A ∗ and the weighted 1 -norm μ E 1 : min { A ∗ + μ E 1 } A,E

subject to

D = A + E.

(5.6.7)

By using the above minimization, an initially unknown low-rank matrix A and unknown sparse matrix E can be recovered from the data matrix D ∈ Rm×n . min{m,n} Here, A ∗ = σi (A) represents the nuclear norm of A, i.e., the sum i of its all singular values, which reflects the cost of the low-rank matrix A and m n

E 1 = i=1 j=1 |Eij | is the sum of the absolute values of all entries of the additive error matrix E, where some of its entries may be arbitrarily large, whereas the role of the constant μ > 0 is to balance the contradictory requirements of low rank and sparsity. The unconstrained minimization of the robust PCA or the PCP problem is expressed as " # 1 min A ∗ + μ E 1 + (A + E − D) . (5.6.8) A,E 2 It can be shown [77] that, under rather weak assumptions, the PCP estimate can exactly recover the low-rank matrix A from the data matrix D = A + E with gross but sparse error matrix E. Thus, to solve the robust PCA or the PCP problem, consider a more general family of optimization problems of the form F (X) = f (X) + μh(X),

(5.6.9)

5.6 Low-Rank–Sparse Matrix Decomposition

301

where f (X) is a convex, smooth (i.e., differentiable) and L-Lipschitz function and h(X) is a convex but nonsmooth function (such as X 1 , X ∗ and so on). Instead of directly minimizing the composite function F (X), we minimize its separable quadratic approximation Q(X, Y), formed at specially chosen points Y: 1 Q(X, Y) = f (Y) + ∇f (Y), X − Y + (X − Y)T ∇2 f (Y)(X − Y) + μh(X) 2 L = f (Y) + ∇f (Y), X − Y + X − Y 2F + μh(X), (5.6.10) 2 where ∇2 f (Y) is approximated by LI. When minimizing Q(X, Y) with respect to X, the function term f (Y) may be regarded as a constant term that is negligible. Hence, we have + 22 0 2 2 L2 1 X − Yk + ∇f (Yk )2 Xk+1 = arg min μh(X) + 2 2 2 2 L X F   1 (5.6.11) = proxμL−1 h Yk − ∇f (Yk ) . L If we let f (Y) = 12 A + E − D 2F and h(X) = A ∗ + λ E 1 then the Lipschitz constant L = 2 and ∇f (Y) = A + E − D. Hence 1 Q(X, Y) = μh(X) + f (Y) = (μ A ∗ + μλ E 1 ) + A + E − D 2F 2 reduces to the quadratic approximation of the robust PCA problem. By Theorem 5.9, one has   1 A Ak+1 = proxμ/2 · ∗ Yk − (Ak + Ek − D) = Usoft(Σ, λ)VT , 2     1 E E μλ , Ek+1 = proxμλ/2 · 1 Yk − (Ak + Ek − D) = soft Wk , 2 2 where UΣVT is the SVD of WkA and 1 WkA = YkA − (Ak + Ek − D), 2 1 WkE = YkE − (Ak + Ek − D). 2 Algorithm 5.6 gives a robust PCA algorithm. The robust PCA has the following convergence.

(5.6.12) (5.6.13)

THEOREM 5.10 [298] Let F (X) = F (A, E) = μ A ∗ +μλ E 1 + 12 A+E−D 2F . Then, for all k > k0 = C1 / log(1/n) with C1 = log(μ0 /μ), one has F (X) − F (X ) ≤

4 Xk0 − X 2F , (k − k0 + 1)2

where X is a solution to the robust PCA problem min F (X). X

(5.6.14)

302

Singular Value Analysis Robust PCA via accelerated proximal gradient [298], [514]

Algorithm 5.6

input: Data matrix D ∈ Rm×n , λ, allowed tolerance . ¯ ← δμ0 . initialization: A0 , A−1 ← O; E0 , E−1 ← O; t0 , t−1 ← 1; μ repeat 1. YkA ← Ak + 2. YkE ← Ek + 3.

WkA



YkA

tk−1 − 1 tk tk−1 − 1



tk 1 (Ak 2

(Ak − Ak−1 ). (Ek − Ek−1 ).

+ Ek − D).

svd(WkA ).

4. (U, Σ, V) = μ 5. r = max j : σj > k . 2

r μ 6. Ak+1 = i=1 σi − k ui viT . 2

1

7. WkE ← YkE − (Ak + Ek − D). 2

λμ 8. Ek+1 = soft WkE , k . 9. tk+1 ←

1+

4t2k + 1 2

2

.

¯). 10. μk+1 ← max(ημk , μ A A E 11. SA k+1 = 2(Yk − Ak+1 ) + (Ak+1 + Ek+1 − Yk − Yk ). E A E 12. SE k+1 = 2(Yk − Ek+1 ) + (Ak+1 + Ek+1 − Yk − Yk ). 2 E 2 13. exit if Sk+1 2F = SA k+1 F + Sk+1 F ≤ .

return k ← k + 1. output: A ← Ak , E ← Ek .

5.7 Matrix Completion Finding a low-rank matrix to fit or approximate a higher-dimensional data matrix is a fundamental task in data analysis. In a slew of applications, two popular empirical approaches have been used to represent the target rank-k matrix. (1) Low-rank decomposition A noisy data matrix X ∈ Rm×n is decomposed as the sum of two matrices A + E, where A ∈ Rm×n is a low-rank matrix with the same rank k as the target matrix to be estimated and E ∈ Rm×n is a fitting or approximating error matrix. (2) Low-rank bilinear modeling: The data matrix X ∈ Rm×n is modeled in a bilinear form X = UVT , where U ∈ Rm×k and V ∈ Rn×k are full-column rank matrices. When most entries of a data matrix are missing, its low-rank decomposition problem is called matrix completion, as presented by Cand`es and Recht in 2009 [79]. The low-rank bilinear modeling of a matrix is known as matrix sensing and was developed by Recht et al. [412] and Lee and Bresler [288].

5.7 Matrix Completion

303

Specifically, the matrix completion problem and the matrix sensing problem can be described as follows. • Matrix completion involves completing a low-rank matrix by taking into account, i.e., observing, only a few of its elements. • Matrix sensing refers to the problem of recovering a low-rank matrix B ∈ Rm×n , given d linear measurements bi = tr(ATi B) and measurement matrices Ai , i = 1, . . . , d. In fact, matrix completion is a special case of the matrix sensing problem where each observed entry in the matrix completion problem represents a single-element measurement matrix Ai = Ai . This section focuses upon matrix completion.

5.7.1 Matrix Completion Problems Matrix completion problems stem from the widespread practical applications. 1. Dynamic scene reconstruction It is a classical and hot issue in computer vision and graphics to make a dynamic scene reconstruction from multiview video sequences. Three-dimensional (3-D) motion estimation from multiview video sequences is of vital importance in achieving high-quality dynamic scene reconstruction and has great application potential in the virtual reality and future movie industry. Because 3-D motion vectors can be treated as an incomplete matrix according to their local distributions and frequently there are outliers in the known motion information, 3-D motion estimation is naturally a matrix completion problem. 2. Radar imaging For high-resolution radar imaging, the resulting high sampling rate poses difficulties for raw data transmission and storage, which results in the undersampling of stepped-frequency-radar data. Radar change imaging aims to recover the changed portion of an imaged region using the data collected before and after a change. Hence, radar change imaging with undersampled data can be cast as a matrix completion problem [45]. Tomographic synthetic aperture radar (TomoSAR) is a new SAR imaging modality, but the massive baselines involved make TomoSAR imaging an expensive and difficult task. However, the use of limited baselines with fixed-elevation aperture length to improve the recovery quality is worth researching. By exploiting multibaseline two-dimensional (2D) focused SAR image data, the missing-data compensation method, based on matrix completion, can solve the 3-D focused TomoSAR imaging problem [46]. 3. MIMO radar Multiple-input multiple-output (MIMO) radar has received significant attention owing to its improved resolution and target estimation performance as compared with conventional radars. For a sufficiently large number of transmission and reception antennas and a small number of targets, the data

304

Singular Value Analysis

matrix is low rank, and it can be recovered from a small number of its entries via matrix completion. This implies that, at each reception antenna, matched filtering does not need to be performed with all the transmit waveforms, but rather with just a small number of randomly selected waveforms from the waveform dictionary [237]. 4. Ranking and collaborative filtering Predicting the user’s preferences is increasingly important in e-commerce and advertisement. Companies now routinely collect user rankings for various products (such as movies, books or games). Socalled ranking and collaborative filtering employs incomplete user rankings on some products for predicting any given user’s preferences on another product. The best known ranking and collaborative filtering is perhaps the Netflix Prize for movie ranking. In this case the data matrix is incomplete, as the data collection process often lacks control or sometimes a small portion of the available rankings could be noisy and even have been subject to tampering. Therefore, it necessary to infer a low-rank matrix L from an incomplete and corrupted matrix [79], [78]. 5. Global positioning Owing to power constraints, sensors in global positioning networks may only be able to construct reliable distance estimates from their immediate neighbors, and hence only a partially observed distance matrix can be formed. However, the problem is to infer all the pairwise distances from just a few observed data so that the locations of the sensors can be reliably estimated. This reduces to a matrix completion problem, where the unknown matrix is of rank 2 (if the sensors are located in the plane) or 3 (if they are located in space) [79], [78]. 6. Remote sensing In array signal processing, it is necessary to estimate the direction of arrival (DOA) of incident signals in a coherent radio-frequency environment. The DOA is frequently estimated by using the covariance matrix for the observed data. However, in remote sensing applications, one may not be able to estimate or transmit all entries of the covariance matrix owing to power constraints. To infer a full covariance matrix from just a few observed partial correlations, one needs to solve a matrix completion problem in which the unknown signal covariance matrix has low rank since it is equal to the number of incident waves, which is usually much smaller than the number of sensors [79], [78]. 7. Multi-label image classification In multi-label image classification, multiple labels are assigned to a given image; thus is useful in web-based image analysis. For example, keywords can be automatically assigned to an uploaded web image so that annotated images may be searched directly using text-based image retrieval systems. An ideal multi-label algorithm should be able to handle missing features, or cases where parts of the training data labels are unknown, and it should be robust against outliers and background noise. To this end, Luo et

5.7 Matrix Completion

305

al. [306] have developed recently a multiview matrix completion for handling multiview features in semi-supervised multi-label image classification. 8. Computer vision and graphics Many problems can be formulated as missingvalue estimation problems, e.g., image in-painting, video decoding, video inpainting, scan completion and appearance acquisition completion [300]. The differences between matrix completion and matrix recovery (or low-rank– sparse matrix decomposition) are given below. (1) Only a small number of matrix elements is known in matrix completion, while the whole data matrix is known in matrix recovery. (2) Matrix completion reconstructs a higher-dimensional matrix, while matrix recovery extracts the lower-dimensional features of a higher-dimensional matrix. (3) The capability of matrix completion is limited by noncoherence and the sampling rate, while the capability of the matrix recovery is limited by real matrix rank and data sampling method.

5.7.2 Matrix Completion Model and Incoherence Let D ∈ R be a higher-dimensional incomplete data matrix, as discussed above: only a small amount of entries are known or observed and the index set of these known or sample entries is Ω, namely only the matrix entries Dij , (i, j) ∈ Ω are known or observed. The support of the index set is called the base number, denoted |Ω|. The base number is the number of sample elements, and the ratio of the number of samples to the dimension number of the data matrix, p = |Ω|/(mn), is called the sample density of the data matrix. In some typical applications (e.g., the Netflix recommendation system), the sample density is only 1% or less. As mentioned earlier, the mathematical problem of recovering a low-rank matrix from an incomplete data matrix is called matrix completion, and can be represented as ˆ = arg min rank(X) X (5.7.1) m×n

X

subject to PΩ (X) = PΩ (D) or

Xij = Dij ,

for (i, j) ∈ Ω,

(5.7.2)

where Ω ⊂ {1, . . . , m} × {1, . . . , n} is a subset of the complete set of entries and PΩ : Rm×n → Rm×n is the projection onto the index set Ω: " Dij , (i, j) ∈ Ω, [PΩ (D)]ij = (5.7.3) 0, otherwise. Unfortunately, the rank minimization problem (5.7.1) is NP-hard owing to the

306

Singular Value Analysis

nonconvexity and discontinuous nature of the rank function. It has been shown [412] that the nuclear norm of a matrix is the tightest convex lower bound of its rank function. Hence, a popular alternative is the convex relaxation of the rank function: ˆ = arg min X ∗ X

subject to Xij = Dij ,

(i, j) ∈ Ω,

(5.7.4)

X

where X ∗ is the nuclear norm of the matrix X, defined as the sum of its singular values. It was observed in [79] that if the entries of D are equal to zero in most rows or columns then it is impossible to complete D unless all of its entries are observed. An interesting question is how to guarantee the unique completion of an incomplete data matrix. The answer to this question depends on an additional property known as the incoherence of the data matrix D. Suppose that the SVD of an m × n matrix M with rank r is M = UΣVT . The matrix M is said to satisfy the standard incoherence condition with parameter μ0 if [98] : # " 2 T 2 μ0 r 2 2 , max U ei 2 ≤ 1≤i≤m m (5.7.5) : # " 2 2 μ0 r max 2VT ej 22 ≤ , 1≤j≤n n where ei ∈ Rm×1 and ej ∈ Rn×1 are the standard basis vectors, with only nonzero ith and jth entries, respectively. The following theorem shows that only the standard incoherence condition is required for the unique recovery of an incomplete data matrix. THEOREM 5.11 [98] Suppose that a data matrix D satisfies the standard incoherence condition (5.7.5) with parameter μ0 . There exist universal constants c0 , c1 , c2 > 0 such that if p ≥ c0

μ0 r log2 (m + n) , min{m, n}

(5.7.6)

then the matrix D is the unique optimal solution to (5.7.4) with probability at least 1 − c1 (m + n)−c2 .

5.7.3 Singular Value Thresholding Algorithm Consider the matrix completion problem # " 1 2 subject to PΩ (X) = PΩ (D), min τ X ∗ + X F 2

(5.7.7)

5.7 Matrix Completion

307

with Lagrangian objective function 1 L(X, Y) = τ X ∗ + X 2F + Y, PΩ (D) − PΩ (X), 2

(5.7.8)

where the inner product A, B = tr(AT B) = B, A. Notice that Y, PΩ (D) and PΩ (Y) 2F = PΩ (Y), PΩ (Y) are constants with respect to X. We delete the former term and add the latter term to yield " # 1 min L(X, Y) = min τ X ∗ + X 2F + Y, PΩ (D − X) 2 X X " # 1 1 = min τ X ∗ + X 2F − X, PΩ (Y) + PΩ (Y) 2F . (5.7.9) 2 2 X Here we have Y, PΩ X = tr(YT PΩ X) = tr(XT PΩT Y) = tr(XT PΩ Y) = X, PΩ Y, because PΩT = PΩ . By Theorem 5.9, from Equation (5.7.9) we have " # 1 1 2 2 Xk+1 = arg min τ X ∗ + X F − X, PΩ Yk  + PΩ Yk F 2 2 X " # 1 = arg min X ∗ + X − PΩ Yk 2F . (5.7.10) 2 X Because Yk = PΩ (Yk ), ∀ k ≥ 0, Theorem 5.9 gives Xk+1 = Dτ (PΩ Yk ) = proxτ · ∗ (Yk ) = Usoft(Σ, τ )VT ,

(5.7.11)

where UΣVT is the SVD of Yk . Moreover, from Equation (5.7.8), the gradient of the Lagrangian function at the point Y is given by ∂L(X, Y) = PΩ (X − D). ∂YT Hence the gradient descent update of Yk is Yk+1 = Yk + μPΩ (D − Xk+1 ), where μ is the step size. Equations (5.7.11) and (5.7.12) give the iteration sequence [75] 0 Xk+1 = Usoft(Σ, τ )(VT ), Yk+1 = Yk + μPΩ (D − Xk+1 ).

(5.7.12)

(5.7.13)

The above iteration sequence has the following two features [75]: (1) Sparsity For each k ≥ 0, the entries of Yk outside Ω are equal to zero, and thus Yk is a sparse matrix. This fact can be used to evaluate the shrink function rapidly.

308

Singular Value Analysis

(2) Low-rank property The matrix Xk has low rank, and hence the iteration algorithm requires only a small amount of storage since we need to keep only the principal factors in the memory. Algorithm 5.7 gives a singular value thresholding algorithm for matrix completion. Algorithm 5.7

Singular value thresholding for matrix completion [75]

given: Dij , (i, j) ∈ Ω, step size δ, tolerance , parameter τ , increment , maximum iteration count kmax . initialization: Define k0 as the integer such that

τ δ PΩ (D) 2

∈ (k0 − 1, k0 ].

0

Set Y = k0 δPΩ (D). Put r0 and k = 1. repeat 1. Compute the SVD (U, Σ, V) = svd(Y k−1 ). 2. Set rk = max{j|σj > τ }. rk 3. Update Xk = j=1 (σj − τ )uj vjT . 4. exit if PΩ (Xk − D)F /PΩ DF ≤  or k = kmax .  k−1 k ), if (i, j) ∈ Ω, Yij + δ(Dij − Xij 5. Compute Yijk = 0, if (i, j) ∈ / Ω. return k ← k + 1. output: Xopt = Xk .

5.7.4 Fast and Accurate Matrix Completion As compared with the nuclear norm, a better convex relaxation of the rank function is truncated nuclear norm regularization, proposed recently in [221]. DEFINITION 5.1 [221] Given a matrix X ∈ Rm×n with r = rank(X), its truncated nuclear norm X ∗,r is defined as the sum of min{m, n} − r minimum singular min{m,n} values, i.e., X ∗,r = i=r+1 σi . Using the truncated nuclear norm instead of the nuclear norm, the standard matrix completion (5.7.4) becomes the truncated nuclear norm minimization min X ∗,r X

subject to PΩ (X) = PΩ (D).

(5.7.14)

Unlike the convex nuclear norm, the truncated nuclear norm X ∗,r is nonconvex, and thus (5.7.14) is difficult to solve. Fortunately, the following theorem can be used to overcome this difficulty.

5.7 Matrix Completion

309

THEOREM 5.12 [221] Given any matrix X ∈ Rm×n and r ≤ min{m, n}, for two matrices A ∈ Rr×m , B ∈ Rr×n , if AAT = Ir×r and BBT = Ir×r then tr(AXBT ) ≤

r 

σi (X).

(5.7.15)

i=1

A simple choice is A = [u1 , . . . , ur ]T , B = [v1 , . . . , vr ]T , and X = is the truncated SVD of X. Under this choice, one has + 0 r  T tr(AXB ) = max σi (X) . AAT =I, BBT =I

r i=1

σi (X)ui viT

(5.7.16)

i=1

Hence the truncated nuclear norm minimization (5.7.14) can be relaxed as " # min X ∗ − max tr(AXBT ) subject to PΩ (X) = PΩ (D). (5.7.17) X

AAT =I,BBT =I

If Al and Bl are found then the above optimization problem becomes 3 4 subject to PΩ (X) = PΩ (D), min X ∗ − tr(Al XBTl )

(5.7.18)

X

and its Lagrangian objective function is L(X) = X ∗ − tr(Al XBTl ) +

λ

PΩ X − PΩ D 2F , 2

(5.7.19)

where λ is a Lagrange multiplier. Let f (X) =

λ

PΩ X − PΩ D 2F − tr(Al XBTl ) 2

and

h(X) = X ∗ ,

then f (X) is a convex, differentiable and L-Lipschitz function, and h(X) is a convex but nonsmooth function. Consider the quadratic approximation of f (X) at a given point Y: L fˆ(X) = f (Y) + X − Y, ∇f (Y) + X − Y 2F . 2 Then, the Lagrangian objective function L(X) = h(X) + fˆ(X) can be reformulated as L Q(X, Y) = X ∗ + f (Y) + X − Y, ∇f (Y) + X − Y 2F . (5.7.20) 2 In minimizing Q(X, Y) with respect to X, the second additive term f (Y) is cons1 tant and can be ignored. On the other hand, another constant term, ∇f (Y) 2F = 2L

310

Singular Value Analysis

2 L 1 ∇f (Y) , 2 L F

can be added in Q(X, Y). Then, we have

min Q(X, Y) X " # L 1 L 1 2 2 = min X ∗ + X − Y F + LX − Y, ∇f (Y) + ∇f (Y) F 2 L 2 L X " # L 1 (by Proposition 5.1) = min X ∗ + X − Y + ∇f (Y) 2F 2 L X which yields Xk+1

" # L 1 2 = arg min X ∗ + X − Yk + ∇f (Yk ) F 2 L X   −1 = proxL−1 · ∗ Yk − L ∇f (Yk ) = Usoft(Σ, t)VT

(by Theorem 5.9),

(5.7.21)

where t = 1/L and UΣVT is the SVD of the matrix Yk − tk ∇f (Yk ). 1 From f (Y) = λ PΩ Y − PΩ D 2F − tr(Al YBTl ) it is easy to see that ∇f (Y) = 2 λ(PΩ Y − PΩ D) − ATl Bl , and thus   (5.7.22) (U, Σ, V) = svd Yk + tk (ATl Bl − λ(PΩ Yk − PΩ D)) . Algorithm 5.8 give the accelerated proximal gradient line (APGL) search algorithm for truncated nuclear norm minimization that was presented in [221]. Algorithm 5.8

Truncated nuclear norm minimization via APGL search

input: Al , Bl ; DΩ = PΩ D and tolerance . initialization: t1 = 1, X1 = DΩ , Y1 = X1 . repeat

  1. Compute Wk+1 = Yk + tk ATl Bl − λ(PΩ Yk − PΩ D) . 2. Perform the SVD (U, Σ, V) = svd(Wk+1 ). min{m,n} max{σi − tk , 0}ui viT . 3. Compute Xk+1 = i=1 4. exit if Xk+1 − Xk F ≤ . 5. Compute tk+1 =

1+

1 + 4t2k 2

6. Update Yk+1 = Xk+1 +

.

tk − 1 (Xk+1 tk+1

− Xk ).

return k ← k + 1. output: Xopt = Xk .

However, the truncated nuclear norm minimization problem (5.7.19) can also be written in the standard form of the alternating-direction method of multipliers

5.7 Matrix Completion

(ADMM): 3 4 min X ∗ − tr(Al WBTl ) X,W

subject to X = W, PΩ W = PΩ D.

311

(5.7.23)

The corresponding augmented Lagrangian objective function is given by L(X, Y, W, β) = X ∗ − tr(Al WBTl ) +

β

X − W 2F + Y, X − W, 2

where β > 0 is the penalty parameter. Ignoring the constant term tr(Al WBTl ) and adding the constant term β −1 Y 2F , we obtain " # β 1 2 2 Xk+1 = arg min X ∗ + X − Wk F + Yk , X − Wk  + Yk F 2 β X " # β 1 = arg min X ∗ + X − (Wk − Yk ) 2F 2 β X = proxβ −1 ·|∗ (Wk − β −1 Yk ) = Usoft(Σ, β −1 )VT ,

(5.7.24)

where UΣVT is the SVD of the matrix Wk − β −1 Yk . Similarly, ignoring the constant term Xk+1 ∗ and adding two constant terms Xk+1 , ATl Bl  and (2β)−1 Yk − ATl Bl 2F , we have Wk+1 = arg min L(Xk+1 , W, Yk , β) PΩ W=PΩ D

" β = arg min −tr(Al WBTl ) + Xk+1 − W 2F + Xk+1 , ATl Bl  2 PΩ W=PΩ D # 1

Yk − ATl Bl 2F +Yk , Xk+1 − W + 2β + 2  22 0 2 β2 1 T 2 W − Xk+1 − (Yk + Al Bl ) 2 . (5.7.25) = arg min 2 2 2 β PΩ W=PΩ D F Its closed solution is given by Wk+1 = Xk+1 −

1 (Yk + ATl Bl ), β

Wk+1 = PΩ (Wk+1 ) + PΩ (D).

(5.7.26) (5.7.27)

Algorithm 5.9 lists the ADMM algorithm for truncated nuclear norm minimization. Unlike traditional nuclear norm heuristics, which take into account all the singular values, the truncated nuclear norm regularization approach in [221] aims to minimize the sum of the smallest min{m, n} − r singular values, where r is the matrix rank. This helps to give a better approximation to the matrix rank when the original matrix has a low-rank structure.

312

Singular Value Analysis

Algorithm 5.9

Truncated nuclear norm minimization via ADMM [221]

input: Al , Bl ; Dij , (i, j) ∈ Ω, tolerance  and singular value threshold τ . initialization: X1 = DΩ , W1 = X1 , Y1 = X1 and β = 1. repeat 1. Compute the SVD (U, Σ, V) = svd(Wk − β −1 Yk ). min{m,n} max{σi − β −1 , 0}ui viT . 2. Compute Xk+1 = i=1 3. exit if Xk+1 − Xk F ≤ . 4. Update Wk+1 = Xk+1 − β −1 (Yk + ATl Bl ). 5. Update Wk+1 = PΩ (Wk+1 ) + PΩ (D). 6. Compute Yk+1 = Yk + β(Xk+1 − Wk+1 ). return k ← k + 1. output: Xopt = Xk .

Exercises 5.1

Given the matrix

5.2

find its SVD. Use the SVD of the matrix



⎤ 1 1 A = ⎣1 1⎦ , 0 0



1 A = ⎣0 1

⎤ 0 1⎦ 1

5.4

to find its Moore–Penrose inverse A† , and verify that it satisfies the four Moore–Penrose conditions. Prove that if A is a normal matrix then each of its singular values is the modulus of the corresponding eigenvalue of AT A. Compute the SVD of the following two matrices: ⎤ ⎡   1 −1 3 4 5 ⎣ ⎦ . A= and A = 3 −3 2 1 7 −3 3

5.5

Given the matrix

5.3



−149 A = ⎣ 537 −27

−50 180 9

⎤ −154 546 ⎦ , −25

find the singular values of A and the left and right singular vectors associated with the smallest singular value.

Exercises

5.6

5.7 5.8 5.9

313

Given a complex matrix A = B + jC, where B, C ∈ Rm×n , use the SVD of B −C the block matrix to represent the SVD of A. C B Let A = xpH + yqH , where x ⊥ y and p ⊥ q. Find the Frobenius norm

A F . (Hint: Compute AH A, and find the singular values of A.) Let A = UΣVH be the SVD of A. What is the relationship between the singular values of AH and A? Given an n × n (where n ≥ 3) matrix ⎡ ⎤ 1 a ··· a ⎢a 1 · · · a ⎥ ⎢ ⎥ A = ⎢. . . ⎥, . . ... ⎦ ⎣ .. .. a a ··· 1

use the SVD to find the value of a such that rank(A) = n − 1. 5.10 Show that if A is a square matrix then | det(A)| is equal to the product of the singular values of A. 5.11 Suppose that A is a invertible matrix. Find the SVD of A−1 . 5.12 Show that if A is an n × n positive definite matrix then the singular values of A and its eigenvalues are the same. 5.13 Let A be an m × n matrix and P be an m × m positive definite matrix. Prove that the singular values of PA and A are the same. What is the relationship between the left and right singular vectors? 5.14 Let A be an m × n matrix and λ1 , . . . , λn be the eigenvalues of the matrix AT A with the corresponding eigenvectors u1 , . . . , un . Prove that the singular values σi of A are equal to the norms Aui , namely σi = Aui , i = 1, . . . , n. 5.15 Let λ1 , . . . , λn and u1 , . . . , un be the eigenvalues and eigenvectors of the matrix AT A, respectively. Suppose that A has r nonzero singular values. Show that {Au1 , . . . , Aur } is an orthogonal basis set of the column space Col(A) and that rank(A) = r. 5.16 Let B, C ∈ Rm×n . Find the relationship between the singular values and the singular vectors  of the complex matrix A = B + j C and those of the real B −C . block matrix C B 5.17 Use the singular vectorsof the matrix A ∈ Rm×n (m ≥ n) to represent the O AT . eigenvectors of A O   ai ∈ Rn+1 to the straight line 5.18 Prove that the distance from the point bi xT a = b is equal to ; |aTi x − bi |2 di = . xT x + 1

314

Singular Value Analysis

5.19 Adopt the MATLAB function [U, S, V] = svd(X) to solve the matrix equation Ax = b, where ⎡ ⎤ ⎡ ⎤ 1 1 1 1 ⎢ 3 1 3⎥ ⎢4⎥ ⎥ ⎢ ⎥ A=⎢ ⎣1 0 1⎦ , b = ⎣3⎦ . 2 2 1 2 5.20 Suppose that the observation data of a computer simulation are generated by √ √ x(n) = 20 sin(2π 0.2n) + 2 sin(2π 0.215n) + w(n), where w(n) is a Gaussian white noise with mean 0 and covariance 1 and n = 1, 2, . . . , 128. Perform 50 independent runs. Determine the effective rank of the autocorrelation matrix ⎡ ⎤ r(0) r(−1) ··· r(−2p) ⎢ r(1) r(0) · · · r(−2p + 1)⎥ ⎢ ⎥ R=⎢ . ⎥, .. . .. .. ⎣ .. ⎦ . . r(M )

r(M − 1)

where M = 50, p = 10, and r(k) =

···

r(M − 2p)

128−k  1 x(i)x(i 128 i=1

+ k) represents sample

autocorrelation functions of the observation data (unknown observation data are taken as zero). 5.21 [179] Apply the SVD to show that if A ∈ Rm×n (m ≥ n) then there are Q ∈ Rm×n and P ∈ Rn×n such that A = QP, where QT Q = In , and P is symmetric and nonnegative definite. This decomposition is sometimes called the polar decomposition, because it is similar to the complex decomposition z = |z|e j arg(z) .

6 Solving Matrix Equations

In a wide variety of science and engineering disciplines, we often encounter three main types of matrix equation: (1) Over-determined matrix equations (ODMEs) Am×n xn×1 = bm×1 with m > n; the data matrix A and the data vector b are known, but one or both may have errors or interference. (2) Blind matrix equations (BMEs) Only the data matrix X is known, while the coefficient matrix A and the parameter matrix S are unknowns in the equation Am×n Sn×p = Xm×p . (3) Under-determined sparse matrix equations (UDSMEs) Ax = b with m < n; the data vector b and the data matrix A are known, but the data vector b is a sparse vector. In this chapter we discuss the methods for solving these three types of matrix equation: ⎧ ⎧ ⎪ Least squares ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨Data least squares ⎪ ⎪ ⎪ ⎪ ODMEs Tikhonov method and Gauss–Seidel methods ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ Total least squares ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ ⎪ Constrained total least squares ⎪ ⎪ ⎪ ⎪ + ⎪ ⎪ ⎪ Subspace methods ⎪ Matrix ⎨BMEs Nonnegative matrix decomposition equations ⎪ ⎪ ⎪ ⎧ ⎪ ⎪ ⎪ ⎪ Orthogonal matching pursuit ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ Gradient projection algorithm ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ Lasso and LARS methods ⎪ ⎪ UDSMEs ⎪ ⎪ ⎪ ⎪ Homotopy algorithm ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ Augmented Lagrange multiplier ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ ⎩ Bregman iteration algorithms 315

316

Solving Matrix Equations

6.1 Least Squares Method Linear parameter estimation is widely used in science and technology problems, and the least squares method is the most common.

6.1.1 Ordinary Least Squares Methods Consider an over-determined matrix equation Ax = b, where b is an m × 1 data vector, A is an m × n data matrix, and m > n. Suppose that in the data vector there is additive observation error or noise, i.e., b = b0 + e, where b0 and e are the errorless data vector and the additive error vector, respectively. In order to counterbalance the influence of the error e on the matrix equation solution, we introduce a correction vector Δb and use it to “perturb” the data vector b. Our goal is to make the correction term Δb that compensates the uncertainty existing in the data vector b as small as possible so that b + Δb → b0 and hence the following transformation is achieved: ⇒

Ax = b + Δb

Ax = b0 .

(6.1.1)

That is to say, if we select the correction vector Δb = Ax − b directly, and make Δb as small as possible, we can achieve the solution of errorless matrix equation Ax = b0 . This idea for solving the matrix equation can be described by the optimization problem   min Δb 2 = Ax − b 22 = (Ax − b)T (Ax − b) . (6.1.2) x

Such a method is known as the ordinary least squares (OLS) method, usually called simply the least squares (LS) method. As a matter of fact, the correction vector Δb = Ax − b is just the error vector of both sides of the matrix equation Ax = b. Hence the central idea of the LS method is to find the solution x such that the sum of the squared error Ax − b 22 is minimized, namely ˆ LS = arg min Ax − b 22 . x x

(6.1.3)

In order to derive an analytic solution for x, expand Equation (6.1.2) to get φ = xT AT Ax − xT AT b − bT Ax + bT b. Find the derivative of φ with respect to x and let the result equal zero; thus dφ = 2AT Ax − 2AT b = 0. dx That is to say, the solution x must satisfy AT Ax = AT b.

(6.1.4)

6.1 Least Squares Method

317

The above equation is “identifiable” or “unidentifiable” depending on the rank of the m × n matrix A. • Identifiable When Ax = b is an over-determined equation, it has a unique solution xLS = (AT A)−1 AT b

if rank(A) = n,

(6.1.5)

or xLS = (AT A)† AT b

if rank(A) < n,

(6.1.6)

where B† denotes the Moore–Penrose inverse matrix of B. In parameter estimation theory the unknown parameter vector x is said to be uniquely identifiable if it is uniquely determined. • Unidentifiable For an under-determined equation Ax = b, if rank(A) = m < n, then different solutions for x give the same value of Ax. Clearly, although the data vector b can provide some information about Ax, we cannot distinguish the different parameter vectors x corresponding to the same Ax. Such a unknown vector x is said to be unidentifiable.

6.1.2 Properties of Least Squares Solutions In parameter estimation, an estimate θˆ of the parameter vector θ is called an unbiased estimator, if its mathematical expectation is equal to the true unknown paˆ = θ. Further, an unbiased estimator is called the optimal rameter vector, i.e., E{θ} unbiased estimator if it has the minimum variance. Similarly, for an over-determined matrix equation Aθ = b + e with noisy data vector b, if the mathematical expectation of the LS solution θˆLS is equal to the true parameter vector θ, i.e., E{θˆLS } = θ, and has the minimum variance, then θˆLS is the optimal unbiased solution. THEOREM 6.1 (Gauss–Markov theorem)

Consider the set of linear equations

Ax = b + e,

(6.1.7)

where the m × n matrix A and the n × 1 vector x are respectively a constant matrix and the parameter vector; b is the m × 1 vector with random error vector e = [e1 , . . . , em ]T whose mean vector and covariance matrix are respectively E{e} = 0,

Cov(e) = E{eeH } = σ 2 I.

ˆ if and only if Then the n × 1 parameter vector x is the optimal unbiased solution x rank(A) = n. In this case the optimal unbiased solution is given by the LS solution ˆ LS = (AH A)−1 AH b, x

(6.1.8)

Var(ˆ xLS ) ≤ Var(˜ x),

(6.1.9)

and its covariance satisfies

318

Solving Matrix Equations

˜ is any other solution of the matrix equation Ax = b + e. where x Proof

From the assumption condition E{e} = 0, it follows that E{b} = E{Ax} − E{e} = Ax.

(6.1.10)

Using Cov(e) = E{eeH } = σ 2 I, and noticing that Ax is statistically uncorrelated with the error vector e, we have E{bbH } = E{(Ax − e)(Ax − e)H } = E{AxxH AH } + E{eeH } = AxxH AH + σ 2 I.

(6.1.11)

Since rank(A) = n and the matrix product AH A is nonsingular, we get E{ˆ xLS } = E{(AH A)−1 AH b} = (AH A)−1 AH E{b} = (AH A)−1 AH Ax = x, ˆ LS = (AH A)−1 AH b is an unbiased solution to the matrix i.e., the LS solution x equation Ax = b + e. ˆ LS has the minimum variance. To this end we suppose that Next, we show that x ˜ that is given by x has an alternative solution x ˜=x ˆ LS + Cb + d, x where C and d are a constant matrix and a constant vector, respectively. The ˜ is unbiased, i.e., solution x E{˜ x} = E{ˆ xLS } + E{Cb} + d = x + CAx + d = x + CAx + d,

∀ x,

if and only if CA = O (the zero matrix),

d = 0.

(6.1.12)

Using these constraint conditions, it can be seen that E{Cb} = CE{b} = CAθ = 0. Then we have 3  H 4 Cov(˜ x) = Cov(ˆ xLS + Cb) = E (ˆ xLS − x) + Cb (ˆ xLS − x) + Cb     xLS − x)(Cb)H + E Cb(ˆ xLS − x)H = Cov(ˆ xLS ) + E (ˆ   + E CbbH CH . (6.1.13)

6.1 Least Squares Method

319

However, from (6.1.10)–(6.1.12), we have       E (ˆ xLS − x)(Cb)H = E (AH A)−1 AH bbH CH − E xbH CH     = (AH A)−1 AH E bbH CH − xE bH CH   = (AH A)−1 AH AxxH AH + σ 2 I CH − xxH AH CH = O,   H E Cb(ˆ xLS − x) = O, = E (ˆ xLS − x)(Cb)H     H H  H H = CE bb C = C AxxH AH + σ 2 I CH E Cbb C 

 H

= σ 2 CCH . Hence, (6.1.13) can be simplified to Cov(˜ x) = Cov(ˆ xLS ) + σ 2 CCH .

(6.1.14)

By the property tr(A + B) = tr(A) + tr(B) of the trace function, and noticing that for a random vector x with zero mean, tr(Cov(x)) = Var(x), we can rewrite (6.1.14) as xLS ), Var(˜ x) = Var(ˆ xLS ) + σ 2 tr(CCH ) ≥ Var(ˆ ˆ LS has the minimum variance, and hence because tr(CCH ) ≥ 0. This shows that x is the optimal unbiased solution to the matrix equation Ax = b + e. Notice that the condition in the Gauss–Markov theorem Cov(e) = σ 2 I implies that all components of the additive noise vector e are mutually uncorrelated and have the same variance σ 2 . Only in this case is the LS solution unbiased and optimal. Now consider the relationship between the LS solution and the maximum likelihood (ML) solution under the conditions of the Gauss–Markov theorem. If the additive noise vector e = [e1 , . . . , em ]T is a complex Gaussian random vector with independently identical distribution (iid) then from (1.5.18) it is known that the pdf of e is given by f (e) =

  1 exp −(e − μe )H Γ−1 e (e − μe ) , π m |Γe |

(6.1.15)

2 . where |Γe | = σ12 · · · σm Under the conditions of the Gauss–Markov theorem (i.e., all the iid Gaussian random variables of the error vector have zero mean and the same variance σ 2 ), the pdf of the additive error vector reduces to     1 1 H 1 1 2 f (e) = (6.1.16) exp − 2 e e = exp − 2 e 2 , (πσ 2 )m σ (πσ 2 )m σ

whose likelihood function is L(e) = log f (e) = −

1 1

e 22 = − m 2(m+1) Ax − b 22 . π m σ 2(m+1) π σ

(6.1.17)

320

Solving Matrix Equations

Thus the ML solution of the matrix equation Ax = b is given by " # 1 2 ˆ ML = arg max − m 2(m+1) Ax − b 2 x x π σ " # 1 2 ˆ LS .

Ax − b 2 = x = arg min x 2

(6.1.18)

That is to say, under the conditions of the Gauss–Markov theorem, the ML solution ˆ LS are equivalent for the matrix equation Ax = b. ˆ ML and the LS solution x x It is easily seen that when the error vector e is a zero-mean Gaussian random vector but its entries have differing variances since the covariance matrix Γe = σ 2 I, the ML solution in this case is given by " #  H −1  1 ˆ ML = arg max exp −e Γe e , x 2 x π m σ12 · · · σm ˆ LS ; namely, the LS solution is no which is obviously not equal to the LS solution x longer optimal in the case of differing variances.

6.1.3 Data Least Squares Consider again an over-determined matrix equation Ax = b. Unlike in an ordinary LS problem, it is assumed that the data vector b does not contain the observed error or noise; on the contrary, the data matrix A = A0 + E contains the observed error or noise, and every element of the error matrix E obeys an independent Gaussian distribution with the zero-mean and equal variance. The perturbation matrix ΔA is used to correct the erroneous-data matrix A in such a way that A + ΔA = A0 + E + ΔA → A0 . As with the ordinary LS method, by forcing (A + ΔA)x = b to compensate the errors in the data matrix, one can obtain (A + ΔA)x = b ⇒ A0 x = b. In this case, the optimal solution of x is given by ˆ DLS = arg min ΔA 22 x x

subject to

b ∈ Range(A + ΔA).

(6.1.19)

This is the data least squares (DLS) method. In the above equation, the constraint condition b ∈ Range(A + ΔA) implies that, for every given exact data vector b ∈ Cm and the erroneous data matrix A ∈ Cm×n , we can always find a vector x ∈ Cn such that (A+ΔA)x = b. Hence the two constraint conditions b ∈ Range(A+ΔA) and (A + ΔA)x = b are equivalent. By using the Lagrange multiplier method, the constrained data LS problem (6.1.19) becomes the unconstrained optimization problem   min L(x) = tr(ΔA(ΔA)H ) + λH (Ax + ΔAx − b) . (6.1.20)

6.2 Tikhonov Regularization and Gauss–Seidel Method

321

Let the conjugate gradient matrix ∂L(x)/(∂ΔAH ) = O (the zero matrix). Then we have ΔA = −λxH . Substitute ΔA = −λxH into the constraint condition (A + ΔA)x = b to yield λ = (Ax − b)/(xH x), and thus ΔA = −(Ax − b)xH /(xH x). Therefore, the objective function is given by   (Ax − b)xH x(Ax − b)H 2 H . J(x) = ΔA 2 = tr(ΔA(ΔA) ) = tr xH x xH x Then, using the trace function property tr(BC) = tr(CB), we immediately have (Ax − b)H (Ax − b) J(x) = , (6.1.21) xH x which yields (Ax − b)H (Ax − b) ˆ DLS = arg min . (6.1.22) x x xH x This is the data least squares solution of the over-determined equation Ax = b.

6.2 Tikhonov Regularization and Gauss–Seidel Method Suppose that we are solving an over-determined matrix equation Am×n xn×1 = bm×1 (where m > n). The ordinary LS method and the data LS method are based on two basic assumptions: (1) the data matrix A is nonsingular or of full column rank; (2) the data vector b or the data matrix A includes additive noise or error. In this section we discuss the regularization method for solving matrix equations with rank deficiency or noise.

6.2.1 Tikhonov Regularization When m = n and A is nonsingular, the solution of the matrix equation Ax = b ˆ = A−1 b; and when m > n and Am×n is of full column rank, the is given by x ˆ LS = A† b = (AH A)−1 AH b. solution is x A problem is that the data matrix A is often rank deficient in engineering applicaˆ = A−1 b or x ˆ LS = (AH A)−1 AH b may diverge; tions. In these cases, the solution x even if a solution exists, it may be a bad approximation to the unknown vector x. If, however, we were lucky and happened to find a reasonable approximation of ˆ ≤ A−1 Aˆ ˆ ≤ A† Aˆ x, the error estimate x − x x − b or x − x x − b would be greatly disappointing [347]. By observation, it can easily be seen that the problem lies in the inversion of the covariance matrix AH A of the rank-deficient data matrix A. As an improvement of the LS cost function 12 Ax − b 22 , Tikhonov [464] in 1963 proposed the regularized least squares cost function  1 J(x) = (6.2.1)

Ax − b 22 + λ x 22 , 2

322

Solving Matrix Equations

where λ ≥ 0 is the regularization parameter. The conjugate gradient of the cost function J(x) with respect to the argument x is given by  ∂J(x) ∂  = (Ax − b)H (Ax − b) + λxH x = AH Ax − AH b + λx. ∂xH ∂xH Let ∂J(x)/∂ xH = 0; then ˆ Tik = (AH A + λI)−1 AH b. x

(6.2.2)

This method using (AH A + λI)−1 instead of the direct inverse of the covariance matrix (AH A)−1 is called the Tikhonov regularization method (or simply the regularized method). In signal processing and image processing, the regularization method is sometimes known as the relaxation method. The Tikhonov-regularization based solution to the matrix equation Ax = b is called the Tikhonov regularization solution, denoted xTik . The idea of the Tikhonov regularization method is that by adding a very small perturbation λ to each diagonal entry of the covariance matrix AH A of a rank deficient matrix A, the inversion of the singular covariance matrix AH A becomes the inversion of a nonsingular matrix AH A + λI, thereby greatly improving the numerical stability of the solution process for the rank-deficient matrix equation Ax = b. Obviously, if the data matrix A is of full column rank but includes error or noise, we must adopt the opposite method to Tikhonov regularization and add a very small negative perturbation, −λ, to each diagonal entry of the covariance matrix AH A. Such a method using a very small negative perturbation matrix −λI is called the anti-Tikhonov regularization method or anti-regularized method, and the solution is given by ˆ = (AH A − λI)−1 AH b. x

(6.2.3)

The total least squares method is a typical anti-regularized method and will be discussed later. As stated above, the regularization parameter λ should take a small value so that (AH A + λI)−1 can better approximate (AH A)−1 and yet can avoid the singularity of AH A; hence the Tikhonov regularization method can obviously improve the numerical stability in solving the singular and ill-conditioned equations. This is so because the matrix AH A is positive semi-definite, so the eigenvalues of AH A + λI lie in the interval [λ, λ + A 2F ], which gives a condition number cond(AH A + λI) ≤ (λ + A 2F )/λ.

(6.2.4)

Clearly, as compared with the condition number cond(AH A) ≤ ∞, the condition number of the Tikhonov regularization is a great improvement. In order to improve further the solution of singular and ill-conditioned matrix equations, one can adopt iterative Tikhonov regularization [347].

6.2 Tikhonov Regularization and Gauss–Seidel Method

323

Let the initial solution vector x0 = 0 and the initial residual vector r0 = b; then the solution vector and the residual vector can be updated via the following iteration formulas: xk = xk−1 + (AH A + λI)−1 AH rk−1 , (6.2.5) rk = b − Axk , k = 1, 2, . . . Let A = UΣVH be the SVD of the matrix A, then AH A = VΣ2 VH and thus the ordinary LS solution and the Tikhonov regularization solution are respectively given by ˆ LS = (AH A)−1 AH b = VΣ−1 UH b, x H

ˆ Tik = (A A + x

2 σmin I)−1 AH b

(6.2.6) 2

= V(Σ +

2 σmin I)−1 ΣUH b,

(6.2.7)

where σmin is the minimal nonzero singular value of A. If the matrix A is singular or ill-conditioned, i.e., σn = 0 or σn ≈ 0, then some diagonal entry of Σ−1 will be 1/σn = ∞, which thereby makes the LS solution diverge. On the contrary, the SVD ˆ Tik has good numerical stability because based Tikhonov regularization solution x the diagonal entries of the matrix   σn σ1 2 2 −1 (Σ + δ I) Σ = Diag (6.2.8) ,..., 2 2 2 σ12 + σmin σn + σmin 2 lie in the interval [0, σ1 (σ12 + σmin )−1 ]. When the regularization parameter λ varies in the definition interval [0, ∞), the family of solutions for a regularized LS problem is known as its regularization path. A Tikhonov regularization solution has the following important properties [248].

ˆ Tik = (AH A + λI)−1 AH b 1. Linearity The Tikhonov regularization LS solution x is a linear function of the observed data vector b. 2. Limit characteristic when λ → 0 When the regularization parameter λ → 0, the Tikhonov regularization LS solution converges to the ordinary LS solution ˆ LS = A† b = (AH A)−1 AH b. The ˆ Tik = x or the Moore–Penrose solution lim x λ→0 ˆ Tik has the minimum 2 -norm among all the feasible points solution point x meeting AH (Ax − b) = 0: ˆ Tik = x

arg min AT (b−Ax)=0

x 2 .

(6.2.9)

3. Limit characteristic when λ → ∞ When λ → ∞, the optimal solution of the ˆ Tik = 0. Tikhonov regularization LS problem converges to the zero vector: lim x λ→∞

4. Regularization path When the regularization parameter λ varies in [0, ∞), the optimal solution of the Tikhonov regularization LS problem is a smooth function of the regularization parameter, i.e., when λ decreases to zero, the optimal solution converges to the Moore–Penrose solution and when λ increases, the optimal solution converges to the zero vector.

324

Solving Matrix Equations

Tikhonov regularization can effectively prevent the divergence of the LS solution ˆ LS = (AT A)−1 AT b when A is rank deficient; thereby it obviously improves the x convergence property of the LS algorithm and the alternating LS algorithm, and so is widely applied.

6.2.2 Regularized Gauss–Seidel Method Let Xi ⊆ Rni be the feasible set of the ni × 1 vector xi . Consider the nonlinear minimization problem   (6.2.10) min f (x) = f (x1 , . . . , xm ) , x∈X

where x ∈ X = X1 × X2 × · · · × Xm ⊆ Rn is the Cartesian product of a closed m  nonempty convex set Xi ⊆ Rni , i = 1, . . . , m, and ni = n. i=1

Equation (6.2.10) is an unconstrained optimization problem with m coupled variable vectors. An efficient approach for solving this class of coupled optimization problems is the block nonlinear Gauss–Seidel method, called simply the GS method [42], [190]. In every iteration of the GS method, m − 1 variable vectors are regarded as known, and the remaining variable vector is minimized. This idea constitutes the basic framework of the GS method for solving nonlinear unconstrained optimization problem (6.2.10). (1) Initialize m − 1 variable vectors xi , i = 2, . . . , m, and let k = 0. (2) Find the solution of the separated suboptimization problem   k k xk+1 = arg min f xk+1 , . . . , xk+1 i = 1, . . . , m. (6.2.11) 1 i i−1 , y, xi+1 , . . . , xm , y∈Xi

At the (k + 1)th iteration of updating the vector xi , the vectors x1 , . . . , xi−1 have been updated as xk+1 , . . . , xk+1 1 i−1 , so these updated subvectors and the vectors xki+1 , . . . , xkm yet be updated are to be regarded as known. (3) Test whether the m variable vectors are all convergent. If they are convergent then output the optimization results (xk+1 , . . . , xk+1 m ); otherwise, let k ← k + 1 1, return to Equation (6.2.11) and continue to iterate until the convergence criterion is met. If the objective function f (x) of the optimization (6.2.10) is an LS error function (for example Ax − b 22 ), then the GS method is customarily called the alternating least squares (ALS) method. EXAMPLE 6.1 Consider the full-rank decomposition of an m × n known data matrix X = AB, where the m × r matrix A is of full column rank and the r × n matrix

6.2 Tikhonov Regularization and Gauss–Seidel Method

325

B is of full row rank. Let the cost function of the matrix full-rank decomposition be 1 f (A, B) = X − AB 2F . (6.2.12) 2 Then the ALS algorithm first initializes the matrix A. At the (k + 1)th iteration, from the fixed matrix Ak we update the LS solution of B as follows: Bk+1 = (ATk Ak )−1 ATk X.

(6.2.13)

Next, from the transpose of the matrix decomposition XT = BT AT , we can immediately update the LS solution of AT : ATk+1 = (Bk+1 BTk+1 )−1 Bk+1 XT .

(6.2.14)

The above two kinds of LS procedures are performed alternately. Once the ALS algorithm converges, the optimization results of the matrix decomposition can be obtained. To analyze the convergence of the GS method, we first introduce the concepts of the limit point and the critical point. Let S be a subset in topological space X; a point x in the space X is called a limit point of the subset S if every neighbourhood of x contains at least one point in S apart from x itself. In other words, the limit point x is a point in topological space X approximated by a point in S (other than x). A point on a function curve at which the derivative is equal to zero or does not exist plays an important role in the optimization problem. A point x is known as a critical point of the function f (x) if x lies in the definition domain of the function and the derivative f (x) = 0 or does not exist. The geometric interpretation of a critical point is that the tangent of the point on the curve is horizontal or vertical, or does not exist.

3 Consider a real function f (x) = x4 − 4x2 whose derivative √ f (x)√= 4x − 8x.

From f (x) = 0 we get three critical points of f (x): x = 0, − 2 and 2. The above concepts on the limit point and critical points of a scalar variable are easily extended to a vector variable. DEFINITION 6.1 The vector x ∈ Rn is said to be a limit point of the vector n ∞ sequence {xk }∞ k=1 in the vector space R if there is a subsequence of {xk }k=1 that converges to x. DEFINITION 6.2 Let f : X → R (where X ⊂ Rn ) be a real function; the vector ¯ ∈ Rn is known as a critical point of the function f (x) if the following condition x is satisfied: ¯ ) ≥ 0, gT (¯ x)(y − x

∀ y ∈ X,

(6.2.15)

where gT (¯ x) represents the transpose of the vector function g(¯ x) and either g(¯ x) =

326

Solving Matrix Equations

∇f (¯ x) is the gradient vector of a continuous differentiable function f (x) at the ¯ or g(¯ point x x) ∈ ∂f (¯ x) is the subgradient vector of a nondifferentiable nonsmooth ¯. function f (x) at the point x ¯ is an interior point in X then the critical-point condition (6.2.15) If X = Rn or x reduces to the stationary-point condition 0 = ∇f (¯ x) (for a continuous differentiable objective function) or 0 ∈ ∂f (¯ x) (for a nonsmooth objective function) for the unconstrained minimization problem min f (x). Let xk = (xk1 , . . . , xkm ) denote the iteration results generated by the GS algorithm; it is expected that the iterative sequence {xk }∞ k=1 has limit points and that each limit point is a critical point of the objective function f . For the optimization problem (6.2.10) this convergence performance of the GS algorithm depends on the quasi-convexity of the objective function f . Put α ∈ (0, 1) and yi = xi . The objective function f (x) in the ALS problem (6.2.10) is called the quasi-convex with respect to xi ∈ Xi if f (x1 , . . . , xi−1 , αxi + (1 − α)yi , xi+1 , . . . , xm ) ≤ max{f (x), f (x1 , . . . , xi−1 , yi , xi+1 , . . . , xm )}. The function f (x) is known as strictly quasi-convex, if f (x1 , . . . , xi−1 , αxi + (1 − α)yi , xi+1 , . . . , xm ) < max{f (x), f (x1 , . . . , xi−1 , yi , xi+1 , . . . , xm )}. The following give the convergence performance of the GS algorithm under different assumption conditions. THEOREM 6.2 [190] Let the function f be strictly quasi-convex with respect to the vectors xi (i = 1, . . . , m − 2) in X, and let the sequence generated by the GS ¯ of {xk } is a critical point method, {xk }, have limit points; then every limit point x of the optimization problem (6.2.10). THEOREM 6.3 [42] For the objective function f in Equation (6.2.10), it is assumed that, for every i and x ∈ X, the minimum point of the optimization algorithm min f (x1 , . . . , xi−1 , y, xi+1 , . . . , xm )

y∈Xi

is uniquely determined. If {xk } is the iterative sequence generated by the GS algorithm then every limit point of xk is a critical point. However, in practical applications, the objective function in the optimization (6.2.10) often does not satisfy the conditions of the above two theorems. For example, by Theorem 6.2, in the case of a rank-deficient matrix A, the quadratic objective function Ax − b 22 is not quasi-convex, so the convergence is not be guaranteed.

6.2 Tikhonov Regularization and Gauss–Seidel Method

327

The sequence generated by the GS method contains limit points, but they may not be the critical points of the optimization problem (6.2.10). The fact that the GS algorithm may not converge was observed by Powell in 1973 [399], who called it the “circle phenomenon” of the GS method. Recently, a lot of simulation experiences have shown [338], [293] that even it converges, the iterative process of the ALS method easily falls into a “swamp”: an unusually large number of iterations leads to a very slow convergence rate. In particular, as long as the column rank of one variable matrix in a set of m variable matrices is deficient, or, although the m variable matrices are of full-column rank, there is collinearity among the column vectors of some of the variable matrices, one easily observes this swamp phenomenon [338], [293]. A simple and effective way for avoiding the circle and swamp phenomena of the GS method is to make a Tikhonov regularization of the objective function in the optimization problem (6.2.10); namely, the separated suboptimization algorithm (6.2.11) is regularized as   1 k k k 2 xk+1 = arg min f (xk+1 , . . . , xk+1 1 i i−1 , y, xi+1 , . . . , xm ) + τi y − xi 2 , 2 y∈Xi (6.2.16) where i = 1, . . . , m. The above algorithm is called the proximal point version of the GS methods [18], [44], abbreviated as the PGS method. The role of the regularization term y − xki 22 is to force the updated vector k+1 xi = y to be close to xki ; this avoids any violent shock in the iterative process and so prevents the divergence of the algorithm. The GS or PGS method is said to be well defined, if each suboptimization problem has an optimal solution [190]. THEOREM 6.4 [190] If the PGS method is well-defined, and in the sequence {xk } ¯ of {xk } is a critical point of the there exist limit points, then every limit point x optimization problem (6.2.10). This theorem shows that the convergence performance of the PGS method is better than that of the GS method. Many simulation experiments show [293] that under the condition of achieving the same error, the iterative number of the GS method in a swamp iteration is unusually large whereas the PGS method tends to converge quickly. The PGS method is also called the regularized Gauss–Seidel method in some literature. The ALS method and the regularized ALS method have important applications in nonnegative matrix decomposition and tensor analysis; these will be discussed in later chapters.

328

Solving Matrix Equations

6.3 Total Least Squares (TLS) Methods Although its original name is different, the total least squares (TLS) method has a long history. The earliest ideas about TLS can be traced back to the paper of Pearson in 1901 [381] who considered an approximate method for solving the matrix equation Ax = b when in both A and b there exist errors. However, as late as 1980, Golub and Van Loan [178] for the first time gave an overall treatment from the point of view of numerical analysis and formally referred to this method as the total least squares. In mathematical statistics the method is called orthogonal regression or errors-in-variables regression [175]. In system identification the TLS method is called the characteristic vector method or the Koopmans–Levin method [486]. Now the TLS method is widely used in statistics, physics, economics, biology and medicine, signal processing, automatic control, system science and many other disciplines and fields.

6.3.1 TLS Problems Let A0 and b0 represent an unobservable error-free data matrix and an error-free data vector, respectively. The actual observed data matrix and data vector are respectively given by A = A0 + E,

b = b0 + e,

(6.3.1)

where E and e express the error data matrix and the error data vector, respectively. The basic idea of the TLS is not only to use a correction vector Δb to perturb the data vector b, but also to use a correction matrix ΔA to perturb the data matrix A, thereby making a joint compensation for errors or noise in both A and b: b + Δb = b0 + e + Δb → b0 , A + ΔA = A0 + E + ΔA → A0 . The purpose is to suppress the influence of the observation error or noise on the matrix equation solution in order to transform the solution of a noisy matrix equation into the solution of an error-free matrix equation: (A + ΔA)x = b + Δb



A0 x = b 0 .

(6.3.2)

Naturally, we want the correction data matrix and the correction data vectors to be as small as possible. Hence the TLS problem can be expressed as the constrained optimization problem TLS:

min

ΔA,Δb,x

[ΔA, Δb] 2F

subject to

(A + ΔA)x = b + Δb

(6.3.3)

or TLS:

min D 2F z

subject to Dz = −Bz,

(6.3.4)

6.3 Total Least Squares (TLS) Methods

where D = [ΔA, Δb], B = [A, b] and z =

x −1



329

is an (n + 1) × 1 vector.

Under the assumption z 2 = 1, from D 2 ≤ D F and D 2 = sup Dz 2

z 2 =1

we have

min D 2F z

=

min Dz 22 z

and thus we can rewrite (6.3.4) as

TLS:

min Dz 22

subject to

z = 1

(6.3.5)

TLS:

min Bz 22

subject to

z = 1,

(6.3.6)

z

or z

since Dz = −Bz.

6.3.2 TLS Solution There are two possible cases in the TLS solution of over-determined equations. Case 1

Single smallest singular value

The singular value σn of B is significantly larger than the singular value σn+1 , i.e., there is only one smallest singular value. Equation (6.3.6) shows that the TLS problem can be summarized as follows. Find a perturbation matrix D ∈ Cm×(n+1) with the minimum norm squared such that B + D is of non full rank (if full rank then there is only the trivial solution z = 0). The TLS problem (6.3.6) is easily solved via the Lagrange multiplier method. To this end, define the objective function J(z) = Bz 22 + λ(1 − zH z),

(6.3.7) ∂J(z)

= where λ is the Lagrange multiplier. Noticing that Bz 22 = zH BH Bz, from ∂z∗ 0 it follows that BH Bz = λz.

(6.3.8)

This shows that the Lagrange multiplier should be selected as the smallest eigenvalue λmin of the matrix BH B = [A, b]H [A, b] (i.e., the square of the smallest singular value of B), while the TLS solution vector z is the

eigenvector corresponding to x λmin . In other words, the TLS solution vector −1 is the solution of the Rayleigh quotient minimization problem ⎧ ⎫ H

x x ⎨ [A, b]H [A, b] −1

Ax − b 22 ⎬ −1 min J(x) = = . (6.3.9) H

x x x ⎩

x 22 + 1 ⎭ −1

−1

Let the SVD of the m × (n + 1) augmented matrix B be B = UΣVH ; let its singular values be arranged in the order σ1 ≥ · · · ≥ σn+1 and let their corresponding

330

Solving Matrix Equations

right singular vectors be v1 , . . . , vn+1 . Then, according to the above analysis, the TLS solution is z = vn+1 , namely ⎡ ⎤ v(1, n + 1) 1 ⎢ ⎥ .. xTLS = − (6.3.10) ⎣ ⎦, . v(n + 1, n + 1) v(n, n + 1) where v(i, n + 1) is the ith entry of (n + 1)th column of V. Remark If the augmented data matrix is given by B = [−b, A] then the TLS solution is provided by ⎡ ⎤ v(2, n + 1) 1 ⎢ ⎥ .. xTLS = (6.3.11) ⎣ ⎦. . v(1, n + 1) v(n + 1, n + 1) Case 2

Multiple smallest singular values

In this case, there are multiple smallest singular values of B, i.e., the smallest singular values are repeated or are very close. Let σ1 ≥ σ2 ≥ · · · ≥ σp > σp+1 ≈ · · · ≈ σn+1 ,

(6.3.12)

and vi be any column in the subspace S = Span{vp+1 , vp+2 , . . . , vn+1 }. Then above   y any right singular vector vp+i = p+i gives a TLS solution αp+i xi = −yp+i /αp+i ,

i = 1, . . . , n + 1 − p.

Hence there are n+1−p TLS solutions. The interesting solution is the TLS solution that is unique in some sense. The possible unique TLS solutions are of two kinds: (1) the minimum norm solution, composed of n parameters; (2) the optimal LS approximate solution, containing only p parameters. 1. Minimum Norm Solution The minimum norm solution of the matrix equation Am×n xn×1 = bm×1 is a TLS solution with n parameters. The TLS algorithm for finding the minimum norm solution was proposed by Golub and Van Loan [178]; it is given as Algorithm 6.1. Remark If the augmented data matrix is given by B = [−b, A], then the Householder transformation in step 3 becomes ⎡ ⎤ .. ⎢ α .. 0 · · · 0 ⎥ ⎥ V1 Q = ⎢ (6.3.13) ⎣ - - ..- - - - - - ⎦ , .. y . ×

6.3 Total Least Squares (TLS) Methods Algorithm 6.1

331

TLS algorithm for minimum norm solution

input: A ∈ Cm×n , b ∈ Cm , α > 0. repeat 1. Compute B = [A, b] = UΣVH , and save V and all singular values. 2. Determine the number p of principal singular values. 3. Put V1 = [vp+1 , . . . , vn+1 ], and compute the Householder transformation

. y .. × V1 Q = - - ...- - - - - - , α .. 0 · · · 0 where α is a scalar, and × denotes the irrelevant block. 4. exit if α = 0. return p ← p − 1. output: xTLS = −y/α.

and the output is given by xTLS = y/α. It should be noticed that, like the unknown parameter vector x of the original matrix equation Ax = b, the minimum norm solution xTLS contains n parameters. From this fact it is seen that even if the effective rank of B is p < n, the minimum norm solution still assumes that n unknown parameters of the vector x are mutually independent. As a matter of fact, because both the augmented matrix B = [A, b] and the original data matrix A have the same rank, the rank of A is also p. This implies that only p columns of A are linearly independent and thus that the number of the principal parameters in the original matrix equation Ax = b is p rather than n. To sum up, the minimum norm solution of the TLS problem contains some redundant parameters that is linearly dependent on other parameters. In signal processing and system theory, the unique TLS solution with no redundant parameters is more interesting, since it is the optimal LS approximate solution. 2. Optimal Least Squares Approximate Solution ˆ be an optimal proximation with rank p of the First let the m × (n + 1) matrix B augmented matrix B, i.e., ˆ = UΣp VH , B

(6.3.14)

where Σp = Diag(σ1 , . . . , σp , 0, . . . , 0). ˆ (p) be a submatrix of the m × (n + 1) optimal Then let the m × (p + 1) matrix B j ˆ defined as approximate matrix B, ˆ (p) : submatrix consisting of the jth to the (j + p)th columns of B. ˆ B j

(6.3.15)

332

Solving Matrix Equations

ˆ (p) , B ˆ (p) , . . . , B ˆ (p) Clearly, there are (n + 1 − p) submatrices B 1 2 n+1−p . As stated before, the fact that the efficient rank of B is equal to p means that p components are linearly independent in the parameter vector x. Let the (p + 1) × 1

(p) vector be a = x−1 , where x(p) is the column vector consisting of the p linearly independent unknown parameters of the vector x. Then, the original TLS problem becomes that of solving the following n + 1 − p TLS problems: ˆ (p) a = 0 , B j

j = 1, 2, . . . , n + 1 − p

or equivalently that of solving the synthetic TLS problem ⎡ ⎤ ˆ : p + 1) B(1 ⎢ ⎥ .. ⎣ ⎦ a = 0, . ˆ + 1 − p : n + 1) B(n

(6.3.16)

(6.3.17)

ˆ : p + i) = B ˆ (p) is defined in (6.3.15). It is not difficult to show that where B(i i ˆ : p + i) = B(i

p 

σk uk (vki )H ,

(6.3.18)

k=1

where vki is a windowed segment of the kth column vector of V, defined as vki = [v(i, k), v(i + 1, k), . . . , v(i + p, k)]T .

(6.3.19)

Here v(i, k) is the (i, k)th entry of V. According to the least squares principle, finding the LS solution of Equation (6.3.17) is equivalent to minimizing the measure (or cost) function ˆ : p + 1)a]H B(1 ˆ : p + 1)a + [B(2 ˆ : p + 2)a]H B(2 ˆ : p + 2)a f (a) = [B(1 ˆ + 1 − p : n + 1)a]H B(n ˆ + 1 − p : n + 1)a + · · · + [B(n  n+1−p  ˆ : p + i) a. ˆ : p + i)]H B(i (6.3.20) = aH [B(i i=1

Define the (p + 1) × (p + 1) matrix S(p) =

n+1−p 

ˆ : p + i), ˆ : p + i)]H B(i [B(i

(6.3.21)

i=1

then the measure function can be simply written as f (a) = aH S(p) a.

(6.3.22)

The minimal variable a of the measure function f (a) is given by ∂f (a)/∂a∗ = 0 below: S(p) a = αe1 ,

(6.3.23)

6.3 Total Least Squares (TLS) Methods

333

where e1 = [1, 0, . . . , 0]T and the constant α > 0 represents the error energy. From (6.3.21) and (6.3.18) we have S(p) =

p 

n+1−p 

j=1

i=1

σj2 vji (vji )H .

(6.3.24)

Solving the matrix equation (6.3.23) is simple. If we let S−(p) be the inverse matrix S(p) , then the solution vector a depends only on the first column of the inverse matrix S−(p) . it is easily seen that the ith entry of x(p) = [xTLS (1), . . . , xTLS (p)]T (p)

in the TLS solution vector a = x−1 is given by xTLS (i) = −S−(p) (i, 1)/S−(p) (p + 1, 1),

i = 1, . . . , p.

(6.3.25)

This solution is known as the optimal least squares approximate solution. Because the number of parameters in this solution and the effective rank are the same, it is also called a low-rank TLS solution [74]. Notice that if the augmented matrix B = [−b, A] then xTLS (i) = S−(p) (i + 1, 1)/S−(p) (1, 1),

i = 1, 2, . . . , p.

(6.3.26)

In summary, the algorithm for the low-rank TLS solution is given in Algorithm 6.2. The basic idea of this algorithm was proposed by Cadzow in 1982 [74]. Algorithm 6.2

SVD-TLS algorithm

input: A ∈ Cm×n , b ∈ Cn . 1. Compute the SVD B = [A, b] = UΣVH , and save V. 2. Determine the effective rank p of B. 3. Use (6.3.24) and (6.3.19) to calculate the (p + 1) × (p + 1) matrix S(p) . 4. Compute S−(p) and xTLS (i) = −S−(p) (i, 1)/S−(p) (p + 1, 1), i = 1, . . . , p. output: xTLS .

6.3.3 Performances of TLS Solution The TLS has two interesting interpretations: one is its geometric interpretation [178] and the other is a closed solution interpretation [510]. 1. Geometric Interpretation of TLS Solution Let aTi be the ith row of the matrix A and bi be the ith entry of the vector b. Then the TLS solution xTLS is the minimal vector such that 0 + n 

Ax − b 22 |aTi x − bi |2 , (6.3.27) min = x

x 22 + 1 xT x + 1 i=1

334

Solving Matrix Equations ! " a

where |aTi x−bi |/(xT x+1) is the distance from the point b i ∈ Rn+1 to the nearest i point in the subspace Px , which is defined as "  # a n×1 T :a∈R , b ∈ R, b = x a . (6.3.28) Px = b Hence the TLS solution can be expressed using the subspace Px [178]: sum of the ! " a

squared distances from the TLS solution point b i to points in the subspace Px is i minimized. Figure 6.1 shows, for comparison, the TLS solution and the LS solution. b

b = xa (a2 , b2 ) (a3 , b3 )

(a1 , b1 )

a LS solution TLS solution

Figure 6.1 LS solution and TLS solution.

In Figure 6.1, each dotted line, which is a vertical distance parallel to the b-axis, is an LS solution; and each solid line, which starts at the point (ai , bi ) and is the vertical distance to the straight line b = xa, is a TLS solution. From this geometric interpretation it can be concluded that the TLS solution is better than the LS solution, because the residual error of curve fitting given by the TLS solution is smaller. 2. Closed Solution of TLS Problems If the singular values of the augmented matrix B are σ1 ≥ · · · ≥ σn+1 then the TLS solution can be expressed as [510] 2 I)−1 AH b. xTLS = (AH A − σn+1

(6.3.29)

Compared with the Tikhonov regularization method, the TLS is a kind of antiregularization method and can be interpreted as a least squares procedure with 2 I from the covariance matrix noise removal: it first removes the noise term σn+1 T T 2 A A and then finds the inverse matrix of A A − σn+1 I to get the LS solution. Letting the noisy data matrix be A = A0 + E, its covariance matrix AH A = H H A0 A 0 + E H A 0 + A H 0 E + E E. Obviously, when the error matrix E has zero mean, the mathematical expectation of the covariance matrix is given by H H H E{AH A} = E{AH 0 A0 } + E{E E} = A0 A0 + E{E E}.

6.3 Total Least Squares (TLS) Methods

335

If the column vectors of the error matrix are statistically independent and have the 2 same variance, i.e., E{ET E} = σ 2 I, then the smallest eigenvalue λn+1 = σn+1 of H the (n + 1) × (n + 1) covariance matrix A A is the square of the singular value of 2 the error matrix E. Because the square of the singular value σn+1 happens to reflect 2 the common variance σ of each column vector of the error matrix, the covariance H 2 matrix AH 0 A0 of the error-free data matrix can be retrieved from A A − σn+1 I, T 2 H namely as A A − σn+1 I = A0 A0 . In other words, the TLS method can effectively restrain the influence of the unknown error matrix. It should be pointed out that the main difference between the TLS method and the Tikhonov regularization method for solving the matrix equation Am×n xn = bm is that the TLS solution contains only p = rank([A, b]) principal parameters, and excludes the redundant parameters, whereas the Tikhonov regularization method can only provide all n parameters including the redundant parameters.

6.3.4 Generalized Total Least Squares The ordinary LS, the data LS, Tikhonov regularization and the TLS method can be derived and explained by a unified theoretical framework. Consider the minimization problem 32 2 2 2 4 2[ΔA, Δb]22 + λ2x22 (6.3.30) min F 2 ΔA,Δb,x

subject to (A + αΔA)x = b + βΔb, where α and β are the weighting coefficients of respectively the perturbation ΔA of the data matrix A and the perturbation Δb of the data vector b, and λ is the Tikhonov regularization parameter. The above minimization problem is called the generalized total least squares (GTLS) problem. 1. Comparison of Optimization Problems (1) Ordinary least squares: α = 0, β = 1 and λ = 0, which gives min

ΔA,Δb,x

Δb 22

subject to Ax = b + Δb.

(6.3.31)

(2) Data least squares: α = 1, β = 0 and λ = 0, which gives min

ΔA 2F

(A + ΔA)x = b.

(6.3.32)

(3) Tikhonov regularization: α = 0, β = 1 and λ > 0, which gives   min

Δb 22 + λ x 22 subject to Ax = b + Δb.

(6.3.33)

ΔA,Δb,x

ΔA,Δb,x

subject to

336

Solving Matrix Equations

(4) Total least squares: α = β = 1 and λ = 0, which gives min

ΔA,Δb,x

ΔA, Δb 22

subject to

(A + ΔA)x = b + Δb.

(6.3.34)

2. Comparison of Solution Vectors The constraint condition (A + αΔA)x = (b + βΔb) can be represented as    αx  −1 −1 = 0. [α A, β b] + [ΔA, Δb] −β  αx , the above equation becomes Letting D = [ΔA, Δb] and z = −β   Dz = − α−1 A, β −1 b z. 

(6.3.35)

Under the assumption zH z = 1, we have 2 22 min D 2F = min Dz 22 = min 2[α−1 A, β −1 b]z22 , and thus the solution to the GTLS problem (6.3.30) can be rewritten as +2 0 2 2[α−1 A, β −1 b]z22 2 2 ˆ GTLS = arg min + λ x 2 . x z zH z

(6.3.36)

Noting that zH z = α2 xH x + β 2 and [α−1 A, β −1 b]z = [α−1 A, β −1 b]



 αx = Ax − b, −β

the solution to the GTLS problem (6.3.36) is given by " #

Ax − b 22 2 ˆ GTLS = arg min + λ x x 2 . x α2 x 22 + β 2

(6.3.37)

When the weighting coefficients α, β and the Tikhonov regularization parameter λ take appropriate values, the GTLS solution (6.3.37) gives the following results: ˆ LS = arg min Ax − b 22 x x

Ax − b 22 x

x 22   = arg min Ax − b 22 + λ x 22

ˆ DLS = arg min x ˆ Tik x

x

ˆ TLS = arg min x x

Ax − b 22

x 22 + 1

(α = 0, β = 1, λ = 0),

(6.3.38)

(α = 1, β = 0, λ = 0),

(6.3.39)

(α = 0, β = 1, λ > 0),

(6.3.40)

(α = 1, β = 1, λ = 0).

(6.3.41)

6.3 Total Least Squares (TLS) Methods

337

3. Comparison of Perturbation Methods (1) Ordinary LS method This uses the possible small correction term Δb to perturb the data vector b in such a way that b − Δb ≈ b0 , and thus compensates for the observed noise e in b. The correction vector is selected as Δb = Ax − b, ˆ LS = (AH A)−1 AH b. and the analytical solution is x (2) Data LS method The correction term ΔA = (Ax − b)xH /(xH x) compensates the observed error matrix E in the data matrix A. The data LS solution is ˆ DLS = arg min x x

(Ax − b)H (Ax − b) . xH x

(3) Tikhonov regularization method This adds the same perturbation term λ > 0 to every diagonal entry of the matrix AH A to avoid the numerical instability ˆ Tik = (AH A + of the LS solution (AH A)−1 AH b. The analytical solution is x −1 H λI) A b. (4) TLS method By subtracting the perturbation matrix λI, the noise or perturbation in the covariance matrix of the original data matrix is restrained. There are three kinds of TLS solution: the minimum norm solution, the antiˆ TLS = (AH A − λI)−1 AH b and the regularization solution with n components x SVD -TLS solution with only p = rank([A, b]) principal parameters. 4. Comparison of Application Ranges (1) The LS method is applicable for a data matrix A with full column rank and a data vector b containing iid Gaussian errors. (2) The data LS method is applicable for a data vector b without error and a data matrix A that has full column rank and iid Gaussian error column vectors. (3) The Tikhonov regularization method is applicable for a data matrix A with deficient column rank. (4) The TLS method is applicable for a data matrix A with full column rank, where both A and the data vector b contain iid Gaussian error.

6.3.5 Total Least Squares Fitting In the numerical analysis of science and engineering problems, it is usually necessary to fit a curve or a curved surface to a given set of points. Because these data points have generally been observed, they inevitably contain errors or are contaminated by noise. In such cases the TLS method is expected to provide better fitting results than the ordinary LS method. Consider a data fitting process: given n data points (x1 , y1 ), . . . , (xn , yn ), we want to fit a straight line to these points. Assume the straight line equation is ax+by−c = 0. If the straight line goes through the point (x0 , y0 ) then c = ax0 +by0 .

338

Solving Matrix Equations

Now consider fitting a straight line through the center x ¯, y¯ of n known data points n n 1 1 xi , y¯ = y. (6.3.42) x ¯= n i=1 n i=1 i Substituting c = a¯ x + b¯ y into ax + by − c = 0, the straight line equation can be written as a(x − x ¯) + b(y − y¯) = 0

(6.3.43)

or equivalently in the slope form m(x − x ¯) + (y − y¯) = 0.

(6.3.44)

The parameter vector [a, b]T is called the normal vector of the fitting straight line, and −m = −a/b is its slope. Then, the straight line fitting problem becomes that of finding the normal vector [a, b]T or the slope parameter m. Evidently, if we are substituting n known data points into a straight line equation then it cannot strictly be satisfied, and thus there are fitting errors. The procedure in LS fitting is to minimize the squared sum of the fitting errors; i.e., the cost function of LS fitting is (1)

n 

(2)

i=1 n 

¯, y¯) = DLS (m, x DLS (m, x ¯, y¯) =

((xi − x ¯) + m(yi − y¯))2 ,

(6.3.45)

(m(xi − x ¯) − (yi − y¯))2 .

(6.3.46)

i=1 (i)

Letting ∂DLS (m, x ¯, y¯)/(∂m) = 0, i = 1, 2, we can find the slope m of the straight line. Then, substituting m into Equation (6.3.44), we can get the fitting equation of the straight line. Unlike LS fitting, TLS fitting minimize the sum of the squared distances from the known data points to the linear equation a(x − x0 ) + b(y − y0 ) = 0. The distance d of a point (p, q) from the straight line ax+by −c = 0 is determined by (ap + bq − c)2 (a(p − x0 ) + b(q − y0 ))2 d2 = = . (6.3.47) a2 + b2 a 2 + b2 Then, the sum of the squared distances from the known n data points to the line a(x − x ¯) + b(y − y¯) = 0 is given by D(a, b, x ¯, y¯) =

n  (a(xi − x ¯) + b(yi − y¯))2 . a 2 + b2 i=1

(6.3.48)

LEMMA 6.1 [350] For a data point (x0 , y0 ) on the line a(x − x ¯)Cb(y − y¯) = 0 we have the relationship D(a, b, x ¯, y¯) ≤ D(a, b, x0 , y0 ),

(6.3.49)

6.3 Total Least Squares (TLS) Methods

339

and the equality holds if and only if x0 = x ¯ and y0 = y¯. Lemma 6.1 shows that the TLS best-fit line must pass through the center of n data points in order to minimize the deviation D. Consider how to minimize the deviation D. To this end, we write D as the norm squared of the product of the 2 × 1 unit vector t = (a2 + b2 )−1/2 [a, b]T and the n × 2 matrix M, i.e., 22 2⎡ ⎤ 2 x1 − x ¯ y1 − y¯  2 2 2 a 2 2⎢ .. ⎥ √ 1 (6.3.50) D(a, b, x ¯, y¯) = Mt 22 = 2⎣ ... 2 , ⎦ . 2 a 2 + b2 b 2 2 2 x −x ¯ yn − y¯ n 2 where



¯ x1 − x ⎢ .. M=⎣ . ¯ xn − x

⎤ y1 − y¯ .. ⎥ . . ⎦

(6.3.51)

yn − y¯

From Equation (6.3.50) one can directly obtain the following result. ¯, y¯) reaches a minimum PROPOSITION 6.1 [350] The distance squared sum D(a, b, x for the unit normal vector t = (a2 + b2 )−1/2 [a, b]T . In this case, the mappingt →  1

Mt 2 reaches the minimum value in the unit sphere S = t ∈ R2  t 2 = 1 . Proposition 6.1 shows that the distance squared sum D(a, b, x ¯, y¯) has a minimum. The following theorem provides a way for finding this minimal distance squared sum. THEOREM 6.5 [350] If the 2 × 1 normal vector t is the eigenvector corresponding to the smallest eigenvalue of the 2 × 2 matrix MT M, then the distance squared sum D(a, b, x ¯, y¯) takes the minimum value σ22 . Here we gives a proof of the above theorem that is simpler than the proof in [350]. Using the fact that t is a unit vector, we have t 2 = 1, and thus the distance squared sum D(a, b, x ¯, y¯) can be written as D(a, b, x ¯, y¯) =

tT MT Mt . tT t

(6.3.52)

This is the typical Rayleigh quotient form. Clearly, the condition that D(a, b, x ¯, y¯) takes a minimum value is that the normal vector t is the eigenvector corresponding to the smallest eigenvalue of the matrix MT M. EXAMPLE 6.2 point to get

Given three data points (2, 1), (2, 4), (5, 1), compute their central 1 3

x ¯ = (2 + 2 + 5) = 3,

1 3

y¯ = (1 + 4 + 1) = 2.

340

Solving Matrix Equations

By subtracting the mean values from the data points, we get the zero-mean data matrix ⎤ ⎤ ⎡ ⎡ −1 −1 2−3 1−2 M = ⎣2 − 3 4 − 2⎦ = ⎣ −1 2 ⎦, 5−3

1−2

and hence

 T

M M= whose EVD is given by ⎡ 1 M M=⎣ T



2 1 −√ 2

1 √ 2 1 √ 2



 ⎦ 9 0

−1

2 6 −3 −3 6

⎡  0 ⎣ 3

1 √ 2 1 √ 2



1

−√

2 1 √ 2

⎤ ⎦=



6 −3

−3 6

 .

√ √ Therefore the normal vector t = [a, b]T = [1/ 2, 1/ 2]T . The TLS best-fit equation is as follows: a(x − x ¯) + b(y − y¯) = 0



1 √ (x 2

− 3) +

1 √ (y 2

− 2) = 0,

i.e., y = −x + 5. In this case, the distance squared sum is 2⎡ ⎤ ⎡ 1 ⎤ 22 2 −1 −1 2 √ 2 2 ⎦2 2 2 2 = 3. ⎣ ⎦ ⎣ ¯, y¯) = Mt 2 = 2 −1 DTLS (a, b, x 2 2 1 √ 2 2 2 −1 2 2 In contrast with the TLS fitting, the cost function of the LS fitting takes the form  1 [m(xi − x ¯) + (yi − y¯)]2 2 m + 1 i=1 3

(1)

DLS (m, x ¯, y¯) = =

1 [(−m − 1)2 + (−m + 2)2 + (2m − 1)2 ]. m2 + 1

(1)

From ∂DLS (m, x ¯, y¯)/(∂m) = 6m − 3 = 0 we get m = 1/2, i.e., its slope is −1/2. In 1 this case, the LS best-fit equation is (x − 3) + (y − 2) = 0, i.e., x + 2y − 7 = 0, 2

(1)

¯, y¯) = 3.6. and the corresponding distance squared sum DLS (m, x Similarly, if instead the LS fitting uses the cost function  1 (m(yi − y¯) + (xi − x ¯))2 m2 + 1 i=1 3

(2)

DLS (m, x ¯, y¯) = =

  1 (−m − 1)2 + (2m − 1)2 + (−m + 2)2 , +1

m2

(2)

then the minimal slope such that DLS (m, x ¯, y¯) is minimized is m =

1 , 2

i.e., the

6.3 Total Least Squares (TLS) Methods

341

best-fit equation is 2x − y − 4 = 0, and the corresponding distance squared sum is (2) DLS (m, x ¯, y¯) = 3.6. Figure 6.2 plots the results of the TLS fitting method and the two LS fitting methods. y ◦ (2, 4)

(3, 2) (2, 1) ◦

◦ (5, 1) x

LS fitting

TLS fitting

LS fitting

Figure 6.2 The LS fitting lines and the TLS fitting line. (1)

(2)

For this example, DLS (m, x ¯, y¯) = DLS (m, x ¯, y¯) > DTLS (a, b, x ¯, y¯), i.e., the two LS fittings have the same fitting deviations, which are larger than the fitting deviation of the TLS. It can be seen that the TLS fitting is thus more accurate than the LS fitting. Theorem 6.5 is easily extended to higher-dimensional cases. Let n data vectors xi = [x1i , . . . , xmi ]T , i = 1, . . . , n represent m-dimensional data points, and 1 x = [¯ x1 , . . . , x ¯ m ]T (6.3.53) n i=1 i n be the mean (i.e., central) vector, where x ¯j = i=1 xji . Now use an m-dimensional normal vector r = [r1 , . . . , rm ]T to fit a hyperplane x to the known data vector, i.e., x satisfies the normal equation n

¯= x

¯ , r = 0. x − x From the n × m matrix ⎡ ⎤ ⎡ ¯ x1 − x ¯1 x11 − x ⎢ .. ⎥ ⎢ . .. M=⎣ . ⎦=⎣ ¯ xn − x

¯1 xn1 − x

(6.3.54)

x12 − x ¯2 .. .

··· .. .

⎤ x1m − x ¯m ⎥ .. ⎦, .

xn2 − x ¯2

···

xnm − x ¯m

(6.3.55)

one can fit the m-dimensional hyperplane, as shown in Algorithm 6.3. It should be noted that if the smallest eigenvalues of the matrix MT M (or the smallest singular values of M) have multiplicity, then correspondingly there are multiple eigenvectors, leading to the result that the hyperplane fitting problem has multiple solutions. The occurrence of this kind of situation shows that a linear data-fitting model may not be appropriate, and thus one should try other, nonlinear, fitting models.

342 Algorithm 6.3

Solving Matrix Equations TLS algorithm for fitting m-dimensional syperplane [350]

input: n data vectors x1 , . . . , xn . ¯= 1 1. Compute the mean vector x n

n

xi .

i=1

2. Use Equation (6.3.55) to form the n × m matrix M. 3. Compute the minimal eigenpair (λ, u) of MT M. Let r = u. ¯ , r = 0. output: Hyperplane x − x

Total least squares methods have been widely used in the following areas: signal processing [241], biomedical signal processing [482], image processing [348], frequency domain system identification [393], [429], variable-error modeling [481], [457], subspace identification of linear systems [485], radar systems [150], astronomy [58], communications [365], fault detection [223] and so on.

6.3.6 Total Maximum Likelihood Method In the previous sections, we presented the LS method, the DLS method, Tikhonov regularization and the TLS method for solving the matrix equation Ax = b + w, where the data matrix A is assumed to be deterministic and known. But, in some important applications, A is random and unobservable. Consider the matrix equation Ax = b + w,

(6.3.56)

where A ∈ Rm×n is the data matrix, x ∈ Rn is an unknown deterministic vector and w ∈ Rm is an unknown perturbation or noise vector subject to the Gaussian 2 distribution N (0, σw ). The data matrix A = [aij ] is a random matrix with random Gaussian variables whose distribution is assumed to be 2 aij , σA ), aij = N (¯

¯ [¯ aij ] = A.

(6.3.57)

Recently, under the assumption that the measurement matrix consists of random Gaussian variables, an alternative approach to solving the matrix equation Ax = b + w has been proposed, [508], [509]. This technique is referred to as the total maximum likelihood (TML) method in [30]. The TML solution of the random matrix equation Ax = b + w is determined by [508] ˆ TML = arg min log p(b; x), x x

(6.3.58)

6.3 Total Least Squares (TLS) Methods

343



 ¯ σ 2 ( x 2 + σ 2 )I . Hence, the solution can be rewritten as where b ∼ N Ax, w A " # ¯ 2  2 

b − Ax 2 2 2 ˆ TML = arg min x . (6.3.59) 2 x 2 + σ 2 + m log σA x 2 + σw x σA w 2 THEOREM 6.6

For any t ≥ 0, let f (t) =

min Ax − b 22 ,

x: x 22 =t

(6.3.60)

and denote the optimal argument by x(t). Then, the maximum likelihood estimator of x in the random matrix equation (6.3.56) is x(t∗ ), where t∗ is the solution to the following unimodal optimization problem: " # f (t) 2 2 + m log(σ t + σ ) . (6.3.61) min A w 2 t + σ2 t≥0 σA w Proof

See [508].

Theorem 6.6 allows for an efficient solution of the ML problem. This is so because [508]: (1) there are standard methods for evaluating f (t) in (6.3.60) for any t ≥ 0; (2) the line search in (6.3.61) is unimodal in t ≥ 0, and thus any simple onedimensional search algorithm can efficiently find its global minima. For the solution of the optimization problem (6.3.60), one has the following lemma: LEMMA 6.2

[159], [508] The solution of optimization problem in (6.3.60), min Ax − b 22 ,

(6.3.62)

x: x 22 =t

has the solution

 T  ¯ + α∗ I † Ab, ¯ ¯ A x(t) = A

¯ is the unique root of the equation ¯ T A) where α∗ ≥ −λmin (A 22 2 † 2 ¯T ¯ ¯ 2

x(t) 2 = 2 A A + α∗ I Ab 2 = t. 2

(6.3.63)

(6.3.64)

2

¯ one can calculate (A ¯TA ¯ + ¯TA, Remark Using the eigenvalue decomposition of A ∗ †¯ 2 α I) Ab 2 for different values of α. The monotonicity of this squared norm enables us to find the α that satisfies (6.3.64) using a simple line search. The above discussion is summarized as Algorithm 6.4. The TML method has been extended to the case in which the measurement matrix is structured, so that the perturbations are not arbitrary but rather follow a fixed pattern. For this case, a structured TML (STML) strategy was proposed in [30].

344

Solving Matrix Equations

Algorithm 6.4

Total maximum likelihood algorithm [508]

¯ t0 , Δt. input: b, A, initialization: k = 1. repeat 1. Compute tk = t0 + Δt. 2. Use one-dimensional line search to solve the equation # #2 † # ¯T ¯ ¯ # # A A + αI Ab # = tk 2

in order to get the optimal solution α∗ . ¯ + α∗ I)† Ab. ¯ ¯TA 3. xk = (A 2 ¯ 4. f (tk ) = b − Ax(t k )2 .

5. J(tk ) =

f (tk ) 2 t + σ2 σA w k

2 2 + m log(σA tk + σ w ).

6. exit if J(tk ) > J(tk−1 ). return k ← k + 1. output: xTML ← xk−1 .

It is interesting and enlightening to compare the TML with the GTLS from the viewpoint of optimization: (1) The optimization solutions of the GTLS problem and of the TML problem are respectively given by " #

b − Ax 22 2 ˆ GTLS = arg min (6.3.65) + λ x GTLS : x 2 , x α2 x 22 + β 2 " # ¯ 2  2 

b − Ax 2 2 2 ˆ TML = arg min + m log σ

x + σ . (6.3.66) TML : x A 2 w 2 x 2 + σ 2 x σA w 2 They consist of a cost function and a penalty function. Taking α = σA and β = σw and replacing the deterministic data matrix A in the GTLS by the ¯ of the random data matrix A, the cost function of the deterministic term A GTLS problem becomes the cost function of the TML problem. (2) The penalty function in the GTLS problem is the regularization term λ x 22 , whereas the penalty function in the TML problem is the barrier function term 2 2 m log(σA

x 22 + σw ).

6.4 Constrained Total Least Squares The data LS method and the TLS method for solving the matrix equation Ax = b consider the case where the data matrix includes observed errors or noise, but the two methods assume that the errors are iid random varibles and have the same variances. However, in some important applications, the noise coefficients of the data

6.4 Constrained Total Least Squares

345

matrix A may be statistically correlated or, although statistically independent, they have different variances. In this section we discuss the solution of over-determined matrix equations when the column vectors of the noise matrix are statistically correlated.

6.4.1 Constrained Total Least Squares Method The matrix equation Am×n xn = bm can be rewritten as     x x = 0, = 0 or C [A, b] −1 −1

(6.4.1)

where C = [A, b] ∈ Cm×(n+1) is the augmented data matrix. Consider the noise matrix D = [E, e] of [A, b]. If the column vectors of the noise matrix are statistically correlated then the column vectors of the correction matrix should also be statistically correlated in order to use an augmented correction matrix ΔC = [ΔA, Δb] to suppress the effects of the noise matrix D = [E, e]. A simple way to make the column vectors of the correction matrix ΔC statistically correlated is to let every column vector be linearly dependent on the same vector (e.g., u): ΔC = [G1 u, . . . , Gn+1 u] ∈ Rm×(n+1) ,

(6.4.2)

where Gi ∈ Rm×m , i = 1, . . . , n+1, are known matrices, while u is to be determined. The constrained TLS problem can be stated as follows [1]: determine a solution vector x and a minimum-norm perturbation vector u such that     x = 0, (6.4.3) C + [G1 u, . . . , Gn+1 u] −1 which can be equivalently expressed as the constrained optimization problem     x T = 0, (6.4.4) min u Wu subject to C + [G1 u, . . . , Gn+1 u] u,x −1 where W is a weighting matrix and is usually diagonal or the identity matrix. The correction matrix ΔA is constrained as ΔA = [G1 u, . . . , Gn u], while the correction vector Δb is constrained as Δb = Gn+1 u. In constrained TLS problems, the linear correlation structure between column vectors of the augmented correction matrix [ΔA, Δb] is kept by selecting appropriate matrices Gi (i = 1, . . . , n + 1). The key to the constrained TLS method is how to choose appropriate matrices G1 , . . . , Gn+1 , depending on the application. THEOREM 6.7 [1]

Let Wx =

n  i=1

xi Gi − Gn+1 .

(6.4.5)

346

Solving Matrix Equations

Then the constrained TLS solution is given by +  H  0 x x H H † C (Wx Wx ) C , min F (x) = x −1 −1

(6.4.6)

where Wx† is the Moore–Penrose inverse matrix of Wx . A complex form of the Newton method was proposed in [1] for calculating the constrained TLS solution. The matrix F (x) is regarded as a complex analytic function with 2n complex variables x1 , . . . , xn , x∗1 , . . . , x∗n . Then the Newton recursive formulas are given by x = x0 + (A∗ B−1 A − B∗ )−1 (a∗ − A∗ B−1 a), where

⎫ T  ⎪ ∂F ∂F ∂F ∂F ⎪ a= , ,..., = complex gradient of F,⎪ = ⎪ ⎪ ∂x ∂x1 ∂x2 ∂xn ⎪ ⎪ ⎬ 2 ∂ F A= = nonconjugate complex Hessian matrix of F, ⎪ ⎪ ∂x∂xT ⎪ ⎪ ⎪ 2 ⎪ ∂ F ⎪ ⎭ B= = conjugate complex Hessian matrix of F. ∗ T ∂x ∂x

(6.4.7)

(6.4.8)

The (k, l)th entries of the two n × n part Hessian matrices are defined as  2     ∂ F ∂F ∂F ∂2F 1 ∂F ∂F , (6.4.9) = = −j −j ∂x∂xT k,l ∂xk ∂xl 4 ∂xkR ∂xkI ∂xlR ∂xlI      ∂F ∂F ∂2F ∂2F 1 ∂F ∂F . (6.4.10) = = + j − j ∂x∗ ∂xT k,l ∂x∗k ∂xl 4 ∂xkR ∂xkI ∂xlR ∂xlI Here xkR and xkI are respectively the real part and the imaginary part of xk . Put   x H −1 , (6.4.11) u = (Wx Wx ) C −1   ˜ = CIn+1,n − G WH u, . . . , G WH u , B (6.4.12) 1 n x x  H  H ˜ G = G1 u, . . . , Gn u , (6.4.13) where In+1,n is a (n + 1) × n diagonal matrix with diagonal entries equal to 1. Hence, a, A and B can be calculated as follows:   ˜ T, a = uH B (6.4.14)  H H T  −1 H H H H −1 ˜− G ˜ W (W W ) B ˜ , ˜ W W W B A = −G (6.4.15) x x x x x x   H   ˜ T +G ˜ (W WH )−1 B ˜ H WH (W WH )−1 W − I G. ˜ B= B (6.4.16) x x x x x x It is shown in [1] that the constrained TLS estimate is equivalent to the constrained maximum likelihood estimate.

6.4 Constrained Total Least Squares

347

6.4.2 Harmonic Superresolution Abatzoglou et al. [1] presented applications of the constrained TLS method in harmonic superresolution. Assume that L narrowband wavefront signals (harmonics) irradiate a uniform linear array of N elements. The array output signals satisfy the forward linear prediction equation [271]   x = 0, k = 1, 2, . . . , M, (6.4.17) Ck −1 where M is the number of snapshots and ⎡ yk (1) yk (2) ··· ⎢ y (2) y (3) ··· k k ⎢ ⎢ . . .. ⎢ .. .. . ⎢ ⎢ ⎢ yk (N − L) yk (N − L + 1) · · · Ck = ⎢ ⎢ ---------------------⎢ ∗ ⎢yk (L + 1) yk∗ (L) ··· ⎢ ⎢ .. .. .. ⎣ . . . ∗ ∗ yk (N − 1) ··· yk (N )

yk (L + 1) yk (L + 2) .. .



⎥ ⎥ ⎥ ⎥ ⎥ ⎥ yk (N ) ⎥ ⎥. ------- ⎥ ⎥ yk∗ (1) ⎥ ⎥ ⎥ .. ⎦ . ∗ yk (N − L)

(6.4.18)

Here yk (i) is the output observation of the kth array element at the time i. The matrix Ck is referred to as the data matrix at the kth snapshot. Combining all the data matrices into one data matrix C, we get ⎤ ⎡ C1 ⎥ ⎢ C = ⎣ ... ⎦ . (6.4.19) CM Then, the harmonic superresolution problem can be summarized as follows: use the constrained TLS method to solve the matrix equation   x = 0, C −1 while the estimation result of Wx is given by   0 ˆ = W1 W , x 0 W2 where

⎡ ⎢ W1 = ⎣

x1 0

x2 .. .

··· .. . x1

xL x2

−1 .. .

..

···

xL

0 .

⎤ ⎥ ⎦

−1

348

Solving Matrix Equations

and

⎡ ⎢ W2 = ⎣

−1

··· .. . −1

xL .. .

0

x2 xL

x1 .. .

0 ..

···

x2

.

⎤ ⎥ ⎦.

x1

Moreover,   H x x CH (Wx WxH )−1 C −1 −1    H  M x x ˆ H )−1 C ˆ W . CH ( W = m x x m −1 −1 

F (x) =

(6.4.20)

m=1

In order to estimate the angles of arrival φi of the L space signals at the linear uniform array, the constrained TLS method consists of the following three steps [1]. (1) Use the Newton method to minimize F (x) in (6.4.20), yielding x. (2) Find the roots zi , i = 1, . . . , L, of the linear-prediction-coefficient polynomial L 

xk z k−1 − z L = 0.

(6.4.21)

k=1

(3) Estimate the angles of arrival φi = arg(zi ), i = 1, . . . , L.

6.4.3 Image Restoration It is important to be able to recover lost information from degraded image data. The aim of image restoration is to find an optimal solution for the original image in the case where there is known record data and some prior knowledge. Let the N × 1 point-spread function (PSF) be expressed by ¯ + Δh, h=h

(6.4.22)

¯ and Δh ∈ RN are respectively the known part and (unknown) error part of where h the PSF. The error components Δh(i), i = 0, 1, . . . , N − 1, of Δh = [Δh(0), Δh(1), . . . , Δh(N −1)]T are independent identically distributed (iid) noises with zero mean and the same variance σh . The observed degraded image is expressed by g, and the imaging equation can be expressed as g = Hf + Δg,

(6.4.23)

where f and Δg ∈ RN are respectively the original image and the additive noise of the observed image. The additive noise vector Δg = [Δg(0), Δg(1), . . . , Δg(N −1)]T is an iid random vector and is statistically independent of the PSF error component

6.4 Constrained Total Least Squares

349

¯ Δh. The matrix H ∈ RN ×N denotes the PSF, which consists of the known part H and the noise part: ¯ + ΔH. H=H (6.4.24) The TLS solution of Equation (6.4.23) is 2 22 2 2 ˆ g ˆ ]2 , f = arg min 2[H, g] − [H, ˆ g]∈ RN ×(N +1) [H,ˆ

(6.4.25)

F

ˆ is where the constraint condition of g ˆ ˆ ∈ Range(H). g

(6.4.26)

By defining an unknown regularized noise vector T  Δh(0) Δh(N − 1) Δg(0) Δg(N − 1) u= ,..., , ,..., , σh σh σg σg

(6.4.27)

Mesarovic et al. [323] proposed the constrained TLS-based image restoration algorithm ¯ − g + Lu = 0, f = arg min u 2 subject to Hf (6.4.28) f

2

where L is an N × 2N matrix defined as ⎡ σh f (N − 1) · · · σh f (1) ⎢ σh f (0) ⎢ ⎢ σh f (1) σh f (0) · · · σh f (2) L=⎢ ⎢ . . .. .. .. .. ⎢ . . ⎣ σh f (N − 1)

σh f (N − 2)

···

σh f (0)

.. . .. . .. . .. .

σg

0

···

0 .. .

σg .. .

··· .. .

0

0

···

⎤ 0⎥ ⎥ 0⎥ ⎥ . (6.4.29) .. ⎥ .⎥ ⎦ σg

¯ In the case where there is a given data vector g and a part of the PSF matrix H is known, solving for the original image f in Equation (6.4.23) is a typical inverse problem: the solution of the image restoration problem corresponds mathematically to the existence and uniqueness of the inverse transformation of Equation (6.4.23). If the inverse transformation does not exist then the image restoration is called a singular inverse problem. Moreover, even if the inverse transformation exists, its solution may be not unique. For a practical physical problem, such a nonunique solution is not acceptable. In this case, the image restoration is said to be an illconditioned inverse problem. This implies that a very small perturbation in the observed data vector g may lead to a large perturbation in the image restoration [16], [465]. An effective way to overcome the ill-conditioned problem in image restoration is to use the regularization method [465], [123], which yields the regularized constrained TLS algorithm [123], [168]. The basic idea of the regularized constrained TLS image restoration algorithm is to introduce a regularizing operator Q and a regularizing parameter λ > 0, and

350

Solving Matrix Equations

to replace the objective function by the sum of two complementary functions; i.e., (6.4.28) becomes   f = arg min u 22 + λ Qf 22 (6.4.30) f

subject to ¯ − g + Lu = 0. Hf The choice of regularizing parameter λ needs to take into account both the fidelity to the observed data and the smoothness of the solution. In order to improve further the performance of the regularized constrained TLS image restoration algorithm, Chen et al. [97] proposed a method for adaptively choosing the regularizing parameter λ, this method is called the adaptively regularized constrained TLS image restoration. The solution of this algorithm is given by   f = arg min u 22 + λ(f ) Qf 22 (6.4.31) f

subject to ¯ − g + Lu = 0. Hf

6.5 Subspace Method for Solving Blind Matrix Equations Consider a blind matrix equation X = AS,

(6.5.1)

where X ∈ CN ×M is a complex matrix whose entries are the observed data and the two complex matrices A ∈ CN ×d and S ∈ Cd×M are unknown. The question is: when only X is known, the solution of the unknown matrix S of blind matrix equation can be found? The answer is yes, but it is necessary to assume two conditions, that the matrix A is of full column rank and the matrix S is of full row rank. These assumptions are often satisfied in engineering problems. For example, in array signal processing, full column rank of the matrix A means that the directions of arrival (DOA) of all source signals are mutually independent, while full row rank of the matrix S requires that each source signal is independently transmitted. Assume that N is the data length, d is the number of sources, M is the number of sensors, usually M ≥ d and N > M . Define the truncated SVD of the data matrix X as ˆΣ ˆV ˆ H, X=U

(6.5.2)

ˆ is a d × d diagonal matrix consisting of the d principal singular values where Σ of X.

6.5 Subspace Method for Solving Blind Matrix Equations

351

ˆ i.e., the matrices A and U ˆ span the same signal subSince Col(A) = Col(U), space, we have ˆ = AT, U (6.5.3) where T is a d × d nonsingular matrix. Let W be a d × N complex matrix that represents a neural network or filter. Premultiply Equation (6.5.1) by W: WX = WAS.

(6.5.4)

Adjusting the matrix W so that WA = Id , the solution of the above equation is given by S = WX.

(6.5.5)

In order to find W, we compute ˆ = WAT = T WU and thus obtain ˆ H. W = TU

(6.5.6)

Summarizing the above discussion, we have the following method for solving the blind matrix (6.5.1): ⎫ Data model X = AS, ⎪ ⎪ ⎪ ⎪ ⎪ ˆ ˆ ˆ ⎬ Truncated SVD X = UΣV, (6.5.7) ˆ = AT for T, ⎪ Solving U ⎪ ) * ⎪ ⎪ ⎭ ˆ T X.⎪ Solution of matrix equation S = TU This method is called the subspace method because it is based on the signal subspace. Hence, the key problem in solving a blind matrix equation is how to find the ˆ = AT in the case where both A and T are unknown. nonsingular matrix T from U Here we consider an example of solving the blind matrix equation (6.5.1) in wireless communications. Without taking into consideration the multipath transmission in wireless communication, the matrix in Equation (6.5.1) is given by [493] X = Aθ B, with

⎡ ⎢ ⎢ Aθ = [a(θ1 ), . . . , a(θd )] = ⎢ ⎣ B = Diag(β1 , β2 , . . . , βd ),

(6.5.8)

1 θ1 .. .

1 θ2 .. .

··· ··· .. .

1 θd .. .

θ1N −1

θ2N −1

···

θdN −1

⎤ ⎥ ⎥ ⎥, ⎦

352

Solving Matrix Equations

in which θi and βi are the unknown direction of arrival and the unknown attenuation coefficient of the ith user, respectively. Define the diagonal matrix Θ = Diag(θ1 , . . . , θd )

(6.5.9)

and the (M − 1) × M selection matrices J1 = [IM −1 , 0],

J2 = [0, IM −1 ]

which select respectively the upper M − 1 rows and the lower M − 1 rows of the matrix Aθ . It is easy to see that (J1 Aθ )Θ = J2 Aθ .

(6.5.10)

ˆ = XT = A BT. U θ

(6.5.11)

Hence

In order to find the nonsingular matrix T, if we premultiply (6.5.11) by the selection matrices J1 and J2 , respectively, and let A θ = J1 Aθ then, using (6.5.10), we get ˆ = (J1 Aθ )BT = A BT, ˆ 1 = J1 U U θ ˆ = (J2 Aθ )BT = A ΘBT. ˆ 2 = J2 U U θ

(6.5.12) (6.5.13)

Since B and Θ are diagonal matrices, we have ΘB = BΘ and thus ˆ = A BΘT = A BTT−1 ΘT = U ˆ T−1 ΘT, U θ θ 2 1 which can be written as −1 ˆ ˆ †U U ΘT, 1 2 =T

(6.5.14)

ˆ )−1 U ˆ H is the generalized inverse of the matrix U ˆ † = (U ˆ HU ˆ . where U 1 1 1 1 1 Because Θ is a diagonal matrix, it is easily seen that Equation (6.5.14) is a typical similarity transformation. Hence, through this similarity transformation of ˆ ˆ †U the matrix U 1 2 , we can obtain the nonsingular matrix T. The above discussion is summarized in Algorithm 6.5 for solving the blind matrix equation X = Aθ B. Although we describe only the case of single-path transmission, the subspace method for solving the matrix equation AS = X is also applicable to the case of multipath transmission. The difference is in the form of the matrix A, so that the method of finding the nonsingular matrix T is also different. The interested reader may refer to [493].

6.6 Nonnegative Matrix Factorization: Optimization Theory Algorithm 6.5

353

Solving blind matrix equation X = Aθ B

input: X ∈ CN ×M . ˆΣ ˆV ˆ H. 1. Compute the truncated SVD X = U ˆ 1 = UJ ˆ 1 and U ˆ 2 = UJ ˆ 2. 2. Calculate U ˆ †U ˆ 3. Make the similarity transformation T−1 U 1 2 T to get T.  H  ˆ T X. output: B = U

6.6 Nonnegative Matrix Factorization: Optimization Theory A matrix with nonnegative real entries is called a nonnegative matrix. Consider the nonnegative blind matrix equation X = AS, where a known matrix X and two unknown matrices A and S are nonnegative matrices. This blind matrix equation is widespread in engineering application problems.

6.6.1 Nonnegative Matrices An n × n matrix A or an n × 1 vector a is said to be

DEFINITION 6.3

(1) positive (or elementwise positive), denoted A > O or a > 0, if all its entries are positive; (2) nonnegative (or elementwise nonnegative), denoted A ≥ O or a ≥ 0, if its all entries are nonnegative. We use the notation A > B (A ≥ B) to mean that A − B is a positive (nonnegative) matrix, i.e., Aij > Bij (Aij ≥ Bij ) for all i, j. DEFINITION 6.4 We say that a matrix A ∈ Rn×n is reducible if there exists a permutation matrix P such that   A11 A12 , (6.6.1) B = PAPT = O A21 where A11 ∈ Rr×r , A22 ∈ Rn−r,n−r , A12 ∈ Rr×(n−r) and O is an (n − r) × r zero matrix. DEFINITION 6.5 ducible.

A matrix A ∈ Rn×n is said to be irreducible, if it is not re-

DEFINITION 6.6 An n×n nonnegative matrix A is said to be regular or primitive, if there is a k ≥ 1 such that all the entries of Ak are positive. EXAMPLE 6.3

Any positive matrix A is regular since, for k = 1, A > O.

354

EXAMPLE 6.4

Solving Matrix Equations

The following matrices are not regular:       1 1 1 0 0 1 A= , A= and A = 0 1 1 1 1 0

because, for any k ≥ 1, Ak ≥ O. But, the matrix   1 1 A= 1 0 is regular, since



1 A= 0 EXAMPLE 6.5

1 1

2



 2 1 = > O. 1 1

The following matrix is ⎡ 1 A = ⎣1 0

regular: ⎤ 0 1 0 0⎦ 1 0

because, for k = 4, we have ⎡

2 A 4 = ⎣2 1

⎤ 1 1 1 1⎦ > O. 1 1

There is the following relationship between an irreducible matrix A and a regular matrix. LEMMA 6.3 If A is a nonnegative and irreducible n×n matrix then (I+A)n−1 > O, i.e., I + A is regular or primitive. Proof [355] It suffices to prove that (I + A)n−1 x > O for any x ≥ 0, x = 0. Define the sequence xk+1 = (I + A)xk ≥ 0, k = 0, 1, . . . , n − 2, x0 = x. Since xk+1 = xk + Axk , xk+1 has no more zero entries than xk . In order to prove by contradiction that xk+1 has fewer zero entries than xk , suppose that xk+1 and xk have exactly the same number of zero entries. Then, there exists a permutation matrix P such that     y z Pxk+1 = , Pxk = , y, z ∈ Rm , y, z > 0, 1 ≤ m < n. 0 0 Then

  y = P(xk + Axk ) = Pxk + PAPT Pxk 0      A11 A12 z z . + = 0 A21 A22 0

Pxk+1 =

This implies that A21 = O, which contradicts the assumption condition that A is

6.6 Nonnegative Matrix Factorization: Optimization Theory

355

irreducible. Thus, x0 = x has at most n − 1 zero entries, xk has at most n − k − 1 zero entries, and hence xn−1 = (I + A)n−1 x0 has no zero entry, i.e., xn−1 is a positive vector, which completes the proof. THEOREM 6.8 (Perron–Frobenius theorem for regular matrices) Suppose that A ∈ Rn×n is a nonnegative and regular matrix, i.e., Ak > 0 for some k ≥ 1; then • there is a Perron–Frobenius (PF) eigenvalue λPF of A that is real and positive, together with positive left and right eigenvectors; • any other eigenvalue λ satisfies |λ| < λPF ; • the eigenvalue λPF is simple, i.e., has multiplicity 1, and corresponds to a 1 × 1 Jordan block. The Perron–Frobenius theorem was proved by Oskar Perron in 1907 [385] for positive matrices and was extended by Georg Frobenius in 1912 [165] to nonnegative and irreducible matrices. The Perron–Frobenius theorem has important applications to probability theory (Markov chains), the theory of dynamical systems, economics, population dynamics (the Leslie population-age-distribution model), power control and so on. Interested readers can refer to the books [39], [324]. In particular, the Perron–Frobenius theorem has been extended to tensors [39].

6.6.2 Nonnegativity and Sparsity Constraints In many engineering applications it is necessary to impose two constraints on the data: a nonnegativity constraint and a sparsity constraint. 1. Nonnegativity Constraint As the name suggests, the nonnegativity constraint constrains the data to be nonnegative. A lot of actual data are nonnegative, and so constitute the nonnegative data matrices. Such nonnegative matrices exist widely in daily life. The following are four important actual examples of nonnegative matrices [277]. 1. In document collections, documents are stored as vectors. Each element of a document vector is a count (possibly weighted) of the number of times a corresponding term appears in that document. Stacking document vectors one after the other creates a nonnegative term-by-document matrix that represents the entire document collection numerically. 2. In image collections, each image is represented by a vector, and each element of the vector corresponds to a pixel. The intensity and color of a pixel is given by a nonnegative number, thereby creating a nonnegative pixel-by-image matrix. 3. For item sets or recommendation systems, the information for a purchase history of customers or ratings on a subset of items is stored in a nonnegative sparse matrix.

356

Solving Matrix Equations

4. In gene expression analysis, gene-by-experiment matrices are formed from observations of the gene sequences produced under various experimental conditions. In addition, in pattern recognition and signal processing, for some particular pattern or target signal a linear combination of all the feature vectors (an “allcombination”) may not be appropriate. On the contrary, a part-combination of some feature vectors is more suitable. For example, in face recognition, the combination of the specific parts of the eye, nose, mouth and so on is often more effective. In an all-combination, the positive and negative combination coefficients emphasize respectively the positive and negative effects of some features, while a zero combination coefficient implies a characteristics that does not play a role. In a part-combination, there are only two kinds of characteristics: those play or do not play a role. Therefore, in order to emphasize the role of some main features, it is natural to add nonnegative constraints to the elements in the coefficient vector. 2. Sparsity Constraint By a sparsity constraint is meant an assumption that the data are not dense but sparse, i.e., most data take a zero value, only a few take nonzero values. A matrix for which most entries are zero and only a few entries are nonzero is called a sparse matrix, while a sparse matrix whose entries take only nonnegative nonzero values is known as a nonnegative sparse matrix. For example, in the commodity recommendation system, a matrix that is composed of the customer’s purchases or scores is a nonnegative sparse matrix. In economics, a lot of variables and data (such as volume and price) are not only sparse but also nonnegative. Sparsity constraints can increase the effectiveness of investment portfolios, while nonnegative constraints can improve investment efficiency and also reduce the investment risks [436], [538]. Although many natural signals and images are not themselves sparse, after a certain transformation they are sparse in the transform domain. For example, the discrete cosine transform (DCT) of a face and a medical image is a set of typical sparse data. The short-time Fourier transform (STFT) of a speech signal is also sparse in the time domain.

6.6.3 Nonnegative Matrix Factorization Model The basic problem of linear data analysis is as follows. By an appropriate transformation or factorization, a higher-dimensional original data vector can be represented as a linear combination of a set of low-dimensional vectors. Because the nature or character of the original data vector is extracted, it can be used for pattern recognition. So these low-dimensional vectors are often called “pattern vectors” or “basis vectors” or “feature vectors”.

6.6 Nonnegative Matrix Factorization: Optimization Theory

357

In the process of data analysis, modeling and processing, the two basic requirements of a pattern vector must be considered. (1) Interpretability The components of a pattern vector should have definite physical or physiological meaning. (2) Statistical fidelity When the data are consistent and do not have too much error or noise, the components of a pattern vector should be able to explain the variance of the data (in the main energy distribution). Vector quantization (VQ) and principal component analysis (PCA) are two widely used unsupervised learning algorithms; they adopt fundamentally different ways to encoding of data. 1. Vector Quantization Method The vector quantization (VQ) method uses a stored prototype vector as the code vector. Let cn be a k-dimensional code vector and let there be N such stored code vectors, i.e., cn = [cn,1 , . . . , cn,k ]T ,

n = 1, . . . , N.

The collection of N code vectors {c1 , . . . , cN } comprises a codebook. The subset consisting of the data vectors closest to the stored pattern vectors or the code vectors cn is called the encoding region of the code vectors cn , denoted Sn and defined as    Sn = x  x − cn 2 ≤ x − cn 2 , ∀ n = 1, . . . , N . (6.6.2) The formulation of the vector quantization problem is as follows: given M k ×1 data vectors xi = [xi,1 , . . . , xi,k ]T , i = 1, . . . , M , determine the encoding region of these vectors, i.e., their corresponding code vectors. Let X = [x1 , . . . , xM ] ∈ Rk×M be a data matrix and C = [c1 , . . . , cN ] ∈ Rk×N denote a codebook matrix. Then the VQ of the data matrix can be described by the model X = CS,

(6.6.3)

where S = [s1 , . . . , sM ] ∈ RN ×M is a quantization coefficient matrix, its columns are called quantization coefficient vectors. From the viewpoint of optimization, the VQ optimization criterion is “winnertake-all”. According to this criterion, the input data is clustered into mutually exclusive patterns [172]. From the viewpoint of encoding, the VQ is a “grandmother cell coding”: all data are explained or clustered simply by a basis vector [506]. Specifically, each quantization coefficient vector is an N × 1 basis vector with only a single entry 1 and the rest zero entries. Therefore, if the (i, j)th entry of the codebook matrix is equal to 1 then the data vector xj is judged to be closest to the code vector ci , that is, the data vector corresponds to only one code vector. The

358

Solving Matrix Equations

VQ method can capture the nonlinear structure of the input data but its capturing ability is weak, because the data vector and the code vector are in one-to-one correspondence in this method. If the dimension of the data matrix is large, a large number of code vectors are needed to represent the input data. 2. Principal Component Analysis Method The linear data model is a widely used data model. Principal component analysis (PCA), linear discriminant analysis (LDA), independent component analysis (ICA) and other multivariate data analysis methods adopt this linear data model. As we saw, the VQ method uses code vectors. Similarly, the base matrix A in the PCA method consists of a set of orthogonal basis vectors ai corresponding to the principal components. These basis vectors are called the pattern or feature vectors. For a data vector x, PCA uses the principle of shared constraints to make the optimization and uses a linear combination of the pattern vectors x = As to represent the input data. From the encoding point of view, PCA provides a distributed encoding, as compared with the grandmother-cell-encoding-based VQ method, thus the PCA method requires only a small group of basic vectors to represent higher-dimensional data. The disadvantages of the PCA method are as follows. 1. It cannot capture any nonlinear structure in the input data. 2. The basis vectors can be statistically interpreted as the directions of maximum difference, but many directions do not have a clear visual interpretation. The reason is that the entries of the basis matrix A and the quantization coefficient vector s can take zero, positive or negative signs. Since the basis vectors are used in the linear combination, and this combination involves a complex cancellation of positive and negative numbers, many individual basis vectors lose their intuitive physical meaning due to this cancellation and do not have an explanatory role for nonnegative data (such as the pixel values of a color image). On the one hand, the entries of a nonnegative pattern vector should all be nonnegative values, but on the other hand mutually orthogonal eigenvectors must contain negative entries: if all the entries of the eigenvector u1 corresponding to the maximum eigenvalue are nonnegative, then any other eigenvector orthogonal to u1 must contain at least one negative entry, otherwise the orthogonality condition of two vectors u1 , uj  = 0, j = 1, cannot be met. This fact indicates that the mutually orthogonal eigenvectors cannot be used as pattern vectors or basis vectors in nonnegative data analysis. In the PCA, LDA and ICA methods, the coefficient vector elements usually take positive or negative values; few take a zero value. That is to say, in these methods, all the basis vectors are involved in the fitting or regression of the data vectors.

6.6 Nonnegative Matrix Factorization: Optimization Theory

359

3. Nonnegative Matrix Factorization Method Unlike in the VQ, PCA, LDA and ICA methods, the basis vectors and the elements of the coefficient vector are treated as nonnegative constraints in nonnegative matrix factorization (NMF). It can be seen that the number of basis vectors involved in the fitting or regression of a data vector is certainly less in the NMF. From this perspective, NMF basis vectors have the role of extracting the principal basis vectors. Another prominent advantage of NMF is that the nonnegative constraint on combination factors facilitates the generation of sparse coding, so that many encoded values are zero. In biology, the human brain encodes information in this sparse coding way [153]. Therefore, as an alternative method of linear data analysis, we should use nonnegative instead of orthogonal constraints on the basis vectors. The NMF method gives a multilinear nonnegative approximation to the data. K×1 Let x(j) = [x1 (j), . . . , xI (j)]T ∈ RI×1 and s(j) = [s1 (j), . . . , sK (j)]T ∈ R+ + denote respectively the nonnegative data vector and K-dimensional nonnegative coefficient vector measured by I sensors at the discrete time j, where R+ denotes the nonnegative quadrant. The mathematical model for nonnegative vectors is described by ⎡ ⎤ ⎡ ⎤ ⎤⎡ x1 (j) s1 (j) a11 · · · a1K ⎢ .. ⎥ ⎢ .. .. ⎥ ⎢ .. ⎥ or x(j) = As(j), .. (6.6.4) ⎣ . ⎦=⎣ . . . ⎦⎣ . ⎦ xI (j)

aI1

···

aIK

sK (j)

in which A = [a1 , . . . , aK ] ∈ RI×K is the basis matrix and ak , k = 1, . . . , K, are the basis vectors. Since the measurement vectors x(j) at different times are expressed in the same set of basis vectors, so these I-dimensional basis vectors can be imagined as the building blocks of data representation, while the element sk (j) of the K-dimensional coefficient vector represents the strength of the kth basis vector (building block) ak in the data vector x(j), reflecting the contribution of ak in the fitting or regression of x. Therefore, the elements sk (j) of the coefficient vectors are often referred to as fitting coefficients, regression coefficients or combination coefficients: (1) sk (j) > 0 represents a positive contribution of the basis vector ak to the additive combination; (2) sk (j) = 0 represents a null contribution of the corresponding basis vector, i.e., it is not involved in the fitting or regression; (3) sk (j) < 0 implies a negative contribution of the basis vector ak . If the nonnegative data vectors measured at the discrete times j = 1, . . . , J are arranged as a nonnegative observation matrix, then [x(1), . . . , x(J)] = A[s(1), . . . , s(J)]



X = AS.

(6.6.5)

360

Solving Matrix Equations

The matrix S is called the coefficient matrix. It is essentially an encoding of the basis matrix. The problem of solving the blind nonnegative matrix equation X = AS can be described as: given a nonnegative matrix X ∈ RI×J (its entries xij ≥ 0) with a + I×r low rank r < min{I, J}, find the basis matrix A ∈ R+ and the coefficient matrix S ∈ Rr×J for X such that + X = AS + N (6.6.6) or Xij = [AS]ij + Nij =

r 

aik skj + nij ,

(6.6.7)

k=1

where N ∈ RI×J is the approximate error matrix. The coefficient matrix S is also called the encoding variable matrix, its entries are unknown hidden nonnegative components. The problem of decomposing a nonnegative data matrix into the product of a nonnegative basis matrix and a nonnegative coefficient matrix is called nonnegative matrix factorization (NMF) and was proposed by Lee and Seung in 1999 in Nature [286]. The NMF is essentially a multilinear nonnegative data representation. If the data matrix X is a positive matrix then the basis matrix A and the coefficient matrix S are required to be positive matrices. Such a matrix factorization is called positive matrix factorization (PMF) and was proposed by Paatero and Tapper in 1994 [367]. Equation (6.6.6) shows that when the data-matrix rank r = rank(X) < min{I, J}, the nonnegative matrix approximation AS can be regarded as a compression and de-noising form of the data matrix X. Table 6.1 shows the connections and differences between VQ, PCA and NMF. Table 6.1 Comparison of VQ, PCA and NMF

Constraints Form

VQ

PCA

NMF

winner-take-all  codebook matrix C coeff. matrix S

all share  basis matrix A coeff. matrix S  combination X = AS

minority share  basis matrix A coeff. matrix S NMF X = AS

Model

clustering X = CS

Analysis ability

nonlinear

linear

multilinear

Encoding way

grandmother cell

distributive

distributive + nonnegative

Machine learning

single mode

distributed

parts combination

6.6 Nonnegative Matrix Factorization: Optimization Theory

361

In the following, we address the features of NMF. 1. Distributed nonnegative encoding Nonnegative matrix factorization does not allow negative elements in the factorization matrices A and S. Unlike the single constraint of the VQ, the nonnegative constraint allows the use of a combination of basis images, or eigenfaces, to represent a face image. Unlike the PCA, the NMF allows only additive combinations; because the nonzero elements of A and S are all positive, the occurrence of any subtraction between basis images can be avoided. In terms of optimization criteria, the VQ adopts the “winnertake-all” constraint and the PCA is based on the “all-sharing” constraint, while the NMF adopts a “group-sharing” constraint together with a nonnegative constraint. From the encoding point of view, the NMF is a distributed nonnegative encoding, which often leads to sparse encoding. 2. Parts combination The NMF gives the intuitive impression that it is not a combination of all features; rather, just some of the features, (simply called the parts), are combined into a (target) whole. From the perspective of machine learning, the NMF is a kind of machine learning method based on the combination of parts which has the ability to extract the main features. 3. Multilinear data analysis capability The PCA uses a linear combination of all the basis vectors to represent the data and can extract only the linear structure of the data. In contrast, the NMF uses combinations of different numbers of and different labels of the basis vectors (parts) to represent the data and can extract its multilinear structure; thereby it has certain nonlinear data analysis capabilities.

6.6.4 Divergences and Deformed Logarithm The NMF is essentially an optimization problem and often uses a divergence as its cost function. The distance D(p g) between probability densities p and q is called the divergence if it meets only the nonnegativity and positive definite condition D(p g) ≥ 0 (the equality holds if and only if p = g). According to the prior knowledge of the statistical noise distribution, common divergences in NMF are the Kullback–Leibler divergence and the alpha–beta (AB) divergence and so on. 1. Squared Euclidean Distance Consider the NMF approximation X ≈ AS. The distance between the nonnegative measurement matrix X and the NMF AS is denoted D(X AS). When the approximation error follows a normal distribution, the NMF generally uses the squared

362

Solving Matrix Equations

Euclidean distance of the error matrix as the cost function: 1  2 (xij − [AS]ij ) . 2 i=1 j=1 I

DE (X AS) = X − AS 22 =

J

(6.6.8)

In many applications, in addition to the nonnegative constraint aik ≥ 0, skj ≥ 0, ∀ i, j, k, it is usually required that the expectation solution of S or of A has some characteristics associated with the cost functions JS (S) or JA (A) respectively, which yields the cost function DE (X AS) =

1

X − AS 22 + αA JA (A) + αS JS (S) 2

(6.6.9)

called the squared Euclidean distance between parameters αS and αA . 2. AB Divergence Let P, G ∈ RI×J be a nonnegative measurement matrix and its approximation matrix; then their alpha–beta divergence is simply called the AB divergence, and is defined as (see e.g., [14], [101], [102]) (α,β)

DAB (P G) ⎧ J ) I * ⎪ 1 α β α+β α+β ⎪ β α ⎪ − p , α, β, α + β = 0, g − p − g ⎪ ij ij ij ij α+β α+β ⎪ αβ i=1 j=1 ⎪ ⎪ ⎪ ) * ⎪1  J I  ⎪ pα ij α α ⎪ ⎪ pα , α=  0, β = 0, α − pij + gij ij ln gij ⎪ α2 ⎪ i=1 j=1  ⎪  ⎨ * ) α −1 J I gα gij 1  = ln pij −1 , α = −β = 0, α + α 2 p ⎪ α ij ij ⎪ i=1 j=1   ⎪ ⎪  J I  ⎪ gβ 1 ⎪ β β β ⎪ gij , α = 0, β = 0, ln pij ⎪ β2 β − gij + pij ⎪ ⎪ ij i=1 j=1 ⎪ ⎪ J I 2 ⎪ 1  ⎪ ⎪ α = 0, β = 0. ln pij − ln gij , ⎩2 i=1 j=1

The AB divergence is a constant function with α and β as parameters, and contains most divergences as special cases. (1) Alpha divergence When α + β = 1, the AB divergence reduces to the alpha divergence (α divergence): (α,1−α)

Dα (P G) = DAB

(P G)

  1 1−α − αpij + (α − 1)gij , pα ij gij α(α − 1) i=1 j=1 I

=

J

(6.6.10)

where α = 0 and α = 1. The following are some common α divergences [102]: • When α = 2, the α divergence reduces to the Pearson χ2 distance. • When α = 0.5, the α divergence reduces to the Hellinger distance.

6.6 Nonnegative Matrix Factorization: Optimization Theory

363

• When α = −1, the α divergence reduces to the Neyman χ2 distance. • As α → 0, the limit of the α divergence is the KL divergence of G from P, i.e., lim Dα (P G) = DKL (G P).

α→0

• As α → 1, the limit of the α is the KL divergence of P from G, i.e., lim Dα (P G) = DKL (P G).

α→1

(2) Beta divergence When α = 1, the AB divergence reduces to the beta divergence (β divergence): (1,β)

Dβ (P G) = DAB (P G)  J  I 1 β 1  β 1+β pij gij , − − =− p1+β g β i=1 j=1 1 + β ij 1 + β ij

(6.6.11)

where β = 0. In particular, if β = 1 then 1  = (p − gij )2 2 i=1 j=1 ij I

Dβ=1 (P G) =

(1,1) DAB (P G)

J

(6.6.12)

reduces to the squared Euclidean distance. (3) Kullback–Leibler (KL) divergence When α = 1 and β = 0, the AB divergence reduces to the standard KL divergence, i.e., (1,0)

DAB (P G) = Dβ=0 (P G) = DKL (P G). (4) Itakura–Saito (IS) divergence When α = 1 and β = −1, the AB divergence gives the standard Itakura–Saito divergence   J I   gij pij (1,−1) ln + −1 . (6.6.13) DIS (P G) = DAB (P G) = pij gij i=1 j=1 3. Kullback–Leibler Divergence Let φ : D → R be a continuous differentiable convex function defined in the closed convex set D ⊆ RK + . The Bregman distance between two vectors x, g ∈ D associated with the function φ is denoted Bφ (x g), and defined as def

Bφ (x g) = φ(x) − φ(g) − ∇φ(x), x − g.

(6.6.14)

Here ∇φ(x) is the gradient of the function φ at x. In particular, if φ(x) =

K 

xi ln xi

(6.6.15)

i=1

then the Bregman distance is called the Kullback–Leibler (KL) divergence, denoted

364

Solving Matrix Equations

DKL (x g). In probability and information theory, the KL divergence is also called the information divergence, the information gain, or the relative entropy. For two probability distribution matrices of a stochastic process P, G ∈ RI×J , if they are nonnegative then their KL divergence DKL (P G) is defined as   J I   pij pij ln (6.6.16) DKL (P G) = − pij + gij . gij i=1 j=1 Clearly, we have DKL (P G) ≥ 0 and DKL (P G) = DKL (G P), i.e., the KL divergence does not have symmetry. 4. Entropy, Tsallis Statistics and Deformed Logarithms In mathematical statistics and information theory, for a set of probabilities {pi }  such that i pi = 1, the Shannon entropy is defined as  S=− pi log2 pi , (6.6.17) i

while the Boltzmann–Gibbs entropy is defined as  pi ln pi . SBG = −k

(6.6.18)

i

For two independent systems A and B, their joint probability density p(A, B) = p(A)p(B), and their Shannon entropy S(·) and Boltzmann–Gibbs entropy SBG (·) have the additivity property S(A + B) = S(A) + S(B),

SBG (A + B) = SBG (A) + SBG (B),

so the Shannon entropy and the Boltzmann–Gibbs entropy are each extensive entropies. In physics, the Tsallis entropy is an extension of the standard Boltzmann–Gibbs entropy. The Tsallis entropy was proposed by Tsallis in 1988, and is also called the q-entropy [473]; it is defined as q 1 − i (pi )q Sq (pi ) = . (6.6.19) q−1 The Boltzmann–Gibbs entropy is the limit of the Tsallis entropy when q → 1, i.e., SBG = lim Dq (pi ). q→1

The Tsallis entropy has only pseudo-additivity: Sq (A + B) = Sq (A) + Sq (B) + (1 − q)Sq (A)Sq (B). Hence it is a nonextensive or nonadditive entropy [474]. The mathematical statistics defined by the Tsallis entropy is often called the Tsallis mathematical statistics. The major mathematical tools of Tsallis mathematical

6.6 Nonnegative Matrix Factorization: Optimization Theory

365

statistics are the q-logarithm and the q-exponential. In particular, the important q-Gaussian distribution is defined by the q-exponential. For nonnegative real numbers q and x, the function ⎧ (1−q) −1 ⎨x , q = 1 1−q (6.6.20) lnq x = ⎩ln x, q=1 is called the Tsallis logarithm of x [473], also called the q-logarithm. For all x ≥ 0, the Tsallis logarithm is an analytic, increasing, concave function. The inverse function of the q-logarithm is called the q-exponential, defined as ⎧ ⎪ (1 + (1 − q)x)1/(1−q) , 1 + (1 − q)x > 0, ⎪ ⎪ ⎪ ⎨0, q < 1, expq (x) = (6.6.21) ⎪ +∞, q > 1, ⎪ ⎪ ⎪ ⎩ exp(x), q = 1. The relationship between the q-exponential and the Tsallis logarithm is given by expq (lnq x) = x,

(6.6.22)

lnq expq (x) = x.

(6.6.23)

The probability density distribution f (x) is known as a q-Gaussian distribution, if √ β f (x) = expq (−βx2 ), (6.6.24) Cq where expq (x) = [1 + (1 − q)x]1/(1−q) is the q-exponential; the normalized factor Cq is given by √  1  ⎧ 2 π Γ 1−q ⎪ ⎪  1  , −∞ < q < 1, √ ⎪ ⎪ (3 − q) 1 − q Γ 1−q ⎪ ⎨√ Cq = π, q = 1, ⎪ √  3−q  ⎪ ⎪ π Γ ⎪ 2(q−1) ⎪√  1 , 1 < q < 3. ⎩ q − 1Γ

q−1

When q → 1, the limit of the q-Gaussian distribution is the Gaussian distribution; the q-Gaussian distribution is widely applied in statistical mechanics, geology, anatomy, astronomy, economics, finance and machine learning. Compared with the Gaussian distribution, a prominent feature of the q-Gaussian distribution with 1 < q < 3 is its obvious tailing. Owing to this feature, the qlogarithm (i.e., the Tsallis logarithm) and the q-exponential are very well suitable for using the AB-divergence as the cost function of the NMF optimization problems. In order to facilitate the application of the q-logarithm and the q-exponential in

366

Solving Matrix Equations

NMF, define the deformed logarithm

+

φ(x) = ln1−α x =

(xα − 1)/α,

α = 0,

ln x,

α = 0.

The inverse transformation of the deformed logarithm, ⎧ exp(x), α = 0, ⎪ ⎨ −1 1/α φ (x) = exp1−α (x) = (1 + αx) , α = 0, 1 + αx ≥ 0, ⎪ ⎩ 0, α = 0, 1 + αx < 0

(6.6.25)

(6.6.26)

is known as the deformed exponential. The deformed logarithm and the deformed exponential have important applications in the optimization algorithms of NMF.

6.7 Nonnegative Matrix Factorization: Optimization Algorithms The NMF is a minimization problem with nonnegative constraints. By the classification of Berry et al. [40], the NMF has three kinds of basic algorithm: (1) multiplication algorithms; (2) gradient descent algorithms; (3) alternating least squares (ALS) algorithms. These algorithms belong to the category of first-order optimization algorithms, where the multiplication algorithms are essentially gradient descent algorithms as well, but with a suitable choice of step size, a multiplication algorithm can transform the subtraction update rule of the general gradient descent method into a multiplicative update. Later, on the basis of the classification of Berry et al. [40], Cichocki et al. [104] added the quasi-Newton method (a second-order optimization algorithm) and the multilayer decomposition method. In the following, the above five representative methods are introduced in turn.

6.7.1 Multiplication Algorithms The gradient descent algorithm is a widely applied optimization algorithm; its basic idea is that a correction term is added during the updating of the variable to be optimized. The step size is a key parameter for this kind of algorithm, and determines the magnitude of the correction. Consider the general updating rule of the gradient descent algorithm for the unconstrained minimization problem min f (X): xij ← xij − ηij ∇f (xij ),

i = 1, . . . , I; j = 1, . . . , J,

(6.7.1)

6.7 Nonnegative Matrix Factorization: Optimization Algorithms

367

where xij is the entry of the variable matrix X, and ∇f (xij ) = [∂f (X)/∂X]ij is the gradient matrix of the cost function f (X) at the point xij . If the step size ηij is chosen in such a way that the addition term xij in the additive update rule is eliminated, then the original gradient descent algorithm with additive operations becomes the gradient descent algorithm with multiplication operations. This multiplication algorithm was proposed by Lee and Seung [287] for the NMF but is available for other many optimization problems as well. 1. Multiplication Algorithm for Squared Euclidean Distance Minimization Consider an unconstrained problem min DE (X AS) = 12 X − AS 22 using the typical squared Euclidean distance as the cost function; its gradient descent algorithm is given by ∂DE (X AS) , ∂aik ∂DE (X AS) − ηkj , ∂skj

aik ← aik − μik

(6.7.2)

skj ← skj

(6.7.3)

where   ∂DE (X AS) = − (X − AS)ST ik , ∂aik   ∂DE (X AS) = − AT (X − AS) kj ∂skj are the gradients of the cost function with respect to the entry aik of the variable matrix A and the entry skj of the variable matrix S, respectively. If we choose skj aik , ηkj = , (6.7.4) μik = T T [A AS]kj [ASS ]ik then the gradient descent algorithm becomes a multiplication algorithm: aik ← aik

[XST ]ik , [ASST ]ik

skj ← skj

[AT X]kj , [AT AS]kj

i = 1, . . . , I, k = 1, . . . , K, k = 1, . . . , K, j = 1, . . . , J.

(6.7.5) (6.7.6)

Regarding on multiplication algorithms, there are four important remarks to be made. Remark 1 The theoretical basis of the multiplication algorithm is the auxiliary function in the expectation-maximization (EM) algorithm. Hence, the multiplication algorithm is practically an expectation maximization maximum likelihood (EMML) algorithm [102].

368

Solving Matrix Equations

Remark 2 The elementwise multiplication algorithm above is easily rewritten as a multiplication algorithm in matrix form: < = A ← A ∗ (XST ) * (ASST ) , (6.7.7)   T (6.7.8) S ← S ∗ (A X) * (AT AS) , where B ∗ C represents the componentwise product (Hadamard product) of two matrices, while B * C denotes the componentwise division of two matrices, namely [B ∗ C]ik = bik cik ,

[B * C]ik = bik /cik .

(6.7.9)

Remark 3 In the gradient descent algorithm, a fixed step length or an adaptive step length is usually taken; this is independent of the index of the updated variable. In other words, the step size may change with the time but at a given update time. In the same update time, the different entries of the variable matrix are updated with the same step size. It contrast, the multiplication algorithm adopts different steps μik for the different entries of the variable matrix. Therefore, this kind of step length is adaptive to the matrix entries. This is an important reason why the multiplication algorithm can improve on the performance of the gradient descent algorithm. Remark 4 The divergence D(X AS) is nonincreasing in the different multiplicative update rules [287], and this nonincreasing property is likely to have the result that the algorithm cannot converge to a stationary point [181]. For the regularized NMF cost function J(A, S) =

1

X − AS 22 + αA JA (A) + αS JS (S), 2

(6.7.10)

since the gradient vectors are respectively given by ∂J (A) ∂DE (X AS) + αA A , ∂aik ∂aik ∂J (S) ∂DE (X AS) ∇skj J(A, S) = + αA S , ∂skj ∂skj ∇aik J(A, S) =

the multiplication algorithm should be modified to [287] < = [XST ]ik − αA ∇JA (aik ) + , aik ← aik T [ASS ]ik + < = [AT X]kj − αS ∇JS (skj ) + skj ← skj , [AT AS]kj +

(6.7.11)

(6.7.12)

where ∇JA (aik ) = ∂JA (A)/∂aik , ∇JS (skj ) = ∂JS (S)/∂skj , and [u]+ = max{u, }. The parameter is usually a very small positive number that prevents the emergence

6.7 Nonnegative Matrix Factorization: Optimization Algorithms

369

of a zero denominator, as needed to ensure the convergence and numerical stability of the multiplication algorithms. The elementwise multiplication algorithm above can also be rewritten in the matrix form ) * A ← A ∗ (XST − αA ΨA ) * (ASST + I) , (6.7.13)  T  S ← S ∗ (A X − αS ΨS ) * (AT AS + I) , (6.7.14) where ΨA = ∂JA (A)/∂A and ΨS = ∂JS (S)/∂ S are two gradient matrices, while I is the identity matrix. 2. Multiplication Algorithms for KL Divergence Minimization Consider the minimization of the KL divergence   I  J  xi 1 j 1 xi1 j1 ln DKL (X AS) = − xi1 j1 + [AS]i1 j1 . [AS]i1 j1 i =1 j =1 1

(6.7.15)

1

Since J  ∂DKL (X AS) =− ∂aik j=1 I  ∂DKL (X AS) =− ∂skj i=1

 

skj xij + skj [AS]ij aik xij + aik [AS]ij

 ,  ,

the gradient descent algorithm is given by ⎛  ⎞ J  skj xij aik ← aik − μik × ⎝− + skj ⎠ , [AS] ij j=1   I   aik xij . + aik skj ← skj − ηkj × − [AS]ij i=1 Selecting μik = J

1

j=1 skj

,

ηkj = I

1

i=1

aik

,

the gradient descent algorithm can be rewritten as a multiplication algorithm [287]: J j=1 skj xik /[AS]ik aik ← aik , (6.7.16) J j=1 skj I a x /[AS]ik skj ← skj i=1ikI ik . (6.7.17) i=1 aik

370

Solving Matrix Equations

The corresponding matrix form is ) *  * ) A ← A * 1I ⊗ (S1K )T ∗ (X * (AS)) ST , ) *  * ) S ← S * (AT 1I ) ⊗ 1K ∗ AT (X * (AS)) ,

(6.7.18) (6.7.19)

where 1I is an I × 1 vector whose entries are all equal to 1. 3. Multiplication Algorithm for AB Divergence Minimization For AB divergence (where α, β, α + β = 0) we have (α,β)

DAB (X AS)

 J  I α β 1  α β α+β α+β xij [AS]ij − ; − =− x [AS]ij αβ i=1 j=1 α + β ij α+β

(6.7.20)

its gradient is given by * ∂DAB (X AS) 1 ) 1−β α+β−1 skj xα =− [AS] − [AS] s ij kj , ij ij ∂aik α j=1

(6.7.21)

* ∂DAB (X AS) 1 ) 1−β α+β−1 aik xα =− [AS] − [AS] a ij ik . ij ij ∂skj α i=1

(6.7.22)

J

(α,β)

I

(α,β)

Hence, letting the step sizes be μik = J

αaik

α+β−1 j=1 skj [AS]ij

ηkj = I i=1

αskj aik [AS]α+β−1 ij

, ,

the multiplication algorithm for the AB divergence minimization is given by J β−1 α j=1 skj xij [AS]ij , aik ← aik J α+β−1 j=1 skj [AS]ij I β−1 α i=1 aik xij [AS]ij skj ← skj I . α+β−1 i=1 aik [AS]ij In order to speed up the convergence of the algorithm, one can use instead the following updating rules [101]  J  β−1 1/α α j=1 skj xij [AS]ij , (6.7.23) aik ← aik J α+β−1 j=1 skj [AS]ij  I  β−1 1/α α i=1 aik xij [AS]ij , (6.7.24) skj ← skj I α+β−1 i=1 aik [AS]ij

6.7 Nonnegative Matrix Factorization: Optimization Algorithms

371

where the positive relaxation parameter 1/α is used to improve the convergence of the algorithm. When β = 1 − α, from the multiplication algorithm for the AB divergence minimization we get the multiplication algorithm for the α divergence minimization,  J 1/α α j=1 (xij /[AS]ij ) skj aik ← aik , (6.7.25) J j=1 skj  I 1/α α i=1 aik (xij /[AS]ij ) . (6.7.26) skj ← skj I i=1 aik For the general cases of α = 0 and/or β = 0, the gradient of the AB divergence is given by [101] J (α,β)  xij ∂DAB (X AS) =− [AS]λ−1 , ij skj ln1−α ∂aik [AS] ij j=1

(6.7.27)

I (α,β)  xij ∂DAB (X AS) =− [AS]λ−1 , ij aik ln1−α ∂skj [AS] ij i=1

(6.7.28)

where λ = α + β. Then, by the relationship between the deformed logarithm and the deformed exponential exp1−α (ln1−α x) = x, it is easy to see that the gradient algorithm for AB divergence minimization is given by   (α,β) ∂DAB (X AS) , (6.7.29) aik ← exp1−α ln1−α aik − μik ∂ ln1−α aik   (α,β) ∂DAB (X AS) . (6.7.30) skj ← exp1−α ln1−α skj − ηkj ∂ ln1−α skj Choosing μik = J ηkj =

a2α−1 ik

,

λ−1 j=1 skj [AS]ij s2α−1 kj , I λ−1 a i=1 ik [AS]ij

then the multiplication algorithm for AB divergence-based NMF is as follows [101]: ⎛ ⎞ J  skj [AS]λ−1 x ij ij ⎠ aik ← aik exp1−α ⎝ ln1−α , (6.7.31) J λ−1 [AS] s [AS] ij ij j=1 kj j=1  I   aik [AS]λ−1 xij ij . (6.7.32) ln1−α skj ← skj exp1−α I λ−1 [AS]ij i=1 aik [AS]ij i=1 78 9 6 78 96 weighting coefficient

α-zooming

372

Solving Matrix Equations

By α-zooming is meant that, owing to the zoom function of the parameter α, the relative errors of updating the elements of the basis matrix A = [aik ] and the coefficient matrix S = [skj ] are mainly controlled by the deformed logarithm ln1−α (xij /[AS]ij ), as follows: (1) When α > 1, the deformed logarithm ln1−α (xij /[AS]ij ) has the “zoom-out” function and emphasizes the role of the larger ratio xij /[AS]ij . Because the smaller ratio is reduced, it can be more easily neglected. (2) When α < 1, the “zoom-in” function of the deformed logarithm highlights the smaller ratio xij /[AS]ij . Because the AB divergence contains several divergences as special cases, the AB multiplication algorithm for NMF is correspondingly available for a variety of NMF algorithms. In the updating formulas of various multiplication algorithms for NMF, it is usually necessary to add a very small perturbation (such as = 10−9 ) in order to prevent the denominator becoming zero.

6.7.2 Nesterov Optimal Gradient Algorithm The gradient descent NMF algorithm in fact consists of two gradient descent algorithms, ∂f (Ak , Sk ) , ∂Ak ∂f (Ak , Sk ) = Sk − μ S . ∂Sk

Ak+1 = Ak − μA Sk+1

In order to ensure the nonnegativity of Ak and Sk , all elements of the updated matrices Ak+1 and Sk+1 need to be projected onto the nonnegative quadrant in each updating step, which constitutes the projected gradient algorithm for NMF [297]:   ∂f (Ak , Sk ) Ak+1 = Ak − μA , (6.7.33) ∂Ak +   ∂f (Ak , Sk ) . (6.7.34) Sk+1 = Sk − μS ∂Sk + Consider the factorization of a low-rank nonnegative matrix X ∈ Rm×n 1 min X − AS 2F 2

subject to A ∈ Rm×r , S ∈ Rr×n + + ,

(6.7.35)

where r = rank(X) < min{m, n}. Since Equation (6.7.35) is a nonconvex minimization problem, one can use alternating nonnegative least squares to represent the local solutions of the minimization

6.7 Nonnegative Matrix Factorization: Optimization Algorithms

373

problem (6.7.35): Sk+1 ATk+1

" # 1 2 = arg min F (Ak , S) = X − Ak S F , 2 S≥O " # 1 T T T T T 2 = arg min F (Sk+1 , A ) = X − Sk+1 A F . 2 A≥O

(6.7.36) (6.7.37)

Here k denotes the kth data block. Recently, Guan et al. [191] showed that the NMF objective function satisfies the two conditions of the Nesterov optimal gradient method: (1) the objective function F (At , S) = 12 X − At S 2F is a convex function; (2) the gradient of the objective function F (At , S) is Lipschitz continuous with Lipschitz constant L = ATt At F . On the basis of the above results, a Nesterov nonnegative matrix factorization (NeNMF) algorithm was proposed in [191], as shown in Algorithm 6.6, where OGM(At , S) is given in Algorithm 6.7. Algorithm 6.6

NeNMF algorithm [191]

input: Data matrix X ∈ Rm×n , 1 ≤ r ≤ min{m, n}. + initialization: A1 ≥ O, S1 ≥ O, k = 1. repeat 1. Update Sk+1 = OGM(Ak , S) and Ak+1 = OGM(STk+1 , AT ). P 2. exit if ∇P S F (Ak , Sk ) = O and ∇A F (Ak , Sk ) = O, where  % $ P [Sk ]ij > 0, [∇S F (Ak , Sk )]ij ,   ∇S F (Ak , Sk ) ij = min 0, [∇S F (Ak , Sk )]ij , [Sk ]ij = 0.  % $ P [Ak ]ij > 0, [∇A F (Ak , Sk )]ij ,   ∇A F (Ak , Sk ) ij = min 0, [∇A F (Ak , Sk )]ij , [Ak ]ij = 0.

return k ← k + 1. r×n output: Basis matrix A ∈ Rm×r , coefficient matrix S ∈ R+ . +

Remark If Ak → STk and Sk → ATk in Algorithm 6.7, and the Lipschitz constant is replaced by L = Sk STk F , then Algorithm 6.7 will give the optimal gradient method OGM(STk+1 , AT ), yielding the output ATk+1 = ATt . It has been shown [191] that owing to the introduction of structural information, when minimizing one matrix with another matrix held fixed, the NeNMF algorithm can converge at the rate O(1/k 2 ).

374

Solving Matrix Equations Optimal gradient method OGM(Ak , S) [191]

Algorithm 6.7

input: Ak and Sk . initialization: Y0 = Sk ≥ O, α0 = 1, L = ATk Ak F , t = 0. repeat

  1. Update St = P+ Yt − L−1 ∇S F (Ak , Yt ) .   1 2. Update αt+1 = 1 + 4αt2 + 1 . 2 α −1 (St − St−1 ). 3. update Yt+1 = St + t αt+1 P 4. exit if ∇S F (Ak , St ) = O.

return t ← t + 1. output: Sk+1 = St .

6.7.3 Alternating Nonnegative Least Squares The alternating least squares method was used for NMF first by Paatero and Tapper [367]. The resulting method is called the alternating nonnegative least squares (ANLS) method. The NMF optimization problem for XI×J = AI×K SK×J , described by 1 min X − AS 2F A,S 2

subject to A, S ≥ O,

can be decomposed into two ANLS subproblems [367] " # 1 (A fixed), ANLS1 : min f1 (S) = AS − X 2F S≥O 2 " # 1 T T T T 2 (S fixed). ANLS2 : min f2 (A ) = S A − X F A≥O 2

(6.7.38)

(6.7.39) (6.7.40)

These two ANLS subproblems correspond to using the LS method to solve alternately the matrix equations AS = X and ST AT = XT , whose LS solutions are respectively given by   (6.7.41) S = P+ (AT A)† AT X ,   T † T T A = P+ (SS ) SX . (6.7.42) If A and/or S are singular in the iteration process, the algorithm cannot converge. In order to overcome the shortcoming of the ALS algorithm due to numerical stability, Langville et al. [277] and Pauca et al. [378] independently proposed a constrained nonnegative matrix factorization (CNMF): " #  1 2 2 2 CNMF : min subject to A, S ≥ O,

X − AS F + α A F + β S F A,S 2 (6.7.43)

6.7 Nonnegative Matrix Factorization: Optimization Algorithms

375

where α ≥ 0 and β ≥ 0 are two regularization parameters that suppress A 2F and

S 2F , respectively. This constrained NMF is a typical application of the regularization least squares method of Tikhonov [464]. The regularization NMF problem can be decomposed into two alternating regularization nonnegative least squares (ARNLS) problems " # 1 1 2 2 ARNLS1 : min J1 (S) = AS − X F + β S F (A fixed), (6.7.44) 2 2 S∈RJ×K + " # 1 1 J2 (AT ) = ST AT − XT 2F + α A 2F (S fixed), ARNLS2 : min 2 2 A∈RI×J + (6.7.45) equivalently written as

+

2   22 0 2 12 A X 2 √ ARNLS1 : min J1 (S) = 2 , S− 2 OJ×K 2F βIJ 2 S∈RJ×K + + 2 T   T 22 0 2 12 S X T T 2 2 J2 (A ) = 2 √ . ARNLS2 : min A − I×J αIJ OJ×I 2F 2 A∈R+

(6.7.46)

(6.7.47)

From the matrix differentials  1  dJ1 (S) = d tr((AS − X)T (AS − X)) + β tr(ST S) 2   = tr (ST AT A − XT A + β ST )dS ,  1  dJ2 (AT ) = d tr((AS − X)(AS − X)T ) + αtr(AT A) 2) * = tr (ASST − XST + αA)dAT ,

we get the gradient matrices ∂J1 (S) = −AT X + AT AS + βS, ∂S ∂J2 (AT ) = −SXT + SST AT + αAT . ∂AT

(6.7.48) (6.7.49)

From ∂J1 (S)/∂S = O and ∂J2 (AT )/∂AT = O, it follows that the solutions of the two regularization least squares problems are respectively given by (AT A + βIJ )S = AT X or T

T

T

(SS + αIJ )A = SX

or

S = (AT A + βIJ )−1 AT X, T

T

A = (SS + αIJ )

−1

T

SX .

(6.7.50) (6.7.51)

The LS method for solving the above two problems is called the alternate constrained least squares method, whose basic framework is as follows [277]. (1) Use the nonzero entries to initialize A ∈ RI×K .

376

Solving Matrix Equations

(2) Find iteratively the regularization LS solutions (6.7.50) and (6.7.51), and force the matrices S and A to be nonnegative: skj = [S]kj = max{0, skj }

and

aik = [A]ik = max{0, aik }.

(6.7.52)

(3) Now normalize each column of A and each row of S to the unit Frobenius norm. Then return to (2), and repeat the iterations until some convergence criteria are met. A better way, however, is to use the multiplication algorithms to solve two alternating least squares problems. From the gradient formulas (6.7.48) and (6.7.49), it immediately follows that the alternating gradient algorithm is given by   skj ← skj + ηkj AT X − AT AS − βS kj ,   aik ← aik + μik XST − ASST − αA ik . Choosing ηkj = 

skj T A AS +

βS

 , kj

μik = 

aik  , ASST + αA ik

the gradient algorithm becomes the multiplication algorithm  T  A X kj  , skj ← skj  T A AS + βS kj   XST ik  . aik ← aik  ASST + αA ik

(6.7.53) (6.7.54)

As long as the matrices A and S are initialized using nonnegative values, the above-described iteration ensures the nonnegativity of the two matrices. By [378], choosing step sizes ηkj = 

skj  T A AS

, kj

μik = 

aik  ASST ik

gives the multiplication algorithm  AT X − βS kj  ← skj  T , A AS kj +   XST − αA ik  . ← aik  ASST ik + 

skj aik

(6.7.55) (6.7.56)

Notice that the above algorithm cannot ensure the nonnegativity of the matrix entries, owing to the subtraction operations in the numerators.

6.7 Nonnegative Matrix Factorization: Optimization Algorithms

377

6.7.4 Quasi-Newton Method 1. Basic Quasi-Newton Method [539] Consider solving the over-determined matrix equation STK×J ATI×K = XTI×K (where J + I) for the unknown matrix AT , an effective way is the quasi-Newton method. The Hessian matrix HA = ∇2A (DE ) = II×I ⊗SST ∈ RIK×IK of the cost function DE (X AS) = (1/2) X − AS 22 is a block diagonal matrix whose block matrix on the diagonal is SST . Then the quasi-Newton method takes the update     , A ← A − ∇A DE (X AS) H−1 A here ∇A (DE (X AS)) = (AS − X)ST is the gradient matrix of the cost function DE (X AS). Hence, the quasi-Newton algorithm is given by <  −1 = A ← A − (AS − X)ST SST . In order to prevent the matrix SST from being singular or having a large condition number, one can use the relaxation method <  −1 = A ← A − (AS − X)ST SST + λIK×K , where λ is a Tikhonov regularization parameter. 2. Multilayer Decomposition Method [105], [106] The basic idea of the multilayer decomposition method is as follows. The largest error is in the first decomposition result X ≈ A(1) S(1) ; hence S(1) is regarded as a new data matrix, and then the NMF of the second layer S(1) ≈ A(2) S(2) is constructed. The second layer decomposition result is still subject to error, and so it is necessary to construct the third layer NMF S(2) ≈ A(3) S(3) , and to continue this process to form an L-layer NMF: X ≈ A(1) S(1) ∈ RI×J S(1) ≈ A(2) S(2) ∈ RK×J .. . S(L−1) ≈ A(L) S(L) ∈ RK×J

(where A(1) ∈ RI×K ), (where A(2) ∈ RK×K ),

(where A(L) ∈ RK×K ).

The decomposition of every layer is made by NMF, and any algorithm described above can be adopted. The final multilayer decomposition result is X ≈ A(1) A(2) · · · A(L) S(L) , from which the NMF is given by A = A(1) A(2) · · · A(L) and S = S(L) .

378

Solving Matrix Equations

6.7.5 Sparse Nonnegative Matrix Factorization When a sparse representation of the data is expected to obtained by NMF, it is necessary to consider NMF with a sparse constraint. Given a vector x ∈ Rn , Hoyer [220] proposed using the ratio of the 1 -norm and the 2 -norm, √ n − x 1 / x 2 √ , (6.7.57) sparseness (x) = n−1 as the measure of the sparseness of the vector. Clearly, if x has only one nonzero element then its sparseness is equal to 1; if and only if the absolute values of all the elements of x are equal, the sparseness of the vector is zero. The sparseness of any vector lies in the region [0, 1]. An NMF with nonnegative constraint is defined as follows [220]: given a nonnegI×K ative data matrix X ∈ RI×J and the + , find the nonnegative basis matrix A ∈ R+ K×J nonnegative coefficient matrix S ∈ R+ such that L(A, S) = X − AS 2F

(6.7.58)

is minimized and A and S satisfy respectively the following sparse constraints: sparseness (ak ) = Sa ,

sparseness (sk ) = Ss ,

k = 1, . . . , K.

Here ak and sk are respectively the kth column of the nonnegative matrix A and the kth row of S, K is the number of components, Sa and Ss are respectively the (expected) sparseness of the columns of A and the rows of S. The following is the basic framework of nonnegative matrix factorization with sparseness constraint [220]: 1. Use two random positive matrices to initialize A and S. 2. If adding a sparseness constraint to the matrix A then: • let A ← A − μA (AS − X)ST ; • project every column vector of A onto a new nonnegative column vector with unchanged 2 -norm but 1 -norm equal to the expected sparseness. 3. If not adding a sparseness constraint to A then A ← A ∗ (XST ) * (ASST ). 4. If adding a sparseness constraint to the matrix S then: • let S ← S − μS AT (AS − X); • project every row vector of S onto a new nonnegative row vector with unchanged 2 -norm but 1 -norm equal to the expected sparseness. 5. If not adding a sparseness constraint to S then S ← S ∗ (AT X) * (AT AS). In the following, we consider a regularization NMF with sparseness constraint on

6.7 Nonnegative Matrix Factorization: Optimization Algorithms

the columns of S: min A,S

⎧ 1⎨ 2⎩

AS − X 2F + α A 2F + β

J 

S:,j 21

j=1

379

⎫ ⎬ ⎭

subject to A, S ≥ O,

(6.7.59)

where S:,j is the jth column of the matrix S. The sparse NMF problem (6.7.59) can be decomposed into two alternating LS subproblems [247] + 2   22 0 12 A X 2 2 2 min J3 (S) = 2 √ T S − , (6.7.60) 2 β1 O 2 A∈RI×K K×J K + F + 2 T   T 22 0 2 12 S X T T 2 √ J4 (A ) = 2 . (6.7.61) min A − 2 αIK OK×I 2F 2 S∈RK×J + From the matrix differentials * 1 )  dJ3 (S) = d tr (AS − X)T (AS − X) + βST 1K 1TK S 2)  *  = tr ST AT A − XT A + βST EK dS , * ) dJ4 (AT ) = dJ2 (AT ) = tr (ASST − XST + αA)dAT , it follows that the gradient matrices of the objective function of the sparse nonnegative LS problem are given by ∂J3 (S) = −AT X + AT AS + βEJ S, ∂S ∂J4 (AT ) = −SXT + SST AT + αAT . ∂AT

(6.7.62) (6.7.63)

Here EK = 1K 1TK is a K × K matrix with all entries equal to 1. Then, the alternating sparse nonnegative LS solutions are respectively given by (AT A + βEJ )S = AT X, (SST + αIJ )AT = SXT , namely S = (AT A + βEJ )−1 AT X, T

T

A = (SS + αIJ )

−1

T

SX .

However, the gradient algorithm for the sparse NMF is [S]kj ← [S]kj + ηkj [AT X − AT AS − βEK S]kj , [AT ]ki ← [AT ]ki + μki [SXT − SST AT − αAT ]ki .

(6.7.64) (6.7.65)

380

Solving Matrix Equations

Choosing the step sizes ηkj =

[S]kj , + βEJ S]kj

[AT AS

μki =

[AT ]ki , + αAT ]ki

[SST AT

the gradient algorithm becomes the multiplication algorithm [S]kj ← [S]kj

[AT X]kj , [AT AS + βEJ S]kj

[AT ]ki ← [AT ]ki

(6.7.66)

[SXT ]ki . + αAT ]ki

[SST AT

(6.7.67)

This is the alternating LS multiplication algorithm for a sparse NMF [247]. The NMF is widely used, and here we present a few typical applications. In image processing the data can be represented as an m × n nonnegative matrix X each of whose columns is an image described by m nonnegative pixel values. Then NMF gives two factor matrices A and S such that X ≈ AS, where A is the basis matrix and S is the coding matrix. In pattern recognition applications such as face recognition, each column of the basis matrix A can be regarded as one key part of the whole face, such as nose, eye, ear etc., and each column of the coding matrix S gives a weight by which the corresponding image can be reconstructed as a linear combination of different parts. Then, NMF can discover the common basis hidden behind an observed image and hence perform face recognition. However, for an image such as a face or brain, etc., the standard version of NMF does not necessarily provide a correct part-of-whole representation. In such cases, it is necessary to add the sparsity constraint of NMF to the face or brain recognition in order to improve the part-of-whole representation of the image. The main purpose of clustering text, image or biology data is to discover patterns from data automatically. Given an m × n nonnegative matrix X = [x1 , . . . , xn ], suppose that there are r cluster patterns. Our problem is to determine to which cluster the m × 1 data vector xj belongs. Via NMF, we can find two factor matrices A ∈ Rm×r and S ∈ Rr×n such that X ≈ AS, where r is the cluster number. The + + factor matrix A can be regarded as a cluster centroid matrix, while S is a cluster membership indicator matrix. The NMF model X = AS can be rewritten as ⎤ ⎡ S11 · · · S1n ⎢ .. ⎥ .. (6.7.68) [x1 , . . . , xn ] = [a1 , . . . , ar ] ⎣ ... . . ⎦ Sr1

···

Srn

or xj =

r  i=1

ai Sij ,

j = 1, . . . , n.

(6.7.69)

6.8 Sparse Matrix Equation Solving: Optimization Theory

381

This shows that if Skj is the largest value of the jth column of S then the jth data vector xj can be determined as belonging to the kth cluster pattern. Nonnegative matrix factorization has been successfully applied in different fields such as metagenes and molecular pattern discovery in bioinformatics [70], text or document clustering [379], [434], the analysis of financial data [134] and so on. The use of K-means provides one of the most famous and traditional methods in clustering analysis, and probabilistic latent semantic indexing (PLSI) is one of the state-of-the-art unsupervised learning models in data mining. The NMF method is closely related to K-means and the PLSI [544] as follows. (1) A model using soft K-means can be rewritten as a symmetric NMF model, and hence K-means and NMF are equivalent. This equivalence justifies the ability of NMF for data clustering, but this does not mean that K-means and NMF will generate identical cluster results since they employ different algorithms. (2) The PLSI and NMF methods optimize the same objective function (the KL divergence), but the PLSI has additional constraints. The algorithms for the two models can generate equivalent solutions, but they are different in essence.

6.8 Sparse Matrix Equation Solving: Optimization Theory In sparse representation and compressed sensing (also known as compressive sensing, compressive sampling, or sparse sampling), it is necessary to solve an underdetermined sparse matrix equation Ax = b, where x is a sparse vector with only a few nonzero entries. In this section we discuss the use of optimization theory for solving sparse matrix equations.

6.8.1 1 -Norm Minimization A full row rank under-determined matrix equation Am×n xn×1 = bm×1 with m < n has infinitely many solutions. Suppose that we seek the sparsest solution, with the fewest nonzero entries. Can it ever be unique? How do we find the sparsest solution? This section answers the first question; the second question will be discussed in the next section. For any positive number p > 0, the p -norm of the vector x is defined as ⎛

x p = ⎝

⎞1/p



|xi |p ⎠

.

(6.8.1)

i∈support(x)

Thus the 0 -norm of an n × 1 vector x can be defined as

x 0 = lim x pp = lim p→0

p→0

n  i=1

|xi |p =

n  i=1

1(xi = 0) = #{i|xi = 0}.

(6.8.2)

382

Solving Matrix Equations

Thus if x 0 ' n then x is sparse. As described in Chapter 1, the core problem of sparse representation is the 0 norm minimization (P0 )

min x 0 x

subject to b = Ax,

(6.8.3)

where A ∈ Rm×n , x ∈ Rn , b ∈ Rm . As an observation signal is usually contaminated by noise, the equality constraint in the above optimization problem should be relaxed to 0 -norm minimization with the inequality constraint ≥ 0 which allows a certain error perturbation: min x 0 x

subject to

Ax − b 2 ≤ .

(6.8.4)

A key term coined and defined in [130] that is crucial for the study of uniqueness is the spark of the matrix A. DEFINITION 6.7 [130] Given a matrix A, its sparsity σ = Spark(A) is defined as the smallest possible number such that there exists a subgroup of σ columns from A that are linearly dependent. The spark gives a simple criterion for the uniqueness of a sparse solution of an under-determined system of linear equations Ax = b, as shown below. THEOREM 6.9 [182], [130], [69] If a system of linear equations Ax = b has a solution x obeying x 0 < spark(A)/2, this solution is necessarily the sparsest solution. To solve the optimization problem (P0 ) directly, one must screen out all nonzero elements in the coefficient vector x. However, this method is intractable, or nondeterministic polynomial-time hard (NP-hard), since the searching space is too large [313], [118], [337]. The index set of nonzero elements of the vector x = [xi , . . . , xn ]T is called its support, denoted support(x) = {i|xi = 0}. The length of the support (i.e., the number of nonzero elements) is measured by the 0 -norm

x 0 = |support(x)|.

(6.8.5)

A vector x ∈ Rn is said to be K-sparse if x 0 ≤ K, where K ∈ {1, . . . , n}. The set of K-sparse vectors is denoted by    ΣK = x ∈ Rn×1  x 0 ≤ K . (6.8.6) ˆ ∈ ΣK then the vector x ˆ ∈ Rn is known as the K-sparse approximation or If x K-term approximation of x ∈ Rn . A given set of L m×1 real input vectors {y1 , . . . , yL } constitutes the data matrix Y = [y1 , . . . , yL ] ∈ Rm×L . The sparse coding problem involves determining n m × 1 basis vectors a1 , . . . , an ∈ Rm , and, for each input vector bl determining an n × 1

6.8 Sparse Matrix Equation Solving: Optimization Theory

383

sparse weighting vector, or coefficient vector, sl ∈ Rn such that the weighted linear combination of a few of these basis vectors can approximate the original input vector, namely n  sl,i ai = Asl , l = 1, . . . , L, (6.8.7) yl = i=1

where sl,i is the ith entry of the sparse weighting vector sl . Sparse coding can be regarded as a form of neural coding: as the weight vector is sparse, so for each input vector, only a small number of neurons (basis vectors) are strongly excited. If the input vectors are different then the excited neurons are also different. ˆ is said to be the optimal K-sparse approximation of x in the p The vector x ˆ reaches an norm condition if the p -norm of the approximation error vector x − x infimum, i.e., ˆ p = inf x − z p .

x − x z∈ΣK

Obviously, there is a close relationship between the 0 -norm definition formula (6.8.5) and the p -norm definition formula (6.8.1): as p → 0, x 0 = lim x pp . p→0

Because, if and only if p ≥ 1, x p is a convex function, the 1 -norm is the objective function closest to 0 -norm. Thus, from the viewpoint of optimization, the 1 -norm is said to be the convex relaxation of the 0 -norm. Therefore, the 0 -norm minimization problem (P0 ) can be transformed into the convex relaxed 1 -norm minimization problem min x 1

(P1 )

x

subject to

b = Ax.

(6.8.8)

This is a convex optimization problem because the 1 -norm x 1 , as the objective function, is itself convex and the equality constraint b = Ax is affine. In an actual situation of existing observation noise, the equality constrained optimization problem (P1 ) can be relaxed to the inequality constrained optimization problem (P10 )

min x 1 x

subject to

b − Ax 2 ≤ .

(6.8.9)

The 1 -norm minimization in Equation (6.8.9) is also called basis pursuit (BP). This is a quadratically constrained linear programming (QCLP) problem. If x1 is the solution to (P1 ) and x0 is the solution to (P0 ), then [129]

x1 1 ≤ x0 1 ,

(6.8.10)

because x0 is only a feasible solution to (P1 ), while x1 is the optimal solution to (P1 ). The direct relationship between x0 and x1 is given by Ax1 = Ax0 . Similarly to the inequality constrained 0 -norm minimization expression (6.8.4),

384

Solving Matrix Equations

the inequality constrained 1 -norm minimization expression (6.8.9) also has two variants: (1) Since x is constrained as a K-sparse vector, the inequality constrained 1 -norm minimization becomes the inequality constrained 2 -norm minimization 1 min b − Ax 22 subject to x 1 ≤ q. (6.8.11) x 2 This is a quadratic programming (QP) problem. (2) Using the Lagrange multiplier method, the inequality constrained 1 -norm minimization problem (P11 ) becomes " # 1 (6.8.12) (P12 ) min

b − Ax 22 + λ x 1 . λ,x 2 (P11 )

This optimization is called basis pursuit de-noising (BPDN) [96]. The optimization problems (P10 ) and (P11 ) are respectively called error constrained 1 -norm minimization and 1 -penalty minimization [471]. The 1 -penalty minimization is also known as regularized 1 linear programming or 1 -norm regularization least squares. The Lagrange multiplier acts as a regularization parameter that controls the sparseness of the sparse solution; the greater λ is, the more sparse the solution x is. When the regularization parameter λ is large enough, the solution x is a zero vector. With the gradual decrease of λ, the sparsity of the solution vector x also gradually decreases. As λ tends to 0, the solution vector x becomes the vector such that b − Ax 22 is minimized. That is to say, λ > 0 can balance the twin objectives by minimizing the error squared sum cost function 12 b − Ax 22 and the 1 -norm cost function x 1 : 1 (6.8.13) J(λ, x) = b − Ax 22 + λ x 1 . 2 6.8.2 Lasso and Robust Linear Regression Minimizing the squared error usually leads to sensitive solutions. Many regularization methods have been proposed to decrease this sensitivity. Among them, Tikhonov regularization [465] and Lasso [462], [143] are two widely known and cited algorithms, as pointed out in [515]. The problem of regression analysis is a fundamental problem within many fields, such as statistics, supervised machine learning, optimization and so on. In order to reduce the computational complexity of solving the optimization problem (P1 ) directly, consider a linear regression problem: given an observed data vector b ∈ Rm and an observed data matrix A ∈ Rm×n , find a fitting coefficient vector x ∈ Rn such that ˆbi = x1 a + x2 a + · · · + xn a , i = 1, . . . , m, (6.8.14) i1

i2

in

6.8 Sparse Matrix Equation Solving: Optimization Theory

385

or ˆ= b

n 

xi ai = Ax,

(6.8.15)

i=1

where

a11 ⎢ .. A = [a1 , . . . , an ] = ⎣ .

··· .. .

⎤ a1n .. ⎥ , . ⎦

am1

···

amn



T

x = [x1 , . . . , xn ] , b = [b1 , . . . , bm ]T . As a preprocessing of the linear regression, it is assumed that m 

bi = 0,

i=1

m 

aij = 0,

i=1

m 

a2ij = 1,

j = 1, . . . , n,

(6.8.16)

i=1

and that the column vectors of the input matrix A are linearly independent. The preprocessed input matrix A is called an orthonormal input matrix, its column vectors ai are known as a predictors and the vector x is simply called the coefficient vector. Tibshirani [462] in 1996 proposed the least absolute shrinkage and selection operator (Lasso) algorithm for solving the linear regression. The basic idea of the Lasso is as follows: under the constraint that the 1 -norm of the prediction vector does not exceed an upper bound q, the squared sum of the prediction errors is minimized, namely Lasso :

min b − Ax 22 x

subject to x 1 =

n 

|xi | ≤ q.

(6.8.17)

i=1

Obviously, the Lasso model and the QP problem (6.8.11) have exactly the same form. The bound q is a tuning parameter. When q is large enough, the constraint has no effect on x and the solution is just the usual multiple linear least squares regression of x on ai1 , . . . , ain and bi , i = 1, . . . , m. However, for smaller values of q (q ≥ 0), some of the coefficients xj will take the value zero. Choosing a suitable q will lead to a sparse coefficient vector x. The regularized Lasso problem is given by   Regularized Lasso: min b − Ax 22 + λ x 1 . (6.8.18) x

The Lasso problem involves both the 1 -norm constrained fitting for statistics and data mining. The Lasso has the following two basic functions.

386

Solving Matrix Equations

(1) Contraction function The Lasso shrinks the range of parameters to be estimated; only a small number of selected parameters are estimated at each step. (2) Selection function The Lasso automatically selects a very small part of the variables for linear regression, yielding a spare solution. Therefore, the Lasso is a shrinkage and selection method for linear regression. It has a close relation to the soft thresholding of wavelet coefficients, forward stagewise regression, boosting methods and so on. The Lasso achieves a better prediction accuracy by shrinkage and variable selection. In practical linear regression, the observed matrix is usually corrupted by some potentially malicious perturbation. In order to find the optimal solution in the worst case, robust linear regression solves the following minimax problem [515] min max b − (A + ΔA)x 2 ,

x∈Rn ΔA∈ U

where ΔA = [δ1 , . . . , δn ] denotes the perturbation of the matrix A and  3 4 def  U = [δ1 , . . . , δn ] δi 2 ≤ ci , i = 1, . . . , n

(6.8.19)

(6.8.20)

is called the uncertainty set, or the set of admissible perturbations of the matrix A. This uncertainty set is said to be featurewise uncoupled, in contrast with coupled uncertainty sets, which require the perturbations of different features to satisfy some joint constraints. Uncoupled 1 norm-bounded uncertainty sets lead to an easily solvable optimization problem, as shown in the following theorem. THEOREM 6.10 [515] The robust regression problem (6.8.19) with uncertainty set U given by (6.8.20) is equivalent to the following 1 -regularized regression problem: + 0 n  minn b − Ax 2 + ci |xi | . (6.8.21) x∈R

i=1

With the 1 -norm changed to an arbitrary norm · a , Theorem 6.10 can be extended to the following result. THEOREM 6.11 [515]

Let  3 4 def  Ua = [δ1 , . . . , δn ] δi a ≤ ci , i = 1, . . . , n .

(6.8.22)

Then the robust regression problem (6.8.19) with uncertainty set Ua is equivalent to the following arbitrary norm · a -regularized regression problem: " # (6.8.23) minn max b − (A + ΔA)x a . x∈R

ΔA∈Ua

6.8 Sparse Matrix Equation Solving: Optimization Theory

THEOREM 6.12

387

The optimization problems

minn ( b − Ax 2 + λ1 x 1 )

x∈R

and

  minn b − Ax 22 + λ2 x 1

x∈R

(6.8.24)

are equivalent, up to changes in the tradeoff parameters λ1 and λ2 . Proof

See Appendix A in [515].

Taking ci = λ and orthonormalizing the column vector ai of A for all i = 1, . . . , n, the 1 regularized regression problem (6.8.21) becomes min ( b − Ax 2 + λ x 1 ) .

x∈Rn

(6.8.25)

By Theorem 6.12, this optimization problem is equivalent to the regularized Lasso problem (6.8.18). That is to say, a robust linear regression problem is equivalent to a regularized Lasso problem. Regarding the solution to Lasso as the solution of a robust least squares problem has two important consequences [515]. (1) Robustness provides a connection of the regularizer to a physical property, namely, protection from noise. (2) Perhaps most significantly, robustness is a strong property that can itself be used as an avenue for investigating different properties of the solution: robustness of the solution can explain why the solution is sparse; moreover, a robust solution is, by definition, the optimal solution under a worst-case perturbation. The Lasso has many generalizations; here are a few examples. In machine learning, sparse kernel regression is referred to as generalized Lasso [418]. The multidimensional shrinkage-thresholding method is known as group Lasso [402]. The sparse multiview feature selection method via low-rank analysis is called MRMLasso (multiview rank minimization-based Lasso) [528]. Distributed Lasso solves distributed sparse linear regression problems [316].

6.8.3 Mutual Coherence and RIP Conditions The 1 -norm minimization problem (P1 ) is a convex relaxation of the 0 -norm minimization (P0 ). Unlike the 0 -norm minimization problem, the 1 -norm minimization problem has the tractability. A natural question to ask is “what is the relationship between these two kinds of optimization problems?” DEFINITION 6.8 (see [313], [131], [69]) The mutual coherence of a given matrix A is the largest absolute normalized inner product between different columns from A. Denoting the kth column in A by ak , the mutual coherence is thus defined as μ(A) =

max

1≤k,j≤n,k=j

|aTk aj | .

ak 2 aj 2

(6.8.26)

388

Solving Matrix Equations

THEOREM 6.13 [130], [69]

For any matrix A ∈ Rm×n , one has spark(A) ≥ 1 +

1 . μ(A)

(6.8.27)

LEMMA 6.4 [130], [69] For an under-determined equation Ax = b (A ∈ Rm×n with full rank), if there exists a solution x such that   1 1 1+ (6.8.28)

x 0 < 2 μ(A) then x is not only the unique solution of (P1 ) but also the unique solution of (P0 ). DEFINITION 6.9 A matrix A is said to satisfy a Kth-order restricted isometry property (RIP) condition, if

x 0 ≤ K



(1 − δK ) x 22 ≤ AK x 22 ≤ (1 + δK ) x 22 ,

(6.8.29)

where 0 ≤ δK < 1 is a constant related to the sparseness of A, while AK is a submatrix consisting of any K columns of the dictionary matrix A. The RIP condition was presented by Cand`es and Tao [81] in 2006 and was refined by Foucart and Lai [160] in 2009. When the RIP condition is satisfied, the nonconvex 0 -norm minimization (P0 ) is equivalent to the convex 1 -norm minimization (P1 ). Let I = {i|xi = 0} ⊂ {1, . . . , n} denote the support of the nonzero elements of the sparse vector x; then |I| denotes the length of the support, i.e., the number of nonzero elements of the sparse vector x. The Kth-order RIP condition with parameter δK is denoted by RIP(K, δK ), where δK is called the restricted isometry constant (RIC) and is defined as the infimum of all the parameters δ such that RIP(K, δK ) holds, i.e.,    δK = inf δ (1 − δ) z 22 ≤ AI z 22 ≤ (1 + δ) z 22 , ∀ |I| ≤ K, ∀ z ∈ R|I| . (6.8.30) By Definition 6.9 it is known that if the matrix AK is orthogonal, then δK = 0 since AK x 2 = x 2 . Therefore, a nonzero value of the RIC δK of a matrix can be used to evaluate the nonorthogonality of the matrix. It is shown in [76] that a more compact lower bound of the RIC is δK < 0.307.

(6.8.31)

Under this condition, if there is no noise then a K-sparse signal can be exactly recovered by 1 -norm minimization and, in the case of noise, the K-sparse signals can be robustly estimated by 1 -norm minimization. The RIC meets the monotonicity condition [76] δK ≤ δK 1

if K ≤ K1 .

(6.8.32)

6.8 Sparse Matrix Equation Solving: Optimization Theory

389

6.8.4 Relation to Tikhonov Regularization In Tikhonov regularization LS problems, when using the 1 -norm of the unknown coefficient vector x in the regularization term instead of the 2 -norm, we have the 1 regularization LS problem " # 1 2 (6.8.33) min

b − Ax 2 + λ x 1 . x 2 The 1 regularization LS problem always has a solution, but not necessarily a unique solution. The following are the 1 regularization properties, which reflect the similarity and differences between 1 -regularization and Tikhonov regularization [248]. 1. Nonlinearity The solution vector x of a Tikhonov regularization problem is a linear function of the observed data vector b, but the solution vector of the 1 -regularization problem is not. 2. Regularization path When the regularization parameter λ varies in the interval [0, ∞), the optimal solution of the Tikhonov regularization problem is a smooth function of the regularization parameter. However, the solution family of the 1 -regularization problem has a piecewise linear solution path [143]: there are regularization parameters λ1 , . . . , λk (where 0 = λk < · · · < λ1 = λmax ) such that the solution vector of the 1 -regularization problem is piecewise linear: x1 =

λ − λi+1 (i) λi − λ (i+1) x − x . λi − λi+1 1 λi − λi+1 1

(6.8.34)

(i)

Here λi+1 ≤ λ ≤ λi , i = 1, . . . , k − 1, and x1 is the solution vector of the 1 -regularization problem when the regularization parameter is λi , while x1 is (1)

the optimal solution of the 1 -regularization problem. Therefore, x1 = 0 and x1 = 0 when λ ≥ λ1 . 3. Limit characteristic of λ → 0 When λ → 0, the limit point of the solution of the Tikhonov regularization problem has the smallest 2 -norm x 2 among all the feasible points such that AH (b − Ax) = 0. In contrast the limit of the solution of the 1 -regularization problem when λ → 0 has the smallest 1 -norm

x 1 among all the feasible points such that AH (b − Ax) = 0. 4. Limit characteristics of λ ≥ λmax When λ → ∞, the optimal solution of the Tikhonov regularization problem converges to a zero vector. However, only if λ ≥ λmax = AH b ∞ ,

(6.8.35)

does the solution of the 1 -regularization problem converge to a zero vector. In the above equation, AH b ∞ = max{[AH b]i } is the ∞ -norm of the vector. The most fundamental difference between the 1 -regularization LS problem and the Tikhonov regularization LS problem is that the solution of the former is usually a sparse vector while the coefficients of the solution of the latter are generally nonzero.

390

Solving Matrix Equations

6.8.5 Gradient Analysis of 1 -Norm Minimization The signum function of a real-valued variable ⎧ ⎪ ⎪ ⎨+1, sign(t) = 0, ⎪ ⎪ ⎩−1,

t ∈ R is defined as t > 0, (6.8.36)

t = 0, t < 0.

The signum multifunction of t ∈ R, denoted SGN(t), also called the set-valued function, is defined as [194] ⎧ ⎪ t > 0, ⎪{+1}, ∂|t| ⎨ = [−1, +1], t = 0, (6.8.37) SGN(t) = ⎪ ∂t ⎪ ⎩{−1}, t < 0. The signum multifunction is also the subdifferential of |t|. For the 1 -norm optimization problem " # 1 min J(λ, x) = min

b − Ax 22 + λ x 1 , x x 2

(6.8.38)

the gradient vector of the objective function is given by ∂J(λ, x) = −AT (b − Ax) + λ∇x x 1 ∂x = −c + λ∇x x 1 .

∇x J(λ, x) =

(6.8.39)

Here c = AT (b − Ax) is the vector of residual correlation vector, and ∇x x 1 = [∇x1 x 1 , . . . , ∇xn x 1 ]T is the gradient vector of the 1 -norm x 1 , with ith entry ⎧ ⎪ xi > 0, ⎪ ⎨{+1}, ∂ x 1 ∇xi x 1 = = {−1}, (6.8.40) xi < 0, ⎪ ∂xi ⎪ ⎩[−1, +1], x = 0, i

for i = 1, . . . , n. From (6.8.39) it can be seen that the stationary point of the 1 -norm minimization problem (P12 ) is determined by the condition ∇x J(λ, x) = −c + λ∇x x 1 = 0, i.e., c = λ∇x x 1 .

(6.8.41)

Letting c = [c(1), . . . , c(n)]T and substituting (6.8.40) into (6.8.41), the stationarypoint condition can be rewritten as ⎧ ⎪ xi > 0, ⎪{+λ}, ⎨ (6.8.42) c(i) = {−λ}, xi < 0, ⎪ ⎪ ⎩[−λ, λ], x = 0, i

for i = 1, . . . , n. Since 1 -norm minimization is a convex optimization problem, the

6.9 Sparse Matrix Equation Solving: Optimization Algorithms

391

above stationary-point condition is in practice a sufficient and necessary condition for the optimal solution of the 1 -norm minimization. The stationary-point condition (6.8.42) can be represented using the vector of residual correlations as c(I) = λsign(x) and

|c(I c )| ≤ λ,

(6.8.43)

in which I c = {1, . . . , n}\I is the complementary set of the support I. This shows that the amplitude of the residual correlation is equal to λ and the sign is consistent with the sign of the corresponding element of the vector x. Equation (6.8.43) can be equivalently written as |c(j)| = λ,

∀j ∈ I

and

|c(j)| ≤ λ

∀ j ∈ I c.

(6.8.44)

That is to say, the absolute value of the residual correlation within the support is equal to λ, while that outside the support is less than or equal to λ, namely

c ∞ = max{c(j)} = λ.

6.9 Sparse Matrix Equation Solving: Optimization Algorithms The common basic idea behind 1 -norm minimization algorithms is as follows: by recognizing the support region of sparse vectors, the solution of an underdetermined sparse matrix equation is transformed into the solution of an overdetermined (nonsparse) matrix equation.

6.9.1 Basis Pursuit Algorithms The orthogonal matching pursuit method is a kind of greedy stepwise least squares method for fitting sparse models. The general method for finding the whole optimal solution of an under-determined matrix equation Am×n xn×1 = bm×1 (m ' n) with sparseness s is first to find the ˜ s×1 = b (usually LS solutions of all the over-determined matrix equations Am×s x m + s) and then to determine the optimal solution among these solutions. Because there are Csm possible combinations of the over-determined equations, the overall solution process is time-consuming and tedious. The basic idea of a greedy algorithm [470] is not to seek a global optimal solution but to try to find a local optimal solution in some sense as quickly as possible. The greedy method cannot get a global optimal solution for all problems, but it may provide a global optimal solution or its approximation in most cases. Typical greedy algorithms use the following matching pursuit methods: 1. Basic matching pursuit (MP) method This was proposed by Mallat and Zhang [313] in 1993; its basic idea is not to minimize some cost function, but to iteratively construct a sparse solution x using a linear combination of a few column

392

2.

3.

4.

5.

Solving Matrix Equations

vectors (called atoms) of a dictionary matrix A to make a sparse approximation to x to get Ax = b. In each iteration the column vector in the dictionary matrix A that is most similar to the present residual vector r = Ax − b is selected as a new column of the action set. If the residual is decreasing with iteration, the convergence of the MP algorithm can be guaranteed. Orthogonal matching pursuit (OMP) Matching pursuit can construct a residual vector orthogonal to every column vector of the selected dictionary matrix at each iteration, but the residual vector generally is not orthogonal to the previously selected column vectors of the dictionary matrix. The OMP [377], [118], [173] ensures that the residual vector is orthogonal to all the previously selected columns vectors after each iteration, and thus guarantees the optimality of the iteration process, and reduces the number of iterations, making the performance more robust. The complexity of the OMP algorithm is O(mn), and it can produce a coefficient vector with the sparseness K ≤ m/(2 log n). Regularization orthogonal matching pursuit (ROMP) On the basis of the OMP algorithm, the regularization procedure is added [340], [341] by first selecting multiple atoms from the relevant atoms as the candidate set and then selecting a fraction of atoms from the candidate set in accordance with the regularization principle. Finally, these atoms are incorporated into the final support set to achieve a fast and effective choice of atoms. Stagewise orthogonal matching pursuit (StOMP) This is a reduced form of the OMP algorithm [133]. At the expense of approximation accuracy, it further improves the computation speed and has complexity O(n), which is more suitable for solving large-scale sparse approximation problems. Compression sampling matching pursuit (CoSaMP) This is an improvement of ROMP [342]. As compared with the OMP algorithm, the approximate accuracy of the CoSaMP algorithm is higher, its complexity O(n log2 n) is lower and the sparseness of the sparse solution is K ≤ m/(2 log(1 + n/K)). Algorithm 6.8 shows the orthogonal matching pursuit algorithm. The following are three different stopping criteria [472].

(1) Stop running after a fixed number of iterations. (2) Stop running when the residual energy (the sum of the squared amplitudes of the residual vector’s components) rk 2 is less than a given value , i.e.,

rk 2 ≤ . (3) Stop running when no column of A has any significant residual energy rk , i.e.,

AH rk ∞ ≤ . A subspace pursuit algorithm for compressed-sensing signal reconstruction is shown in Algorithm 6.9. At the kth iteration, the OMP, stagewise OMP and regular OMP methods combine the index of the new candidate column of the dictionary matrix A with the

6.9 Sparse Matrix Equation Solving: Optimization Algorithms Algorithm 6.8

393

Orthogonal matching pursuit algorithm [377], [118], [69]

task: Approximate the solution of (P0 ): argmin x0 subject to Ax = b. x

input: A ∈ Rm×n , b ∈ Rm and the error threshold . initialization: Initial solution x0 = 0, initial residual r0 = Ax0 − b = b, initial solution support Ω0 = support(x0 ) = ∅. Put k = 1. repeat 1. Sweep: use zj = aTj rk−1 /a2j to compute (j) = min aj zj − rk−1 22 , ∀j. zj

/ Ωk−1 , and 2. Update support: find a minimizer j0 of (j) satisfying (j0 ) ≤ (j), ∀ j ∈ update Ωk = Ωk−1 ∪ j0 . −1 H 3. Update provisional solution: compute xk = (AH AΩk b. Ωk AΩk ) 4. Update residual: calculate rk = b − AΩk xk . 5. exit if the stopping criterion is satisfied. return k ← k + 1.  xk (i), i ∈ Ωk , output: The sparse coefficient vector x(i) = 0, i∈ / Ωk .

Algorithm 6.9

Subspace pursuit algorithm [115]

task: Approximate the solution of (P0 ): argmin x0 subject to Ax = b. x

input: Sparsity K, A ∈ Rm×n , b ∈ Rm . initialization: 1. Ω0 = {K| indexes corresponding to the largest-magnitude entries in AT b}. 2. r0 = b − AΩ0 A†Ω0 b. repeat ˜ k = Ωk−1 & {K| indexes corresponding to the largest magnitude entries in the 1. Ω vector ATΩk−1 rk−1 }. 2. Compute xp = A†Ω˜ b. k

3. Ωk = {K| indexes corresponding to the largest magnitude entries in xp }. 4. rk = b − AΩk A†Ωk b. 5. exit if rk 2 ≤ rk−1 2 . 6. Let Ωk = Ωk−1 . return k ← k + 1. output: x = xp .

index set at the (k − 1)th iteration Ωk−1 . Once a candidate is selected, it will remain in the list of selected columns until the algorithm ends. Unlike these methods, for a K-sparse signal, Algorithm 6.9 keeps the index set of K candidate columns unchanged and allows the candidate column to be continuously updated during the iteration process.

394

Solving Matrix Equations

6.9.2 First-Order Augmented Lagrangian Algorithm In practice, solving the LP problem (P1 ) is hard, because the matrix A is large and dense and an LP problem is often ill-conditioned. In typical compressed sensing (CS) applications, the dimension of A is large (n ≈ 106 ), hence general interiorpoint methods are not practical for solving (P1 ) in CS applications owing to the need to factorize an m × n matrix A. However, the LP problem (P1 ) can be efficiently solved in the Lagrangian form (see e.g., [154], [194], [20]) " # 1 2 (6.9.1) min λ x 1 + Ax − b 2 x∈Rn 2 with penalty parameter λ , 0. An augmented Lagrange function for the above LP problem is given by [20] 1 J(x) = λ x 1 − λθ T (Ax − b) + Ax − b 22 , 2

(6.9.2)

where θ is a Lagrange multiplier vector for the constraints Ax = b. Algorithm 6.10 shows the first-order augmented Lagrangian (FAL) algorithm. This algorithm solves the LP problem by inexactly solving a sequence of optimization problems of the form " # 1 T 2 min λk x 1 − λk θk (Ax − b) + Ax − b 2 , (6.9.3) 2 x∈Rn | x 1 ≤ηk for an appropriately chosen sequence of {λk , θk , ηk }, k = 0, 1, . . . The APG stopping criterion APGstop is given by step 8 in Algorithm 6.10, and the FAL stopping criterion FALstop is given by 2 2  FALstop = 2u(l) − u(l−1) 2∞ ≤ γ (6.9.4) for noiseless measurements or

2 (l) 2 2u − u(l−1) 2 2 2 2 ≤γ FALstop = 2u(l−1) 2 2

(6.9.5)

for noisy measurements, where γ is the threshold value. Algorithm FAL produces xsol = u(l) when FALstop is true. The APG function used in step 9 in Algorithm 6.10 is given by Algorithm 6.11.

6.9.3 Barzilai–Borwein Gradient Projection Algorithm Consider the standard bound-constrained quadratic program (BCQP) problem " # 1 T T min q(x) = x Ax − b x subject to v ≤ x ≤ u. (6.9.6) z 2

6.9 Sparse Matrix Equation Solving: Optimization Algorithms Algorithm 6.10

395

FAL ({λk , k , τk }k∈Z+ , η) [20]

  task: Find x = argmin λk x1 − λk θkT (Ax − b) + 12 Ax − b22 . x

input: x0 , L = σmax (AAT ). initialization: θ1 = 0, k = 0. 1. while (FALSTOP not true) do 2.

k ← k + 1.

3.

pk = λk x1 , fk (x) = 12 Ax − b − λk θk 22 .

4.

hk (x) = 12 x − xk−1 22 .

5.

ηk ← η + (λk /2)θk 22 .

6.

Fk = {x ∈ Rn | x1 ≤ ηk }.

7.

k,max ← σmax (A)(ηk + xk−1 2 )

8.

APGSTOP = { ≥ k,max } or {∃ g ∈ ∂Pk (x)|v with g2 ≤ τk }.

9.

xk ← APG (pk , fk , L, Fk , xk−1 , hk , APGSTOP).

10.



2/k .

θk+1 ← θk − (Axk − b)/λk .

11. end while 12. output: x ← xk .

Algorithm 6.11

APG (pk , fk , L, Fk , xk−1 , hk , APGSTOP) [20]

initialization: u0 = x0 , w0 ← argminx∈p h(x), θ0 ← 1, l ← 0. 1. while (APGSTOP not true) do

3.

vl ← (1 − θl )ul + θl wl .      l −1 wl+1 ← argmin p(z) + [∇f (vl )]T z + (L/c)h(z)z ∈ F . i=0 θi

4.

ˆ l+1 ← (1 − θl )ul + θl wl+1 . u

5.

7.

Hl (x) ← p(x) + [∇f (vl )]T x + x − vl 22 . 2  ul+1 ← argmin{Hl (x)x ∈ F }.

1  4 θl+1 ← θl − 4θl2 − θl2 .

8.

l ← l + 1.

2.

6.

L

2

9. end while 10. return ul or vl depending on APGSTOP.

Gradient projection (GP) or projected gradient methods provide an alternative way of solving large-scale BCQP problems.

396

Solving Matrix Equations

Let Ω be the feasible set of (6.9.6) such that    Ω = x ∈ R n v ≤ x ≤ u ,

(6.9.7)

and let P denote the projection operator onto Ω, that is P(x) = mid(v, x, u),

(6.9.8)

where mid(v, x, u) is the vector whose ith component is the median of the set {vi , xi , ui }. Suppose that a feasible point xk is generated, the gradient projection method computes the next point as [114] xk+1 = P(xk − αk gk ),

(6.9.9)

where αk > 0 is some step size and gk = Axk − b. Barzilai and Borwein [26] proposed a method for greatly improving the effectiveness of gradient projection methods by appropriate choices of the step size αk . Two useful choices of the step size were given in [26]: αkBB1 =

sTk−1 sk−1 , sTk−1 yk−1

αkBB2 =

sTk−1 yk−1 , T y yk−1 k−1

(6.9.10)

where sk−1 = xk − xk−1 and yk−1 = gk − gk−1 . Gradient projection using the Barzilai–Borwein step size is called the Barzilai– Borwein gradient projection (BBGP) method. One of its features is that it is nonmonotonic, i.e., q(xk ) may increase on some iterations. Nevertheless, the BBGP method is simple and easy to implement and avoids any matrix factorization. Interestingly, the convex unconstrained optimization problem " # 1 (6.9.11)

b − Ax 22 + τ x 1 , x ∈ Rn , A ∈ Rm×n , b ∈ Rm min x 2 can be transformed into a bound-constrained quadratic program (BCQP) problem. To this end, define ui = (xi )+ and vi = (−xi )+ for all i = 1, . . . , n, where (x)+ = max{x, 0} denotes the positive-part operator. Then, the coefficient vector x can be split into a difference between nonnegative vectors: x = u − v,

u ≥ 0,

v ≥ 0.

(6.9.12)

The above equation can also be written as v ≤ x ≤ u, where, for vectors a, b, a ≤ b means ai ≤ bi , ∀ i. Obviously, x 1 = 1Tn u + 1Tn v, where 1n = [1, . . . , 1]T is an n × 1 summing vector consisting of n ones. By setting x = u − v, the convex unconstrained optimization problem (6.9.11) can be rewritten as the following BCQP problem [154]: " # 1 T T min c z + z Bz ≡ F (z) subject to z ≥ 0. (6.9.13) z 2

6.9 Sparse Matrix Equation Solving: Optimization Algorithms

397

Here   u z= , v



 −AT b , c = τ 12n + AT b



 AT A −AT A B= . −AT A AT A

(6.9.14)

Algorithm 6.12 shows the GPSR-BB algorithm (Barzilai–Borwein gradient projection for sparse reconstruction), which was proposed in [154]. Algorithm 6.12 GPSR-BB algorithm [154]     task: Solve min 12 b − Ax22 + τ x1 or min cT z + 12 zT Bz ≡ F (z) . x

z≥0

input: B and F (z). initialization: Given z0 , choose αmin , αmax , α0 ∈ [αmin , αmax ], and set k = 0. repeat 1. Compute δk = (zk − αk ∇F (zk ))+ − zk . 2. Compute γk = δkT Bδk . 3. Line search: If γk = 0, let λk = 1, otherwise   λk = mid 0, δkT ∇F (zk )/γk , 1 . 4. Update zk+1 = zk + λk δk . 5. exit if zk+1 − (zk+1 − α∇F ¯ (zk+1 ))+ 2 ≤ tol, where tol is a small parameter and α ¯ is a positive constant. 6. if γk = 0, let αk+1 = αmax , otherwise   αk+1 = mid αmin , δk 22 /γk , αmax . 7. return k ← k + 1. output: z = zk+1 .

6.9.4 ADMM Algorithms for Lasso Problems The alternating direction method of multipliers (ADMM) form of the Lasso problem  min 12 Ax − b 22 + λ x 1 is given by " # 1 2 min subject to x − z = 0. (6.9.15)

Ax − b 2 + λ z 1 2 The corresponding ADMM algorithm is [52]: xk+1 = (AT A + ρI)−1 (AT b + ρzk − yk ), k+1

= Sλ/ρ (x

k+1

k

z y

k+1

= y + ρ(x

k

+ y /ρ),

k+1

−z

k+1

(6.9.16) (6.9.17)

),

(6.9.18)

398

Solving Matrix Equations

where the soft thresholding operator S is defined as ⎧ ⎪ ⎨ x − λ/ρ, x > λ/ρ, Sλ/ρ (x) = 0, |x| ≤ λ/ρ, ⎪ ⎩ x + λ/ρ, x < λ/ρ. The Lasso problem can be generalized to " # 1 min

Ax − b 22 + λ Fx 1 , 2

(6.9.19)

(6.9.20)

where F is an arbitrary linear transformation. The above problem is called the generalized Lasso problem. The ADMM form of the generalized Lasso problem can be written as " # 1 min subject to Fx − z = 0. (6.9.21)

Ax − b 22 + λ z 1 2 The corresponding ADMM algorithm can be described as [52] xk+1 = (AT A + ρFT F)−1 (AT b + ρFT zk − yk ), zk+1 = Sλ/ρ (Fx y

k+1

k

k+1

= y + ρ(Fx

+ yk /ρ),

k+1

−z

k+1

(6.9.22) (6.9.23)

).

Consider the group Lasso problem [537] + 0 n  1 2 min

xi 2

Ax − b 2 + λ 2 i=1

(6.9.24)

(6.9.25)

N with feature groups Ax = i=1 Ai xi , where Ai ∈ Rm×ni , xi ∈ Rni , i = 1, . . . , N N and i=1 ni = n. Another difference from the generalized Lasso is the Euclideannorm (not squared) regularization of x. The group Lasso is also called the sum-ofnorms regularization [356]. N Put Axk = N −1 i=1 Ai xki . The ADMM algorithm for the group Lasso is given by [52] 3ρ 4 k k k k 2 ¯ xk+1 (6.9.26) = arg min x − A x − z + Ax + y

+ λ x

A i i i i i 2 , 2 i 2 xi 1 ¯k+1 = (6.9.27) (b + ρAxk+1 + ρyk ), z N +ρ ¯k+1 ). yk+1 = yk + Axk+1 − z (6.9.28) 6.9.5 LARS Algorithms for Lasso Problems An efficient approach for solving Lasso problems is the least angle regressive (LARS) algorithm [143]. Algorithm 6.13 shows the LARS algorithm with Lasso modification.

6.9 Sparse Matrix Equation Solving: Optimization Algorithms Algorithm 6.13

399

LARS algorithm with Lasso modification [143]

given: The data vector b ∈ Rm and the input matrix A ∈ Rm×n . ˆ = 0 and AΩ = A. Put k = 1. initialization: Ω0 = ∅, b 0

repeat ˆ k−1 ). ˆk = ATΩk−1 (b − b 1. Compute the correlation vector c

ck (j)| = C} with 2. Update the active set Ωk = Ωk−1 ∪ {j (k) | |ˆ ck (n)|}. C = max{|ˆ ck (1)|, . . . , |ˆ ck (j)). 3. Update the input matrix AΩk = [sj aj , j ∈ Ωk ], where sj = sign(ˆ 4. Find the direction of the current minimum angle GΩk = ATΩk AΩk ∈ Rk×k , αΩk = (1Tk GΩk 1k )−1/2 ,

k wΩk = αΩk G−1 Ωk 1k ∈ R ,

μk = A Ω k w Ω k ∈ R k . 5. Compute b = ATΩk μk = [b1 , . . . , bm ]T and estimate the coefficient vector T ˆ k = (ATΩk AΩk )−1 ATΩk = G−1 x Ωk AΩk .  +  x C − cˆk (j) C + cˆk (j) , , γ˜ = min − wj 6. Compute γˆ = minc j∈Ωk

αΩ − b j k

αΩ + bj k

j∈Ωk

j

+

, where wj is the jth

of entry wΩk = [w1 , . . . , wn ]T and min{·}+ denotes the positive minimum term. If

there is not positive term then min{·}+ = ∞.

7. If γ˜ < γˆ then the fitted vector and the active set are modified as follows: ˆk = b ˆ k−1 + γ˜ μk , Ωk = Ωk − {˜j}, b where the removed index ˜j is the index j ∈ Ωk such that γ˜ is a minimum. ˆ k and Ωk are modified as follows: Conversely, if γˆ < γ˜ then b ˆk = b ˆ k−1 + γˆ μk , Ωk = Ωk ∪ {ˆj}, b where the added index ˆj is the index j ∈ Ωk such that γˆ is a minimum. 8. exit If some stopping criterion is satisfied. 9. return k ← k + 1. ˆk. output: The coefficient vector x = x

The LARS algorithm is a stepwise regression method, and its basic idea is this: while ensuring that the current residual is equal to the selected correlation coefficient, select as the solution path the projection of the current residual onto the space constituted by the selected variables. Then, continue to search on this solution path, absorbing the new added variable, and adjusting the solution path. Suppose that Ωk is the active set of variables at the beginning of the kth iteration step, and let xΩk be the coefficient vector for these variables at this step. Letting the initial active set Ω0 = ∅, the initial value of the residual vector is

400

Solving Matrix Equations

given by r = b. At the first iteration find the column (i.e., the predictor) of the (1) input matrix A that is correlated with the residual vector r = b; denote it ai , and enter its index in the active set Ω1 . Thus, one obtains the first regression-variable set AΩ1 . The basic step of the LARS algorithm is as follows: suppose that after k−1 LARS steps the regression variable set is AΩk−1 . Then, one has the vector expression  −1/2 T wΩk−1 = 1TΩk−1 (ATΩk−1 AΩk−1 )−1 1Ωk−1 (AΩk−1 AΩk−1 )−1 1Ωk−1 .

(6.9.29)

Hence AΩk−1 wΩk−1 is the solution path of the LARS algorithm in the current regression variable set Ωk−1 , and wΩk−1 is the path for which x continues to search. The core idea of the LARS algorithm with Lasso modification, as shown in Algorithm 6.13, is “one at a time”: at the every iteration step, one must add or remove a variable. Specifically, from the existing regression variable set and the current residuals, a solution path is determined. The maximum possible step forward on this path is denoted by γˆ and the maximum step needed to find a new variable is denoted by γ˜ . If γ˜ < γˆ then the new variable xj (γ) corresponding to the LARS estimate is not a Lasso estimate, and hence this variable should be deleted from the regression variable set. Conversely, if γ˜ > γˆ then the new estimate xj (γ) corresponding to the LARS estimate should become a new Lasso estimate, and hence this variable should be added to the regression set. After deleting or adding a variable, one should stop searching on the original solution path. Then, one recomputes the correlation coefficient of the current residual and current new variable set to determine a new solution path, and continues the “one at a time” form of the LARS iteration steps. This process is repeated until all Lasso estimates are obtained by using the LARS algorithm.

6.9.6 Covariance Graphical Lasso Method The estimation of a covariance matrix via a sample of vectors drawn from a multivariate Gaussian distribution is among the most fundamental problems in statistics [48]. Suppose n samples of p normal random vectors are given by x1 , . . . , xn ∼ Np (0, Σ). The log likelihood is [48] L(Σ) = −

n np n − log det Σ − tr(Σ−1 S), 2 2 2

n where S = n−1 i=1 xi xTi is the sample covariance matrix. This approximates the true covariance matrix Σ = [σij ] with σij = E{xTi xj }. Bien and Tibshirani [48] in 2011 proposed a covariance graphical Lasso method using a Lasso penalty on the elements of the covariance matrix. The basic version of the covariance graphical Lasso problem minimizes the ob-

6.9 Sparse Matrix Equation Solving: Optimization Algorithms

401

jective function [48], [498]:   min g(Σ) = min ρ Σ 1 + log(det Σ) + tr(SΣ−1 ) Σ

Σ

(6.9.30)

over the space of positive definite matrices M+ with shrinkage parameter ρ ≥ 0. Then, covariance graphical Lasso estimation is used to solve the optimization problem (6.9.30). The covariance graphical Lasso method is particularly useful owing to the following two facts [498]: (1) by using the 1 -norm term, the covariance graphical Lasso is able to set some offdiagonal elements of Σ exactly equal to zero at the minimum point of (6.9.30); (2) the zeros in Σ encode the marginal independence structures among the components of a multivariate normal random vector with covariance matrix Σ. However, the objective function is not convex, which makes the optimization challenging. To address this challenge, Bien and Tibshirani [48] proposed a majorize– minimize approach to minimize (6.9.30) approximately, and Wang [498] in 2014 proposed a coordinate descent algorithm for covariance graphical Lasso. In comparison with the majorize–minimize algorithm of Bien and Tibshirani, Wang’s coordinate descent algorithm is simpler to implement, substantially faster to run and numerically more stable as shown in experimental results. The idea of coordinate descent is simple: to update one column and one row of Σ at a time while holding all the rest of the elements in Σ fixed. For example, to update the last column and row, Σ and S are partitioned as follows:     Σ11 σ12 S11 s12 Σ= , S = , (6.9.31) T σ12 σ22 sT12 s22 where Σ11 and S11 are respectively the covariance matrix and the sample covariance matrix of the first p − 1 variables, and σ12 and s12 are respectively the covariance and the sample covariance between the first p − 1 variables and the last variable while σ22 and s22 respectively are the variance and the sample variance of the last variable. T Σ−1 Letting β = σ12 and γ = σ22 − σ12 11 σ12 , we have   −1 T −1 −1 −1 −1 T −1 + Σ ββ Σ γ −Σ β γ Σ 11 11 11 11 Σ−1 = . (6.9.32) −1 −β T Σ−1 γ −1 11 γ Therefore, the three terms in (6.9.30) can be represented as follows [498]: β Σ 1 = 2ρ β 1 + ρ(β T Σ−1 11 β + γ) + c1 , log(det Σ) = log γ + c2 , −1 −1 −1 tr(SΣ−1 ) = β T Σ−1 − 2sT12 Σ−1 + s22 γ −1 + c3 . 11 SΣ11 βγ 11 βγ

402

Solving Matrix Equations

Dropping the constants c1 , c2 , c3 , we get the objection function in (6.9.30): −1 T −1 −1 f (β, γ) = 2ρ β 1 + ρ(β T Σ−1 11 β + γ) + β Σ11 SΣ11 βγ −1 − 2sT12 Σ−1 + s22 γ −1 + log γ. 11 βγ

(6.9.33)

This objective function of the covariance graphical Lasso method can be written as two separated objective functions f1 (γ) = log γ + aγ −1 + ργ,

(6.9.34)

f2 (β) = 2ρ β 1 + β Vβ − 2u β, T

T

(6.9.35)

−1 −1 −1 T −1 −1 where a = β T Σ−1 Σ11 SΣ−1 11 SΣ11 β − 2s12 Σ11 β + s22 , V = γ 11 + ρΣ11 and u = −1 −1 γ Σ11 s12 . From (6.9.34) it can be seen that " a, ρ = 0, (6.9.36) γ = arg min f1 (γ) =    γ − 1 + 1 + 4aρ /(2ρ), ρ = 0.

Equation (6.9.35) shows that f2 (β) is still a Lasso problem and can be efficiently solved by coordinate descent algorithms. For j ∈ {1, . . . , p − 1}, the minimum point of (6.9.35) along the coordinate direction in which βj varies is given by  > p−1  v βˆ , ρ v , j = 1, . . . , p − 1 (6.9.37) βˆ = soft u − j

j

kj k

jj

k=1,k=j

where soft(x, ρ) = sign(x)(|x| − ρ)+ is the soft thresholding operator. A coordinate descent algorithm for covariance graphical Lasso is proposed in [498] based on the above results, as shown in Algorithm 6.14.

6.9.7 Homotopy Algorithm In topology, the concept of homotopy describes a “continuous change” between two objects. The homotopy algorithm is a kind of searching algorithm that starts from a simple solution and uses iterative calculation to find the desired complex solution. Therefore, the key of a homotopy algorithm is to determine its initial simple solution appropriately. Consider the relationship between the 1 -norm minimization problem (P1 ) and the unconstrained 2 -minimization problem (P12 ). Suppose that there is a corresponding unique solution xλ for every minimization problem (P12 ) with λ ∈ [0, ∞). Then the set {xλ |λ ∈ [0, ∞)} determines a solution path and has xλ = 0 for a ˜ λ of (P12 ) converges to the sufficiently large λ; and when λ → 0, the solution x solution of the 1 -norm minimization problem (P1 ). Hence, xλ = 0 is the initial solution of the homotopy algorithm for solving the minimization problem (P1 ). The homotopy algorithm for solving the unconstrained 2 -norm minimization

6.9 Sparse Matrix Equation Solving: Optimization Algorithms Algorithm 6.14

403

Coordinate descent algorithm [498]

task: Find Σ = argmin{ρΣ1 + log(det Σ) + tr(SΣ−1 )}. Σ

input: S = Y T Y, ρ.





initialization: Given Σ0 = S and partition S = ⎣

S11

s12

sT12

s22

repeat





1. Σk+1 = Σk , and Partition Σk+1 = ⎣

Σ11

σ12

T σ12

σ22

⎦. Put k = 0.

⎦.

2. Put β = σ12 . −1 T SΣ−1 3. a = β T Σ−1 11 β − 2s12 Σ11 β + s22 .  11 a, ρ = 0,  4. γ = (−1 + 1 + 4aρ)/(2ρ), ρ = 0. −1 −1 5. V = γ −1 Σ−1 11 SΣ11 + ρΣ11 .

6. u = γ −1 Σ−1 11 s12 .

'  vjj , j = 1, . . . , p. 7. Compute βˆj = soft uj − k=j vkj βˆk , ρ ⎡ ⎤−1 Σ−1 + Σ−1 ββ T Σ−1 γ −1 −Σ−1 βγ −1 11 11 11 11 ⎦ . 8. Calculate Σk = ⎣ −1 −β T Σ−1 γ −1 11 γ 9. exit if Σk+1 − Σk 2 ≤ . 10. return k ← k + 1. output: Σk .

problem (P12 ) starts from the initial value x0 = 0, and runs in an iteration form to compute the solutions xk at the kth step for k = 1, 2, . . . In the whole calculation, the following active set is kept unchanged:    (6.9.38) I = j  |ck (j)| = ck ∞ = λ . Algorithm 6.15 shows the homotopy algorithm for solving the 1 -norm minimization problem. With the reduction of λ, the objective function of (P11 ) will go through a homotopy process from the 1 -norm constraint to the 2 -norm objective function. It has been shown [143], [363], [132] that the homotopy algorithm is an efficient solver for the 1 -norm minimization problem (P1 ).

6.9.8 Bregman Iteration Algorithms The sparse optimization models for solving the matrix equation Au = b discussed above can be summarized as follows:

404

Solving Matrix Equations

Algorithm 6.15

Homotopy algorithm [132]

input: Observed vector b ∈ Rm , input matrix A and parameter λ. initialization: x0 = 0, c0 = AT b. repeat 1. Use (6.9.38) to form the active set I, and put ck (I) = [ck (i), i ∈ I] and AI = [ai , i ∈ I]. 2. Compute the residual correlation vector ck (I) = ATI (b − AI xk ). 3. Solve the matrix equation ATI AI dk (I) = sign(ck (I)) for dk (I). 4. Compute



 λ − ck (i) λ + ck (i) , , i∈I 1 − φTi vk 1 + φTi vk   x (i) . γk− = min − k i∈I dk (i) 5. Determine the breakpoint γk = min{γk+ , γk− }. γk+ = minc +

6. Update the solution vector xk = xk−1 + γk dk . 7. exit if xk ∞ = 0. 8. return k ← k + 1. output: x = xk .

(1) 0 norm minimization model min u 0 subject to Au = b; u

(2) basis-pursuit (BP)/compressed sensing model min u 1 subject to Au = b; u

(3) basis-pursuit de-noising model min u 1 subject to Au − b 2 < ; u

(4) Lasso model min Au − b 2 subject to u 1 ≤ s. u

Among these models, the BP model is a relaxed form of the 0 -norm minimization model, and the Lasso model is a linear prediction representation equivalent to the basis pursuit de-noising model. Now, consider the general form of the above four optimization models: uk+1 = arg min {J(u) + λH(u)} , u

(6.9.39)

where J : X → R and H : X → R are nonnegative convex functions (X is a closed convex set), but J is a nonsmooth, while H is differentiable. Figuratively speaking, a function is of the bounded-variation type if the oscillation of its graphics (i.e. its swing or variation) is manageable or “tamed” in a certain range. The bounded-variation norm of a vector u, denoted u BV , is defined as [8]  |∇u|dx, (6.9.40)

u BV = u 1 + J0 (u) = u 1 + Ω

6.9 Sparse Matrix Equation Solving: Optimization Algorithms

405

where J0 (u) denotes the total variation of u. Let J(u) = u BV and H(u) = 12 u − f 22 . Then the optimization model (6.9.39) becomes a total variation or Rudin–Osher–Fatemi (ROF) de-noising model [420]: " # λ (6.9.41) uk+1 = arg min u BV + u − f 22 . 2 u A well-known iteration method for solving the optimization problem (6.9.39) is Bregman iteration, based on the Bregman distance [59]. DEFINITION 6.10 Let J(u) be a convex function, let the vectors u, v ∈ X and let g ∈ ∂J(v) be the subgradient vector of the function J at the point v. The Bregman distance between the points u and v, denoted DJg (u, v), is defined as DJg (u, v) = J(u) − J(v) − g, u − v.

(6.9.42)

The Bregman distance is not a distance in traditional sense, because DJg (u, v) = However, it has the following properties that make it an efficient tool for solving the 1 -norm regularization problem: DJg (v, u).

(1) For all u, v ∈ X and g ∈ ∂J(v), the Bregman distance is nonnegative, i.e., DJg (u, v) ≥ 0. (2) The Bregman distance for the same point is zero, i.e., DJg (v, v) = 0. (3) The Bregman distance can measure the closeness of two points u and v since DJg (u, v) ≥ DJg (w, v) for any point w on the line connecting u and v. Consider the first-order Taylor series approximation of the nonsmooth function J(u) at the kth iteration point uk , denoted J(u) = J(uk ) + gk , u − uk . The approximate error is measured by the Bregman distance DJgk (u, uk ) = J(u) − J(uk ) − gk , u − uk .

(6.9.43)

Early in 1965, Bregman proposed that the unconstrained optimization problem (6.9.39) can be modified as [59]   g uk+1 = arg min DJk (u) + λH(u) u

= arg min {J(u) − gk , u − uk  + λH(u)} . u

(6.9.44)

This is the well-known Bregman iteration. In what follows we give the Bregman iteration algorithm and its generalization. The objective function of the Bregman iterative optimization problem is denoted by L(u) = J(u)−gk , u−uk +λH(u). By the stationary-point condition 0 ∈ ∂L(u) it follows that 0 ∈ ∂J(u) − gk + λ∇H(u). Hence, at the (k + 1)th iterative point uk+1 one has gk+1 = gk − λ∇H(uk+1 ),

gk+1 ∈ ∂J(uk+1 ).

(6.9.45)

406

Solving Matrix Equations

Equations (6.9.44) and (6.9.45) constitute the Bregman iterative algorithm, which was proposed by Osher et al. in 2005 [364] for image processing. THEOREM 6.14 [364] Suppose that J and H are convex functions and that H is differentiable. If a solution to Equation (6.9.44) exists then the following convergence results are true. • The function H is monotonically decreasing in the whole iterative process, namely H(uk+1 ) ≤ H(uk ). • The function H will converge to the optimal solution H(u ), since H(uk ) ≤ H(u ) + J(u )/k. The Bregman iterative algorithm has two versions [532]: Version 1 k = 0, u0 = 0, g0 = 0. Iterations: " # 1 g uk+1 = arg min DJk (u, uk ) + Au − b 22 , 2 u gk+1 = gk − AT (Auk+1 − b). If uk does not converge then let k ← k + 1 and return to the iteration. Version 2 k = 0, b0 = 0, u0 = 0. Iterations: bk+1 = b + (bk − Auk ). " # 1 uk+1 = arg min J(u) + Au − bk+1 22 . 2 u If uk does not converge then let k ← k + 1 and return to the iteration. It has been shown [532] that the above two versions are equivalent. The Bregman iterative algorithm provides an efficient tool for optimization, but, g at every step, one must minimize the objective function DJk (u, uk )+H(u). In order to improve the computational efficiency of the Bregman iterative algorithm, Yin et al. [532] proposed a linearized Bregman iterative algorithm. The basic idea of the linearized Bregman iteration is as follows: for the Bregman iteration, use a first-order Taylor series expansion to linearize the nonlinear function H(u) to H(u) = H(uk )+∇H(uk ), u−uk  at the point uk . Then, the optimization problem (6.9.39) with λ = 1 becomes   g uk+1 = arg min DJk (u, uk ) + H(uk ) + ∇H(uk ), u − uk  . u

Note that the first-order Taylor series expansion is exact only for the neighborhood of u at the point uk , and the additive constant term H(uk ) can be made negative

6.9 Sparse Matrix Equation Solving: Optimization Algorithms

407

in the optimization with respect to u, so a more exact expression of the above optimization problem is " # 1 gk 2 uk+1 = arg min DJ (u, uk ) + ∇H(uk ), u − uk  + u − uk 2 . (6.9.46) 2δ u Importantly, the above equation can be equivalently written as " # 1 g uk+1 = arg min DJk (u, uk ) + u − (uk − δ∇H(uk )) 22 , 2δ u

(6.9.47)

because (6.9.46) and (6.9.47) differ by only a constant term that is independent of u. In particular, if H(u) = 12 Au − b 22 then from ∇H(u) = AT (Au − b) one can write (6.9.47) as " #  2 1 2 g 2u − uk − δAT (Auk − b) 22 . uk+1 = arg min DJk (u, uk ) + (6.9.48) 2 2δ u Consider the objective function in (6.9.48) as L(u) = J(u) − J(uk ) − gk , u − uk  +

 2 1 2 2u − uk − δAT (Auk − b) 22 . 2 2δ

By the subdifferential stationary-point condition 0 ∈ ∂L(u) it is known that *  1) 0 ∈ ∂J(u) − gk + u − uk − δAT (Auk − b) . δ Denoting gk+1 ∈ ∂J(uk+1 ), from the above equation one has [532] gk+1 = gk − AT (Auk − b) − =

k 

AT (b − Aui ) −

i=1

uk+1 − uk = ··· δ

uk+1 . δ

(6.9.49)

Letting vk =

k 

AT (b − Aui ),

(6.9.50)

i=1

one can obtain two important iterative formulas. First, from (6.9.49) and (6.9.50) we get the update formula of the variable u at the kth iteration: uk+1 = δ(vk − gk+1 ).

(6.9.51)

Then, Equation (6.9.50) directly yields an iterative formula for the intermediate variable vk : vk+1 = vk + AT (b − Auk+1 ).

(6.9.52)

408

Solving Matrix Equations

Equations (6.9.51) and (6.9.52) constitute the linearized Bregman iterative algorithm [532] for solving the optimization problem " # 1 2 (6.9.53) min J(u) + Au − b 2 . u 2 In particular, for the optimization problem " # 1 min μ u 1 + Au − b 22 , u 2 we have

⎧ ⎪ ⎪ ⎨{+1}, ∂( u 1 )i = [−1, +1], ⎪ ⎪ ⎩{−1},

(6.9.54)

if ui > 0, (6.9.55)

if ui = 0, if ui < 0.

Hence Equation (6.9.51) can be written in the component form as uk+1 (i) = δ(vk (i) − gk+1 (i)) = δ shrink(vk (i), μ),

i = 1, . . . , n,

(6.9.56)

where ⎧ ⎪ ⎪ ⎨y − α, shrink(y, α) = sign(y) max{|y| − α, 0} = 0, ⎪ ⎪ ⎩y + α,

y ∈ (α, ∞), y ∈ [−α, α], y ∈ (−∞, −α)

is the shrink operator. The above results can be summarized as the linearized Bregman iterative algorithm for solving the basis-pursuit de-noising or total variation de-noising problem [532], see Algorithm 6.16. Consider the optimization problem uk+1 = arg min { Au 1 + H(u)} . u

(6.9.57)

Introducing the intermediate variable z = Au, the unconstrained optimization problem (6.9.39) can be written as the constrained optimization (uk+1 , zk+1 ) = arg min { z 1 + H(u)} u,z

subject to z = Au.

(6.9.58)

By adding an 2 -norm penalty term, this constrained optimization problem becomes the unconstrained optimization problem " # λ 2 (6.9.59) (uk+1 , zk+1 ) = arg min z 1 + H(u) + z − Au 2 . 2 u,z

Exercises

409

Linearized Bregman iterative algorithm [532]

Algorithm 6.16

  task: Solve basis-pursuit/total variation de-noising problem min μu1 + 12 Au − b22 . u

input: Input matrix A ∈ Rm×n and observed vector b ∈ Rm . initialization: k = 0, u0 = 0, v0 = 0. repeat 1. for i = 1, . . . , n do 2. uk+1 (i) = δ shrink(vk (i), μ). 3. end for 4. vk+1 = vk + AT (b − Auk+1 ). 5. exit if uk+1 − uk 2 ≤ . return k ← k + 1. output: uk .

The split Bregman iterative algorithm is given by [177] " # λ uk+1 = arg min H(u) + zk − Au − bk 22 , 2 u " # λ zk+1 = arg min z 1 + z − Auk+1 − bk 22 , 2 z bk+1 = bk + [Auk+1 − zk+1 ].

(6.9.60) (6.9.61) (6.9.62)

The three iterations of the split Bregman iteration algorithm have the following characteristics. (1) The first iteration is a differentiable optimization problem that can be solved by the Gauss–Seidel method. (2) The second iteration can be solved efficiently using the shrink operator. (3) The third iteration is a explicit calculation.

Exercises 6.1

Consider the matrix equation Ax + = x, where is an additive color noise vector satisfying the conditions E{ } = 0 and E{ T } = R. Let R be known, and use the weighting error function Q(x) = T W as the cost function for ˆ WLS . Such a method is called the weighting finding the optimal estimate x least squares method. Show that ˆ WLS = (AT WA)−1 AT Wx, x where the optimal choice of the weighting matrix W is Wopt = R−1 .

410

6.2

Solving Matrix Equations

Given an over-determined linear equation ZTt Xt = ZTt Yt x with Zt ∈ R(t+1)×K is known as the instrumental variable matrix and t + 1 > K. ˆ , find an (a) Letting the estimate of the parameter vector x at time t be x expression for it. This is called the instrumental variable method. (b) Setting       Yt Zt Xt Yt+1 = , Zt+1 = , Xt+1 = , bt+1 zt+1 xt+1 find a recursive computation formula for xt+1 .

6.3

6.4

Consider the matrix equation y = Aθ + e, where e is an error vector. Define the weighting-error squared sum Ew = eH We, where the weighting matrix W is Hermitian positive definite. Find a solution such that Ew is minimized. Such a solution is called the weighted least squares (WLS) solution. Let λ > 0, and let Ax = b is an over-determined matrix equation. Show that the anti-Tikhonov regularized optimization problem # " 1 1 min

Ax − b 22 − λ x 22 2 2

6.5

has optimal solution x = (AH A − λI)−1 AH b. [179] The TLS solution of the linear equation Ax = b can be also expressed as min

b+e∈Range(A+E)

D[E, e]T F ,

E ∈ Rm×n ,

e ∈ Rm ,

where D = Diag(d1 , . . . , dm ) and T = Diag(t1 , . . . , tn+1 ) are nonsingular. Show the following results. (a) If rank(A) < n then the above TLS problem has one solution if and only if b ∈ Range(A). (b) If rank(A) = n, AT D2 b = 0, |tn+1 | Db 2 ≥ σn (DAT1 ), where T1 = Diag(t1 , . . . , tn ) then the TLS problem has no solution. Here σn (C) denotes the nth singular value of the matrix C. 6.6

6.7 6.8

Consider the TLS in the previous problem. Show that if C = D[A, b]T = [A1 , d] and σn (C) > σn+1 (C) then the TLS solution satisfies (AT1 A1 − 2 σn+1 (C)I)x = AT1 d. Given the data points (1, 3), (3, 1), (5, 7), (4, 6), (7, 4), find respectively TLS and LS linear fittings and analyze their squared sum of distances. Prove that the solution of the optimization problem   min tr(AT A) − 2tr(A) subject to AX = O A

ˆ = I − XX† , where O is a null matrix. is given by A

Exercises

6.9

411

Consider the harmonic recovery problem in additive white noise x(n) =

p 

Ai sin(2π fi n + φi ) + e(n),

i=1

where Ai , fi and φi are the amplitude, frequency and phase of the ith harmonic, and e(n) is the additive white noise. The above harmonic process obeys the modified Yule–Walker (MYW) equation Rx (k) +

2p 

ai Rx (k − i) = 0,

∀ k,

i=1

and the harmonic frequency can be recovered from   1 Imzi , i = 1, 2, . . . , p, fi = arctan 2π Rezi where zi is a root in the conjugate root pair (zi , zi∗ ) of the characteristic 2p polynomial A(z) = 1 + i=1 ai z −i . If √ √ x(n) = 20 sin(2π 0.2n) + 2 sin(2π0.213n) + e(n), where e(n) is a Gaussian white noise with mean 0 and variance 1, take n = 1, 2, . . . , 128 and use the LS method and SVD-TLS algorithm to estimate the AR parameters ai of the ARMA model and the harmonic frequencies f1 and f2 . Assume that there are 40 MYW equations, and that p = 2 and p = 3 respectively in the LS method, while the number of unknown parameters is 14 in the TLS algorithm. When using the TLS algorithm, the number of harmonics is determined by the number of leading singular values and then the roots of the characteristic polynomial are computed. Compare the computer simulation results of the LS method and the SVD-TLS algorithm. 6.10 The TLS problem for solving the matrix equation Ax = b can be equivalently represented as min

b+Δb∈Range(A+ΔA)

C[ΔA, Δb]T 2F ,

ΔA ∈ Rm×n ,

Δb ∈ Rm ,

where C = Diag(c1 , . . . , dm ) and T = Diag(t1 , . . . , tn+1 ). Show the following results. (a) If rank(A) < n then the above TLS problem has a solution if and only if b ∈ Range(A). (b) If rank(A) = n, AT Cb = 0 and |tn+1 | Cb 2 ≥ σn (CAT1 ) with T1 = Diag(t1 , . . . , tn ) then the TLS problem has no solutions, σn (CAT1 ) is the nth singular value of the matrix CAT1 .

412

Solving Matrix Equations

6.11 Show the nonnegativity of the KL divergence  I  K   pij DKL (P Q) = − pij + qij pij log qij i=1 j=1 and show that the KL divergence is equal to zero if and only if P = Q. 6.12 Letting DE (X AS) = 12 X − AS 22 , show that ∂DE (X AS) = −(X − AS)ST , ∂A ∂DE (X AS) ∇DE (X AS) = = −AT (X − AS). ∂S ∇DE (X AS) =

7 Eigenanalysis

Focusing on the eigenanalysis of matrices, i.e. analyzing matrices by means of eigenvalues, in this chapter we first discuss the eigenvalue decomposition (EVD) of matrices and then present various generalizations of EVD: generalized eigenvalue decomposition, the Rayleigh quotient, the generalized Rayleigh quotient, quadratic eigenvalue problems and the joint diagonalization of multiple matrices. In order to facilitate readers’ understanding, this chapter will present some typical applications.

7.1 Eigenvalue Problem and Characteristic Equation The eigenvalue problem not only is a very interesting theoretical problem but also has a wide range of applications.

7.1.1 Eigenvalue Problem If L[w] = w holds for any nonzero vector w, then L is called an identity transformation. More generally, if, when a linear operator acts on a vector, the output is a multiple of this vector, then the linear operator is said to have an input reproducing characteristic. This input reproducing has two cases. (1) For any nonzero input vector, the output of a linear operator is always completely the same as the input vector (e.g., it is the identity operator). (2) For some special input vector only, the output vector of a linear operator is the same as the input vector up to a constant factor. DEFINITION 7.1 Suppose that a nonzero vector u is the input of the linear operator L. If the output vector is the same as the input vector u up to a constant factor λ, i.e., L[u] = λu,

u = 0,

(7.1.1)

then the vector u is known as an eigenvector of the linear operator L and the scalar λ is the corresponding eigenvalue of L. 413

414

Eigenanalysis

The most commonly used linear operator in engineering applications is undoubtedly a linear time-invariant system. From the above definition, it is known that if each eigenvector u is regarded as an input of the linear time-invariant system, then the eigenvalue λ associated with eigenvector u is equivalent to the gain of the linear system L with u as the input. Only when the input of the system is an eigenvector u, its output is the same as the input up to a constant factor. Thus an eigenvector can be viewed as describing the characteristics of the system, and hence is also called a characteristic vector. This is a physical explanation of eigenvectors in relation to linear systems. If a linear transformation w = L[x] can be represented as w = Ax, then A is called a standard matrix of the linear transformation. Clearly, if A is a standard matrix of a linear transformation, then its eigenvalue problem (7.1.1) can be written as Au = λu,

u = 0.

(7.1.2)

The scalar λ is known as an eigenvalue of the matrix A, and the vector u as an eigenvector associated with the eigenvalue λ. Equation (7.1.2) is sometimes called the eigenvalue–eigenvector equation. EXAMPLE 7.1 Consider a linear time-invariant system h(k) with transfer function ∞ H(e jω ) = k=−∞ h(k)e−jωk . When inputting a complex exponential or harmonic signal ejωn , the system output is given by ∞ 

L[e jωn ] =

k=−∞

h(n − k)e jωk =

∞ 

h(k)e jω(n−k) = H(e jω )e jωn .

k=−∞

Letting u(ω) = [1, e jω , . . . , e jω(N −1) ]T be an input vector, the system output is given by ⎡ ⎡ ⎤ ⎤ 1 1 ⎢ e jω ⎥ ⎢ e jω ⎥ ⎢ ⎥ ⎥ jω ⎢ = H(e ) L⎢ ⎢ ⎥ ⎥ ⇒ L[u(ω)] = H(e jω )u(ω). .. .. ⎣ ⎣ ⎦ ⎦ . . e jω(N −1)

e jω(N −1)

This shows that the harmonic signal vector u(ω) = [1, e jω , . . . , e jω(N −1) ]T is an eigenvector of the linear time-invariant system, and the system transfer function H(e jω ) is the eigenvalue associated with u(ω). From Equation (7.1.2) it is easily seen that if A ∈ Cn×n is a Hermitian matrix then its eigenvalue λ must be a real number, and A = UΣUH ,

(7.1.3)

where U = [u1 , . . . , un ]T is a unitary matrix, and Σ = Diag(λ1 , . . . , λn ). Equation (7.1.3) is called the eigenvalue decomposition (EVD) of A.

7.1 Eigenvalue Problem and Characteristic Equation

415

Since an eigenvalue λ and its associated eigenvector u often appear in pairs, the two-tuple (λ, u) is called an eigenpair of the matrix A. Although an eigenvalue can take a zero value, an eigenvector cannot be a zero vector. Equation (7.1.2) means that the linear transformation Au does not “change the direction” of the input vector u. Hence the linear transformation Au is a mapping “keeping the direction unchanged”. In order to determine the vector u, we can rewrite (7.1.2) as (A − λI)u = 0.

(7.1.4)

If the above equation is assumed to hold for certain nonzero vectors u, then the only condition for a nonzero solution to Equation (7.1.4) to exist is that, for those vectors u, the determinant of the matrix A − λI is equal to zero, namely det(A − λI) = 0.

(7.1.5)

It should be pointed out that some eigenvalues λ may take the same value. The repetition number of an eigenvalue is said to be the eigenvalue multiplicity. For example, all n eigenvalues of an n × n identity matrix I are equal to 1, so its multiplicity is n. It is easily concluded that if the eigenvalue problem (7.1.4) has a nonzero solution u = 0, then the scalar λ must make the n × n matrix A − λI singular. Hence the eigenvalue problem solving consists of the following two steps: (1) find all scalars λ (eigenvalues) such that the matrix A − λI is singular; (2) given an eigenvalue λ such that the matrix A − λI is singular, find all nonzero vectors u satisfying (A − λI)u = 0; these are the eigenvector(s) corresponding to the eigenvalue λ.

7.1.2 Characteristic Polynomial As discussed above, the matrix (A − λI) is singular if and only if its determinant det(A − λI) = 0, i.e., (A − λI) singular



det(A − λI) = 0,

(7.1.6)

where the matrix A − λI is called the characteristic matrix of A. The determinant   a11 − x a12 ··· a1n    a21 a22 − x · · · a2n   p(x) = det(A − xI) =  .  .. .. ..   .. . . .     a a · · · a − x nn n1 n2 = pn xn + pn−1 xn−1 + · · · + p1 x + p0

(7.1.7)

416

Eigenanalysis

is known as the characteristic polynomial of A, and p(x) = det(A − xI) = 0

(7.1.8)

is said to be the characteristic equation of A. The roots of the characteristic equation det(A − xI) = 0 are known as the eigenvalues, characteristic values, latent values, the characteristic roots or latent roots. Obviously, computing the n eigenvalues λi of an n × n matrix A and finding the n roots of the nth-order characteristic polynomial p(x) = det(A − xI) = 0 are two equivalent problems. An n × n matrix A generates an nth-order characteristic polynomial. Likewise, each nth-order polynomial can also be written as the characteristic polynomial of an n × n matrix. THEOREM 7.1 [32]

Any polynomial p(λ) = λn + a1 λn−1 + · · · + an−1 λ + an

can be written as the characteristic polynomial of the n × n matrix ⎤ ⎡ −a1 −a2 · · · −an−1 −an ⎢ −1 0 ··· 0 0 ⎥ ⎥ ⎢ ⎢ 0 −1 · · · 0 0 ⎥ A=⎢ ⎥, ⎢ . .. .. .. ⎥ .. ⎣ .. . . . . ⎦ 0 0 ··· −1 0 namely p(λ) = det(λI − A).

7.2 Eigenvalues and Eigenvectors In this section we discuss the computation and properties of the eigenvalues and eigenvectors of an n × n matrix A.

7.2.1 Eigenvalues Even if an n × n matrix A is real, the n roots of its characteristic equation may be complex, and the root multiplicity can be arbitrary or even equal to n. These roots are collectively referred to as the eigenvalues of the matrix A. Regarding eigenvalues, it is necessary to introduce the following terminology [422, p. 15]. (1) An eigenvalue λ of a matrix A is said to have algebraic multiplicity μ, if λ occurs μ times as a root of the characteristic polynomial det(A − zI) = 0. (2) If the algebraic multiplicity of an eigenvalue λ is 1, then it is called a single eigenvalue. A nonsingle eigenvalue is said to be a multiple eigenvalue.

7.2 Eigenvalues and Eigenvectors

417

(3) An eigenvalue λ of A is said to have geometric multiplicity γ if the number of linearly independent eigenvectors corresponding to the eigenvalue is γ. In other words, the geometric multiplicity γ is the dimension of the eigenspace Null(A − λI). (4) A matrix A is known as a derogatory matrix if there is at least one of its eigenvalues with geometric multiplicity greater than 1. (5) An eigenvalue is referred to as semi-single, if its algebraic multiplicity is equal to its geometric multiplicity. A semi-single eigenvalue is also called a defective eigenvalue. It is well known that any nth-order polynomial p(x) can be written in the factorized form p(x) = a(x − x1 )(x − x2 ) · · · (x − xn ).

(7.2.1)

The n roots of the characteristic polynomial p(x), denoted x1 , x2 , . . . , xn , are not necessarily different from each other, and also are not necessarily real. In general the eigenvalues of a matrix A are different from each other, but a characteristic polynomial has multiple roots then we say that the matrix A has degenerate eigenvalues. It has already been noted that even if A is a real matrix, its eigenvalues may be complex. Taking the Givens rotation matrix   cos θ − sin θ A= sin θ cos θ as an example, its characteristic equation is   cos θ − λ − sin θ   = (cos θ − λ)2 + sin2 θ = 0. det(A − λI) =  sin θ cos θ − λ However, if θ is not an integer multiple of π then sin2 θ > 0. In this case, the characteristic equation cannot give a real value for λ, i.e., the two eigenvalues of the rotation matrix are complex, and the two corresponding eigenvectors are complex vectors. The eigenvalues of an n × n matrix (not necessarily Hermitian) A have the following properties [214]. 1. An n×n matrix A has a total of n eigenvalues, where multiple eigenvalues count according to their multiplicity. 2. If a nonsymmetric real matrix A has complex eigenvalues and/or complex eigenvectors, then they must appear in the form of a complex conjugate pair. 3. If A is a real symmetric matrix or a Hermitian matrix then its all eigenvalues are real numbers. 4. The eigenvalues of a diagonal matrix and a triangular matrix satisfy the following:

418

Eigenanalysis

• if A = Diag(a11 , . . . , ann ) then its eigenvalues are given by a11 , . . . , ann ; • if A is a triangular matrix then all its diagonal entries are eigenvalues. 5. Given an n × n matrix A: • • • •

if if if if

λ is an eigenvalue of A then λ is also an eigenvalue of AT ; λ is an eigenvalue of A then λ∗ is an eigenvalue of AH ; λ is an eigenvalue of A then λ + σ 2 is an eigenvalue of A + σ 2 I; λ is an eigenvalue of A then 1/λ is an eigenvalue of its inverse matrix A−1 .

6. All eigenvalues of an idempotent matrix A2 = A are 0 or 1. 7. If A is a real orthogonal matrix, then all its eigenvalues are located on the unit circle. 8. The relationship between eigenvalues and the matrix singularity is as follows: • if A is singular then it has at least one zero eigenvalue; • if A is nonsingular then its all eigenvalues are nonzero. 9. The relationship between eigenvalues and trace: the sum of all eigenvalues of A n is equal to its trace, namely i=1 λi = tr(A). 10. A Hermitian matrix A is positive definite (semi-definite) if and only if its all eigenvalues are positive (nonnegative). 11. The relationship between eigenvalues and determinant: the product of all eigen?n values of A is equal to its determinant, namely i=1 λi = det(A) = |A|. 12. If the eigenvalues of A are different from each other then there must be a similar matrix S (see Subsection 7.3.1) such that S−1 AS = D (diagonal matrix); the diagonal entries of D are the eigenvalues of A. 13. The relationship between eigenvalues and rank is as follows: • if an n × n matrix A has r nonzero eigenvalues then rank(A) ≥ r; • if 0 is a single eigenvalue of an n × n matrix A then rank(A) = n − 1; • if rank(A − λI) ≤ n − 1 then λ is an eigenvalue of the matrix A. 14. The geometric multiplicity of any eigenvalue λ of an n × n matrix A cannot be greater than the algebraic multiplicity of λ. 15. The Cayley–Hamilton theorem: if λ1 , λ2 , . . . , λn are the eigenvalues of an n × n matrix A then n ( (A − λi I) = 0. i=1

16. On the eigenvalues of similar matrices: • if λ is an eigenvalue of an n × n matrix A and another n × n matrix B is nonsingular then λ is also an eigenvalue of the matrix B−1 AB; • if λ is an eigenvalue of an n × n matrix A and another n × n matrix B is unitary then λ is also an eigenvalue of the matrix BH AB; • if λ is an eigenvalue of an n × n matrix A and another n × n matrix B is orthogonal then λ is also an eigenvalue of BT AB.

7.2 Eigenvalues and Eigenvectors

419

17. The eigenvalues λi of the correlation matrix R = E{x(t)xH (t)} of a random vector x(t) = [x1 (t), . . . , xn (t)]T are bounded by the maximum power Pmax = maxi E{|xi (t)|2 } and the minimum power Pmin = mini E{|xi (t)|2 } of the signal components, namely (7.2.2) Pmin ≤ λi ≤ Pmax . 18. The eigenvalue spread of the correlation matrix R of a random vector x(t) is λ (7.2.3) X (R) = max . λmin 19. The matrix products Am×n Bn×m and Bn×m Am×n have the same nonzero eigenvalues. 20. If an eigenvalue of a matrix A is λ then the corresponding eigenvalue of the matrix polynomial f (A) = An + c1 An−1 + · · · + cn−1 A + cn I is given by f (λ) = λn + c1 λn−1 + · · · + cn−1 λ + cn .

(7.2.4)

21. If λ is an eigenvalue of a matrix A then eλ is an eigenvalue of the matrix exponential function eA .

7.2.2 Eigenvectors If a matrix An×n is a complex matrix, and λ is one of its eigenvalues then the vector v satisfying (A − λI)v = 0

or Av = λv

(7.2.5)

is called the right eigenvector of the matrix A associated with the eigenvalue λ, while the vector u satisfying uH (A − λI) = 0T

or uH A = λuH

(7.2.6)

is known as the left eigenvector of A associated with the eigenvalue λ. If a matrix A is Hermitian then its all eigenvalues are real, and hence from Equation (7.2.5) it is immediately known that ((A − λI)v)T = vT (A − λI) = 0T , yielding v = u; namely, the right and left eigenvectors of any Hermitian matrix are the same. It is useful to compare the similarities and differences of the SVD and the EVD of a matrix. (1) The SVD is available for any m × n (where m ≥ n or m < n) matrix, while the EVD is available only for square matrices. (2) For an n × n non-Hermitian matrix A, its kth singular value is defined as the spectral norm of the error matrix Ek making the rank of the original matrix A decreased by 1:   : rank(A + E) ≤ k − 1 , k = 1, . . . , min{m, n},

E σk = min spec m×n E∈ C

(7.2.7)

420

Eigenanalysis

while its eigenvalues are defined as the roots of the characteristic polynomial det(A − λI) = 0. There is no inherent relationship between the singular values and the eigenvalues of the same square matrix, but each nonzero singular value of an m × n matrix A is the positive square root of some nonzero eigenvalue of the n × n Hermitian matrix AH A or the m × m Hermitian matrix AAH . (3) The left singular vector ui and the right singular vector vi of an m × n matrix A associated with the singular value σi are defined as the two vectors satisfying H H uH i Avi = σi , while the left and right eigenvectors are defined by u A = λi u and Avi = λi vi , respectively. Hence, for the same n × n non-Hermitian matrix A, there is no inherent relationship between its (left and right) singular vectors and its (left and right) eigenvectors. However, the left singular vector ui and right singular vector vi of a matrix A ∈ Cm×n are respectively the eigenvectors of the m × m Hermitian matrix AAH and of the n × n matrix AH A. From Equation (7.1.2) it is easily seen that after multiplying an eigenvector u of a matrix A by any nonzero scalar μ, then μu is still an eigenvector of A. For convenience, it is generally assumed that eigenvectors have unit norm, i.e., u 2 = 1. Using eigenvectors we can introduce the condition number of any single eigenvalue. DEFINITION 7.2 [422, p. 93] any matrix A is defined as

The condition number of a single eigenvalue λ of

cond(λ) =

1 , cos θ(u, v)

(7.2.8)

where θ(u, v) represents the acute angle between the left and right eigenvectors associated with the eigenvalue λ. DEFINITION 7.3 The set of all eigenvalues λ ∈ C of a matrix A ∈ Cn×n is called the spectrum of the matrix A, denoted λ(A). The spectral radius of a matrix A, denoted ρ(A), is a nonnegative real number and is defined as ρ(A) = max |λ| : λ ∈ λ(A).

(7.2.9)

DEFINITION 7.4 The inertia of a symmetric matrix A ∈ Rn×n , denoted In(A), is defined as the triplet In(A) = (i+ (A), i− (A), i0 (A)), where i+ (A), i− (A) and i0 (A) are respectively the numbers of the positive, negative and zero eigenvalues of A (each multiple eigenvalue is counted according to its multiplicity). Moreover, the quality i+ (A) − i− (A) is known as the signature of A. The following summarize the properties of the eigenpair (λ, u) [214]. 1. If (λ, u) is an eigenpair of a matrix A then (cλ, u) is an eigenpair of the matrix cA, where c is a nonzero constant.

7.2 Eigenvalues and Eigenvectors

421

2. If (λ, u) is an eigenpair of a matrix A then (λ, cu) is also an eigenpair of the matrix A, where c is a nonzero constant. 3. If (λi , ui ) and (λj , uj ) are two eigenpairs of a matrix A, and λi = λj , then the eigenvector ui is linearly independent of uj . 4. The eigenvectors of a Hermitian matrix associated with different eigenvalues are orthogonal to each other, namely uH i uj = 0 for λi = λj . 5. If (λ, u) is an eigenpair of a matrix A then (λk , u) is the eigenpair of the matrix Ak . 6. If (λ, u) is an eigenpair of a matrix A then (eλ , u) is an eigenpair of the matrix exponential function eA . 7. If λ(A) and λ(B) are respectively the eigenvalues of the matrices A and B, and u(A) and u(B) are respectively the eigenvectors associated with λ(A) and λ(B), then λ(A)λ(B) is the eigenpair of the matrix Kronecker product A ⊗ B, and u(A) ⊗ u(B) is the eigenvector associated with the eigenvalue λ(A)λ(B). The SVD of an m × n matrix A can be transformed to the EVD of the corresponding matrix. There are two main methods to achieve this transformation. Method 1 The nonzero singular value σi of a matrix Am×n is the positive square root of the nonzero eigenvalue λi of the m × m matrix AAT or the n × n matrix AT A, and the left singular vector ui and the right singular vector vi of A associated with σi are the eigenvectors of AAT and AT A associated with the nonzero eigenvalue λi . Method 2 The SVD of a matrix Am×n is transformed to the EVD of an (m + n) × (m + n) augmented matrix   O A . (7.2.10) AT O THEOREM 7.2 (Jordan–Wielandt theorem) [455, Theorem I.4.2] If σ1 ≥ σ2 ≥ · · · ≥ σp−1 ≥ σp are the singular values of Am×n (where p = min{m, n}) then the augmented matrix in (7.2.10) has the eigenvalues −σ1 , . . . , −σp , 0, . . . , 0, σp , . . . , σ1 , 6 78 9 |m−n|

and the eigenvectors associated with ±σj are given by   uj , j = 1, 2, . . . , p. ±vj If m = n then we have   uj , n + 1 ≤ j ≤ m, m > n 0

 or

 0 , m + 1 ≤ j ≤ n, m < n. vj

422

Eigenanalysis

On the eigenvalues of the matrix sum A + B, one has the following result. THEOREM 7.3 (Weyl theorem) [275] Let A, B ∈ Cn×n be Hermitian matrices, and let their eigenvalues be arranged in ascending order: λ1 (A) ≤ λ2 (A) ≤ · · · ≤ λn (A), λ1 (B) ≤ λ2 (B) ≤ · · · ≤ λn (B), λ1 (A + B) ≤ λ2 (A + B) ≤ · · · ≤ λn (A + B). Then

⎧ λi (A) + λ1 (B), ⎪ ⎪ ⎪ ⎨ λi−1 (A) + λ2 (B), λi (A + B) ≥ .. ⎪ ⎪ . ⎪ ⎩ λ1 (A) + λi (B), ⎧ λi (A) + λn (B), ⎪ ⎪ ⎪ ⎨ λi+1 (A) + λn−1 (B), λi (A + B) ≤ .. ⎪ ⎪ . ⎪ ⎩ λn (A) + λi (B),

(7.2.11)

(7.2.12)

where i = 1, 2, . . . , n. In particular, when A is a real symmetric matrix and B = αzzT (see below), then one has the following interlacing eigenvalue theorem [179, Theorem 8.1.8]. THEOREM 7.4 Let A ∈ Rn×n be a symmetric matrix with eigenvalues λ1 , . . . , λn satisfying λ1 ≥ λ2 ≥ · · · ≥ λn , and let z ∈ Rn be a vector with norm z = 1. Suppose that α is a real number, and the eigenvalues of the matrix A + αzzT are arranged as ξ1 ≥ ξ2 ≥ · · · ≥ ξn . Then one has ξ 1 ≥ λ1 ≥ ξ 2 ≥ λ2 ≥ · · · ≥ ξ n ≥ λn

(for α > 0)

(7.2.13)

λ 1 ≥ ξ 1 ≥ λ 2 ≥ ξ 2 ≥ · · · ≥ λn ≥ ξ n

(for α < 0),

(7.2.14)

or

and, whether α > 0 or α < 0, the following result is true: n 

(ξi − λi ) = α.

(7.2.15)

i=1

7.3 Similarity Reduction The normalized representation of a matrix is called the canonical form (or normal or standard form) of the matrix. In most fields, a canonical form specifies a unique representation while a normal form simply specifies the form without the

7.3 Similarity Reduction

423

requirement of uniqueness. The simplest canonical form of a matrix is its diagonalized representation. However, many matrices are not diagonalizable. The problem is that in many applications it is necessary to reduce a given matrix to as simple a form as possible. In these applications the similarity reduction of a matrix is a very natural choice. The standard form of the similarity reduction is the Jordan canonical form. Therefore, the core problem of matrix similarity reduction is how to obtain the Jordan canonical form. There are two different ways to find the Jordan canonical form of a matrix. (1) For a constant matrix, use a direct similarity reduction of the matrix itself. (2) First make a balanced reduction of the polynomial matrix corresponding to the original constant matrix, and then transform the Smith normal form of the balanced reduction into the Jordan canonical form of the similarity reduction of the original matrix. In this section we discuss the implementation of the first way, and in the next section we describe similarity reduction based on the second way.

7.3.1 Similarity Transformation of Matrices The theoretical basis and mathematical tools of matrix similarity reduction are provided by the similarity transformation of matrices. Let P ∈ Cm×m be a nonsingular matrix, and use it to make the linear transformation of a matrix A ∈ Cm×m : B = P−1 AP.

(7.3.1)

Suppose that an eigenvalue of the linear transformation B is λ and that a corresponding eigenvector is y, i.e., By = λy.

(7.3.2)

Substitute (7.3.1) into (7.3.2) to get P−1 APy = λy or A(Py) = λ(Py). Letting x = Py or y = P−1 x, we have Ax = λx.

(7.3.3)

Comparing (7.3.2) with (7.3.3), it can be seen that the two matrices A and B = P−1 AP have the same eigenvalues and the eigenvector y is a linear transformation of the eigenvector x of the matrix A, i.e., y = P−1 x. Because the eigenvalues of the two matrices A and B = P−1 AP are the same and their eigenvectors have a linear transformation relationship, the matrices A and B are said to be similar. DEFINITION 7.5 If there is a nonsingular matrix P ∈ Cm×m such that B = P−1 AP then the matrix B ∈ Cm×m is said to be similar to the matrix A ∈ Cm×m and P is known as the similarity transformation matrix.

424

Eigenanalysis

“B is similar to A” is denoted as B ∼ A. Similar matrices have the following basic properties. (1) Reflexivity A ∼ A, i.e., any matrix is similar to itself. (2) Symmetry If A is similar to B, then B is also similar to A. (3) Transitivity If A is similar to B, and B is similar to C, then A is similar to C, i.e., A ∼ C. In addition, similar matrices have also the following important properties. 1. Similar matrices B ∼ A have the same determinant, i.e., |B| = |A|. 2. If the matrix P−1 AP = T (an upper triangle matrix) then the diagonal entries of T give the eigenvalues λi of A. 3. Two similar matrices have exactly the same eigenvalues. 4. For the similar matrix B = P−1 AP we have B2 = P−1 APP−1 AP = P−1 A2 P, and thus Bk = P−1 Ak P. This is to say, if B ∼ A then Bk ∼ Ak . This property is called the power property of similar matrices. 5. If both B = P−1 AP and A are invertible then B−1 = P−1 A−1 P, i.e., when two matrices are similar, their inverse matrices are similar as well. If P is a unitary matrix then B = P−1 AP is called the unitary similarity transformation of A. The following example shows how to perform the diagonalization of an m × m matrix A. EXAMPLE 7.2

Find the similarity transformation of the 3 × 3 real matrix ⎡ ⎤ 1 1 1 A = ⎣ 0 3 3 ⎦. −2 1 1

Solution Direct computation gives the characteristic polynomial    λ−1 −1 −1   |λI − A| =  0 λ−3 −3  = λ(λ − 2)(λ − 3).  2 −1 λ−1  On solving the characteristic equation |λI − A| = 0, the three eigenvalues of A are given by λ = 0, 2, 3. (a) For the eigenvalue λ = 0, we have (0I − A)x = 0, i.e., x1 + x2 + x3 = 0, 3x2 + 3x3 = 0, 2x1 − x2 − x3 = 0,

7.3 Similarity Reduction

425

with solution x1 = 0 and x2 = −x3 , where x3 is arbitrary. Hence, the eigenvector associated with the eigenvalue λ = 0 is ⎡ ⎤ ⎡ ⎤ 0 0 x = ⎣ −a ⎦ = a ⎣ −1 ⎦ , a = 0. a 1 Taking a = 1 yields the eigenvector x1 = [0, −1, 1]T . (b) For λ = 2, the characteristic polynomial is given by (2I − A)x = 0, i.e., x1 − x2 − x3 = 0, x2 + 3x3 = 0, 2x1 − x2 + x3 = 0, whose solution is x1 = −2x3 , x2 = −3x3 , where x3 is arbitrary. Thus we get the eigenvector ⎡ ⎤ ⎡ ⎤ −2a −2 x = ⎣ −3a ⎦ = ⎣ −3 ⎦ , a = 1. a 1 (c) Similarly, the eigenvector associated with the eigenvalue λ = 3 is x3 = [1, 2, 0]T . The three eigenvectors constitute the similarity transformation matrix, which is given, along with its inverse, by ⎡ ⎤ ⎡ ⎤ 0 −2 1 1 −0.5 0.5 −1 P = ⎣ −1 −3 2 ⎦ , P = ⎣ −1 0.5 0.5 ⎦ . 1 1 0 −1 1.0 1.0 Therefore, the diagonalized matrix corresponding to A is given by ⎡ ⎤⎡ ⎤⎡ ⎤ 1 −0.5 0.5 1 1 1 0 −2 1 P−1 AP = ⎣ −1 0.5 0.5 ⎦ ⎣ 0 3 3 ⎦ ⎣ −1 −3 2 ⎦ −1 1.0 1.0 −2 1 1 1 1 0 ⎡ ⎤⎡ ⎤ 1 −0.5 0.5 0 −4 3 ⎣ ⎦ ⎣ = −1 0.5 0.5 0 −6 6 ⎦ −1 1.0 1.0 0 2 0 ⎡ ⎤ 0 0 0 = ⎣ 0 2 0 ⎦, 0

0

3

which is the diagonal matrix consisting of the three different eigenvalues of the matrix A. DEFINITION 7.6 An m × m real matrix A is said to be a diagonalization matrix if it is similar to a diagonal matrix.

426

Eigenanalysis

The following theorem gives the necessary and sufficient condition for the diagonalization of the matrix A ∈ Cm×m . THEOREM 7.5 An m × m matrix A is diagonalizable if and only if A has m linearly independent eigenvectors. The following theorem gives the necessary and sufficient condition for all eigenvectors to be linearly independent, and thus it is the necessary and sufficient condition for matrix diagonalization. THEOREM 7.6 (Diagonalization theorem) [433, p. 307] Given a matrix A ∈ Cm×m whose eigenvalues λk have the algebraic multiplicity dk , k = 1, . . . , p, where p k=1 dk = m. The matrix A has m linearly independent eigenvectors if and only if rank(A − λk I) = m − dk , k = 1, . . . , p. In this case, the matrix U in AU = UΣ is nonsingular, and A is diagonalized as U−1 AU = Σ.

7.3.2 Similarity Reduction of Matrices A nondiagonalizable matrix A ∈ Cm×m with multiple eigenvalues can be reduced to the Jordan canonical form via a similarity transformation. DEFINITION 7.7 Let a matrix A with r = rank(A) have d different nonzero eigenvalues, and let the nonzero eigenvalue λi have multiplicity mi , i.e., m1 + · · · + md = r. If the similarity transformation has the form P−1 AP = J = Diag(J1 , . . . , Jd , 0, . . . , 0), 6 78 9

(7.3.4)

m−r

then J is called the Jordan canonical form of the matrix A and Ji , i = 1, . . . , d, are known as Jordan blocks. A Jordan block corresponding to an eigenvalue λi with multiplicity 1 is of first order and is a 1 × 1 Jordan block matrix J1×1 = λi . A kth-order Jordan block is defined as ⎤ ⎡ λ 1 0 ⎥ ⎢ . ⎥ ⎢ λ .. ⎥ ∈ Ck×k , ⎢ Jk×k = ⎢ (7.3.5) ⎥ . . . 1⎦ ⎣ 0 λ in which the k entries on the main diagonal line are λ, and the k − 1 entries on the subdiagonal on the right of main diagonal are equal to 1, while all other entries are equal to zero. For example, second-order and third-order Jordan blocks are as

7.3 Similarity Reduction

follows:  J2×2 =

⎤ λ 1 0 = ⎣0 λ 1⎦ . 0 0 λ ⎡



λ 1 , 0 λ

427

J3×3

The Jordan canonical form J has the following properties. 1. J is an upper bidiagonal matrix. 2. J is a diagonal matrix in the special case of n Jordan blocks of size nk = 1. 3. J is unique up to permutations of the blocks (so it is called the Jordan canonical form). 4. J can have multiple blocks with the same eigenvalue. A multiple eigenvalue λ with algebraic multiplicity mi may have one or more Jordan blocks, depending on the geometric multiplicity of λ. For example, if a 3 × 3 matrix A has an eigenvalue λ0 with algebraic multiplicity 3, then the Jordan canonical form of the matrix A may have three forms: ⎤ ⎡ ⎤ ⎡ 0 0 0 J1×1 λ0 0 J1 = ⎣ 0 J1×1 0 ⎦ = ⎣ 0 λ0 0 ⎦ (geometric multiplicity 3), 0 0 J1×1 0 0 λ0 ⎤ ⎡   0 λ0 0 J1×1 0 ⎣ = 0 λ0 1 ⎦ (geometric multiplicity 2), J2 = 0 J2×2 0 0 λ0 ⎤ ⎡ 0 λ0 1 J3 = J3×3 = ⎣ 0 λ0 1 ⎦ (geometric multiplicity 1), 0 0 λ0 because the meaning of the geometric multiplicity α of the eigenvalue λ is that the number of linearly independent eigenvectors corresponding to λ is α. Jordan canonical forms with different permutation orders of the Jordan blocks are regarded as the same Jordan canonical form, as stated above. For example, the second-order Jordan canonical form J2 can also be arranged as ⎤ ⎡   0 λ0 1 J2×2 0 = ⎣ 0 λ0 0 ⎦ . J2 = 0 J1×1 0 0 λ0 In practical applications, a nondiagonalizable m × m matrix usually has multiple eigenvalues. Because the number of Jordan blocks associated with an eigenvalue λi is equal to its geometric multiplicity αi , a natural question is how to determine αi for a given eigenvalue λi ? The method for determining the geometric multiplicity (i.e. the number of Jordan blocks) of a given m × m matrix A is as follows.

428

Eigenanalysis

(1) The number of Jordan blocks with order greater than 1 (i.e., ≥ 2) is determined by α1 = rank(λi I − A) − rank(λi I − A)2 . (2) The number of Jordan blocks with order greater than 2 (i.e., ≥ 3) is given by α2 = rank(λi I − A)2 − rank(λi I − A)3 . (3) More generally, the number of Jordan blocks with the order greater than k − 1 (i.e., ≥ k) is determined by αk−1 = rank(λi I − A)k−1 − rank(λi I − A)k . (4) The sum of the orders of the Jordan blocks corresponding to an eigenvalue λi is equal to its algebraic multiplicity. EXAMPLE 7.3

Find the Jordan canonical form of a 3 × 3 matrix ⎡ ⎤ 1 0 4 A = ⎣ 2 −1 4 ⎦. −1 0 −3

Solution From the characteristic determinant   λ−1 0 −4  |λI − A| =  −2 λ+1 −4  1 0 λ+3

(7.3.6)

    = (λ + 1)3 = 0,  

we find that get the eigenvalue of the matrix A is λ = −1 with (algebraic) multiplicity 3. For λ = −1, since ⎡

−2 0 λI − A = ⎣−2 0 1 0

⎤ −4 −4⎦ , 2



⎤ 0 0 0 (λI − A)2 = ⎣0 0 0⎦ , 0 0 0

the number of Jordan blocks with order ≥ 2 is given by rank(λI − A) − rank(λI − A)2 = 1 − 0 = 1, i.e., there is one Jordan block with order ≥ 2. Moreover, since rank(λI − A)2 − rank(λI − A)3 = 0 − 0 = 0, there is no third-order Jordan block. Therefore, the eigenvalue −1 with multiplicity 3 corresponds to two Jordan blocks, of which one Jordan block has order 2, and another has order 1. In other words, the Jordan canonical form of the given matrix A is  J J = 1×1 0

0 J2×2





⎤ −1 0 0 = ⎣ 0 −1 1 ⎦ . 0 0 −1

The Jordan canonical form P−1 AP = J can be written as AP = PJ. When A has d different eigenvalues λi with multiplicity mi , i = 1, . . . , d, and m1 +· · ·+md = r with r = rank(A), the Jordan canonical form is J = Diag(J1 , . . . , Jd , 0, . . . , 0),

7.3 Similarity Reduction

429

so AP = PJ simplifies to ⎡

J1 ⎢ .. A[P1 , . . . , Pd ] = [P1 , . . . , Pd ] ⎣ . O

··· .. . ···

⎤ O .. ⎥ .⎦ Jd

= [P1 J1 , . . . , Pd Jd ], where the Jordan blocks are given by ⎡ ⎤ λi 1 · · · 0 ⎢ .⎥ ⎢ 0 . . . . . . .. ⎥ ⎥ ∈ Rmi ×mi , Ji = ⎢ ⎢. . ⎥ .. λ ⎣ .. 1⎦ i 0 ··· 0 λi

(7.3.7)

i = 1, . . . , d.

Letting Pi = [pi,1 , . . . , pi,mi ], Equation (7.3.7) gives Api,j = pi,j−1 + λi pi,j ,

i = 1, . . . , d, j = 1, . . . , mi ,

(7.3.8)

where pi,0 = 0. Equation (7.3.8) provides a method of computation for the similarity transformation matrix P. Algorithm 7.1 shows how to find the Jordan canonical form of a matrix A ∈ m×m C via similarity reduction. Algorithm 7.1

Similarity reduction of matrix A

input: A ∈ Rm×m . 1. Solve |λI − A| = 0 for λi with multiplicity mi , where i = 1, . . . , d, and m1 + · · · + md = m. 2. Solve Equation (7.3.8) for Pi = [pi,1 , . . . , pi,mi ], i = 1, . . . , d. 3. Set P = [P1 , . . . , Pd ], and find its inverse matrix P−1 .

4. Compute P−1 AP = J. output: Similarity transformation matrix P and Jordan canonical form J.

EXAMPLE 7.4 matrix

Use similarity reduction to find the Jordan canonical form of the ⎡

1 0 A = ⎣ 2 −1 −1 0

⎤ 4 4 ⎦. −3

Solution From |λI − A| = 0 we get the single eigenvalue λ = −1 with algebraic multiplicity 3.

430

Eigenanalysis

Let j = 1, and solve Equation (7.3.8): ⎤⎡ ⎡ ⎤ ⎡ ⎤ p11 1 0 4 p11 ⎣ 2 −1 4 ⎦ ⎣ p21 ⎦ = − ⎣ p21 ⎦ −1 0 −3 p31 p31 from which we have p11 = −p31 ; p21 can take an arbitrary value. Take p11 = p31 = 0 and p21 = 1. Solving Equation (7.3.8) for j = 2, ⎡ ⎤⎡ ⎤ ⎡ ⎤ ⎤ ⎡ 1 0 4 p12 0 p12 ⎣ 2 −1 4 ⎦ ⎣ p22 ⎦ = ⎣ 1 ⎦ − ⎣ p22 ⎦ , −1 0 −3 p32 p32 0 we get p12 = −2p32 ; p22 can take an arbitrary value. Take p12 = 2, p22 = 2 and p32 = −1. Similarly, from Equation (7.3.8) with j = 3, ⎡ ⎤⎡ ⎤ ⎡ ⎤ ⎤ ⎡ 1 0 4 p13 2 p13 ⎣ 2 −1 4 ⎦ ⎣ p23 ⎦ = ⎣ 2 ⎦ − ⎣ p23 ⎦ , −1

0 −3

p33

−1

p33

we obtain p13 +2p33 = 1; p32 can take an arbitrary value. Take p13 = 1, p23 = 0 and p33 = 0. Hence, the similarity transformation matrix and its inverse are respectively as follows: ⎡ ⎤ ⎡ ⎤ 0 2 1 0 1 2 P=⎣ 1 (7.3.9) 2 0 ⎦ , P−1 = ⎣ 0 0 −1 ⎦ . 0 −1 0 1 0 2 Therefore the similarity reduction of A is given by ⎡ ⎤ −1 0 0 J = P−1 AP = ⎣ 0 −1 1 ⎦. 0 0 −1

(7.3.10)

7.3.3 Similarity Reduction of Matrix Polynomials Consider a polynomial f (x) = an xn + an−1 xn−1 + · · · + a1 x + a0 .

(7.3.11)

When an = 0, n is called the order of the polynomial f (x). An nth-order polynomial is known as a monic polynomial if the coefficient of xn is equal to 1. Given a matrix A ∈ Cm×m and a polynomial function f (x), we say that f (A) = an An + an−1 An−1 + · · · + a1 A + a0 I is an nth-order matrix polynomial of A.

(7.3.12)

7.3 Similarity Reduction

431

Suppose that the characteristic polynomial p(A) = λI − A has d different roots which give the eigenvalues λ1 , . . . , λd of the matrix A and that the multiplicity of the eigenvalue λi is mi , i.e., m1 + · · · + md = m. If the Jordan canonical form of the matrix A is J, i.e., A = PJP−1 = PDiag(J1 (λ1 ), . . . , Jd (λd ))P−1 ,

(7.3.13)

then f (A) = an An + an−1 An−1 + · · · + a1 A + a0 I = an (PJP−1 )n + an−1 (PJP−1 )n−1 + · · · + a1 (PJP−1 ) + a0 I = P(an Jn + an−1 Jn−1 + · · · + a1 J + a0 I)P−1 = Pf (J)P−1

(7.3.14)

is known as the similarity reduction of the matrix polynomial f (A), where f (J) = an Jn + an−1 Jn−1 + · · · + a1 J + a0 I ⎡ ⎤n ⎡ O J1 J1 ⎢ ⎥ ⎢ . .. .. = an ⎣ ⎦ + · · · + a1 ⎣ . O =

Jp

Diag(an Jn1

O

+ · · · + a1 J1 +

O





⎥ ⎢ ⎦ + a0 ⎣

Jd

a0 I1 , . . . , an Jnd

O

I1 .. O

.

⎤ ⎥ ⎦

Id

+ · · · + a1 Jd + a0 Id )

= Diag(f (J1 ), . . . , f (Jd )).

(7.3.15)

Substitute Equation (7.3.15) into Equation (7.3.14) to yield f (A) = PDiag(f (J1 ), . . . , f (Jd ))P−1 ,

(7.3.16)

where f (Ji ) ∈ Cmi ×mi is the Jordan representation of the matrix function f (A) associated with the eigenvalue λi and is defined as ⎤ ⎡ 1

1 f (λi ) f (λi ) f (λ) · · · f (mi −1) (λi ) 2! (mi − 1)! ⎥ ⎢ ⎥ ⎢ 1

(mi −2) ⎢ f (λi ) f (λi ) ··· f (λi ) ⎥ ⎥ ⎢ (mi − 2)! ⎥ , (7.3.17) f (Ji ) = ⎢ .. .. .. ⎥ ⎢ . . ⎥ ⎢ . ⎥ ⎢

⎦ ⎣ f (λi ) f (λi ) 0 f (λi ) where f (k) (x) = dk f (x)/dxk is the kth-order derivative of the function f (x). Equation (7.3.16) is derived under the assumption that f (A) is a the matrix polynomial. However, using (PAP−1 )k = PJk P−1 , it is easy to see that Equation (7.3.16) applies to other common matrix functions f (A) as well. (1) Powers of a matrix AK = PJK P−1 = Pf (J)P−1 . def

432

Eigenanalysis

In this case, f (x) = xK . (2) Matrix logarithm

 ∞ ∞   (−1)n−1 (−1)n−1 n ln(I + A) = A =P An P−1 n n n=1 n=1 def

= Pf (J)P−1 . In this case, f (x) = ln(1 + x). (3) Sine and cosine functions

 ∞ ∞   (−1)n (−1)n 2n+1 2n+1 sin A = P−1 =P A J (2n + 1)! (2n + 1)! n=0 n=0 def

= Pf (J)P−1 ,

 ∞ ∞   (−1)n (−1)n 2n cos A = A =P J2n P−1 = Pf (J)P−1 , (2n)! (2n)! n=0 n=0 def

where f1 (x) = sin x and f2 (x) = cos x, respectively. (4) Matrix exponentials  ∞ ∞   1 1 n A def n e = P−1 = Pf (J)P−1 , A =P J n! n! n=0 n=0  ∞ ∞   1 1 −A def n n n n e P−1 = Pf (J)P−1 . = (−1) A = P (−1) J n! n! n=0 n=0 Here the scalar functions corresponding to the matrix exponentials f1 (A) = eA and f2 (A) = e−A are respectively f1 (x) = ex and f2 (x) = e−x . (5) Matrix exponential functions  ∞ ∞   1 1 n n At def n n e = P−1 = Pf (J)P−1 , A t =P J t n! n! n=0 n=0  ∞ ∞   1 1 −At def n n n n n n e P−1 = Pf (J)P−1 , = (−1) A t = P (−1) J t n! n! n=0 n=0 where the scalar functions corresponding to the matrix exponential functions f1 (A) = eAt and f2 (A) = e−At are respectively f1 (x) = ext and f2 (x) = e−xt . The following example shows how to compute matrix functions using (7.3.16) and (7.3.17). EXAMPLE 7.5

Given the matrix ⎡

⎤ 1 0 4 A = ⎣ 2 −1 4 ⎦, −1 0 −3

7.3 Similarity Reduction

433

compute (a) the matrix polynomial f (A) = A4 −3A3 +A−I, (b) the matrix power f (A) = A1000 , (c) the matrix exponential f (A) = eA , (d) the matrix exponential function f (A) = eAt and (e) the matrix trigonometric function sin A. Solution In Example 7.4, we found the similarity transformation matrix P, its inverse matrix P−1 and the Jordan canonical form of the above matrix, as follows: ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 0 1 2 −1 0 0 0 2 1 P = ⎣1 1 ⎦. 2 0 ⎦, P−1 = ⎣ 0 0 −1 ⎦, J = P−1 AP = ⎣ 0 −1 1 0 2 0 0 −1 0 −1 0 Thus we have the matrix polynomial f (A) = Pf (J)P−1 ⎤⎡ ⎡ ⎤⎡ 0 0 2 1 f (−1) 0 0 =⎣ 1 2 0 ⎦⎣ 0 f (−1) f (−1) ⎦ ⎣ 0 0 0 f (−1) 1 0 −1 0 ⎤ ⎡



0 4f (−1) f (−1) + 2f (−1) ⎦. =⎣ 2f (−1) f (−1) 4f (−1) −f (−1) 0 f (−1) − 2f (−1)

1 0 0

⎤ 2 −1 ⎦ 2 (7.3.18)

(a) To compute the matrix polynomial f (A) = A4 − 3A3 + A − I, consider the corresponding polynomial function f (x) = x4 −3x3 +x−1 with first-order derivative f (x) = 4x3 − 9x2 + 1. Hence, for the triple eigenvalue λ = −1, f (−1) = 5 and the first-order derivative f (−1) = −12. Substituting f (−1) = 5 and f (−1) = −12 into Equation (7.3.18), we have ⎡ ⎤ −19 0 −48 f (A) = A4 − 3A3 + A − I = ⎣ −24 5 −48 ⎦ . 12 0 29 (b) To compute the matrix power f (A) = A1000 , consider the corresponding polynomial function f (x) = x1000 , whose first-order derivative f (x) = 1000x999 . Substitute f (−1) = 1 and f (−1) = −1000 into Equation (7.3.18) to yield directly ⎡ ⎤ −1999 0 −4000 A1000 = ⎣ −2000 1 −4000 ⎦ . 1000 0 2001 (c) To compute the matrix exponential f (A) = eA , consider the polynomial functions f (x) = ex and f (x) = ex ; we have f (−1) = e−1 and f (−1) = e−1 . From these values and Equation (7.3.18), we immediately get ⎤ ⎡ −1 0 4e−1 3e eA = ⎣ 2e−1 e−1 4e−1 ⎦ . −e−1 0 −e−1

434

Eigenanalysis

(d) To compute the matrix exponential function f (A) = eAt , consider the polynomial functions f (x) = ext and f (x) = text ; it is well known that f (−1) = e−t , f (−1) = te−t . Substituting these two values into (7.3.18), we get ⎤ ⎡ −t 0 4te−t e + 2te−t eAt = ⎣ 2te−t e−t 4te−t ⎦ . −t −t −te 0 e − 2te−t (e) To compute the matrix trigonometric function sin A, consider the polynomial function f (x) = sin x, whose first-order derivative is f (x) = cos x. Substituting f (−1) = sin(−1) and f (−1) = cos(−1) into Equation (7.3.18), we immediately get ⎡ ⎤ sin(−1) + 2 cos(−1) 0 4 cos(−1) ⎦. sin A = ⎣ 2 cos(−1) sin(−1) 4 cos(−1) − cos(−1) 0 sin(−1) − 2 cos(−1) 7.4 Polynomial Matrices and Balanced Reduction The matrix polynomial and its similarity reduction presented in the above section can be used to calculate matrix powers and the matrix functions, but the determination of the Jordan canonical forms requires three key steps. (1) Solve the characteristic equation |λI − A| = 0 to get the eigenvalues λ1 , . . . , λm of A ∈ Cm×m . (2) Solve Api,j = pi,j−1 + λi pi,j , i = 1, . . . , d, j = 1, . . . , mi to determine the similarity transformation sub-matrices Pi = [pi,1 , . . . , pi,mi ], i = 1, . . . , d. (3) Find the inverse matrix P−1 = [P1 , . . . , Pd ]−1 . This section presents an alternative way to determine the Jordan canonical form of A ∈ Cm×m . The basic starting point of this idea is to transform the similarity reduction of the constant matrix A into a balanced reduction of the characteristic polynomial matrix λI − A and then to transform its Smith normal form into the Jordan canonical form of A.

7.4.1 Smith Normal Forms DEFINITION 7.8 An m × n matrix A(x) = [aij (x)], with polynomials of the argument x as entries, is called a polynomial matrix. Denote R[x]m×n as the collection of m × n real polynomial matrices and C[x]m×n as the collection of m × n complex polynomial matrices. It is easily to verify that C[x]m×n and R[x]m×n are respectively linear spaces in the complex field C and the real field R. The polynomial matrix and the matrix polynomial are two different concepts: the matrix polynomial f (A) = an An + an−1 An−1 + · · · + a1 A + a0 I is a polynomial

7.4 Polynomial Matrices and Balanced Reduction

435

with the matrix A as its argument, while the polynomial matrix A(x) is a matrix with polynomials of x as entries aij (x). DEFINITION 7.9 Let A(λ) ∈ C[λ]m×n ; then r is called the rank of the polynomial matrix A(x), denoted r = rank[A(x)], if any k ≥ (r + 1)th-order minor is equal to zero and there is at least one kth-order minor that is the nonzero polynomial in C[x]. In particular, if m = rank[A(x)], then an mth-order polynomial matrix A(x) ∈ C[x]m×m is said to be a full-rank polynomial matrix or a nonsingular polynomial matrix. DEFINITION 7.10 Let A(x) ∈ C[x]m×m . If there is another polynomial matrix B(x) ∈ C[x]m×m such that A(x)B(x) = B(x)A(x) = Im×m , then the polynomial matrix A(x) is said to be invertible, and B(x) is called the inverse of the polynomial matrix A(x), denoted B(x) = A−1 (x). For a constant matrix, its nonsingularity and invertibility are two equivalent concepts. However, the nonsingularity and invertibility of a polynomial matrix are two different concepts: nonsingularity is weaker than invertibility, i.e., a nonsingular polynomial matrix is not necessarily invertible while an invertible polynomial matrix must be nonsingular. A constant matrix A ∈ Cm×m can be reduced to the Jordan canonical form J via a similarity transformation. A natural question to ask is: what is the standard form of polynomial matrix reduction? To answer this question, we first discuss how to reduce a polynomial matrix. DEFINITION 7.11 Let A ∈ C[x]m×n , P ∈ Rm×m and d(x) ∈ R[x]. Elementary row matrices are of the following three types. The Type-I elementary row matrix Pij = [e1 , . . . , ei−1 , ej , ei+1 , . . . , ej−1 , ei , ej+1 , . . . , em ] is such that Pij A interchanges rows i and j of A. The Type-II elementary row matrix Pi (α) = [e1 , . . . , ei−1 , αei , ei+1 , . . . , em ] is such that Pi (α)A multiplies row i of A by α = 0. The Type-III elementary row matrix Pij [d(x)] = [e1 , . . . , ei , . . . , ej−1 , ej + d(x)ei , . . . , em ] is such that Pij [d(x)]A adds d(x) times row i to row j. Similarly, letting Q ∈ Rn×n , we can define three types of elementary column matrices, Qij , Qi (α) and Qij [d(x)], respectively, as follows.

436

Eigenanalysis

DEFINITION 7.12 Two m × n polynomial matrices A(x) and B(x) are said to be balanced, denoted A(x) ∼ = B(x), if there is a sequence of elementary row matrices P1 (x), . . . , Ps (x) and a sequence of elementary column matrices Q1 (x), . . . , Qt (x) such that (7.4.1) B(x) = Ps (x) · · · P1 (x)A(x)Q1 (x) · · · Qt (x) = P(x)A(x)Q(x), where P(x) is the product of the elementary row matrices and Q(x) is the product of the elementary column matrices. Balanced matrices have the following basic properties. (1) Reflexivity Any polynomial matrix A(x) is balanced with respect to itself: A(x) ∼ = A(x). (2) Symmetry B(x) ∼ = A(x) ⇔ A(x) ∼ = B(x). (3) Transitivity If C(x) ∼ = B(x) and B(x) ∼ = A(x) then C(x) ∼ = A(x). The balance of polynomial matrices is also known as their equivalence. A reduction which transforms a polynomial matrix into another, simpler, balanced polynomial matrix is known as balanced reduction. It is self-evident that, given a polynomial matrix A(x) ∈ C[x]m×n , it is desirable that it is balanced to the simplest possible form. This simplest polynomial matrix is the well-known Smith normal form. THEOREM 7.7 If the rank of a polynomial matrix A(x) ∈ C[x]m×n is equal to r, then A(x) can be balanced to the Smith normal form S(x) as follows: A(x) ∼ (7.4.2) = S(x) = Diag[σ1 (x), σ2 (x), . . . , σr (x), 0, . . . , 0], where σi (x) divides σi+1 (x) for 1 ≤ i ≤ r − 1. EXAMPLE 7.6

Given a polynomial matrix ⎡ x+1 2 A(x) = ⎣ 1 x 1 1

⎤ −6 −3 ⎦ , x−4

perform the elementary transformations of A(x) to reduce it. Solution



x+1 A(x) = ⎣ 1 1 ⎡ x−1 →⎣ 0 1 ⎡ x−1 →⎣ 0 x−1

⎤ ⎡ ⎤ ⎡ ⎤ 2 −6 x − 1 0 −2x + 2 x−1 0 0 x −3 ⎦ → ⎣ 1 x −3 ⎦ → ⎣ 1 x −1 ⎦ 1 x−4 1 1 x−4 1 1 x−2 ⎤ ⎡ ⎤ x−1 0 0 0 0 x−1 0 ⎦ x − 1 −x + 1⎦ → ⎣ 0 1 1 x−1 1 x−2 ⎤ ⎤ ⎡ x−1 0 0 0 0 ⎦→⎣ 0 ⎦ = B(x). x−1 0 x−1 0 2 2 0 0 (x − 1) x − 1 (x − 1)

7.4 Polynomial Matrices and Balanced Reduction

437

The matrix B(x) is not in Smith normal form, because |B(x)| = (x−1)4 is a fourthorder polynomial of x while |A(x)| is clearly a third-order polynomial of x. Since b11 , b22 , b33 contain the first-order factor x − 1, the first row, say, of B(x) can be divided by this factor. After this elementary row transformation, the matrix B(x) is further reduced to ⎤ ⎡ 1 0 0 ⎦. A(x) ∼ 0 = ⎣0 x − 1 2 0 0 (x − 1) This is the Smith normal form of the polynomial matrix xI − A. COROLLARY 7.1 The Smith normal form of a polynomial matrix A(x) ∈ C[x]m×n is unique. That is, every polynomial matrix A(x) can be balanced to precisely one matrix in Smith normal form. The major shortcoming of the elementary transformation method for reducing a polynomial matrix A(x) to the Smith normal form is that a sequence of elementary transformations requires a certain skill that is not easy to program, and is sometimes more trouble, than just using the matrix A(x) as it stands.

7.4.2 Invariant Factor Method To overcome the shortcoming of the elementary transformation method, it is necessary to introduce an alternative balanced reduction method. Extract any k rows and any k columns of an m × m matrix A to obtain a square matrix whose determinant is the kth minor of A. DEFINITION 7.13 The determinant rank of A is defined to be the largest integer r for which there exists a nonzero r × r minor of A. DEFINITION 7.14 Given a polynomial matrix A(x) ∈ C[x]m×n , if its rank is rank[A(x)] = r then, for a natural number k ≤ r, there is at least one nonzero kth minor of A(x). The greatest common divisor of all nonzero kth minors, denoted dk (x), is known as the kth determinant divisor of the polynomial matrix A(x). THEOREM 7.8 The kth determinant divisors dk (x) = 0 for 1 ≤ k ≤ r with r = rank(A). Also, dk−1 (x) divides dk (x), denoted dk−1 |dk , for 1 ≤ k ≤ r. Proof (from [319]) Let r = rank(A). Then there exists an r × r nonzero minor of A and hence dr (x) = 0. Then, because each r × r minor is a linear combination of (r − 1) × (r − 1) minors of A, it follows that some (r − 1) × (r − 1) minor of A is also nonzero and thus dr−1 (x) = 0. Also, dr−1 (x) divides each minor of size r − 1 and consequently divides each minor of size r, and hence dr−1 (x) divides dr (x), the greatest common divisor of all minors of size r. This argument can be repeated with r replaced by r − 1 and so on.

438

Eigenanalysis

The kth determinant divisors dk (x), k = 1, . . . , r in Theorem 7.8 are the polynomials with leading coefficient 1 (i.e., monic polynomials). DEFINITION 7.15 If rank(A(x)) = r then the invariant factors of A(x) are defined as σi (x) = di (x)/di−1 (x), 1 ≤ i ≤ r, where d0 (x) = 1. The numbers of the various minors of a given m × m polynomial matrix are as follows: 1 1 × Cm = m2 first minors; (1) Cm 2 2 (2) Cm × Cm second minors; =2 < m(m − 1) · · · (m − k + 1) k k (3) Cm × Cm = kth minors, where k = 1, . . . , m − 1; 2 × 3 × ··· × k

m = 1 mth minor. (4) Cm

For example, for the polynomial matrix ⎤ ⎡ x 0 1 A(x) = ⎣x2 + 1 x 0 ⎦, x − 1 −x x + 1 its various minors are as follows: The number of first minors is 9, in which the number of nonzero minors is 7. The number of second minors is C32 × C32 = 9 and all are nonzero minors. The number of third minors is just 1, and |A(x)| = 0. EXAMPLE 7.7

Given a polynomial matrix ⎡ 0 x(x − 1) A(x) = ⎣x 0 0 0

⎤ 0 x + 1 ⎦, −x + 1

find its Smith normal form. Solution The nonzero first minors are given by |x(x − 1)| = x2 − x,

|x| = x,

|x + 1| = x + 1,

| − x + 1| = −x + 1.

So, the first determinant divisor is d1 (x) = 1. The nonzero second minors,        0 x(x − 1)  = −x2 (x − 1), x x + 1  = −x(x − 2),    0 −x + 2 x 0   x(x − 1) 0   = x(x − 1)(x + 1),  0 x + 1 have greatest common divisor x, so the second determinant divisor is d2 (x) = x.

7.4 Polynomial Matrices and Balanced Reduction

The third determinant,   0 x(x − 1)  x 0  0 0

439

 0  x + 1  = x2 (x − 1)(x − 2), −x + 1

yields the third determinant divisor d3 (x) = x2 (x − 1)(x − 2) directly. Hence, the invariant factors are as follows: d1 (x) d2 (x) = 1, σ2 (x) = = x, d0 (x) d1 (x) d3 (x) σ3 (x) = = x(x − 1)(x − 2). d2 (x)

σ1 (x) =

Thus the Smith normal form is



1 S(x) = ⎣0 0

0 x 0

⎤ 0 ⎦. 0 x(x − 1)(x − 2)

The most common polynomial matrix is λI − A, where λ is an eigenvalue of the m × m matrix A. The polynomial matrix λI − A, in automatic control, signal processing, system engineering and so on, is often called the λ-matrix of A, and is denoted A(λ) = λI − A. EXAMPLE 7.8

Find the Smith normal form of the λ-matrix of the 3 × 3 matrix ⎡ ⎤ λ+1 2 −6 A(λ) = λI − A = ⎣ 1 λ −3 ⎦ . 1 1 λ−4

Solution Every entry consists of a first minor, and their common divisor is 1; hence the first determinant divisor d1 (λ) = 1. The second minors are respectively given by       λ + 1 2   = (λ − 1)(λ + 2), λ + 1 −6 = −3(λ − 1),   1  1 −3 λ      λ + 1 2  −6   = λ − 1, λ + 1  = (λ − 1)(λ − 2),  1  1 λ − 4 1        2 −6 2 1 λ −6       1 λ − 4 = 2(λ − 1), λ −3 = 6(λ − 1), 1 1  = −(λ − 1),     λ 1 −3  −3    1 λ − 4 = λ − 1,  1 λ − 4 = (λ − 1)(λ − 3). The greatest common divisor is λ − 1, so the second determinant divisor d2 (λ) = λ − 1.

440

Eigenanalysis

The only third minor has  λ + 1   1   1

2 λ 1

 −6  −3  = (λ − 1)3 , λ − 4

which gives the third determinant divisor d3 (λ) = (λ − 1)3 . From the above results, it is known that the invariant factors are given by σ1 (λ) =

d1 (λ) = 1, d0 (λ)

σ2 (λ) =

d2 (λ) = λ − 1, d1 (λ)

σ3 (λ) =

d3 (λ) = (λ − 1)2 , d2 (λ)

which gives the Smith normal form ⎤ ⎡ 1 0 0 ⎦. S(λ) = ⎣0 λ − 1 0 2 0 0 (λ − 1) EXAMPLE 7.9 λ-matrix

Find the invariant factors and the Smith normal form of the ⎡

λ−1 0 A(λ) = ⎣ −2 λ+1 1 0

⎤ −4 −4 ⎦ . λ+3

Solution Its determinant divisors are as follows. (1) The greatest common divisor of the nonzero first minors is 1, so the first determinant divisor d1 (λ) = 1. (2) The greatest common divisor of the nonzero second minors is λ+1, which yields directly d2 (λ) = λ + 1. (3) The determinant    λ−1 0 −4    −2 λ+1 −4  = (λ + 1)3   1 0 λ+3  gives directly the third determinant divisor d3 (λ) = (λ + 1)3 . Hence the invariant factors are σ1 (λ) =

d1 = 1, d0

σ2 (λ) =

d2 = λ + 1, d1

σ3 (λ) =

d3 = (λ + 1)2 . d2

From these invariant factors we have the Smith normal form ⎤ ⎡ 1 0 0 ⎦. S(λ) = ⎣ 0 λ + 1 0 2 0 0 (λ + 1)

7.4 Polynomial Matrices and Balanced Reduction

441

7.4.3 Conversion of Jordan Form and Smith Form We have presented the similarity reduction of matrices and the balanced reduction of polynomial matrices, respectively. An interesting issue is the relationship between the Jordan canonical form of similarity reduction and the Smith normal form of balanced reduction. Via the linear transformation ⎡ ⎤ −a12 ··· −a1m λ − a11 ⎢ −a21 λ − a22 · · · −a2m ⎥ ⎢ ⎥ (7.4.3) A(λ) = λI − A = ⎢ ⎥, .. .. .. . . ⎣ ⎦ . . . . −am2 · · · λ − amm −am1 a constant matrix A easily becomes a polynomial matrix A(λ). THEOREM 7.9 A∼B

Let A and B be two m × m (constant) matrices. Then



(xIm − A) ∼ = (xIm − B)



xIm − A and xIm − B have the same Smith normal form.

Proof [319] For the forward implication: if P−1 AP = B, where P ∈ Rm×m then P−1 (xIm − A)P = xIm − P−1 AP = xIm − B. For the backward implication: if xIm − A and xIm − B are balanced then they have the same invariant factors and so have the same nontrivial invariant factors. That is, A and B have the same invariant factors, and hence are similar. Theorem 7.9 establishes the relationship between the Jordan canonical form J of the matrix A and the Smith normal form of the λ-matrix λI − A. It may be noticed that if the polynomial matrix A(x) is not in the form of xI − A then Theorem 7.9 no longer holds: A ∼ B does not mean A(x) ∼ = B(x) if A(x) = xI − A. Let the Jordan canonical form of a matrix A be J, i.e., A ∼ J. Hence, by Theorem 7.9, we have λI − A ∼ (7.4.4) = λI − J. If the Smith normal form of the λ-matrix λI − A is S(λ), i.e., λI − A ∼ = S(λ),

(7.4.5)

then from Equations (7.4.4) and (7.4.5) and the transitivity of balanced matrices, it can immediately be seen that S(λ) ∼ = λI − J.

(7.4.6)

This is the relationship between the Jordan canonical form J of the matrix A and the Smith normal form S(λ) of the λ-matrix λI − A.

442

Eigenanalysis

Let J = Diag(J1 , . . . , Jd , 0, . . . , 0) denote the Jordan canonical form of a constant matrix A ∈ Cm×m . From Equation (7.4.6) it is known that the Smith normal form of the λ-matrix A(λ) = λI − J should also have d Smith blocks: S(λ) = Diag[S1 (λ), . . . , Sd (λ), 0 . . . , 0],

(7.4.7)

where the Smith block Si (λ) = Diag(σi,1 , . . . , σi,mi ) and m1 + · · · + md = r. Thus the Smith block Si and the Jordan blocks Ji have the same dimensions, mi × mi . Therefore, the conversion relationship between the Jordan canonical form and the Smith normal form becomes that between the Jordan block and the Smith block corresponding to the same eigenvalue λi .

7.4.4 Finding Smith Blocks from Jordan Blocks Our problem is: given the Jordan blocks Ji , i = 1, . . . , d of a constant matrix A, find the Smith blocks Si (λ), i = 1, . . . , d of the λ-matrix λI − A. According to the respective eigenvalue multiplicities, there are different corresponding relationships between Jordan blocks and Smith blocks. (1) The first-order Jordan block corresponding to the single eigenvalue λi is given by Ji = λi . From Equation (7.4.6) it is known that Ji = [λ] ⇒ Si (λ) = [λ − λi ]. (2) The second-order Jordan block Ji corresponding to the eigenvalue λi with algebraic multiplicity mi = 2 is given by     λi 0 λ − λi 0 2 2 2 ∼ Ji2 = ⇒ Si2 (λ) = λI2 − Ji2 = , 0 λi 0 λ − λi     λ λ − λi 1 −1 ⇒ S1i2 (λ) ∼ J1i2 = i = λI2 − J1i1 = 0 λi 0 λ − λi   1 0 ∼ , = 0 (λ − λi )2 because the determinant divisors of λI2 − Ji1 are d1 = 1 and d2 = (λ − λi )2 , and hence σ1 (λ) = d1 /d0 = 1 and σ2 (λ) = d2 /d1 = (λ − λi )2 . (3) The third-order Jordan block Ji is given by ⎤ ⎤ ⎡ ⎡ 0 0 λi 0 0 λ − λi J3i3 = ⎣ 0 λi 0 ⎦ ⇒ S3i3 (λ) ∼ λ − λi 0 ⎦, = λI3 − J31 ∼ =⎣ 0 0 0 λi 0 0 λ − λi ⎤ ⎤ ⎡ ⎡ λi 0 0 1 0 0 ⎦, J2i3 = ⎣ 0 λi 1 ⎦ ⇒ S2i3 (λ) ∼ 0 = λI3 − J32 ∼ = ⎣ 0 λ − λi 2 0 0 λi 0 0 (λ − λi )

7.4 Polynomial Matrices and Balanced Reduction

and



J1i3

λi =⎣ 0 0

1 λi 0

⎤ 0 1⎦ λi

443

⎤ 1 0 0 ∼ ⎦, 0 = ⎣0 1 3 0 0 (λ − λi ) ⎡

S1i3 (λ) ∼ = λI3 − J33



because the invariant factors of λI3 − J2i3 are σ1 = d1 /d0 = 1,

σ2 = d2 /d1 = λ − λi ,

σ3 = d3 /d2 = (λ − λi )2 ,

while the invariant factors of λI3 − J1i3 are σ1 = 1,

σ2 = 1,

σ3 = (λ − λi )3 .

Summarizing the above analysis, we have the corresponding relationship between the Jordan canonical form and the Smith canonical form:

J21 J22 and



J31

J32

J33

λi = ⎣0 0 ⎡ λi = ⎣0 0 ⎡ λi ⎣ = 0 0

0 λi 0 0 λi 0 1 λi 0

J1 = [λi ]   λi 0 = 0 λi   λ 1 = i 0 λi ⎤ 0 0⎦ λi ⎤ 0 1⎦ λi ⎤ 0 1⎦ λi

⇔ ⇔ ⇔

S1 (λ) = [λ − λi ];   0 λ − λi S21 (λ) = , 0 λ − λi   1 0 S22 (λ) = ; 0 (λ − λi )2

⎤ 0 0 λ − λi S31 (λ) = ⎣ 0 λ − λi 0 ⎦, 0 0 λ − λi ⎤ ⎡ 1 0 0 ⎦, S32 (λ) = ⎣0 λ − λi 0 2 0 0 (λ − λi ) ⎤ ⎡ 1 0 0 ⎦. S33 (λ) = ⎣0 1 0 3 0 0 (λ − λi )

(7.4.8) (7.4.9) (7.4.10)









(7.4.11)

(7.4.12)

(7.4.13)

7.4.5 Finding Jordan Blocks from Smith Blocks Conversely, given the Smith normal form S(λ) of a λ-matrix λI − A, how do we find the Jordan canonical form J of the matrix A? Equations (7.4.8)–(7.4.13) can also be used to find the Jordan blocks from the Smith blocks. The question is how to separate the Smith blocks from the Smith normal form. Using block-divided method of the Smith normal form, we exclude the zero-order invariant factor 1 and then separate the Smith blocks from the Smith normal form according the following two cases.

444

Eigenanalysis

Case 1

Single eigenvalue

If some ith invariant factor dependents only on one eigenvalue λk , i.e., σi (λ) = (λ − λk )ni , where ni ≥ 1, then ⎡ ⎢ ⎢ Si (λ) = ⎢ ⎣

1 ..





0

⎥ ⎥ ⎥ ⎦

. 1



(λ − λj )ni

0

⎢ ⎢ Ji = ⎢ ⎣

λj

1 ..

.

0 ..

.

λj 0



⎥ ⎥ ⎥ ∈ Cni ×ni . 1⎦

λj (7.4.14)

For example, ⎡ S(λ) = ⎣

1 (λ − λ0 ) 0



Case 2



0

⎧ ⎪ ⎨ J 1 = λ0  λ ⎪ ⎩ J2 = 0 0

(λ − λ0 )  1 , λ0

⎦ 2



⎧ ⎪ ⎨ S1 (λ) = λ − λ0 ,   ⇔ 1 0 ⎪ , ⎩ S2 (λ) = 0 (λ − λ0 )2 ⎤ ⎡ 0 λ0 0 J(λ) = ⎣ 0 λ0 1 ⎦ . 0 0 λ0

Multiple eigenvalues

If the kth invariant factor is a product of factors depending on s eigenvalues, σk (λ) = (λ − λj1 )nk,1 (λ − λj2 )nk,2 · · · (λ − λjs )nk,s ,

i = 1, . . . , s,

then the invariant factor σk (λ) can be decomposed into s different eigenvalue factors σk1 (λ) = (λ−λj1 )nk,1 , . . . , σks (λ) = (λ−λjs )nk,s . Then, using σki (λ) = (λ−λji )nk,i , the Jordan block can be obtained from the Smith block. EXAMPLE 7.10 The third invariant factor σ3 (λ) = (λ − λ1 )(λ − λ2 )2 is decomposed into two single-eigenvalue factors σ31 (λ) = λ − λ1 and σ32 (λ) = (λ − λ2 )2 ; hence ⎧ ⎤ ⎡ ⎪ 1 0 ⎨ S1 (λ) = λ − λ1 ,   ⎦ ⎣ S(λ) = ⇔ 1 1 0 ⎪ , ⎩ S2 (λ) = 0 (λ − λ1 )(λ − λ2 )2 0 (λ − λ2 )2 ⎧ ⎡ ⎤ ⎪ 0 λ1 0 ⎨ J1 = λ1 ,   ⇔ J(λ) = ⎣ 0 λ2 1 ⎦ . ⇔ λ2 1 ⎪ , = J ⎩ 2 0 0 λ2 0 λ2 EXAMPLE 7.11 The nonzero-order invariant factor of the 5 × 5 Smith normal form is (λ − λ1 )2 (λ − λ2 )3 , which is decomposed into two single-eigenvalue factors

7.4 Polynomial Matrices and Balanced Reduction

445

σ51 (λ) = (λ − λ1 )2 and σ52 (λ) = (λ − λ2 )3 . Hence we have S(λ) = Diag[1, 1, 1, 1, (λ − λ1 )2 (λ − λ2 )3 ]   ⎧ 0 ⎪ ⎪ S1 (λ) = 1 , ⎪ ⎪ 0 (λ − λ1 )2 ⎪ ⎨ ⎤ ⎡ ⇔ 1 0 0 ⎪ ⎪ ⎪ ⎦, S (λ) = ⎣0 1 0 ⎪ ⎪ ⎩ 2 0 0 (λ − λ2 )3   ⎧ ⎡ ⎪ λ1 ⎪ J 1 = λ1 1 , ⎪ ⎪ 0 λ ⎢ ⎪ 1 ⎨ ⎢0 ⎤ ⎡ ⇔ J(λ) = ⎢ ⇔ 0 λ2 1 ⎢0 ⎪ ⎪ ⎣0 ⎪ J2 = ⎣ 0 λ ⎦ , 1 ⎪ 2 ⎪ ⎩ 0 0 0 λ2

1 λ1 0 0 0

0 0 λ2 0 0

0 0 1 λ2 0

⎤ 0 0⎥ ⎥ 0⎥ ⎥. 1⎦ λ2

In addition to determinant divisors and invariant factors, there is another divisor of a polynomial matrix. DEFINITION 7.16 Decompose an invariant factor into the product of a few nonzero-order factors, i.e., write σk (λ) = (λ − λk1 )ck1 · · · (λ − λklk )cklk , where cij > 0, j = 1, . . . , lk ; then every factor (λ − λklk )cklk is called an elementary divisor of A(λ). The elementary divisors are arranged in no particular order. All elementary divisors of a polynomial matrix A(λ) constitute the elementary divisor set of A(λ). EXAMPLE 7.12 Given a polynomial matrix A(λ) ∈ C[λ]5×5 with rank 4 and Smith normal form S(λ) = Diag(1, λ, λ2 (λ − 1)(λ + 1), λ2 (λ − 1)(λ + 1)3 , 0), find the elementary divisors of A(λ). Solution From the second invariant factor σ2 (λ) = λ it is known that λ is an elementary divisor of A(λ). From the third invariant factor σ3 (λ) = λ2 (λ − 1)(λ + 1) it follows that other three elementary divisors of A(λ) are λ2 , λ − 1 and λ + 1. From the fourth invariant factor σ4 (λ) = λ2 (λ − 1)(λ + 1)3 we have the corresponding three elementary divisors of A(λ), given by λ2 , λ − 1 and (λ + 1)3 . Hence there are seven elementary divisors of the polynomial matrix A(λ): λ, λ2, λ2, λ − 1, λ − 1, λ + 1, (λ + 1)3 . Given all the elementary divisors of a polynomial matrix A(λ), its invariant factors can be reconstructed by the following method. (1) Use all the elementary divisors to form a rectangular array: the same order divisors are arranged as a row, in order from low to high according to their

446

Eigenanalysis

powers. If the number of the same-order divisors is less than r = rank(A(λ)), then make up the row with the zero-order divisor 1. (2) The invariant factors of A(λ) are obtained by respectively multiplying along the columns. EXAMPLE 7.13 Given a polynomial matrix A(λ) with rank r = 4, and seven elementary divisors, λ, λ2 , λ2 , λ − 1, λ − 1, λ + 1, (λ + 1)3 , find the invariant divisors of A(λ). Solution Use the given elementary divisors to form the rectangular array 1 λ 1 1 1 1

λ2 λ−1 λ+1

λ2 λ−1 (λ + 1)3

Multiply respectively down the columns to obtain the invariant factors of A(λ): σ1 (λ) = 1,

σ2 (λ) = λ,

σ3 (λ) = λ2 (λ − 1)(λ + 1),

σ4 (λ) = λ2 (λ − 1)(λ + 1)3 . Once the Smith normal form of the λ-matrix λI − A is transformed to the Jordan canonical form of the matrix polynomial, the matrix function can be computed. However, although this method overcomes the disadvantage of the elementary transformations, it still needs to use the similarity transformation matrix P and its inverse matrix P−1 . Can we avoid both elementary transformations and also the similarity transformation matrix together with its inverse matrix to calculate a matrix function? This is the topic of the next section.

7.5 Cayley–Hamilton Theorem with Applications The eigenvalues of an n × n matrix A are determined by roots of its characteristic polynomial det(A − λI). Moreover, the characteristic polynomial is also closely related to the calculation of the matrix inverse, matrix power and matrix exponential function, so a more in-depth discussion and analysis on characteristic polynomial is needed.

7.5.1 Cayley–Hamilton Theorem The Cayley–Hamilton theorem is an important result of regarding the characteristic polynomial of a general matrix. Using this theorem many problems can be easily solved, such as finding the inverse of matrices and the computation of several matrix functions. In order to introduce the Cayley–Hamilton theorem, we first present a few important concepts on polynomials. As stated earlier, when pn = 0, n is called the order of the polynomial p(x) =

7.5 Cayley–Hamilton Theorem with Applications

447

pn xn + pn−1 xn−1 + · · · + p1 x + p0 . An nth-order polynomial is said to be monic if the coefficient of xn is equal to 1. If p(A) = pn An + pn−1 An−1 + · · · + p1 A + p0 I = O, then p(x) = pn xn + pn−1 xn−1 + · · · + p1 x + p0 is the polynomial “annihilating” the matrix A, and it is simply called the annihilating polynomial. For an n × n matrix A, let m be the smallest integer such that the powers I, A, . . . , Am are linearly dependent. Hence, we have pm Am + pm−1 Am−1 + · · · + p1 A + p0 I = On×n .

(7.5.1)

The polynomial p(x) = pm xm + pm−1 xm−1 + · · · + p1 x + p0 is said to be the minimal polynomial of the matrix A. As atated in the following theorem, the characteristic polynomial p(x) = det(A− xI) is the annihilating polynomial of the n × n matrix A. THEOREM 7.10 (Cayley–Hamilton theorem) Any square matrix An×n satisfies its own characteristic equation: if the characteristic equation is p(x) = |A − xI| = pn xn + pn−1 xn−1 + · · · + p1 x + p0 = 0 then pn An + pn−1 An−1 + · · · + p1 A + p0 I = O,

(7.5.2)

where I and O are the n × n identity matrix and null matrix, respectively. Proof Let p(A) = pn An + pn−1 An−1 + · · · + p1 A + p0 I. If (λ, u) is an eigenpair of the matrix A, i.e., Au = λu, then A2 u = λAu = λ2 u,

A3 u = λA2 u = λ3 u,

··· ,

An u = λn u.

Therefore we have p(A)u = (pn An + pn−1 An−1 + · · · + p1 A + p0 I)u = pn An u + pn−1 An−1 u + · · · + p1 Au + p0 u = pn λn u + pn−1 λn−1 u + · · · + p1 λu + p0 u = (pn λn + pn−1 λn−1 + · · · + p1 λ + p0 )u.

(7.5.3)

Because the eigenvalue λ is a root of the characteristic equation p(x) = 0, we get p(λ) = |A − λI| = pn λn + pn−1 λn−1 + · · · + p1 λ + p0 = 0.

(7.5.4)

Substitute this result into Equation (7.5.3) to yield p(A)u = 0 · u = 0, Hence, p(A) = O.

for any eigenvector u = 0.

(7.5.5)

448

Eigenanalysis

The Cayley–Hamilton theorem has many interesting and important applications. For example, by using this theorem, it can be shown directly that two similar matrices have the same eigenvalues. Consider the characteristic polynomials of two similar matrices. Let B = S−1 AS be a matrix similar to A, and let its characteristic polynomial p(x) = det(A−xI) = pn xn + pn−1 xn−1 + · · · + p1 x + p0 . By the Cayley–Hamilton theorem, we know that p(A) = pn An + pn−1 An−1 + · · · + p1 A + p0 I = O. For the similar matrix B, because Bk = (S−1 AS)(S−1 AS) · · · (S−1 AS) = S−1 Ak S, we have p(B) = pn Bn + pn−1 Bn−1 + · · · + p1 B + p0 I = pn S−1 An S + pn−1 S−1 An−1 S + · · · + p1 S−1 AS + p0 I   = S−1 pn An + pn−1 An−1 + · · · + p1 A + p0 I S = S−1 p(A)S =O since p(A) = O by the Cayley–Hamilton theorem. In other words, two similar matrices A ∼ B have the same characteristic polynomials and hence have the same eigenvalues. In the following, we discuss some important applications of the Cayley–Hamilton theorem.

7.5.2 Computation of Inverse Matrices If a matrix An×n is nonsingular, then premultiply (or postmultiply) both sides of Equation (7.5.2) by A−1 to yield directly pn An−1 + pn−1 An−2 + · · · + p2 A + p1 I + p0 A−1 = O, from which we get a computation formula for the inverse matrix: A−1 = − EXAMPLE 7.14 matrix

1 (p An−1 + pn−1 An−2 + · · · + p2 A + p1 I). p0 n

(7.5.6)

Use the Cayley–Hamilton theorem to compute the inverse of the ⎡

2 A = ⎣0 0

0 2 0

⎤ 1 0⎦ . 3

7.5 Cayley–Hamilton Theorem with Applications

Solution The second power ⎡ 2 A2 = ⎣ 0 0

449

of A is given by ⎤ ⎤ ⎡ ⎤⎡ 4 0 5 2 0 1 0 1 2 0⎦ ⎣0 2 0⎦ = ⎣0 4 0⎦ 0 0 9 0 0 3 0 3

and the characteristic polynomial is   2 − x 0 1   det(A − xI) =  0 2−x 0  = (2 − x)2 (3 − x)  0 0 3 − x = −x3 + 7x2 − 16x + 12, i.e., p0 = 12, p1 = −16, p2 = 7, p3 = −1. Substitute these values together with the matrix A and its second power A2 into the matrix inverse formula (7.5.6):  1  −A2 + 7A − 16I 12 ⎛ ⎡ ⎤ ⎡ 4 0 5 2 1 = − ⎝− ⎣0 4 0⎦ + 7 ⎣0 12 0 0 9 0 ⎡ ⎤ 3 0 −1 1 = ⎣0 3 0 ⎦ . 6 0 0 2

A−1 = −

0 2 0

⎤ ⎡ 1 1 ⎦ ⎣ − 16 0 0 3 0

0 1 0

⎤⎞ 0 0⎦⎠ 1

The Cayley–Hamilton theorem also can be used to compute the generalized inverse matrix of any complex matrix. THEOREM 7.11 [119]

Let A be an m × n complex matrix, and

f (λ) = (−1)m (a0 λm + a1 λm−1 + · · · + am−1 λ + am ),

a0 = 1,

(7.5.7)

be the characteristic polynomial det(AAH − λI) of the matrix product AAH . If k is the largest integer such that ak = 0 then the generalized inverse matrix of A is determined by ) * H (AAH )k−1 + a1 (AAH )k−2 + · · · + ak−2 (AAH ) + ak−1 I . A† =−a−1 k A (7.5.8) When k = 0 is the largest integer such that ak = 0, the generalized inverse matrix A† = O. The above theorem is sometimes called the Decell theorem. Notice that the largest integer k in the Decell theorem is the rank of the matrix A, i.e., k = rank(A). From this theorem, Decell presented the following method for computing the Moore– Penrose inverse matrix [119].

450

Eigenanalysis

(1) Form the matrix sequence A0 , A1 , . . . , Ak , where the scalars qi are defined in the second column and where the matrices Bi are recursively computed from the first two columns: −1 = q0 ,

A0 = O,

B0 = I

A1 = AA ,

tr(A1 ) = q1 ,

B1 = A1 − q1 I

A2 = AAH B1 ,

tr(A2 ) = q2 , 2 .. .

B2 = A2 − q2 I

H

.. . Ak−1 = AAH Bk−2 , Ak = AAH Bk−1 ,

tr(Ak−1 ) = qk−1 , k−1 tr(Ak ) = qk , k

.. . Bk−1 = Ak−1 − qk−1 I Bk = Ak − qk I.

Faddeev [149, pp. 260–265] showed that the coefficients constructed in this way satisfy qi = −ai , i = 1, . . . , k. (2) Compute the Moore–Penrose inverse matrix H k−1 H A† = −a−1 + a1 (AAH )k−2+ · · · + ak−2 (AAH )+ ak−1 I] k A [(AA ) H = −a−1 k A Bk−1 .

(7.5.9)

7.5.3 Computation of Matrix Powers Given a matrix An×n and an integer k, we say that Ak is the kth power of the matrix A. If k is large, calculation of the matrix power Ak is a very tedious process. Fortunately, the Cayley–Hamilton theorem provides a simplest solution to this computation problem. Consider the polynomial division f (x)/g(x), where g(x) = 0. According to Euclidean division, it is known that there exist two polynomials q(x) and r(x) such that f (x) = g(x)q(x) + r(x), where q(x) and r(x) are respectively called the quotient and the remainder, and either the order of the remainder r(x) is less than the order of the quotient g(x) or r(x) = 0. Let the characteristic polynomial of the matrix A be given by p(x) = pn xn + pn−1 xn−1 + · · · + p1 x + p0 . Then, for any x, its Kth power is given by xK = p(x)q(x) + r(x).

(7.5.10)

When x is a root of the characteristic equation p(x) = 0, the above equation

7.5 Cayley–Hamilton Theorem with Applications

451

becomes xK = r(x) = r0 + r1 x + · · · + rn−1 xn−1 ,

(7.5.11)

because the order of r(x) is less than the order n of p(x). Replacing the scalar x by the matrix A, Equation (7.5.10) becomes AK = p(A)q(A) + r(A).

(7.5.12)

According to the Cayley–Hamilton theorem, if p(x) is the characteristic polynomial of the matrix A then p(A) = O. Hence Equation (7.5.12) reduces to AK = r(A) = r0 I + r1 A + · · · + rn−1 An−1 .

(7.5.13)

Equation (7.5.13) provides Algorithm 7.2 for computing AK . Algorithm 7.2

Calculation of matrix powers [229, p. 172]

input: A ∈ Rn×n and the integer K. 1. Compute n roots (eigenvalues) λ1 , . . . , λn of the characteristic polynomial p(x) = det(A − xI) = pn xn + pn−1 xn−1 + · · · + p1 x + p0 . 2. Solve the system of linear equations r0 + λ1 r1 + · · · + λn−1 rn−1 = λK 1 , 1 r0 + λ2 r1 + · · · + λn−1 rn−1 = λK 2 , 2 .. . r0 + λn r1 + · · · + λn−1 rn−1 = λK n n , to obtain the solutions r0 , r1 , . . . , rn−1 . 3. Calculate the matrix power AK = r0 I + r1 A + · · · + rn−1 An−1 . output: AK .

 1 1 , calculate the matrix power A731 . Given the matrix A = 2 1 

EXAMPLE 7.15

Solution Form the characteristic polynomial   1 − x 1   = x2 − 2x − 1. p(x) = det(A − xI) =  2 1 − x √ √ Put p(x) = 0 to get the eigenvalues of A: λ1 = 1 + 2 and λ2 = 1 − 2. Solve the system of linear equations √ √ r0 + (1 + 2)r1 = (1 + 2)731 , √ √ r0 + (1 − 2)r1 = (1 − 2)731 ,

452

Eigenanalysis

to get the solutions 

 √  √  √ 731 √ 1 1 2 2 r0 = − (1 + 2) + + (1 − 2)731 , 2 4 2 4 √ √ √ 2 r1 = [(1 + 2)731 − (1 − 2)731 ]. 4

Finally, A731 = r0 I2 + r1 A.

7.5.4 Calculation of Matrix Exponential Functions Mimicking the scalar exponential function eat = 1 + at +

1 2 2 1 a t + · · · + ak tk + · · · , 2! k!

for a given matrix A, one can define the matrix exponential function as eAt = I + At +

1 2 2 1 A t + · · · + Ak t k + · · · . 2! k!

(7.5.14)

The matrix exponential function can be used to represent the solutions of certain first-order differential equations. In engineering applications we often meet the firstorder linear differential equation ˙ x(t) = Ax(t),

x(0) = x0 ,

where A is a constant matrix. The solution of the above differential equation can be written as x(t) = eAt x0 . Hence, solving the linear first-order differential equation is equivalent to computing the matrix exponential function eAt . Using the Cayley–Hamilton theorem, it can be shown that the solution of an nth-order linear matrix differential equation is uniquely determined. THEOREM 7.12 [289] nomial is

Let A be an n × n constant matrix; its characteristic poly-

p(λ) = det(λI − A) = λn + cn−1 λn−1 + · · · + c1 λ + c0 .

(7.5.15)

If the nth-order matrix differential equation Φ(n) (t) + cn−1 Φ(n−1) (t) + · · · + c1 Φ (t) + c0 Φ(t) = O

(7.5.16)

satisfies the initial conditions Φ(0) = I, Φ (0) = A, Φ

(0) = A2 ,

...

, Φ(n−1) = An−1

(7.5.17)

then Φ(t) = eAt is the unique solution of the nth-order matrix differential equation (7.5.16).

7.5 Cayley–Hamilton Theorem with Applications

THEOREM 7.13 [289] polynomial be

453

Let A be an n × n constant matrix and its characteristic

p(λ) = det(λI − A) = λn + cn−1 λn−1 + · · · + c1 λ + c0 , then the solution of the matrix differential equation (7.5.16) satisfying the initial condition (7.5.17) is given by Φ(t) = eAt = x1 (t)I + x2 (t)A + x3 (t)A2 + · · · + xn (t)An−1 ,

(7.5.18)

where xk (t) is the solution of the nth-order scalar differential equation x(n) (t) + cn−1 x(n−1) (t) + · · · + c1 x (t) + c0 x(t) = 0 satisfying the conditions ⎫ x1 (0) = 1⎪ ⎪ ⎪ ⎪ x 1 (0) = 0⎪ ⎬ .. ⎪ , . ⎪ ⎪ ⎪ ⎪ ⎭ (n−1) (0) = 0 x1

⎫ x2 (0) = 0⎪ ⎪ ⎪ ⎪ x 2 (0) = 1⎪ ⎬ .. . (n−1)

x2

(0) = 0

⎪ ⎪ ⎪ ⎪ ⎪ ⎭

(7.5.19)

⎫ xn (0) = 0⎪ ⎪ ⎪ ⎪ ⎪ x n (0) = 0⎬ ,

...

,

.. . (0) = 1 x(n−1) n

⎪ ⎪ ⎪ ⎪ ⎪ ⎭

.

Equation (7.5.18) gives an efficient method for computing the matrix exponential function eAt . EXAMPLE 7.16

Given a matrix



⎤ 1 0 1 A = ⎣0 1 0⎦ , 0 0 2

use Theorem 7.13 to compute eAt . Solution From the characteristic equation   λ − 1 0 −1   det(λI − A) =  0 λ−1 0  = (λ − 1)2 (λ − 2) = 0,  0 0 λ − 2 we get the eigenvalues of A: λ1 = 1, λ2 = 1, λ3 = 2, where the multiplicity of the eigenvalue 1 is 2. From Theorem 7.13 it is known that the matrix exponential function is given by eAt = x1 (t)I + x2 (t)A + x3 (t)A2 , where x1 (t), x2 (t), x3 (t) are the solutions of the third-order scalar differential equation x

(t) + c2 x

(t) + c1 x (t) + c0 x(t) = 0

454

Eigenanalysis

subject to the initial conditions ⎫ x1 (0) = 1⎪ ⎬

x 1 (0) = 0 , ⎪ ⎭ x

1 (0) = 0

⎫ x2 (0) = 0⎪ ⎬

x 2 (0) = 1 , ⎪ ⎭ x

2 (0) = 0

⎫ x3 (0) = 0⎪ ⎬ x 3 (0) = 0

x

3 (0)

=1

⎪ ⎭

.

From the eigenvalues λ1 = 1, λ2 = 1, λ3 = 2 it is known that the general solution of the scalar differential equation x

(t) + c2 x

(t) + c1 x (t) + c0 x(t) = 0 is given by x(t) = a1 tet + a2 et + a3 e2t . From the general solution formula it is easy to proved as follows. (a) Substitute the initial condition x1 (0) = 1, x 1 (0) = 0, x

1 (0) = 0 into the general solution formula to give ⎧ ⎪ ⎨ ⎪ ⎩

a2 + a 3 = 1 a1 + a2 + 2a3 = 0



2a1 + a2 + 4a3 = 0

⎧ ⎪ ⎨ a1 = −2 a2 = 0 ⎪ ⎩ a3 = 1

i.e., the special solution satisfying those initial conditions is x1 (t) = −2tet + e2t . (b) From the initial conditions x2 (0) = 0, x 2 (0) = 1, x

2 (0) = 0 we obtain ⎧ ⎪ ⎨ ⎪ ⎩

a2 + a 3 = 0 a1 + a2 + 2a3 = 1



2a1 + a2 + 4a3 = 0

⎧ ⎪ ⎨ a1 = 1 a2 = 2 ⎪ ⎩ a3 = −1

and hence we have that the special solution satisfying those initial conditions is x2 (t) = tet + 2et − e2t . (c) From the initial conditions x3 (0) = 0, x 3 (0) = 0, x

3 (0) = 1 we get ⎧ ⎪ ⎨ ⎪ ⎩

a2 + a 3 = 0 a1 + a2 + 2a3 = 0 2a1 + a2 + 4a3 = 1



⎧ ⎪ ⎨ a1 = −1 a2 = −1 ⎪ ⎩ a3 = 1.

In other words, the special solution satisfying those initial conditions is x3 (t) = −tet − et + e2t . Moreover, from ⎡

⎤⎡ ⎤ ⎡ 1 0 1 1 0 1 1 A2 = ⎣0 1 0⎦ ⎣0 1 0⎦ = ⎣0 0 0 2 0 0 2 0

0 1 0

⎤ 3 0⎦ 4

7.6 Application Examples of Eigenvalue Decomposition

455

and Theorem 7.13 we have eAt = x1 (t)I + x2 (t)A + x3 (t)A2 ⎡ ⎤ ⎡ 1 0 0 1 = (−2tet + e2t ) ⎣0 1 0⎦ + (tet + 2et − e2t ) ⎣0 0 0 1 0 ⎡ ⎤ 1 0 3 + (−tet − et + e2t ) ⎣0 1 0⎦ , 0 0 4

0 1 0

⎤ 1 0⎦ 2

which gives ⎡

eAt

−2tet + et + e2t =⎣ 0 0

0 −2tet + et + e2t 0

⎤ −2tet − et + 2e2t ⎦. 0 −4tet + 3e2t

Summarizing the above discussions, we may conclude that the Cayley–Hamliton theorem provides a very effective tool for matrix inversion, computation of matrix powers, solving linear differential equations (or equivalently computing the matrix exponential function) and so on.

7.6 Application Examples of Eigenvalue Decomposition Eigenvalue decomposition has wide applications. In this section we present three typical applications of EVD: Pisarenko harmonic decomposition, the Karhunen– Loeve transformation and principal component analysis.

7.6.1 Pisarenko Harmonic Decomposition Harmonic processes are often encountered in many engineering applications, and often it is necessary to determine the frequencies and powers of these harmonics. This is so-called harmonic retrieval. Here we present the harmonic retrieval method of Pisarenko [394]. Consider a harmonic process consisting of p real sinusoids: x(n) =

p 

Ai sin(2πfi n + θi ).

(7.6.1)

i=1

When each phase θi is a constant, the above harmonic process is a deterministic process that is nonstationary. In order to guarantee the stationarity of the harmonic process, it is generally assumed that the phases θi are random numbers distributed uniformly in [−π, π]. In this case, the harmonic process becomes stochastic. A harmonic process can be described by a difference equation. First consider a

456

Eigenanalysis

single sinusoid. For simplicity, let the harmonic signal be x(n) = sin(2πf n + θ). Recall the trigonometric identity sin(2πf n + θ) + sin(2πf (n − 2) + θ) = 2 cos(2πf ) sin(2πf (n − 1) + θ). Substituting x(n) = sin(2πf n+θ) into the above identity, we have the second-order difference equation x(n) − 2 cos(2πf )x(n − 1) + x(n − 2) = 0. ∞ Make the z transformation X(z) = n=−∞ x(n)z −n to get (1 − 2 cos(2πf )z −1 + z −2 )X(z) = 0. Hence the characteristic equation is 1 − 2 cos(2πf )z −1 + z −2 = 0 which has a pair of complex conjugate roots, i.e., z = cos(2πf ) ± j sin(2πf ) = e±j 2πf . Note that the modulus of the conjugate roots is equal to 1, i.e., |z1 | = |z2 | = 1. From the roots we can determine the frequencies of the sinusoids:   1 Imzi . (7.6.2) fi = arctan 2π Rezi If p real sinusoidal signals have no repeated frequencies then the p frequencies are determined by the roots of the characteristic polynomial p (

(z − zi )(z −

zi∗ )

i=1

=

2p 

ai z 2p−i = 0

i=0

or 1 + a1 z −1 + · · · + a2p−1 z −(2p−1) + a2p z −2p = 0.

(7.6.3)

It is easily seen that all the moduli of these roots are equal to 1. Because these roots appear in the form of conjugate pairs, in the coefficients of the characteristic polynomial (7.6.3) there exist the symmetries ai = a2p−i ,

i = 0, 1, . . . , p,

(7.6.4)

and the difference equation corresponding to (7.6.3) is x(n) +

2p  i=1

where n is a time variable.

ai x(n − i) = 0,

(7.6.5)

7.6 Application Examples of Eigenvalue Decomposition

457

A sinusoidal signal is generally observed to have additive white noise. Letting the additive white noise be e(n), then the observed signal is given by y(n) = x(n) + e(n) =

p 

Ai sin(2πfi n + θi ) + e(n),

(7.6.6)

i=1

where e(n) ∼ N (0, σe2 ) is a Gaussian white noise that is statistically independent of the sinusoidal signal x(n). Substituting x(n) = y(n) − e(n) into (7.6.5), we immediately have a difference equation satisfied by the sinusoidal signal with white noise as follows: y(n) +

2p 

ai y(n − i) = e(n) +

i=1

2p 

ai e(n − i).

(7.6.7)

i=1

This is a special case of an autoregressive moving-average (ARMA) process. To derive the normal equation obeyed by the autoregressive parameters of this special ARMA process, define the vector y(n) = [y(n), y(n − 1), . . . , y(n − 2p)]T , w = [1, a1 , . . . , a2p ]T ,

(7.6.8)

e(n) = [e(n), e(n − 1), . . . , e(n − 2p)] . T

Then, (7.6.7) can be written as yT (n)w = eT (n)w.

(7.6.9)

Premultiplying (7.6.9) by the vector y(n) and taking the expectation yields E{y(n)yT (n)}w = E{y(n)eT (n)}w.

(7.6.10)

Letting Ry (k) = E{y(n + k)y(n)} denote the autocorrelation function of the observation data y(n), we have ⎡ ⎤ Ry (0) Ry (−1) ··· Ry (−2p) ⎢ Ry (1) Ry (0) · · · Ry (−2p + 1)⎥ ⎥ def ⎢ E{y(n)yT (n)} = ⎢ . ⎥ = R, . .. .. .. ⎦ ⎣ .. . . Ry (0) Ry (2p) Ry (2p − 1) · · · E{y(n)eT (n)} = E{[x(n) + e(n)]eT (n)} = E{e(n)eT (n)} = σe2 I. Substituting the above two relations into (7.6.10), we get the important normal equation Rw = σe2 w.

(7.6.11)

Here σe2 is the smallest eigenvalue of the autocorrelation matrix R = E{y(n)yT (n)} of the observation process {y(n)}, and the coefficient vector w is the eigenvector associated with this smallest eigenvalue. This is the basis of the Pisarenko harmonic decomposition method.

458

Eigenanalysis

Note that the smallest eigenvalue σe2 of the autocorrelation matrix R is just the variance of the white noise e(n), and the other eigenvalues correspond to the powers of the harmonic signals. When the signal-to-noise ratio is large, the eigenvalue σe2 is clearly smaller than other eigenvalues. Hence, the Pisarenko harmonic decomposition informs us that the harmonic retrieval problem becomes the EVD of the sample correlation matrix R: the coefficient vector w of the characteristic polynomial of the harmonic process is the eigenvector corresponding to the smallest eigenvalue σe2 of R.

7.6.2 Discrete Karhunen–Loeve Transformation In many engineering applications, it is necessary to represent observed samples of a random signal as a set of coefficients, and it is to be hoped that this new signal representation has some desirable properties. For example, for encoding it is desired that the signal can be represented by a small number of coefficients and that these coefficients represent the power of the original signal. For optimal filtering, it is desirable that the converted samples are statistically uncorrelated, so that we can reduce the complexity of the filter or enhance the signal to noise ratio. The common way to achieve these goals is to expand the signal as a linear combination of orthogonal basis functions. If a set of orthogonal basis functions is selected according to the autocorrelation matrix of the signal observation samples, it is possible to obtain the signal representation with the minimum mean square error (MMSE) among all possible orthogonal basis functions. In the MMSE sense, such a signal representation is the optimal signal representation. This signal transformation, proposed by Karhunen and Loeve for a continuous random signal, is called the Kauhunen–Loeve (KL) transformation. Later, Hotelling extended this transformation to a discrete random signal, so it is also called the Hotelling transformation. However, in most of the literature, this is still called the discrete Karhunen–Loeve transformation. Let x = [x1 , . . . , xM ]T be a zero-mean random vector whose autocorrelation matrix is Rx = E{xxH }. Consider the linear transformation w = QH x,

(7.6.12)

where Q is a unitary matrix, i.e., Q−1 = QH . Then, the original random signal vector x can be expressed as the linear combination of the coefficient of the vector w, namely x = Qw =

M 

wi q i ,

qH i qj = 0,

i = j,

(7.6.13)

i=1

where qi are the column vectors of Q. In order to reduce the number of transform coefficients wi , it is assumed that

7.6 Application Examples of Eigenvalue Decomposition

459

only the first m coefficients w1 , . . . , wm , m = 1, . . . , M are used to approximate the random signal vector x, i.e., ˆ= x

m 

1 ≤ m ≤ M.

wi q i ,

(7.6.14)

i=1

Then, the error of the mth-order approximation of a random-signal vector is given by ˆ= em = x − x

M 

m 

wi q i −

i=1

M 

wi q i =

i=1

w i qi .

(7.6.15)

i=m+1

This yields the mean squared error Em =

E{eH m em }

=

M 

2 qH i E{|wi | }qi

=

i=m+1

M 

E{|wi |2 }qH i qi .

(7.6.16)

i=m+1

2 H From wi = qH i x it is easily seen that E{|wi | } = qi Rx qi . If we have the H constraint qi qi = 1 then the mean squared error expressed by (7.6.16) can be rewritten as M M   E{|wi |2 } = qH (7.6.17) Em = i Rx q i i=m+1

i=m+1

subject to qH i qi = 1,

i = m + 1, m + 2, . . . , M.

In order to minimize the mean squared error, use the Lagrange multiplier method to form the cost function J=

M  i=m+1

Let

M 

qH i Rx qi +

λi (1 − qH i qi ).

i=m+1

∂J = 0, i = m + 1, m + 2, . . . , M , i.e., ∂q∗i

∂ ∂q∗i



M 

qH i Rx qi

i=m+1

M 

+

λi (1 −

 qH i qi )

= Rx q i − λ i q i = 0

i=m+1

for i = m + 1, m + 2, . . . , M ; then we have R x q i = λi q i ,

i = m + 1, m + 2, . . . , M.

(7.6.18)

This transformation is known as the Karhunen–Loeve (KL) transformation. The above discussion shows that when using (7.6.14) to approximate a random signal vector x, in order to minimize the mean squared error of approximation, we should select Lagrange multipliers λi and the orthogonal basis vector of the cost function gi to be respectively the last M − m eigenvalues and the corresponding

460

Eigenanalysis

eigenvectors of Rx . In other words, the orthogonal basis for the random-signal vector in (7.6.14) should be the first m eigenvectors of Rx . Let the EVD of the M × M autocorrelation Rx be Rx =

M 

λi u i u H i ,

(7.6.19)

i=1

then the selected orthogonal basis vectors in (7.6.14) are qi = ui , i = 1, . . . , m. If the autocorrelation matrix Rx has only K large eigenvalues, and the other M − K eigenvalues are negligible, then the approximation order m in (7.6.14) should equal K, and hence the Kth-order discrete Kauhunen–Loeve expansion is obtained: K  ˆ= x w i ui , (7.6.20) i=1

where wi , i = 1, . . . , K, is the ith element of the K × 1 vector w = UH 1 x.

(7.6.21)

Here U1 = [u1 , . . . , uK ] consists of the eigenvectors corresponding to the K leading eigenvalues of the autocorrelation matrix. In this case, the mean squared error of the Kth-order discrete Karhunen–Loeve expansion is ⎛ ⎞ M M M M     H⎝ H⎠ EK = uH R u = u λ u u = λi . (7.6.22) u x i j j j i i i i=K+1

i=K+1

j=1

i=K+1

Because λi , i = K + 1, . . . , M , are the minor eigenvalues of the autocorrelation matrix Rx , the mean squared error EK is very small. Suppose that the original data to be transmitted are x1 , . . . , xM , the direct transmitting of these data arriving at the transmitter may bring two problems: they may easily be received by others; and the data length M can be very large in many cases. For example, an image needs to be converted into data by rows, and then all these data are synthesized to a long segment. By using the discrete Karhunen– Loeve expansion, we can avoid the above two difficulties of direct data emission. Let M samples of the image or voice signals to be compressed and transmitted be xc (0), xc (1), . . . , xc (M − 1), where M is large. If we perform an EVD of the autocorrelation matrix of the given data and determine the number K of principal eigenvalues, then we obtain K linear transformation coefficients w1 , . . . , wK and K orthogonal eigenvectors u1 , . . . , uK . Thus we need only to transmit K coefficients w1 , . . . , wK at the transmitter. If the K eigenvectors are stored at the receiver then we can utilize K  ˆ= wi u i (7.6.23) x i=1

7.6 Application Examples of Eigenvalue Decomposition

461

to reconstruct the M pieces of transmitted data xc (i), i = 0, 1, . . . , M − 1. The process of transforming the M pieces of signal data xc (0), xc (1), . . . , xc (M − 1) into K coefficients w1 , . . . , wK is called signal encoding or data compression, while the process of reconstructing M data signals from K coefficients is known as signal decoding. Figure 7.1 shows this signal encoding and decoding schematically. x(n)

discrete KL

uncoded signal

transformation

wn

low dimension signal orthogonal basis choice

ˆi w encoding signal

discrete KL inverse

x ˆ(n)

transformation

Figure 7.1 Discrete KL transformation based signal encoding.

The ratio M/K is called the compression ratio. If K is much smaller than M then the compression ratio is large. Obviously, when the discrete KL transformation is used to encode the original data, not only is the length of the transmitted data greatly compressed but also, even if the K encoding coefficients are received by others, it is difficult for them to accurately reconstruct the original data without knowing the K principal eigenvectors.

7.6.3 Principal Component Analysis Consider an index set of P statistically correlated characters {x1 , . . . , xP }. Due to correlation, there is redundant information among the P characters. We hope to find a new set of K mutually orthogonal characters {˜ xp 1 , . . . , x ˜pK } where x ˜ pi , i = 1, . . . , K, is the ith principal component in which there is no redundant information. Such a process is known as feature extraction. Via feature extraction, a higherdimensional vector space with redundant information becomes a lower-dimensional vector space without redundant information. Such an operation is called dimensionality reduction. A typical dimensionality reduction method is principal component analysis. Through an orthogonal transformation, P statistically correlated original characters are transformed to K mutually orthogonal new characters. Because the K new characters {˜ xp1 , . . . , x ˜pK } have the larger powers |˜ xp1 |2 , . . . , |˜ xpK |2 , they can be regarded as the principal components of the P original character indexes x1 , . . . , xP . Hence, data or signal analysis based on K principal components is known as the principal component analysis (PCA). Thus the main objective of PCA is to use K (< P ) principal components to express P statistically correlated variables. In order to fully reflect the useful information carried by the P variables, each principal component should be a linear combination of the original P variables. DEFINITION 7.17

Let Rx be the autocorrelation matrix of a P × 1 data vector

462

Eigenanalysis

x. If the P × P matrix Rx has K principal eigenvalues then their corresponding K eigenvectors are called the principal components of the data vector x. The main steps and ideas of the PCA are as follows. (1) Dimensionality reduction The P original variables are synthesized into K principal components: x ˜j =

P 

a∗ij xi = aH j x,

j = 1, 2, . . . , K,

(7.6.24)

i=1

where aj = [a1j , . . . , aP j ]T and x = [x1 , . . . , xP ]T . (2) Orthogonalization In order to make K principal components orthonormal, i.e., ˜ xi , x ˜j  = xH xaH i aj = δij , the coefficient vectors ai must satisfy the orthonormal condition aH i aj = δij , H as all components of x are statistically correlated, i.e., x x = 0. (3) Power maximization Selecting ai = ui , i = 1, . . . , K, where ui are the K eigenvectors corresponding to the K leading eigenvalues λ1 ≥ · · · ≥ λK of the autocorrelation matrix Rx = E{xxH }, it is easy to compute the energy of each nonredundant component: H ∗ H H H Ex˜i = E{|˜ xi |2 } = E{aH i x(ai x) } = ui E{xx ui } = ui Rx ui = λi .

Because the eigenvalues are arranged in nondescending order Ex˜1 ≥ Ex˜2 ≥ · · · ≥ Ex˜k , x ˜1 is often referred to as the first principal component of x, x ˜2 as the second principal component of x and so on. For the P × P autocorrelation matrix ⎡ E{|x1 |2 } E{x1 x∗2 } · · · ⎢ ⎢ E{x2 x∗1 } E{|x2 |2 } · · · Rx = E{xxH } = ⎢ .. .. ⎢ .. ⎣ . . . E{xP x∗1 } E{xP x∗2 } · · ·

⎤ E{x1 x∗P } ⎥ E{x2 x∗P }⎥ ⎥, .. ⎥ ⎦ . E{|xP |2 }

(7.6.25)

by the definition and property of the matrix trace, it can be seen that tr(Rx ) = E{|x1 |2 } + · · · + E{|xP |2 } = λ1 + · · · + λP .

(7.6.26)

If the autocorrelation matrix Rx has only K leading eigenvalues then E{|x1 |2 } + · · · + E{|xP |2 } ≈ λ1 + · · · + λK .

(7.6.27)

Summarizing the above discussion, we may conclude that the basic idea of PCA is to transform the original P statistically correlated random data into K mutually

7.7 Generalized Eigenvalue Decomposition (GEVD)

463

orthogonal principal components by using dimensionality reduction, orthogonalization and energy maximization. The sum of the energies of these principal components is approximately equal to the sum of the energies of the original P random data. DEFINITION 7.18 [520] Let Rx be the autocorrelation matrix of a P × 1 data vector x with K principal eigenvalues and P − K minor eigenvalues (i.e., small eigenvalues). The P − K eigenvectors corresponding to these minor eigenvalues are called the minor components of the data vector x. Data or signal analysis based on the P − K minor components is known as minor component analysis (MCA). The PCA can provide the contour and main information about a signal or an image. In contrast, MCA can provide the details of a signal or the texture of an image. It has been widely used in many fields such as frequency estimation [317], [318], blind beamforming [189], moving-target display [251], clutter cancellation [24], etc. In signal or image analysis and pattern recognition, when the PCA cannot distinguish two object signals or images, the MCA is used to compare their details. 7.7 Generalized Eigenvalue Decomposition (GEVD) The EVD of an n × n single matrix can be extended to the EVD of a matrix pair or pencil consisting of two matrices. The EVD of a matrix pencil is called generalized eigenvalue decomposition (GEVD). As a matter of fact, the EVD is a special example of the GEVD. 7.7.1 Generalized Eigenvalue Decomposition The basis of the EVD is the eigensystem expressed by the linear transformation L[u] = λu: taking the linear transformation L[u] = Au, one gets the EVD Au = λu. Now consider the generalization of the eigensystem composed of two linear systems La and Lb both of whose inputs are the vector u, but the output La [u] of the first system La is some constant times λ the output Lb [u] of the second system Lb . Hence the eigensystem is generalized as [231] La [u] = λLb [u] (u = 0);

(7.7.1)

this is called a generalized eigensystem, denoted (La , Lb ). The constant λ and the nonzero vector u are known as the generalized eigenvalue and the generalized eigenvector, respectively. In particular, if two linear transformations are respectively La [u] = Au and Lb [u] = Bu then the generalized eigensystem becomes Au = λBu.

(7.7.2)

464

Eigenanalysis

The two n × n matrices A and B compose a matrix pencil (or matrix pair), denoted (A, B). The constant λ and the nonzero vector u are respectively referred to as the generalized eigenvalue and the generalized eigenvector of the matrix pencil (A, B). A generalized eigenvalue and its corresponding generalized eigenvector are collectively called a generalized eigenpair, denoted (λ, u). Equation (7.7.2) shows that the ordinary eigenvalue problem is a special example of the generalized eigenvalue problem when the matrix pencil is (A, I). Although the generalized eigenvalue and the generalized eigenvector always appear in pairs, the generalized eigenvalue can be found independently. To find the generalized eigenvalue independently, we rewrite the generalized characteristic equation (7.7.2) as (A − λB)u = 0.

(7.7.3)

In order to ensure the existence of a nonzero solution u, the matrix A − λB cannot be nonsingular, i.e., the determinant must be equal to zero: (A − λB) singular



det(A − λB) = 0.

(7.7.4)

The polynomial det(A − λB) is called the generalized characteristic polynomial of the matrix pencil (A, B), and det(A − λB) = 0 is known as the generalized characteristic equation of the matrix pencil (A, B). Hence, the matrix pencil (A, B) is often expressed as A − λB. The generalized eigenvalues of the matrix pencil (A, B) are all solutions z of the generalized characteristic equation det(A − zB) = 0, including the generalized eigenvalues equal to zero. If the matrix B = I then the generalized characteristic polynomial reduces to the characteristic polynomial det(A − λI) = 0 of the matrix A. In other words, the generalized characteristic polynomial is a generalization of the characteristic polynomial, whereas the characteristic polynomial is an special example of the generalized characteristic polynomial of the matrix pencil (A, B) when B = I. The generalized eigenvalues λ(A, B) are defined as    λ(A, B) = z ∈ C det(A − zB) = 0 . (7.7.5) THEOREM 7.14 [224] The pair λ ∈ C and u ∈ Cn are respectively the generalized eigenvalue and the generalized eigenvector of the n × n matrix pencil (A, B) if and only if |A − λB| = 0 and u ∈ Null(A − λB) with u = 0. The following are the properties of the GEVD Au = λBu [230, pp. 176–177]. 1. If A and B are exchanged, then every generalized eigenvalue will become its reciprocal, but the generalized eigenvector is unchanged, namely Au = λBu



Bu =

1 Au. λ

7.7 Generalized Eigenvalue Decomposition (GEVD)

465

2. If B is nonsingular then the GEVD becomes the standard EVD of B−1 A: Au = λBu



(B−1 A)u = λu.

3. If A and B are real positive definite matrices then the generalized eigenvalues must be positive. 4. If A is singular then λ = 0 must be a generalized eigenvalue. 5. If both A and B are positive definite Hermitian matrices then the generalized eigenvalues must be real, and H uH i Auj = ui Buj = 0,

i = j.

Strictly speaking, the generalized eigenvector u described above is called the right generalized eigenvector of the matrix pencil (A, B). The left generalized eigenvector corresponding to the generalized eigenvalue λ is defined as the column vector v such that vH A = λvH B.

(7.7.6)

Let both X and Y be nonsingular matrices; then from (7.7.2) and (7.7.6) one has XAu = λXBu,

vH AY = λvH BY.

(7.7.7)

This shows that premultiplying the matrix pencil (A, B) by a nonsingular matrix X does not change the right generalized eigenvector u of the matrix pencil (A, B), while postmultiplying (A, B) by any nonsingular matrix Y does not change the left generalized eigenvector v. Algorithm 7.3 uses a contraction mapping to compute the generalized eigenpairs (λ, u) of a given n × n real symmetric matrix pencil (A, B). Algorithm 7.3

Lanczos algorithm for GEVD [422, p. 298]

given: n × n real symmetric matrices A, B. initialization: Choose u1 such that u1 2 = 1, and set α1 = 0, z0 = u0 = 0, z1 = Bu1 and i = 1. repeat 1. Compute u = Aui − αi zi−1 . βi = u, ui . u = u − β i zi . w = B−1 u. αi+1 = w, u. ui+1 = w/αi+1 . zi+1 = u/αi+1 . λi = βi+1 /αi+1 . 2. exit if i = n. return i ← i + 1. output: (λi , ui ), i = 1, . . . , n.

466

Eigenanalysis

Algorithm 7.4 is a tangent algorithm for computing the GEVD of a symmetric positive definite matrix pencil. Algorithm 7.4

Tangent algorithm for computing the GEVD [136]

given: n × n real symmetric matrices A, B. repeat 1. Compute ΔA = Diag(A11 , . . . , Ann )−1/2 , As = ΔA AΔA , B1 = ΔA BΔA . 2. Calculate the Cholesky decomposition RTA RA = As , RTB RB = ΠT B1 Π.

3. Solve the matrix equation FRB = AΠ for F = AΠR−1 B . 4. Compute the SVD Σ = VFUT of F. 5. Compute X = ΔA ΠR−1 B U. 6. exit if AX = BXΣ2 . repeat output: X, Σ.

When the matrix B is singular, Algorithms 7.3 and 7.4 will be unstable. The GEVD algorithm for the matrix pencil (A, B) with singular matrix B was proposed by Nour-Omid et al. [354], this is shown in Algorithm 7.5. The main idea of this algorithm is to introduce a shift factor σ such that A − σB is nonsingular. GEVD algorithm for singular matrix B [354], [422]  initialization: Choose a basis w, z1 = Bw, α1 = w, z1 . Set u0 = 0, i = 1. repeat 1. Compute ui = w/αi . zi = (A − σB)−1 w. w = w − αi ui−1 . βi = w, zi . zi+1 = Bw. ( αi+1 = zi+1 , w.

Algorithm 7.5

λi = βi /αi+1 . 2. exit if i = n. return i ← i + 1. output: (λi , ui ), i = 1, . . . , n.

DEFINITION 7.19 Two matrix pencils with the same generalized eigenvalues are known as equivalent matrix pencils. From the definition of the generalized eigenvalue det(A − λB) = 0 and the

7.7 Generalized Eigenvalue Decomposition (GEVD)

467

properties of a determinant it is easy to see that det(XAY − λXBY) = 0



det(A − λB) = 0.

Therefore, premultiplying and/or postmultiplying one or more nonsingular matrices, the generalized eigenvalues of the matrix pencil are unchanged. This result can be described as the following proposition. PROPOSITION 7.1 If X and Y are two nonsingular matrices then the matrix pencils (XAY, XBY) and (A, B) are equivalent. The GEVD can be equivalently written as αAu = βBu as well. In this case, the generalized eigenvalue is defined as λ = β/α.

7.7.2 Total Least Squares Method for GEVD In applications of the GEVD we are only interested in the nonzero generalized eigenvalues, because their number reflects the number of signal components and the nonzero generalized eigenvalues themselves often imply useful information about the signal parameters. However, the number of signal components is usually unknown and needs to be estimated. Generally, the dimension of the matrix pencil (A, B) is larger than the number of signal components. Moreover, the two matrices A and B are usually the autocorrelation and cross-correlation matrices consisting of the observed data vectors, respectively. Because the observation data are short and noisy, the sample autocorrelation and cross-correlation matrices have the large estimation errors. Owing to the large dimension of the matrix pencil and the large estimation errors of the sample autocorrelation and cross-correlation matrices, the estimation of the nonzero generalized eigenvalues of the matrix pencil is a least squares problem. As shown by Roy and Kailath [419], use of a least squares estimator can lead to some potential numerical difficulties in solving the generalized eigenvalue problem. To overcome this difficulty, a higher-dimensional ill-conditioned LS problem needs to be transformed into a lower-dimensional well-conditioned LS problem. Without changing its nonzero generalized eigenvalues, a higher-dimensional matrix pencil can be transformed into a lower-dimensional matrix pencil by using a truncated SVD. Consider the GEVD of the matrix pencil (A, B). Let the SVD of A be given by    Σ1 O V1H H A = UΣV = [U1 , U2 ] , (7.7.8) O Σ2 V2H where Σ1 is composed of p principal singular values. Under the condition that the generalized eigenvalues remain unchanged, we can premultiply the matrix pencil A − γB by UH 1 and postmultiply it by V1 to get [543]: H UH 1 (A − γB)V1 = Σ1 − γU1 BV1 .

(7.7.9)

468

Eigenanalysis

Hence the original n × n matrix pencil (A, B) becomes a p × p matrix pencil H (Σ1 , UH 1 BV1 ). The GEVD of the lower-dimensional matrix pencil (Σ1 , U1 BV1 ) is called the total least squares (TLS) method for the GEVD of the higher-dimensional matrix pencil (A, B), which was developed in [543].

7.7.3 Application of GEVD: ESPRIT Consider a linear combination x(n) at time n of p harmonic signals subject to white noise: p  si e jn ωi + w(n), (7.7.10) x(n) = i=1

where si and ωi ∈ (−π, π) are the complex amplitude and frequency of the ith harmonic signal. Suppose that w(n) is a complex Gaussian white noise with zero mean and variance σ 2 : E{w(k)w∗ (l)} = σ 2 δ(k − l),

E{w(k)w(l)} = 0,

∀ k, l.

Our problem is to estimate the number p and frequencies ω1 , . . . , ωp of the harmonic signals using only the observation data x(1), . . . , x(N ). def

Define y(n) = x(n + 1). Choose m > p, and introduce the m × 1 vectors def

x(n) = [x(n), x(n + 1), . . . , x(n + m − 1)]T , def

w(n) = [w(n), w(n + 1), . . . , w(n + m − 1)]T , def

y(n) = [y(n), y(n + 1), . . . , y(n + m − 1)]T = [x(n + 1), x(n + 2), . . . , x(n + m)]T , def

a(ωi ) = [1, e j ωi , . . . , e j (m−1)ωi ]T . Then Equation (7.7.10) can be written as in vector form x(n) = As(n) + w(n).

(7.7.11)

y(n) = AΦs(n) + w(n + 1),

(7.7.12)

Moreover,

where def

A = [a(ω1 ), a(ω2 ), . . . , a(ωp )], def

s(n) = [s1 e j ω1 n , s2 e j ω2 n , . . . , sp e j ωp n ]T , def

Φ = Diag(e j ω1 , e j ω2 , . . . , e j ωp ). Because y(n) = x(n + 1), the vector y(n) can be regarded as a shifted version of x(n). Hence Φ is called the rotation operator since the shift is a simple rotation.

7.7 Generalized Eigenvalue Decomposition (GEVD)

469

The autocorrelation matrix of the observation vector x(n) is given by Rxx = E{x(n)xH (n)} = APAH + σ 2 I,

(7.7.13)

where P = E{s(n)sH (n)} is the autocorrelation matrix of the signal vector s(n). The cross-correlation matrix of x(n) and y(n) is defined as Rxy = E{x(n)yH (n)} = APΦH AH + σ 2 Z,

(7.7.14)

where σ 2 Z = E{w(n)wH (n + 1)}. It is easy to verify that Z is an m × m special matrix given by ⎡ ⎤ 0 0 ⎢1 0 ⎥ ⎢ ⎥ (7.7.15) Z=⎢ ⎥. .. .. ⎣ ⎦ . . 0 1 0 That is, all entries on the subdiagonal below the main diagonal are equal to 1 while all other entries are equal to 0. The problem is to estimate the number p, the frequencies ωi and the harmonic powers |si |2 (i = 1, . . . , p) of the harmonic signals by using the autocorrelation matrix Rxx and the cross-correlation matrix Rxy . Importantly, the data vector x and its shifted vector y have the same autocorredef def lation matrices, i.e., Rxx = E{x(n)xH (n)} = E{x(n + 1)xH (n + 1)} = Ryy . Making the EVD of Rxx , we get its minimum eigenvalue λmin = σ 2 . Construct a new matrix pair Cxx = Rxx − λmin I = Rxx − σ 2 I = APAH ,

(7.7.16)

Cxy = Rxy − λmin Z = Rxy − σ Z = APΦ A . 2

H

H

(7.7.17)

Now use Cxx and Cxy to constitute a matrix pencil (Cxx , Cxy ). We have Cxx − γCxy = AP(I − γΦH )AH .

(7.7.18)

Owing to the full column rank of A and the nonsingularity of P, the above equation can be written as rank(Cxx − γCxy ) = rank(I − γΦH ).

(7.7.19)

When γ = e j ωi , i = 1, . . . , p, the matrix I − γΦ is nonsingular, whereas it is singular or rank-deficit when γ = e j ωi , because γe−j ωi = 1. This shows that e j ωi , i = 1, . . . , p, are the generalized eigenvalues of the matrix pencil (Cxx , Cxy ). This result is given in the following theorem. THEOREM 7.15 [419] Define Γ as the generalized eigenvalue matrix of a matrix pencil (Cxx , Cxy ), where Cxx = Rxx − λmin I, Cxy = Rxy − λmin Z and λmin is the minimum eigenvalue of the autocorrelation matrix Rxx . If the matrix P is

470

Eigenanalysis

nonsingular then the matrix Γ and the rotation operator matrix Φ have the following relationship:   Φ O Γ= , (7.7.20) O O i.e., the nonzero entries of Γ are an arrangement of the entries of the rotation operator Φ. Roy and Kailath [419] proposed a method for estimating signal parameters via rotational invariance techniques (ESPRIT). See Algorithm 7.6. Algorithm 7.6

Basic ESPRIT algorithm 1 [419]

input: x(1), . . . , x(N ). 1. Estimate Rxx (i, j) =

N 1  x(n)x∗ (n − i + j), N n=1

Rxy (i, j) =

N 1  x(n)x∗ (n + 1 − i + j) N n=1

for i, j = 0, 1, . . . , m − 1. 2. Form Rxx = [Rxx (i, j)]m−1,m−1 , Rxy = [Rxy (i, j)]m−1,m−1 . i,j=0 i,j=0 3. Make the EVD of Rxx to estimate the minimum eigenvalue as σ 2 . 4. Compute Cxx = Rxx − σ 2 I and Cxy = Rxy − σ 2 Z. 5. Make the GEVD of (Cxx , Cxy ) to find p generalized eigenvalues e j ωi . output: p and ω1 , . . . , ωp .

This basic ESPRIT method is sometimes called the LS-ESPRIT algorithm. The TLS-ESPRIT algorithm, shown in Algorithm 7.7, can improve the numerical performance of the LS-ESPRIT algorithm. Algorithm 7.7

TLS–ESPRIT algorithm [543]

input: x(1), . . . , x(N ). 1. Estimate Rxx (i, j) =

N 1  x(n)x∗ (n − i + j), N n=1

Rxy (i, j) =

N 1  x(n)x∗ (n + 1 − i + j) N n=1

for i, j = 0, 1, . . . , m − 1. 2. Form Rxx = [Rxx (i, j)]m−1,m−1 , Rxy = [Rxy (i, j)]m−1,m−1 . i,j=0 i,j=0 3. Make the EVD of Rxx to estimate the minimum eigenvalue σ 2 . 4. Compute Cxx = Rxx − σ 2 I and Cxy = Rxy − σ 2 Z. 5. Make the SVD of Cxx , and determine p and the matrices Σ1 , U1 , V1 . 6. Make the GEVD of (Σ1 , UH 1 Cxy V1 ) to estimate the frequencies ω1 , . . . , ωp . output: p and ω1 , . . . , ωp .

It has been shown that the LS-ESPRIT and the TLS-ESPRIT algorithms give the

7.7 Generalized Eigenvalue Decomposition (GEVD)

471

same asymptotic estimation accuracy for large samples, but the estimation accuracy of the TLS-ESPRIT is better than that of the LS-ESPRIT for small samples.

7.7.4 Similarity Transformation in GEVD Consider an isometric line array that consists of m elements, as shown in Figure 7.2. y

∇1

∇2





m−1 ∇

m ∇

x Subarray 1 Subarray 2

Figure 7.2 An array is divided into two subarrays.

This isometric line array is divided into two subarrays, where subarray 1 consists of the first to the (m − 1)th elements, and subarray 2 consists of the second to the mth elements. Let the m×N matrix X = [x(1), . . . , x(N )] denote the observation data matrix of the original array, where x(n) = [x1 (n), . . . , xm (n)]T is the observation data vector, consisting of m elements, at time n and N is the data length, i.e., n = 1, . . . , N . Letting S = [s(1), . . . , s(N )]

(7.7.21)

denote the signal matrix, where s(n) = [s1 (n), . . . , sp (n)]T are the signal vectors, then for N snapshots, Equation (7.7.10) can be represented in matrix form as X = [x(1), . . . , x(N )] = AS,

(7.7.22)

where A is an m × p array response matrix. Let J1 and J2 be two (m − 1) × m selection matrices such that J1 = [Im−1 , 0m−1 ],

(7.7.23)

J2 = [0m−1 , Im−1 ],

(7.7.24)

where Im−1 is a (m − 1) × (m − 1) identity matrix, and 0m−1 is a (m − 1) × 1 zero vector. Premultiply the observation matrix X respectively by the selection matrices J1

472

Eigenanalysis

and J2 : X1 = J1 X = [x1 (1), . . . , x1 (N )],

(7.7.25)

X2 = J2 X = [x2 (1), . . . , x2 (N )],

(7.7.26)

where x1 (n) = [x1 (n), x2 (n), . . . , xm−1 (n)]T , T

x2 (n) = [x2 (n), x3 (n), . . . , xm (n)] ,

n = 1, . . . , N, n = 1, . . . , N.

(7.7.27) (7.7.28)

That is, the observation data submatrix X1 consists of the top m − 1 rows of X, which corresponds to the observation data matrix of subarray 1; and X2 consists of the lower m − 1 rows of X, which corresponds to the observation data matrix of subarray 2. Put     first row A1 = A= ; (7.7.29) last row A2 then, from the array response matrix A of the isometric line array it can be seen that there is the following relationship between the submatrices A1 and A2 : A2 = A1 Φ,

(7.7.30)

where Φ is a rotation operator matrix defined earlier. It can be shown that X1 = A1 S,

(7.7.31)

X2 = A2 S = A1 ΦS.

(7.7.32)

Because Φ is a unitary matrix, X1 and X2 have the same signal subspace and the same noise subspace; namely subarray 1 and subarray 2 have the same observation space (signal subspace + noise subspace). This is the physical explanation of the shift invariance of an isometric line array. From Equation (7.7.13) we get the following EVD:   H  Us Σs O H 2 Rxx = APA + σ I = [Us Un ] 2 O σ I UH n  H U   s 2 H = U s Σs U H (7.7.33) = U s Σs σ 2 U n s + σ Un Un . H Un H Since [Us , Un ][Us , Un ]H = I, i.e., I − Un UH n = Us Us , Equation (7.7.33) yields H APAH + σ 2 Us UH s = U s Σs U s .

(7.7.34)

Postmultiplying both sides of the above equation by Us , and noticing that UH s Us = I, we have Us = AT,

(7.7.35)

7.7 Generalized Eigenvalue Decomposition (GEVD)

473

where T is a nonsingular matrix given by T = AH Us (Σs − σ 2 I)−1 .

(7.7.36)

Although T is unknown, it is simply a virtual parameter in the following analysis in which we use only its nonsingularity. Postmultiply (7.7.29) by T to yield     A1 T first row . (7.7.37) = AT = A2 T last row Adopting the same block form, we divide Us into     U1 first row = . Us = U2 last row

(7.7.38)

Because AT = Us , comparing (7.7.37) with (7.7.38), we immediately have U1 = A1 T and

U2 = A2 T.

(7.7.39)

Substituting (7.7.30) into (7.7.39), we get U2 = A1 ΦT.

(7.7.40)

By (7.7.39) and (7.7.40), we have also U1 T−1 ΦT = A1 TT−1 ΦT = A1 ΦT = U2 .

(7.7.41)

Ψ = T−1 ΦT;

(7.7.42)

Define

then the matrix Ψ is similar to Φ and thus they have the same eigenvalues, i.e., the eigenvalues of Ψ are also e j φm , m = 1, 2, . . . , M . Finally, substituting (7.7.42) into (7.7.41), we get an important relationship: U2 = U1 Ψ.

(7.7.43)

Equation (7.7.43) suggests an alternative to the basic ESPRIT algorithm, which is given in Algorithm 7.8. Algorithm 7.8

Basic ESPRIT algorithm 2

input: x(1), . . . , x(N ). 1. Estimate Rxx (i, j) =

N N 1  1  x(n)x∗ (n − i + j), Rxy (i, j) = x(n)x∗ (n + 1 − i + j) N n=1 N n=1

for i, j = 0, 1, . . . , m − 1. m−1,m−1 2. Form Rxx = [Rxx (i, j)]m−1,m−1 , Rxy = [Rxy (i, j)]i,j=0 . i,j=0

3. Compute the EVD Rxx = UΣUH . 4. Use U and the p principal eigenvalues of Rxx to form Us . 5. Extract the upper and lower m − 1 rows of Us to constitute U1 and U2 . −1 H 6. Compute the EVD of Ψ = (UH U1 U2 to get the p eigenvalues e j ωi . 1 U1 )

output: p and ω1 , . . . , ωp .

474

Eigenanalysis

7.8 Rayleigh Quotient In physics and information technology, one often meets the maximum or minimum of the quotient of the quadratic function of a Hermitian matrix. This quotient has two forms: the Rayleigh quotient (also called the Rayleigh–Ritz ratio) of a Hermitian matrix and the generalized Rayleigh quotient (or generalized Rayleigh–Ritz ratio) of two Hermitian matrices.

7.8.1 Definition and Properties of Rayleigh Quotient In his study of the small oscillations of a vibrational system, in order to find the appropriate generalized coordinates Rayleigh [411] in the 1930s proposed a special form of quotient, later called the Rayleigh quotient. DEFINITION 7.20 The Rayleigh quotient or Rayleigh–Ritz ratio R(x) of a Hermitian matrix A ∈ Cn×n is a scalar and is defined as R(x) = R(x, A) =

xH Ax , xH x

(7.8.1)

where x is a vector to be selected such that the Rayleigh quotient is maximized or minimized. The Rayleigh quotient has the following important properties [374], [375], [93], [205]. 1. Homogeneity If α and β are scalars then R(αx, βA) = βR(x, A).

(7.8.2)

2. Shift invariance R(x, A − αI) = R(x, A) − α.   3. Orthogonality x ⊥ A − R(x)I x. 4. Boundedness When the vector x lies in the range of all nonzero vectors, the Rayleigh quotient R(x) falls in a complex plane region (called the range of A). This region is closed, bounded and convex. If A is a Hermitian matrix such that A = AH then this region is a closed interval [λ1 , λn ]. 5. Minimum residual For all vectors x = 0 and all scalars μ,

(A − R(x)I)x ≤ (A − μI)x .

(7.8.3)

The boundedness of the Rayleigh quotient of a Hermitian matrix can be strictly described as in the following theorem. THEOREM 7.16 (Rayleigh–Ritz theorem) Let A ∈ Cn×n be a Hermitian matrix whose eigenvalues are arranged in increasing order: λmin = λ1 ≤ λ2 ≤ · · · ≤ λn−1 ≤ λn = λmax ;

(7.8.4)

7.8 Rayleigh Quotient

475

then max

xH Ax xH Ax = max = λmax , H x x xH x=1 xH x

if Ax = λmax x,

(7.8.5)

min

xH Ax xH Ax = min = λmin , xH x xH x=1 xH x

if Ax = λmin x.

(7.8.6)

x=0

and x=0

More generally, the eigenvectors and eigenvalues of A correspond respectively to the critical points and the critical values of the Rayleigh quotient R(x). This theorem has various proofs, see e.g., [179], [205] or [107].

7.8.2 Rayleigh Quotient Iteration Let A ∈ C be a diagonalizable matrix, and let its eigenvalues be λi and the corresponding eigenvectors be ui . For convenience, suppose that the matrix A is nonsingular, its first eigenvalue is larger than the others and λ1 > λ2 ≥ · · · ≥ λn ; then the eigenvalue λ1 and its corresponding eigenvector u1 are called the principal eigenvalue and the principal eigenvector of A, respectively. Since λi = R(xi ), Rayleigh proposed an iteration method for computing the principal eigenvalues and the principal eigenvectors when A is a Hermitian matrix. This method was later called Rayleigh quotient iteration. Algorithm 7.9 shows the standard Rayleigh quotient iteration algorithm for the Hermitian matrix A. n×n

Algorithm 7.9

Rayleigh quotient iteration algorithm

input: Hermitian matrix A. initialization: Choose x1 with x1 2 = 1. repeat 1. Compute Rk = R(xk ) = xH k Axk . 2. exit if A − Rk I is singular. 3. Calculate xk+1 =

(Ak − Rk I)−1 xk .

(Ak − Rk I)−1 xk

return k ← k + 1. Find the nonzero solution of (A − Rk I)xk+1 = 0. output: x = xk+1 .

The iterative result {Rk , xk } is called the Rayleigh sequence generated by Rayleigh iteration. The Rayleigh sequence has the following properties [374]. (1) Scale invariance Two matrices αA (α = 0) and A generate the same Rayleigh sequence.

476

Eigenanalysis

(2) Shift invariance If A generates a Rayleigh sequence {Rk , xk } then the Rayleigh sequence generated by the matrix A − αI is {Rk − α, xk }. (3) Unitary similarity Let U be a unitary matrix, then the Rayleigh sequence generated by UAUH is {Rk , Uxk }. The extended Rayleigh quotient iteration algorithm [374] is available for a general matrix A, as shown in Algorithm 7.10. If A = AH , then the above extension algorithm reduces to the standard Rayleigh quotient iteration algorithm. Algorithm 7.10

Rayleigh quotient iteration for general matrix [374]

input: A. initialization: Choose x0 with x0 2 = 1. repeat 1. Compute Rk = R(xk ) = xH k Axk . H 2. Solve xH k+1 (A − Rk I) = τk xk for xk+1 , and let xk+1 =

xk+1 .

xk+1 2

3. Calculate Rk+1 = R(xk+1 ) = xH k+1 Axk+1 . 4. exit if A − Rk I or A − Rk+1 I is singular. 5. Calculate xk+1 =

(Ak − Rk I)−1 xk .

(Ak − Rk I)−1 xk 2

6. Solve (A − Rk+1 I)xk+2 = τk+1 xk+1 for xk+2 , and let xk+2 =

xk+2 .

xk+2 2

return k ← k + 2. Find the nonzero solution of (A − Rk I)xk+1 = 0 or (A − Rk+1 I)xk+1 = 0. output: x = xk+1 .

7.8.3 Algorithms for Rayleigh Quotient Choose the negative direction of the gradient of the Rayleigh quotient as the gradient flow of the vector x, i.e., x˙ = −(A − R(x)I)x;

(7.8.7)

then the vector x can be iteratively computed using a gradient algorithm as follows [205]: xk+1 = xk + μx˙ k = xk − μ(A − R(xk )I)xk .

(7.8.8)

As described in the following theorem, the gradient algorithm for solving the Rayleigh quotient problem has a faster convergence rate than the standard Rayleigh quotient iteration algorithm. THEOREM 7.17 Suppose that λ1 > λ2 . For an initial value x0 such that x0 = 1, the vector xk computed iteratively by the gradient algorithm converges exponentially to the maximum eigenvector u1 or −u1 of the matrix A at a rate λ1 − λ2 .

7.8 Rayleigh Quotient

Proof

477

[205, p. 18].

We now consider a conjugate gradient algorithm for solving the Rayleigh quotient problem (7.8.1), where A is a real symmetric matrix. Starting from some initial vector x0 , the conjugate gradient algorithm uses the iteration formula xk+1 = xk + αk pk

(7.8.9)

to update and approximate the minimum (or maximum) eigenvector of the matrix A. The real coefficient αk is given by [437], [526] *  1 ) αk = ± (7.8.10) −B + B 2 − 4CD , 2D where the positive sign is applied to update the minimum eigenvector, and the negative sign corresponds to the update of the maximum eigenvector. The calculation formulas of the parameters D, B, C in Equation (7.8.10) are as follows: D = Pb (k)Pc (k) − Pa (k)Pd (k),

(7.8.11)

B = Pb (k) − λk Pd (k),

(7.8.12)

C = Pa (k) − λk Pc (k),

(7.8.13)

Pa (k) = pTk Axk /(xTk xk ),

(7.8.14)

where

Pb (k) = Pc (k) = Pd (k) = λk =

pTk Apk /(xTk xk ), pTk xk /(xTk xk ), pTk pk /(xTk xk ), R(xk ) = xTk Axk /(xTk xk ).

(7.8.15) (7.8.16) (7.8.17) (7.8.18)

At the (k + 1)th iteration, the search direction is selected in the following way: pk+1 = rk+1 + b(k)pk ,

(7.8.19)

where b(−1) = 0 and rk+1 is the residual error vector at the (k + 1)th iteration and is given by 1 rk+1 = − ∇x∗ R(xk+1 ) = (λk+1 xk+1 − Axk+1 )/(xTk+1 xk+1 ). 2

(7.8.20)

The selection of b(k) in Equation (7.8.19) should make the search directions pk+1 and pk conjugate or orthogonal with respect to the Hessian matrix of the Rayleigh quotient: pTk+1 Hpk = 0.

(7.8.21)

478

Eigenanalysis

Compute the Hessian matrix of the Rayleigh quotient directly to get [526]:    T ∂R(x) 2 ∂R(x) T − R(x)I , (7.8.22) H(x) = T A− x −x x x ∂x ∂x where ∂R(x) 2 x R(x) 2 = − T 2 xT Ax + T Ax = − T x + T Ax. ∂x (x x) x x x x x x

(7.8.23)

When x = xk+1 , denote the Hessian matrix as Hk+1 = H(xk+1 ). In this case, the parameter b(k) is given by b(k) = −

rT Apk + (rTk+1 rk+1 )(xTk+1 pk ) rTk+1 Hk+1 pk = − k+1 T . pk Hk+1 pk pk (Apk − λk+1 I)pk

(7.8.24)

Equations (7.8.9)–(7.8.19) and Equation (7.8.24) together form a conjugate gradient algorithm for solving the Rayleigh quotient problem (7.8.1). This algorithm was proposed by Yang et al. [526].

7.9 Generalized Rayleigh Quotient The extended form of the Rayleigh quotient is the generalized Rayleigh quotient. This section presents the definition of and the solution method for the generalized Rayleigh quotient together with its typical applications in pattern recognition and wireless communications.

7.9.1 Definition and Properties DEFINITION 7.21 Let A and B be two n × n Hermitian matrices, and let B be positive definite. The generalized Rayleigh quotient or generalized Rayleigh–Ritz ratio R(x) of the matrix pencil (A, B) is a scalar, and is defined as xH Ax , (7.9.1) xH Bx where x is a vector to be selected such that the generalized Rayleigh quotient is maximized or minimized. R(x) =

˜ = B1/2 x, In order to find the generalized Rayleigh quotient define a new vector x 1/2 where B denotes the square root of the positive definite matrix B. Substitute −1/2 ˜ x=B x into the definition formula of the generalized Rayleigh quotient (7.9.1); then we have  H   ˜ ˜ H B−1/2 A B−1/2 x x R(˜ x) = . (7.9.2) ˜H x ˜ x This shows that the generalized Rayleigh quotient of the matrix pencil (A, B) is

7.9 Generalized Rayleigh Quotient

479

equivalent to the Rayleigh quotient of the matrix product C = (B−1/2 )H A(B−1/2 ). ˜ is selected as By the Rayleigh–Ritz theorem, it is known that, when the vector x the eigenvector corresponding to the minimum eigenvalue λmin of C, the general˜ is ized Rayleigh quotient takes a minimum value λmin , while when the vector x selected as the eigenvector corresponding to the maximum eigenvalue λmax of C, the generalized Rayleigh quotient takes the maximum value λmax . Consider the EVD of the matrix product (B−1/2 )H A(B−1/2 ):  −1/2 H  −1/2  ˜ = λ˜ A B B x x. (7.9.3) n If B = i=1 βi vi viH is the EVD of the matrix B then B1/2 =

n  

βi vi viH

i=1

and B1/2 B1/2 = B. Because the matrix B1/2 and its inverse B−1/2 have the same eigenvector and reciprocal eigenvalues, we have B−1/2 =

n  1  vi viH . βi i=1

(7.9.4)

This shows that B−1/2 is also a Hermitian matrix, so that (B−1/2 )H = B−1/2 . Premultiplying Equation (7.9.3) by B−1/2 and substituting (B−1/2 )H = B−1/2 , it follows that ˜ = λB−1/2 x ˜ or B−1 AB−1/2 x

B−1 Ax = λx,

˜ . Hence, the EVD of the matrix product (B−1/2 )H A(B−1/2 ) is as x = B−1/2 x equivalent to the EVD of B−1 A. Because the EVD of B−1 A is the GEVD of the matrix pencil (A, B), the above analysis can be summarized as follows: the conditions for the maximum and the minimum of the generalized Rayleigh quotient are: xH Ax = λmax , xH Bx xH Ax R(x) = H = λmin , x Bx

R(x) =

if we choose Ax = λmax Bx,

(7.9.5)

if we choose Ax = λmin Bx.

(7.9.6)

That is to say, in order to maximize the generalized Rayleigh quotient, the vector x must be taken as the generalized eigenvector corresponding to the maximum generalized eigenvalue of the matrix pencil (A, B). In contrast, for the generalized Rayleigh quotient to be minimized, the vector x should be taken as the generalized eigenvector corresponding to the minimum generalized eigenvalue of the matrix pencil (A, B). In the following, two applications of the generalized Rayleigh quotient are presented.

480

Eigenanalysis

7.9.2 Effectiveness of Class Discrimination Pattern recognition is widely applied in recognitions of human characteristics (such as a person’s face, fingerprint, iris) and various radar targets (such as aircraft, ships). In these applications, the extraction of signal features is crucial. For example, when a target is considered as a linear system, the target parameter is a feature of the target signal. The divergence is a measure of the distance or dissimilarity between two signals and is often used in feature discrimination and evaluation of the effectiveness of class discrimination. Let Q denote the common dimension of the signal-feature vectors extracted by various methods. Assume that there are c classes of signals; the Fisher class separability measure, called simply the Fisher measure, between the ith and the jth classes is used to determine the ranking of feature vectors. Consider the class (l) discrimination of c classes of signals. Let vk denote the kth sample feature vector of the lth class of signals, where l = 1, . . . , c and k = 1, · · · , lK , with lK the number of sample feature vectors in the lth signal class. Under the assumption that the (l) prior probabilities of the random vectors v(l) = vk are the same for k = 1, . . . , lK (i.e., equal probabilities), the Fisher measure is defined as ) *2  (l) (l) mean (v ) − mean (mean(v )) k l l=i,j k k , (7.9.7) m(i,j) =  (l) l=i,j vark (vk ) where: (l) meank (vk ) is the mean (centroid) of all signal-feature vectors of the lth class; (l) var(vk ) is the variance of all signal-feature vectors of the lth class; (l) meanl (meank (vk )) is the total sample centroid of all classes. As an extension of the Fisher measure, consider the projection of all Q×1 feature vectors onto the (c − 1)th dimension of class discrimination space. Put N = N1 + · · · + Nc , where Ni represents the number of feature vectors of the ith class signal extracted in the training phase. Suppose that si,k = [si,k (1), . . . , si,k (Q)]T denotes the Q × 1 feature vector obtained by the kth group of observation data of the ith signal in the training phase, while mi = [mi (1), . . . , mi (Q)]T is the sample mean of the ith signal-feature vector, where mi (q) =

Ni 1  si,k (q), Ni k=1

i = 1, . . . , c, q = 1, . . . , Q.

7.9 Generalized Rayleigh Quotient

481

Similarly, let m = [m(1), . . . , m(Q)]T denote the ensemble mean of all the signal-feature vectors obtained from all the observation data, where 1 m (q), c i=1 i c

m(q) =

q = 1, . . . , Q.

Using the above vectors, one can define a Q × Q within-class scatter matrix [138] ⎛ ⎞ Ni c   1 def 1 ⎝ (si,k − mi )(si,k − mi )T ⎠ Sw = (7.9.8) c i=1 Ni k=1

and a Q × Q between-class scatter matrix [138] 1 (m − m)(mi − m)T . c i=1 i c

def

Sb =

(7.9.9)

For the optimal class discrimination matrix U to be determined, define a criterion function  T U Sb U def diag , (7.9.10) J(U) =  UT S w U diag

where



A denotes the product of diagonal entries of the matrix A. As it is a

diag

measure of class discrimination ability, J should be maximized. We refer to Span(U) as the class discrimination space, if 

U = arg max J(U) = U∈RQ×Q

UT Sb U

diag



UT S w U

.

(7.9.11)

diag

This optimization problem can be equivalently written as ⎧ Q ⎫  T ⎪ ⎪ ⎪ ⎪ Q ⎨ ui Sb ui ( uTi Sb ui ⎬ i=1 [u1 , . . . , uQ ] = arg max Q =  uT S u ⎪ ui ∈RQ ⎪ ⎪ i=1 i w i ⎪ ⎩ uTi Sw ui ⎭

(7.9.12)

i=1

whose solution is given by ui = arg max ui ∈RQ

uTi Sb ui , uTi Sw ui

i = 1, . . . , Q.

(7.9.13)

This is just the maximization of the generalized Rayleigh quotient. The above equation has a clear physical meaning: the column vectors of the matrix U constituting

482

Eigenanalysis

the optimal class discrimination subspace which should maximize the between-class scatter and minimize the within-class scatter, namely which should maximize the generalized Rayleigh quotient. For c classes of signals, the optimal class discrimination subspace is (c − 1)dimensional. Therefore Equation (7.9.13) is maximized only for c − 1 generalized Rayleigh quotients. In other words, it is necessary to solve the generalized eigenvalue problem S b ui = λ i S w ui ,

i = 1, 2, . . . , c − 1,

(7.9.14)

for c − 1 generalized eigenvectors u1 , . . . , uc−1 . These generalized eigenvectors constitute the Q × (c − 1) matrix Uc−1 = [u1 , . . . , uc−1 ]

(7.9.15)

whose columns span the optimal class discrimination subspace. After obtaining the Q × (c − 1) matrix Uc−1 , we can find its projection onto the optimal class discrimination subspace, yi,k = UTc−1 si,k ,

i = 1, . . . , c, k = 1, . . . , Ni ,

(7.9.16)

for every signal-feature vector obtained in the training phase. When there are only three classes of signals (c = 3), the optimal class discrimination subspace is a plane, and the projection of every feature vector onto the optimal class discrimination subspace is a point. These projections directly reflect the discrimination ability of different feature vectors in signal classification.

7.9.3 Robust Beamforming In wireless communications, if a base station uses an antenna array which is composed of multiple antennas (called array elements), it can separate multiple users in a common channel through spatial processing and thereby detect the desired user signals. Consider an array consisting of M omnidirectional antennas, and suppose that K narrowband signals are located in the far field. The observation signal vector received by the M array elements at time n is given by y(n) = d(n) + i(n) + e(n),

(7.9.17)

where e(n) is the vector whose entries are the additive white noises at the M array elements, while d(n) = a(θ0 (n))s0 (n), i(n) =

K−1  k=1

a(θk (n))sk (n),

(7.9.18) (7.9.19)

7.9 Generalized Rayleigh Quotient

483

are respectively the desired signal vector received by the M array elements and the interference signal vector from the other K − 1 users; θ0 (n) and θk (n) respectively denote the directions of arrival (DOAs) of the desired user and the kth interference user, and the vectors a(θ0 (n)) and a(θk (n)) are respectively the array response vectors of the desired user and the kth interference user. Assume that these signal sources are statistically independent of each other, and that the additive white noises of different array elements are statistically independent of each other and have the same variance σ 2 . Therefore the autocorrelation matrix of the observation signal is given by Ry = E{y(n)yH (n)} = Rd + Ri+e ,

(7.9.20)

  Rd = E d(n)dH (n) = P0 a(θ0 )aH (θ0 ),     Ri+e = E i(n) + e(n) i(n) + e(n) H

(7.9.21)

where

=

K−1 

Pk a(θk )aH (θk ) + σ 2 I;

(7.9.22)

k=1

the constants P0 and Pk represent the powers of the desired user and of the kth interference user, respectively. Let w(n) be the weighting vector of the beamformer at time n; then its output is z(n) = wH (n)y(n).

(7.9.23)

It is easy to find the signal-to-interference-plus-noise ratio (SINR) of the beamformer output:   E |wH d(n)|2 wH R w  = H d . SINR(w) = (7.9.24) w Ri+e w E |wH (i(n) + e(n))|2 In order to achieve the purpose of interference suppression, we need to maximize the generalized Rayleigh quotient SINR(w). That is to say, the optimal beamformer for interference suppression should be selected as the generalized eigenvector corresponding to the maximum generalized eigenvalue of the matrix pencil {Rd , Ri+e }. However, in this method we need to compute respectively the autocorrelation matrix Rd of the desired user and autocorrelation matrix Ri+e of the interference plus noise. These autocorrelation matrices are hard to compute directly. Substituting Rd = P0 a(θ0 )aH (θ0 ) into the SINR formula (7.9.24), the unconstrained optimization problem can be represented as max SINR(w) = max

P0 |wH a(θ0 )|2 . wH Ri+e w

(7.9.25)

Adding the constraint condition wH a(θ0 ) = 1, the unconstrained optimization

484

Eigenanalysis

problem becomes the constrained optimization problem min wH Ri+e w

subject to

wH a(θ0 ) = 1.

(7.9.26)

This optimization problem is still not easy to solve, because Ri+e cannot be calculated. Note that wH Ry w = wH Rd w + wH Ri+e w = P0 + wH Ri+e w and that the power of the desired signal P0 is a constant that is independent of the beamformer, so the optimization problem in Equation (7.9.26) is also equivalent to min wH Ry w

subject to wH a(θ0 ) = 1.

(7.9.27)

In contrast with Equation (7.9.26), the autocorrelation matrix of the signal vector Ry is easily estimated. Using the Lagrange multiplier method, it is easy to find that the solution of the constrained optimization problem (7.9.27) is wopt (n) =

R−1 y a(θ0 (n)) aH (θ0 (n))R−1 y a(θ0 (n))

.

(7.9.28)

This is the robust beamformer for interference suppression proposed in [414].

7.10 Quadratic Eigenvalue Problems We often encounter a common problem – the quadratic eigenvalue problem (QEP) – in linear stability studies in fluid mechanics [61], [204], the dynamic analysis of acoustic systems, the vibrational analysis of structural systems in structural mechanics, circuit simulation [466], the mathematical modeling of microelectronic mechanical systems (MEMSs) [108], biomedical signal processing, time series prediction, the linear-prediction coding of voices [116], multiple input–multiple output (MIMO) system analysis [466], finite element analysis of partial differential equations in industrial applications [260], [359], and in a number of applications of linear algebra problems. The quadratic eigenvalue problem is closely linked to the eigenvalue problem (especially the generalized eigenvalue problem), but there is a clear difference.

7.10.1 Description of Quadratic Eigenvalue Problems Consider a system with viscous damping and without any external forces; its motion equation is the differential equation [230], [422] M¨ x + Cx˙ + Kx = 0,

(7.10.1)

7.10 Quadratic Eigenvalue Problems

485

where M, C, K are respectively n × n mass, damping and stiffness matrices, while the vectors x ¨, x˙ and x are respectively the acceleration vector, the velocity vector and the displacement vector. In vibrational analysis, the general solution of the homogeneous linear equation (7.10.1) is given by x = eλt u, where u is usually a complex vector while λ is an eigenvalue that is generally a complex number. Substituting x = eλt u and its derivatives with respect to time into (7.10.1), one gets the quadratic characteristic equation (λ2 M + λC + K)u = 0,

or vH (λ2 M + λC + K) = 0T

if instead we write x = vH e λt . The determinant |λ2 M + λC + K| is known as the quadratic characteristic polynomial, and |λ2 M + λC + K| = 0 is called the quadratic characteristic equation. In general, the quadratic eigenvalue problem can be described as follows [274]: given n × n matrices M, C, K, find scalars λi and nonzero vectors ui , vi such that (λ2i M + λi C + K)ui = 0, viH (λ2i M

T

+ λi C + K) = 0 .

(7.10.2) (7.10.3)

The scalars λi satisfying the above equation are called quadratic eigenvalues, and the nonzero vectors ui and vi are respectively known as the right quadratic eigenvector and the left quadratic eigenvector corresponding to the quadratic eigenvalue λi . Every quadratic eigenvalue λi and its corresponding quadratic eigenvectors ui , vi constitute a quadratic eigentriple (λi , ui , vi ) of the quadratic eigenvalue problem. A quadratic eigenvalue problem has 2n eigenvalues (finite or infinitely large), 2n right eigenvectors and 2n left eigenvectors. In engineering applications the eigenvalues are usually complex; the imaginary parts are the resonant frequencies and the real parts denote exponential damping. We hope to find all the eigenvalues in the frequency range of interest. In some applications the eigentriples usually number dozens to hundreds. The quadratic eigenvalue problem is an important subclass of nonlinear eigenvalue problems. Let Q(λ) = λ2 M + λC + K.

(7.10.4)

This is a quadratic n × n matrix polynomial. In other words, the coefficients of the matrix Q(λ) constitute the quadratic polynomial of λ. As discussed in Subsection 7.4.2, the matrix Q(λ) is called the λ-matrix [273]. The quadratic eigenvalues are given by the solutions to the characteristic equation |Q(z)| = |z 2 M + zC + K| = 0.

(7.10.5)

DEFINITION 7.22 [466] The matrix Q(λ) is called a regular λ-matrix if the characteristic polynomial |Q(z)| is not identically zero for all z. Conversely, if |Q(z)| ≡ 0 for all z, then say that the matrix Q(z) is nonregular λ-matrix.

486

Eigenanalysis

The λ-matrix Q(λ) is usually assumed to be regular. For a regular λ-matrix Q(λ), two different eigenvalues are likely to have the same eigenvector. Table 7.1 summarizes the properties of the eigenvalues and eigenvectors of the quadratic eigenvalue problem (QEP). Table 7.1 Properties of quadratic eigenvalues [466] No. Matrix

Eigenvalue

Eigenvectors

1 2 3 4

2n, finite finite and infinite real or (λ, λ∗ ) pairs real or (λ, λ∗ ) pairs

(λ, u), (λ∗ , u∗ ) (λ, u), (λ, u∗ )

5

6

7

M nonsingular M singular M, C, K: real M,  C, K : Hermitian M: Hermitian pdf C, K: Hermitian psdf

⎧ ⎪ ⎨ M, C: symmetric pdf K: symmetric psdf ⎪ ⎩ γ(M, C, K) > 0  M, K: Hermitian 

8

M: pdf, C = −C

H

M, K: real symmetric pdf, C = −CT

Re(λ) ≤ 0 



λ = 0, n principal and n minor eigenvalues imaginary eigenvalues or conjugate pair (λ, λ∗ )





2n eigenvectors are independent (λ, u): right eigenpair (−λ∗ , u): left eigenpair

imaginary eigenvalues

In Table 7.1, pdf means positive definite, psdf means positive semi-definite and 3 4 2    γ(M, C, K) = min uH Cu − 4 uH Mu uH Ku | u 2 = 1 . According to the value of the quadratic function, the QEP can be further classified as follows [197]: (1) Elliptic QEP The QEP (7.10.1) with (uH Cu)2 < 4(uH Mu)(uH Ku) is known as an elliptic quadratic eigenvalue problem. (2) Hyperbolic QEP The QEP (7.10.1) is called a hyperbolic quadratic eigenvalue problem if the function (uH Cu)2 > 4(uH Mu)(uH Ku).

7.10.2 Solving Quadratic Eigenvalue Problems There are two main methods in solving quadratic eigenvalue problems: (1) Decomposition method: On the basis of the generalized Bezout theorem, a quadratic eigenvalue problem can be divided into two first-order eigenvalue problems.

7.10 Quadratic Eigenvalue Problems

487

(2) Linearized method: Via linearization, a nonlinear quadratic eigenvalue problem becomes a linear generalized eigenvalue problem. 1. Decomposition Method Define the matrix Q(S) = MS2 + CS + K,

S ∈ Cn×n ;

(7.10.6)

then the difference between Q(S) and Q(λ) is Q(λ) − Q(S) = M(λ2 I − S2 ) + C(λI − S) = (λM + MS + C)(λI − S).

(7.10.7)

This result is known as the generalized Bezout theorem for a second-order matrix polynomial [171]. If for the second-order matrix equation Q(S) = MS2 + CS + K = O

(7.10.8)

there exists one solution S ∈ Cn×n , then we say that this solution is the right solvent of the second-order matrix equation. Similarly, the solution of the equation S2 M + SC + K = O is called the left solvent of the second-order matrix equation [466]. Clearly, if S is a solution of the second-order matrix equation Q(S) = O then the generalized Bezout theorem (7.10.7) simplifies to Q(λ) = (λM + MS + C)(λI − S).

(7.10.9)

Equation (7.10.9) shows that if S is the solution of the second-order matrix equation (7.10.8) then the quadratic eigenvalue problem |Q(λ)| = |λ2 M+λC+K| = 0 is equivalent to |Q(λ)| = |(λM + MS + C)(λI − S)| = 0. From the determinant property |AB| = |A| |B| it is known that the quadratic eigenvalue problem becomes two (linear) eigenvalue subproblems: • n of the quadratic eigenvalues are solutions of the characteristic equation |λM + MS + C| = 0; • other n quadratic eigenvalues are solutions of the characteristic equation |λI − S| = 0. Because |λM + MS + C| = |MS + C − λ(−M)|, the eigenvalue problem |λM + MS + C| = 0 is equivalent to the generalized eigenvalue problem (MS + C)u = λ(−M)u. Summarizing the above discussion, it may be concluded that, by the generalized Bezout theorem, the quadratic eigenvalue problem (λ2 M + λC + K)u = 0 can be decomposed into two linear eigenvalue subproblems: (1) a generalized eigenvalue problem for the matrix pencil (MS + C, −M), i.e., (MS + C)u = −λMu;

488

Eigenanalysis

(2) the standard eigenvalue problem Su = λu for the matrix S. 2. Linearized Method A common way to solve nonlinear equations is to linearize them. This idea is also available for solving quadratic eigenvalue problems, because they themselves are an important subclass of nonlinear equations. < = 2 Let z = λx x ; then the quadratic characteristic equation (λ M + λC + K)x = 0 can be written as the equivalent form Lc (λ)z = 0, where     M O −C −K − (7.10.10) Lc (λ) = λ O I I O or

 M Lc (λ) = λ O

  O C − I I

 −K . O

(7.10.11)

Here Lc (λ) is called the companion form or linearized λ-matrix of Q(λ). The companion forms λB − A are divided into the first companion form     M O −C −K (7.10.12) , B= L1 : A= O I I O and the second companion form L2 :

 O A= I

 −K , O

 M B= O

 C . I

(7.10.13)

Via linearization, a quadratic eigenvalue problem Q(λ)x = 0 becomes a generalized eigenvalue problem Lc (λ)z = 0 or Az = λBz. As stated in Section 7.7, in order to ensure that the GEVD of a matrix pencil (A, B) is uniquely determined, the matrix B has to be nonsingular. From (7.10.12) and (7.10.13) it can be seen that this is equivalent to the matrix M being nonsingular. Algorithm 7.11 shows a linearized algorithm for the quadratic eigenvalue problem (QEP). The QZ decomposition in step 2 of Algorithm 7.11 can be run by using the qz function in MATLIB directly. It may be noticed that when some entry of the upper triangular matrix S is equal to zero then the eigenvalue λ = ∞. When M, C and K are symmetric matrices, and A − λB is the symmetric linearization of P(λ) = λ2 M + λC + K, Algorithm 7.11 will not ensure the symmetry of A − λB because the QZ decomposition in step 2 cannot ensure the symmetry of (A, B). In this case, one needs to use the following method [466]. (1) First compute the Cholesky decomposition B = LLT , where L is a lower triangular matrix.

7.10 Quadratic Eigenvalue Problems Algorithm 7.11

489

Linearized algorithm for QEP

input: λ-matrix P(λ) = λ2 M + λC + K. 1. Form the matrix pencil (A, B) via linearization in (7.10.12) or (7.10.13). 2. Use QZ decomposition to compute the generalized Schur decomposition T = QH AZ,

S = QH BZ.

3. Calculate the quadratic eigenvalues and eigenvectors: for k = 1 : 2n (i.e., k = 1, . . . , 2n), λk = tkk /skk . Solve (T − λk S)φ = 0, and let ξ = Zφ. ξ1 = ξ(1 : n); ξ2 = ξ(n + 1 : 2n). r1 = P(λ  k )ξ1 /ξ1 ; r2 = P(λ2 )ξ2 /ξ2 . ξ(1 : n), if r1  ≤ r2 , uk = ξ(n + 1 : 2n), otherwise. endfor output: (λk , uk ), k = 1, . . . , 2n.

(2) Substitute B = LLT into the symmetric GEVD problem Aξ = λBξ to get the standard eigenvalue problem L−1 AL−T φ = λφ, where φ = LT ξ. (3) Use the symmetric QR decomposition to compute the eigenpair (λi , φi ) of the symmetric matrix L−1 AL−T . These eigenpairs are the quadratic eigenpairs of the original quadratic eigenvalue problem. Meerbergen [321] presented a generalized Davidson algorithm, a modified Davidson algorithm and a second-order residual-error iteration method, etc., for the linearized quadratic eigenvalue problem. Algorithm 7.11 is only applicable to the case of a nonsingular matrix M. However, in some industrial applications the matrix M is singular. For example, in the finite element analysis of a damping structure, a so-called massless degree of freedom is often encountered, which corresponds to some zero column vectors of the mass matrix M [260]. When the matrix M is singular, and thus B is also singular, there are two ways to improve the original quadratic eigenvalue problem [260]. The first way uses spectral transformation, i.e., the introduction of an appropriate shift λ0 to change the eigenvalue to μ as follows: μ = λ − λ0 . Then, the Type I linearization becomes     M −C − λ0 M −K u˙ =μ O I −λ0 I u

(7.10.14)

O I

  u˙ . u

(7.10.15)

490

Eigenanalysis

 −C − λ0 M −K is The shift parameter λ0 is chosen to ensure that the matrix I −λ0 I nonsingular. The alternative way is to set 1 α= . (7.10.16) μ 

Substitute μ = 1/α into Equation (7.10.15) and rearrange the result, obtaining −1       M O u˙ u˙ −C − λ0 M −K =α . (7.10.17) I −λ0 I O I u u The above equation is the standard eigenvalue decomposition Ax = αx, where A=

 −C − λ0 M I

−K −λ0 I

−1 

M O

(7.10.18)  O , I

x=

  u˙ . u

(7.10.19)

Therefore the solution to the eigenvalue problem Ax = αx yields the eigenvalue α and its corresponding eigenvector. Then we use 1 (7.10.20) + λ0 α to determine the eigenvalues of the quadratic eigenvalue problem. It is necessary to point out that any higher-order eigenvalue problem λ = μ + λ0 =

(λm Am + λm−1 Am−1 + · · · + λA1 + A0 )u = 0 can be linearized [321]: ⎡ ⎤⎡ ⎡ ⎤ A1 A 2 · · · u −A0 ⎢ I ⎥ ⎢ λu ⎥ ⎢ I O ··· ⎢ ⎥⎢ ⎢ ⎥ ⎥⎢ . ⎥ = λ⎢ . . ⎢ . . . ⎦ ⎣ .. ⎦ ⎣ .. . . . . . ⎣ I O ··· I λm−1 u

⎤ ⎤⎡ Am u ⎥ ⎢ O⎥ ⎥ ⎢ λu ⎥ ⎥ ⎢ .. . ⎥. . ⎦ ⎣ .. ⎦ O λm−1 u

(7.10.21)

(7.10.22)

That is, an mth-order quadratic eigenvalue problem can be linearized to a standard generalized eigenvalue problem Ax = λBx.

7.10.3 Application Examples In the following, three application examples of quadratic eigenvalues are presented. 1. Autoregressive Parameter Estimation Consider the modeling of a real random process as an autoregressive (AR) process x(n) + a(1)x(n − 1) + · · · + a(p)x(n − p) = e(n),

(7.10.23)

7.10 Quadratic Eigenvalue Problems

491

where a(1), . . . , a(p) are the AR parameters, p is the AR order and e(n) is an unobservable excitation signal that is usually a white noise with variance σe2 . Multiplying Equation (7.10.23) by x(n − τ ), τ ≥ 1, and taking the mathematical expectation, we get the linear normal equation Rx (τ ) + a(1)Rx (τ − 1) + · · · + a(p)Rx (τ − p) = 0,

τ = 1, 2, . . . ,

(7.10.24)

where Rx (τ ) = E{x(n)x(n − τ )} is the autocorrelation of the AR process. This normal equation is called the Yule–Walker equation. Taking τ = 1, . . . , p, then Equation (7.10.24) can be written as ⎡ ⎤ ⎡ ⎤⎡ ⎤ Rx (1) Rx (0) a(1) Rx (−1) · · · Rx (−p + 1) ⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢R (2)⎥ ⎢ Rx (1) Rx (0) · · · Rx (−p + 2)⎥ ⎢a(2)⎥ ⎥ = −⎢ x ⎥. ⎢ ⎥⎢ (7.10.25) .. .. .. ⎢ .. ⎥ ⎢ ⎥ ⎢ .. ⎥ .. ⎣ . ⎦ ⎣ ⎦⎣ . ⎦ . . . . a(p) Rx (p − 1) Rx (p − 2) · · · Rx (p) Rx (0) However, in many cases there is an observation noise v(n), i.e., the observation signal is y(n) = x(n) + v(n), where v(n) is a white noise with variance σ 2 and is statistically independent of x(n). Under this assumption, there is the following relationship between the observation signal y(n) and the AR random process x(n): + Ry (0) − σ 2 , τ = 0, 2 Rx (τ ) = Ry (τ ) − σ δ(τ ) = τ = 0. Ry (τ ), After substituting this relation into Equation (7.10.25), the Yule–Walker equation becomes ⎡ ⎡ ⎤⎡ ⎤ ⎤ Ry (1) Ry (0) − σ 2 a(1) Ry (−1) · · · Ry (−p + 1) ⎢ ⎢ ⎥⎢ ⎥ ⎥ ⎢R (2)⎥ ⎢ Ry (1) Ry (0) − σ 2 · · · Ry (−p + 2)⎥ ⎢a(2)⎥ ⎢ ⎥⎢ ⎥ = − ⎢ y ⎥ , (7.10.26) .. .. .. ⎢ .. ⎥ ⎢ ⎥ ⎢ .. ⎥ .. ⎣ . ⎦ ⎣ ⎦⎣ . ⎦ . . . . 2 Ry (p − 2) · · · Ry (0) − σ a(p) Ry (p − 1) Ry (p) which is the Yule–Walker equation with noise compensation [466]. Let a = [−a(1), . . . , −a(p)]T , r1 = [Ry (1), . . . , Ry (p)]T and ⎡ ⎤ Ry (0) Ry (−1) · · · Ry (−p + 1) ⎢ ⎥ ⎢ Ry (1) Ry (0) · · · Ry (−p + 2)⎥ ⎢ ⎥. Ry = ⎢ .. .. .. ⎥ .. ⎣ ⎦ . . . . Ry (0) Ry (p − 1) Ry (p − 2) · · · Then Equation (7.10.26) can be rewritten as (Ry − σ 2 Ip )a = r1 .

(7.10.27)

492

Eigenanalysis

Since Ry (τ ) = Rx (τ ), τ ≥ 1, if we let τ = p + 1, p + 2, . . . , p + q then from Equation (7.10.24) it follows that giT a = Ry (p + i),

i = 1, . . . , q,

(7.10.28)

where gi = [Ry (p + i − 1), Ry (p + i − 2), . . . , Ry (i)]T . Now combine Equations (7.10.27) and (7.10.28) to yield the matrix equation ¯ − λD)v = 0 , (R y p+q

(7.10.29)

¯ and D are (p + q) × (p + 1) matrices and v is a (p + 1) × 1 vector, defined where R y respectively as ⎤ ⎡ Ry (1) Ry (0) Ry (−1) · · · Ry (−p + 1) ⎥ ⎢ ⎢ Ry (2) Ry (1) Ry (0) · · · Ry (−p + 2)⎥ ⎥ ⎢ .. .. .. .. ⎥ ⎢ .. ⎥ ⎢ . . . . . ⎥ ⎢ ¯ = ⎢ R (p) ⎥, R R (p − 1) R (p − 2) · · · R (0) y y y y y ⎥ ⎢ ⎥ ⎢ ⎢Ry (p + 1) Ry (p) Ry (2) ··· Ry (1) ⎥ ⎥ ⎢ .. .. .. .. ⎥ ⎢ .. ⎦ ⎣ . . . . . Ry (q) Ry (p + q) Ry (p + q − 1) Ry (p + q − 2) · · · ⎤ ⎡ 0 1 0 ··· 0 ⎢0 0 1 · · · 0⎥ ⎡ ⎤ ⎥ ⎢ 1 ⎥ ⎢. . . . ⎢a(1)⎥ ⎢ .. .. .. . . ... ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ D = ⎢0 0 0 · · · 1⎥ and v = ⎢a(2)⎥ . ⎢ . ⎥ ⎥ ⎢ ⎢0 0 0 · · · 0⎥ ⎣ .. ⎦ ⎥ ⎢ .. ⎥ ⎢ .. .. .. . . a(p) ⎣. . . . .⎦ 0 0 0 ··· 0 ¯ − λD)T , it follows that Premultiplying Equation (7.10.29) by (R y (λ2 M + λC + K)v = 0p+1 ,

(7.10.30)

where ¯ , ¯TR M=R y y

¯ ), ¯ T D + DT R C = −(R y y

K = DT D.

(7.10.31)

Because the matrices M, C and K are symmetric, Equation (7.10.30) is a symmetric quadratic eigenvalue problem. As before, let     K O −C −M A= , B= . (7.10.32) O I I O Then the quadratic eigenvalue problem becomes a 2(p + 1) generalized eigenvalue problem (A − λB)u = 0.

(7.10.33)

7.10 Quadratic Eigenvalue Problems

493

Because the eigenvalues are real or appear in conjugate pairs (λ, λ∗ ), the right eigenvectors corresponding to the conjugate pair (λ, λ∗ ) are the conjugate pair (u, u∗ ). Among the 2(p+1) eigenpairs (λi , ui ), the eigenvalue λ and its corresponding eigenvector u satisfying both Equation (7.10.27) and Equation (7.10.28) are respectively estimates of the variance σ 2 of the observation noise and the AR parameter vector [1, a(1), . . . , a(p)]T . The above method for estimating AR parameters from observation data noisy with the additive white noise was proposed by Davila in 1998 [116]. 2. Constrained Least Squares Consider the constrained least squares problem   x = arg min xT Ax − 2bT x subject to xT x = c2 ,

(7.10.34)

where A ∈ Rn×n is a symmetric matrix and c is a real nonzero constant that takes the value c = 1 in many applications. This constrained optimization problem can be solved by the Lagrange multiplier method. To this end, let the cost function be J(x, λ) = xT Ax − 2bT x + λ(c2 − xT x). From

(7.10.35)

∂J(x, λ) ∂J(x, λ) = 0 and = 0, we have ∂x ∂λ

(A − λI)x = b

and

xT x = c 2 .

(7.10.36)

Let x = (A − λI)y; substituting this into the first formula of Equation (7.10.36), we get (A − λI)2 y = b

or (λ2 I − 2λA + A2 )y − b = 0.

(7.10.37)

Using the above assumption that x = (A−λI)y and A are the symmetric matrices, it is easily seen that xT x = yT (A − λI)T (A − λI)y = yT (A − λI)[(A − λI)y] = yT (A − λI)x = yT b. Then, the second formula of Equation (7.10.36) can be equivalently written as y T b = c2

or

1 = yT b/c2 ,

from which we have b = byT b/c2 = bbT y/c2 , because yT b = bT y. Substitute the above equation into Equation (7.10.37) to yield )  * λ2 I − 2λA + A2 − c−2 bbT y = 0. (7.10.38) This is just a symmetric quadratic eigenvalue problem.

494

Eigenanalysis

To sum up the above discussion, the solution steps for constrained least squares problem (7.10.34) are as follows [169]. (1) Solve the quadratic eigenvalue problem (7.10.38) to get the eigenvalues λi (the eigenvectors may not be needed). (2) Determine the minimum eigenvalue λmin . (3) The solution of the constrained least squares problem (7.10.34) is given by x = (A − λmin I)−1 b. 3. Multiple-Input–Multiple-Output System The multiple-input–multiple-output (MIMO) system is a linear system often encountered in communication, radar, signal processing, automatic control and system engineering. Consider a linear controlled system with m input and n outputs: ˙ M¨ q(t) + Cq(t) + Kq(t) = Bu(t),

(7.10.39)

y(t) = Lq(t),

(7.10.40)

where u(t) ∈ Cm , m ≤ r, is some input signal vector, q(t) ∈ Cr is the state vector of the system, y(t) ∈ Cn is the system output vector, B ∈ Cr×m is the input function matrix, L ∈ Cn×r is the output function matrix and M, C, K are r × r matrices. Making a Laplace transformation of the MIMO system, and assuming a zero initial condition, we have s2 M¯ q(s) + sC¯ q(s) + K¯ q(s) = B¯ u(s),

(7.10.41)

¯ (s) = L¯ y q(s).

(7.10.42)

Hence the transfer function matrix of the MIMO system is given by ¯ (s) y = L(s2 M + sC + K)−1 B. G(s) = ¯ (s) u

(7.10.43)

Put Q(s) = s2 M + sC + K; then, from Equations (7.10.12) and (7.10.13) one gets the inverse matrices in the cases of the first companion form L1 and the second companion form L2: 2n  λi ui viH L1 : (s2 M + sC + K)−1 = U(sI − Λ)−1 ΛVH = , s − λi i=1 L2 :

(s2 M + sC + K)−1 = U(sI − Λ)−1 VH =

2n  ui viH . s − λi i=1

That is to say, the eigenvalues of the quadratic characteristic polynomial |s2 M + sC + K| provide the poles of the controlled MIMO system transfer function. Hence, the quadratic eigenvalues provide the basis for studying the control performances and response performances of a controlled MIMO system [466]. Many applications of quadratic eigenvalue problems can be found in [466], which is a wonderful review of quadratic eigenvalue problems.

7.11 Joint Diagonalization

495

7.11 Joint Diagonalization The EVD is the diagonalization of one Hermitian matrix, and the GEVD can be viewed as the joint diagonalization of two matrices. A natural question to ask is whether the more than two matrices can be simultaneous diagonalized or jointly diagonalized? This is the theme of this section.

7.11.1 Joint Diagonalization Problems Consider the following model for an array that is receiving signals x(t) = As(t) + v(t),

t = 1, 2, . . . ,

(7.11.1)

in which x(t) = [x1 (t), . . . , xm (t)]T is the observed signal vector, m is the number of sensors observing the spatial signals, s(t) = [s1 (t), . . . , sn (t)]T is the source vector, v(t) = [v1 (t), . . . , vm (t)]T is the additive noise vector on the sensor array, while A ∈ Cm×n (m > n) is the matrix representing the mixing of the n sources, called the mixture matrix. The so-called blind source separation (BSS) problem uses only the observed data vector x(t) to determine the unknown mixture matrix A and then to separate the source vector s(t) = Wx(t) via the separation or demixing matrix W = A† ∈ Cn×m . Hence the key to BSS is the determination of the mixture matrix A. To this end, the following assumptions are made. (1) The additive noise is time-domain white and spatial-domain colored, i.e., its correlation matrix is Rv (k) = E{v(t)vH (t − k)} = δ(k)Rv + Rv , k = 0 (spatial-domain colored), = O, k =  0 (time-domain white). Here “time-domain white” means that the additive noise on every sensor is white noise, while “spatial-domain colored” means that the additive white noise on different sensors may be correlated.   (2) The n sources are statistically independent, i.e., E s(t)sH (t − k) = Dk (a diagonal matrix). (3) The  source vector s(t) is statistically independent of the noise vector v(t), i.e., E s(t)vH (t − k) = O (zero matrix). Under the above three assumptions, the autocorrelation matrix of the array out-

496

Eigenanalysis

put vector is given by   Rx (k) = E x(t)xH (t − k)   = E (As(t) + v(t))(As(t − k) + v(t − k))H     = AE s(t)sH (t − k) AH + E v(t)vH (t − k) + AD0 AH + Rv , k = 0, = k = 0. ADk AH ,

(7.11.2)

Hence, for K autocorrelation matrices Rx (k), k = 1, . . . , K with no noise (time lag k = 0), making a joint diagonalization Rx (k) = UΣk UH ,

(7.11.3)

then Rx (k) = UΣk UH = ADk AH ,

k = 1, . . . , K.

(7.11.4)

The question is whether the joint diagonalization result U is the mixture matrix A? The answer is no, because the Σk do not happen to be the same as Dk , k = 1, . . . , K. On the other hand, in the signal model (7.11.1) there exists an uncertainty x(t) =

n  i=1

α i ai ·

si (t) + v(t) = AGG−1 s(t) + v(t), αi

(7.11.5)

where G is an n×n generalized permutation matrix. In this case, the autocorrelation matrix of the array output vector is given by   Rx (k) = E x(t)xH (t − k) 3  H 4 = E AGG−1 s(t) + v(t) AGG−1 s(t − k) + v(t − k)     = AGG−1 E s(t)sH (t − k) (AGG−1 )H + E v(t)vH (t − k) + AGG−1 D0 G−H GH AH + Rv , k = 0, = (7.11.6) k = 0. AG(G−1 Dk G−H )GH AH , Thus, there is the following relationship between the joint diagonalization result and the model parameters: Rx (k) = UΣk UH = AG(G−1 Dk G−H )GH AH

(7.11.7)

for k = 1, . . . , K. This shows that U = AG, i.e., we can only identify AG through the joint diagonalization result U. DEFINITION 7.23 Two matrices M and N are said to be essentially equal matri. ces, denoted M = N, if there exists a generalized permutation matrix G such that M = NG.

7.11 Joint Diagonalization

497

When using U = AG as the separating matrix, the separated output is given by ˆs(t) = U† x(t) = G−1 A† x(t) = G−1 s(t) + G−1 A† v(t).

(7.11.8)

Under an appropriate SNR the effect of the additive noise v can be ignored, so the separated output is given by ˆs(t) ≈ G−1 s(t),

(7.11.9)

which is just a copy of the source vector up to factor uncertainty and ordering uncertainty. Such a separation result is allowed by the source separation. This is the reason why the separating matrix matrix U and the mixture matrix A are called essentially equal matrices. The joint diagonalization of multiple matrices was first proposed by Flury in 1984 in the principal component analysis of K covariance matrices. Later, Cardoso and Souloumiac [85] in 1996 and Belochrani et al. [33] in 1997 respectively proposed the approximate joint diagonalization of multiple cumulant matrices or covariance matrices from the BSS angle. From then on, joint diagonalization has been widely studied and applied. The mathematical problem of BSS is as follows: given K matrices A1 , . . . , AK ∈ Cm×m , the joint diagonalization problem seeks a nonsingular diagonalizing matrix W ∈ Cm×m and K associated diagonal matrices Λ1 , . . . , ΛK ∈ Cm×m such that Ak = WΛk WH ,

k = 1, . . . , K,

(7.11.10)

where W is called sometimes the joint diagonalizer. The above diagonalization is called exact joint diagonalization. However, a practical form of joint diagonalization is approximate joint diagonalization: this approach seeks a joint diagonalizer W and K associated diagonal matrices Λ1 , . . . , ΛK to minimize the objective function [84], [85] according to min J1 (W, Λ1 , . . . , ΛK ) = min

K 

2 22 αk 2WH Ak W − Λk 2F

(7.11.11)

2 22 αk 2Ak − WΛk WH 2F ,

(7.11.12)

k=1

or [504], [530] min J2 (W, Λ1 , . . . , ΛK ) = min

K  k=1

where α1 , . . . , αK are positive weighting coefficients. To simplify the description, we assume that α1 = · · · = αK = 1. 7.11.2 Orthogonal Approximate Joint Diagonalization In many engineering applications only the joint diagonalizer W is used, while the diagonal matrices Λ1 , . . . , ΛK are not needed. In this case, it is an interesting problem to transform the objective function of approximate joint diagonalization into

498

Eigenanalysis

an objective function containing only the joint diagonalizer W. This optimization problem has the following solution methods. 1. Off-Function Minimization Method In numerical analysis, the sum of the squared absolute values of all off-diagonal entries of an m × m matrix B = [Bij ] is defined as its off function, namely def

off(B) =

m n  

|Bij |2 .

(7.11.13)

i=1, i=j j=1

The m × m matrix M constituting of all the off-diagonal entries, + 0, i = j, [Moff ]ij = Mij , i = j,

(7.11.14)

is the off matrix of the original matrix B. Therefore the off function is the square of the Frobenius norm of the off matrix: off(M) = Moff 2F .

(7.11.15)

Using the off function, the orthogonal approximate joint diagonalization problem can be represented as [84], [85] 0 + K K n n     H H 2 off(W Ak W) = |(W Ak W)ij | . min J1a (W) = k=1 i=1, i=j j=1

k=1

(7.11.16) Making a Givens rotation of the off-diagonal entries of A1 , . . . , AK , one can implement the orthogonal joint diagonalization of these matrices. The product of all the Givens rotation matrices gives the joint diagonalizer W. This is the Jacobi algorithm for approximate joint diagonalization proposed by Cardoso and Souloumiac [84], [85]. 2. Diagonal Function Maximization Method A diagonal function of a square matrix may be a scalar, vector or matrix. (1) Diagonal function diag(B) ∈ R is the diagonal function of the m × m matrix B, and is defined as m  def diag(B) = |Bii |2 . (7.11.17) i=1

(2) Diagonal vector function The diagonal vector function of an m × m matrix B, denoted diag(B) ∈ Cm , is a column vector consisting of the diagonal entries of B arranged in order, i.e., def

diag(B) = [B11 , . . . , Bmm ]T .

(7.11.18)

7.11 Joint Diagonalization

499

(3) Diagonal matrix function The diagonal matrix function of an m × m matrix B, denoted Diag(B) ∈ Cm×m , is a diagonal matrix consisting of the diagonal entries of B, i.e., ⎤ ⎡ 0 B11 def ⎢ ⎥ .. Diag(B) = ⎣ (7.11.19) ⎦. . 0

Bmm

Minimizing off(B) is equivalent to maximizing the diagonal function diag(B), i.e., min off(B) = max diag(B). (7.11.20) So, Equation (7.11.16) can be rewritten as [504] 0 + K K  n   H  H    2  W Ak W  . diag W Ak W = max J1b (W) = ii

(7.11.21)

k=1 i=1

k=1

As a matter of fact, WH Ak W should be a diagonal matrix, and for any n × n diagonal matrix D, we have n  |Dii |2 = tr(DH D). (7.11.22) i=1

Therefore the optimization problem (7.11.21) can be written as the trace form + 0 K   H H  H max J1b (W) = tr W Ak WW Ak W . (7.11.23) k=1

In particular, in the case of orthogonal approximate joint diagonalization, since WWH = WH W = I and AH k = Ak , k = 1, . . . , K, the above equation reduces to 0 + K   H 2  tr W Ak W . (7.11.24) max J1b (W) = k=1

3. Subspace Method In the case of orthogonal approximate joint diagonalization, the joint diagonalizer W is the unitary matrix whose column vectors have unit norm, i.e., uk 2 = 1. Define mk = diag(Λk ) as the vector consisting of the diagonal entries of the diagonal matrix Λ, and let ˆ = [vec(A ), . . . , vec(A )], A 1 K

M = [m1 , . . . , mK ];

then the cost function of the joint diagonalization can be equivalently written as K K   2 2 2 2 2Ak − WΛk WH 22 = 2vec Ak − (W∗  W)Diag(Λk )22 F F k=1

k=1

2 2 ˆ − (W∗  W)M22 , = 2A F

(7.11.25)

500

Eigenanalysis

where R  D is the Khatri–Rao product of two matrices R and D, which is the Kronecker product of the column vectors of R and D: R  D = [c1 ⊗ d1 , c2 ⊗ d2 , . . . , cn ⊗ dn ].

(7.11.26)

Hence the solution of the joint diagonalization (7.11.25) can be rewritten as 2 2 ˆ − BM22 , B = W∗  W. (7.11.27) {W, M} = arg min 2A F W, M

This optimization problem is separable, since the least squares solution of M is ˆ M = (BH B)−1 BH A.

(7.11.28)

Substituting (7.11.28) into (7.11.27), we can cancel M to get 2 2 2 2 ˆ − B(BH B)−1 BH A ˆ 22 = arg min 2P⊥ A ˆ 22 , W = arg min 2A B F F W

W

(7.11.29)

H −1 H B is the orthogonal projection matrix. An algorithm where P⊥ B = I − B(B B) for solving the optimization problem (7.11.29) can be found in [480]. The three methods described above belong in the orthogonal approximate joint diagonalization category, because the joint diagonalizer provided by these methods is a unitary matrix.

7.11.3 Nonorthogonal Approximate Joint Diagonalization In orthogonal joint diagonalization, the solution cannot be trivial or ill-conditioned because of the unitarity of W. The limitation of orthogonal joint diagonalization is that it is available only for a whitened observation data vector. However, the prewhitening of data has the following two disadvantages. (1) The whitening operation, as a preprocessing, can adversely affect the performance of the separated signals, since the statistical error involved in this preprocessing stage cannot be corrected in the “effective separation” stage [83]. (2) In practice the whitening can distort the weighting least squares criterion, leading to the exact diagonalization of one selected matrix at the possible cost of the poor diagonalization of others [530]. Nonorthogonal joint diagonalization is joint diagonalization without the constraint WH W = I, and has become a mainstream joint diagonalization method. The advantage of nonorthogonal joint diagonalization is that it avoids two disadvantages of data prewhitening, while the disadvantage is that it may give a trivial or a degenerate solution. Typically, algorithms for nonorthogonal joint diagonalization are of the iterative type. • The iterative algorithm in [387] uses an information-theoretic criterion.

7.11 Joint Diagonalization

501

• The subspace-fitting algorithm in [480] uses Newton-type iterations. • The AC–DC algorithm in [530] minimizes the weighted least squared (WLS) error. • The gradient search algorithm in [233], the Newton search algorithm in [234] and the penalty function approach in [501] minimize the sum of the WLS error and some penalty term. • The quadratic optimization algorithm in [496] minimizes the WLS error under various constraints on the diagonalizing matrix. The AC–DC algorithm separates the coupled optimization problem into two single optimization problems: JWLS2 (W, Λ1 , . . . , ΛK ) =

=

K 

2 22 wk 2Ak − WΛk WH 2F

k=1

2 22 m 2 2  2 H2 w k 2A k − λ[k] n wk w k 2 . 2 2

K  k=1

n=1

F

The AC–DC algorithm consists of two phases. (1) Alternating columns (AC) phase Fix all but one column of W and the matrices Λ1 , . . . , ΛK and minimize the objective function JWLS2 (W) with respect to the remaining column vector of W. (2) Diagonal centers (DC) phase Fix W and minimize the cost function JWLS2 (W, Λ1 , . . . , ΛK ) with respect to the matrices Λ1 , . . . , ΛK . The simplest way of avoiding a trivial solution is to add the constraint condition Diag(W) = I. However, the biggest drawback of the nonorthogonal joint diagonalization is the possibility that its K diagonal matrices Λ1 , . . . , ΛK and/or the joint diagonalizer W may be singular; this leads to degenerate solutions of the nonorthogonal joint diagonalization problem [295]. In order to avoid both trivial solutions and degenerate solutions of the nonorthogonal joint diagonalization problem, Li and Zhang [295] proposed another objective function: ⎧ ⎫ K m m ⎨     H 2  ⎬ [W Ak W]ij  − β log  det(W) , (7.11.30) min J(W) = αk ⎩ ⎭ k=1

i=1 j=1,j=i

where αk (1 ≤ k ≤ K) are the positive weighting coefficients, and β > 0. The above cost function can be divided into the sum of a squared off-diagonal error term, J1 (W) =

K  k=1

m m    H    W Ak W 2 , αk ij i=1 j=1,j=i

(7.11.31)

502

Eigenanalysis

and a negative log determinant term,   J2 (W) = − log  det(W).

(7.11.32)

A significant advantage of the cost function (7.11.30) is that when W is equal a zero matrix or is singular, J2 (W) → +∞. Hence, minimization of the cost function J(W) can avoid both a trivial solution and a degenerate solution. PROPOSITION 7.2 [295] The criterion (7.11.30) is lower unbounded if and only if there is a nonsingular matrix W such that the given matrices A1 , . . . , AK are all exactly diagonalized. When optimizing at the mth iteration and collecting all the terms related to wi in J1 (W), the cost function in (7.11.30) can be rewritten as   J(W) = wiH Qi wi − β log  det(W) + Ci , (7.11.33) where Ci is a quantity independent of wi and the matrix Qi is defined as Qi =

K m  

  H H αk Ak wj wjH AH k + A k wj w j A k .

(7.11.34)

k=1 j=1,j=i

Here Qi is always Hermitian and (semi-)positive definite. Based on the cost function (7.11.30), a fast approximate joint diagonalization (FAJD) algorithm was developed in [295], as shown in Algorithm 7.12. PROPOSITION 7.3 [295] When the matrix Qi is positive definite and the constant β > 0, the global minimum point of the cost function J(wi ) = wiH Qi wi − β log |cH s wi | is determined by √ −1 βQi cs (opt) jφ =e  , (7.11.35) wi −1 2cH s Qi cs where φ is an ambiguous rotation angle (with real-valued numbers if φ is 0 or π and with complex-valued numbers for 0 ≤ φ < 2π). Proposition 7.3 shows that the minimization of the cost function J(W) is independent of the penalty parameter β, which implies that β can be any finite large positive number; the simplest choice is β = 1. This fact avoids the influence of the choice of penalty parameter on the performance of the penalty function method. Joint diagonalization is widely applied in blind source separation (see, e.g. [10], [33], [332], [530], [295]), blind beamforming [84], time delay estimation [531], frequency estimation [326], array signal processing [505], MIMO blind equalization [112], blind MIMO system identification [94], and so on.

Exercises Algorithm 7.12

503

Fast approximate joint diagonalization algorithm [295]

input: Target matrices A1 , . . . , AK ∈ Cm×m and weights α1 , . . . , αK and β. initialization: W (a unitary matrix or any invertible matrix). Set i = 1. repeat 1. Compute Qi =

m

K k=1

j=1,j=i

H H αk (Ak wj wjH AH k + Ak wj wj Ak ).

2. Form the exchange matrix  1, when i = s, j = t or i = t, j = s, [Es,t ]ij = 1, when i = j, i = s, t, 0,

otherwise.

3. For some integer s, 1 ≤ s ≤ m − 1 make the matrix partition

B u Es,m WEi,m = bH v such that the block matrix B is invertible. H −1 , 1]Es,m . 4. Calculate cH s = [−b B

'( √ −1 5. Replace the ith column in W with wiopt = ( βQ−1 2cH s Qi c s . i cs ) 6. exit if i = m. return i ← i + 1. output: Diagonalizing matrix W.

Exercises 7.1

Show the following properties: if λ is an eigenvalue of the n × n matrix A then: (a) λk is an eigenvalue of the matrix Ak . (b) If A is nonsingular, then A−1 has the eigenvalue 1/λ. (c) An eigenvalue of the matrix A + σ 2 I is λ + σ 2 .

7.2 7.3 7.4

Show that when A is an idempotent matrix, the matrices BA and ABA have the same eigenvalues. Assuming that all the entries of the n × n matrix A are equal to 1, find the n eigenvalues of A. Consider the matrix ⎡ ⎤ 0 1 0 0 ⎢1 0 0 0⎥ ⎥ A=⎢ ⎣0 0 y 1⎦ . 0 0

1

2

(a) An eigenvalue of A is known to be 3; find the value of y. (b) Find the matrix P such that (AP)T AP is diagonal.

504

7.5

Eigenanalysis

For initial values u(0) = 2, v(0) = 8, use the EVD to solve the coupled differential equations u (t) = 3u(t) + v(t), v (t) = −2u(t) + v(t).

7.6

A 4 × 4 Hessenberg matrix has the form ⎡ a 1 b1 c1 ⎢a 2 b2 c2 H=⎢ ⎣0 b c 3 3 0 0 c4

⎤ d1 d2 ⎥ ⎥. d ⎦ 3

d4

Show the following two results. (a) If a2 , b3 , c4 are nonzero, and any eigenvalue λ of H is a real number, then the geometric multiplicity of λ must be equal to 1. (b) If H is similar to the symmetric matrix A, and the algebraic multiplicity of some eigenvalue λ of A is larger than 1, then of a2 , b3 , c4 at least one equals zero. 7.7

Show the following results: (a) If A = A2 = AH ∈ Cn×n and rank(A) = r < n then there is an n × n unitary matrix V such that VH AV = Diag(Ir , 0). (b) If A = AH and A2 = In then there is an unitary matrix V such that VH AV = Diag(Ir , In−r ).

7.8

Let H = I − 2uuH be a Householder transformation matrix. (a) Show that the vector u with unit norm is an eigenvector of the Householder transformation matrix, and find its corresponding eigenvalue. (b) If a vector w is orthogonal to u, show that w is also an eigenvector of the matrix H, and find its corresponding eigenvalue.

7.9

Assume A and B are similar matrices, where ⎡ ⎤ ⎡ −2 0 0 −1 A = ⎣ 2 x 2 ⎦, B=⎣ 0 3 1 1 0

0 2 0

⎤ 0 0 ⎦. y

(a) Find the values of x and y. (b) Find the invertible matrix P such that PT AP = B. 7.10 A block matrix can be used to reduce the dimension of an eigenvalue problem. Let   B X (O is the zero matrix). A= O R Show that det(A − λI) = det(B − λI) det(R − λI). 7.11 Let u be an eigenvector associated with the eigenvalue λ of the matrix A.

Exercises

505

(a) Show that u is also an eigenvector of the matrix A − 3I. (b) Show that u is also an eigenvector of the matrix A2 − 4A + 3I. 7.12 [229] Let



−2 A=⎣ 0 −1

4 0 5

⎤ 3 0 ⎦, 2

Show that

f (x) = x593 − 2x15 .



A593 − 2A15

2 = −A = ⎣ 0 1

7.13 Given the matrix



2 0 ⎣ A= 0 2 0 0 find eAt . 7.14 [229] Given the matrix



1 A = ⎣ −1 0

1 2 1

⎤ −4 −3 0 0 ⎦. −5 −2

⎤ 1 0⎦ , 3

⎤ 2 1 ⎦, 3

find the nonsingular matrix S such that B = S−1 AS is diagonal. 7.15 Show that the matrix An×n satisfying A2 − A = 2I is diagonalizable. 7.16 Given that u = [1, 1, −1]T is an eigenvector of the matrix ⎡ ⎤ 2 −1 2 A=⎣ 5 a 3 ⎦, −1 b −2 (a) find a, b and the eigenvalue corresponding to the eigenvector u; (b) investigate whether the matrix A is similar to a diagonal matrix and explain the reason for your answer. 7.17 Show that a nonzero eigenvalue λ(A ⊗ B) of the Kronecker product A ⊗ B of the matrices A and B is equal to the product of an eigenvalue of A and an eigenvalue of B, i.e., λ(A ⊗ B) = λ(A)λ(B). 7.18 Suppose that both A and B are Hermitian matrices, and λi and μi are respectively the eigenvalues of the matrices A and B. Show that if c1 A + c2 B has the eigenvalue c1 λi + c2 μi , where c1 , c2 are any scalars, then AB = BA. 7.19 Show that if A, B, C are positive definite then the root of |Aλ2 + Bλ + C| = 0 has a negative real part.

506

Eigenanalysis

7.20 Let P, Q, R, X be 2 × 2 matrices. Show that every eigenvalue of the solution X of the matrix equation PX2 + QX + R = O (the zero matrix) is a root of |Pλ2 + Qλ + C| = 0. 7.21 Put     0 1 −1 0 . , B= A= 1 0 0 1

7.22

7.23

7.24

7.25

7.26

Defining the generalized eigenvalues of the matrix pencil (A, B) as the scalar quantities α and β satisfying det(βA − αB) = 0, find α and β. Let the matrix G be the generalized inverse of a matrix A, and let both A and GA be symmetric. Show that the reciprocal of a nonzero eigenvalue of A is an eigenvalue of the generalized inverse matrix G. Use the characteristic equation to show that if λ is a real eigenvalue of the matrix A then the absolute value of the eigenvalue of A + A−1 is equal to or larger than 2. Let A4×4 satisfy three conditions |3I4 + A| = 0, AAT = 2I4 and |A| < 0. Find an eigenvalue of the adjoint matrix adj(A) = det(A)A−1 of the matrix A. Show that the maximum of the quadratic form f = xT Ax at x = 1 is equal to the maximal eigenvalue of the symmetric matrix A. (Hint: Write f in the standard quadratic form.) Let the eigenvalues of An×n be λ1 , λ2 , . . . , λn and the eigenvector corresponding to λi be ui . Find: (a) the eigenvalues of P−1 AP and the corresponding eigenvectors; (b) the eigenvalues of (P−1 AP)T and the corresponding eigenvectors.

7.27 Let p(z) = a0 + a1 z + · · · + an z n be a polynomial. Show that each eigenvector of the matrix A must be an eigenvector of the matrix polynomial p(A), but that the eigenvectors of p(A) are not necessarily eigenvectors of A. 7.28 Let A be an orthogonal matrix and λ be an eigenvalue of A not equal to ±1, but with modulus 1. If u + j v is the eigenvector corresponding to λ, where u and v are real vectors, show that u and v are orthogonal. n 7.29 Show that an n×n real symmetric matrix A can be written as A = i=1 λi Qi , where λi is an eigenvalue of A while Qi is a nonnegative definite matrix that not only meets the orthogonality condition Qi Qj = O,

i = j,

but also is an idempotent matrix, i.e., Q2i = Qi . This representation of the matrix A is known as its spectral factorization [32, p. 64]. 7.30 Given ⎡ ⎤ 3 −1 0 A=⎣ 0 5 −2 ⎦ , 0 0 9

Exercises

507

find a nonsingular matrix S such that the similarity transformation S−1 AS = B is diagonal, and find B. 7.31 The matrix ⎡ ⎤ 1 −1 1 A=⎣ x 4 y ⎦ −3 −3 5 has three linearly independent eigenvectors and eigenvalues λ1 = λ2 = 2. Find the invertible matrix P such that P−1 AP is a diagonal matrix. 7.32 Let the vectors α = [α1 , . . . , αn ]T and β = [β1 , . . . , βn ]T be orthogonal nonzero vectors. Put A = αβ T and find: (a) A2 ; (b) the eigenvalues and eigenvectors of A. 7.33 Given the matrix



1 ⎣ A= 0 1

0 2 0

⎤ 1 0⎦ 1

and B = (kI + A)2 , where k is real number, (a) find the diagonalized matrix Λ corresponding to the matrix B; (b) find for which values of k the matrix is positive definite. 7.34 Given a matrix



⎤ −1 1 0 A = ⎣−4 3 0⎦ , 1 0 2

find its Jordan canonical form and its similarity transformation matrix. 7.35 Use the Jordan canonical form to solve the linear differential equations dx1 = −x1 + x2 , dt dx2 = −4x1 + 3x2 , dt dx3 = x1 + 2x3 . dt 7.36 Given a matrix



⎤ 2 −1 −2 A = ⎣−1 2 2 ⎦, 0 0 1

find g(A) = A8 − 9A6 + A4 − 3A3 + 4A2 − I.

508

Eigenanalysis

7.37 Let

  a  |A| =  5 1 − c

−1 b 0

 c  3  = −1, −a

and its adjoint matrix adj(A) = |A|A−1 have an eigenvalue λ and corresponding eigenvector u = [−1, −1, 1]T . Find the values of a, b, c and λ. 7.38 Find an orthogonal basis of the matrix ⎡ ⎤ 1 −1 −1 −1 ⎢ −1 1 −1 −1 ⎥ ⎥. A=⎢ ⎣ −1 −1 1 −1 ⎦ −1 −1 −1

1

7.39 For a solution of the generalized eigenvalue problem Ax = λBx, the matrix B must be nonsingular. Now assume that B is singular and its generalized inverse matrix is B† . (a) Let (λ, x) be an eigenpair of the matrix B† A. Show that this eigenpair is a generalized eigenpair of the matrix pencil (A, B) if Ax is the eigenvector corresponding to the eigenvalue 1 of the matrix BB† . (b) Let λ, x satisfy Ax = λBx. Show that if x is also the eigenvector corresponding to the eigenvalue 1 of the matrix B† B then (λ, x) is an eigenpair of the matrix B† A. 7.40 Let u1 and v1 be respectively the right and left eigenvectors corresponding to the generalized eigenvalue λ1 of the matrix pencil (A, B), and Bu1 , Bv1  = 1. Show that two matrix pencils (A, B) and (A1 , B1 ), where A1 = A − σ1 Bu1 v1H BH ,

B1 = B − σ2 Au1 v1H BH ,

have the same left and right eigenvectors; it may be assumed that the shift factors σ1 and σ2 satisfy the condition 1 − σ1 σ2 = 0. 7.41 If both A and B are positive definite Hermitian matrices, show that any generalized eigenvalue must be real and that the generalized eigenvectors corresponding to different generalized eigenvalues are orthogonal with respect to the positive matrices A and B, respectively, i.e., H xH i Axj = xi Bxj = 0,

i = j.

7.42 If M, C, K are symmetric and positive definite, show that the roots of the quadratic eigenvalue problem |λ2 M + λC + K| = 0 have a negative real part.

Exercises

509

7.43 Assume that the eigenvalues of the n × n Hermitian matrix A are arranged in the order that λ1 (A) ≥ λ2 (A) ≥ · · · ≥ λn (A). Use the Rayleigh quotient to show that (a) λ1 (A + B) ≥ λ1 (A) + λn (B), (b) λn (A + B) ≥ λn (A) + λn (B). 7.44 Use the Rayleigh quotient to show that for any n×n symmetric matrix A and any n × n positive semi-definite matrix B, the eigenvalues obey the inequality λk (A + B) ≥ λk (A),

k = 1, 2, . . . , n,

and that if B is positive definite then λk (A + B) > λk (A),

k = 1, 2, . . . , n.

7.45 Let A be an n × n matrix (not necessarily positive definite). Show that, for any n × 1 vector x, the following two inequalities hold: (xT Ax)2 ≤ (xT AAT x)(xT x) and

1/2    T x AAT x 1  XT (A + AT )x  . ≤ 2 xT x xT x

7.46 Consider the symmetric matrix sequence Ar = [aij ],

i, j = 1, 2, . . . , r,

where r = 1, 2, . . . , n. Let λi (Ar ), i = 1, 2, . . . , r, be the ith eigenvalue of the matrix Ar and let λ1 (Ar ) ≥ λ2 (Ar ) ≥ · · · ≥ λr (Ar ); then λk+1 (Ai+1 ) ≤ λk (Ai , ) ≤ λk (Ai+1 ). This result is called the Sturmian separating theorem [32, p. 117]. Use the Rayleigh quotient to show this theorem. 7.47 Show that Equation (7.10.1) is a hyperbolic quadratic eigenvalue problem if and only if Q(λ) is negative definite for some eigenvalues λ ∈ R. 7.48 For any two square matrices A, B with the same dimension, show that (a) 2(AAT + BBT ) − (A + B)(A + B)T is positive semi-definite. (b) tr((A + B)(A + B)T ) ≤ 2(tr(AAT ) + tr(BBT )). (c) λ((A + B)(A + B)T ) ≤ 2(λ(AAT ) + λ(BBT )). 7.49 Let A and B be two positive semi-definite matrices with the same dimension; show that [525]  1 tr(AB) ≤ (tr(A) + tr(B)) 2 and that the equality holds if and only if A = B and rank(A) ≤ 1.

510

Eigenanalysis

7.50 For the polynomial p(x) = xn + an−1 xn−1 + · · · + a1 x + a0 , the n × n matrix ⎡ ⎤ 0 1 0 ··· 0 ⎢ 0 0 1 ··· 0 ⎥ ⎢ ⎥ ⎢ .. . . .. ⎥ . .. .. .. Cp = ⎢ . . ⎥ ⎢ ⎥ ⎣ 0 0 0 ··· 1 ⎦ −a0

−a1

−a2

···

−an−1

is known as the companion matrix of the polynomial p(x). Use mathematical induction to prove that for n ≥ 2, one has det(Cp − λI) = (−1)n (a0 + a1 λ + · · · + an−1 λn−1 + λn ) = (−1)n p(λ). 7.51 Let p(x) = x3 + a2 x2 + a1 x + a0 and let λ be a zero point of the polynomial p(x). (a) Write down the companion matrix Cp of the polynomial p(x). (b) Explain why λ3 = −a2 λ2 − a1 λ − a0 , and prove that (1, λ, λ2 ) are the eigenvalues of the companion matrix Cp of the polynomial p(x). 7.52 Show that, for any an n×n symmetric matrix A and any positive semi-definite matrix B, the eigenvalue inequality λk (A + B) ≥ λk (A),

k = 1, 2, . . . , n

is true, and if B is positive definite then λk (A + B) > λk (A),

k = 1, 2, . . . , n.

(Hint: Use the maximum–minimum principle of the Rayleigh quotient.)

8 Subspace Analysis and Tracking

In many considerations involving approximation, optimization, differential equations, communications, signal processing and system science, subspaces play a very important role. The framework of vector subspaces can help us answer some important questions: for example, how to get a good polynomial approximation for complex functions, how to get a good approximate solution of a differential equation, how to design a better signal processor. Questions like these are a core problem in many engineering applications. The vector subspace provides a class of effective methods for addressing these issues, which are known as subspace methods. This chapter focuses on the theoretical analysis of subspaces, and presents some typical applications of subspace methods. In many engineering problems, subspaces are time-variant, there is a need for real-time processing of the received signal or for a real-time control system. In these cases we need to keep track of the subspaces. Therefore, this chapter will focus on the subspace tracking and updating.

8.1 General Theory of Subspaces Before discussing subspaces, it is necessary to present the basic concepts of subspaces and the algebraic and geometric relationships between subspaces.

8.1.1 Bases of Subspaces Let V = Cn be an n-dimensional complex vector space. Consider a subset of m n-dimensional vectors with m < n. DEFINITION 8.1 If S = {u1 , . . . , um } is a vector subset in the vector space V , then the set W of the linear combinations of u1 , . . . , um is called the subspace spanned by u1 , . . . , um and is defined as    W = Span{u1 , . . . , um } = uu = a1 u1 + · · · + am um . (8.1.1) Every vector spanning the subspace W is called a subspace generator of W and the set consisting of all the generators {u1 , . . . , um } is known as the spanning set 511

512

Subspace Analysis and Tracking

of the subspace. A vector subspace containing only the zero vector is said to be a trivial subspace. THEOREM 8.1 (Spanning set theorem) [285, p. 234] Let S = {u1 , . . . , um } be a subset in the vector space V , and W = Span{u1 , . . . , up } (where p ≤ m) be a subspace spanned by p column vectors in S. • If some vector (say uk ) in S is a linear combination of other vectors, then after removing the vector uk from S, the other vectors still span the subspace W . • If W = {0}, i.e., W is a nontrivial subspace, then there must be some subset in S consisting of p linearly independent vectors which span the subspace W . Let all redundant vectors be deleted from a vector set S = {u1 , . . . , um }, and the remaining p linearly independent vectors be {u1 , . . . , up }; then they still span the subspace W . In the sense of spanning the same subspace W , {u1 , . . . , up } and {u1 , . . . , um } are said to be equivalent spanning sets. DEFINITION 8.2 Let W be a vector subspace. A vector set {u1 ,. . ., up } is called a set of basis vectors or a basis set or just a basis if the following two conditions are satisfied: (a) the subspace W = Span{u1 , . . . , up }; (b) the vector set B = {u1 , . . . , up } is a set of linearly independent vectors. On the basis vectors of a subspace, the following two observations are useful. (1) When using the spanning set theorem to delete some vector from a set S, once S becomes a set of linearly independent vectors, we have to stop deleting vectors from S. If we delete a supposedly supernumerary vector that is not in fact a linear combination of the remaining vectors, then the smaller vector set will not span the original subspace W . Therefore, a basis set is the possible small spanning set. In other words, the number of basis vectors spanning the subspace W cannot be fewer. (2) A basis set is also the largest possible set of linearly independent vectors. Let S be a basis set of the subspace W . If adding a vector (say w) from the subspace W to S, then the new vector set cannot be linearly independent, because S spans the subspace W , and the vector w in W itself is a linear combination of all the basis vectors in S. Thus, the number of basis vectors spanning the subspace W cannot be greater. It should be noted that, when referring to a vector space basis set, we do not imply this is unique, it should be emphasized that this is just one basis set. Although there is a variety of choices for the basis set of a vector subspace, all the basis sets necessarily have the same number of linearly independent vectors; a spanning set containing more vectors cannot be counted as a basis set. From this discussion, it is easy to introduce the concept of subspace dimension.

8.1 General Theory of Subspaces

513

DEFINITION 8.3 The number of vectors in any basis set of a subspace W is called the subspace dimension of W , denoted dim(W ). If any basis set does not consist of a finite number of linearly independent vectors then W is said to be an infinite-dimensional vector subspace. Since any zero vector is linearly dependent on other vectors, without loss of generality it is usually assumed that the spanning set of a subspace does not contain any zero vector. For a given spanning set {u1 , . . . , un }, it is easy to construct a basis set for the subspace Span{u1 , . . . , un }; see the discussion in Section 8.2. An important reason for specifying a basis set for the subspace W is to provide coordinates for the subspace W . The following theorem shows the existence of such a coordinate system. THEOREM 8.2 [285] Let W be an n-dimensional vector subspace, and B = {b1 , . . . , bn } be a basis set for W . Then, for any vector x ∈ W , there is a unique set of scalars c1 , . . . , cn such that x can be expressed as x = c 1 b1 + · · · + c n b n .

(8.1.2)

This is called the unique representation theorem for subspace vectors. The coefficients c1 , . . . , cn are unique, so that they form n coordinates that represent the subspace W and thus constitute a coordinate system for the subspace W .

8.1.2 Disjoint Subspaces and Orthogonal Complement In subspace analysis, the relation between subspaces is described by the relation between the subspace elements (i.e., vectors). In the following, we discuss the algebraic relations between subspaces. The intersection of the subspaces S1 , . . . , Sn , S = S1 ∩ · · · ∩ Sn ,

(8.1.3)

is the set consisting of all common vectors of the subspaces S1 , . . . , Sn . If the unique common vector of these subspaces is the zero vector, i.e., S = S1 ∩ · · · ∩ Sn = {0}, then the subspaces S1 , . . . , Sn are said to be disjoint. The union of the disjoint subspaces S = S1 ∪ · · · ∪ Sn is called the direct sum of subspaces, also denoted as S = S1 ⊕ · · · ⊕ Sn .

(8.1.4)

In this case, every vector x ∈ S has the unique decomposition representation x = a1 + · · · + an , where ai ∈ Si . If a vector is orthogonal to all the vectors in subspace S then we say that the vector is orthogonal to the subspace S. By extension, the subspaces S1 , . . . , Sn are called orthogonal subspaces, denoted Si ⊥ Sj , i = j, if ai ⊥ aj holds for all ai ∈ Si , aj ∈ Sj (i = j).

514

Subspace Analysis and Tracking

In particular, the set of all vectors orthogonal to the subspace S comprises a vector subspace called the orthogonal complement subspace of S, denoted S ⊥ . Specifically, let S be a vector space, then the vector space S ⊥ is the orthogonal complement of S if    (8.1.5) S ⊥ = xxT y = 0, ∀ y ∈ S . The dimensions of both the subspace S and its orthogonal complement S ⊥ satisfy the relationship dim(S) + dim(S ⊥ ) = dim(V ).

(8.1.6)

As the name suggests, the orthogonal complement subspace S ⊥ of the subspace S in the vector space V has the double attributes of orthogonality and complementarity. (1) The subspace S ⊥ is orthogonal to S. (2) The vector space V is the direct sum of the subspaces S and S ⊥ , i.e., V = S ⊕ S ⊥ . In other words, the vector space V is the combination of the subspace S and its complement S ⊥ . In particular, every vector u in the vector space Rm can be uniquely decomposed into the sum of a vector x in the subspace S and a vector y in the orthogonal complement S ⊥ , namely u = x + y,

x ⊥ y.

(8.1.7)

This decomposition form is called the orthogonal decomposition of a vector and has the wide applications in signal processing, pattern recognition, automatic control, wireless communications, system science and so on. EXAMPLE 8.1 L2 (R), if

A function u(t) is called strictly square integrable, denoted u(t) ∈ ∞ −∞

|u(t)|2 dt < ∞.

In wavelet analysis, one normally uses multiple resolutions to approximate a square integrable function or signal u(t) ∈ L2 (R). This is known as the multiple resolution analysis of functions or signals. In multiple resolution analysis it is necessary to construct a subspace sequence or chain {Vj |j ∈ Z} in the space L2 (R) such that the subspace sequence has some desired properties. For example, this subspace sequence must have the inclusion · · · ⊂ V−2 ⊂ V−1 ⊂ V0 ⊂ V1 ⊂ V2 ⊂ · · · . From this inclusion it is seen that Vj is a subspace of Vj+1 . Therefore, there must exist the orthogonal complement Wj = Vj⊥ such that Vj+1 = Vj ⊕ Wj , where Vj and Wj are respectively called the scale subspace and the wavelet subspace

8.1 General Theory of Subspaces

515

at resolution 2−j . The multiple resolution analysis satisfying the above condition is known as orthogonal multiple resolution analysis. The following are relationships among disjoint subspaces, orthogonal subspaces and orthogonal complement subspaces. (1) Disjointness is a weaker condition than orthogonality, because that two subspaces are disjoint means only that they have no pair of nonzero common vectors; it does not imply any other relation of these subspaces. On the contrary, when subspaces S1 and S2 are mutually orthogonal, any two vectors x ∈ S1 and y ∈ S2 are orthogonal and therefore have no correlated components, i.e., S1 and S2 must be disjoint. Therefore, two disjoint subspaces are not necessarily orthogonal but two orthogonal subspaces must be disjoint. (2) The concept of an orthogonal complement subspace is a stricter concept than that of an orthogonal subspace: the orthogonal complement S ⊥ of a subspace S in the vector space V must be orthogonal to S but a subspace orthogonal to S is generally not the orthogonal complement of S. There may be more subspaces S1 , . . . , Sp orthogonal to S in V , but there is only one orthogonal complement S ⊥ ; it is uniquely determined. A subspace set {S1 , . . . , Sm } satisfying the relation {0} ⊂ S1 ⊂ · · · ⊂ Sm is called a subspace nest. DEFINITION 8.4 [179] if

For a matrix A, a subspace S ⊆ Cn is called A-invariant x∈S



Ax ∈ S.

EXAMPLE 8.2 Let the eigenvectors of the n × n matrix A be u1 , . . . , un , and let S = Span{u1 , . . . , un }; then because Aui = λi ui , i = 1, 2, . . . , n, we have ui ∈ S



Aui ∈ S,

i = 1, . . . , n.

This shows that the subspace S spanned by the eigenvectors of A is an A-invariant subspace. For any eigenvalue λ of an n × n matrix A, the subspace Null(A − λI) is an A-invariant subspace because u ∈ Null(A − λI)



(A − λI)u = 0



Au = λu ∈ Null(A − λI).

The null space Null(A−λI) is called the eigenspace of the matrix A corresponding to the eigenvalue λ. Let A ∈ Cn×n , B ∈ Ck×k , X ∈ Cn×k and X = [x1 , . . . , xk ]; then the jth column

516

Subspace Analysis and Tracking

of AX = XB is

⎤ b1j x11 + b2j x12 + · · · + bkj x1k ⎢ b1j x21 + b2j x22 + · · · + bkj x2k ⎥ ⎥ ⎢ (AX)j = ⎢ ⎥ .. ⎦ ⎣ . b1j xn1 + b2j xn2 + · · · + bkj xnk ⎡

= b1j x1 + b2j x2 + · · · + bkj xk . Hence, if S = Span{x1 , . . . , xk } then Axj ∈ S,

j = 1, . . . , k.

In other words, the subspace S = Span{x1 , . . . , xk } is an A-invariant subspace, if AX = XB,

A ∈ Cn×n , B ∈ Ck×k , X ∈ Cn×k .

(8.1.8)

In this case, if X has full column rank, and (λ, u) is an eigenpair of the matrix B, i.e., Bu = λu, then premultiplying Bu = λu by the full column rank matrix X gives Bu = λu



XBu = λXu



A(Xu) = λ(Xu).

(8.1.9)

Then, λ(B) ⊆ λ(A), and the equality holds if and only if X is a square nonsingular matrix. That is to say, if X is a nonsingular matrix, then B = X−1 AX is similar to A, and λ(B) = λ(A). This shows again, from the viewpoint of invariant subspaces, that two similar matrices may have the same eigenvalues but their eigenvectors could still be different. The concept of invariant subspaces plays an important role in using the subspaces to track and update iteratively the eigenvalues of a large-dimensional sparse matrix.

8.2 Column Space, Row Space and Null Space Before presenting vector subspace analysis, it is necessary to discuss the basic spaces relating to matrices. These basic spaces are column space, row space and null space.

8.2.1 Definitions and Properties As previously, for a matrix A ∈ Cm×n , its m row vectors are denoted as r1 = [a11 , . . . , a1n ],

...,

and its n column vectors are denoted as ⎡ ⎤ a11 ⎢ ⎥ a1 = ⎣ ... ⎦ , . . . , am1

rm = [am1 , . . . , amn ], ⎡

⎤ a1n ⎢ ⎥ an = ⎣ ... ⎦ . amn

8.2 Column Space, Row Space and Null Space

517

DEFINITION 8.5 If A = [a1 , . . . , an ] ∈ Cm×n then the set of all linear combinations of its column vectors comprises a subspace that is called the column space or column span of the matrix A, denoted Col(A) or Span{a1 , . . . , an }, i.e., Col(A) = Span{a1 , . . . , an } 0 +  n   m α j aj , α j ∈ C . = y ∈ C y =

(8.2.1) (8.2.2)

j=1

Similarly, the set of all linear combination of complex conjugate row vectors r∗1 , . . . , r∗m ∈ Cn of the matrix A is known as the row space or row span of the matrix A, denoted Row(A) or Span{r∗1 , . . . , r∗m }, namely Row(A) = Span{r∗1 , . . . , r∗m } 0 +  m   n ∗ β i ri , β i ∈ C . = y ∈ C y =

(8.2.3) (8.2.4)

i=1

The column space of the matrix A is sometimes denoted as Span{A}, i.e., Col(A) = Span(A) = Span{a1 , . . . , an }.

(8.2.5)

Similarly, the symbol Span(AH ) represents the column space of the complex conjugate transpose matrix AH . Because the column vectors of AH are the conjugate row vectors of A, one has Row(A) = Col(AH ) = Span(AH ) = Span{r1∗ , . . . , r∗m }.

(8.2.6)

That is, the row space of the complex matrix A and the column space of the complex conjugate transpose matrix AH are equivalent. The row space and the column space are two vector subspaces defined directly by the matrix Am×n . In addition, there are other two subspaces defined directly by the matrix transform Ax rather than A. These two subspaces are the range and the null space of the mapping or transform. In Chapter 1, the range of the mapping T was defined as the set of all values of T (x) = 0, while the kernel or null space is defined as the set of all nonzero vectors satisfying T (x) = 0. Naturally, if the linear mapping y = T (x) is a matrix transform from the space Cn to the space Cm , i.e., ym×1 = Am×n xn×1 , then, for a given matrix A, the range of the matrix transform Ax is defined as the set of all possible values of the vectors y = Ax and the null space is defined as the set of the vectors x satisfying Ax = 0. In some publications (especially in engineering), the range and the null space are defined as follows. DEFINITION 8.6 as

If A is an m × n complex matrix then the range of A is defined    Range(A) = y ∈ Cm Ax = y, x ∈ Cn .

(8.2.7)

518

Subspace Analysis and Tracking

The null space of A is also called the kernel of A and is defined as the set of all solution vectors of the homogeneous linear equation Ax = 0, i.e.,    (8.2.8) Null(A) = Ker(A) = x ∈ Cn Ax = 0 . Similarly, the null space of the complex conjugate transpose matrix AH is defined as    (8.2.9) Null(AH ) = Ker(AH ) = x ∈ Cm AH x = 0 . The dimension of the null space is called the nullity of A, namely nullity(A) = dim[Null (A)].

(8.2.10) n

Let A = [a1 , . . . , an ] and x = [α1 , . . . , αn ]T , then Ax = j=1 αj aj , so we have + 0  n   m Range(A) = y ∈ C  y = αj aj , αj ∈ C = Span{a1 , . . . , an }. j=1

Therefore we have Range(A) = Col(A) = Span{a1 , . . . , an }, H

H

Range(A ) = Col(A ) =

Span{r∗1 , . . . , r∗m }.

(8.2.11) (8.2.12)

THEOREM 8.3 If A is an m × n complex matrix then the orthogonal complement (Row(A))⊥ of the row space of A is the null space of A, and the orthogonal complement (Col(A))⊥ of the column space of A is the null space of AH , i.e., (Row(A))⊥ = Null(A),

(Col A)⊥ = Null(AH ).

(8.2.13)

Proof Let the row vectors of A be ri , i = 1, . . . , m. By the matrix multiplication rule it is known that if x lies in the null space Null(A), i.e., Ax = 0, then ri x = 0. This result can be written in the standard orthogonal form of two column vectors: H (rH i ) x = 0, i = 1, . . . , m. This implies that the vector x is orthogonal to the linear H span of rH 1 , . . . , rm , i.e.,   H  H = Row(A). x ⊥ Span rH 1 , . . . , rm = Col A Hence Null(A) and Row(A) are orthogonal. On the contrary, if x and Row(A) = H H Col(AH ) are orthogonal then x is also orthogonal to rH i , i = 1, . . . , m, i.e., (ri ) x = 0; this is equivalent to ri x = 0, i = 1, . . . , m, or Ax = 0. This shows that the set of vectors x orthogonal to Row(A) is the null space of A, and thus (Row(A))⊥ = Null(A). ⊥  In the general equality Row(Bn×m ) = Null(B), let B = AH m×n ; then ⊥     ⊥   Row(AH ) = Null AH ⇒ Col(A) = Null AH . This completes the proof of the theorem.

8.2 Column Space, Row Space and Null Space

519

In summary, there are the following relationships between the vector subspaces of a matrix A. (1) The range and column space of a matrix are equal, i.e., Range(A) = Col(A) = Span(A) = Span{a1 , . . . , an }. (2) The row space of A and the column space of AH are the same, i.e.,       Row(A) = Col AH = Span AH = Range AH . (3) The orthogonal complement of the row space of A is equal to its null space:  ⊥ Row(A) = Null(A). (4) The orthogonal complements of the column space of A and the null space of AH are the same, i.e.,    ⊥ Col(A) = Null AH . Since the column space Col(A) of the matrix A is the collection of linear combinations of its column vectors, Col(A) is determined only by the linearly independent column vectors ai1 , . . . , aik , while the other column vectors, which are linearly dependent on ai1 , . . . , aik , are redundant for generating the column space. The subset {ai1 , . . . , aik } is the maximal linearly independent subset of the column vector set {a1 , . . . , an } if ai1 , . . . , aik are linearly independent and these linearly independent column vectors are not contained in any other linearly independent subset of {a1 , . . . , an }. If {ai1 , . . . , aik } is the maximal linearly independent subset, then Span{a1 , . . . , an } = Span{ai1 , . . . , aik },

(8.2.14)

and the maximal linearly independent subset {ai1 , . . . , aik } is known as a basis of the column space Col(A). Clearly, for a given matrix Am×n , its basis can have different combination forms, but all the bases must have the same number of basis vectors. This common number of vectors is called the dimension of the column space Col(A), denoted dim[Col(A)]. Since the rank of A is defined as the number of its linearly independent column vectors, the rank of A and the dimension of the column space Col(A) are the same quantity. In other words, the rank can also be defined as rank(A) = dim[Col(A)] = dim[Range(A)].

(8.2.15)

For a given matrix A ∈ Cm×n , the following is a comparison between column space and null space [285, p. 226]. 1. The column space Col(A) is a subspace of Cn , while the null space Null(A) is a subspace of Cm .

520

Subspace Analysis and Tracking

2. Col(A) is explicitly defined and is spanned by all the columns of A, while Null(A) is implicitly defined, and has no relation with column vectors of A. 3. Each basis vector of Col(A) is a pivot column (see Definition 1.13) of A, while any basis vector v of Null(A) must meet the condition Av = 0. 4. Each column of A is in Col(A), while there is no obvious relation of Null(A) to the entries of A. 5. A typical vector v in Col(A) has the property that Ax = v has an exact solution x, i.e., [A, v] and A have the same rank; while a typical vector v ∈ Null(A) satisfies Av = 0. 6. Col(A) = Rm if and only if the equation Ax = b has a solution for all b ∈ Rm , while Null(A) = {0} if and only if the equation Ax = 0 has only a zero solution.

8.2.2 Subspace Basis Construction As stated above, the column space and the row space of the matrix Am×n are spanned by n column vectors and m row vectors of A, respectively. But, if the matrix rank is r = rank(A), then only r linearly independent column or row vectors can generate the column space Span(A) and the row space Span(AH ), respectively. Obviously, the use of basis vectors is a more economical and better way for subspace representation. So, how do we find the required basis vectors? In the following, we discuss the constructions of basis vectors for the row space Row(A), the column space Col(A) and the null spaces Null(A) and Null(AH ). It is easy to show the following two results [285]. (1) Elementary row transformations do not change the row space Row(A) or the null space Null(A) of the matrix A. (2) Elementary column transformations do not change the column space Col(A) of A or the null space Null(AH ) of AH . The following theorem provides a method for using elementary row or column transformations to construct the required subspaces. THEOREM 8.4 [285] Let the matrix Am×n become the echelon matrix B via elementary row transformations; then • the nonzero rows of the echelon matrix B constitute a basis set for the row space of A; • the pivot columns constitute a basis set for the column space Col(A) of A. The above discussion can be summarized in terms of the elementary transformation methods for constructing the basis vectors of the column space and the row space for a given matrix.

8.2 Column Space, Row Space and Null Space

521

1. Elementary Row Transformation Method Let a matrix A become the reduced echelon matrix Br via the elementary row transformation, then (1) The nonzero rows corresponding to the pivots of the reduced echelon matrix Br constitute the basis vectors of the row space Row(A). (2) The pivot columns of the matrix A constitute the basis of the column space Col(A). (3) The nonpivot columns of the matrix A constitute the basis of the null space Null(AH ). 2. Elementary Column Transformation Method Let a matrix A become the reduced echelon matrix in column form Bc ; then (1) The nonzero columns corresponding to the pivots in the columns of the reduced echelon matrix Bc constitute the basis vectors of the column space Col(A). (2) The pivot rows of the matrix A constitute the basis of the row space Row(A). (3) The nonpivot rows of the matrix A constitute the basis of the null space Null(A). EXAMPLE 8.3

Find the row space and the column space of a 3 × 3 matrix ⎡ ⎤ 1 2 1 A = ⎣ −1 −1 1 ⎦ . 1 4 5

Solution Method 1: By the elementary column transformations C2 − 2C1 (multiplying the first column by −2, then adding it to the second column), C3 −C1 , C1 +C2 and C3 − 2C2 , we have ⎡ ⎤ 1 0 0 Bc = ⎣ 0 1 0 ⎦ . 3 2 0 This yields two linearly independent column vectors c1 = [1, 0, 3]T and c2 = [0, 1, 2]T which are two basis vectors of the column space Col(A), i.e., ⎧ ⎡ ⎤ ⎡ ⎤⎫ 0 ⎬ ⎨ 1 Col(A) = Span ⎣0⎦ , ⎣1⎦ . ⎩ ⎭ 3 2 From the pivots in the columns of the reduced echelon matrix in column form, Bc , the pivot rows of the matrix A are the first row and the second row, i.e., the row space Row(A) can be written as Row(A) = Span{[1, 2, 1], [−1, −1, 1]}.

522

Subspace Analysis and Tracking

Method 2: Making the elementary row transformations: R2 + R1 (the first row is added to the second row), R3 − R1 and R3 − 2R2 , we have ⎡ ⎤ 1 2 1 B r = ⎣ 0 1 2⎦ , 0 0 0 which yields two linearly independent row vectors, r1 = [1, 2, 1] and r2 = [0, 1, 2]. They constitute the basis vectors of the row space Row(A), namely Row(A) = Span{[1, 2, 1], [0, 1, 2]}. The pivot columns of the matrix A are its first and second columns, and they constitute the basis vectors of the column space Col(A), i.e., ⎧⎡ ⎤ ⎡ ⎤⎫ 1 2 ⎬ ⎨ Col(A) = Span ⎣ −1 ⎦ , ⎣ −1 ⎦ . ⎩ ⎭ 1 4 As a matter of fact, the two solutions are equivalent, because making elementary column transformations of the basis vectors of the column space in method 2, we have ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 1 1 0 1 1 0 1 2 −C1 +C2 C2 −C1 C1 ↔C2 ⎣ −1 −1 ⎦ −−−−−−→ ⎣ −1 0 ⎦ −−−−−→ ⎣1 0⎦ −−−−−→ ⎣0 1⎦. 1 3 2 3 3 2 1 4 These are the same as the basis vectors of the column space obtained from method 1. Similarly, it can be shown that the basis vectors of the row spaces obtained in method 1 and method 2 are equivalent as well.

8.2.3 SVD-Based Orthonormal Basis Construction Because the basis sets obtained by elementary row transformations and by elementary column transformations are equivalent, we can choose any elementary transformation. Elementary row transformations are usually applied. However, if the number of matrix columns is significantly less than the number of rows, then elementary column transformations require fewer computations. Elementary transformation methods give only the linearly independent basis vectors. However, in many applications, it is required to obtain the orthonormal basis vectors of the basic spaces (column space, row space and null space) of a given matrix. Of course, the Gram–Schmidt orthogonalization of the linearly independent basis vectors will give the orthonormal basis vectors, but a simpler method is based on the SVD of a matrix. Let the matrix Am×n with the rank rank(A) = r have the SVD A = UΣVH ,

(8.2.16)

8.2 Column Space, Row Space and Null Space

where



˜ r ], U = [Ur , U

˜ r ], V = [Vr , V

Σ=

523

Σr

Or×(n−r)

O(m−r)×(n−r)

O(n−r)×(n−r)

 .

˜ r are respectively m × r and m × (m − r) matrices, Vr and V ˜ r are Here Ur and U respectively n × r and n × (n − r) matrices and Σr = Diag(σ1 , . . . , σr ). Obviously, the SVD of A can be simplified as follows: r  H A = U r Σr V r = σi ui viH , (8.2.17) i=1

AH = Vr Σ r U H r =

r 

σ i v i uH i .

(8.2.18)

i=1

1. Orthonormal Basis Construction of Column Space Substituting (8.2.17) into the definition formula of Range(A), we have    Range(A) = y ∈ Cm  y = Ax, x ∈ Cn 0 +  r   m H n σi ui vi x, x ∈ C = y ∈ C y = i=1 0 +  r   m H n = y ∈ C y = ui (σi vi x), x ∈ C i=1 + 0  r   m H = y ∈ C y = αi ui , αi = σ i v i x ∈ C i=1

= Span{u1 , . . . , ur }. Using the equivalent relation between the range and the column space, we have Col(A) = Range(A) = Span{u1 , . . . , ur }. This shows that r left singular vectors u1 , . . . , ur constitute a basis set (a group of basis vectors) for the column space Col(A). 2. Orthonormal Basis Construction of Row Space Computing the range of the complex conjugate transposed matrix AH , we get    Range(AH ) = y ∈ Cn  y = AH x, x ∈ Cm 0 +  r   n H m σi vi ui x, x ∈ C = y ∈ C y = + =

i=1

0  r   y ∈ Cn  y = α i v i , α i = σi u H i x∈C i=1

= Span{v1 , . . . , vr },

524

Subspace Analysis and Tracking

and hence Row(A) = Range(AH ) = Span{v1 , . . . , vr }. That is, the right singular vectors v1 , . . . , vr corresponding to the r nonzero singular values constitute a basis set (group of basis vectors) for the row space Row(A). 3. Orthonormal Basis Construction of Null Space Since the rank of A is assumed to be r, the dimension of the null space Null(A) is equal to n − r. Hence we need to search for n − r linearly independent orthonormal vectors as the orthonormal basis of the null space. To this end, consider the vectors satisfying Ax = 0. From the property of singular vectors we get viH vj = 0, ∀ i = 1, . . . , r, j = r + 1, . . . , n. From this it can be seen that Avj =

r 

σi ui viH vj = 0,

∀ j = r + 1, . . . , n.

i=1

Because the n − r right singular vectors vr+1 , . . . , vn corresponding to the zero singular values are linearly independent, and satisfy Ax = 0, they constitute the basis of the null space Null(A), i.e., Null(A) = Span{vr+1 , . . . , vn }. Similarly, we have AH u j =

r 

σi v i u H i uj = 0,

∀ j = r + 1, . . . , m.

i=1

Owing to the linear independence of the m − r right singular vectors ur+1 , . . . , um and the condition AH x = 0, these vectors constitute the basis of the null space Null(AH ), namely Null(AH ) = Span{ur+1 , . . . , um }. Because the left singular-vector matrix U and the right singular-vector matrix V of the matrix A ∈ Cm×n are unitary, the basis set provided by the above SVD methods is an orthonormal basis of the column space, the row space or the null space of A. Summarizing the above discussions, we have the following conclusions on a complex matrix A ∈ Cm×n with rank r. (1) The r left singular vectors u1 , . . . , ur corresponding to the nonzero singular values form an orthonormal basis for the column space Col(A), i.e., Col(A) = Span{u1 , . . . , ur }.

(8.2.19)

(2) The m − r left singular vectors ur+1 , . . . , um corresponding to the zero singular values form an orthonormal basis for the null space Null(AH ), namely Null(AH ) = (Col(A))⊥ = Span{ur+1 , . . . , um }.

(8.2.20)

8.2 Column Space, Row Space and Null Space

525

(3) The r right singular vectors v1 , . . . , vr corresponding to the nonzero singular values form an orthonormal basis for the row space Row(A): Row(A) = Span{v1 , . . . , vr }.

(8.2.21)

(4) The n − r right singular vectors vr+1 , . . . , vn corresponding to the zero singular values form an orthonormal basis for the null space Null(A), i.e.,  ⊥ (8.2.22) Null(A) = Row(A) = Span{vr+1 , . . . , vn }. By [502], the QR decomposition A = QR is an alternative method for constructing an orthogonal basis for the column space of the matrix A. Let the n × n matrix A = [a1 , . . . , an ] be nonsingular, and let Q = [q1 , . . . , qn ]. From a1 = r11 q1 , a2 = r12 q1 + r22 q2 and so on, we have ak = r1k q1 + · · · + rkk qk ,

k = 1, . . . , n.

Then it follows that Span{a1 } = Span{q1 }, Span{a1 , a2 } = Span{q1 , q2 } and Span{a1 , . . . , ak } = Span{q1 , . . . , qk },

k = 1, . . . , n.

Finally, we get Col(A) = Col(Q). In other words, the column vectors of the unitary matrix Q are an orthonormal basis set for the column space of the matrix A.

8.2.4 Basis Construction of Subspaces Intersection We now consider how to construct an orthonormal basis for the intersection of the null spaces Null(A) ∩ Null(B) for two given matrices A ∈ Cm×n and B ∈ Cp×n . Obviously, letting   A ∈ C(m+p)×n , C= B then Cx = 0



Ax = 0 and

Bx = 0.

That is, the null space of C is equal to the intersection of the null space of A and the null space of B, i.e., Null(C) = Null(A) ∩ Null(B). This shows that if the rank of an (m + p) × n matrix C is r = rank(C), then, among its right singular vectors v1 , . . . , vn , the right singular vectors vr+1 , . . . , vn corresponding to the n − r zero singular values constitute an orthonormal basis for the intersection of the null spaces Null(A) ∩ Null(B). However, this is related to the SVD of the (m + p) × n matrix C.

526

Subspace Analysis and Tracking

THEOREM 8.5 [179, p. 583] Let A ∈ Rm×n and {z1 , . . . , zt } be an orthogonal basis set for the null space Null(A). Denote Z = [z1 , . . . , zt ], and define {w1 , . . . , wq } as an orthogonal basis set for the null space Null(BZ), where B ∈ Rp×n . If W = [w1 , . . . , wq ] then the column vectors of ZW constitute an orthogonal basis set for the intersection of the null space Null(A) ∩ Null(B). Given the matrices Am×n , Bp×n , the above theorem provides the construction method for an orthogonal basis set for Null(A) ∩ Null(B). T , determine the effective rank r of the matrix (1) Compute the SVD A = UA ΣA VA A and thus obtain orthogonal basis vectors vr+1 , . . . , vn for the null space Null(A), where vi is the right singular vector of A. Let Z = [vr+1 , . . . , vn ]. (2) Calculate the matrix product Cp×(n−r) = BZ and its SVD C = UC ΣC VCT and determine its effective rank q; this yields orthogonal basis vectors wq+1 , . . . , wn−r for the null space Null(BZ), where wi is the right singular vector of C = BZ. Put W = [wq+1 , . . . , wn−r ]. (3) Compute the matrix product ZW whose column vectors give the orthogonal basis vectors of the intersection of null spaces Null(A) ∩ Null(B) (as Z and W are respectively the matrices consisting of the right singular vectors of A and of BZ, ZW has the orthogonality property).

8.3 Subspace Methods An observed data matrix X inevitably contains observation errors and/or noises as well as the true data matrix A: X = A + W = [x1 , . . . , xn ] ∈ Cm×n ,

(8.3.1)

where xi ∈ Cm×1 is an observation data vector and W represents the additive observation error matrix. In engineering, the column space of an observed data matrix, Span(X) = Span{x1 , . . . , xn },

(8.3.2)

is called the observation data space, while the column space of the observation error matrix, Span(W) = Span{w1 , . . . , wn },

(8.3.3)

is known as the noise subspace.

8.3.1 Signal Subspace and Noise Subspace Define the autocorrelation matrix     RX = E XH X = E (A + W)H (A + W) .

(8.3.4)

8.3 Subspace Methods

527

Assume that the error matrix W = [w1 , . . . , wn ] and the true data matrix A are statistically uncorrelated; then       (8.3.5) R X = E XH X = E AH A + E W H W .  H   H  2 I (i.e., the observation noises are statisPut E A A = R and E W W = σw 2 tically mutually uncorrelated, and have the same variances σw ). Then 2 I. RX = R + σw

Letting rank(A) = r, the EVD of the autocorrelation matrix RX is given by   2 2 I = U Λ + σw I UH = UΠUH , RX = UΛUH + σw where

  2 2 2 2 2 I = Diag σ12 + σw , . . . , σr2 + σw , σw , . . . , σw Π = Σ + σw

with Σ = Diag(σ12 , . . . , σr2 , 0, . . . , 0); σ12 ≥ · · · ≥ σr2 are the nonzero eigenvalues of the true autocorrelation matrix E{AH A}. Obviously, if the signal-to-noise ratio is large enough, i.e., σr2 is significantly 2 greater than σw , then the r largest eigenvalues of the noisy autocorrelation matrix RX , 2 2 , . . . , λr = σr2 + σw λ1 = σ12 + σw

are called the principal eigenvalues and other n − r smaller eigenvalues, 2 2 , . . . , λn = σ w , λr+1 = σw

are known as the minor eigenvalues. Then the EVD of the autocorrelation matrix RX can be written as    UH Σs O s RX = [Us , Un ] = SΣs SH + GΣn GH , O Σn U H n

(8.3.6)

where def

S = [s1 , . . . , sr ] = [u1 , . . . , ur ], def

G = [g1 , . . . , gn−r ] = [ur+1 , . . . , un ],   2 2 Σs = Diag σ12 + σw , . . . , σr2 + σw ,  2  2 Σn = Diag σw , . . . , σw . DEFINITION 8.7 Let X = UΣVH be the SVD of the observed data matrix X with U = [S, G]. If S is the eigenvector matrix corresponding to the r principal eigenvalues λ1 , . . . , λr of X, then Span(S) = Span{u1 , . . . , ur } is called the signal subspace of the observation data space Span(X) and the column space Span(G) = Span{ur+1 , . . . , un } is known as the noise subspace of the observation data space.

528

Subspace Analysis and Tracking

Let us analyze the geometric significance of the signal and noise subspaces. From the characteristics of the construction of subspaces and unitary matrices, we know that the signal and noise subspaces are mutually orthogonal, i.e., Span{s1 , . . . , sr } ⊥ Span{g1 , . . . , gn−r }.

(8.3.7)

Since U is a unitary matrix, we have  H S H = SSH + GGH = I, UU = [S, G] GH namely GGH = I − SSH .

(8.3.8)

Define the projection matrix onto the signal subspace as Ps = SS, S−1 SH = SSH , def

(8.3.9)

since S, S = SH S = I. Then, Ps x = SSH x can be viewed as the projection of the vector x onto the signal subspace, so SSH is often spoken of as the signal subspace. Moreover, (I − Ps )x represents the projection of the vector x that is orthogonal to the signal subspace. From G, G = GH G = I we get the projection matrix onto the noise subspace Pn = GG, G−1 GH = GGH . Therefore we often say that Pn = GGH = I − SSH = I − Ps

(8.3.10)

is the noise subspace. Signal analysis methods using only the signal subspace SSH or the noise subspace GGH are called the signal subspace method and the noise subspace method, respectively. In pattern recognition, the signal subspace method is known as the principal component analysis (PCA) method while the noise subspace method is called the minor component analysis (MCA) method. Subspace applications have the following characteristics [522]. (1) The signal subspace method and the noise subspace method need only a few singular vectors or eigenvectors. If the number of the large singular values (or eigenvalues) of Am×n is less than the number of its small singular values (or eigenvalues) then one should use the signal subspace with the smaller dimension; otherwise use the noise subspace. (2) In many applications, it is not necessary to know singular values or eigenvalues; it is sufficient to know the rank of the matrix and the singular vectors or eigenvectors. (3) In most cases, it is not necessary to know the singular vectors or eigenvectors exactly; it is sufficient to know the basis vectors spanning the signal subspace or noise subspace.

8.3 Subspace Methods

529

(4) Conversion between the signal subspace SSH and the noise subspace GGH can be made using GGH = I − SSH . In the following, we present two typical applications of subspace methods.

8.3.2 Multiple Signal Classification (MUSIC) Let x(t) be a data vector observed at time snapshot t. In array signal processing and spatial spectrum estimation, the vector x(t) = [x1 (t), x2 (t), . . . , xn (t)]T consists of the observation data of n array elements (antennas or sensors). In time-domain spectrum estimation, x(t) = [x(t), x(t−1), . . . , x(t−n−1)]T consists of n observation data samples. Assume that the data vector x(t) is an observation data vector of an array with n elements illuminated by r narrowband signals that form a superposition of incoherent complex harmonics: x(t) =

r 

si (t)a(ωi ) + v(t) = As(t) + v(t),

(8.3.11)

i=1

where A = [a(ω1 ), . . . , a(ωr )] is an n × r array response matrix and a(ωi ) = [1, e j ωi , · · · , e j (n−1)ωi ]T is a direction vector or frequency vector; s(t) = [s1 (t), . . . , sr (t)]T is the random signal vector with zero mean vector and autocorrelation matrix Rs = E{s(t)sH (t)}; v(t) = [v1 (t), . . . , vn (t)]T is the additive noise vector whose components are Gaussian white noises, each with the zero mean and the same variance σ 2 . In harmonic retrieval, the parameters ωi are the frequencies of complex harmonics; in array signal processing, ωi is a spatial parameter equal to 2π(d/λ) sin θi , where d is the distance between the adjacent array elements (assuming the array elements are uniformly arranged on a straight line), λ is the wavelength and θi denotes the incident direction of the ith narrowband signal arriving at the array element, referred to as the direction of arrival (DOA). The question now is how to use N snapshots of the observation data vector x(t) (t = 1, . . . , N ) to estimate the r signal parameters ωi . Suppose that the noise vector v(t) and the signal vector s(t) are statistically uncorrelated, and let the EVD of the autocorrelation matrix Rxx = E{x(t)xH (t)} be given by Rxx = APss AH + σ 2 I = UΣUH  H   S Σ O , = [S, G] O σ 2 In−r GH

(8.3.12)

where Pss = E{s(t)sH (t)} and Σ contains r large or principal eigenvalues that are significantly larger than σ 2 .

530

Subspace Analysis and Tracking

Consider on the one hand

 SH G GH    O Σ O = σ 2 G. = [S, G] I O σ 2 In−r 

Rxx G = [S, G]

Σ O O σ 2 In−r



(8.3.13)

On the other hand, from Rxx = APss AH +σ 2 I we have Rxx G = APss AH G+σ 2 G. Using the result of (8.3.13), we immediately have APss AH G = O, and hence GH APss AH G = O.

(8.3.14)

It is well known that, when Q is nonsingular, tH Qt = 0 if and only if t = 0, so the necessary and sufficient condition for (8.3.14) to hold is AH G = O,

(8.3.15)

as Pss = E{s(t)sH (t)} is nonsingular. Equation (8.3.15) can be rewritten as aH (ω)G = 0T ,

ω = ω 1 , . . . , ωp .

(8.3.16)

Clearly, when ω = ω1 , . . . , ωp , we have aH (ω)G = 0T . Rewriting (8.3.16) in a scalar form, one can define a function similar to the spectrum: 1 1 P (ω) = . (8.3.17) = H

GH a(ω) 22 a (ω)GGH a(ω) The p peak values P (ω1 ), . . . , P (ωp ) give the DOA of the p signals ωi =2π(d/λ) sin θi , i = 1, . . . , p. The function P (ω) defined in (8.3.17) describes the distribution of the spatial parameters, so is called the spatial spectrum. The spatial-spectrum-based signal classification method is called the multiple signal classification (MUSIC) method, which was proposed independently by Schmidt [426] and Biemvenu and Kopp [47] in 1979. Later, in 1986 Schmidt [427] republished his paper. Substituting (8.3.10) into (8.3.17) yields P (ω) =

aH (ω)

1   . I − SSH a(ω)

(8.3.18)

Since GGH and SSH represent the signal subspace and the noise subspace, respectively, Equations (8.3.17) and (8.3.18) can be viewed as the noise subspace method and the signal subspace method, respectively. In practical applications, ω is usually divided into hundreds of equal interval units, ωi = 2πiΔf.

(8.3.19)

For example, for Δf = 0.5/500 = 0.001, P (ω) is computed for every ωi to determine

8.3 Subspace Methods

531

the ω values corresponding to all the spectrum peaks. Hence, the MUSIC algorithm needs to make a global search in the frequency axis.

8.3.3 Subspace Whitening Let a be an m × 1 random vector with zero mean and covariance matrix Ca = E{aaH }. If the m × m covariance matrix Ca is nonsingular, and is not equal to the identity matrix, then the random vector a is said to be a colored or nonwhite random vector. Assume that the EVD of the covariance matrix is Ca = E{aaH } = VDVH and set W = VD−1/2 VH = C−1/2 ; a

(8.3.20)

then the covariance matrix of the linear transform b = Wa = C−1/2 a a

(8.3.21)

is equal to the identity matrix, i.e.,  H Cb = E{bbH } = WCa WH = C−1/2 VDVH C−1/2 = I, a a

(8.3.22)

−1/2

as Ca = VD−1/2 VH is a Hermitian matrix. The above equation shows that, after the linear transformation Wa, the original colored random vector becomes a −1/2 white random vector. The linear transformation matrix W = Ca is called the whitening matrix of the random vector a. However, if the m × m covariance matrix Ca is singular or rank-deficient, for example rank(Ca ) = n < m, then there is no whitening matrix W such that WCa WH = I. In this case, one should consider whitening a random vector a in the range space V = Range(Ca ) = Col(Ca ). This whitening is known as subspace whitening and was proposed by Eldar and Oppenheim in 2003 [145]. Let the EVD of the rank-deficient covariance matrix Ca be  Ca = [V1 , V2 ]

Dn×n

On×(m−n)

O(m−n)×n

O(m−n)×(m−n)



V1H V2H

 ,

(8.3.23)

and let W = V1 D−1/2 V1H .

(8.3.24)

Then the covariance matrix of the linear transform b = Wa = V1 D−1/2 V1H a is

532

Subspace Analysis and Tracking

given by Cb = E{bbH } = WCa WH

   D O V1H V1 D−1/2 V1H O O V2H   O D−1/2 V1H O O

= V1 D−1/2 V1H [V1 , V2 ] = [V1 D =

 In O

−1/2

 O . O

 , O]

D O

That is, b = Wa is a white random vector in the subspace Range(Ca ). Therefore, W = V1 D−1/2 V1H is called the subspace whitening matrix.

8.4 Grassmann Manifold and Stiefel Manifold Consider the minimization of the objective function J(W), where W is an n × r matrix. The common constraints on W are of two types, as follows. (1) Orthogonality constraint W is required to satisfy an orthogonality condition, either WH W = Ir (n ≥ r) or WWH = In (n < r). The matrix W satisfying such a condition is called semi-orthogonal. (2) Homogeneity constraint It is required that J(W) = J(WQ), where Q is an r × r orthogonal matrix. There are two classes of optimization problems: in one class both the orthogonality constraint and the homogeneity constraint are adapted; the other uses only the orthogonality constraint.

8.4.1 Equivalent Subspaces DEFINITION 8.8 (Linear manifold) Let H be a subspace in the space V , and let L denote the set of linear combinations of finite elements in H, i.e., +  0 n    L = ξ ξ = ai ηi , ηi ∈ H . i=1

Then L is known as the linear manifold spanned by H. Two matrices are said to be equivalent, if the subspaces spanned by their column vectors are the same. In other words, sets of equivalent matrices have the same column spaces, i.e., such a subspace is invariant with respect to any choice of basis. Therefore, this class of subspaces is known as invariant subspaces. In many engineering applications, the projection matrix PH onto a subspace H

8.4 Grassmann Manifold and Stiefel Manifold

533

and the orthogonal projection matrix P⊥ H = I − PH play a key role. In particular, invariant subspaces can be described using a projection matrix. 1. Tall-Skinny Matrix When n > r, the n × r matrix W is called a tall-skinny matrix. Let M be an r × r nonsingular matrix, and W1 = W2 M be a tall-skinny matrix. If H = Span(W1 ) = Col(W1 ) and S = Col(W2 ) then the projection matrix onto the subspace H is given by PH = W1 (W1H W1 )† W1H = W2 M(MH W2H W2 M)† MH W2H = W2 (W2H W2 )† W2H = PS . Because the projection matrix PH onto the subspace H is equal to the projection matrix PS , one has PH x = PS x,

∀ x ∈ Cn .

(8.4.1)

The column subspaces H and S such that PH x = PS x are called equivalent subspaces. In other words, if M is an r × r nonsingular matrix, then the two tall-skinny matrices W and WM span the same or equivalent subspaces. 2. Short Matrix When n < r, the n × r matrix W is called a short matrix. Let M be an n × n nonsingular matrix, and W1 = MW2 be a short matrix. If H = Row(W1 ) and S = Row(W2 ) then the projection matrix onto the subspace H is given by PH = W1H (W1 W1H )† W1 = W2H MH (MW2 W2H MH )† MW2 = W2H (W2 W2H )† W2 = PS . Hence, for any row vector x ∈ C1×r , one has xPH = xPS . This shows that the row subspaces H = Row(W1 ) = Row(MW2 ) and S = Row(W2 ) are equivalent subspaces.

8.4.2 Grassmann Manifold In optimization problems, a full column (or row) rank matrix M is usually constrained to be semi-orthogonal. 1. Tall-Skinny Semi-Orthogonal Matrix If a tall-skinny matrix W∈ Cn×r satisfies the semi-orthogonality constraint WH W=

534

Subspace Analysis and Tracking

Ir , then W is known as a tall-skinny semi-orthogonal matrix. For any a tall-skinny semi-orthogonal matrix W, its column space H = Col(W) can be equivalently represented by the projection matrix PH = W(WH W)−1 WH = WWH .

(8.4.2)

For an unconstrained optimization problem min J(W) with tall-skinny matrix W, its solution is a single matrix W. However, for an optimization problem subject to the semi-orthogonality constraint WH W = Ir and the homogeneity constraint J(W) = J(WQ), where Q is an r × r orthogonal matrix, i.e., min J(W)

subject to

WH W = Ir , J(W) = J(WQ),

(8.4.3)

its solution is any member of the set of tall-skinny semi-orthogonal matrices defined by    Gr(n, r) = W ∈ Cn×r  WH W = Ir , W1 W1H = W2 W2H . (8.4.4) This set is known as the Grassmann manifold of the tall-skinny semi-orthogonal matrices. The Grassmann manifold was proposed by Grassmann in 1848, but it only became widely known in the 1980s [2]. The original definition of the Grassmann manifold can be found in [184, Chapter 3, Section 1]. 2. Short Semi-Orthogonal Matrix Let W ∈ Cn×r with n < r be a short matrix. If the short matrix W meets the semi-orthogonality constraint WWH = In then W is called a short semi-orthogonal matrix. For the unconstrained optimization problem min J(W) with short matrix W, its solution is a single short matrix W. However, for an optimization problem subject to the semi-orthogonality constraint WWH = In and the homogeneity constraint J(W) = J(QW), where Q is an n × n orthogonal matrix, i.e., min J(W) subject to

WWH = In , J(W) = J(QW),

(8.4.5)

its solution is any member of the set of short semi-orthogonal matrices given by    Gr(n, r) = W ∈ Cn×r  WWH = In , W1H W1 = W2H W2 , (8.4.6) which is the Grassmann manifold of short semi-orthogonal matrices. Clearly, both the semi-orthogonality constraint and the homogeneous constraint greatly ease the solution of an optimization problem. The Grassmann manifold has important applications in sciences and engineering. On the geometric characteristics of the Grassmann manifold, Edelman et al. [142] in 1998 gave a systematic interpretation.

8.4 Grassmann Manifold and Stiefel Manifold

535

8.4.3 Stiefel Manifold Next we consider an optimization problem with only the semi-orthogonality constraint: W=

argmin

J(W)

subject to

W H W = Ir ,

(8.4.7)

J(W)

subject to

WWH = In .

(8.4.8)

W∈Cn×r ,n>r

or W=

argmin W∈Cn×r ,n r or

   St(n, r) = W ∈ Cn×r  WWH = In , n < r .

(8.4.10)

This manifold was proposed by Stiefel [456] in the 1930s in a topological study (Stiefel together with Hestenes [210] proposed the famous conjugate gradient algorithm in 1952). There are the following relationships between the Grassmann manifold and the Stiefel manifold. (1) The Stiefel manifold St(n, r) is a set of n×r semi-orthogonal matrices WH W = Ir (n > r) or WWH = In (n < r). Any point on the Stiefel manifold St(n, r) represents a matrix in the set of n × r semi-orthogonal matrices. (2) The Grassmann manifold Gr(n, r) consists of the matrices spanning the same column space W1 W1H = W2 W2H (n > r) or the same row space W1H W1 = W2H W2 (n < r) in the Stiefel manifold St(n, r) . The set of all the r ×r orthogonal matrices Q forms an orthogonal group, denoted Or , and is defined as    H Or = Qr ∈ Cr×r  QH (8.4.11) r Qr = Qr Qr = I r . The orthogonal group, the Grassmann manifold and the Stiefel manifold are three classes of subspace manifolds. To discuss the relations among these three classes of subspace manifolds, let W be a point on the Stiefel manifold, i.e., W ∈ St(n, r) is an n × r semi-orthogonal matrix. Collecting the n × (n − r) matrices W⊥ ∈ St(n, n − r) satisfying the H H orthogonality conditions W⊥ W⊥ = In−r and W⊥ W = O(n−r)×r , then [W, W⊥ ] constitutes an orthogonal group On . If we let Q be any (n − r) × (n − r) orthogonal matrix such that W⊥ Q = O then the set of matrices  forms another orthogonal  Q I = W. This shows that the group, On−r . Notice that matrix product [W, W⊥ ] Q

536

Subspace Analysis and Tracking

semi-orthogonal matrix Wn×r can be determined by the n × n orthogonal group On and the (n−r)×(n−r) orthogonal group On−r . From the matrix multiplication in the above equation it can be seen that a point on the Stiefel manifold St(n, r) can be represented by the quotient of two orthogonal groups On /On−r : St(n, r) = On /On−r .

(8.4.12)

Moreover, using the semi-orthogonal matrix Wn×r to represent a point on the Stiefel manifold, the matrices Us such that W = Us Q (where Q is any r × r orthogonal matrix) or Us = WQ−1 constitute a point (equivalent to a subspace class) on the Grassmann manifold Gr(n, r) . Therefore, if the inverse matrix operation is regarded as matrix division then the Grassmann manifold Gr(n, r) can be represented as the quotient of the Stiefel manifold St(n, r) and the orthogonal matrix Q, i.e., Gr(n, r) = St(n, r) /Or ,

(8.4.13)

where Or denotes an r × r orthogonal group. If we substitute (8.4.12) into (8.4.13) then we see that the Grassmann manifold can also be represented as the quotient of two orthogonal groups: Gr(n, r) = On /(Or × On−r ).

(8.4.14)

Table 8.1 summarizes the relationships among the Grassmann manifold, the Stiefel manifold and the orthogonal group. Table 8.1 Subspace manifold representations [142] Subspace manifold

Symbol

Matrix form

Quotient

Orthogonal group

On

n × n matrix

—–

Stiefel manifold

St(n, r)

n × r matrix

On /On−r  St(n, r) /Or or On /(Or × On−r )

Grassmann manifold Gr(n, r)

—–

8.5 Projection Approximation Subspace Tracking (PAST) The tracking and updating of eigenspaces are mainly applied in real-time signal processing, so they should be fast. The following factors need to be taken into account in the design of a fast subspace tracking algorithm. (1) The subspace at time n can be updated from the subspace at time n − 1.

8.5 Projection Approximation Subspace Tracking (PAST)

537

(2) The change in the covariance matrix from time n − 1 to time n should be of as low a rank change as possible (best are rank-1 or rank-2 changes). (3) Only the low-dimensional subspaces need to be tracked. Eigenspace tracking and updating methods fall into four classes, as follows. 1. Orthogonal basis tracking This method uses only the orthogonal basis without using the eigenvectors themselves, which can simplify the adaptive tracking problem of eigenspaces. 2. Rank-1 updating The covariance matrix of a nonstationary signal at time k can be regarded as the sum of the covariance matrix at time k −1 and another rank-1 matrix. Therefore, the tracking of the EVD of the covariance matrix is closely related to the rank-1 updating. 3. Projection approximation The determination of an eigenspace is solved by unconstrained optimization; this method is called projection approximation subspace tracking [522]. 4. Lanczos subspace tracking The subspace tracking of a time-varying data matrix can be performed using the Lanczos-type iteration and random approximation [166]. Xu et al. [516], [518] proposed the tri-Lanczos and the bi-Lanczos subspace tracking algorithms: the tri-Lanczos algorithm tracks the EVD of a covariance matrix, and the bi-Lanczos algorithm tracks the SVD of a data matrix, and the numbers of principal eigenvalues and principal singular values can be detected and estimated. This section focuses upon method 3, the projection approximation subspace tracking method.

8.5.1 Basic PAST Theory The minimization of an objective function J(W) with orthogonality constraint H Wn×r Wn×r = Ir and homogeneity constraint J(W) = J(WQr×r ) can be made equivalent to an unconstrained optimization problem. Let C = E{xxH } denote the autocorrelation matrix of an n × 1 random vector, and let the objective function be 32 22 4 J(W) = E 2x − WWH x22 3 H  4 = E x − WWH x x − WWH x       (8.5.1) = E xH x − 2E xH WWH x + E xH WWH WWH x .

538

Subspace Analysis and Tracking

Because n        E |xi |2 = tr E{xxH } = tr(C), E xH x = i=1

)  *   E x WW x = tr E WH xxH W = tr WH CW , )    * E xH WWH WWH x = tr E WH xxH WWH W   = tr WH CWWH W , 

H

H



the objective function can be represented as the trace form J(W) = tr(C) − 2tr(WH CW) + tr(WH CWWH W),

(8.5.2)

where W is an n × r matrix whose rank is assumed to be r. Regarding the minimization problem min J(W), important questions are: • Is there a global minimizer W of J(W)? • What is the relationship between the minimizer W and the signal subspace of the autocorrelation matrix C? • Are there other local minima of J(W)? The following two theorems provide answers to the above three questions. THEOREM 8.6 [522] The matrix W is a stationary point of J(W) if and only if W = Ur Q, where Ur ∈ Cn×r consists of r different eigenvectors of the autocorrelation matrix C, and Q ∈ Cr×r is any unitary matrix. At every stationary point, the value of the objective function J(W) is equal to the sum of the eigenvalues whose corresponding eigenvectors are not in Ur . THEOREM 8.7 [522] All its stationary points are saddle points of the objective function J(W) unless Ur consists of r principal eigenvectors of the autocorrelation matrix C. In this particular case J(W) reaches a global minimum. Theorems 8.6 and 8.7 show the following facts. 1. In the unconstrained minimization of the objective function J(W) in (8.5.1), it is not required that the columns of W are orthogonal, but the above two theorems show that minimization of the objective function J(W) in (8.5.1) will automatically result in a semi-orthogonal matrix W such that WH W = I. 2. Theorem 8.7 shows that when the column space of W is equal to the signal subspace, i.e., Col(W) = Span(Ur ), the objective function J(W) reaches a global minimum, and it has no other local minimum. 3. From the definition formula (8.5.1) it is easy to see that J(W) = J(WQ) holds for all r×r unitary matrices Q, i.e., the objective function automatically satisfies the homogeneity constraint.

8.5 Projection Approximation Subspace Tracking (PAST)

539

4. Because the objective function defined by (8.5.1) automatically satisfies the homogeneity constraint, and its minimization automatically has the result that W satisfies the orthogonality constraint WH W = I, the solution W is not uniquely determined but is a point on the Grassmann manifold. 5. The projection matrix P = W(WH W)−1 WH = WWH = Ur UH r is uniquely determined. That is to say, the different solutions W span the same column space. 6. When r = 1, i.e., the objective function is a real function of a vector w, the solution w of the minimization of J(w) is the eigenvector corresponding to the largest eigenvalue of the autocorrelation matrix C. Therefore, the minimization of the objective function J(W) with orthogonality constraint and homogeneity constraint becomes an SVD or EVD problem, as follows. (1) Use the data vector x(k) to form the data matrix X = [x(1), . . . , x(N )]; then compute the SVD of X and determine its effective rank r, which yields r principal singular values and the corresponding left singular-vector matrix Ur . The optimal solution of the minimization problem is given by W = Ur . (2) Compute the EVD of the autocorrelation matrix C = XXH to get the eigenvector matrix Ur corresponding to the r principal eigenvalues. This eigenvector matrix Ur is the optimal solution of the minimization problem. However, in practical applications, the autocorrelation matrix C is usually timevarying, and thus its eigenvalues and eigenvectors are time-varying as well. From Equation (8.5.2) it can be seen that in the time-varying case the matrix differential of the objective function J(W(t)) is given by dJ(W(t)) * ) = −2tr WH (t)C(t)dW(t) + (C(t)W(t))T dW∗ (t) ) + tr (WH (t)W(t)WH (t)C(t) + W(t)C(t)W(t)WH (t))dW(t) * + (C(t)W(t)WH (t)W(t) + W(t)WH (t)C(t)W(t))T dW∗ (t) , which yields the gradient matrix ∇W J(W(t)) = −2C(t)W(t) + C(t)W(t)WH (t)W(t) + W(t)WH (t)C(t)W(t) = W(t)WH (t)C(t)W(t) − C(t)W(t), where the semi-orthogonality constraint WH (t)W(t) = I has been used. Substituting C(t) = x(t)xH (t) into the above gradient matrix formula, one obtains a gradient descent algorithm for solving the minimization problem W(t) =

540

Subspace Analysis and Tracking

W(t − 1) − μ∇W J(W(t)), as follows: y(t) = WH (t)x(t)





(8.5.3)

W(t) =W(t − 1) + μ x(t) − W(t − 1)y(t) y (t). H

(8.5.4)

However, this gradient descent algorithm for updating W(t) has a low convergence rate, and its ability to track a time-varying subspace is poor. A better method is the following recursive least squares (RLS) algorithm. Define an exponential weighting objective function J1 (W(t)) =

t 

22 2 β t−i 2x(i) − W(t)WH (t)x(i)22

i=1

=

t 

2 22 β t−i 2x(i) − W(t)y(i)22 ,

(8.5.5)

i=1

where 0 < β ≤ 1 is a forgetting factor and y(i) = WH (t)x(i). By the adaptive filtering theorem, the optimal solution of the minimization problem min J1 (W) is the Wiener filter, given by W(t) = Cxy (t)C−1 yy (t),

(8.5.6)

where Cxy (t) = Cyy (t) =

t  i=1 t 

β t−i x(i)yH (i) = βCxy (t − 1) + x(t)yH (t),

(8.5.7)

β t−i y(i)yH (i) = βCyy (t − 1) + y(t)yH (t).

(8.5.8)

i=1

8.5.2 PAST Algorithms Substituting Equations (8.5.7) and (8.5.8) into Equation (8.5.6), and applying the matrix inverse lemma, one obtains the projection approximation subspace tracking (PAST) algorithm [522]; see Algorithm 8.1. In Algorithm 8.1, Tri[A] denotes an operation computing only the upper (or lower) triangular part of A then copying this as the lower (or upper) triangular part of A. The PAST algorithm extracts the signal subspace from the data vector and is a PCA method. In particular, if the first formula in the above algorithm is replaced by   y(t) = g WH (t − 1)x(t) , (8.5.9) where g(z(t)) = [g(z1 (t)), g(z2 (t)), . . . , g(zn (t))]T is the nonlinear vector function, then the PAST algorithm becomes the blind source separation algorithm based on

8.5 Projection Approximation Subspace Tracking (PAST) Algorithm 8.1

541

PAST algorithm [522]

  task: Solve min E x − WWH x22 . W

input: Observation vector x(t). initialize: P(0) and W(0). repeat 1. y(t) = WH (t − 1)x(t). 2. h(t) = P(t − 1)y(t). 3. g(t) = h(t)/(β + yH (t)h(t)). 4. P(t) = β −1 Tri[P(t − 1) − g(t)hH (t)]. 5. e(t) = x(t) − W(t − 1)y(t). 6. W(t) = W(t − 1) + e(t)gH (t). 7. exit if W(t) − W(t − 1)F ≤ . return t ← t + 1. output: W = W(t).

nonlinear PCA. The LMS algorithm and the RLS algorithm for nonlinear PCA were developed in [358] and [371], respectively. Algorithm 8.2 shows the PAST via deflation (PASTd) algorithm.

PAST via deflation (PASTd) algorithm [522]   task: Solve min E x − WWH x22 .

Algorithm 8.2 W

input: Observation vector x(t). initialize: di (0) and wi (0). repeat 1. x1 (t) = x(t). 2. for i = 1, . . . , r do 2.1

yi (t) = wiH (t − 1)xi (t).

2.2

di (t) = βdi (t − 1) + |yi (t)|2 .

2.3 2.4

ei (t) = xi (t) − wi (t − 1)yi (t).   wi (t) = wi (t − 1) + ei (t) yi∗ (t)/di (t) .

2.5

xi+1 (t) = xi (t) − wi (t)yi (t).

3. end for  4. exit if ri=1 wi (t) − wi (t − 1)2 ≤ . return t ← t + 1. output: W = [w1 (t), . . . , wr (t)].

542

Subspace Analysis and Tracking

The PASTd algorithm can be generalized to an algorithm for tracking both the rank and the subspace; see [523].

8.6 Fast Subspace Decomposition In the terminology of Krylov subspaces, the signal subspace tracking of a sample ˆ becomes the tracking of the Rayleigh–Ritz (RR) vectors autocorrelation matrix R ˆ of R. The starting point of this method is that both the span of the principal ˆ and the span of its RR vectors are eigenvectors of a sample autocorrelation matrix R asymptotically equivalent estimates of the signal subspace of R. Because RR vectors can be found efficiently using the Lanczos algorithm, this method can implement the fast decomposition of signal subspaces.

8.6.1 Rayleigh–Ritz Approximation Given an observed data vector x(t) = s(t) + n(t) ∈ CM , where s(t) and n(t) are the signal and additive noise vectors, respectively; and the entries of n(t) have the zero mean and the same variance σ. Consider the autocorrelation matrix of the observed data vector,       Rxx = E x(t)xH (t) = E s(t)sH (t) + E n(t)nH (t) = Rss + σI (8.6.1) whose sample autocorrelation matrix is  ˆ xx (n) = 1 R x(t)xH (t) n t=1 n

1 n−1 ˆ Rxx (n − 1) + x(n)xH (n) n n ˆ ss (n) + σI. =R

=

(8.6.2)

ˆ xx (n) at the time n can be updated in Thus the sample autocorrelation matrix R time by ˆ xx (n) = n − 1 R ˆ xx (n − 1) + 1 x(n)xH (n), R (8.6.3) n n ˆ ss (n) is diand the sample autocorrelation matrix of the unknown signal vector R ˆ ˆ rectly estimated by Rss (n) = Rxx (n) − σI. The question now is how to track the ˆ ss (n)} or equivalently how to track the principal eigenvecsignal subspace Span{R ˆ ss (n) using the sample autocorrelation matrix R ˆ xx (n). The direct EVD tors of R ˆ of Rxx (n) at any given time does not meet the requirement of real-time tracking of the principal eigenvectors or the signal subspace. The Krylov subspace and the Rayleigh–Ritz (RR) approximation provide a fast subspace decomposition. ˆ xx (n) and B = R ˆ ss (n) + σI. Then (8.6.2) becomes For convenience, let A = R A = B + σI

or

B = A − σI.

(8.6.4)

8.6 Fast Subspace Decomposition

543

Let the EVD of the Hermitian matrix A ∈ CM ×M be A=

M 

λ k uk u H k ,

(8.6.5)

k=1

where (λk , uk ) are the kth eigenvalue and eigenvector of A and it is assumed that λ1 > · · · > λd > λd+1 = · · · = λM = σ. That is to say, {λk − σ, uk }dk=1 are respectively the signal (i.e., principal) eigenvalues and the signal eigenvectors of B, owing to B = A − σI. Now consider the RR approximation problem for the signal eigenvalues and the signal eigenvectors. To this end, introduce the following two definitions. DEFINITION 8.9

For an m-dimensional subspace S m , if (m)

Ayi (m)

(m) (m) yi

− θi

⊥ Sm

(8.6.6)

(m)

then θi and yi are respectively the RR values and the RR vectors of the M ×M Hermitian matrix A with respect to the subspace S m . (m)

The RR values and the RR vectors are combined in Ritz pairs (θi

(m)

, yi

).

DEFINITION 8.10 The order-m Krylov subspace Km (A, f ) generated by an M ×M matrix A and a vector f of dimension M is the linear subspace spanned by the images of f under the first m powers of A (starting from A0 = I), that is, Km (A, f ) = Span{Km (A, f )} = Span{f , Af , . . . , Am−1 f },

(8.6.7)

where Km (A, f ) = [f , Af , . . . , Am−1 f ] is called the Krylov matrix. An important property of the Krylov subspace is that [519] Km (A, f ) = Km (A − σI, f )

(8.6.8)

for any scalar α. Choose the order-m Krylov subspace Km (A, f ) as the subspace S m in (8.6.6), and suppose that f is not orthogonal to any of the d principal eigenvectors or to at least one of the nonprincipal eigenvectors and that the d principal eigenvalues are distinct. Then, by [517], dim(Kd+1 (A, f )) = d + 1. (m)

(m)

LEMMA 8.1 [375] Let (θi , yi ) i = 1, . . . , m, be the RR values and the RR vectors of a Hermitian matrix A with respect to the subspace S m , and let Q = [q1 , . . . , qm ] be an orthogonal basis of the same subspace. If (αi , ui ) is the ith eigenpair of the m × m matrix QH AQ, where i = 1, . . . , m, then (m)

θi

= αi (m)

and

(m)

yi

(m)

= Qui .

(8.6.9)

If d = rank(A) then Span{y1 , . . . , yd } = Span{u1 , . . . , ud }, i.e., for an mdimensional subspace S m = Km (A, f ), the signal subspace can be approximated

544

Subspace Analysis and Tracking (m)

(m)

by the span of its RR vectors y1 , . . . , yd . This approximation is called the Rayleigh–Ritz approximation, simply called the RR approximation. (m)

The RR pairs (θi , yi ) of an M × M higher-dimensional matrix A can be equivalently estimated via (8.6.9) by the eigenpairs (αi , ui ) of an m × m low-dimensional matrix QH AQ. The remaining issue is how to construct the orthogonal basis matrix Q ∈ CM ×m of the Krylov subspace Km (A, f ). The properties of the Rayleigh–Ritz approximation are evaluated by the asymptotic properties of the RR values and the RR vectors: for m > d, their errors are given respectively by θk

ˆ = O(N −m−d ), −λ k

(m) yk

ˆ k = O(N −(m−d)/2 ), −u

(m)

k = 1, . . . , d,

(8.6.10)

k = 1, . . . , d.

Here N is the data length. Therefore, once m ≥ d + 2, then √ √ (m) ˆ k − uk ), k = 1, . . . , d, lim N (yk − uk ) = lim N ( u N →∞

N →∞

(m)

(8.6.11)

(8.6.12)

(m)

ˆ d } are asymptotically equivu1 , . . . , u i.e., both Span{y1 , . . . , yd } and Span{ˆ alent estimates of the signal subspace Span{u1 , . . . , ud }, so the tracking of the signal subspace of a Hermitian matrix A becomes the tracking of the RR vectors (m) (m) y1 , . . . , yd of A.

8.6.2 Fast Subspace Decomposition Algorithm (m)

(m)

By Lemma 8.1, the tracking of the RR vectors y1 , . . . , yd of A is equivalent to (m) (m) the tracking of the principal eigenvectors u1 , . . . , ud of Qm AQm . The Lanczos method provides an efficient way of finding an orthonormal basis Qm = [q1 , . . . , qm ] for Km (A, f ). The orthonormal basis constructed by the Lanczos method is called the Lanczos basis. The use of a Lanczos basis means that (m) (m) the RR pairs (θi , yi ), i = 1, . . . , d are closely related to the eigenpairs of the tridiagonalized matrix of A. Let Qm = [q1 , . . . , qm ] be a Lanczos basis. It is known from [375] that ⎡ ⎤ α 1 β1 0 ⎢ β 1 α2 β 2 ⎥ ⎢ ⎥ ⎢ ⎥ . . . H ˆ . . . ⎢ ⎥. . . . (8.6.13) Qm A Qm = Tm = ⎢ ⎥ ⎢ ⎥ . . ⎣ . αm−1 βm−1 ⎦ 0

βm−1

αm

The iteration based on (8.6.13) is called the tri-Lanczos iteration for tridiagonalization of the Hermitian matrix A. Once the tridiagonalization Tm of the Hermi(m) (m) tian matrix A has been achieved, one can obtain the eigenpairs (αi , ui ), i =

8.6 Fast Subspace Decomposition

545

1, . . . , m, by the EVD of the m × m tridiagonal matrix Tm , and thus find the RR (m) (m) vectors y1 , . . . , yd of A by (8.6.9). The most attractive property of a Lanczos algorithm is that the original larger problem of finding the expected eigenvalues and eigenvectors of an M ×M (complex) ˆ is transformed into the smaller problem sample covariance (Hermitian) matrix A of computing the eigenvalue decomposition of an m × m (real) tridiagonal matrix Tm , because m is usually much smaller than M . Lanczos algorithms are divided into two kinds: one is the tri-Lanczos iteration for the tridiagonalization of the Hermitian matrix, and another is the bi-Lanczos iteration for bidiagonalization of any matrix. The tri-Lanczos iteration is available only for the tridiagonalization of a sample ˆ = N −1 N x(n)xH (n), and is not applicable for any autocorrelation matrix A n=1 nonsquare data matrix. For an N × M data matrix XN = N −1/2 [x(1), . . . , x(N )]H ˆ = XH X , the bi-Lanczos iteration can be used directly for the data such that A N N matrix XN , as shown in Algorithm 8.3. The bi-Lanczos iteration gives the left Lanczos basis Uj = [u1 , . . . , uj ], the right Lanczos basis Vj = [v1 , . . . , vj ] and the bidiagonal matrix ⎡ ⎢ ⎢ ⎢ Bj = ⎢ ⎢ ⎣

(b)

(b)

α1

β1

(b)

α2

0

..

.

..

.

0



⎥ ⎥ ⎥ ⎥, (b) ⎥ βj−1 ⎦ (b) αj

j = 1, . . . , m.

(8.6.14)

Algorithm 8.4 shows the bi-Lanczos-iteration-based fast subspace decomposition algorithm developed in [516]. Regarding the RR approximation, Xu and Kailath [518] showed that the d largest RR values found at the mth Lanczos iteration can be used to replace the signal eigenvalues. However, it is necessary first to estimate d. To this end, construct the testing statistic : ˆ log φdˆ = N (M − d)

) * ˆ −1 A ˆ 2 − dˆ θ(m)2 (M − d) F k=1 k ) * ,  ˆ ˆ −1 tr( A) ˆ − d θ(m) (M − d) k=1 k

(8.6.15)

where ˆ = tr( A)

M  k=1

ˆ , λ k

ˆ 2 =

A F

M 

ˆ2 . λ k

(8.6.16)

k=1

For applications of the fast subspace decomposition in signal processing and wireless communications, see [516].

546

Subspace Analysis and Tracking Bi-Lanczos iteration [179]

Algorithm 8.3

input: Data matrix XN . initialize: p0 = f (unit-norm vector); β0 = 1; u0 = 0; j = 0. (b)

while (βj

= 0) do (b)

1. vj+1 = pj /βj . 2. j = j + 1. (b)

3. rj = XN vj − βj−1 uj−1 . (b)

4. αj

= rj . (b)

5. uj = rj /αj . (b)

6. pj = XH N uj − αj v . (b)

7. βj

= pj .

end

Algorithm 8.4

Fast subspace decomposition algorithm

input: Data matrix XN . initialize: Select r0 = f (unit-norm vector). Let m = 1, β0 = r0  = 1, dˆ = 1. repeat 1. Carry out the j = mth bi-Lanczos iteration (Algorithm 8.3). 2. Use α and β estimated by Algorithm 8.3 to form an m × m bidiagonal matrix Bm (m)

and find its singular values θi , i = 1, . . . , m. 3. For dˆ = 1, . . . , m − 1, calculate the testing statistics √ φdˆ = N | log(ˆ σdˆ/ˆ σd+1 ˆ )| )# * #2 j # 1 #  1 (m) 2 # √ XN # − (θ ) . where σ ˆi = i # # √

4. If φdˆ < γdˆ

M −j

N

2

i=1

ˆ and go to step 5; otherwise let m ← m + 1, and log N , then let d = d,

return to step 1. 5. Compute the SVD of the m × m bidiagonal matrix Bm to get d principal right (m) ˆ f ). singular vectors v associated with the Krylov subspace Km (A, k

(m)

output: Signal subspace estimate Span{v1

(m)

, . . . , vd

}.

Exercises 8.1

Let V be the vector space of all the 2 × 2 matrices; show that the subspace  #  "   a b  , ad = 0, bc = 0 W = AA = c d

Exercises

8.2

is not a subspace in V . Let V be the vector space of all "    a  W = AA = 0 is a subspace in V . If   0 2 B1 = , 1 0

547

the 2 × 2 matrices, and  # 0 , a, b are any real numbers b 

B2 =

0 0

 3 , 0

 B3 =

 0 3 , 2 0

(a) show that the matrix set {B1 , B2 , B3 } is linearly dependent, and represent B3 as a linear combination of B1 and B2 ; (b) show that {B1 , B2 } is a linearly independent matrix set. 8.3

Let u1 , . . . , up be vectors in a finite-dimensional vector space V , and S = {u1 , . . . , up } be a vector set. Determine whether the following results are true or false. (a) The set of all the linear combinations of u1 , . . . , up is a vector space. (b) If {u1 , . . . , up−1 } is linearly independent, then S is also the linearly independent vector set. (c) If the vector set S is linearly independent, then S is a basis set in V . (d) If V = Span{u1 , . . . , up }, then some subset of S is a basis set in V . (e) If dim(V ) = p and V = Span{u1 , . . . , up } then the vector set S cannot be linearly dependent.

8.4

Determine whether the following results are true. (a) The row space of a matrix A is the same as the column space of AT . (b) The dimensions of the row space and the column space of the matrix A are the same, even if A is not a square matrix. (c) The sum of the dimensions of the row space and the column space of a matrix A is equal to the number of rows of A. (d) The row space of the matrix AT and the column space of A are the same.

8.5

8.6

Let A be an m × n matrix. There are several different subspaces among the subspaces Row(A), Col(A), Null(A), Row(AT ), Col(AT ) and Null(AT ). Which are located in Rm and Rn , respectively? Let A ∈ Cm×n , Row(A) be the row space of A and Col(A) be the column space of A. Show that: (a) the orthogonal complement (Row(A))⊥ is the null space of A, namely, (Row(A))⊥ = Null(A); (b) the orthogonal complement (Col(A))⊥ is the null space of AH , i.e., (Col(A))⊥ = Null(AH ).

8.7

For each case, show that the vector set W is a vector subspace or give a counter example to show that it is not a vector subspace:

548

Subspace Analysis and Tracking

⎫ ⎧ ⎡ ⎤ a  ⎪ ⎪ ⎪ ⎪ ⎬ ⎨ ⎢ ⎥ b ⎥ ⎢ (a) W = ⎣ ⎦ 2a + b = c, a + b + c = d ; ⎪ ⎪ c  ⎪ ⎪ ⎭ ⎩ d  ⎫ ⎧⎡ ⎤  ⎪ ⎪ ⎪ ⎪ a−b  ⎬ ⎨ ⎢ 3b ⎥ ⎥ a, b are real numbers ; (b) W = ⎢ ⎣ ⎦ ⎪ ⎪ ⎪ ⎪ 3a − 2b  ⎭ ⎩  a ⎧⎡ ⎫ ⎤ 2a + 3b  ⎪ ⎪ ⎪ ⎪ ⎨⎢ ⎬ ⎥ c + a − 2b ⎢ ⎥  (c) W = ⎣ a, b, c are real numbers .  ⎪ ⎪ 4c + a ⎦ ⎪ ⎪ ⎩ ⎭  3c − a − b 8.8

Given



8 2 A = ⎣ −3 −2 5 0

⎤ 9 −4 ⎦ , 5



⎤ 2 w = ⎣ 1 ⎦, −2

determine whether w is in the column space Col(A) or in the null space Null(A). 8.9 In statistical theory a matrix is often required to be of full rank. If the matrix A is an m × n matrix, where m > n, explain why the condition for A to be full rank is that its columns are linearly independent. 8.10 Show that v is in Col(A) of the matrix A if Av = λv and λ = 0. 8.11 Letting the column vectors of V1 and V2 be respectively the orthogonal basis in the same subspace of Cn , show that V1 V1H x = V2 V2H x, ∀ x. 8.12 Assume that V is a subspace and S is the generator or spanning set of V . Given that ⎧⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤⎫ 1 −1 1 2 ⎬ ⎨ S = ⎣ 0 ⎦ , ⎣ −2 ⎦ , ⎣1⎦ , ⎣ −1 ⎦ , ⎩ ⎭ −1 −3 2 0 find the basis of V , and compute dim(V ). 8.13 Let the matrix A ∈ Cm×n . Show that ⎛  min rank(X)≤k

⎝ A − X F =

n 

1/2 ⎞ ⎠, σi2

i=k+1

where X ∈ Cm×n , k < n, and σ1 ≥ · · · ≥ σn are the n singular values of A. 8.14 The cosine of the angle of two vectors x and y is defined as cos θ(x, y) =

|x, y ,

x 2 y 2

0 ≤ θ(x, y) ≤

π . 2

Suppose that X = [xi , . . . , xk ], Y = [y1 , . . . , yk ] ∈ Cn×k (n ≥ k) and XH X =

Exercises

549

YH Y = Ik . If the cosine function of the ith angle between the spaces Col(X) and Col(Y) is defined as cos θi (Col(X), Col(Y)) = cos θ(xi , yi ), prove that cos θi (Col(X), Col(Y)) is the ith singular value of the matrix XH Y (where the singular values are in decreasing order). 8.15 Assume that X, Y ∈ Cm×n (m ≥ n), and XH X = YH Y = In . Define the distance between the subspaces S1 = Col(X) and S2 = Col(Y) as d(S1 , S2 ) =

PS1 − PS2 F . (a) Find the projection matrices PS1 and PS2 . (b) Show that  1/2 , d(S1 , S2 ) = 1 − σmin (XH Y) where σmin (XH Y) is the minimum singular value of the matrix XH Y. 8.16 Let A and B be two m × n matrices, and m ≥ n. Show that min

QT Q=In

A − BQ 2F =

n ) 

* (σi (A))2 − 2σi (BT A) + (σi (B))2 ,

i=1

where σi (A) is the ith singular value of the matrix A. 8.17 Assume that T is a one-to-one linear transformation and that T (u) = T (v) always means u = v. Prove that if the set {T (u1 ), T (u2 ), . . . , T (up )} is linearly dependent then the vector set {u1 , u2 , . . . , up } is linearly dependent. (Note that this result shows that a one-to-one linear transformation maps a linearly independent vector set into another linearly independent vector set.) 8.18 Given the matrix ⎡ ⎤ 2 5 −8 0 17 ⎢ 1 3 −5 1 5 ⎥ ⎥, A=⎢ ⎣ −3 −11 19 −7 −1 ⎦ 1

7

−13

5

find the bases of its column space, row space and 8.19 Letting ⎡ −2 1 −1 6 ⎢ 1 −2 −4 3 A=⎢ ⎣ 7 −8 −10 −3 4 −5 −7 0 and



1 −2 −4 ⎢ 0 3 9 B=⎢ ⎣ 0 0 0 0 0 0 be two row-equivalent matrices, find

3 −12 0 0

−3

null space. ⎤ −8 −2 ⎥ ⎥ 10 ⎦ 4 ⎤ −2 12 ⎥ ⎥ 0 ⎦ 0

550

Subspace Analysis and Tracking

(a) (b) (c) (d)

the rank of the matrix A and the dimension of the null space Null(A), bases of the column space Col(A) and the row space Row(A), a basis of the null space Null(A), the number of pivot columns in the row echelon of AT .

8.20 Let A be an n × n symmetric matrix. Show that (a) (Col(A))⊥ = Null(A), ˆ + z, where x ˆ ∈ Col(A) and (b) every vector x in Rn can be written as x = x z ∈ Null(A).

9 Projection Analysis

In many engineering applications (such as wireless communication, radar, sonar, time series analysis, signal processing, and so on), the optimal solution of many problems consists of extracting a desired signal and restraining all other interferences, clutters or noises. Projection is one of the most important mathematical tools for solving this kind of problem There are two types of projection, orthogonal projection and oblique projection, as we will see. This chapter focuses upon the projection analysis and applications.

9.1 Projection and Orthogonal Projection In the process of learning mechanics, we become familiar with the decomposition of the force of gravity of an object on a slope, shown in Figure 9.1.

v w u g Figure 9.1 Decomposition of the force of gravity on an object.

In Figure 9.1, the force of gravity g on the object is vertically downward. It can be decomposed into two components: one is perpendicular to the slant and acts as the normal pressure force u on the object; the other is parallel to the slope and acts as sliding force v on the object. Thus g = u + v. Since the pressure force u and the sliding force v are perpendicular to each other, the decomposition g = u + v is an example of orthogonal decomposition. We can describe this as follows. The projection of g onto the direction of u is obtained as the arrowhead of g moves 551

552

Projection Analysis

along the direction of v. In Figure 9.1 u and v are orthogonal but this need not be the case provided that u and v still satisfy u + v = g. 9.1.1 Projection Theorem More generally, we now consider projection, and in particular the orthogonal projection, in a vector space. Let x ∈ Rn , and let S and H be any two subspaces in Rn . Consider using a linear matrix transform P to map the vector x in Rn onto a vector x1 in the subspace S. Such a linear matrix transform is called a projector onto subspace S along the “direction” of subspace H, denoted PS|H , and PS|H x is said to be a projection of x onto S along H. In particular, if the subspace H = S ⊥ = Rn \ S is the orthogonal complement of S in Rn then PS|S ⊥ x is called the orthogonal projection of x onto S and is simply denoted PS x or Px. DEFINITION 9.1 [179, p. 75] The matrix P ∈ Cn×n is known as an orthogonal projector onto a subspace S, if: (1) Range(P) = S; (2) P2 = P; (3) PH = P. Regarding the above three conditions, we have the following interpretations. • The condition Range(P) = S means that the column space of P must be equal to the subspace S. If the subspace S is the subspace spanned by the n column vectors of an m×n matrix A, i.e., S = Span(A), then Range(P) = Span(A) = Range(A). This means that if the matrix A is orthogonally projected onto the subspace S then the projection result PA must be equal to the original matrix A, namely PA = A. • The condition P2 = P means that the orthogonal projector must be an idempotent operator. • The condition PH = P shows that the orthogonal projector must have complex conjugate symmetry, i.e., Hermiticity. For a given orthogonal projector P onto a subspace S, P⊥ = I − P is called the orthogonal projector onto the orthogonal complement S ⊥ of S, because (I − P)x = x − Px ∈ Rn \ S = S ⊥ . Let P1 and P2 be two distinct orthogonal projectors onto the subspace S; then, for any vector x ∈ Rn , we have P1 x ∈ S, (I − P2 )x ∈ S ⊥



(P1 x)H (I − P2 )x = 0,

P2 x ∈ S, (I − P1 )x ∈ S ⊥



(P2 x)H (I − P1 )x = 0.

9.1 Projection and Orthogonal Projection

553

Hence

(P1 − P2 )x 22 = (P1 x − P2 x)H (P1 x − P2 x) = (P1 x)H (I − P2 )x + (P2 x)H (I − P1 )x ≡ 0,

∀ x.

That is to say, P1 = P2 , namely the orthogonal projector onto a subspace is uniquely determined. It is well known that in elementary geometry, the shortest distance from a point to a straight line is the perpendicular distance. By extension, if x ∈ V , while S is a subspace of the vector space V and x is not in the subspace S then the shortest distance problem from x to S involves finding the vector y ∈ S such that the length of the vector x − y is shortest. In other words, given a vector x ∈ V , we want to ˆ ∈ S such that find a vector x ˆ 2 ≤ x − y 2 ,

x − x

∀ y ∈ S.

(9.1.1)

THEOREM 9.1 (Projection theorem) Let V be an n-dimensional vector space and ˆ in the S be an n-dimensional subspace. If for the vector x ∈ V , there is a vector x ˆ is orthogonal to every vector y ∈ S, i.e., subspace S such that x − x ˆ , y = 0, x − x

∀ y ∈ S,

(9.1.2)

ˆ 2 ≤ x − y 2 holds for all the vectors y ∈ S, but the then the relation x − x ˆ. equality holds if and only if y = x Proof

Compute the square of the vector norm: ˆ ) + (ˆ

x − y 22 = (x − x x − y) 22 ˆ )T (x − x ˆ ) + 2(x − x ˆ )T (ˆ = (x − x x − y) + (ˆ x − y)T (ˆ x − y).

ˆ , y = 0 holds for every vector y ∈ M , this equation holds ˆ )T y = x − x Since (x − x ˆ ∈ M as well, i.e., (x − x ˆ )T x ˆ = 0. Then for the vector x ˆ )T x ˆ − (x − x ˆ )T y = 0. ˆ )T (ˆ x − y) = (x − x (x − x ˆ 22 + ˆ x −y 22  From the above two equation we immediately have x−y 22 = x− x 2 ˆ 2 , and the equality holds if and only if y = x ˆ.

x − x ˆ = PS x, i.e., x ˆ The solution to Equation (9.1.2) in Theorem 9.1 is given by x is the orthogonal projection of the vector x onto the subspace S. In this case, ˆ = x − PS x ∈ V \ S = S ⊥ , which implies that x − x ˆ is orthogonal to any x−x vector y in S, i.e., Equation (9.1.2) holds. The orthogonal projection PS x has the following properties [67]: 1. PS (αx + βy) = αPS x + βPS y, for x, y ∈ V and α, β ∈ C. 2. x 2 = PS x 2 + (I − PS )x 2 .

554

Projection Analysis

3. Every x ∈ V has the following unique representation x = PS x + (I − PS )x.

(9.1.3)

That is, x can be uniquely decomposed into the sum of the component PS x on S and the component (I − PS )x on S ⊥ . 4. PS xn → PS x if xn − x → 0. 5. x ∈ S if and only if PS x = x. 6. x ∈ S ⊥ if and only if PS x = 0.

9.1.2 Mean Square Estimation A set M ⊆ L2 is called an orthogonal system, if ξ ⊥ η for every ξ, η ∈ M (ξ = η). In particular, M is an orthonormal system if ξ = 1 for every ξ ∈ M . n Consider n data vectors η1 , . . . , ηn . Using the linear combination ξˆ = i=1 ai ηi to fit an unknown random variable ξ, the fitted (or estimated) error variable is given by = ξ − ξˆ = ξ −

n 

ai ηi .

(9.1.4)

i=1

Many engineering problems can be described as follows: find n constants a1 , . . . , an such that the mean square of the fitted error variable ⎧ 2 ⎫ n ⎨  ⎬    P = E ξ − ai ηi  (9.1.5)  ⎭ ⎩ i=1

is minimized. In parameter estimation theory [329], such an estimate is called the best linear mean square estimate of ξ. THEOREM 9.2 [329] If the data vectors η1 , . . . , ηn constitute an orthonormal system, then the best mean square estimate of the random variable ξ is determined by ξˆ =

n 

ξ, ηi ηi .

(9.1.6)

ˆ ξ = ξˆ + (ξ − ξ).

(9.1.7)

i=1

Consider the decomposition

ˆ It can be shown that the above decomposition is orthogonal, i.e., ξˆ ⊥ (ξ − ξ).

9.1 Projection and Orthogonal Projection

555

The proof is simple:  0 + n n   ˆ − ξ)} ˆ =E ξ− ξ, ηj ηj ξ, ηi ηi E{ξ(ξ j=1

+ =E ξ

n 

0

+ −E

ξ, ηj ηj

j=1

=

n  

E{ξ, ηj }

j=1

i=1 n 

ξ, ηi ηi

i=1

2



n 

0 ξ, ηj ηj

j=1

n n         E ξ, ηi  E ξ, ηj  E ηi , ηj  i=1 j=1

n n  2  2     = − E ξ, ηj  E ξ, ηi  j=1

i=1

= 0. Figure 9.2 shows, for a simple case, the orthogonal decomposition (9.1.7); see also (9.1.4).

ξ − ξˆ

ξ

a 1 η1

a 2 η2 η2

ξˆ

Figure 9.2 Orthogonal decomposition.

The projection Proj{ξ |η1 , . . . , ηn } denotes the mean square estimate of the unknown parameter vector ξ for known data vectors η1 , . . . , ηn . This estimate is ˆ |η , . . . , ηn }. sometimes written as E{ξ 1 The above projection theorem reveals a method for finding the best linear mean square estimate, but it requires all the given data vectors η1 , . . . , ηn to be orthonormal. When the known data vectors are not mutually orthogonal, we need to use the prewhitening to render the original nonorthogonal data vectors as standard orthonormal white noise vectors with zero mean and unit variance. Then, we apply the projection theorem to the whitened data vectors to find a mean square estimate. In some cases, we can easily find the orthogonal projection of the vector x onto the subspace M . THEOREM 9.3 [232] Let H be an inner product space, and x be a vector in H. If M is an n-dimensional subspace in H, and {u1 , . . . , un } is a set of orthogonal

556

Projection Analysis

ˆ ≤ x − y if and only if basis vectors of the subspace M then x − x ˆ= x

x, u2  x, un  x, u1  u + u + ··· + u . u1 , u1  1 u2 , u2  2 un , un  n

(9.1.8)

The significance of this theorem is that when M is a finite-dimensional subspace in the inner product space H, we can first find a group of orthogonal basis vectors {u1 , . . . , un } in the subspace M (for example, using Gram–Schmidt orthogonalizaˆ of the vector x onto the subspace tion) and use (9.1.8) to compute the projection x M.

9.2 Projectors and Projection Matrices The tool for projection is the projection matrix.

9.2.1 Projector and Orthogonal Projector DEFINITION 9.2 [410] Consider any vector x ∈ Cn in the direct sum decomposition of the vector space Cn = S ⊕H. If x = x1 +x2 satisfies x1 ∈ S and x2 ∈ H, and x1 and x2 are uniquely determined, then the mapping Px = x1 is the projection of the vector x as its arrowhead moves along the “direction” of the subspace H onto the subspace S, and P is called in brief the projector onto subspace S along the “direction” of subspace H, denoted PS|H . Let y = PS|H x represent the projection of x onto S along H, then y ∈ S. If we project y onto S along H, then PS|H y = y, and thus PS|H y = y



PS|H (PS|H x) = PS|H PS|H x = PS|H x



P2S|H = PS|H .

DEFINITION 9.3 A homogeneous linear operator P is called a projection operator if it is idempotent, i.e., P2 = PP = P. The unique decomposition x = x1 + x2 = Px + (I − P)x

(9.2.1)

maps any vector x in Cn onto the component x1 in the subspace S and the component x2 in the subspace H. The unique decomposition in (9.2.1) does not ensure that x1 and x2 are mutually orthogonal. However, in many practical applications it is required that the projections x1 and x2 of any vector x in complex vector space Cn onto two subspaces are mutually orthogonal. From this orthogonality requirement on x1 and x2 , x1 , x2  = (Px)H (I − P)x = 0



xH PH (I − P)x = 0, ∀ x = 0,

9.2 Projectors and Projection Matrices

557

we immediately have PH (I − P) = O



PH P = PH = P.

From this we can introduce the following definition. DEFINITION 9.4 The projector P⊥ = I − P is known as the orthogonal projector of P, if P is not only an idempotent matrix but is also Hermitian. An orthogonal projector has the following properties [382]. 1. If I − P is the orthogonal projector of P then P is also the orthogonal projector of I − P. 2. If both E1 and E2 are orthogonal projectors, and E1 and E2 have no cross-term, i.e., E1 E2 = E2 E1 = O, then E1 + E2 is an orthogonal projector. 3. If both E1 and E2 are orthogonal projectors, and E1 E2 = E2 E1 = E2 then E1 − E2 is an orthogonal projector. 4. If both E1 and E2 are orthogonal projectors and E1 E2 = E2 E1 then E1 E2 is an orthogonal projector. In the following, we discuss orthogonal projectors from the viewpoint of signal processing. As shown in Figure 9.3, a discrete time filter can be regarded as a projector, denoted P. Let x(n) = [x(1), . . . , x(n)]T = s(n) + v(n) be the input vector of the filter at the discrete time n, where s(n) is the signal vector and v(n) is the additive white noise vector. signal estimate ˆs(n)

input data P x(n) = s(n) + v(n)

Figure 9.3 Projector representation of the filter.

The filter outputs a signal estimate ˆs(n) = Px(n). For simplicity we omit the time variable in the vectors s(n) and x(n) and denote them simply as s and x. The following are the basic requirements of the filter operator P. (1) In order to ensure that the signal through the filter does not incur “distortion”, the projector P must be a linear operator. (2) If the filter output ˆs is again passed through the filter, the signal estimate ˆs should not change. This implies that PPx = Px = ˆs must be satisfied. This def condition is equivalent to P2 = PP = P, i.e., the projector P must be an idempotent operator. (3) The signal estimate is ˆs = Px, so x − Px denotes the estimation error of the filter operator P. By the orthogonality-principle lemma, when a filter is optimal, its estimation error x − Px should be orthogonal to the estimate Px

558

Projection Analysis

of the expected response, i.e., (x − Px) ⊥ Px. This implies that PT = P, i.e., the projector must have Hermiticity. Here we give a proof that (x − Px) ⊥ Px ⇒ PT = P. First, (x − Px) ⊥ Px can be equivalently written as (Px)T (x − Px) = 0



xT PT x − xT PT Px = 0



xT (PT − PT P)x = 0, ∀ x = 0,

and thus PT = PT P, whose transpose is P = PT P. Therefore, since PT = PT P and P = PT P, we have PT = P.

9.2.2 Projection Matrices The matrix form of a projector is called a projection matrix. Consider the construction of a projection matrix: it is an idempotent matrix that has only the eigenvalues 1 and 0. Let an m × m projection matrix P have r eigenvalues equal to 1, other m − r eigenvalues equal to 0. Then, the projection matrix can be, in EVD form, written as m r   P= λi u i u H ui uH (9.2.2) i = i . i=1

i=1

Consider the projection y = Px of any m × 1 vector x: y = Px =

r  i=1

ui uH i x=

r 

(xH ui )H ui .

(9.2.3)

i=1

Equation (9.2.3) reveals the two essential roles of a projection matrix: (1) After the projection of a vector x by the projection matrix P, the components of x correlated with the eigenvectors corresponding to the eigenvalue 1 of P, xH ui , i = 1, 2, . . . , r, are fully retained in the projection result Px. (2) The other components of x, correlated with the eigenvectors corresponding to the eigenvalues 0 of P, xH ui , i = r + 1, r + 2, . . . , m, are fully cancelled, and do not appear in the projection result Px. Therefore, when, for example, a signal-to-noise ratio is high enough, the signal components of the observed data vector are fully retained by the projection matrix, and the noise components are fully cancelled by the projection matrix. Let an m × n matrix A be a matrix with full column rank, i.e., rank(A) = n. Denote the column space of A by A = Col(A) = Range(A). An interesting question is, how do we construct a projection matrix PA onto the column space A? Because the rank of the matrix A is n, there are only n nonzero singular values

9.2 Projectors and Projection Matrices

σ1 , . . . , σn in the SVD of A:



A = UΣVH = [U1 , U2 ]

Σ1 O(m−n)×n

559

 VH

= U1 Σ 1 V H ,

(9.2.4)

where the diagonal matrix Σ1 = Diag(σ1 , . . . , σn ), and U1 = [u1 , . . . , un ],

U2 = [un+1 , . . . , um ]

(9.2.5)

are respectively the singular-vector matrices corresponding to the n nonzero singular values and the m − n zero singular values. In Chapter 8, we obtained two important results on column space as follows. (1) The n left singular vectors u1 , . . . , un corresponding to the nonzero singular values are the orthonormal basis vectors of the column space Col(A), i.e., Col(A) = Span{u1 , . . . , un } = Span(U1 ).

(9.2.6)

(2) The m−n left singular vectors un+1 , . . . , um corresponding to the zero singular values are the orthonormal basis vectors of the null space Null(AH ), i.e., Null(AH ) = (Col A)⊥ = Span{un+1 , . . . , um } = Span(U2 ).

(9.2.7)

Using the matrix A to make a linear transformation of the n × 1 vector x, we have the result n n   σi ui viH x = σi α i u i , (9.2.8) y = Ax = i=1

i=1

viH x

where αi = is the inner product of the eigenvector v and the data vector x. Equation (9.2.8) shows that the linear transformation result y = Ax is a linear combination of the left singular vectors u1 , . . . , un of A. The projection matrix PA of the m × n linear transformation matrix A projects all the m × 1 data vectors x onto the subspace defined by the matrix A. The projection matrix PA and the matrix A have the same eigenvectors: n of these vectors correspond to the nonzero eigenvalues of PA or A and the other m − n eigenvectors correspond to the zero eigenvalues. Because the projection matrix P has only the eigenvalues 1 and 0, n eigenvectors of P correspond to its eigenvalues 1 and the other m − n eigenvectors correspond to its eigenvalues 0. In other words, the EVD of the projection matrix has the form    On×(m−n) In UH 1 PA = [U1 , U2 ] O(m−n)×n O(m−n)×(m−n) UH 2 = U 1 UH 1 .

(9.2.9)

560

Projection Analysis

On the other hand, from (9.2.4) it follows that AA, A−1 AH = A(AH A)−1 AH H −1 = U1 Σ1 VH (VΣ1 UH VΣ1 UH 1 U1 Σ 1 V ) 1

= U1 Σ1 VH (VΣ21 VH )−1 VΣ1 UH 1 H H = U1 Σ1 VH VΣ−2 1 V VΣ1 U1

= U 1 UH 1 .

(9.2.10)

Comparing (9.2.9) with (9.2.10), one immediately has a definition for the projection matrix PA of the linear transformation matrix A: PA = AA, A−1 AH .

(9.2.11)

The above derivation of the definition formula (9.2.11) of the projection matrix PA is the method commonly used in the mathematical literature. The key of the derivation is first to guess that the projection matrix has the form of AA, A−1 AH , and then to verify this. Here we present another derivation that is based on some basic facts about the Moore–Penrose inverse matrix and the projection matrix. Postmultiply both sides of the mapping function PU = V by U† U: PUU† U = PU = VU† U, ∀ U. Hence PU = V



P = VU† = V(UH U)† UH .

(9.2.12)

Let A = Col(A) be the column space of the matrix A ∈ Cm×n , and let A⊥ = (Col(A))⊥ = Null(AH ) be the orthogonal complement of the column space A. Hence, if S ∈ A⊥ then S and A are mutually orthogonal, i.e., S ⊥ A. Therefore, we have two basic facts of the projection: PA A = A,

PA S = O, ∀ S ∈ Range(A)⊥ ,

(9.2.13)

which can be rewritten as PA [A, S] = [A, O],

(9.2.14)

where O is the zero matrix. From (9.2.12) and (9.2.14) we immediately have PA = [A, O]([A, S]H [A, S])† [A, S]H †  H   H A A A AH S = [A, O] H S A SH S SH †  H   H O A A A = [A, O] O SH S SH = A(AH A)† AH .

(9.2.15)

9.2 Projectors and Projection Matrices

561

It is easy to be verify that the projection matrix PA = A(AH A)† AH has idempotence, PA PA = PA and Hermiticity, PAH = PA . The projection matrix orthogonal to PA can be defined as follows: PA⊥ = I − PA = I − AA, A−1 AH .

(9.2.16)

9.2.3 Derivatives of Projection Matrix Let a projection matrix be the function of some vector θ:  −1 H A (θ) = A(θ)A† (θ) PA (θ) = A(θ) AH (θ)A(θ)  −1 H where A† (θ) = AH (θ)A(θ) A (θ) is the pseudo-inverse matrix of A(θ). For simplicity of representation, PA (θ) will be denoted simply as P. Consider the first-order and the second-order derivatives of the projection matrix P with respect to the elements θi of the vector θ = [θ1 , . . . , θn ]T . Definite the first-order partial derivative of the projection matrix with respect to θi as def

Pi =

∂P . ∂θi

(9.2.17)

Using the chain rule for the derivatives, one has Pi = Ai A† + AA†i ,

(9.2.18)

where def

Ai =

∂A , ∂θi

A†i =

def

∂A† ∂θi

(9.2.19)

are respectively the partial derivatives of the matrix A(θ) and of its Moore–Penrose inverse matrix A† (θ) with respect to θi . The first-order partial derivative of the pseudo-inverse matrix A† is given by ⊥ † † A†i = (AH A)−1 AH i P − A Ai A .

(9.2.20)

Combine (9.2.18) and (9.2.20): Pi = P⊥ Ai A† + (P⊥ Ai A† )H .

(9.2.21)

The second-order partial derivative of the projection matrix is given by † ⊥ † ⊥ H Pi,j = P⊥ j Ai A + P Ai,j A + P Ai Aj † ⊥ † ⊥ H H + (P⊥ j Ai A + P Ai,j A + P Ai Aj ) .

(9.2.22)

562

Projection Analysis

Noting that P⊥ j = −Pj and using (9.2.20), (9.2.22) can be expressed as [495] ⊥ H ⊥ † Pi,j = − P⊥ Aj A† Ai A† − (A† )H AH j P Ai A + P Ai,j A ⊥ ⊥ † † + P⊥ Ai (AH A)−1 AH j P − P A i A Aj A ) ⊥ H ⊥ † + − P⊥ Aj A† Ai A† − (A† )H AH j P Ai A + P Ai,j A *H ⊥ ⊥ † † + P⊥ Ai (AH A)−1 AH . (9.2.23) j P − P Ai A Aj A

The derivative formulas of the projection matrix are very useful in analyzing the statistical properties of some estimators relating to the projection matrix. Interested readers can refer to the literature [495].

9.3 Updating of Projection Matrices The adaptive filter is a filter whose coefficients can be adapted over time. This kind of adaptive adjustment depends on the use of a simple time updating formula, as complex calculations cannot meet the demands of real-time signal processing. Therefore, in order to apply the projection matrix and the orthogonal projection matrix to the adaptive filter, it is necessary to deduce time-updating formulas for these two matrices.

9.3.1 Updating Formulas for Projection Matrices Suppose that the present data space is {U}, the projection matrix corresponding to U is PU and the orthogonal projection matrix is P⊥ U . Here, U takes the form X1,m (n) or X0,m−1 (n) etc. Now we assume that a new data vector u is added to the original basis set of {U}. Generally, the new data vector u will provide some new information not included in the original basis set of {U}. Because the data subspace is enlarged from {U} to {U, u}, we should search for a “new” projection matrix PU,u , and its orthogonal projection matrix P⊥ U,u , corresponding to the new subspace. From the perspective of adaptive updating, the simplest method for finding the updated projection matrix PU,u from the known projection matrix PU is to decompose PU,u into two parts: one is the nonadaptive or known part, while the other is the adaptive updating part. There is a particularly useful way of decomposition, i.e., the nonadaptive part and the adaptive updating part are required to be mutually orthogonal. Specifically, the orthogonal decomposition of the projection matrix PU,u is PU,u = PU + Pw , where Pw = ww, w

−1

w

H

(9.3.1)

should be selected to satisfy the orthogonal condition

PU , Pw  = (vec PU )H vec Pw = 0,

(9.3.2)

9.3 Updating of Projection Matrices

563

simply denoted as Pw ⊥ PU . Because the updating is implemented through the original projection matrix PU and the new data vector u, while PU contain no component of the new data vector, the updating part Pw in the orthogonal decomposition should contain the new data vector u. Let w = Xu, i.e., Pw = XuXu, Xu−1 (Xu)T . Substituting Pw into the orthogonal condition formula (9.3.2), we obtain PU , Pw  = PU XuXu, Xu−1 (Xu)T = 0. As w = Xu = 0, the Moore–Penrose inverse matrix Xu, Xu−1 (Xu)T cannot be a zero matrix, and thus the above equation implies that PU Xu = 0, ∀ u, holds, namely PU X = O. That is to say, X should be the orthogonal projection matrix P⊥ U. Summarizing the above discussion, we obtain the vector w as follows: w = P⊥ U u.

(9.3.3)

That is, w is the orthogonal projection of the data vector u onto the column space of the data matrix U. By using the definition of the projection matrix and the symmetry of the orthogT ⊥ onal projection matrix [P⊥ U ] = PU , we easily get ⊥ ⊥ −1 T ⊥ Pw = ww, w−1 wT = P⊥ u PU . U uPU u, PU u

(9.3.4)

Substituting (9.3.4) into the orthogonal decomposition (9.3.1), we obtain an updating formula giving the “new” projection matrix: ⊥ ⊥ −1 T ⊥ PU,u = PU + P⊥ u PU . U uPU u, PU u

(9.3.5)

From P⊥ U,u = I−PU,u , we immediately have the “new” orthogonal projection matrix ⊥ ⊥ ⊥ ⊥ −1 T ⊥ P⊥ u PU . U,u = PU − PU uPU u, PU u

(9.3.6)

By postmultiplying Equations (9.3.5) and (9.3.6) by a vector y, we get the updating formulas ⊥ ⊥ −1 u, P⊥ PU,u y = PU y + P⊥ U uPU u, PU u U y,

(9.3.7)

P⊥ U,u y

(9.3.8)

=

P⊥ Uy



⊥ ⊥ −1 P⊥ u, P⊥ U uPU u, PU u U y.

Premultiplying (9.3.7) and (9.3.8) respectively by a vector zH , we obtain ⊥ ⊥ −1 z, PU,u y = z, PU y + z, P⊥ u, P⊥ U uPU u, PU u U y,

z, P⊥ U,u y

=

z, P⊥ U y



⊥ ⊥ −1 z, P⊥ u, P⊥ U uPU u, PU u U y.

We conclude with the following remarks:

(9.3.9) (9.3.10)

564

Projection Analysis

(1) Equations (9.3.5) and (9.3.6) are respectively the updating formulas for a projection matrix and its orthogonal projection matrix. (2) Equations (9.3.7) and (9.3.8) are respectively the updating formulas for the projection and the orthogonal projection of a vector. (3) Equations (9.3.9) and (9.3.10) are respectively the updating formulas for a scalar related to a projection matrix and its orthogonal projection matrix. In the following, we present an application of the above updating formulas in prediction filter updating.

9.3.2 Prediction Filters Assume that the data input and weighting tap coefficients are real numbers. For convenience, first introduce the time-shifting operator z j : z −j x(n) = [0, . . . , 0, x(1), . . . , x(n − j)]T .

(9.3.11)

Here z −j x(n) denotes a time-shifting operation rather than a multiplication. In addition, the starting point of time is assumed to be 1, i.e., x(n) = 0 for all n ≤ 0. Consider the mth-order forward prediction filter x ˆ(k) =

m 

wif (n)x(k − i),

k = 1, 2, . . . , n,

(9.3.12)

i=1

where wif (n), i = 1, . . . , m, denotes the weighting coefficient of the filter at time n. The above equation can be rewritten as a matrix equation: ⎤ ⎡ ⎤ ⎡ ⎤⎡ f w1 (n) x ˆ(1) 0 0 ··· 0 ⎥ ⎢ x(1) ⎥⎢ ˆ(2) ⎥ 0 ··· 0 w2f (n) ⎥ ⎢ ⎢x ⎥ ⎢ ⎥⎢ ⎢ = ⎢ . ⎥. (9.3.13) ⎢ ⎥⎢ . ⎥ .. .. .. .. ⎥ . ⎣ ⎣ ⎦ . . ⎦ . . . . ⎣ . ⎦ f x ˆ(n) x(n − 1) x(n − 2) · · · x(n − m) (n) wm Define a data matrix X1,m (n) = [z −1 x(n), z −2 x(n), . . . , z −m x(n)] ⎡ 0 0 ··· 0 ⎢ x(1) 0 · · · 0 ⎢ =⎢ .. .. .. .. ⎣ . . . . def

x(n − 1)

x(n − 2)

···

⎤ ⎥ ⎥ ⎥, ⎦

x(n − m)

f and define respectively an mth-order forward prediction coefficient vector wm (n) ˆ (n): and a forward predictor vector x def

(9.3.14)

def

(9.3.15)

f f wm (n) = [w1f (n), w2f (n), . . . , wm (n)]T ,

ˆ (n) = [ˆ x x(1), x ˆ(2), . . . , x ˆ(n)]T ;

9.3 Updating of Projection Matrices

565

then (9.3.13) can be rewritten in the simpler form f ˆ (n). (n) = x X1,m (n)wm

(9.3.16)

In order to find an LS estimate of the forward predictor vector, use x(n) instead of ˆ (n) in (9.3.16): x f (n) = X1,m (n), X1,m (n)−1 XT1,m (n)x(n). wm

(9.3.17)

By substituting (9.3.17) into (9.3.16), we can express the forward predictor vector as ˆ (n) = P1,m (n)x(n), x

(9.3.18)

where P1,m (n) = X1,m (n)X1,m (n), X1,m (n)−1 XT1,m (n) denotes the projection matrix of the data matrix X1,m (n). Defining the forward prediction error vector as ˆ (n), efm (n) = [efm (1), . . . , efm (n)]T = x(n) − x

(9.3.19)

where efm (k), k = 1, . . . , n, is the forward prediction error of the filter at time k, then from (9.3.18) and the definition of the orthogonal projection matrix, we have efm (n) = P⊥ 1,m (n)x(n).

(9.3.20)

The geometric interpretations of (9.3.18) and (9.3.20) are that the forward predicˆ (n) and the forward prediction error vector efm (n) are respectively the tor vector x projection and the orthogonal projection of the data vector x(n) onto the subspace spanned by the data matrix X1,m (n). Now consider the backward prediction filter x ˆ(k − m) =

m 

wib (n)x(k − m + i),

k = 1, . . . , n;

(9.3.21)

i=1

here wib (n), i = 1, . . . , m are the weighting coefficients of an mth-order backward prediction filter at the time n. The above equation can be rewritten in the matrix– vector form ⎡ ⎤ ⎡ ⎤ 0 x(1) 0 ··· 0 ⎤ ⎢ ⎥ ⎥⎡ b ⎢ .. .. .. .. .. ⎢ ⎥ ⎥ wm (n) ⎢ . . . . . ⎢ ⎥ ⎥⎢ ⎢ ⎥ ⎥ ⎥ ⎢wb (n)⎥ ⎢ ⎢ 0 x(m − 1) · · · 0 ⎥ ⎢ x(m) ⎥ ⎢ m−1 ⎥ ⎢ = ⎢ ⎥ (9.3.22) ⎥⎢ ⎢ .. ⎥ ⎢ x ⎥ ⎥⎣ ⎢x(m + 1) ˆ (1) x(m) ··· x(1) ⎦ . ⎢ ⎥ ⎥ ⎢ .. .. .. .. .. ⎢ ⎥ ⎢ ⎥ b ⎣ ⎦ ⎣ ⎦ w1 (n) . . . . . x ˆ(n − m) x(n) x(n − 1) · · · x(n − m + 1) or b ˆ (n), X0,m−1 (n)wm (n) = z −m x

(9.3.23)

566

Projection Analysis

where X0,m−1 (n) = [z 0 x(n), z −1 x(n), . . . , z −m+1 x(n)] ⎡ x(1) 0 ··· 0 ⎢ x(2) x(1) · · · 0 ⎢ =⎢ . . . .. .. .. ⎣ .. . x(n)

x(n − 1)

···

⎤ ⎥ ⎥ ⎥, ⎦

(9.3.24)

x(n − m + 1)

b b b (n) = [wm (n), wm−1 (n), . . . , w1b (n)]T , wm

ˆ (n) = [0, . . . , 0, x ˆ (n − m) = z −m x ˆ(1), . . . , x ˆ(n − m)]T . x

(9.3.25) (9.3.26)

ˆ (n) Using the known data vector x(n) instead of the unknown predictor vector x in (9.3.23), we get an LS solution for the weighting vector of the backward prediction filter: b (n) = X0,m−1 (n), X0,m−1 (n)−1 XT0,m−1 (n)x(n − m). wm

(9.3.27)

Substituting (9.3.27) into (9.3.23), the backward predictor vector can be expressed as follows: ˆ (n − m) = P0,m−1 (n)x(n − m) = P0,m−1 (n)z −m x(n), x

(9.3.28)

where P0,m−1 (n) = X0,m−1 (n)X0,m−1 (n), X0,m−1 (n)−1 XT0,m−1 (n) is the prediction matrix of X0,m−1 (n). Define the backward prediction error vector ˆ (n − m) ebm (n) = [ebm (1), . . . , ebm (n)]T = x(n − m) − x

(9.3.29)

in which ebm (k), k = 1, . . . , n, is the backward prediction error at time k. Then, by substituting (9.3.28) into (9.3.29), we can use the orthogonal prediction matrix to express the backward prediction error vector: −m ebm (n) = P⊥ x(n). 0,m−1 (n)z

(9.3.30)

The geometric interpretation of (9.3.28) and (9.3.29) is as follows: the backward ˆ (n − m) and the backward prediction error vector ebm (n) are predictor vector x respectively the projection and the orthogonal projection of the shifted data vector z −m x(n) onto the subspace spanned by the data matrix X0,m−1 (n). Define an n × 1 unit vector π = [0, . . . , 0, 1]T ,

(9.3.31)

called an extracting vector since the last entry of u = [u1 , . . . , un ]T is extracted by π T u. Then, the current forward and backward prediction errors can be expressed

9.3 Updating of Projection Matrices

567

using the inner products of the prediction error vectors and the extracting vector π: efm (n) = π, efm (n) = π, P⊥ 1,m (n)x(n), ebm (n)

=

π, ebm (n)

=

−m π, P⊥ x(n). 0,m−1 (n)z

(9.3.32) (9.3.33)

The forward and backward prediction residuals at time n are defined respectively as fm (n) = efm (n), efm (n),

(9.3.34)

ebm (n), ebm (n).

(9.3.35)

bm (n)

=

Using the shifting property of z −1 , we obtain the forward and backward prediction residuals at the time n − 1: fm (n − 1) = z −1 efm (n), z −1 efm (n),

(9.3.36)

z −1 ebm (n), z −1 ebm (n).

(9.3.37)

bm (n

− 1) =

9.3.3 Updating of Lattice Adaptive Filter A lattice adaptive filter is a least squares type of filter consisting of coupled forward and backward prediction filters. The forward and backward prediction error equations of a lattice filter are given by b efm+1 (n) = efm (n) − Km+1 (n)ebm (n − 1),

(9.3.38)

f −Km+1 (n)efm (n)

(9.3.39)

ebm+1 (n)

=

+

ebm (n

− 1),

f b (n) and Km+1 (n) are respectively the forward- and backward-reflecting where Km+1 coefficients of the (m + 1)th-order lattice filter. The lattice adaptive filter design is to derive the recursive formulas of the forwardand backward-reflecting coefficients. Owing to the coupling of the forward and backward predictions, the updating design of a lattice adaptive filter is complicated. Fortunately, the updating formulas for the projection matrix and the orthogonal projection matrix provide an effective method for designing a lattice adaptive filter [12]. Before deriving the recursive formulae for the lattice filter, we first prove an important relationship: −m −m−1 z −1 P⊥ = P⊥ . 0,m−1 (n)z 1,m (n)z

Proof

(9.3.40)

Using the shifting property of z −1 , it is easy to see that for any vector y(n), −m −m z −1 P⊥ y(n) = P⊥ y(n − 1) 0,m−1 (n)z 0,m−1 (n − 1)z −m−1 = P⊥ y(n). 0,m−1 (n − 1)z

(9.3.41)

568

Projection Analysis

Noting that X0,m−1 (n − 1) = z −1 [z 0 x(n), z −1 x(n), . . . , z −m+1 x(n)] = [z −1 x(n), z −2 x(n), . . . , z −m x(n)] = X1,m (n), we have P⊥ 0,m−1 (n − 1) = I − X0,m−1 (n − 1)X0,m−1 (n − 1), X0,m−1 (n − 1)−1 XT0,m−1 (n − 1) = I − X1,m (n)X1,m (n), X1,m (n)−1 XT1,m (n) = P⊥ 1,m (n). Then, Equation (9.3.41) can be written as −m −m−1 y(n) = P⊥ y(n). z −1 P⊥ 0,m−1 (n)z 1,m (n)z

As the above equation holds for any nonzero vector y(n), Equation (9.3.40) is true. For the recursion of the forward prediction error efm (n) = π, P⊥ 1,m (n)x(n) in (9.3.32), from (9.3.40) it follows that −m−1 −m x(n) = z −1 P⊥ x(n) P⊥ 1,m (n)z 0,m−1 z

= z −1 ebm (n) = ebm (n − 1).

(9.3.42)

Letting z = π, U = X1,m (n), u = z −m−1 x(n) and y = x(n) in (9.3.10), we have {U, u} = {X1,m (n), z −m−1 x(n)} = {X1,m+1 (n)},

(9.3.43)

and hence ⎫ ⎪ ⎪ ⎪ ⎪ ⊥ ⊥ ⎪ PU,u = P1,m+1 (n), ⎪ ⎪ ⎪ ⎬ ⊥ ⊥ −m−1 b PU u = P1,m (n)z x(n) = em (n − 1), ⎪ ⎪ ⎪ ⊥ ⎪ P⊥ y = P (n)x(n), ⎪ U 1,m ⎪ ⎪ ⎪ ⎭ ⊥ ⊥ PU,u y = P1,m+1 (n)x(n). ⊥ P⊥ U = P1,m (n),

(9.3.44)

9.3 Updating of Projection Matrices

569

Substituting (9.3.44) into (9.3.10) and using (9.3.42), we get efm+1 (n) = π, P⊥ 1,m+1 (n)x(n) −m−1 x(n) π, ebm (n − 1)P⊥ m (n)x(n), z b b em (n − 1), em (n − 1) ⊥ −m−1 x(n) π, ebm (n − 1)P⊥ m (n)x(n), Pm (n)z = efm (n) − ebm (n − 1), ebm (n − 1) b π, ebm (n − 1)P⊥ m (n)x(n), em (n − 1) = efm (n) − ebm (n − 1), ebm (n − 1) f e (n), z −1 ebm (n) b = efm (n) − m b (9.3.45) em (n − 1). m (n − 1)

= π, P⊥ 1,m (n)x(n) −

By comparing (9.3.45) with (9.3.39), the backward reflecting coefficient is given by b Km+1 (n) =

efm (n), z −1 ebm (n) Δm+1 (n) = b , bm (n − 1) m (n − 1)

(9.3.46)

where Δm+1 (n) = efm (n), z −1 ebm (n)

(9.3.47)

is called the partial correlation coefficient. Mimicking the above derivation, the forward-reflecting coefficient is found as f (n) = Km+1

Δm+1 (n) . fm (n)

(9.3.48)

Summarizing the above discussions, we have the following conclusion: the design of a lattice adaptive filter amounts to deriving recursive formulas for the partial correlation coefficient Δm+1 (n), the forward prediction residual fm (n), the backf ward prediction residual bm (n), the forward reflecting coefficient Km+1 (n) and the b backward reflecting coefficient Km+1 (n). In order to derive the recursive formula for the forward prediction residual fm (n), substitute efm (n) = P⊥ 1,m (n)x(n) into (9.3.34): ⊥ fm (n) = P⊥ 1,m (n)x(n), P1,m (n)x(n) T ⊥ = xT (n)[P⊥ 1,m (n)] P1,m (n)x(n)

= xT (n)P⊥ 1,m (n)x(n) = x(n), P⊥ 1,m (n)x(n).

(9.3.49)

Similarly, from (9.3.37) it can be shown that bm (n) = z −m x(n), P⊥ 0,m−1 (n)x(n − m).

(9.3.50)

Letting z = y = x, U = X1,m (n) and u = z −m−1 x(n) in (9.3.10), from (9.3.49)

570

Projection Analysis

it follows that fm+1 (n) = fm (n) −

Δ2m+1 (n) . bm (n − 1)

(9.3.51)

Similarly, letting z = y = z −m−1 x, U = X1,m (n) and u = x(n) in (9.3.10), then from (9.3.50) we have bm+1 (n) = fm (n − 1) −

Δ2m+1 (n) . fm (n)

(9.3.52)

By [12], there is the following relationship between the projection matrix PU,π (n) of the subspace {U, π} and the projection matrix PU (n − 1) of the subspace {U}:   PU (n − 1) 0 , (9.3.53) PU,π (n) = O 1 Likewise, we have P⊥ U,π (n) =

 ⊥ PU (n − 1) O

 0 . 0

(9.3.54)

Letting z = x(n), y = z −m−1 x(n), U = X1,m (n) and u = π, and using (9.3.54), the partial correlation coefficient can be recursively computed from Δm+1 (n) = Δm+1 (n − 1) +

fm (n) bm (n − 1) , γm (n − 1)

(9.3.55)

where γm (n − 1) = π, P⊥ 1,m x(n)π

(9.3.56)

is known as the angular parameter. Letting z = y = π, U + X1,m (n) and u = z −m−1 x(n) in (9.3.10), (9.3.56) then yields a recursive formula for the angular parameter: 2  b em (n − 1) γm+1 (n − 1) = γm (n − 1) − b . (9.3.57) m (n − 1) 9.4 Oblique Projector of Full Column Rank Matrix As described in the previous sections, the orthogonal projection of a vector z onto a subspace H can be decomposed into the sum of its orthogonal projections onto the subspaces H1 and H2 , where H2 is required to be the orthogonal complement of H1 , and the orthogonal projector must be idempotent and Hermitian. If the subspace H2 is not the orthogonal complement of H1 , then the idempotent operator is no longer Hermitian. An idempotent operator without Hermiticity is an oblique projector. The oblique projector was first proposed by Murray [335] and Lorch [303] in the 1930s. Later, Afriat [11], Lyantse [308], Rao and Mitra [410], Halmos [195], Kato [240] and Takeuchi et al. [461] made further studies. In 1989, Kayalar and Weinert

9.4 Oblique Projector of Full Column Rank Matrix

571

[242] presented an application of the oblique projector to array signal processing and derived new formulas and an iterative algorithm for computing the oblique projector. In 1994, Behrens and Scharf [31] derived more practical computation formulas for the oblique projector onto the column space of a matrix. In 2000, Vandaele and Moonen [487] presented formulas for oblique projection formulas onto the row space of a matrix using the LQ decomposition of a matrix. Owing to the wide applications of the oblique projector, in this section and the next we discuss the theory, methods and applications of oblique projectors of matrices with full column rank and full row rank, respectively.

9.4.1 Definition and Properties of Oblique Projectors Let V be a Hilbert space and H be a closed subspace in V spanned by the observation vectors. Assume the data are divided into two subsets which span respectively the closed subspaces H1 and H2 such that H = H1 + H2 , where the intersection of H1 and H2 satisfies H1 ∩ H2 = {0}, i.e., the two subspaces H1 and H2 are nonoverlapping or disjoint. Note the fact that the two subspaces are disjoint shows only they have no common nonzero element, and does not implies their orthogonality. Let P, P1 , P2 respectively be orthogonal projectors onto the subspaces H, H1 , H2 . Given a vector z ∈ V , we hope compute the orthogonal projection of z onto the subspace H, denoted Pz. Aronszajn [17] presented a method for computing the projection Pz: first find the orthogonal projections of z onto the subspaces H1 and H2 to get P1 z and P2 z, respectively. Then use the synthesizing formula Pz = (I − P2 )(I − P1 P2 )−1 P1 z + (I − P1 )(I − P2 P1 )−1 P2 z

(9.4.1)

to yield Pz. The above Aronszajn synthetic formula can be directly used to solve some statistical interpolation problems [9], [380], [423]. And, as pointed in [242], the dual filter smoothing formula is simply a special case of the Aronszajn synthetic formula. It should be pointed out that when the subspace H2 is the orthogonal complement ⊥ of H1 , the orthogonal projectors are P2 = P⊥ 1 and P1 = P2 . In this case, the Aronszajn synthetic formula (9.4.1) reduces to ⊥ Pz = P1 z + P⊥ 1 z = P2 z + P2 z,

(9.4.2)

i.e., a typical orthogonal decomposition. Hence, the Aronszajn synthetic formula is a generalization of the orthogonal decomposition. The Aronsajn synthetic method applies two orthogonal projectors P1 and P2 onto two nonorthogonal subspaces to compute the orthogonal projection Pz. This method has the following two disadvantages: (1) It needs a large amount of calculation.

572

Projection Analysis

(2) It cannot easily be generalized to cases where the subspace H is decomposed into three or more subspaces. Two inherent disadvantages of the Aronszajn synthetic method can be avoided by the use of nonorthogonal projection. This nonorthogonal projection is known as oblique projection, to be discussed below. In order to make the calculation of the orthogonal projection Pz as simple as possible, the easiest way is to let Pz be the direct sum of the projections of the vector z onto the subspaces H1 and H2 , i.e., Pz = E1 z + E2 z,

(9.4.3)

where E1 and E2 are projectors. Although the above equation still synthesizes the orthogonal projection Pz, it is different from the Aronszajn synthetic formula (9.4.2) in that the two projection matrices in (9.4.3) are no longer orthogonal: E1 = E⊥ 2 , i.e., the two subspaces H1 and H2 are not mutually orthogonal. In this case, we say that H = H 1 ⊕ H2

(9.4.4)

is the direct sum decomposition of the subspace H. Note that the symbols P and E are used to represent respectively an orthogonal projector and an nonorthogonal projector. A nonorthogonal projector is known as an oblique projector. It should be noted that both orthogonal projectors and oblique projectors must satisfy the idempotence of any projector. It is easy to verify that (I − P2 )(I − P1 P2 )−1 P1 and (I−P1 )(I−P2 P1 )−1 P2 in the Aronszajn synthetic formula (9.4.1) are not orthogonal or oblique projectors, because they are not idempotent operators. The question is, how do we construct the two oblique projectors E1 and E2 in (9.4.3)? To this end, let us consider a full column rank matrix. Let the n × m matrix H be a full column rank matrix whose range (space) is    Range(H) = y ∈ Cm y = Hx, x ∈ Cn ,

(9.4.5)

and whose null space is    Null(H) = x ∈ Cn Hx = 0 .

(9.4.6)

Now let the n × m full column rank matrix H and an n × k full row rank matrix be combined into an n × (m + k) matrix [H, S], where m + k < n such that the column rank of the matrix [H, S] is less than the number of its rows and the column vectors of H and the column vectors of S are linearly independent. Owing to this linear independence, the two ranges Range(H) and Range(S) are disjoint. According to the definition of the projection matrix, the orthogonal projector

9.4 Oblique Projector of Full Column Rank Matrix

573

onto the range space Range(H, S) is given by −1

PHS = [H, S] [H, S], [H, S] [H, S]H −1  H   H H H H HH S . = [H, S] H S H SH S SH

(9.4.7)

Equation (9.4.7) shows that the orthogonal projector PHS can be decomposed as follows [31]: PHS = EH|S + ES|H ,

(9.4.8)

where  EH|S = [H, O]  ES|H = [O, S]

HH H SH H

HH H SH H

−1  H  H HH S , SH S SH −1  H  HH S H , SH S SH

(9.4.9) (9.4.10)

with zero matrix O. From the the inversion lemma for a block matrix (1.7.11), it is easy to find that  H −1 H H HH S = SH H SH S   (HH PS⊥ H)−1 −(HH PS⊥ H)−1 HH S(SH S)−1 , (9.4.11) −1 H −1 −(SH P⊥ S H(HH H)−1 (SH P⊥ H S) H S) where PS⊥ = I − PS = I − S(SH S)−1 SH , H −1 H P⊥ H . H = I − PH = I − H(H H)

By substituting (9.4.11) into (9.4.9), we have EH|S = [H, O]



(HH PS⊥ H)−1

−(HH PS⊥ H)−1 HH S(SH S)−1

−1 H −(SH P⊥ S H(HH H)−1 H S)

−1 (SH P⊥ H S)



HH SH



namely EH|S = H(HH PS⊥ H)−1 HH − H(HH PS⊥ H)−1 HH PS or EH|S = H(HH PS⊥ H)−1 HH PS⊥ .

(9.4.12)

Similarly, substitute (9.4.11) into (9.4.10) to yield −1 H ⊥ ES|H = S(SH P⊥ S PH . H S)

(9.4.13)

574

Projection Analysis

Equations (9.4.12) and (9.4.13) were obtained by Behrens and Scharf in 1994 [31]. From (9.4.12) and (9.4.13) we see that the idempotent operators EH|S and ES|H are not Hermitian, so they are oblique projectors rather than orthogonal projectors. DEFINITION 9.5 An idempotent operator E without complex conjugate symmetry is called an oblique projector. By definition, both the operator EH|S defined in (9.4.12) and the operator ES|H defined in (9.4.13) are oblique projectors. The oblique projector EH|S reads as “the projector onto the subspace Range(H) ‘along’ the subspace Range(S)”. Similarly, the oblique projector ES|H reads as “the projector onto the subspace Range(S) ‘along’ the subspace Range(H)”. The geometric interpretation of an oblique projector is as follows: for an oblique projector EH|S , its projection direction is along to the subspace Range(S). Thus the subspace of the projector EH|S and the subspace Range(S) cannot have any intersection. Because EH|S H is the projection of H onto the subspace Range(H) along the subspace Range(S), while H itself lies in the subspace Range(H), the result of the oblique projection is EH|S H = H. The oblique projector is an extension of the orthogonal projector, while the orthogonal projector is a special case of the oblique projector. The following summarizes the important properties of oblique projectors [31]. 1. Both EH|S and ES|H are idempotent operators, i.e., E2H|S = EH|S ,

E2S|H = ES|H .

2. EH|S [H, S] = [H, O] and ES|H [H, S] = [O, S], or equivalently EH|S H = H,

EH|S S = O,

ES|H H = O,

ES|H S = S.

3. The cross-terms of the oblique projectors EH|S and ES|H are equal to zero, i.e., EH|S ES|H = O,

ES|H EH|S = O.

4. Oblique projection followed by orthogonal projection does not change the original oblique projection: EH|S = PH EH|S ,

ES|H = PS ES|H .

5. Let B† = (BH B)−1 BH denote the generalized inverse matrix of B; then H† EH|S = (PS⊥ H)† ,

† S† ES|H = (P⊥ H S) .

6. There exists the following relationship between an oblique projection matrix and the corresponding orthogonal projection matrix: PS⊥ EH|S PS⊥ = PS⊥ PP⊥ H PS⊥ = PP⊥ H , S

S

9.4 Oblique Projector of Full Column Rank Matrix

575

where PPS⊥ H = PS⊥ HPS⊥ H, PS⊥ H−1 (PS⊥ H)H . 7. If the subspaces Range(H) and Range(S) are orthogonal, i.e., Range(H) ⊥ Range(S), then EH|S = PH ,

ES|H = P⊥ H.

On the above properties, we have the following four remarks: Remark 1

Property 2 shows that:

(1) the range of EH|S is Range(H) and the null space of EH|S contains Range(S), i.e., Range(EH|S ) = Range(H),

Range(S) ⊂ Null(EH|S );

(9.4.14)

(2) the range of ES|H is Range(S) and the null space of ES|H contains Range(H), i.e., Range(ES|H ) = Range(S),

Range(H) ⊂ Null(ES|H ).

(9.4.15)

In other words, the range of the oblique projector EH|S is in the null space of ES|H , whereas the range of ES|H is in the null space of EH|S . Remark 2 Since the cross-terms of the oblique projectors EH|S and ES|H are zero, Range(EH|S ) and Range(ES|H ) are disjoint. Remark 3 Property 4 shows that EH|S is the oblique projection onto the subspace Range(H) along the subspace Range(S), i.e., Range(EH|S ) = Range(H). Therefore, if the oblique projection EH|S is again projected onto the subspace Range(H) then the projection result will not change. Remark 4 Properties 5 and 6 can be regarded as two relationships between the oblique projector and the orthogonal projector.

9.4.2 Geometric Interpretation of Oblique Projectors From the above discussions, we can summarize three projectors below. (1) The orthogonal projector PHS is a projector onto the subspace Range(H, S) of the composite matrix [H, S]. (2) The oblique projector EH|S is a projector onto the subspace Range(H) along another range Range(S). (3) The oblique projector ES|H is a projector onto the subspace Range(S) along the range Range(H). As explained, the range of the oblique projector EH|S is Range(H), and its null space contains Range(S). In order to describe the null space of the oblique projector

576

Projection Analysis

EH|S , define the matrix A as the matrix orthogonal to the composite matrix [H, S], i.e., [H, S]H A = O and thus HH A = O,

SH A = O.

(9.4.16)

Premultiplying the second equation above by the full column rank matrix S(SH S)−1, SH A = O can be equivalently written as PS A = O



PS⊥ A = A.

(9.4.17)

Therefore, from (9.4.12) we have EH|S A = H(HH PS⊥ H)−1 HH PS⊥ A = H(HH PS⊥ H)−1 HH A = O. This shows that the subspace Range(A) spanned by the columns of A is in the null space of the oblique projector EH|S , i.e., Range(A) ⊂ Null(EH|S ). Equations (9.4.16) show that the subspace Range(A) is orthogonal to the two subspaces Range(H) and Range(S), respectively. Therefore, we can use three space directions Range(H), Range(S) and Range(A) as coordinate axes of the Euclidean space C m+k spanned by the column vectors of the n × (m + k) matrices [H, S], as shown in Figure 9.4. In Figure 9.4(b), the coordinate axis Range(A) is perpendicular to each of the other two coordinate axes, Range(H) and Range(S), but Range(H) and Range(S) are not mutually perpendicular. Nevertheless, the Euclidean space can be decomposed into the direct sum of the three directions: C m+k = Range(H) ⊕ Range(S) ⊕ Range(A).

(9.4.18)

The orthogonal projection in Figure 9.4(a) has been explained earlier. The oblique projection in Figure 9.4(b) is explained as follows: when the vector y is in the horizontal plane defined by the two coordinate axes Range(H) and Range(S), the oblique projection EH|S y of y onto Range(H) along Range(S) satisfies the following two conditions: • the oblique projection EH|S y lies in the coordinate axis Range(H), i.e., EH|S y ∈ Range(H); • the connection line from the arrowhead of the oblique projection vector EH|S y to the arrowhead of the vector y (when in the horizontal plane) is parallel to the coordinate axis Range(S), i.e., EH|S y ∈ / Range(S). As shown in Figure 9.4(b), for the vector y in the Euclidean space, the construction of the oblique projection EH|S y can be divided into the following two steps. (1) The vector y is first projected orthogonally onto the plane defined by the coordinate axes Range(H) and Range(S), i.e., the projection PHS y is the orthogonal projection of the vector y onto the range Range(H, S) spanned by the column vectors of the matrix [H, S], denoted yHS = PHS y.

9.4 Oblique Projector of Full Column Rank Matrix

577

(a) Geometric interpretation of orthogonal projection

(b) Geometric interpretation of oblique projections Figure 9.4 Orthogonal projection and oblique projections [383].

(2) The method illustrated in Figure 9.4(b) is used to find the oblique projection of the orthogonal projection of yHS onto Range(H) along Range(S), denoted EH|S yHS . The physical meaning of the oblique projection operator is obtained as follows: the sentence “the oblique projection EH|S y of a vector y is its projection onto Range(H) ‘along’ Range(S)” is translated into signal processing language as “the oblique projection EH|S y extracts the components of the vector y in the special direction Range(H) and fully cancels all the components of y in another direction, Range(S)”. From Figure 9.4 it can be deduced that the projection PA y is the orthogonal projection of the vector y onto the range A = Range(A), where the projection PA y is not only orthogonal to EH|S y but also is orthogonal to ES|H y, i.e., PA y ⊥ EH|S y and PA y ⊥ ES|H y. In mathematics, oblique projections are a type of parallel projection. The parallel projection of a three-dimensional point (x, y, z) onto the plane (x, y) is given by

578

Projection Analysis

(x + az, y + bz, 0). The constants a and b determine uniquely a parallel projection. If a = b = 0, then the parallel projection is called an “orthographic” or orthogonal projection. Otherwise, it is an oblique projection. In image processing, oblique projection is a technique providing a two-dimensional picture or image of a three-dimensional object.

9.4.3 Recursion of Oblique Projectors Let H be a known data matrix, H = Range(H) be the range (i.e., column space) of H and EH|S x be the oblique projection of the vector x onto H along a disjoint subspace S. If a new data matrix V is added, the question is how to compute recursively the new oblique projection EH|S ˜ x using the known oblique projection ˜ and H ˜ = [H, V]. ˜ = Range(H), EH|S x and the new data matrix V, where H ˜ = [H, V], H ˜ and H ˜ = Range(H) ˜ and S = Range(S) THEOREM 9.4 [383] If H are disjoint then the new oblique projection matrices are given by EH|S = EH|S + EV˜ |S , ˜

(9.4.19)

ES|H˜ = ES|H − PS EV˜ |S ,

(9.4.20)

˜ V ˜ = V − EH|S V and PH is the projection matrix of the where V˜ = Range(V), matrix H. The following three remarks on Theorem 9.4 help us understand the role of oblique projection matrices and the relationship between oblique projection recursion and orthogonal projection recursion. Remark 1

˜ and S are orthogonal then If two subspaces H EH|S = PH˜ , ˜

EH|S = PH ,

ES|H˜ = P⊥ ˜, H

ES|H = P⊥ H.

˜ = (I − PH )V = P⊥ V, and hence In this case, V H EV˜ = PV˜ ,

PS PV˜ = PV˜ .

Substitute the above relations into (9.4.19) and (9.4.20) to yield ⊥ ⊥ −1 T ⊥ PH˜ = PH + P⊥ V PH , H VPH , PH V ⊥ ⊥ ⊥ ⊥ −1 T ⊥ P⊥ V PH . ˜ = PH − PH VPH , PH V H

These are just the recursive formulas for the projection matrix and the orthogonal projection matrix, respectively. Therefore Theorem 9.4 is a generalization of the definitions of projection matrix and the orthogonal projection matrix to the case ˜ and S. of nonorthogonal subspaces H

9.5 Oblique Projector of Full Row Rank Matrices

579

Remark 2 Theorem 9.4 can be explained from the angle of innovation processes. By the Kalman filtering theorem, in the case of an orthogonal projection, PH V ˜ = V − PH V represents the mean square estimate of the data matrix V and V ˜ can be regarded as the innovation matrix of the data matrix V, while Range(V) denotes the innovation subspace in orthogonal projection. Similarly, EH|S V is the mean square estimate of the data matrix V in the subspace H along the disjoint ˜ = V − EH|S V can be regarded as the innovation matrix of the subspace S, and V ˜ denotes the innovation subspace in data matrix V in H along S, while Range(V) oblique projection. Remark 3 From the viewpoint of subspaces, the vector space C n equals H ⊕ S ⊕ (H ⊕ S)⊥ , where H, S and (H ⊕ S)⊥ represent respectively the desired signal (the range), the structured noise (the interference) and the unstructured noise subspace. By Theorem 9.4 and PHS = EH|S + ES|H , it is easy to see that ˜ = (I − EH|S )V = ES|H V + P⊥ V = VS + V(HS)⊥ . V HS ˜ of the data matrix V in H along S consists of That is, the innovation matrix V two components: VS = ES|H V is the innovation matrix of the structured noise in the subspace S and V(H,S)⊥ V is the innovation matrix of the unstructured noise in the orthogonal complement subspace (H ⊕ S)⊥ . If the structured noise is negligible ˜ = V(HS)⊥ . In compared with the unstructured noise then the innovation matrix V ⊥˜ ⊥ this case, PS V ≈ PS V(HS)⊥ = V(HS)⊥ and ˜ V ˜ H P⊥ V) ˜ −1 V ˜ H P⊥ = V(HS)⊥ (VH ⊥ V(HS)⊥ )−1 VH ⊥ EV˜ |S = V( S S (HS) (HS) which is just the projection matrix of V(HS)⊥ .

9.5 Oblique Projector of Full Row Rank Matrices In the previous sections, our focus was the action of orthogonal projectors and the oblique projectors upon full column rank matrices. If the matrix is of full row rank, then the corresponding orthogonal projector and oblique projector have different definitions and forms of expression.

9.5.1 Definition and Properties If D ∈ Cm×k has the full row rank, i.e., rank(D) = m (m < k) then its projection PD is a k × k matrix and is defined as PD = DH (DDH )−1 D.

(9.5.1)

Let the matrices B ∈ Cm×k and C ∈ Cn×k be two full row

rank matrices, i.e., B rank(B) = m, rank(C) = n, and let their combined matrix C ∈ C(m+n)×k be also of full row rank, i.e., m + n, where m + n < k. This implies that the row vectors of

580

Projection Analysis

the matrix B and the row vectors of C are linearly independent. Hence the range subspace spanned by row vectors of the matrix B, denoted ZB and simply called the row subspace of

B, and the row subspace ZC of the matrix C are disjoint. B Putting D = C and substituting D into (9.5.1), we find the orthogonal projector 

PD

BBH = [B , C ] CBH H

BCH CCH

H

−1   B , C

(9.5.2)

which can be decomposed as follows: PD = EZB |ZC + EZC |ZB , where



EZB |ZC EZC |ZB

BBH = [B , C ] CBH  BBH H H = [B , C ] CBH H

H

(9.5.3)

−1   B , O −1   BCH O . C CCH BCH CCH

(9.5.4) (9.5.5)

  B

Premultiplying the above two equations by the matrix C , then     B B EZB |ZC = , C O     B O E = . C ZC |ZB C

(9.5.6) (9.5.7)

These results can be equivalently written as BEZB |ZC = B,

CEZB |ZC = O,

(9.5.8)

BEZC |ZB = O,

CEZC |ZB = C.

(9.5.9)

This shows that the operators EZB |ZC and EZC |ZB have no cross-term, i.e., EZB |ZC EZC |ZB = O

and

EZC |ZB EZB |ZC = O.

(9.5.10)

By the idempotence of the orthogonal projector PD , we have P2D = (EZB |ZC + EZC |ZB )(EZB |ZC + EZC |ZB ) = E2ZB |ZC + E2ZC |ZB = PD = EZB |ZC + EZC |ZB which yields E2ZB |ZC = EZB |ZC ,

E2ZC |ZB = EZC |ZB .

(9.5.11)

As they are idempotence, EZB |ZC , defined in (9.5.4) and EZC |ZB , defined in (9.5.5), are regarded as oblique projectors of the full row rank matrices B and C, respectively.

9.5 Oblique Projector of Full Row Rank Matrices

581

In particular, if the row subspace of B and the row subspace of C are orthogonal, i.e., BCH = O and CBH = O, then the oblique projectors are reduced to orthogonal projectors [487]: EZB |ZC = BH (BBH )−1 B = PB , H

H −1

EZC |ZB = C (CC )

C = PC .

(9.5.12) (9.5.13)

Proof Substituting the orthogonality condition BCH = O and CBH = O into (9.5.4), then −1    B BBH O H H EZB |ZC = [B , C ] H O O CC    H −1 O B (BB ) = [BH , CH ] O (CCH )−1 O = BH (BBH )−1 B = PB . Similarly, we can show that EZC |ZB = CH (CCH )−1 C = PC . 9.5.2 Calculation of Oblique Projection The QR decomposition of an m × n (m > n) real matrix A is defined as   R T , (9.5.14) Q A= O where Q is an m × m orthogonal matrix and R is an upper triangular matrix. If B is a real matrix with the number of columns greater than the number of rows then, letting B = AT and taking the transpose of the QR decomposition formula (9.5.14), we obtain the LQ decomposition of B: BQ = AT Q = [L, O]

(9.5.15)

B = [L, O]QT ,

(9.5.16)

or

where L = RT is a lower triangular matrix. Putting B ∈ Rm×k , C ∈ Rn×k and A ∈ Rp×k , where m + n + p < k, the LQ  T is given by decomposition of the matrix BT , CT , AT ⎤ ⎡ ⎤ ⎡ B L11 ⎦, ⎣C⎦ [Q1 , Q2 , Q3 ] = ⎣L21 L22 (9.5.17) L31 L32 L33 A where Q1 ∈ Rk×m , Q2 ∈ Rk×n , Q3 ∈ Rk×p are orthogonal matrices, i.e., QTi Qi = I and QTi Qj = O, i = j, and Lij is a lower triangular matrix.

582

Projection Analysis

Equation (9.5.17) can be equivalently written as ⎡ ⎤ ⎡ L11 B ⎣C⎦ = ⎣L21 A L31

L22 L32

⎤ ⎡ QT ⎤ 1 ⎥ ⎦⎢ ⎣QT2 ⎦ . L33

(9.5.18)

QT3

By the oblique projection definition (9.5.4), and using QTi Qi = I and QTi Qj = O, i = j, then       BBT BCT −1 B EZB |ZC = BT , CT O CBT CCT T  L = [Q1 , Q2 ] 11 L21 L22    T T −1   Q1 Q1 QT1 Q2 L11 B L11 × T T O L21 L22 Q2 Q1 Q2 Q2 L21 L22 −1    B L = [Q1 , Q2 ] 11 . (9.5.19) L21 L22 O Similarly, one has EZC |ZB = [Q1 , Q2 ]

 L11 L21

−1   O . C L22

(9.5.20)

Note that the LQ decomposition of A is ⎡

QT1



⎢ ⎥ A = [L31 , L32 , L33 ] ⎣QT2 ⎦ . QT3 From (9.5.19) one can find the oblique projection of the p × k matrix A along the direction parallel to the row space of the n × k matrix C onto the row space of the m × k matrix B as follows: ⎡ T⎤ Q1 −1    B L11 ⎢ T⎥ AEZB |ZC = [L31 , L32 , L33 ] ⎣Q2 ⎦ [Q1 , Q2 ] O L21 L22 QT3 −1    B L11 . = [L31 , L32 ] O L21 L22 If we let

−1



L [L31 , L32 ] 11 L21

L22

= [LB , LC ],

(9.5.21)

9.5 Oblique Projector of Full Row Rank Matrices

then [487] AEZB |ZC

  B = [LB , LC ] = LB B. O

583

(9.5.22)

Similarly, the oblique projection of the p×k matrix A, along the direction parallel to the row space of the m × k matrix B, onto the row space of the n × k matrix C is given by   O = LC C. AEZC |ZB = [LB , LC ] (9.5.23) C From (9.5.21) one has





L [L31 , L32 ] = [LB , LC ] 11 L21

L22

,

namely LC = L32 L−1 22 ,

(9.5.24)

−1 −1 LB = (L31 − LC L21 )L−1 11 = (L31 − L32 L22 L21 )L11 .

(9.5.25)

Substituting the above two equations into (9.5.22) and (9.5.23), we have −1 T AEZB |ZC = (L31 − L32 L−1 22 L21 )L11 L11 Q1

AEZC |ZB

T = (L31 − L32 L−1 22 L21 )Q1 ,   QT1 = L32 L−1 22 [L21 , L22 ] QT2 T T = L32 L−1 22 L21 Q1 + L32 Q2 .

(9.5.26)

(9.5.27)

9.5.3 Applications of Oblique Projectors Oblique projectors have been used in generalized image restoration [534], solving large unsymmetric systems [421], fast system identification [424], multivariate analysis [461], partial correlation (PARCOR) estimation [249], impulse noise cancellation [512], error correction coding [315], burst error correction decoding [270], model reduction [227], system modeling [31], blind channel estimation [487], joint channel and symbol estimation [535] and so on. In actual situations of system identification, parameter estimation, signal detection and so on, in addition to the signal of interest (referred to as the desired signal), there are often other interfering signals or additive colored noise. Moreover, measurement errors are always inevitable, and they are usually expressed as Gaussian white noise. Let θ be the parameter vector of the signal x to be estimated, and let the linear system H provide this signal, x = Hθ. Assume that other interference signal vectors and/or the additive colored noise vectors i are generated by another synthesizing linear system S with parameter vector φ, i.e., i = Sφ. If the

584

Projection Analysis

observation data vector is y, and the additive white observation error vector is e then y = Hθ + Sφ + e.

(9.5.28)

Figure 9.5 shows a block diagram of this observation model. θ

H

φ

S

+

+

y

e

Figure 9.5 Observation model.

In general, the column vectors of the system matrix H generating the desired signal are not only linearly independent of each other but also are linearly independent of the column vectors of the linear system matrix S generating interference or colored noise. Therefore, the range Range(H) of H and the range Range(S) of S are disjoint, but in general they will be nonorthogonal. Any undesired signal generated by a linear system is known as structured noise. Assuming that this structured noise is orthogonal to the additive Gaussian white noise e, we have Sφ, e = 0



φH SH e = 0, ∀ φ = 0



SH e = 0.

(9.5.29)

Similarly, letting the desired signal also be orthogonal to the additive Gaussian white noise, then HH e = 0.

(9.5.30)

Given two matrices H and S, the objective of system modeling is to estimate the system parameter vector θ relating to the desired signal. To this end, premultiplying (9.5.28) by the orthogonal projection matrix PS⊥ , we have PS⊥ y = PS⊥ Hθ + e.

(9.5.31)

Here the relation PS⊥ S = O and PS⊥ e = [I − S(SH S)−1 SH ]e = e have been used. Premultiplying both sides of (9.5.31) by the matrix HH , and using HH e = 0, then θ = (HH PS⊥ H)−1 HH PS⊥ y.

(9.5.32)

Hence an estimate of the desired signal is given by ˆ = Hθ = H(HH PS⊥ H)−1 HH PS⊥ y = EH|S y, x

(9.5.33)

i.e., the estimate of the desired signal is the oblique projection of the observation

Exercises

585

data vector y onto the column space of H along the direction parallel to the column space of S.

Exercises 9.1 9.2 9.3

9.4

9.5

9.6

9.7

9.8 9.9

Let P be a projector, and Range(P) = Col(U), Null(P) = Col(V). Use the matrices U and V to represent the projector P. Let both E1 and E2 be orthogonal projectors, and let E1 E2 = E2 E1 . Show that E1 E2 is an orthogonal projector. Let X denote an observation data matrix. Use it to estimate a vector y as follows. Given two observation data vectors x1 = [2, 1, 2, 3]T , x2 = [1, 2, 1, 1]T in X, use them to estimate y = [1, 2, 3, 2]T and find the sum of the squared estimation errors. (Hint: Let the optimal filter vector be wopt ; then the observation equation is Xwopt = y.) Assuming that V1 , V2 are two sets of orthonormal basis vectors of the subspace W of a complex vector space Cn , prove that V1 V1H x = V2 V2H x holds for all vectors x. Show that if the projectors P1 and P2 are commute, i.e., P1 P2 = P2 P1 , then their product P = P1 P2 is also a projector and find the range Range(P) and the null space Null(P) of P, respectively.     0 1 1 6 2 , find their respective and A = Given the matrices A = 1 1 0 −7 6 Moore–Penrose inverse matrices A† , and explain why AA† and A† A are orthogonal projections respectively onto the column space and the row space of the matrix A. Assuming two basis vectors u1 = [−1, 2, −4, 3, 1]T and u2 = [5, 6, 2, −2, −1]T , generate the column space Range(U) = Span{u1 , u2 }. Is the vector v = [−31, −18, −34, 28, 11]T in the vector space Range(U)? Explain your answer. Use the inverse matrix of inner product X1,p (n − 1), X1,p (n − 1)−1 to represent the inverse matrix of inner product X1,p (n), X1,p (n)−1 . For matrices H, S, define EH|S = H(HH PS⊥ H)−1 HH PS⊥ −1 H ⊥ ES|H = S(SH P⊥ S PH . H S)

Show that (a) EH|S and ES|H are idempotent operators, i.e., E2H|S = EH|S and E2S|H = ES|H ; (b) the oblique projection of [H, S] along the direction of Range(H) onto Range(S) is given by EH|S [H, S] = [H, O] and ES|H [H, S] = [O, S], or

586

Projection Analysis

equivalently by EH|S H = H and

EH|S S = O,

ES|H H = O

ES|H S = S.

and

9.10 Prove the following results: (a) PH EH|S = EH|S and PS ES|H = ES|H ; (b) EH|S PH = PH and ES|H PS = PS . 9.11 Let H = Range(H) and S = Range(S). Prove the following results: PS⊥ EH|S PS⊥ = PS⊥ PPS⊥ H PS⊥ = PPS⊥ H . 9.12 If H = Range(H), S = Range(S), and H ⊥ S show that EH|S = PH and ES|H = PS . 9.13 Prove that ES|H = PS − PS EH|S = PS (I − EH|S ). 9.14 Prove that the products of the generalized inverse matrices H† and S† and † their oblique projection satisfy H† EH|S = (PS⊥ H)† and S† ES|H = (P⊥ H S) , † H −1 H where B = (B B) B denotes the generalized inverse matrix of B. 9.15 Prove that the cross-terms of the oblique projectors EH|S and ES|H is equal to zero, i.e., EH|S ES|H = O and ES|H EH|S = O.

PART III HIGHER-ORDER MATRIX ANALYSIS

10 Tensor Analysis

In many disciplines, for more and more problems three or more subscripts are needed to describe the data. Data with three or more subscripts are referred to as multi-channel data, whose representation is the tensor. Data analysis based on vectors and matrices belongs to linear data analysis, while data analysis based on tensors is known as tensor analysis, which belongs to the category of multilinear data analysis. Linear data analysis is a single-factor analysis method, and multilinear data analysis is a multi-factor analysis method. Just as tensors are the generalization of matrices, multilinear data analysis is a natural extension of linear data analysis.

10.1 Tensors and their Presentation A column or row vector, with data arranged all in the same direction, is known as a one-way data array, while a matrix is a two-way data array, with data arranged along both the horizontal and vertical directions. A tensor is a multi-way data array that is an extension of a matrix. Thus a three-way tensor has data arranged in three directions and an N -way tensor has N “directions” or modes. The tensor in mathematics refers specifically to such a multi-way data array and should not be confused with the tensors in physics and engineering, which are often referred to as tensor fields [447].

10.1.1 Tensors Here, tensors will be represented by mathcal symbols, such as T , A, X and so on. A tensor represented in an N -way array is called an N th-order tensor, and is defined as a multilinear function on an N -dimensional Cartesian product vector space, denoted T ∈ KI1 ×I2 ×···×IN , where K denotes either the real field R or the complex field C and where In is the number of entries or the dimension in the nth “direction” or mode. For example, for a third-order tensor T ∈ K4×2×5 , the first, second and third modes have dimensions respectively equal to 4, 2 and 5. A scalar is a zero589

590

Tensor Analysis

order tensor, a vector is a first-order tensor and a matrix is a second-order tensor. An N th-order tensor is a multilinear mapping T : KI1 × KI2 × · · · × KIN → KI1 ×I2 ×···×IN .

(10.1.1)

Because T ∈ KI1 ×···×IN can be regarded as an N th-order matrix, tensor analysis is essentially higher-order matrix analysis. With development of modern applications, multi-way higher-order data models have been successfully applied across many fields, which include, among others, chemometrics, chemistry, large data analysis and fusion, medicine and neuroscience, signal processing, text mining, clustering, internet traffic, EEG, computer vision, communication records, large-scale social networks etc. The following are typical applications of multi-way data models in various subjects [7]. 1. Social network analysis/Web mining Multi-way data analysis is often used to extract relationships in social networks. The purpose of social network analysis is to study and discover the hidden structure in social networks, for example, to extract the patterns of communication between people or between people and organizations. In [6], chat room communication data is arranged into a three-way array “user × keyword × time sample”, and the performance of this multi-way model in capturing the potential user-group structure is compared with the twoway model. Not only chat rooms, but also e-mail communication data can also be expressed using the three-way model “sender × receiver × time”. In the analysis of network links, by combining the hyperlink with the anchor text information, the page graphic data are rearranged into a sparse three-way model “page × page × anchor text” [255], [256]. In Web personalization, click data are arranged into a three-way array “user × query word × page” [458]. 2. Computer vision Tensor approximation has been shown to have important applications in computer vision, such as in image compression and face recognition. In computer vision and computer graphics, the data are usually multi-way. For example, a color image is a three-way array: its x and y coordinates are two modes and the color is the third mode. Fast image encoding technology regards an image as a vector or matrix. It has been shown that when an image is expressed as a tensor, a rank-1 approximation to the tensor obtained by iteration can be used to compress the image [499]. In addition, not only can this tensor structure of an image retain the two-dimensional features of the image but also it avoids the loss of image information [202]. Therefore, in image compression, a tensor representation is more effective than a matrix representation [435]. 3. Hyperspectral image A hyperspectral image is usually a set of two-dimensional images, and thus it can be represented by three-way data: two-way spatial data and one-way spectral data [376], [91], [290]: the two-way data determine the location of the pixel and the one-way spectral data are related to the spectral band.

10.1 Tensors and their Presentation

591

4. Medicine and neuroscience Multi-way electroencephalography (EEG) data are typically represented as an m × n matrix whose elements are the signal values from m time samples and n electrodes. However, in order to find the hidden power structure of a brain, we need to consider the frequency components of the EEG signals (such as the instantaneous signal powers at a particular frequency p). In this case, the EEG data can be arranged in a three-way data set. A threeway data array corresponds to channel or space (different electrode position), time of the data sample and frequency component. If adding theme and condition, then one obtains a five-way array or tensor representing “channel × time × frequency × theme × condition”. Multi-way models were used to study the effects of new drugs on brain activity in 2001 [148]. In this study, the experimental data from the EEG, and different dosages of the drug for several patients, in some cases, were arranged in a six-way array corresponding to EEG, dosage, patient, condition etc. The results showed that through the multi-way model rather than a two-way model (such as used in principal component analysis), important information was successfully extracted from complex drug data. 5. Multilinear image analysis of face recognition [491], [492] Faces are modeled using a fifth-order tensor D ∈ R28×5×3×3×7943, which represents the face images from 28 personal shots in five viewpoints, three illuminations and three expressions, while every image contain 7943 pixels. 6. Epilepsy tensors By identifying the spatial, spectrum and temporal features of epileptic seizure and false appearance, one can determine the focal point of epileptic seizure or exclude the false appearance of epilepsy. Therefore, it is necessary to use a three-way array consisting of time, frequency and space (electrode position) to model epilepsy signals [5]. 7. Chemistry In the chemical, pharmaceutical and food sciences, commonly used fluorescence excitation–emission data contain several chemical compositions at different concentrations. The fluorescence spectra generate a three-way data set with the model “sample × excitation × emission”. The main purpose of analyzing this type of data is to determine the chemical components and their relative concentration in every sample. Figure 10.1 plots examples of a three-way array and a five-way tensor modeling a face. frequency

viewpoint

space (channel, subject, condition)

illumination

times

people pixel

expression

Figure 10.1 Three-way array (left) and a tensor modeling a face (right).

592

Tensor Analysis

The main objective of this chapter is to investigate the relationship between matrices and tensors and to address the following questions: how to matricize (tensorize) data and how to apply the inherent tensor structure in data in highdimensional problems.

10.1.2 Tensor Representation In linear algebra (i.e., matrix algebra in a finite-dimensional vector space) a linear operator is defined in a finite-dimensional vector space. Because a matrix expresses a second-order tensor, linear algebra can also be regarded as the algebra of secondorder tensors. In multiple linear algebra (i.e., higher-order tensor algebra) a multiple linear operator is also defined in a finite-dimensional vector space. A matrix A ∈ Km×n is represented by its entries and the symbol [·] as A = I1 ×I2 ×···×In [aij ]m,n is, using a dual matrix i,j=1 . Similarly, an nth-order tensor A ∈ K I1 ,...,In symbol [[·]], represented as A = [[ai1 ···in ]]i1 ,...,in =1 , where ai1 i2 ···in is the (i1 , . . . , in )th entry of the tensor. An nth-order tensor is sometimes called an n-dimensional hypermatrix [111]. The set of all (I1 × I2 × · · · × In )-dimensional tensors is often denoted T (I1 , I2 , . . . , In ). I×J×K The most common tensor is a third-order tensor, A = [[aijk ]]I,J,K .A i,j,k ∈ K third-order tensor is sometimes called a three-dimensional matrix [476]. A square third-order tensor X ∈ KI×I×I is said to be cubical. In particular, a cubical tensor is supersymmetric [110], [257], if its entries have the following symmetry: ∀ i, j, k = 1, . . . , I.

xijk = xikj = xjik = xjki = xkij = xkji ,

Figure 10.2(a) depicts a third-order tensor A ∈ RI×J×K .

11

11

11

11

11

11

11

j=1,...,J

(a) A ∈ RI×J×K

k=

1,

...

,K

i=1,...,I

111

(b) Unit tensor I ∈ RN ×N ×N

Figure 10.2 Third-order tensors.

For a tensor A = [[ai1 ···im ]] ∈ RN ×···×N , the line connecting a11···1 to aN N ···N is called the superdiagonal line of the tensor A. A tensor A is known as unit tensor, if all its entries on the superdiagonal line are equal to 1 and all other entries are

10.1 Tensors and their Presentation

593

equal to zero, i.e., + ai1 ···im = δi1 ···im =

1,

if i1 = · · · = im ,

0,

otherwise.

(10.1.2)

A unit third-order tensor is denoted I ∈ RN ×N ×N , as shown in Figure 10.2(b). In tensor analysis, it is convenient to view a third-order tensor as a set of vectors or matrices. Suppose that the ith entry of a vector a is denoted by ai and that a matrix A = [aij ] ∈ KI×J has I row vectors ai: , i = 1, . . . , I and J column vectors a:j , j = 1, . . . , J; the ith row vector ai: is denoted by [ai1 , . . . , aiJ ], and the jth column vector a:j is denoted by [a1j , . . . , aIj ]T . However, the concepts of row vectors and column vectors are no longer directly applicable for a higher-order tensor. The three-way arrays of a third-order tensor are not called row vectors and column vectors, but are renamed tensor fibers. A fiber is a one-way array with one subscript that is variable and the other subscripts fixed. The fibers of a third-order tensor are vertical, horizontal and “depth” fibers. The vertical fibers of the thirdorder tensor A ∈ KI×J×K are also called column fibers, denoted a:jk ; the horizontal fibers are also known as the row fibers, denoted ai:k ; the depth fibers are also called the tube fiber, denoted aij: . Figure 10.3(a)–(c) show respectively the column fibers, row fibers and tube fibers of a third-order tensor.

(a) Column fibers a:jk

(b) Row fibers ai:k

(c) Tube fibers aij:

Figure 10.3 The fiber patterns of a third-order tensor.

Obviously, a third-order tensor A ∈ KI×J×K has respectively JK column fibers, KI row fibers and IJ tube fibers. An N th-order tensor has N different kinds of fibers, called mode-n fibers or mode-n vectors, where n = 1, . . . , N . DEFINITION 10.1 [280] Let A = [[ai1 ···iN ]] ∈ KI1 ×···×IN be an N th-order tensor; its mode-n vectors, for n = 1, . . . , N , are In -dimensional vectors with in as the entry subscript that is variable and all the other subscripts {i1 , . . . , iN } \ in fixed for a given mode-n vector, denoted as ai1 ···in−1 :in+1 ···iN .

594

Tensor Analysis

The difference between the order and the dimensions of a tensor A ∈ RI1 ×···×IN is that N is the order of the tensor, while In is the dimension of the mode-n vectors. For a matrix, its column vectors are the mode-1 vectors and its row vectors are the mode-2 vectors. For example, for a 2 × 3 matrix, with two rows and three columns, the mode-1 vectors (the three column vectors) have dimension 2 and the mode-2 vectors (the two row vectors) have dimension 3. In a third-order tensor A ∈ KI×J×K , the column fibers a:jk , row fibers ai:k and tube fibers aij: are the mode-1, mode-2 and mode-3 vectors, respectively. There are JK mode-1 vectors, denoted a:jk , and every mode-1 vector has I entries, i.e., a:jk = (a1jk , . . . , aIjk ); there are KI mode-2 vectors, denoted ai:k , and every mode-2 vector consists of J entries, i.e., ai:k = (ai1k , . . . , aiJk ); while there are IJ mode-3 vectors, denoted aij: = (aij1 , . . . , aijK ). The subspace spanned by the mode-n vectors is called a mode-n space. Generally, a:i2 ,...,iN , ai1 :i3 ,...,iN , ai1 ,...,in−1 :in+1 ,...,iN are respectively the mode-1, mode-2 and mode-n vectors of the N th-order (N > 3) tensor A ∈ KI1 ×I2 ×···×IN . Clearly, an N th-order tensor has a total of I2 · · · IN mode-1 vectors and I1 · · · In−1 In+1 · · · IN mode-n vectors. Note that we do not write the mode-n vector representation as a column vector [· · · ]T or row vector [· · · ], but deliberately use the symbol (· · · ) because the same mode vector is sometimes a column vector and sometimes a row vector, depending on the slice matrices (see below) of the tensor. A higher-order tensor can also be represented by matrix sets. For a third-order tensor, there exist three kinds of slice matrices: the horizontal slice, the lateral slice and the frontal slice, as shown in Figure 10.4.

(a) Horizontal slices

(b) Lateral slices

(c) Frontal slices

Figure 10.4 Slices of a third-order tensor.

The horizontal slices, lateral slices and frontal slices of a third-order tensor are respectively denoted by the matrix notations Ai:: , A:j: and A::k . Figure 10.5 shows representations of these three slice matrices. In Figure 10.5, the arrow direction denotes the column direction for each slice matrix. In fact, the indexes i, j, k of a third-order tensor constitute three index sets (i, j), (j, k), (k, i) in order. The first and second elements of each set represent respectively the row and column indexes of the corresponding slice matrix. The relationships between the rows and columns of each slice matrix of a thirdorder tensor are as follows:

10.1 Tensors and their Presentation



horizontal slice

k



slice matrix

595

Ai::

j: row

j k: column



i

lateral slice



slice matrix

A:j:

k: row

k i: column



i

frontal slice

j

slice matrix

A::k

i: row



j: column

Figure 10.5 Representation of slice matrices.

For the horizontal slice matrix Ai:: , j is the row index, k is the column index; For the lateral slice matrix A:j: , k is the row index, i is the column index; For the frontal slice matrix A::k , i is the row index, j is the column index. The following are mathematical representations of the three kinds of slice matrices of a third-order tensor. (1) The third-order tensor A ∈ KI×J×K has I horizonal slice matrices ⎤ ⎤ ⎡ ⎡ ai11 · · · ai1K ai1: def ⎢ .. ⎥ = [a , . . . , a ] = ⎢ .. ⎥ .. Ai:: = ⎣ ... ⎣ . ⎦ . i:1 i:K . ⎦ aiJ1 · · · aiJK aiJ: for i = 1, . . . , I. (2) The third-order tensor A ∈ KI×J×K has J lateral slice matrices ⎤ ⎤ ⎡ ⎡ a:j1 a1j1 · · · aIj1 def ⎢ .. ⎥ = [a , . . . , a ] = ⎢ .. ⎥ .. A:j: = ⎣ ... ⎣ . ⎦ . 1j: Ij: . ⎦ a:jK a1jK · · · aIjK for j = 1, . . . , J. (3) The third-order tensor A ∈ KI×J×K has K frontal slice matrices ⎡ ⎡ ⎤ ⎤ a11k · · · a1Jk a1:k def ⎢ .. ⎥ = [a , . . . , a ] = ⎢ .. ⎥ .. A::k = ⎣ ... ⎣ . ⎦ . :1k :Jk . ⎦ aI1k · · · aIJk aI:k for k = 1, . . . , K.

(10.1.3)

(10.1.4)

(10.1.5)

596

Tensor Analysis

From the above analysis we have the following results: " a:1k , . . . , a:Jk denote J column vectors of A::k , for the mode-1 vectors a:jk : a:j1 , . . . , a:jK denote K row vectors of A:j: ; " ai:1 , . . . , ai:K denote K column vectors of Ai:: , for the mode-2 vectors ai:k : a1:k , . . . , aI:k denote I row vectors of A::k ; " a1j: , . . . , aIj: denote I column vectors of A:j: , for the mode-3 vectors aij: : ai1: , . . . , aiJ: denote J row vectors of Ai:: . EXAMPLE 10.1 i.e., A::1

Given two frontal slice matrices of the tensor A ∈ R3×4×2 [254], ⎡

⎤ 1 4 7 10 = ⎣2 5 8 11⎦ , 3 6 9 12



A::2

13 = ⎣14 15

16 17 18

19 20 21

⎤ 22 23⎦ 24

∈ R3×4 ,

find all the mode-n vectors and the horizontal and lateral slice matrices of A. Solution The total number of mode-1 vectors (i.e., column fibers) is JK = 4×2 = 8; they are ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 1 4 7 10 13 16 19 22 ⎣2⎦ , ⎣5⎦ , ⎣8⎦ , ⎣11⎦ , ⎣14⎦ , ⎣17⎦ , ⎣20⎦ , ⎣23⎦ , 3 6 9 12 15 18 21 24 and they span the mode-1 space of the tensor A ∈ R3×4×2 . The total number of mode-2 vectors is KI = 2 × 3 = 6; they are [1, 4, 7, 10], [2, 5, 8, 11], [3, 6, 9, 12]; [13, 16, 19, 22], [14, 17, 20, 23], [15, 18, 21, 24]. The total number of mode-3 vectors is IJ = 3 × 4 = 12; they are [1, 13], [4, 16], [7, 19], [10, 22]; [2, 14], [5, 17], [8, 20], [11, 23]; [3, 15], [6, 18], [9, 21], [12, 24]. Moreover, the three horizontal slice matrices of the tensor are ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 1 13 2 14 3 15 ⎢ 4 16⎥ ⎢ 5 17⎥ ⎢ 6 18⎥ ⎥ ⎢ ⎥ ⎢ ⎥ A1:: = ⎢ ⎣ 7 19⎦ , A2:: = ⎣ 8 20⎦ , A3:: = ⎣ 9 21⎦ 10 22 11 23 12 24 The four lateral slice matrices are   1 2 3 , A:1: = 13 14 15   7 8 9 , A:3: = 19 20 21



A:2: A:4:

4 = 16  10 = 22

5 17 11 23

⎫ 6 ⎪ ⎪ ⎪ 18 ⎬  12 ⎪ ⎪ ⎪ ⎭ 24

∈ R4×2 .

∈ R2×3 .

10.2 Vectorization and Matricization of Tensors

597

By extending the concept of the slices of a third-order tensor, we can divide a fourth-order tensor A ∈ KI×J×K×L into L third-order tensor slices A1 , . . . , AL ∈ KI×J×K . By analogy, a fifth-order tensor AI×J×K×L×M can be divided into M fourth-order tensor slices, and each fourth-order tensor slice can be further divided into L third-order tensor slices. In this sense, the third-order tensor is the basis of tensor analysis.

10.2 Vectorization and Matricization of Tensors A third-order tensor A ∈ KI×J×K has I horizontal slice matrices Ai:: ∈ KJ×K , i = 1, . . . , I, J lateral slice matrices A:j: ∈ KK×I , j = 1, . . . , J and K frontal slice matrices A::k ∈ KI×J , k = 1, . . . , K. However, in tensor analysis and computation, it is frequently desirable to use a single matrix to represent a third-order tensor. In this case, we need an operation such that a third-order tensor (a three-way array) is reorganized or rearranged into a matrix (a two-way array). The transformation of a three-way or N -way array into a matrix form is called tensor matricization [245]. The matricization of a tensor A is denoted by unvec(A). The matricization of a tensor is sometimes known as tensor unfolding or tensor flattening [65]. It should be noted that the unfolding or flattening here refers only to the unfolding of a cubic or three-dimensional array into a planar or two-dimensional array. In addition to a unique matrix formulation, a unique vector formulation of a higher-order tensor is also of interest in many applications. Tensor vectorization is a transformation that rearranges the tensor as a unique vector. The vectorization of a tensor A is denoted by vec(A). 10.2.1 Vectorization and Horizontal Unfolding As mentioned above, the tensor vectorization of an N th-order tensor A = [[Ai1 ,...,iN ]] ∈ KI1 ×···×IN is denoted by a = vec(A), that is, a mapping a : KI1 ×···×IN → KI1 I2 ···IN ×1 . For all the indices in ∈ {1, . . . , In }, n = 1, . . . , N , the entries of the vector a are given by   N n−1  ( (in − 1) (10.2.1) Ik . al = Ai1 ,i2 ,...,iN , l = i1 + n=2

k=1

For example, a1 = A1,1,...,1 , aI1 = AI1 ,1,...,1 , aI2 = A1,I2 ,1,...,1 , aI1 I2 ···IN = AI1 ,I2 ,...,IN . Given an N th-order tensor A ∈ KI1 ×···×IN, the matrix A(n) =KIn ×(I1 ···In−1 In+1 ···IN ) is known as the (mode-n) horizontal unfolding matrix of the tensor A; and the matrix A(n) = K(I1 ···In−1 In+1 ···IN )×In is called the (mode-n) longitudinal unfolding matrix of A. Hence, the mappings A → A(n) and A → A(n) are said to be the horizontal unfolding and longitudinal unfolding of the tensor A, respectively.

598

Tensor Analysis

We discuss first the horizontal unfolding of a tensor here. In a horizontal unfolding of a third-order tensor, matrices from the same slice set are arranged side by side in a row in the same plane. There are three horizontal unfolding methods. 1. Kiers Horizontal Unfolding Method This is the tensor matricization method proposed by Kiers in 2000 [245]. In this method, a third-order tensor A ∈ KI×J×K is matricized as one of the following three unfolding matrices: ⎫ (I×JK) aKiers ⇔ AKiers = AKiers(1) = [A::1 , . . . , A::K ], ⎪ i,(k−1)J+j = aijk ⎪ ⎪ ⎪ ⎬ (J×KI) Kiers (10.2.2) aj,(i−1)K+k = aijk ⇔ AKiers = AKiers(2) = [A1:: , . . . , AI:: ], ⎪ ⎪ ⎪ ⎪ ⎭ (K×IJ) aKiers ⇔ AKiers = AKiers(3) = [A:1: , . . . , A:J: ]. k,(j−1)I+i = aijk Figure 10.6 plots the Kiers horizontal unfolding of a third-order tensor A ∈ K . The three unfoldings are pictured on the right of the figure. I×J×K

J =⇒ AKiers(1) : I

=⇒

I

A::1

A::2

J

···

A::K

···

A::I

···

A:J:

K K =⇒ AKiers(2) : J

=⇒

I J

K

J

A1::

A2::

K I

=⇒

I =⇒ AKiers(3) : K

I

A:1:

A:2:

K J

Figure 10.6 The horizontal unfolding of a tensor (Kiers method).

By extension, given an N th-order tensor A ∈ KI1 ×I2 ×···×IN , the Kiers horizontal unfolding method maps its entry ai1 i2 ···iN to the (in , j)th entry of the matrix AKiers(n) ∈ KIn ×(I1 ···In−1 In+1 ···IN ) , i.e., AKiers(n) (in , j) = aKiers in ,j = ai1 ,i2 ,...,iN , where in = 1, . . . , In and   N +n−p−1 N −2  ( (iN +n−p − 1) Iq + in+1 , j= p=1

q=n+1

n = 1, . . . , N

(10.2.3)

(10.2.4)

10.2 Vectorization and Matricization of Tensors

599

with IN +m = Im and iN +m = im (m > 0). 2. LMV Horizontal Unfolding Method Lathauwer, Moor and Vanderwalle [281] in 2000 proposed the following horizontal unfolding of a third-order tensor (simply called the LMV method): ⎫ (I×JK) aLMV ⇔ ALMV = ALMV(1) = [AT:1: , . . . , AT:J: ], ⎪ ⎪ i,(j−1)K+k = aijk ⎪ ⎪ ⎬ (J×KI) LMV T T (10.2.5) aj,(k−1)I+i = aijk ⇔ ALMV = ALMV(2) = [A::1 , . . . , A::K ], ⎪ ⎪ ⎪ ⎪ (K×IJ) aLMV ⇔ ALMV = ALMV(3) = [AT1:: , . . . , ATI:: ]. ⎭ k,(i−1)J+j = aijk Figure 10.7 shows the LMV horizontal unfolding of a third-order tensor. K =⇒ ALMV(1) : I

=⇒

I

AT :1:

AT :2:

···

AT :J:

···

AT ::K

···

AT I::

K J I

=⇒

I J

=⇒ ALMV(2) : J

I

K

AT ::1

AT ::2

J

K

=⇒

J =⇒ ALMV(3) : K J

AT 1::

AT 2::

K I

Figure 10.7 The horizontal unfolding of a tensor (LMV method).

This unfolding maps the entry ai1 ,...,iN of an N th-order tensor A ∈ KI1 ×···×IN In ×(I1 ···In−1 In+1 ···IN ) to the entry aLMV , where in ,j of the mode-n matrix ALMV(n) ∈ K j = (in+1 − 1)In+2 In+3 · · · IN I1 I2 · · · In−1 + (in+2 − 1)In+3 In+4 · · · IN I1 I2 · · · In−1 + · · · + (iN − 1)I1 I2 · · · In−1 + (i1 − 1)I2 I3 · · · In−1 + (i2 − 1)I3 I4 · · · In−1 + · · · + in−1 or, written in a compact form,   n−1  N   n−2 n−1 N  ( (  ( (ik −1) (ik −1) , (10.2.6) j = in−1 + Im + Ip Iq+1 k=1

where Iq = 1 if q > N .

m=k+1

p=1

k=n+1

q=k

600

Tensor Analysis

3. Kolda Horizontal Unfolding Method The Kolda horizontal unfolding [254] maps the entry ai1 ,...,iN of an N th-order tensor of the mode-n matrix AKolda(n) ∈ KIn ×(I1 ···In−1 In+1 ···IN ) , where to the entry aKolda in ,j   N k−1  ( j =1+ (ik − 1) (10.2.7) Im . k=1,k=n

m=1,m=n

Figure 10.8 shows the Kolda horizontal unfolding of a third-order tensor. J =⇒ AKolda(1): I

=⇒

I

A::1

A::2

J

···

A::K

···

AT ::K

···

A:J:

K I

=⇒

I J

=⇒ AKolda(2): J

I

K

AT ::1

AT ::2

J

K

=⇒

I =⇒ AKolda(3): K

I

A:1:

A:2:

K J

Figure 10.8 The horizontal unfolding of a tensor (Kolda method).

The entries of the Kolda matricization of a third-order tensor are as follows: ⎫ (I×JK) aKolda ⇔ AKolda = AKolda(1) = [A::1 , . . . , A::K ], ⎪ ⎪ i,(k−1)J+j = aijk ⎪ ⎪ ⎬ (J×KI) Kolda T T (10.2.8) aj,(k−1)I+i = aijk ⇔ AKolda = AKolda(2) = [A::1 , . . . , A::K ], ⎪ ⎪ ⎪ ⎪ (K×IJ) aKolda ⇔ AKolda = AKolda(3) = [A:1: , . . . , A:J: ]. ⎭ k,(j−1)I+j = aijk Comparing the above three horizontal unfolding methods, we have their relationships. (1) The mode-1 horizontal unfolding matrices of both the Kolda method and the Kiers method are the same, and their mode-3 horizontal unfolding matrices are also the same, namely AKolda(1) = AKiers(1) ,

AKolda(3) = AKiers(3) .

(10.2.9)

(2) The mode-2 horizontal unfolding matrices of both the Kolda method and the LMV method are the same, i.e., AKolda(2) = ALMV(2) .

(10.2.10)

10.2 Vectorization and Matricization of Tensors

EXAMPLE 10.2

601

Consider a third-order tensor A with entries

A1,1,1 = 1,

A1,2,1 = 3,

A1,1,2 = 5,

A1,2,2 = 7,

A2,1,1 = 2,

A2,2,1 = 4,

A2,1,2 = 6,

A2,2,2 = 8,

by (10.2.1), we have the tensor vectorization a = [1, 2, 3, 4, 5, 6, 7, 8]T . From (10.2.2) we get the Kiers horizontal unfolding matrices:  A111 A211  A111 = [A:1: , A:2: ] = A211  A111 = [A1:: , A2:: ] = A121

AKiers(1) = [A::1 , A::2 ] =

A121 | A112 A221 | A212

AKiers(2)

A112 | A121 A212 | A221

AKiers(3)

A112 | A211 A122 | A221

  1 A122 = 2 A222   A122 1 = A222 2   A212 1 = A222 3

 3 | 5 7 , 4 | 6 8  5 | 3 7 , 6 | 4 8  5 | 2 6 . 7 | 4 8

10.2.2 Longitudinal Unfolding of Tensors If the same kind of slice matrices of a tensor are arranged in a longitudinal direction, then we get the longitudinal unfolding of a tensor. This is the transpose of the horizontal unfolding. Thus longitudinal unfolding has also three methods, which correspond to the three methods of horizontal unfolding. The mode-n longitudinal unfolding of an N th-order tensor A ∈ KI1 ×I2 ×···×IN is denoted by the notation A(n) ∈ K(I1 ···In−1 In+1 ···IN )×In . A third-order tensor A ∈ KI×J×K can be matricized as a (JK) × I matrix, a (KI) × J matrix or an (IJ) × K matrix, denoted A(1) = A(JK×I) ,

A(2) = A(KI×J) ,

A(3) = A(IJ×K) .

(10.2.11)

1. Kiers Longitudinal Unfolding Method Given an N th-order tensor A ∈ KI1 ×I2 ×···×IN , let A(I1 ···In−1 In+1 ···IN )×In be its mode-n longitudinal unfolding. Then, the entry of the jth row and the in th column of the longitudinal unfolding is defined as (I ···In−1 In+1 ···IN )×In

aj,i1n

= ai1 ,i2 ,...,iN ,

(10.2.12)

where j is given by (10.2.4). More specifically, the Kiers method for the longitudinal unfolding of a third-order

602

Tensor Analysis

tensor gives ⎤ AT::1 ⎥ ⎢ = ⎣ ... ⎦ , ⎡

(JK×I)

a(k−1)J+j,i = aijk



(1)

A(JK×I) = AKiers

AT::K ⎤ AT1:: ⎢ ⎥ = ⎣ ... ⎦ , ⎡

(KI×J)

a(i−1)K+k,j = aijk



(2)

A(KI×J) = AKiers

⎫ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎬

⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎢ .. ⎥ ⎪ ⎪ = ⎣ . ⎦. ⎪ ⎪ ⎪ ⎪ ⎭ T A:J:

(10.2.13)

AT ⎤ ⎡ I:: AT:1:

(IJ×K)

a(j−1)I+i,k = aijk



(3)

A(IJ×K) = AKiers

2. LMV Longitudinal Unfolding Method The LMV longitudinal unfolding of a third-order tensor is represented as [282] ⎤ ⎫ ⎡ ⎪ A:1: ⎪ ⎪ ⎢ .. ⎥ ⎪ (JK×I) (1) (JK×I) ⎪ a(j−1)K+k,i = aijk ⇔ A = ALMV = ⎣ . ⎦ , ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ A:J: ⎪ ⎪ ⎤ ⎪ ⎡ ⎪ ⎪ A::1 ⎪ ⎬ ⎢ .. ⎥ (KI×J) (2) (KI×J) (10.2.14) = ALMV = ⎣ . ⎦ , a(k−1)I+i,j = aijk ⇔ A ⎪ ⎪ ⎪ ⎪ A::K ⎪ ⎤ ⎪ ⎡ ⎪ ⎪ ⎪ A1:: ⎪ ⎪ ⎪ ⎥ ⎢ (IJ×K) (3) . (IJ×K) ⎪ . = ALMV = ⎣ . ⎦ , ⎪ a(i−1)J+j,k = aijk ⇔ A ⎪ ⎪ ⎪ ⎭ AI:: where the row subscript j is determined by (10.2.6). 3. Kolda Longitudinal Unfolding Method The Kolda longitudinal unfolding of a third-order tensor is given by ⎡ T ⎤ ⎫ ⎪ A::1 ⎪ ⎪ ⎢ .. ⎥ ⎪ (JK×I) (1) (JK×I) ⎪ a(k−1)J+j,i = aijk ⇔ A = AKolda = ⎣ . ⎦ , ⎪ ⎪ ⎪ ⎪ ⎪ T ⎪ A::K ⎪ ⎪ ⎤ ⎪ ⎡ ⎪ ⎪ A::1 ⎪ ⎬ ⎥ ⎢ (KI×J) (2) . (KI×J) . = AKolda = ⎣ . ⎦ , a(k−1)I+i,j = aijk ⇔ A ⎪ ⎪ ⎪ ⎪ A::K ⎪ ⎡ T ⎤ ⎪ ⎪ ⎪ ⎪ A:1: ⎪ ⎪ ⎢ .. ⎥ ⎪ (IJ×K) (3) (IJ×K) ⎪ = AKolda = ⎣ . ⎦ . ⎪ a(j−1)I+i,k = aijk ⇔ A ⎪ ⎪ ⎪ ⎭ T A:J:

(10.2.15)

10.2 Vectorization and Matricization of Tensors

603

The entries of the Kolda longitudinal matricization of an N th-order tensor are also given by (10.2.12), but its row subscript j is determined by (10.2.7). Regarding the horizontal and longitudinal unfoldings, there are the following relationships. (1) The relationships between the longitudinal unfoldings of a third-order tensor are (1)

(1)

AKiers = AKolda ,

(3)

(3)

AKiers = AKolda ,

(2)

(2)

ALMV = AKolda .

(10.2.16)

(2) The relationships between the longitudinal unfolding and the horizontal unfolding of a third-order tensor are (n)

AKiers = ATKiers(n) ,

(n)

ALMV = ATLMV(n) ,

(n)

AKolda = ATKolda(n) .

(10.2.17)

(3) The LMV mode-n longitudinal unfolding matrix is a longitudinal arrangement of the Kiers mode-n horizontal unfolding matrix. Conversely, the Kiers mode-n longitudinal unfolding matrix is a longitudinal arrangement of the LMV mode-n horizontal unfolding matrix. Some publications (e.g., [65], [245], [5], [7]) use the horizontal unfolding of tensors, while others (e.g., [302], [282], [257]) use longitudinal unfolding. From the following example we can see the connection and difference between the Keirs, LMV and Kolda matricization results for the same tensor. EXAMPLE 10.3 R3×4×2 given by ⎡ 1 ⎣ A::1 = 2 3

Consider again the two frontal slice matrices of the tensor A ∈ [254] ⎤ ⎡ ⎤ 4 7 10 13 16 19 22 (10.2.18) 5 8 11⎦ , A::2 = ⎣14 17 20 23⎦ ∈ R3×4 , 6 9 12 15 18 21 24

from Example 10.1. The horizontal follows: (1) The Keirs horizontal unfolding ⎡ 1 4 7 10 13 AKeirs(1) = ⎣2 5 8 11 14 3 6 9 12 15 ⎡ 1 13 2 14 ⎢ 4 16 5 17 AKeirs(2) = ⎢ ⎣ 7 19 8 20 10 22 11 23  1 2 3 4 AKeirs(3) = 13 14 15 16

and longitudinal unfolding matrices are as

16 19 17 20 18 21 3 6 9 12

⎤ 22 23⎦ , 24 ⎤

15 18 ⎥ ⎥, 21 ⎦ 24

5 6 7 8 9 10 11 12 17 18 19 20 21 22 23 24

 .

604

Tensor Analysis

(2) The LMV horizontal unfolding ⎡ 1 13 4 16 7 ALMV(1) = ⎣2 14 5 17 8 3 15 6 18 9 ⎡ 1 2 3 13 ⎢ 4 5 6 16 ALMV(2) = ⎢ ⎣ 7 8 9 19 10 11 12 22  1 4 7 10 ALMV(3) = 13 16 19 22 (3) The Kolda horizontal unfolding ⎡ 1 4 7 10 13 AKolda(1) = ⎣2 5 8 11 14 3 6 9 12 15 ⎡ 1 2 3 13 ⎢ 4 5 6 16 AKolda(2) = ⎢ ⎣ 7 8 9 19 10 11 12 22  1 2 3 4 AKolda(3) = 13 14 15 16

⎤ 19 10 22 20 11 23⎦ , 21 12 24 ⎤ 14 15 17 18 ⎥ ⎥, 20 21 ⎦ 23 24 2 5 8 11 3 6 9 12 14 17 20 23 15 18 21 24

16 19 17 20 18 21

(4) The Kiers longitudinal unfolding ⎤ ⎡ 1 2 3 ⎡ ⎢4 5 6⎥ 1 ⎥ ⎢ ⎢7 8 9⎥ ⎢13 ⎥ ⎢ ⎢ ⎥ ⎢ ⎢ ⎢10 11 12⎥ ⎢2 (1) (2) AKeirs = ⎢ ⎥, AKeirs = ⎢ ⎢13 14 15⎥ ⎢14 ⎥ ⎢ ⎢ ⎢16 17 18⎥ ⎣3 ⎥ ⎢ ⎣19 20 21⎦ 15 22 23 24 (5) The LMV longitudinal unfolding ⎤ ⎡ 1 2 3 ⎢ 13 14 15 ⎥ ⎥ ⎢ ⎢ 4 5 6 ⎥ ⎥ ⎢ ⎥ ⎢ 16 17 18 ⎥ ⎢ (1) ALMV = ⎢ ⎥, ⎢ 7 8 9 ⎥ ⎥ ⎢ ⎢ 19 20 21 ⎥ ⎥ ⎢ ⎣ 10 11 12 ⎦ 22 23 24

6 18

4 16 5 17 6 18

7 19

(2)

8 20

9 10 11 12 21 22 23 24



7 10 19 22⎥ ⎥ ⎥ 8 11⎥ ⎥, 20 23⎥ ⎥ 9 12⎦ 21 24



ALMV

.

⎤ 22 23⎦ , 24 ⎤

14 15 17 18 ⎥ ⎥, 20 21 ⎦ 23 24 5 17



⎢ ⎢ ⎢ ⎢ =⎢ ⎢ ⎢ ⎣



1 ⎢2 ⎢ ⎢3 ⎢ ⎢. (3) AKeirs = ⎢ ⎢ .. ⎢ ⎢10 ⎢ ⎣11 12

⎤ 1 4 7 10 2 5 8 11 ⎥ ⎥ ⎥ 3 6 9 12 ⎥ ⎥, 13 16 19 22 ⎥ ⎥ 14 17 20 23 ⎦ 15 18 21 24

 .

⎤ 13 14⎥ ⎥ 15⎥ ⎥ .. ⎥ ⎥ . ⎥. ⎥ 22⎥ ⎥ 23⎦ 24

10.2 Vectorization and Matricization of Tensors

605

and  (3)

ALMV =

1 13

4 16

7 10 19 22

2 14

5 17

8 11 20 23

(6) The Kolda longitudinal unfolding ⎤ ⎡ 1 2 3 ⎡ ⎢4 5 6⎥ 1 4 ⎥ ⎢ ⎥ ⎢7 ⎢2 8 9 5 ⎥ ⎢ ⎢ ⎥ ⎢ ⎢ 10 11 12 3 6 ⎥ ⎢ ⎢ (1) (2) AKolda = ⎢ ⎥, AKolda = ⎢ ⎢13 14 15⎥ ⎢13 16 ⎥ ⎢ ⎢ ⎢16 17 18⎥ ⎣14 17 ⎥ ⎢ ⎣19 20 21⎦ 15 18 22 23 24

3 15

6 18



7 10 8 11⎥ ⎥ ⎥ 9 12⎥ ⎥, 19 22⎥ ⎥ 20 23⎦ 21 24

9 12 21 24



T

1 ⎢2 ⎢ ⎢3 ⎢ ⎢. (3) AKolda = ⎢ ⎢ .. ⎢ ⎢10 ⎢ ⎣11 12

.

⎤ 13 14⎥ ⎥ 15⎥ ⎥ .. ⎥ ⎥ . ⎥. ⎥ 22⎥ ⎥ 23⎦ 24

From the above results, it can be seen that (a) the difference between the same mode-n horizontal unfoldings of the different methods is in the order in which the column vectors are arranged; (b) the difference between the same mode-n longitudinal unfoldings of the different methods is in the order in which the row vectors are arranged. The matricization of tensors brings convenience to tensor analysis, but it should be noted [65] that matricization may lead to the following problems: (1) (2) (3) (4)

the the the the

mathematical stability of the model is slightly worse; explanation of the model is slightly less clear; predictability of the model is slightly worse; number of the model parameters may be greater.

The matricization of tensors is an effective mathematical tool of tensor analysis. However, the ultimate goal of tensor analysis is sometimes to restore the original tensor from its matricization. The operation of reconstructing the original tensor from its vectorization or matricization is called tensorization or tensor reshaping or tensor reconstruction. Any horizontal unfolding matrices A(I×JK) , A(J×KI) or A(K×IJ) or any longitudinal unfolding matrices A(KI×J) , A(IJ×K) or A(JK×I) can be tensorized into the third-order tensor A ∈ KI×J×K . For example, for the Kiers matricization matrix, its tensorization entry is defined as (I×JK)

(J×KI)

(K×IJ)

(JK×I)

(KI×J)

(IJ×K)

aijk = Ai,(k−1)J+j = Aj,(i−1)K+k = Ak,(j−1)I+i = A(k−1)J+j,i = A(i−1)K+k,j = A(j−1)I+i,k ,

(10.2.19)

606

Tensor Analysis

and, for the LMV matricization matrix, (I×JK)

(J×KI)

(K×IJ)

(JK×I)

(KI×J)

(IJ×K)

aijk = Ai,(j−1)K+k = Aj,(k−1)I+i = Ak,(i−1)J+j = A(j−1)K+k,i = A(k−1)I+i,j = A(i−1)J+j,k .

(10.2.20)

The tensorization of the column vector a(IJK×1) or the row vector a(1×IJK) is defined as (IJK×1)

(1×IJK)

aijk = a(k−1)IJ+(j−1)I+i = a(i−1)JK+(j−1)K+k ,

(10.2.21)

where i = 1, . . . , I, j = 1, . . . , J, k = 1, . . . , K. More generally, an N th-order tensor can be reshaped or reconstructed using one of the mode-n horizontal unfolding formulas (10.2.4), (10.2.6) or (10.2.7).

10.3 Basic Algebraic Operations of Tensors The basic algebraic operations of tensors involve mainly the products of tensors, the product of a tensor and a matrix and the rank of a tensor.

10.3.1 Inner Product, Norm and Outer Product The inner product of two tensors is a generalization of the inner product of two vectors: first vectorize two tensor and then find the inner product of the two vectorized tensors to yield the inner product of the two tensors. DEFINITION 10.2 For two tensors A, B ∈ T (I1 , I2 , . . . , IN ), the tensor inner product A, B is a scalar and is defined as the inner product of the column vectorizations of the two tensors, i.e., def

A, B = vec(A), vec(B) = (vec(A))H vec(B) =

I1 I2   i1 =1 i2 =1

···

IN 

a∗i1 i2 ···in bi1 i2 ···in .

(10.3.1)

in =1

Using the concept of the inner product of tensors, we can directly introduce the definition of the Frobenius norm of a tensor. The Frobenius norm of a tensor A is defined as ⎞1/2 ⎛ I2 I1 IN     def ··· |ai1 i2 ···in |2 ⎠ .

A F = A, A = ⎝

DEFINITION 10.3

i1 =1 i2 =1

(10.3.2)

in =1

The outer product of two vectors is a matrix, e.g., X = uvT . The outer product of more than two vectors yields a tensor.

10.3 Basic Algebraic Operations of Tensors

607

DEFINITION 10.4 The outer product of n vectors a(m) ∈ Km×1 , m = 1, . . . , n, denoted a(1) ◦ · · · ◦ a(n) , is an nth-order tensor, namely A = a(1) ◦ a(2) ◦ · · · ◦ a(n) ,

(10.3.3)

whose entries are (1) (2)

(n)

ai1 i2 ···in = ai1 ai2 · · · ain , (m)

where aj

(10.3.4)

is the jth entry of the mode-m vector a(m) .

The outer product of two vectors u ∈ Km×1 and v ∈ Kn×1 is ⎤ ⎡ ⎡ ⎤ u1 v 1 · · · u1 v n u1 ⎢ ⎢ ⎥ .. ⎥ ∈ Km×n . .. X = u ◦ v = ⎣ ... ⎦ [v1 , . . . , vn ] = ⎣ ... . . ⎦

EXAMPLE 10.4

um v 1

um

···

um v n

The outer product of three vectors u ∈ KI×1 , v ∈ KJ×1 , w ∈ KK×1 is ⎤ ⎡ u1 v1 · · · u1 vJ ⎢ .. ⎥ ◦ [w , · · · , w ] ∈ KI×J×K , .. A = u ◦ v ◦ w = ⎣ ... . 1 K . ⎦ uI v 1 whose frontal slice matrix ⎡ u1 v 1 w k ⎢ .. A::k = ⎣ . uI v 1 w k

··· ··· .. . ···

uI v J ⎤ u1 v J w k ⎥ .. ⎦, .

k = 1, . . . , K.

uI v J w k

That is, aijk = ui vj wk , i = 1, . . . , I, j = 1, . . . , J, k = 1, . . . , K. The outer product of vectors is easily extended to the outer product of tensors. DEFINITION 10.5 Given tensors A ∈ KI1 ×···×IP and B ∈ KJ1 ×···×JQ , their tensor outer product is denoted A ◦ B ∈ KI1 ×···×IP ×J1 ×···×JQ , and is defined as (A ◦ B)i1 ···iP j1 ···jQ = ai1 ···iP bj1 ···jQ ,

∀ i1 , . . . , iP , j1 , . . . , jQ .

(10.3.5)

The inner product and the norm of tensors have the following properties [254]. 1. The norm of a tensor can be transformed to the norm of its matricization function: 2 2 2 2

A = 2A(In ×I1 ···In−1 In+1 ···IN ) 2 = 2A(I1 ···In−1 In+1 ···IN ×In ) 2. 2. The norm of a tensor can be transformed to the norm of its vectorization function: 2 2 2 2

A = 2a(I1 I2 ···IN ×1) 2 = 2a(1×I1 I2 ···IN ) 2.

608

Tensor Analysis

3. The square of the norm of difference of two tensors satisfies

A − B 2 = A 2 − 2A, B + B 2 . 4. If Q ∈ KJ×In is an orthonormal matrix, i.e., QQH = IJ×J or QH Q = IIn ×In , then the norm satisfies

A ×n Q = A , where ×n is the mode-n product, to be defined below. 5. Let A, B ∈ KI1 ×I2 ×···×IN , an , bn ∈ KJ×In , such that A = a1 ◦ a2 ◦ · · · ◦ aN and B = b1 ◦ b2 ◦ · · · ◦ bN ; then A, B =

N (

an , bn .

n=1

6. If A ∈ KI1 ×···×In−1 ×J×In+1 ×···×IN , B ∈ KI1 ×In−1 ×K×In+1 ×···×IN and C ∈ KJ×K then A, B ×n C = A ×n CH , B.

10.3.2 Mode-n Product of Tensors The product of a third-order tensor and a matrix was defined by Tucker [476], [477]. It is called the Tucker product of a third-order tensor. DEFINITION 10.6 Let X ∈ KI1 ×I2 ×I3 , A ∈ KJ1 ×I1 , B ∈ KJ2 ×I2 and C ∈ KJ3 ×I3 . The Tucker mode-1 product X ×1 A, the Tucker mode-2 product X ×2 B and the Tucker mode-3 product X ×3 C are respectively defined as [282] (X ×1 A)j1 i2 i3 =

I1 

xi1 i2 i3 aj1 i1 , ∀ j1 , i2 , i3 ,

(10.3.6)

xi1 i2 i3 bj2 i2 , ∀ i1 , j2 , i3 ,

(10.3.7)

xi1 i2 i3 cj3 i3 , ∀ i1 , i2 , j3 .

(10.3.8)

i1 =1

(X ×2 B)i1 j2 i3 =

I2  i2 =1

(X ×3 C)i1 i2 j3 =

I3  i3 =1

The following is an interpretation of the Tucker mode-n tensor product. First, the mode-1 tensor product can be represented in tensor symbol as Y = X ×1 A.

(10.3.9)

Second, from the entry formulas (10.2.8) for the mode-1 horizontal unfolding of the

10.3 Basic Algebraic Operations of Tensors

609

third-order tensor and from the longitudinal unfolding definition formula (10.2.14), we have respectively yj1 i2 i3 =

I1 

xi1 i2 i3 aj1 i1 =

i1 =1

)

= AX

(I1 ×I2 I3 )

I1 

(I1 ×I2 I3 ) aj1 i1 1 ,(i3 −1)I2 +i2

Xi

i1 =1

*

j1 ,(i3 −1)I2 +i2

,

and yj1 i2 i3 =

I1 

xi1 i2 i3 aj1 i1 =

i1 =1

* ) = X(I2 I3 ×I1 ) AT

I1 

(I I ×I )

3 1 X(i 2−1)I 3 +i 2

3 ,i1

aj1 i1

i1 =1 (i2 −1)I3 +i3 ,j1

. (J1 ×I2 I3 ) 1 ,(i3 −1)I2 +i2

Moreover, from (10.2.8) and (10.2.14) it is known that Yj (I I3 ×J1 ) Y(i 2−1)I 3 +i3 ,j1 2

= yj1 i2 i3 and

= yj1 i2 i3 . Therefore, the mode-1 product of the third-order tensor can be represented in matrix form as Y(J1 ×I2 I3 ) = (X ×1 A)(J1 ×I2 I3 ) = AX(I1 ×I2 I3 ) , Y(I2 I3 ×J1 ) = (X ×1 A)(I2 I3 ×J1 ) = X(I2 I3 ×I1 ) AT . Mimicking the mode-1 tensor product, we obtain matrix representations of the mode-2 and the mode-3 products below: + Y(J1 ×I2 I3 ) = AX(I1 ×I2 I3 ) , Y = X ×1 A ⇔ (10.3.10) Y(I2 I3 ×J1 ) = X(I2 I3 ×I1 ) AT . + Y(J2 ×I3 I1 ) = BX(I2 ×I3 I1 ) , Y = X ×2 B ⇔ (10.3.11) Y(I3 I1 ×J2 ) = X(I3 I1 ×I2 ) BT . + Y(J3 ×I1 I2 ) = CX(I3 ×I1 I2 ) , Y = X ×3 C ⇔ (10.3.12) Y(I1 I2 ×J3 ) = X(I1 I2 ×I3 ) CT . EXAMPLE 10.5 We revisit X ∈ R3×4×2 given by ⎡ 1 2 3 X::1 = ⎣5 6 7 9 10 11

again the two frontal slices of the third-order tensor ⎤ 4 8 ⎦, 12



X::2

13 = ⎣17 21

14 18 22

15 19 23

⎤ 16 20⎦ .

(10.3.13)

24

Now we will compute respectively the products of the tensor X and the following matrices: ⎡ ⎤   1 2 1 2 3 A= , B = ⎣3 4⎦ . 4 5 6 5 6

610

Tensor Analysis

Solution From (10.3.6), the two frontal slices of X ×1 A ∈ R2×4×2 are given by   38 44 50 56 (X ×1 A)::1 = , 83 98 113 138   110 116 122 128 . (X ×1 A)::2 = 263 278 293 308 Moreover, from (10.3.8) the three frontal slices of X ×3 B ∈ R3×4×3 are as follows: ⎡ ⎤ 27 30 33 36 (X ×3 B)::1 = ⎣39 42 45 48⎦ , 51 54 57 60 ⎡ ⎤ 55 62 69 76 (X ×3 B)::2 = ⎣ 83 90 97 104⎦ , 111 118 125 132 ⎡ ⎤ 83 94 105 116 (X ×3 B)::3 = ⎣127 138 149 160⎦ . 171 182 193 204 Figure 10.9 shows the operational principle of the three-mode product X ×1 u1 ×2 u2 ×3 u3 for the third-order tensor X ∈ R8×7×4 , where three vectors are u1 ∈ R8×1×1 , u2 ∈ R1×7×1 , u3 ∈ R1×1×4 . u1 u2

 ×1

×2

 ×3

u3

= (8 × 7 × 4)

(8 × 1 × 1)

(1 × 7 × 4)

(1 × 1 × 1)

(1 × 7 × 1) (1 × 1 × 4) (1 × 1 × 4)

Figure 10.9 The three-mode product of a third-order tensor.

From Figure 10.9 we can conclude that (a) each mode-vector-product reduces the tensor order by one, so that the tensor finally becomes a zero-order tensor or scalar; (b) the orders of the three mode vectors of a third-order tensor can be exchanged in the three-mode product. The Tucker product can be extended to the mode-n (matrix) product of an N th-order tensor. DEFINITION 10.7

The mode-n product of an N th-order tensor X ∈ KI1 ×···×IN

10.3 Basic Algebraic Operations of Tensors

611

and a Jn × In matrix U(n) is denoted X ×n U(n) . This is an I1 × · · · × In−1 × Jn × In+1 · · · × IN tensor whose entry is defined as [281] def

(X ×n U(n) )i1 ···in−1 jin+1 ···iN =

In 

xi1 i2 ···iN ajin ,

(10.3.14)

in =1

where j = 1, . . . , Jn , ik = 1, . . . , Ik , k = 1, . . . , N . From the above definition it is easy to seen that the mode-n product of an N thorder tensor X ∈ KI1 ×I2 ×···×IN and the identity matrix IIn ×In is equal to the original tensor, namely X ×n IIn ×In = X .

(10.3.15)

As for a third-order tensor, the mode-n product of an N th-order tensor X ∈ RI1 ×IN and a matrix U(n) ∈ RJn ×In can be represented in the mode-n matricization of the tensor as follows: Y = X ×n U(n)



Y(n) = U(n) X(n)

or

Y(n) = X(n) (U(n) )T

(10.3.16)

in which X(n) = X(In ×I1 ···In−1 In+1 ···IN ) and X(n) = X(I1 ···In−1 In+1 ···IN ×In ) are the mode-n horizontal and longitudinal unfoldings of the tensor X , respectively. Let X ∈ KI1 ×I2 ×···×IN be an N th-order tensor; then its mode-n product has the following properties [281]. 1. Given matrices A ∈ KJm ×Im , B ∈ KJn ×In , if m = n, then X ×m A ×n B = (X ×m A) ×n B = (X ×n B) ×m A = X ×n B ×m A. 2. Given matrices A ∈ KJ×In , B ∈ KIn ×J , then X ×n A ×n B = X ×n (BA). 3. If A ∈ KJ×In has a full column rank then Y = X × n A ⇒ X = Y × n A† . 4. If A ∈ KJ×In is semi-orthonormal, i.e., AH A = IIn , then Y = X ×n A



X = Y × n AH .

The above properties show that the tensor X can be reconstructed by the mode -n product Y = X ×n A via X = Y ×n A† or X = Y ×n AH . I1 ×I2 ×I3 slices of are EXAMPLE 10.6 Two frontal   tensor X ∈ R  a third-order 1 2 3 7 8 9 and X::2 = , and given by X::1 = 4 5 6 10 11 12 ⎡ ⎤ −0.7071 0.5774 A = ⎣ 0.0000 0.5774⎦ 0.7071 0.5774

612

Tensor Analysis

is a semi-orthonormal matrix, i.e., AT A = I2 . Find the frontal slices of the tensor X. Solution Since I1 = 2, I2 = 3, I3 = 2, J = 3, we obtain the mode-1 horizontal unfolding Y(J×I2 I3 ) = AX(I1 ×I2 I3 ) of Y = X ×1 A ∈ RJ×I2 ×I3 as follows: Y(J×I2 I3 ) ⎡ ⎤  −0.70711 0.57735  1 2 3 7 8 9 ⎣ ⎦ = 0.00000 0.57735 4 5 6 10 11 12 0.70711 0.57735 ⎤ ⎡ 1.60229 1.47253 1.34277 0.82373 0.69397 0.56421 = ⎣ 2.30940 2.88675 3.46410 5.77350 6.35085 6.92820 ⎦ . 3.01651 4.30097 5.58543 10.72327 12.00773 13.29219 Therefore, the mode-1 horizontal unfolding of the tensor X = Y ×1 AT can be retrieved as X(I1 ×I2 I3 ) = AT Y(J×I2 I3 ) , yielding the estimate ˆ (I1 ×I2 I3 ) X   −0.70711 0.00000 0.70711 = 0.57735 0.57735 0.57735 ⎡ ⎤ 1.60229 1.47253 1.34277 0.82373 0.69397 0.56421 × ⎣ 2.30940 2.88675 3.46410 5.77350 6.35085 6.92820 ⎦ 3.01651 4.30097 5.58543 10.72327 12.00773 13.29219   1.00000 2.00002 3.00003 7.00006 8.00007 9.00008 , = 4.00000 5.00000 5.99999 9.99999 10.99999 11.99999 and thus we have the following estimate for the frontal slices of the tensor as follows:   1.00000 2.00002 3.00003 ˆ X::1 = , 4.00000 5.00000 5.99999   ˆ ::2 = 7.00006 8.00007 9.00008 . X 9.99999 10.99999 11.99999

10.3.3 Rank of Tensor If the tensor A ∈ T (I1 , I2 , . . . , In ) can be decomposed as A = a(1) ◦ a(2) ◦ · · · ◦ a(n) ,

(10.3.17)

then it is said to be decomposed, and the vectors a(i) ∈ KIi , i = 1, . . . , n, are known as the components or factors of the decomposed tensor A. Since all the factors are rank-1 vectors, the above decomposition is called a rank-1 decomposition of the tensor.

10.3 Basic Algebraic Operations of Tensors

613

An entry of the decomposed tensor is defined as (1) (2)

(n)

ai1 i2 ···in = ai1 ai2 · · · ain .

(10.3.18)

The set of all I1 × I2 × · · · × In decomposed tensors is the decomposed tensor set, denoted D(I1 , I2 , . . . , In ) or simply D. LEMMA 10.1 [253] A=a

(1)

◦a

For two decomposed tensors A, B ∈ D, if (2)

◦ · · · ◦ a(n) ,

B = b(1) ◦ b(2) ◦ · · · ◦ b(n) ,

(10.3.19)

then the following results are true: • A, B = • A F =

n 

a(i) , b(i) ;

i=1 n 

n 

i=1

i=1

a(i) 2 and B F =

b(i) 2 ;

• A + B ∈ D if and only if A and B are different in only one component, all other components being the same. Let A F = 1 and B F = 1. Two decomposed tensors A and B are said to be orthogonal, denoted A ⊥ B, if their inner product is equal to zero, i.e., A, B =

n (

a(i) , b(i)  = 0.

i=1

In matrix algebra, the number of linearly independent rows (columns) of a matrix A is called the row (column) rank of A, equivalently stated as: the column (row) rank of A is the dimension of the column (row) space of A. The column rank and the row rank of a matrix are always equal. Therefore the rank, column rank and row rank are the same for a matrix. However, this important property of the matrix rank is no longer true for a higher-order tensor. In analogy to the the column rank and row rank of a matrix (the mode-1 and mode-2 vectors), the rank of the mode-n vectors of a tensor is called the mode-n rank of the tensor. DEFINITION 10.8 [280] Given an N th-order tensor A ∈ KI1 ×···×IN , among its I1 · · · In−1 In+1 · · · IN mode-n vectors with dimension In , the maximum number of linearly independent vectors is called the mode-n rank of the tensor A, denoted rn = rankn (A), equivalently stated as: the dimension of the subspace spanned by the mode-n vectors of a tensor is known as its mode-n rank. For example, the mode-n rank for n = 1, 2, 3 of a third-order tensor A ∈ RI×J×K are denoted rn (A), and are defined as   def r1 (A) = dim spanR {a:jk | j = 1, . . . , J, k = 1, . . . , K} ,   def r2 (A) = dim spanR {ai:k | i = 1, . . . , I, k = 1, . . . , K} ,   def r3 (A) = dim spanR {aij: | i = 1, . . . , I, j = 1, . . . , J} ,

614

Tensor Analysis

where a:jk , ai:k and aij: are the mode-1, mode-2 and mode-3 vectors of the tensor, respectively. The column rank and the row rank of a matrix are always the same, but the mode-i rank and the mode-j of a higher-order tensor are generally not the same if i = j. The third-order tensor A ∈ KI×J×K is said to be of rank (r1 , r2 , r3 ) if it has mode-1 rank r1 , mode-2 rank r2 and mode-3 rank r3 . More generally, an N th-order tensor A is of rank (r1 , r2 , . . . , rN ), if its model-n rank is equal to rn , n = 1, . . . , N . In particular, if the rank of every mode-n matricization is equal to 1 then the tensor is of the rank (1, 1, . . . , 1). There are N mode-n ranks of an N th-order tensor, which is inconvenient in use. For this purpose, it is necessary to define only one rank for a given tensor, the tensor rank. Consider a tensor A ∈ T that can be decomposed into the weighted sum of R decomposed tensors: R  A= σi U i , (10.3.20) i=1

where σi > 0, i = 1, . . . , R, Ui ∈ D and Ui F = 1, i = 1, . . . , R. From [267], the tensor rank of A is denoted rank(A), and is defined as the minimum value of R such that (10.3.20) holds. Then (10.3.20) is called the rank decomposition of the tensor. In particular, if R = 1, i.e., A = a(1) ◦ a(2) ◦ · · · ◦ a(n) , then A is said to be a rank-1 tensor. Therefore, a third-order tensor A is a rank-1 tensor, if it can be represented as the outer product of three vectors, i.e., A = a(1) ◦a(2) ◦a(3) . Similarly, a third-order tensor A is a rank-2 tensor (i.e., the rank of the tensor is 2) if it can be represented as A = a(1) ◦ a(2) ◦ a(3) + b(1) ◦ b(2) ◦ b(3) . 10.4 Tucker Decomposition of Tensors In order to carry out the information mining of a tensor, tensor decomposition is necessary. The concept of tensor decomposition stems from two papers of Hitchcock in 1927 [212], [213] who expressed a tensor as the sum of finite rank-1 tensors, and called it a canonical polyadic decomposition. The concept of multi-way models was proposed by Cattell in 1944 [88]. However, two aspects of tensor decomposition and multi-way models attracted people’s attention in succession in the 1960s. Tucker published his three papers on factor decomposition of the tensor [475]–[477]. Later, Carroll and Chang [87] and Harshman [198] independently proposed canonical factor decomposition (CANDECOMP) and parallel factor decomposition (PARAFAC), respectively. These studies brought into being two major categories of tensor decomposition methods:

10.4 Tucker Decomposition of Tensors

615

(1) Tucker decomposition, later known as higher-order SVD; (2) canonical factor decomposition (CANDECOMP/PARAFAC), now simply called CP decomposition. In this section we discuss the Tucker decomposition of tensors, while the CP decomposition of tensors will be considered in the next section.

10.4.1 Tucker Decomposition (Higher-Order SVD) Tucker decomposition is related closely to the Tucker operator, while the Tucker operator is in turn an effective representation of the multimode product of a tensor and a matrix. DEFINITION 10.9 Given an N th-order tensor G ∈ KJ1 ×J2 ×···×JN and matrices U(n) ∈ KIn ×Jn , where n ∈ {1, . . . , N }, the Tucker operator is defined as [254] def

[[G; U(1) , U(2) , . . . , U(N ) ]] = G ×1 U(1) ×2 U(2) ×3 · · · ×N U(N ) .

(10.4.1)

The result is an N th-order I1 × I2 × · · · × IN tensor. Given an N th-order tensor G ∈ KJ1 ×···×JN and an index set N = {1, . . . , N }, then the Tucker operator has the following properties [254]: 1. If matrices U(n) ∈ KIn ×Jn , V ∈ KKn ×In for all n ∈ N , then   [[G; U(1) , . . . , U(N ) ]]; V(1) , . . . , V(N ) =[[G; V(1) U(1) , . . . , V(N ) U(N ) ]]. 2. If U(n) ∈ KIn ×Jn , n ∈ N has full-column rank then X = [[G; U(1) , . . . , U(N ) ]]



G = [[X ; U(1)† , . . . , U(N )† ]].

3. If U(n) ∈ KIn ×Jn is an orthonormal matrix such that U(n)T U(n) = IJn , then X = [[G; U(1) , . . . , U(N ) ]]



G = [[X ; U(1)T , . . . , U(N )T ]].

PROPOSITION 10.1 [254] Given a tensor G ∈ RJ1 ×J2 ×···×JN , let N = {1, . . . , N } and the matrices U(n) ∈ RIn ×Jn , n ∈ N . If every matrix has the QR decomposition U(n) = Qn Rn , ∀ n ∈ N , where Qn is an orthonormal matrix and Rn is an upper triangular matrix, then 2 2 2 2 2[[G; U(1) , . . . , U(n) ]]2 = 2[[X ; R1 , . . . , Rn ]]2. The matrix A ∈ RI1 ×I2 is a two-mode mathematical object: it has two companion vector spaces, column space and row space. The SVD makes these vector spaces orthogonal and decomposes A into the product of three matrices A = U1 ΣUT2 , where the J1 left singular vectors of U1 ∈ RI1 ×J1 span the column space of A, and the J2 right singular vectors of the U2 ∈ RI2 ×J2 span the row space of A. Because the role of the diagonal singular values is usually more important than

616

Tensor Analysis

the left and right singular vectors, the diagonal singular value matrix Σ is viewed as the core matrix of A. If the diagonal singular value matrix Σ is regarded as a second-order tensor then Σ is naturally the core tensor of the second-order tensor A, while the product of three matrices A = U1 ΣUT2 in the SVD of the matrix A can be rewritten as the mode-n product of the tensor A = Σ ×1 U1 ×2 U2 . This mode-n product of the SVD of a matrix is easily extended to the SVD of an N th-order tensor or an N -dimensional supermatrix A ∈ CI1 ×···×IN . THEOREM 10.1 (N th-order SVD) [281] Every I1 × I2 × · · · × IN real tensor X can be decomposed into the mode-n product X = G ×1 U(1) ×2 U(2) ×3 · · · ×N U(N ) = [[G; U(1) , U(2) , . . . , U(N ) ]]

(10.4.2)

with entries xi1 i2 ···iN =

J1 J2  

···

j1 =1 j2 =1 (n)

JN 

(1)

(2)

(N ) , N jN

gi1 i2 ···iN ui1 j1 ui2 j2 · · · ui

(10.4.3)

jN =1

(n)

where U(n) = [u1 , . . . , uJn ] is an In × Jn semi-orthogonal matrix such that (U(n) )T U(n) = IJn , with Jn ≤ In , the core tensor G is a J1 × J2 × · · · × JN tensor and the subtensor Gjn =α is the tensor X with the fixed index jn = α. The subtensors have the following properties. • All-orthogonality: two subtensors Gjn =α and Gjn =β , for α = β, are orthogonal: Gjn =α , Gjn =β  = 0,

∀ α = β, n = 1, . . . , N.

(10.4.4)

• Ordering:

Gin =1 F ≥ Gin =2 F ≥ · · · ≥ Gin =N F .

(10.4.5)

Unlike the diagonal singular value matrix, the core tensor G does not have a diagonal construction, i.e., its off-diagonal entries are not equal to zero [253]. Thus the entries gj1 ···jN of the core tensor G = [[gj1 ···jN ]] guarantee the interaction of the mode matrices U(n) , n = 1, . . . , N . The mode-n matrix U(n) is required to have an orthogonal column construction, similarly to the left singular matrix U and the right singular matrix V of SVD, i.e., any two columns of U(n) are mutually orthogonal. Because Equation (10.4.2) is a decomposition form of SVD in a higher-order tensor, (10.4.2) is naturally called the higher-order singular value decomposition of tensor [281]. As early as the 1960s, Tucker proposed the factor decomposition of third-order tensors [475], [476], [477], now commonly called the Tucker decomposition. In comparison with the Tucker decomposition, higher-order SVD more closely reflects the connection to and generalization of the SVD of a matrix. However, Tucker decomposition and higher-order SVD are often used without any distinction being drawn between them.

10.4 Tucker Decomposition of Tensors

617

PROPOSITION 10.2 There exists the following transform relation between the higher-order SVD of an N th-order tensor X ∈ CI1 ×···×IN and its core tensor: X = G ×1 U(1) ×2 U(2) · · · ×N U(N ) ⇒

G = X ×1 U(1)T ×2 U(2)T · · · ×N U(N )T ,

(10.4.6)

where G ∈ CJ1 ×···JN , U(n) ∈ CIn ×Jn and Jn ≤ In . Proof Continuously applying the property of the mode-n product of a tensor and a matrix X ×m A ×n B = X ×n B ×m A, it can be seen that Equation (10.4.2) can be equivalently written as X = G ×N U(N ) ×N −1 U(N −1) · · · ×1 U(1) . Use the property of the mode-n product X ×n A ×n B = X ×n (BA) and notice that (U(1) )T U(1) = IJ1 ; then X ×1 (U(1) )T = G ×N U(N ) ×N −1 U(N −1) · · · ×1 U(1) ×1 (U(1) )T = G ×N U(N ) ×N −1 U(N −1) · · · ×2 U(2) ×1 ((U(1) )T U(1) ) = G ×N U(N ) ×N −1 U(N −1) · · · ×2 U(2) . Mimicking this process, we have also X ×1 (U(1) )T ×2 (U(2) )T = G ×N U(N ) ×N −1 U(N −1) · · ·×3 U(3) . Finally, we have G = X ×1 (U(1) )T ×2 (U(2) )T · · ·×N (U(N ) )T . Define (n)

U⊗ = (U(n−1) ⊗ · · · ⊗ U(1) ⊗ U(N ) ⊗ U(n+1) )T

(Kiers),

(10.4.7)

U⊗ = (U(n+1) ⊗ · · · ⊗ U(N ) ⊗ U(1) ⊗ U(n−1) )T

(LMV),

(10.4.8)

(n) U⊗

(Kolda).

(10.4.9)

(n)

= (U

(N )

⊗ ··· ⊗ U

(n+1)

⊗U

(n−1)

⊗U

(1) T

)

Then the equivalent matrix representation of the higher-order SVD is given by (n)

X(n) = U(n) G(n) U⊗ , X

(n)

=

(n) (U⊗ )T G(n) (U(n) )T ,

(10.4.10) (10.4.11)

(n)

where X(n) and X are horizontal and longitudinal unfoldings, respectively. When using the above matrix equivalent representation, two basic principles should be followed: there is no factor matrix U(n) , n ≤ 0, and there are no factor matrices with the same subscript. Table 10.1 summarizes a variety of mathematical representation of Tucker decompositions.

10.4.2 Third-Order SVD The Tucker decomposition (third-order SVD) of a third-order tensor X = G ×1 A ×2 B ×3 C has the following properties [282].

∈ KI×J×K ,

(10.4.12)

618

Tensor Analysis

Table 10.1 Mathematical representations of the Tucker decomposition Mathematical formula Operator form X = [[G; U(1) , U(2) , . . . , U(N ) ] Mode-N form

X = G ×1 U(1) ×2 U(2) · · · ×N U(N )

Entry form

xi1 ···iN =

Outer product X =  Kiers  LMV  Kolda

J2 J1  

J

j1 =1 j2 =1

J2 J1   j1 =1 j2 =1

N 

··· j

N

=1

(1) (2) u 1 ,j1 i2 ,j2

gj1 j2 ···jN ui

(N ) ,j N N

· · · ui

J

N 

··· j

N

=1

(1)

(2)

(N )

1

2

N

gj1 j2 ···jN uj ◦ uj ◦ · · · ◦ uj

X(n) = U(n) G(n) (U(n−1) ⊗ · · · ⊗ U(1) ⊗ U(N ) ⊗ · · · ⊗ U(n+1) )T X(n) = (U(n−1) ⊗ · · · ⊗ U(1) ⊗ U(N ) ⊗ · · · ⊗ U(n+1) )G(n) U(n)T X(n) = U(n) G(n) (U(n+1) ⊗ · · · ⊗ U(N ) ⊗ U(1) · · · ⊗ U(n−1) )T X(n) = (U(n+1) ⊗ · · · ⊗ U(N ) ⊗ U(1) · · · ⊗ U(n−1) )G(n) U(n)T X(n) = U(n) G(n) (U(N ) ⊗ · · · ⊗ U(n+1) ⊗ U(n−1) ⊗ · · · ⊗ U(1) )T X(n) = (U(N ) ⊗ · · · ⊗ U(n+1) ⊗ U(n−1) ⊗ · · · ⊗ U(1) )G(n) U(n)T

1. The mode-1 matrix A ∈ KI×P , the mode-2 matrix B ∈ KJ×Q and the mode-3 matrix C ∈ KK×R are all columnwise orthonormal, i.e., AH A = I P ,

BH B = I Q ,

CH C = IR .

2. The longitudinal unfoldings G(QR×P ) , G(RP ×Q) and G(P Q×R) of the third-order core tensor G ∈ KP ×Q×R are respectively columnwise orthogonal: (QR×P ) ) G:p , G(QR×P  = σ12 (p1 )δp1 ,p2 , :p2 1

1 ≤ p1 , p2 ≤ P,

×Q) ×Q) G(RP , G(RP  = σ22 (q1 )δq1 ,q2 , :q1 :q2

1 ≤ q1 , q2 ≤ Q,

(P Q×R) Q×R) , G(P  = σ32 (r1 )δr1 ,r2 , G:r :r2 1

1 ≤ r1 , r2 ≤ R.

Here the singular values are arranged as follows: σ12 (1) ≥ σ12 (2) ≥ · · · ≥ σ12 (P ), σ22 (1) ≥ σ22 (2) ≥ · · · ≥ σ22 (Q), σ32 (1) ≥ σ32 (2) ≥ · · · ≥ σ32 (R). The SVD of the third-order tensor X = [[xijk ]] ∈ RI×J×K has a total of three subscript variables i, j, k, and thus has P3 = 3! = 6 possible horizontal unfolding

10.4 Tucker Decomposition of Tensors

619

matrices, which are given by (I×JK)

xi,(k−1)J+j = xijk



X(I×JK) = [X::1 , . . . , X::K ] = AG(P ×QR) (C ⊗ B)T , (J×KI) xj,(i−1)K+k

= xijk



X(J×KI) = [X1:: , . . . , XI:: ] = BG(Q×RP ) (A ⊗ C)T , (K×IJ) xk,(j−1)I+i

X

(K×IJ)

= xijk

(I×JK)

(10.4.14)



= [X:1: , . . . , X:J: ] = CG(R×P Q) (B ⊗ A)T ,

xi,(j−1)K+k = xijk

(10.4.13)

(10.4.15)

⇒ = AG(P ×QR) (B ⊗ C)T ,

(10.4.16)

X(J×KI) = [XT::1 , . . . , XT::K ] = BG(Q×RP ) (C ⊗ A)T ,

(10.4.17)

X

(I×JK)

(J×KI) xj,(k−1)I+i

(K×IJ) xk,(i−1)J+j

=

[XT:1: , . . . , XT:J: ]

= xijk

= xijk

⇒ ⇒

X(K×IJ) = [XT1:: , . . . , XTI:: ] = CG(R×P Q) (A ⊗ B)T ;

(10.4.18)

and the following six possible longitudinal unfolding matrix representations: (JK×I)



X(JK×I) = (C ⊗ B)G(QR×P ) AT ,

(10.4.19)



X(KI×J) = (A ⊗ C)G(RP ×Q) BT ,

(10.4.20)



X(IJ×K) = (B ⊗ A)G(P Q×R) CT ,

(10.4.21)



X(JK×I) = (B ⊗ C)G(QR×P ) AT ,

(10.4.22)

(KI×J)



X(KI×J) = (C ⊗ A)G(RP ×Q) BT ,

(10.4.23)

(IJ×K)



X(IJ×K) = (A ⊗ B)G(P Q×R) CT .

(10.4.24)

x(k−1)J+j,i = xijk (KI×J)

x(i−1)K+k,j = xijk (IJ×K)

x(j−1)I+i,k = xijk (JK×I)

x(j−1)K+k,i = xijk x(k−1)I+i,j = xijk x(i−1)J+j,k = xijk

Here are three equivalent matricization representation methods for the thirdorder SVD. (1) The third-order SVD matricization representations (Kiers method) [245], [491]: ⎧ (I×JK) (I×JK) (P ×QR) ⎪ x = xijk ⇒ XKiers = AGKiers (C ⊗ B)T, ⎪ ⎪ ⎨ i,(k−1)J+j (J×KI) (J×KI) (Q×RP ) horizontal xj,(i−1)K+k = xijk ⇒ XKiers = BGKiers (A ⊗ C)T, ⎪ ⎪ ⎪ (K×IJ) (R×P Q) ⎩ x(K×IJ) = x ⇒ XKiers = CGKiers (B ⊗ A)T. ijk k,(j−1)I+i ⎧ (JK×I) (JK×I) (QR×P ) T ⎪ ⎪ x(k−1)J+j,i = xijk ⇒ XKiers = (C ⊗ B)GKiers A , ⎪ ⎨ (KI×J) (KI×J) (RP ×Q) longitudinal x(i−1)K+k,j = xijk ⇒ XKiers = (A ⊗ C)GKiers BT, ⎪ ⎪ ⎪ (IJ×K) (P Q×R) ⎩ x(IJ×K) = x ⇒ XKiers = (B ⊗ A)GKiers CT. ijk (j−1)I+i,k

620

Tensor Analysis

(2) The third-order SVD matricization representations (LMV method) [281], [282]: ⎧ (I×JK) (I×JK) (P ×QR) ⎪ xi,(j−1)K+k = xijk ⇒ XLMV = AGLMV (B ⊗ C)T, ⎪ ⎪ ⎨ (J×KI) (J×KI) (Q×RP ) horizontal xj,(k−1)I+i = xijk ⇒ XLMV = BGLMV (C ⊗ A)T, ⎪ ⎪ ⎪ (K×IJ) (R×P Q) ⎩ x(K×IJ) ⇒ XLMV = CGLMV (A ⊗ B)T. k,(i−1)J+j = xijk ⎧ (JK×I) (JK×I) (QR×P ) T ⎪ ⎪ x(j−1)K+k,i = xijk ⇒ XLMV = (B ⊗ C)GLMV A , ⎪ ⎨ (KI×J) (KI×J) (RP ×Q) longitudinal x(k−1)I+i,j = xijk ⇒ XLMV = (C ⊗ A)GLMV BT, ⎪ ⎪ ⎪ (IJ×K) (P Q×R) ⎩ x(K×IJ) ⇒ XLMV = (A ⊗ B)GLMV CT. (i−1)J+j,k = xijk (3) The third-order SVD matricization [164], [254], [7]: ⎧ (I×JK) ⎪ = xijk x ⎪ ⎪ ⎨ i,(k−1)J+j (J×KI) horizontal xj,(k−1)I+i = xijk ⎪ ⎪ ⎪ ⎩ x(K×IJ) = x ijk k,(j−1)I+i ⎧ (JK×I) ⎪ = xijk x ⎪ ⎪ ⎨ (k−1)J+j,i (KI×J) longitudinal x(k−1)I+i,j = xijk ⎪ ⎪ ⎪ ⎩ x(IJ×K) = x ijk (j−1)I+i,k (J×KI)

Below we give a proof of XKiers be similarly proved. Proof form

representations (Kolda method) [65], [245], (P ×QR)

(C ⊗ B)T,

(Q×RP )

(C ⊗ A)T,

(R×P Q)

(B ⊗ A)T.

(I×JK)

= AGKolda

(J×KI)

= BGKolda

(K×IJ)

= CGKolda

(JK×I)

= (C ⊗ B)GKolda

(KI×J)

= (C ⊗ A)GKolda

(IJ×K)

= (B ⊗ A)GKolda



XKolda



XKolda



XKolda



XKolda



XKolda



XKolda (Q×RP )

= BGKiers

(QR×P )

AT,

(RP ×Q)

BT,

(P Q×R)

CT.

(A ⊗ C)T . The other forms can

First, the Tucker decomposition can be written in the horizontal unfolding

Xi:: =

Q  R P  

gpqr aip B:q CT:r

p=1 q=1 r=1

=

Q  R P  

(Q×RP )

Gq,(p−1)R+r aip bq cTr

p=1 q=1 r=1



⎤ (Q×RP ) G1,(p−1)R+r ⎢ ⎥ .. ⎥ aip cTr = [b1 , . . . , bQ ] ⎢ . ⎣ ⎦ p=1 r=1 (Q×RP ) GQ,(p−1)R+r R P  

=

R P   p=1 r=1

(Q×RP )

BG:(p−1)R+r aip cTr .

10.4 Tucker Decomposition of Tensors

621

Expand the above equation to get ⎤ ai1 cT1 ⎢ . ⎥ ⎢ .. ⎥ ⎥ ⎢ ⎢ a cT ⎥ i1 R ⎥ =⎢ < ⎥ (Q×RP ) (Q×RP ) (Q×RP ) (Q×RP ) ⎢ ⎢ .. ⎥ , = B G:1 , . . . , G:R , . . . , G:(P −1)R+1 , . . . , G:RP ⎢ . ⎥ ⎥ ⎢ ⎢aiP cT1 ⎥ ⎥ ⎢ ⎢ .. ⎥ ⎣ . ⎦ aiP cTR ⎡

Xi::

(J×KI)

where a is a row vector and c is a column vector. Hence, from XKiers = [X1:: , . . . , XI:: ] we have ⎡ ⎤ ⎡ T T⎤ a11 cT1 · · · aI1 cT1 a1 c1 ⎢ . ⎥ ⎢ . ⎥ . . .. .. ⎥ ⎢ .. ⎢ .. ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ a cT · · · a cT ⎥ ⎢aT cT ⎥ ⎢ 11 R ⎢ 1 R⎥ I1 R ⎥ ⎥ (J×KI) (Q×RP ) ⎢ (Q×RP ) ⎢ . .. ⎥ .. ⎥ ⎢ . ⎥ XKiers = BGKiers ⎢ . . ⎥ = BGKiers ⎢ .. ⎥ ⎢ .. ⎢ ⎥ ⎢ T T⎥ ⎢a1P cT1 · · · aIP cT1 ⎥ ⎢ aP c 1 ⎥ ⎢ ⎥ ⎢ ⎥ .. ⎥ ⎢ .. ⎢ .. ⎥ .. ⎣ . ⎣ . ⎦ . . ⎦ a1P cTR · · · aIP cTR aTP cTR (Q×RP )

= BGKiers

(A ⊗ C)T .



The above Tucker decomposition form is called the Tucker3 decomposition and has the following two simplified forms: Tucker2: xijk =

Q P  

gpqk aip bjq + eijk ,

(10.4.25)

p=1 q=1

Tucker1: xijk =

P 

gpjk aip + eijk .

(10.4.26)

p=1

As compared with the Tucker3 decomposition, we have C = IK and G ∈ RP ×Q×K in the Tucker2 decomposition; and B = IJ , C = IK and G ∈ RP ×J×K in the Tucker1 decomposition.

10.4.3 Alternating Least Squares Algorithms The Tucker or CP decomposition of an N th-order tensor can be written as the unified mathematical model X = f (U(1) , U(2) , . . . , U(N ) ) + E,

(10.4.27)

622

Tensor Analysis

where U(n) , n = 1, . . . , N , are the decomposed factor or component matrices, and E is an N th-order noise or error tensor. Hence, the factor matrices can be found by solving the following optimization problem: 2 2 ˆ (N ) ) = arg min 2X − f (U(1) , . . . , U(N ) )22 . ˆ (1) , . . . , U (10.4.28) (U F U(1) ,...,U(N )

This is an N -argument coupled optimization problem. As stated in Chapter 6, an effective method for solving this class of coupled optimization problems is the alternating least squares (ALS) algorithm. The basic idea of the ALS algorithm for Tucker decomposition is: fixing the (1) (i−1) factor matrices Uk+1 , . . . , Uk+1 as they were updated at the (k + 1)th iteration (i+1)

(N )

, . . . , Uk as they were updated at the kth iteration, find the LS solution and Uk of the factor matrix U(i) at the (k + 1)th iteration: 2  2 ˆ (i) = arg min 2X − f U(1) , . . . , U(i−1) , U(i) , U(i+1) , . . . , U(N ) 22 , (10.4.29) U k+1 k+1 k+1 k k F U(i)

where i = 1, . . . , N . For k = 1, 2, . . ., apply LS methods alternately until all the factor matrices converge. As an example, we discuss the solution of the following optimization problem for a Tucker decomposition: 2 (I×JK) 22 2X min − AG(P ×QR) (C ⊗ B)T 2F . (10.4.30) A,B,C,G(P ×QR)

By the principle of the ALS, if we assume that the mode-2 matrix B, the horizontal unfoldings of the mode-3 matrix C and the core tensor G are fixed then the above optimization problem can be decoupled as an optimization problem containing only the mode-1 matrix A: 2 22 min 2X(I×JK) − AG(P ×QR) (C ⊗ B)T 2F . A

This corresponds to finding an LS solution A for X(I×JK) = AG(P ×QR) (C ⊗ B)T . Postmultiplying both sides of the matrix equation by (C ⊗ B), we get X(I×JK) (C ⊗ B) = AG(P ×QR) (C ⊗ B)T (C ⊗ B).

(10.4.31)

Make the SVD X(I×JK) (C ⊗ B) = U1 S1 V1T , and take the P principal left sinˆ = U1 (:, 1 : P ). This operation gular vectors as an estimate of the matrix A, i.e., A (I×JK) can be simply expressed as [A, S, T] = SVD[X (C ⊗ B), P ]. Similarly, we can respectively find the estimates of B and C. Then, fixing the ˆ and C, ˆ we can re-solve A and next recalculate B and estimated factor matrices B C until all three factor matrices A, B, C converge. When all the factor matrices converge and satisfy the orthogonality conditions AT A = IP , BT B = IQ and CT C = IR , we can premultiply both sides of (10.4.31) by the factor matrix AT : G(P ×QR) = AT X(I×JK) (C ⊗ B),

(10.4.32)

10.4 Tucker Decomposition of Tensors

623

since AT A = IP and (CT ⊗ BT )(C ⊗ B) = (CT C) ⊗ (BT B) = IR ⊗ IQ = IQR . Similarly, we can find the other two horizontal unfolding matrices G(Q×RP ) and (R×P Q) of the core tensor, and hence get the core tensor itself. Thus one has an G ALS algorithm for the Tucker decomposition, as shown in Algorithm 10.1. Algorithm 10.1

ALS Algorithm for Tucker decomposition [7]

input: Third-order tensor X . initialize: Choose factor matrices B0 , C0 , and put k = 0. repeat (I×JK)

1. Use xi,(k−1)J+j = xijk to form the horizontal unfolding matrix X(I×JK) . 2. Compute the truncated SVD of X(I×JK) (Ck ⊗ Bk ) [Ak+1 , S1 , T1 ] = SVD[X(I×JK) (Ck ⊗ Bk ), P ]. 3. Compute the truncated SVD of X(J×KI) (Ak+1 ⊗ Ck ) [Bk+1 , S2 , T2 ] = SVD[X(J×KI) (Ak+1 ⊗ Ck ), Q]. 4. Compute the truncated SVD of X(K×IJ) (Bk+1 ⊗ Ak+1 ) [Ck+1 , S3 , T3 ] = SVD[X(K×IJ) (Bk+1 ⊗ Ak+1 ), R]. 5. exit if Ak+1 − Ak F ≤ , Bk+1 − Bk F ≤ , Ck+1 − Ck F ≤ . return k ← k + 1. 6. Compute G(P ×QR) = AT X(I×JK) (C ⊗ B). 7. Compute G(Q×RP ) = BT X(J×KI) (A ⊗ C). 8. Compute G(R×P Q) = CT X(K×IJ) (B ⊗ A). output: Factors A ∈ RI×P , B ∈ RJ×Q , C ∈ RK×R ; core tensor G ∈ RP ×Q×R .

Algorithm 10.1 is also available for the Kiers horizontal unfolding of a third-order tensor, and is easily extended to other matricization cases. For example, if the Kiers horizontal unfolding matrices are replaced by the Kolda horizontal unfolding matrices (I×JK)

xi,(k−1)J+j = xijk ,

(J×KI)

xj,(k−1)I+i = xijk ,

(K×IJ)

xk,(j−1)I+i = xijk ,

and correspondingly [A, S, V] = SVD(X(I×JK) (C ⊗ B), P ), [B, S, V] = SVD(X(J×KI) (C ⊗ A), Q), [C, S, V] = SVD(X(K×IJ) (B ⊗ A), R), then Algorithm 10.1 becomes the ALS algorithm for the Tucker decomposition proposed by Bro in 1998 [65], where P, Q, R are the number of principal singular values of the corresponding matrices. Therefore, the equivalent matrix forms of the higher-order SVD must correspond strictly to the matricization method of the tensor. Consider the Tucker3 decomposition or the higher-order SVD of an N th-order

624

Tensor Analysis

tensor X : min

U(1) ,...,U(N ) ,G(n)

2 22 2X(n) − U(n) G(n) U(n) 2 , ⊗ F

(10.4.33)

(n)

where the Kronecker product U⊗ is defined by (10.4.7) or (10.4.8) or (10.4.9). In the higher-order SVD, the factor matrix U(n) ∈ RIn ×Jn satisfies the semiorthogonality condition (U(n) )T U(n) = IJn , so we have (n)

(n)

U⊗ (U⊗ )T = IJ1 ···Jn−1 Jn+1 ···JN .

(10.4.34)

(n) In order to solve the matrix equation X(n) ≈ U G(n) U⊗ , let SVD X(n) = (n) U(n) S(n) (V(n) )T , then U(n) S(n) (V(n) )T = U(n) G(n) U⊗ . Premultiplying both (n) sides by (U(n) )T and postmultiplying by (U⊗ )T , then from (10.4.34) we have (n) G(n) = S(n) (V(n) )T (U⊗ )T . (10.4.35) (n)

Algorithm 10.2 is a higher-order SVD (HOSVD) algorithm, while Algorithm 10.3 is a higher-order orthogonal iteration (HOOI) algorithm. Algorithm 10.2

HOSVD(X , R1 , . . . , RN ) [257]

input: N th-order tensor X . initialize: The horizontal unfolding matrices X(n) , n = 1, . . . , N . for n = 1, . . . , N do 1. Compute the SVD X(n) = Uk Σk VkT . 2. Determine the effective rank Rn of X(n) . 3. Update U(n) ← Uk (:, 1 : Rn ). end for 4. Calculate G = X ×1 U(1)T ×2 U(2)T ×3 · · · ×N U(N )T . output: Core tensor G and factor matrices U(1) , . . . , U(N ) . Algorithm 10.3

HOOI(X , R1 , . . . , RN ) [257]

input: N th-order tensor X . initialize: The horizontal unfolding matrices X(n) , n = 1, . . . , N . for n = 1, . . . , N do 1. Compute the SVD X(n) = Uk Σk VkT . 2. Determine the effective rank Rn of X(n) . 3. Update U(n) ← Uk (:, 1 : Rn ) and let k = 1. end for repeat for n = 1, . . . , n do Y ← X ×1 U(1)T · · · ×n−1 U(n−1) ×n+1 U(n+1)T ×n+2 · · · ×N U(N ) . U(n) ← Rn leading left singular vectors of Y(n) . end for 4. Gk = X ×1 U(1)T ×2 U(2)T ×3 · · · ×N U(N )T . 5. exit if Gk+1 − Gk F ≤ , k > 1 or maximum iterations exhausted. return k ← k + 1. output: Core tensor G and factor matrices U(1) , . . . , U(N ) .

10.5 Parallel Factor Decomposition of Tensors

625

The HOOI algorithm is the more effective technique for computing the factor matrices and the core tensor. The feature of HOOI is to compute the leading singular vectors of the horizontal unfolding matrix X(n) even if they just form an orthonormal basis of its dominant subspace.

10.5 Parallel Factor Decomposition of Tensors As mentioned early, canonical decomposition (CANDECOMP) and parallel factor (PARAFAC) decomposition are two data analysis methods for multi-way data, proposed independently by Carroll and Chang in 1970 [87] and Harshman in 1970 [198] on the basis of the principle of parallel proportional profiles (PP) proposed by Cattell in 1944 [88]. PARAFAC can be seen as a generalization of two-way factor analysis to multiway data. The CANDECOMP/PARAFAC decomposition is usually simply called the CP decomposition.

10.5.1 Bilinear Model Tensor analysis is essentially multilinear analysis, which in turn is a generalization of bilinear analysis; principal component analysis (PCA) and independent component analysis (ICA) are two typical bilinear analysis methods. Via a bilinear or multilinear model, the factors (or components, or loadings) reflecting the linear combination of variables are extracted. These extracted factors are then used to explain the basic information content of the data. In data analysis, given a two-way data matrix X ∈ KI×J , two-way bilinear analysis uses the model R  xij = air bjr + eij (10.5.1) r=1

for data fitting, where the meaning of each parameter is as follows: xij is the entry of the ith row and the jth column of the data matrix X; R is the number of factors; air is the “factor loading”; bjr is the “factor score”; eij is the measurement error of the data xij and is the entry of the ith row and jth column of the I × J error matrix E. R If bjr = βr is fixed at a constant value, then xij = r=1 βr air is a linear model for the factor loadings air . Conversely, if air = αr is fixed at a constant value, then R xij = r=1 αr bjr is a linear model for the factor scores bjr . Therefore, the two-way data model (10.5.1) is often called a two-way bilinear model.

626

Tensor Analysis

Define respectively the factor loading vector and the factor score vector as ar = [a1r , . . . , aIr ]T ,

br = [b1r , . . . , bJr ]T ;

then the two-way bilinear model can be rewritten in matrix form as X=

R  r=1



ar ◦ b r =

R 

ar bTr

(10.5.2)

r=1

a11 b11 + · · · + a1R b1R ⎢ .. =⎣ . aI1 b11 + · · · + aIR b1R

··· .. . ···

⎤ a11 bJ1 + · · · + a1R bJR ⎥ .. ⎦. .

(10.5.3)

aI1 bJ1 + · · · + aIR bJR

The above equation can be equivalently given as X = ABT =

R 

ar ◦ b r ,

(10.5.4)

⎤ a1R .. ⎥ ∈ KI×R , . ⎦

(10.5.5)

r=1

where



a11 ⎢ .. A=⎣ . ⎡

aI1

b11 ⎢ .. B=⎣ . bJ1

··· .. . ··· ··· .. . ···

aIR

⎤ b1R .. ⎥ ∈ KJ×R . ⎦

(10.5.6)

bJR

are the factor loading matrix and the factor score matrix, respectively. Data analysis emphasizes the uniqueness of a structured model. That a structured model is unique means that the model identification requires no further constraint conditions. An unconstrained two-way bilinear model has no uniqueness: from (10.5.4) it is easy to see that in the two-way bilinear model there exists infinite rotational freedom. This is so because X = AQ(BQ)T = ABT

or

X = AQT (BQT )T = ABT.

That is, given only the two-way data matrix X ∈ RI×J , there are infinitely many sets of solutions (A, B) satisfying the two-way bilinear model. Hence, in order to keep the uniqueness of two-way bilinear model fitting, some constraints must be imposed on the factor matrices. The PCA is a two-way bilinear analysis method with an orthogonality constraint on factor matrices. Assume that the data matrix X ∈ RI×J has R leading singular values; then the PCA uses the truncated SVD X = U1 Σ1 V1T as the two-way data model, where Σ1 is a R × R diagonal matrix whose diagonal entries are the R leading singular values while U1 ∈ RI×R and V1 ∈ RJ×R respectively consist of

10.5 Parallel Factor Decomposition of Tensors

627

the R left and right singular vectors corresponding to the R leading singular values. Letting A = U1 Σ 1 ,

B = V1 ,

(10.5.7)

the PCA data model can be written as the orthogonally constrained bilinear model X = ABT

subject to

AT A = D, BT B = I,

(10.5.8)

where D is an R × R diagonal matrix and I is an R × R identity matrix. That is, the PCA requires the column vectors of the factor loading matrix A to be mutually orthogonal and each column vector of the factor score matrix B to be orthonormal. Table 10.2 compares the PCA with an unconstrained two-way bilinear analysis. Table 10.2 Comparison of PCA with two-way bilinear analysis Method

Bilinear analysis

PCA analysis

Constructed model X = ABT

X = ABT

Constraint

AT A = D, BT B = I # # #X − ABT #2 F # #2 # # min #X − ABT #

Cost function

no # # #X − ABT #2 F

Optimization

# #2 min #X − ABT #F A,B

A,B

F

s.t. AT A = D, BT B = I

Two-way bilinear data analysis as such is not available for processing a multi-way data set, but it can be generalized, as we now discuss.

10.5.2 Parallel Factor Analysis In order to generalize unconstrained two-way bilinear analysis to a multi-way data set, it is required to answer the following questions. (1) How do we name the factors when there are three or more? (2) How do we cope with the uncertainty caused by the degrees of rotational freedom? In fact, in some modern applications (e.g., chemistry), it is already difficult to distinguish factor loadings and factor scores. However, if we want to promote the analysis to deal with three-way data, it is necessary to add a new factor, and this factor is all the more difficultly to name. A simple solution to this difficulty is to use different modes to distinguish the different factors. Specifically, we call the factor loading vector ar of fitting data matrix X the mode-A vector, and the factor score vector br the mode-B vector. If we add a factor vector cr then it is called the mode-C vector, and so on; there could be mode-D, mode-E vectors etc.

628

Tensor Analysis

To cope with the uncertainty in parallel factor decomposition caused by the degrees of rotational freedom, Cattell in 1944 proposed the parallel proportional profiles principle [88]: a multi-way data model with no degrees of rotational freedom can be obtained only by assigning different proportions or weighting coefficients to the same profile or loading vectors of two-way or more than two-way data sets. Thus the two-way bilinear analysis representation (10.5.1) generalizes naturally to the parallel factor decomposition of a three-way tensor: xijk =

R 

air bjr ckr + eijk ,

(10.5.9)

r=1

where xijk are the entries of the tensor X = [[xijk ]] ∈ KI×J×K , while eijk are the entries of the additive noise tensor E = [[eijk ]] ∈ KI×J×K . Define the mode-A, mode-B and mode-C vectors respectively as ar = [a1r , . . . , aIr ]T ∈ KI×1 ,

(10.5.10)

br = [b1r , . . . , bJr ] ∈ K

(10.5.11)

T

J×1

cr = [c1r , . . . , cKr ] ∈ K T

,

K×1

;

(10.5.12)

then entry representation (10.5.9) of the factor decomposition of a three-way tensor can be equivalently expressed as X =

R 

ar ◦ br ◦ cr + E.

(10.5.13)

r=1

This is the three-way generalization of the matrix representation (10.5.2) of two-way bilinear analysis. Figure 10.10 shows, for comparison, two-way factor analysis and parallel factor analysis.

=

X

R

ar

r=1

xij =

R

air bjr + eij

r=1

(a) Factor analysis

X

=

br

R

ar

r=1

xijk =

R

cr br

air bjr ckr + eijk

r=1

(b) Parallel factor analysis

Figure 10.10 Factor analysis and parallel factor analysis.

Obviously, if only one mode factor, for example mode-A, can be varied, and the other two mode factors, mode-B and mode-C, are fixed, i.e., bjr = αr and

10.5 Parallel Factor Decomposition of Tensors

629

ckr = βr , then the three-way data model reduces to a linear combination of modeA factor vectors. Similarly, the three-way data model can also be regarded as a linear combination of mode-B factors (mode-A and mode-C fixed) or the modeC factors (mode-A and mode-B fixed), so the three-way data model (10.5.9) is a trilinear factor model. Figure 10.11 shows a comparison of parallel factor decomposition with Tucker decomposition. c1

=

X

cR

b1

+···+

a1

bR

+

aR

E

(a) Parallel factor decomposition Q J

P

I

R P

=

X

I

A

B R

G

C

K

Q

+

E

K J

(b) Tucker decomposition Figure 10.11 Parallel factor decomposition and Tucker decomposition.

Table 10.3 gives mathematical formulas for Tucker decomposition, CP decomposition and SVD. Table 10.3 Three decomposition methods Decomposition method Mathematical formula Tucker decomposition

xijk =

Q P R    p=1

CP decomposition

xijk = xij =

R 

q=1

r=1

grr air bjr

r=1

From Table 10.3 we have the following results.

gpqr aip bjq ckr

r=1

Q P R    p=1

SVD

q=1

aip bjq ckr

630

Tensor Analysis

(1) In the Tucker decomposition, the entry gpqr of the core tensor G shows that there are interaction forms involving the entry aip of the mode-A vector ai = [ai1 , . . . , aiP ]T , the entry bjq of the mode-B vector bj = [bj1 , . . . , bjQ ]T and the entry ckr of the mode-C vector ck = [ck1 , . . . , ckR ]T . (2) In the CP decomposition, the core tensor is a unit tensor. Because the entries on the superdiagonal p = q = r ∈ {1, . . . , R} are equal to 1, and all the other entries are zero, there are interactions only between the rth factor air of the mode-A vector ai , the rth factor bjr of the mode-B vector bj and the rth factor ckr of the mode-C vector ck . This means that the mode-A, mode-B and mode-C vectors have the same number R of factors, i.e., all modes extract the same number of factors. The horizontal unfolding CP decomposition has six possible forms: X(I×JK) = [X::1 , . . . , X::K ] = A(C  B)T ,

(10.5.14)

= [X1:: , . . . , XI:: ] = B(A  C) ,

(10.5.15)

X(K×IJ) = [X:1: , . . . , X:J: ] = C(B  A)T ,

(10.5.16)

=

[XT:1: , . . . , XT:J: ]

= A(B  C) ,

(10.5.17)

=

[XT::1 , . . . , XT::K ]

T

= B(C  A) ,

(10.5.18)

X(K×IJ) = [XT1:: , . . . , XTI:: ] = C(A  B)T ,

(10.5.19)

X

(J×KI)

X

(I×JK)

X

(J×KI)

T

T

where ⎡

a11 ⎢ .. A = [a1 , . . . , aR ] = ⎣ . aI1 ⎡ b11 ⎢ .. B = [b1 , . . . , bR ] = ⎣ . bJ1 and



c11 ⎢ .. C = [c1 , . . . , cR ] = ⎣ . cK1

··· .. . ···

··· .. . ··· ··· .. . ···

⎤ a1R .. ⎥ ∈ RI×R , . ⎦

(10.5.20)

aIR

⎤ b1R .. ⎥ ∈ RJ×R , . ⎦

(10.5.21)

bJR

⎤ c1R .. ⎥ ∈ RK×R . . ⎦

(10.5.22)

cKR

Here we demonstrate Equation (10.5.14), the other expressions can be similarly proved. Proof By Cattell’s principle of parallel proportional profiles, for the front slice matrix of a third-order tensor we have X::k = a1 bT1 ck1 + · · · + aR bTR ckR .

(10.5.23)

10.5 Parallel Factor Decomposition of Tensors

631

Thus X(I×JK) = [X::1 , . . . , X::K ] can be written as   X(I×JK) = a1 bT1 c11 + · · · + aR bTR c1R , . . . , a1 bT1 cK1 + · · · + aR bTR cKR ⎤ ⎡ c11 bT1 · · · cK1 bT1 ⎢ .. ⎥ .. = [a1 , . . . , aR ] ⎣ ... . . ⎦ c1R bTR

···

cKR bTR 

= A(C  B)T .

Table 10.4 summarizes the mathematical expressions for the horizontal unfolding matrices and for the CP decomposition of the third-order tensor. Table 10.4 Horizontal unfolding and CP decomposition of third-order tensor Matricization

Horizontal unfolding matrices and CP decompositions ⎧ (I×JK) ⎪ xi,(k−1)J+j = xijk ⇒ X(I×JK) = [X::1 , . . . , X::K ] = A(C  B)T ⎪ ⎪ ⎪ ⎨ (J×KI) Kiers method xj,(i−1)K+k = xijk ⇒ X(J×KI) = [X1:: , . . . , XI:: ] = B(A  C)T ⎪ ⎪ ⎪ ⎪ ⎩ x(K×IJ) ⇒ X(K×IJ) = [X:1: , . . . , X:J: ] = C(B  A)T k,(j−1)I+i = xijk ⎧ (I×JK) ⎪ xi,(j−1)K+k = xijk ⇒ X(I×JK) = [XT:1: , . . . , XT:J: ] = A(B  C)T ⎪ ⎪ ⎪ ⎨ (J×KI) LMV method xj,(k−1)I+i = xijk ⇒ X(J×KI) = [XT::1 , . . . , XT::K ] = B(C  A)T ⎪ ⎪ ⎪ ⎪ ⎩ x(K×IJ) ⇒ X(K×IJ) = [XT1:: , . . . , XTI:: ] = C(A  B)T k,(i−1)J+j = xijk ⎧ (I×JK) ⎪ xi,(k−1)J+j = xijk ⇒ X(I×JK) = [X::1 , . . . , X::K ] = A(C  B)T ⎪ ⎪ ⎪ ⎨ (J×KI) Kolda method xj,(k−1)I+i = xijk ⇒ X(J×KI) = [XT::1 , . . . , XT::K ] = B(C  A)T ⎪ ⎪ ⎪ ⎪ ⎩ x(K×IJ) ⇒ X(K×IJ) = [X:1: , . . . , X:J: ] = C(B  A)T k,(j−1)I+i = xijk

If we apply the PCA or ICA to analyze (10.5.14), then it requires the use of a truncated SVD, A(I×JK) =

R 

σr ur vrH ,

(10.5.24)

r=1

where R is the number of principal singular values. This involves R(I + JK + 1) parameters, because ur ∈ KI×1 , vr ∈ KJK×1 . However, the CP method based on (10.5.13) requires only R(I + J + K) parameters. Therefore, compared with the CP analysis method, the PCA or ICA methods requires a much larger number of free parameters, as R(I +JK +1) + R(I +J +K). This is one of the outstanding advantages of the CP method.

632

Tensor Analysis

Although most multi-way data analysis techniques can maintain the multi-way properties of the data, some simplified multi-way analysis technologies (such as Tucker1) are based on a matricization of a multi-way tensor and first transform a third-order or higher-order tensor into a two-way data set. Once a three-way tensor is unfolded to a two-way data set, two-way analysis methods (such as SVD, PCA, ICA) can be applied to extract the data construction. However, it should be noted that when a multi-way array is rearranged to a two-way data set, it is possible that loss of information and error interpretation will result. If the data are contaminated by noise, this phenomenon will be more serious. A typical example is a sensory data set, where eight judges evaluate 10 kinds of bread according to 11 class attributes [65]. When this data set is modeled using the PARAFAC model, a common evaluation guide is assumed, and each judge, in varying degrees, must comply with this evaluation guide. But, when the sensory data are unfolded to a two-way array, and the two-way factor model is used, there is no longer a common evaluation guide and each judge can make a completely independent evaluation. In this case, in order to explain the variations in the data, the two-way factor model needs to extract as many factors as possible. Compared with the PARAFAC model, which explains the data variations just in forms of basic assumptions, the additional variations captured by the two-way factor model are in fact only a reflection of the role of noise rather than of the internal structure of the data. As a result, the multi-way model is superior to the two-way model in terms of interpretation and accuracy. An important fact is that the multilinear models (such as PARAFAC, Tucker decomposition and their variants) can capture the multi-variant linear structure in the data, but a bilinear model (such as SVD, PCA and ICA) cannot. Similarly to the horizontal unfolding CP decomposition, the longitudinal unfolding CP decomposition also has six forms: ⎤ ⎡ ⎡ T ⎤ X:1: X::1 ⎢ .. ⎥ ⎢ .. ⎥ (JK×I) T (JK×I) X = ⎣ . ⎦ = (B  C)A , X = ⎣ . ⎦ = (C  B)AT , X:J:

X(KI×J)

⎤ X::1 ⎥ ⎢ = ⎣ ... ⎦ = (C  A)BT ,

X(IJ×K)

X::K ⎤ ⎡ X1:: ⎥ ⎢ = ⎣ ... ⎦ = (A  B)CT ,



XI::

X(KI×J)

X(IJ×K)

XT::K ⎡ T ⎤ X1:: ⎢ .. ⎥ = ⎣ . ⎦ = (A  C)BT , XTI:: ⎡ T ⎤ X:1: ⎢ .. ⎥ = ⎣ . ⎦ = (B  A)CT . XT:J:

10.5 Parallel Factor Decomposition of Tensors

633

Here we give a proof that X(KI×J) = (C  A)BT : ⎤ ⎡ ⎤ ⎡ X::1 a1 bT1 c11 + · · · + aR bTR c1R ⎥ ⎢ ⎥ ⎢ .. X(KI×J) = ⎣ ... ⎦ = ⎣ ⎦ . ⎡ ⎢ =⎣

c11 a1 .. .

a1 bT1 cK1 + · · · + aR bTR cKR ⎤⎡ T⎤ · · · c1R aR b1 .. ⎥ ⎢ .. ⎥ = [c ⊗ a , . . . , c ⊗ a ]BT .. . 1 1 R R . ⎦⎣ . ⎦

cK1 a1

···

X::K

cKR aR

bTR

= (C  A)BT . The other formulas can be similarly proved. Table 10.5 summarizes the mathematical representations of the longitudinal unfolding and the CP decomposition of third-order tensors. Table 10.5 Longitudinal unfolding and CP decomposition of third-order tensors Matricizing method

Kiers

LMV

Kolda

Longitudinal unfolding and CP decomposition ⎧ (JK×I) ⎪ x(k−1)J+j,i = xijk ⇔ ⎪ ⎪ ⎪ ⎨ (KI×J) x(i−1)K+k,j = xijk ⇔ ⎪ ⎪ ⎪ ⎪ ⎩ x(IJ×K) ⇔ (j−1)I+i,k = xijk ⎧ (JK×I) ⎪ x(j−1)K+k,i = xijk ⇔ ⎪ ⎪ ⎪ ⎨ (KI×J) x(k−1)I+i,j = xijk ⇔ ⎪ ⎪ ⎪ ⎪ (IJ×K) ⎩ x(i−1)J+j,k = xijk ⇔ ⎧ (JK×I) ⎪ ⎪ x(k−1)J+j,i = xijk ⇔ ⎪ ⎪ ⎨ (KI×J) x(k−1)I+i,j = xijk ⇔ ⎪ ⎪ ⎪ ⎪ (IJ×K) ⎩ x(j−1)I+i,k = xijk ⇔

X(JK×I) = [X::1 , . . . , X::K ]T = (C  B)AT X(KI×J) = [X1:: , . . . , XI:: ]T = (A  C)BT X(IJ×K) = [X:1: , . . . , X:J: ]T = (B  A)CT X(JK×I) = [XT:1: , . . . , XT:J: ]T = (B  C)AT X(KI×J) = [XT::1 , . . . , XT::K ]T = (C  A)BT X(IJ×K) = [XT1:: , . . . , XTI:: ]T = (A  B)CT X(JK×I) = [X::1 , . . . , X::K ]T = (C  B)AT X(KI×J) = [XT::1 , . . . , XT::K ]T = (C  A)BT X(IJ×K) = [XT:1: , . . . , XT:J: ]T = (B  A)CT

From Tables 10.4 and 10.5 it is easy to see that   T T (n) (n) XKiers = XKiers(n) , XLMV = XLMV(n) ,

 T (n) XKolda = XKolda(n) .

Given an N th-order tensor X ∈ RI1 ×I2 ×···×IN , its CP decomposition has the elementwise form R  (1) (2) (N ) xi1 ···iN = ui1 ,r ui2 ,r · · · ui ,r , (10.5.25) r=1

N

634

Tensor Analysis

which can be equivalently written as X =

R 

(2) (N ) u(1) r ◦ u r ◦ · · · ◦ ur .

(10.5.26)

r=1

The CP decomposition can be also written as the Kruskal operator form [254] X = [[U(1) , . . . , U(N ) ]] = [[I; U(1) , . . . , U(N ) ]].

(10.5.27)

Clearly, CP decomposition is a special example of the Tucker decomposition when the core tensor G ∈ KJ1 ×···×JN is an N th-order unit tensor I ∈ RR×···×R . Table 10.6 summarizes the mathematical representation of the CP decomposition, where L = I1 · · · In−1 In+1 · · · IN and where U(n) ∈ RIn ×R , n = 1, . . . , N , are the factor matrices. Table 10.6 Mathematical formulation of CP decomposition Representation Mathematical formulation Operator form

X = [[U(1) , U(2) , . . . , U(N ) ] = [[I; U(1) , . . . , U(N ) ]]

Mode-n form

X = I ×1 U(1) ×2 U(2) · · · ×N U(N )

Entry form

xi1 ···iN =

Outer product

X = 

Kiers  LMV  Kolda

R  (1)

ui

r=1

R  (1)

1 ,r

(2) 2 ,r

ui

(N ) ,r N

· · · ui

(2)

(N )

ur ◦ ur ◦ · · · ◦ ur

r=1

(I ×L)

xinnj

= xi1 ···iN

(j determined by (10.2.4))



X(In ×L) = U(n) (U(n−1)  · · ·  U(1)  U(N )  · · ·  U(n+1) )T (I ×L)

xinnj X

(In ×L)

(I ×L)

xinnj X

(In ×L)

= xi1 ···iN (n)

=U

(U

= xi1 ···iN (n)

=U

(U

(j determined by (10.2.6)) (n+1)

 ···  U

(N )

U

(1)

 · · ·  U(n−1) )T

(j determined by (10.2.7)) (N )

 ···  U

(n+1)

U



(n−1)

⇔  · · ·  U(1) )T

Table 10.7 summarizes several variants of the CP decomposition of three-way data: PARAFAC2 is the relaxed form of the PARAFAC model, S-PARAFAC is shifted PARAFAC and cPARAFAC is convolution PARAFAC; PARALIND is a linearly dependent parallel factor analysis. All the methods listed in the table can be extended to N -way data. In Table 10.7, the matrix Dk is the diagonal matrix whose diagonal entries are the entries of the kth row of the mode-C matrix C, H is the interaction matrix of the factor matrices, while the matrix Ak is the mode-A matrix corresponding to

10.5 Parallel Factor Decomposition of Tensors

635

Table 10.7 Comparison of multi-way models [7] Model

Mathematical formulation

PARAFAC

xijk =

P 

Q 

R 

Literature

grrr aip bjq ckr

[87], [198]

p=1 q=1 r=1

PARAFAC2

X::k = Ak Dk BT + E::k

S-PARAFAC xijk =

R 

a(i+s

r=1

jr )r

bjr ckr + eijk

PARALIND

X::k = AHDk BT + E::k

cPARAFAC

xijk =

R 

[199] [200] [64]

air b(j−θ)r cθkr + eijk

[334]

r=1

xijk =

Tucker3

Q  P  R 

gpqr aip bjq ckr

[477]

p=1 q=1 r=1

Tucker2

xijk =

Q P  

gpqk aip bjq

[477], [479]

p=1 q=1

Tucker1

xijk =

P 

gpjk aip

[477], [479]

p=1

S-Tucker3

xijk =

Q  P  R  p=1 q=1 r=1

gpqr a(i+sp )p bjq ckr [200]

the front slice X::k ; this mode-A matrix is required to obey the constraint ATk Ak = Φ, k = 1, . . . , K, where Φ is a matrix independent of Ak , k = 1, . . . , K.

10.5.3 Uniqueness Condition The CP decomposition has two inherent indeterminacies: the permutation indeterminacy and the scaling indeterminacy of the factor vectors. First, the factor vectors (ar , br , cr ) and (ap , bp , cp ) can be exchanged; the result X =

R 

ar ◦ br ◦ cr

r=1

will remain unchanged. This permutation indeterminacy can be also represented as X = [[A, B, C]] = [[AP, BP, CP]],

∀ R × R permutation matrices P,

where A = [a1 , . . . , aR ], B = [b1 , . . . , bR ] and C = [c1 , . . . , cR ]. Moreover, if αr βr γr = 1 holds for r = 1, . . . , R then X =

R  r=1

(αr ar ) ◦ (βr br ) ◦ (γr cr ) =

R  r=1

ar ◦ b r ◦ c r .

636

Tensor Analysis

In other words, the CP decomposition of a tensor is blind to the scale (i.e., the norm) of the factor vectors. The permutation indeterminacy and the scaling indeterminacy of the CP decomposition do not affect the multilinear analysis of a tensor, and therefore are allowable. Apart from these two inherent indeterminacies, the uniqueness of the CP decomposition is that in the condition of the sum of all decomposition components is equal to the original tensor, the possible combination of the rank-1 tensors is unique. The CP decomposition still has uniqueness under a more relaxed condition, in fact. This relaxed condition is related to the Kruskal rank. DEFINITION 10.10 [267] The Kruskal rank of a matrix A ∈ RI×J , simply called the k-rank, is denoted kA = rankk (A) = r and is equal to r, where r is the maximum integer such that any r column vectors of A are linearly independent. Because the matrix rank, rank(A) = r, requires only that r is the maximum number of columns such that a group of column vectors is linearly independent, whereas the Kruskal rank requires that r is the maximum number of columns such that every group of r column vectors is linearly independent, the Kruskal rank is always less than or equal to the rank of A, i.e., rankk (A) ≤ rank(A) ≤ min{I, J}, ∀ A ∈ RI×J . The sufficient condition for CP decomposition to have uniqueness, proposed by Kruskal in 1977 [267], is given by kA + kB + kC ≥ 2R + 2.

(10.5.28)

Berge and Sidiropoulos [37] showed, for R = 2 and R = 3, that Kruskal’s above sufficient condition is not only a sufficient condition, but also a necessary condition of CP decomposition of a tensor. However, this conclusion is not true for the case R > 3. Sidiropoulos and Bro [442] generalized Kruskal’s sufficient condition, and showed that if the rank of an N -way tensor X ∈ RI1 ×···×IN is R, and the CP decomposition of the tensor is R  (2) (N ) X = a(1) (10.5.29) r ◦ ar ◦ · · · ◦ ar , r=1

then the sufficient condition for unique decomposition is given by N 

rankk (X(n) ) ≥ 2R + (N − 1),

(10.5.30)

n=1

where rankk (X(n) ) is the Kruskal rank of the mode-n matricization X(n) = X(In ×L) of the tensor X . From the longitudinal unfoldings X(JK×I) = (B C)AT , X(KI×J) = (CA)BT

10.5 Parallel Factor Decomposition of Tensors

637

and X(IJ×K) = (A  B)CT , Liu and Sidiropoulos [302] showed that the necessary condition for the CP decomposition of a third-order tensor is min{rank(A  B), rank(B  C), rank(C  A)} = R,

(10.5.31)

and the necessary condition for the uniqueness for the CP decomposition of an N th-order tensor is min

n=1,...,N

rank (A1  · · ·  An−1  An+1  · · ·  AN ) = R,

(10.5.32)

where An , n = 1, . . . , N are the component matrices satisfying the longitudinal unfolding X((I2 ×···×IN )×I1 ) = (AN  · · ·  A3  A2 )AT1 .

(10.5.33)

Because rank(A  B) ≤ rank(A ⊗ B) ≤ rank(A)rank(B), a simpler necessary condition is [302] + 0 N ( min rank(Am ) ≥ R. (10.5.34) n=1,...,N

m=1,m=n

Lathauwer [279] showed that the CP decomposition of a third-order tensor A ∈ RI×J×K is generally unique if R ≤ K and R(R − 1) ≤ I(I − 1)J(J − 1)/2.

(10.5.35)

By [257], the CP decomposition of a fourth-order tensor A ∈ RI×J×K×L with rank R is generally unique if R ≤ L and R(R − 1) ≤ IJK(3IJK − IJ − IK − JK − I − J − K + 3)/4.

(10.5.36)

10.5.4 Alternating Least Squares Algorithm Given a third-order tensor X , consider the separated optimization problems for the Kiers horizontal unfolding of the CP decomposition of X : 2 22 A = arg min 2X(I×JK) − A(C  B)T 2F , (10.5.37) A

2 22 B = arg min 2X(J×KI) − B(A  C)T 2F ,

(10.5.38)

2 22 C = arg min 2X(K×IJ) − C(B  A)T 2F .

(10.5.39)

B C

Their LS solutions are respectively given by  † A = X(I×JK) (C  B)T ,  † B = X(J×KI) (A  C)T ,  † C = X(K×IJ) (B  A)T .

(10.5.40) (10.5.41) (10.5.42)

638

Tensor Analysis

Using the property of the Moore–Penrose inverse of the Khatri–Rao product (C  B)† = (CT C ∗ BT B)† (C  B)T ,

(10.5.43)

we obtain the solution of the factor matrix A as follows: A = X(I×JK) (C  B)(CT C ∗ BT B)† .

(10.5.44)

Mimicking this operation, we also obtain the LS solutions for the mode-B matrix and for mode-C matrix: B = X(J×KI) (A  C)(AT A ∗ CT C)† , C=X

(K×IJ)



(B  A)(B B ∗ A A) . T

T

(10.5.45) (10.5.46)

Equations (10.5.44)–(10.5.46) give an ALS algorithm for CP decomposition shown in Algorithm 10.4 [65], [7]. Algorithm 10.4

CP–ALS algorithm via Kiers horizontal unfolding

input: Kiers unfolding matrices X(I×JK) , X(J×KI) , X(K×IJ) of X . initialize: B0 ,C0 and the number of factors R. Put k = 0. repeat

 † 1. Ak+1 ← X(I×JK) (Ck  Bk ) CTk Ck ∗ BTk Bk .  † 2. Bk+1 ← X(J×KI) (Ak+1  Ck ) ATk+1 Ak+1 ∗ CTk Ck .  † 3. Ck+1 ← X(K×IJ) (Bk+1  Ak+1 ) BTk+1 Bk+1 ∗ ATk+1 Ak+1 . #2 # 4. exit if #X(I×JK) − Ak+1 (Ck+1  Bk+1 )T #F ≤ .

return k ← k + 1. output: Factor matrices A ∈ RI×R , B ∈ RJ×R , C ∈ RK×R .

Similarly, one has an ALS algorithm based on the LMV longitudinal unfolding for the CP decomposition of a third-order tensor, as given in Algorithm 10.5 [65], [282]. Algorithm 10.6 is an ALS algorithm for the CP decomposition of an N th-order tensor via the Kolda horizontal unfolding. The main advantage of ALS algorithms is that they are simple and easily implemented, but the main disadvantage is that most ALS algorithms converge slowly and may even be divergent. To cope with these disadvantages, it is necessary to regularize the cost function of an ALS algorithm. The following optimization problems give regularized ALS iterations for the CP decomposition of a third-order tensor.

10.5 Parallel Factor Decomposition of Tensors CP–ALS algorithm via LMV longitudinal unfolding

Algorithm 10.5

input: LMV longitudinal unfolding matrices X(JK×I) , X(KI×J) , X(IJ×K) of X . initialize: B0 , C0 and the number of factors R. Put k = 0. repeat

 T 1. Ak+1 ← (Bk  Ck )† X(JK×I) .   ˜ ← (Ck  Ak+1 )† X(KI×J) T . 2. B ˜ :,r /B ˜ :,r 2 , r = 1, . . . , R. 3. [Bk+1 ]:,r = B   ˜ ← (Ak+1  Bk+1 )† X(IJ×K) T . 4. C

˜ :,r /C ˜ :,r 2 , r = 1, . . . , R. 5. [Ck+1 ]:,r = C #2 # (JK×I) − (B  C)AT # ≤ . 6. exit if #X F

return k ← k + 1. output: The factor matrices A ∈ RI×R , B ∈ RJ×R , C ∈ RK×R .

CP–ALS (X , R) [257]

Algorithm 10.6

input: Kolda horizontal unfolding matrices X(n) , n = 1, . . . , N and the number of factor R. initialize: An ∈ RIn ×R , n = 1, . . . , N . Put k = 0. repeat for n = 1, . . . , N do 1. V ← AT1 A1 ∗ · · · ∗ ATn−1 An−1 ∗ ATn+1 An+1 ∗ · · · ∗ ATN AN . 2. An ← X(n) (AN  · · ·  An+1  An−1  · · ·  A1 )V† . 3. λn ← An . (k)

4. An ← An /λn . end for # (k+1) #  (k) # − An #F ≤ . exit if N n=1 An return k ← k + 1. output: λ = [λ1 , . . . , λN ]T and An ∈ RIn ×R , n = 1, . . . , N .

(1) Kolda horizontal unfolding [293] 32 4 22 Ak+1 = arg min 2X(I×JK) − A(Ck  Bk )T 2F + τk A − Ak 2F , A 32 4 22 Bk+1 = arg min 2X(J×KI) − B(Ck  Ak+1 )T 2F + τk B − Bk 2F , B 32 4 22 Ck+1 = arg min 2X(K×IJ) − C(Bk+1  Ak+1 )T 2F + τk C − Ck 2F . C

639

640

Tensor Analysis

(2) LMV longitudinal unfolding [338] 32 22 2 22 4 ATk+1 = arg min 2X(JK×I) − (Bk  Ck )AT 2F + τk 2AT − ATk 2F , A 32 22 2 22 4 T Bk+1 = arg min 2X(KI×J) − (Ck  Ak+1 )BT 2 + τk 2BT − BTk 2 , B

CTk+1

F

F

32 22 2 22 4 = arg min 2X(IJ×K) − (Ak+1  Bk+1 )CT 2F + τk 2CT − CTk 2F . C

With the updating of the factor matrix A in the Kolda horizontal unfolding as an example, the regularization term A − Ak 2F can force the updated matrix A not to deviate from Ak too much, so as to avoid the divergence of the iterative process. Finding the gradient matrix of the objective function with respect to the argument matrix A in the suboptimization problem, and then letting the gradient matrix equal the zero matrix, we obtain   (Ck  Bk )T (Ck  Bk ) + τk I AT = (Ck  Bk )T (XI×JK )T + τk (Ak )T . From this one can deduce a regularized LS solution of the suboptimization problem as follows: −1   A = XI×JK (Ck  Bk ) + τk Ak (Ck  Bk )T (Ck  Bk ) + τk I  −1  = XI×JK (Ck  Bk ) + τk Ak CTk Ck ∗ BTk Bk + τk I . Similarly, one has also −1   , B = X(J×KI) (Ck  Ak+1 ) + τk Bk CTk Ck ∗ ATk+1 Ak+1 + τi I  (K×IJ) −1  T C= X (Bk+1  Ak+1 ) + τk Ck Bk+1 Bk+1 ∗ ATk+1 Ak+1 + τk I . Algorithm 10.7 gives a regularized ALS algorithm for the Kolda horizontal unfolding of the CP decomposition of a third-order tensor. If τk ≡ 0, ∀ k, then Algorithm 10.7 reduces to the ordinary ALS algorithm. It can be seen that, when updating the factor matrix A, the improvement of the regularized ALS algorithm over the ALS algorithm is mainly reflected in the following two aspects. (1) The numerical stability problem caused by the possible singularity of CTk Ck ∗  −1 BTk Bk is avoided by the use of CTk Ck ∗ BTk Bk + τk I instead of (CTk Ck ∗ T † B k Bk ) . (2) The sudden jump of Ak+1 is avoided by the use of XI×JK (Ck  Bk ) + τk Ak instead of XI×JK (Ck  Bk ) because, through the weighting coefficient τk , the update of Ak+1 is weakly correlated with Ak . It should be pointed out that if the regularization term is τk A 2F rather than

10.6 Applications of Low-Rank Tensor Decomposition Algorithm 10.7

641

Regularized ALS algorithm CP–RALS (X , R, N, λ) [293]

input: Kolda horizontal unfolding matrices X(I×JK) , X(J×KI) , X(K×IJ) and the number R of factors. initialize: A0 ∈ RI×R , B0 ∈ RJ×R , C0 ∈ RK×R , τ0 = 1. Put k = 0. repeat 1. W ← XI×JK (Ck  Bk ) + τk Ak . 2. S ← CTk Ck ∗ BTk Bk + τk I. 3. Ak+1 ← WS−1 . 4. W ← X(J×KI) (Ck  Ak+1 ) + τk Bk . 5. S ← CTk Ck ∗ ATk+1 Ak+1 + τi I. 6. Bk+1 ← WS−1 . 7. W ← X(K×IJ) (Bk+1  Ak+1 ) + τk Ck . 8. S ← BTk+1 Bk+1 ∗ ATk+1 Ak+1 + τk I. 9. Ck+1 ← WS−1 . 10. τk+1 ← δ · τk . 11. exit if Ak+1 − Ak F + Bk+1 − Bk F + Ck+1 − Ck F ≤ . return k ← k + 1. output: A ∈ RI×R , B ∈ RJ×R , C ∈ RK×R .

τk A−Ak 2F then the regularized ALS algorithm only has advantage (1) and cannot have advantage (2). Regarding CP decompositions, in practice, if a tensor has linear dependence among the components of factor matrices, or large differences in magnitude between the components [366], [468], then only a few algorithms can accurately retrieve the hidden components of the tensor [468], [389]. This involves the stability of the CP tensor decomposition. By the stability of the CP tensor decomposition is meant the existence of a finite Cramer–Rao bound in a stochastic set-up where tensor entries are corrupted by additive Gaussian-distributed noises. On Cramer–Rao-induced bounds for CANDECOMP/PARAFAC tensor decompositions, the interested reader may refer to [463].

10.6 Applications of Low-Rank Tensor Decomposition Without doubt this is the era of “big data”. Technological advances in data acquisition are spurring on the generation of unprecedentedily massive data sets, thus posing a permanent challenge to data analysts [238]. Many such big data are expressed in tensor form. It is a key problem to extract useful information from big tensor data. Perhaps the most simple, yet useful, approach for solving this key

642

Tensor Analysis

problem is low-rank tensor decomposition. This section presents typical applications of low-rank tensor decomposition in various disciplines.

10.6.1 Multimodal Data Fusion In various disciplines, information about the same phenomenon can be acquired from different types of detectors, at different conditions, in multiple experiments or from multiple subjects. Each such acquisition framework is known as a modality [272], and the existence of several such acquisition frameworks is called multimodality. DEFINITION 10.11 [193] Data fusion is the process integrating multiple data and knowledge representing the same real-world object into a consistent, accurate and useful representation. The key to data fusion is diversity. DEFINITION 10.12 [272] Diversity (due to multimodality) is the property that allows one to enhance the uses, benefits and insights of data in a way that cannot be achieved with a single modality. The following are some practical examples of diversity and multimodality [272]. 1. Audiovisual multimodality A large number of audiovisual applications involve human speech and vision. In such applications, it is usually an audio channel that conveys the information of interest. Audiovisual multimodality is used for a broad range of applications, such as speech processing (including speech recognition, speech activity detection, speech enhancement, speaker extraction and separation), scene analysis (including tracking a speaker within a group), biometrics and monitoring for safety and security applications [28] and more. 2. Human–machine interaction (HMI) An important task of HMI is to design modalities that will make HMI as natural, efficient, and intuitive as possible [478]. The idea is to combine multiple interaction modes based on audiovision, touch, smell, movement (e.g., gesture detection and user tracking), interpretation of human language commands, and other multisensory functions [441], [478]. 3. Remote sensing and earth observations Sensor technologies in remote sensing and earth observation are divided into passive and active sensors. Optical hyperspectral and multispectral imaging are passive, and they rely on natural illumination. Typical active sensors have light detection and ranging (LiDAR) and synthetic aperture radar (SAR), and they can operate at night and in shadowed areas [38]. Both technologies can provide information about elevation, the three-dimensional structure of the observed objects and their surface properties.

10.6 Applications of Low-Rank Tensor Decomposition

643

4. Meteorological monitoring Accurate measurements of atmospheric phenomena such as rain, water vapor, dew, fog and snow are required for meteorological analysis and forecasting, as well as for numerous applications in hydrology, agriculture and aeronautical services. 5. Biomedicine and health This includes understanding brain functionality, medical diagnosis and smart patient-monitoring. Data fusion can be of two types: model-driven and data-driven. A model-driven approach relies on an explicit realistic model of the underlying processes [113], [49]. A data-driven approach relies only on the observations and an assumed model for them: it avoids external input [198]. Data-driven approaches are usually an extension of matrix-based approaches. Given a point x in measurement space, consider an analytical model of x given by x = f (z),

(10.6.1)

where f is an intermediate transform with unknown elements z = {z1 , . . . , zp }. The unknown elements of the model are known as latent variables. The goal of data fusion is to estimate values of the latent variables z1 , . . . , zp as precisely as possible by combining relevant information from two or more data sources. A key property in any analytical model is the uniqueness of the estimated latent variables. 1. Diversity in Matrix Decomposition Models When x in (10.6.1) is the (i, j)th entry of a data matrix X = [xij ] ∈ KI×J , perhaps the simplest yet most useful implementation of (10.6.1) is xij =

R 

air bjr ,

i = 1, . . . , I, j = 1, . . . , J,

(10.6.2)

r=1

which can be written in the matrix decomposition form X=

R 

ai bTj = ABT ,

(10.6.3)

r=1

where A = [a1 , . . . , ar ] ∈ KI×R with ar = [a1r , . . . , aIr ]T and B = [b1 , . . . , br ] ∈ KJ×R with br = [b1r , . . . , bJr ]T . The model in (10.6.3) provides I linear combinations of the columns of B and J linear combinations of the columns of A. By [443], X provides I-fold diversity for B and J-fold diversity for A. However, the underlying factor matrices A and B are not uniquely determined, because for any invertible matrix T ∈ RR×R , it always holds that X = ABT = (AT)(T−1 B).

(10.6.4)

644

Tensor Analysis

2. Diversity in Tensor Decomposition Models When x in (10.6.1) is the (i, j, k)th entry of a data tensor X ∈ KI×J×K , a simple yet efficient implementation of (10.6.1) is the CP decomposition given by R  xijk = air bjr ckr , i = 1, . . . , I, j = 1, . . . , J, k = 1, . . . , K, (10.6.5) r=1

or written as the third-order tensor form R  X = ar ◦ br ◦ cr ,

(10.6.6)

r=1

where ar = [a1r , . . . , aIr ]T , br = [b1r , . . . , bJr ]T and cr = [c1r , . . . , cKr ]T . The third-order tensor given by (10.6.6) provides three modes of linear diversity: JK linear combinations of the columns of A, IK of the columns of B and IJ of the columns of C [443]. Equation (10.6.6) is easily extended to the N th-order tensor model X =

R 

(2) (N ) u(1) r ◦ u r ◦ · · · ◦ ur ,

(10.6.7)

r=1

which provides N modes of linear diversity. Unlike the matrix decomposition, which, as explained above, does not have uniqueness, the CP decomposition (10.6.6) is inherently “essentially unique” up to a scaled permutation matrix. By [272], if a model has a unique decomposition, whether a structural constraint of factor matrices or an observational constraint (i.e., any nondegenerate mode of a higher-order tensor), it can be regarded as a node of “diversity” since it is “disjoint” from other such modes. In particular, each observational mode in the N th-order tensor (10.6.7) is a mode of diversity.

10.6.2 Fusion of Multimodal Brain Images The human brain is today perhaps the most challenging biological object under study. The reason is that the identification of the brain structures involved in cognitive processes would not only yield essential understanding about the human condition, but would also provide leverage to help deal with the staggering global burden of disease due to brain disorders [507], [238]. It is assumed for electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) recorded concurrently that the time samples for the fMRI are obtained at integer multiples of those for EEG. Consider PARAFAC in the context of time and frequency decompositions of EEG, which naturally lead to three-way tensors S. Calculating the wavelet or Gabor spectrum for each channel V(i, :), i = 1, . . . , IE , yields a matrix ST (i, :, :) for given time frames and frequencies. The spectra for all channels may be shaped into a

10.6 Applications of Low-Rank Tensor Decomposition

645

three-way tensor ST ∈ RIE ×IT δ ×IF δ , where IE is the number of sensors, IT δ is the number of time samples and IF δ is the number of frequency samples. Define the following signature matrices of EEG: K ∈ RIE ×ICx : EEG lead-field matrix, G ∈ RICx ×IT : primary current density matrix (generators) of EEG, MV ∈ RIE ×R : spectral signature of EEG atoms over electrodes, MG ∈ RICx ×R : spectral signature of EEG atoms over the cortical grid, TV ∈ RIT δ ×R : total temporal signature of EEG atoms, FV ∈ RIF δ ×R : total spectral signature of EEG atoms. In PARAFAC, ST is decomposed into R components, which can be expressed in tensor notation as [238] ST = MV ×R TV ×R FV + E = [[MV , TV , FV ]] + E

(10.6.8)

or in scalar notation as ST (iE , iT δ , iF δ ) =

R 

MV (iE , r)TV (iT δ , r)FV (iF δ , r) + E(iE , iT δ , iF δ ), (10.6.9)

r=1

where iE , iT δ and iF δ are indices for space, time and frequency, respectively, and E denotes the noise. Moreover, MV (:, r), TV (:, r), FV (:, r) are respectively the spatial, temporal and spectral signatures for atom r. By the Lagrange multiplier method, one has ˆ ,F ˆ ) ˆ ,T (M V V V = arg min

MV ,TV ,FV

"

1

S − [[MV , TV , FV ]] 2F + λ1 MV 2F + λ2 LMV 2F 2 T

#

(10.6.10) subject to MTV MV = I,

MV ≥ O,

FV ≥ O,

(10.6.11)

with Laplacian matrix L ∈ RIE ×IE . Integrating PARAFAC and STONNICA (spatio-temporal orthogonal nonnegative independent component analysis), the model (10.6.8) is easily combined with EEG source estimation as follows [238]: ST = [[KMG , TV , FV ]] + E

(10.6.12)

whose solution is given by ˆ ,F ˆ ) ˆ ,T (M G V V " # 1 2 2 2 = arg min

S − [[KMG , TV , FV ]] F + λ1 MG F + λ2 LG MG F 2 T MG ,TV ,FV (10.6.13)

646

Tensor Analysis

subject to MTG MG = I,

MG ≥ O,

FV ≥ O

(10.6.14)

with the Laplacian matrix LG ∈ RICx ×ICx ; FV is the spectral signature of the generator sources of EEG. The coupled optimization problem (10.6.13) can be solved by using the alternating nonnegative least squares (ANLS) method; see Section 10.9.

10.6.3 Process Monitoring Image data are being increasingly used for face recognition, video surveillance, monitoring of manufacturing processes etc. One of the main challenges in analyzing time-ordered image data is the high-dimensionality and complex correlation structure (i.e., temporal, spatial, and spectral correlations) of such data. Thus, effective dimension reduction and low-rank representation of images are essential for better face recognition, video surveillance and process monitoring [521]. As a low-rank decomposition method, the PCA has been traditionally used for feature extraction and the analysis of high-dimensional data such as nonlinear profiles and images. However, the regular PCA is available only for vectors and matrices and cannot be applied to color images, which are characterized by tensors. This subsection addresses extended PCA-based low-rank tensor recovery for image data. The following are various extended PCA approaches. 1. Unfold Principal Component Analysis (UPCA) [521] So-called unfold principal component analysis (UPCA) is an approach applying PCA to an unfolded tensor. Let the tensor X ∈ RI1 ×I2 ×I3 ×N denote a sample of N images, where I1 and I2 are respectively the numbers of horizontal and vertical image pixels and I3 is the number of color channels. For example, I3 is 1 for grayscale and 3 for red, green and blue (RGB)-color images. In UPCA, the sample of N images (X ∈ RI1 ×I2 ×I3 ×N ) is represented as an matrix X(4) ∈ RN ×(I1 I2 I3 ) by unfolding the tensor along the 4mode. Then, the PCA algorithm is applied to X(4) to solve the eigendecomposition problem 1 ˜T ˜ X X(4) = UDUT . (10.6.15) N − 1 (4) ˜ (4) = X(4) − X ¯ (4) is the centered matrix of X(4) , U ∈ R(I1 I2 I3 )×(I1 I2 I3 ) is Here X the orthonormal matrix of the eigenvectors and D is a diagonal matrix consisting ˜T X ˜ of the I1 I2 I3 eigenvalues of (N − 1)−1 X (4) (4) . ˜T X ˜ (4) have p principal eigenvalues λ , i = 1, . . . , p, and let Let (N − 1)−1 X (4)

i

their corresponding eigenvector matrix be Up ∈ R(I1 I2 I3 )×p . Then, the low-rank

10.6 Applications of Low-Rank Tensor Decomposition

647

unfolding matrix of the image tensor X can be obtained as ˆ (4) = Up Σp UT ∈ R(I1 I2 I3 )×(I1 I2 I3 ) , X p   where Σp = Diag( λ1 , . . . , λp ).

(10.6.16)

2. Multilinear Principal Component Analysis (MPCA) Multilinear principal component analysis (MPCA), developed by Lu et al. [304], applies PCA directly to tensors rather than unfolded data. Given N training samples of existing images, denoted Yn ∈ RI1 ×I2 ×I3 , n = 1, . . . , N , the MPCA finds a set of semi-orthogonal factor matrices U = {U(k) ∈ T RIk ×Pk |U(k) U(k) = IPk , Pk < Ik , k = 1, 2, 3} such that the projected low-dimensional tensor captures the most of the variation observed in the original tensor. Let Zn = Y˜n ×1 U(1) ×2 U(2) ×3 U(3) , n = 1, . . . , N, (10.6.17) where Y˜n = Yn − Y¯ is the centered tensor of Yn . The MPCA model procedure is to find U(k) ∈ RIk ×Pk as follows: 

 U(1) , U(2) , U(3) =

arg max

N 

U(1) ,U(2) ,U(3) n=1

Zn 2F .

(10.6.18)

PROPOSITION 10.3 Let X¯ ∈ RI1 ×I2 ×I3 ×N denote a centered tensor of a sample of N images. The semi-orthogonal factor matrices obtained by the following Tucker decomposition of X are the same as the solutions to the MPCA model in (10.6.18): 2 22   2¯ 2 G, U(1) , U(2) , U(3) = arg max 2X −G ×1 U(1) ×2 U(2) ×3 U(3) 2 . (10.6.19) G,U(1) ,U(2) ,U(3)

Proof

F

See [521].

This proposition indicates that one can use the MPCA to implement the Tucker decomposition. For a new image Ynew ∈ RI1 ×I2 ×I3 , the following give a comparison between UPCA and MPCA in process monitoring [521]: (1) In UPCA, the feature vector and the residual vector are calculated from znew = vec(Ynew )Up and enew = vec(Ynew ) − znew UTp , respectively. (2) In MPCA, the feature vector is calculated from Znew = Ynew ×1 U(1) ×2 U(2) ×3 U(3) , and the residual vector is obtained from Enew = Ynew − Znew ×1 U(1)T ×2 U(2)T ×3 U(3)T . The monitoring features are defined by θnew = vec Znew and enew = vec(Enew ). Since it breaks the tensor structure and has high computational complexity, UPCA is not an effective low-rank recovery technique for tensor data [203] although it has been extensively used in the analysis of image data.

648

Tensor Analysis

10.6.4 Note on Other Applications The high-order nonlinearities in a system are characterized by high-dimensional tensors such that these nonlinearities can be represented by just a few vectors obtained from the canonical tensor decomposition. A tensor-based nonlinear model order reduction (TNMOR) scheme was proposed in [299] for typical circuits with a few nonlinear components. The key feature of TNMOR is that it preserves the inherent sparsity through the use of low-rank tensors, thereby easing the memory requirement and speeding up computation via the exploitation of structures in the data. Combining imaging technology and high spectral resolution spectroscopy in a unique system, the hyperspectral sensor provides a powerful means to discriminate the targets of interest in a scene [541]. A hyperspectral image consisting of twodimensional spatial information and one-dimensional spectral information can be regarded as third-order tensor data. Recently, nonnegative CP decomposition has been used effectively to analyze hyperspectral large data [494], and an algorithm that combines tensor decomposition and PCA jointed algorithm was proposed in [322] for hyperspectral image denoising. On the other hand, in most practical situations it is difficult to specify the prior spectrum of the target of interest in advance. Hence, target detection without any target information, known as anomaly detection, is preferred and is more practicable (see [453], [314], [336] and the literature therein). A tensor-decomposition-based anomaly detection algorithm was recently developed in [541] for hyperspectral images. Medical images are commonly used to visualize non-invasively the motion of the heart as it beats, using magnetic resonance imaging (MRI), echocardiography (ECHO) or computed tomography (CT). Such medical image sequences can be used to analyze the motion dynamics qualitatively. A number of nonrigid image registration techniques for cardiac motion tracking have been proposed in recent years; see [161], [500]. Quantifying cardiac motion at a population level is not a simple task, since images can vary widely in terms of image quality, size, resolution and pose. In order to study the components individually, McLeod et al. [320] proposed using tensor decomposition to decouple the spatial and temporal aspects. The tensor clustering problem divides several N th-order input tensors into groups by minimizing some criterion. A robust tensor clustering (RTC) was presented in [82] for robust face clustering; it first finds a lower-rank approximation to the original tensor data using a 1 -norm optimization function and then computes a higher-order SVD of this approximate tensor to obtain the final clustering results. In modern sensing, networking and communication, multidimensional data usually have high dimensionality and are represented as tensor data. These tensorial data have a large amount of redundancy, so they need to be carefully analyzed. In order to avoid the so-called “curse of dimensionality” (the various phenomena that arise in analyzing and organizing high-dimensional spaces and which do not occur

10.7 Tensor Eigenvalue Decomposition

649

in low-dimensional settings), it is desirable to have a mechanism to reduce such redundancy for the efficient application of learning algorithms. Recently, a heterogeneous Tucker decomposition model for tensor clustering was proposed in [459]. This is a novel subspace-clustering algorithm that exploits the tensorial structure of data. Tensor decomposition relates to big data analytics. An excellent tutorial on stateof-the-art tensor decomposition, with original research on the parallel and distributed computation of low-rank decomposition for big tensors, can be found in [446].

10.7 Tensor Eigenvalue Decomposition Eigenvalues of tensors, proposed by Qi [403] and Lim [296] independently in 2005, have attracted much attention in the literature and have found applications in science and engineering, such as in molecular conformation [125], blind source separation [252], signal processing [404], magnetic resonance imaging [405], [431], spectral hypergraph analysis [294], quantum entanglement [349] and so on.

10.7.1 Tensor–Vector Products Eigenvalues of tensors are closely related to tensor–vector products. Suppose that A = [[ai1 ···im ]] ∈ Rn×n×···×n is a real-valued mth-order n-dimensional tensor with nm entries. A real-valued mth-order n-dimensional tensor A is said to be supersymmetric if its entries are invariant under permutation [259], i.e., ai

p(1)

...ip(m)

= ai1 ...im

for all

i1 , . . . , im ∈ {1, . . . , n} and

p ∈ Πm ,

where Πm denotes the space of all m-permutations. A supersymmetric tensor is usually referred to just as a symmetric tensor. Let R[m,n] and S[m,n] denote the sets of all real-valued mth-order n-dimensional tensors and of all real symmetric tensors, respectively. Putting A ∈ S[m,n] and x ∈ Rn , then we can define the following tensor–vector products [404], [259]: Axm =

n 

···

i1 =1

(Axm−1 )i1 =

n 

n  i3 =1

ai1 ···im xi1 · · · xim ,

(10.7.1)

im =1

···

i2 =1

(Axm−2 )i1 ,i2 =

n 

n 

ai1 ···im xi2 · · · xim

∀ i1 = 1, . . . , n,

(10.7.2)

ai1 ···im xi3 · · · xim

∀ i1 , i2 = 1, . . . , n.

(10.7.3)

im =1

···

n  im =1

More generally, we have the following definition.

650

Tensor Analysis

DEFINITION 10.13 [258] Let A ∈ S[m,n] and x ∈ Rn . Then, for 0 ≤ r ≤ m − 1, the (m − r)-times product of a symmetric tensor A with the vector x is denoted by Axm−r ∈ R[r,n] and defined as n n   (Axm−r )i1 ,...,ir = ··· ai1 ···im xir+1 · · · xim ir+1 =1

im =1

for all i1 , . . . , ir ∈ {1, . . . , n}.

(10.7.4)

Regarding (m − r)-times tensor–vector products, we make the following remarks. Remark 1 By the definition formulas (10.7.1)–(10.7.3), it easily follows that there exist the following relationships between the (m − r)-times tensor–vector products: Axm = xT Axm−1

and

Axm−1 = (Axm−2 )x.

(10.7.5)

Remark 2 Since Axm = xT Axm−1 , the scalar Axm can actually be regarded as the quadratic form of the symmetric tensor A and is suitable for using as a measure of the positive definiteness of tensors: the tensor A ∈ S[m,n] is called an mth-order n-dimensional positive definite tensor (positive semi-definite tensor), [m,n] [m,n] denoted A ∈ S+ (A ∈ S+0 ), if Axm > 0 (Axm ≥ 0) for all x = 0. Similarly, through Axm , we can also define a negative definite tensor, negative semi-definite tensor and indefinite tensor, respectively. Remark 3 The (m − r)-times tensor–vector product Axm−r is actually a tensor contraction [276]: Axm−1 contracts the tensor A ∈ S[m,n] into a real-valued nvector Axm−1 ∈ R[1,n] , i.e., an n-dimensional rank-1 tensor, and Axm−2 contracts the tensor A ∈ S[m,n] into an n × n real-valued matrix whereas Axm−r contracts the tensor A ∈ S[m,n] into an rth-order n-dimensional tensor. Remark 4 by [259]

The derivatives of the tensor–vector product with respect to x are given

∇(Axm ) = mAxm−1

and

∇2 (Axm ) = m(m − 1)Axm−2 .

(10.7.6)

As we have seen, an n × n matrix with entries δij is called a unit or identity matrix, denoted I ∈ Rn×n , where δ is the standard Kronecker delta function, i.e., + 1, i = j, δij = 0, i = j. In contrast, the unit tensor has two different definitions. DEFINITION 10.14 A real-valued mth-order n-dimensional tensor I ∈ R[m,n] is called a unit tensor if its entries are defined by + 1, i1 = · · · = im , ii1 ···im = δi1 ···im = (10.7.7) 0, otherwise.

10.7 Tensor Eigenvalue Decomposition

651

From the above definition, we have (Ixm−1 )i = xm−1 i

or

Ixm−1 = x[m−1] = [xm−1 , . . . , xm−1 ]T . n 1

(10.7.8)

DEFINITION 10.15 [258] For m even, if the entries of a real-valued mth-order n-dimensional tensor E ∈ R[m,n] are given by 1  δp(1)p(2) δp(3)p(4) · · · δp(m−1)p(m) (10.7.9) ei1 ···im = m! p∈Πm

for i1 , . . . , im ∈ {1, . . . , n}, then it is known as a unit tensor. From the definition (10.7.9), we have [258] Exm−1 = x

for all

x ∈ Σ,

where Σ is the unit sphere defined by    Σ = x ∈ Rn  x = 1 .

(10.7.10)

(10.7.11)

10.7.2 Determinants and Eigenvalues of Tensors The ith eigenvalue of the tensor A is denoted by λi (A). DEFINITION 10.16 [403] Let A ∈ S[m,n] be an mth-order n-dimensional symmetric tensor. For some λ ∈ C, if the polynomial system det(A − λI) = 0 (see below) has a solution x ∈ Cn \ {0} then λ is called a tensor eigenvalue of A and x is known as a tensor eigenvector of A associated with λ. The two-tuple (λ, x) is known as a tensor eigenpair of A. The symbol det(A) denotes the hyperdeterminant (or simply determinant) of the tensor A. For the tensor A ∈ S[m,n] , some properties of its determinants are summarized below. PROPOSITION 10.4 [403] For an mth-order n-dimensional tensor A, its determinant det(A) is the resultant of Axm−1 = 0, and is a homogeneous polynomial in the entries of A, with degree d = n(m − 1)n−1 . The degree of [A]i,...,i in det(A) is not greater than (m − 1)n−1 . The determinant of an mth-order n-dimensional tensor A has the following properties [403], [126], [221]: 1. 2. 3. 4.

det(A) is the resultant of Axm−1 = 0; det(A) = 0 if and only if Axm−1 = 0 has a nonzero solution; det(I) = 1, where I is the unit tensor; the determinant of a nonsingular tensor is nonzero, i.e., det(A) = 0;

652

Tensor Analysis

5. det(A) is an irreducible polynomial in the entries of A; 6. if, for some i and a tensor A, its entries aii2 ···im = 0 for all i2 , . . . , im = {1, . . . , n} then det(A) = 0; in particular, the determinant of the zero tensor is zero; n−1 7. det(γA) = γ n(m−1) det(A); 8. suppose that T ∈ C[m,n] is a complex-valued triangular tensor; then det(T ) =

n (

(ti ···i )(m−1)

(n−1)

.

(10.7.12)

i=1

The set of eigenvalues of a tensor A is called the spectrum of A, denoted σ(A), and is defined as    σ(A) = λ ∈ C det(A − λI) = 0 . (10.7.13) Suppose that T ∈ C[m,n] is a triangular tensor. Then    σ(T ) = ti···i ∈ C i ∈ {1, . . . , n} ,

(10.7.14)

and the algebraic multiplicity of the eigenvalue ti···i is (m − 1)n−1 for all i ∈ {1, . . . , n}. DEFINITION 10.17 [222] Let A be an mth-order n-dimensional tensor of indeterminate variables and λ be an indeterminate variable. The determinant χ(λ) = det(A − λI) is called the characteristic polynomial of the tensor A. Let A be an mth-order n-dimensional tensor of indeterminate variables. By [222], χ(λ) is homogeneous of degree n(m − 1)n−1 . There are various definitions of the eigenvalue of a symmetric tensor. Among them, H-eigenvalues and Z-eigenvalues are two particularly interesting definitions. 1. H-Eigenpair [403] A real number λ is called an H-eigenvalue of the tensor A if λ and a nonzero real vector x are solutions of the homogeneous polynomial equation Axm−1 = λx[m−1] , where x

[m−1]

denotes a vector in R with ith entry (x n

x[m−1] = [xm−1 , . . . , xm−1 ]T n 1

and

(10.7.15) [m−1]

)i =

Ixm−1 = x[m−1] .

xm−1 , i

i.e., (10.7.16)

Moreover, the solution x is known as an H-eigenvector of A associated with the H-eigenvalue λ, and (λ, x) is called an H-eigenpair of A. The term “H-eigenvalue” is attributable to the fact that λ is a solution to the homogeneous polynomial equation (10.7.15). A tensor all of whose entries are nonnegative is called a nonnegative tensor. THEOREM 10.2 [527] An mth-order n-dimensional nonnegative tensor A = [[ai1 ···im ]] has at least one H-eigenvalue, which has the largest modulus of all its eigenvalues.

10.7 Tensor Eigenvalue Decomposition

653

THEOREM 10.3 [406] Let A = [[ai1 ···im ]] be an mth-order n-dimensional symmetric nonnegative tensor with m ≥ 2. Then the H-eigenvalues of A are always nonnegative. 2. Z-Eigenpair [403] A real number λ and a real vector x ∈ Rn are called a Z-eigenvalue of A and a Z-eigenvector of A associated with the Z-eigenvalue λ, respectively, if they are solutions of the following system: Axm−1 = λx

and

xT x = 1.

(10.7.17)

A Z-eigenvalue and a corresponding Z-eigenvector form a Z-eigenpair. The term “Z-eigenvalue” is attributed to Zhou, who suggested to the author of [403] the above definition in a private communication. Since the definition (10.7.17) is associated with the Euclidean norm x 2 = 1, the number λ ∈ R is also known as an E-eigenvalue of A if it and a nonzero vector x ∈ Cn are solutions of the polynomial equation system (10.7.17), and the nonzero solution vector x is also called an E-eigenvector of A associated with the eigenvalue λ [403]. The Z-eigenpair defined by Qi [403] and the 2 -eigenpair defined by Lim [296] are equivalent. Importantly, Lim [296] observed that any eigenpair (λ, x) is a KKT point of the nonlinear optimization problem max Axm

x∈Rn

subject to

xT x = 1.

Define the maximum Z-eigenvalue function λZ 1 : S → R as follows:     λZ 1 = λ ∈ R λ is the largest Z-eigenvalue of A . LEMMA 10.2 [296], [403]

(10.7.18)

(10.7.19)

Let A ∈ S[m,n] , where m is even. Then, we have m λZ 1 = max Ax .

x 2 =1

(10.7.20)

In spectral hypergraph analysis, by introducing the characteristic tensor of a hypergraph, Li et al. [294] recently showed that the maximum Z-eigenvalue function of the associated characteristic tensor provides a natural link between the combinatorial structure and the analytic structure of the underlying hypergraph. PROPOSITION 10.5 [406] A real-valued mth-order n-dimensional symmetric tensor A always has Z-eigenvalues. If m is even then A always has at least one Heigenvalue. When m is even, the following three statements are equivalent: • A is positive semi-definite; • all the H-eigenvalues of A are nonnegative; • all the Z-eigenvalues of A are nonnegative.

654

Tensor Analysis

The relationship between H-eigenvalues and Z-eigenvalues is as follows. (1) For an even-order symmetric tensor, both Z-eigenvalues and H-eigenvalues always exist [403], [92]. (2) Finding an H-eigenvalue of a symmetric tensor is equivalent to solving a homogeneous polynomial equation, whereas calculating a Z-eigenvalue is equivalent to solving a nonhomogeneous polynomial equation. (3) In general, the properties of Z-eigenvalues and H-eigenvalues are different. For example, a diagonal symmetric tensor A has exactly n H-eigenvalues but may have more than n Z-eigenvalues [403]. 3. D-Eigenpair [405] Let D be a symmetric n × n matrix and assume that the order m of a tensor A is 4. Then λ and x = 0 are called a D-eigenvalue and a D-eigenvector of the tensor A, if Axm−1 = λDx

and

xT Dx = 1;

(10.7.21)

(λ, x) is said to be a D-eigenpair of the tensor A. Diffusion kurtosis imaging is a new model in medical engineering, in which a diffusion kurtosis tensor is involved. By introducing D-eigenvalues for a diffusion kurtosis tensor, the largest, the smallest and the average D-eigenvalues correspond to the largest, the smallest and the average apparent kurtosis coefficients of a water molecule in the space, respectively. See [405] for more details. 4. U-Eigenpair and US-Eigenpair [349] Define the tensor–vector product of a complex-valued tensor A ∈ C[m,n] and a complex-valued vector x ∈ Cn as (A∗ xm−1 )k =

n 

···

i2 =1

(Ax(m−1)∗ )k =

n  i2 =1

n 

a∗ki2 ···im xi2 · · · xim ,

k = 1, . . . , n,

(10.7.22)

aki2 ···im x∗i2 · · · x∗im ,

k = 1, . . . , n.

(10.7.23)

im =1

···

n  im =1

A number λ ∈ C and a vector x are respectively called a unitary eigenvalue (Ueigenvalue) and a U-eigenvector of the tensor A if λ and x are solutions of the following equation system: ⎫ A∗ xm−1 = λx[m−1]∗ ,⎪ ⎬ (10.7.24) Ax(m−1)∗ = λx[m−1] , ⎪ ⎭

x = 1. Let S ∈ S[m,n] be a complex-valued symmetric tensor and x = [x1 , . . . , xn ]T ∈

10.7 Tensor Eigenvalue Decomposition

Cn . Denote (S ∗ xm−1 )k =

n  i2 =1

···

n 

655

s∗ki2 ···im xi2 · · · xim .

im =1

A number λ ∈ C and a vector x = 0 are respectively called a unitary symmetric eigenvalue (US-eigenvalue) and US-eigenvector of S if λ and a nonzero vector x ∈ Cn are solutions of the following equation system: ⎫ S ∗ xm−1 = λx∗ ,⎪ ⎬ (10.7.25) Sx(m−1)∗ = λx, ⎪ ⎭

x = 1. THEOREM 10.4 [349] For complex-valued mth-order n-dimensional tensors A, B, S ∈ H = Cn ×···×n , the following results hold: • A, B = B, A∗ ; • all U-eigenvalues are real numbers; • the US-eigenpair (λ, x) of a complex-valued symmetric mth-order n-dimensional tensor S can also be defined by the following equation system: 0 S ∗ xm−1 = λx∗ , (10.7.26)

x = 1, λ ∈ R, or Sx(m−1)∗ = λx,

x = 1,

0 (10.7.27)

λ ∈ R.

5. B-Eigenpair [92] Let A be an mth-order n-dimensional tensor consisting of nm entries in R: A = [[ai1 ···im ]],

Ai1 ···im ∈ R, 1 ≤ i1 , . . . , im ≤ n.

DEFINITION 10.18 Let A and B be two mth-order n-dimensional tensors on R. Assume that both Axm−1 and Bxm−1 are not identically equal to zero. Then (λ, x) ∈ C × (Cn \ {0}) is called an eigenpair of A relative to B, if the n-system of equations (A − λB)xm−1 = 0,

(10.7.28)

i.e., n  i2 =1

···

n 

(aii2 ···im − λbii2 ···im )x2 · · · xm = 0,

im =1

possesses a solution.

i = 1, . . . , n

(10.7.29)

656

Tensor Analysis

The scalar λ is known as a B-eigenvalue of A, and x is called a B eigenvector of A. In tensor analysis problems motivated by the geometric measurement of quantum entanglement, the largest U-eigenvalue |λ| is the entanglement eigenvalue, and the corresponding U-eigenvector x is the closest separable state (for more details, see [349]).

10.7.3 Generalized Tensor Eigenvalues Problems Let A = [[ai1 ···im ]] ∈ Cn×···×n and B = [[bi1 ···im ]] ∈ Cn×···×n be two complex-valued mth-order n-dimensional tensors. By [126], (A, B) is called a regular tensor pair, if det(βA − αB) = 0

for some (α, β) ∈ C × C;

and (A, B) is known as a singular tensor pair if det(βA − αB) = 0

for some (α, β) ∈ C × C.

A tensor pair is also called a tensor pencil. In the following, we focus on the regular tensor pairs. DEFINITION 10.19 [126] For a regular tensor pair (A, B), if there exists (α, β) ∈ C × C with a nonzero vector x ∈ Cn such that βAxm−1 = αBxm−1

(10.7.30)

then λ = α/β is called a generalized eigenvalue of the regular tensor pair (A, B) if det(B) = 0. The set of eigenvalues (α, β) is called the spectrum of the tensor pair (A, B), and is denoted as    σ(A, B) = (α, β) ∈ C × C det(βA − αB) = 0 , (10.7.31) or, for a nonsingular B, the spectrum is denoted as    σ(A, B) = λ ∈ C det(A − λB) = 0 .

(10.7.32)

In particular, when β = 1, the generalized eigenvalue λ and the generalized eigenvector x reduce to the B-eigenvalue and the B-eigenvector of A. Several tensor eigenpair definitions described above can be unified under a generalized tensor eigenpair framework, as follows. (1) When B = I is a unit tensor defined by (10.7.7), the generalized tensor eigenvalue directly reduces to the H-eigenvalue of A, because Ixm−1 = x[m−1] and thus Axm−1 = λIxm−1 = λx[m−1] . (2) If B = E is a unit tensor defined by (10.7.9) with x = 1 then the generalized tensor eigenvalue reduces to the Z-eigenvalue of A.

10.7 Tensor Eigenvalue Decomposition

657

(3) Put Bxm−1 = Dx and Bxm = 1 (i.e., xT Bxm−1 = xT Dx = 1). Then the generalized tensor eigenvalue becomes the D-eigenvalue of A. (4) When A and x are a complex-valued tensor and vector, respectively, and B = I, the definition of the generalized eigenvalue becomes the definition of the Ueigenvalue of A. (5) If A is a conjugate-symmetric tensor, x ∈ C and B = E with x = 1 then the generalized tensor eigenvalue gives the US-eigenvalue of A. Premultiplying both sides of (10.7.30) by xT and using (10.7.5), it can be seen that any generalized tensor eigenpair satisfies λ=

Axm . Bxm

(10.7.33)

Consider a computation method for generalized tensor eigenvalue decomposition. [m,n] and define the nonlinear optimization To this end, let A ∈ S[m,n] and B ∈ S+ problem [259] " # Axm m max f (x) = subject to x ∈ Σ. (10.7.34)

x x Bxm Here the constraint makes the term x m in f (x) superfluous. THEOREM 10.5 [259] Let f (x) be defined as in (10.7.34). For x ∈ Σ, the gradient and the Hessian matrix are respectively given by     m Axm m m−1 m−1 g(x) = ∇f (x) = (Ax Bx , (10.7.35) )x + Ax − Bxm Bxm m2 Axm m ) H(x) = ∇2 f (x) = (m − 1)Axm−2 (Bxm−1  Bxm−1 ) + m 3 (Bx ) Bxm *   + Axm I + (m − 2)xxT + m(Axm−1  x) m ) − (m − 1)Axm Bxm−2 + m(Axm−1  Bxm−1 ) (Bxm )2  * + mAxm x  Bxm−1 , (10.7.36) where the symbol  means the symmetrized outer product: a  b = abT + baT . Kolda and Mayo [259], in 2014, developed a generalized eigenproblem adaptive power (GEAP) method, as shown in Algorithm 10.8. The GEAP method can be specialized to the Z-eigenpair adaptive power method, as shown in Algorithm 10.9. In these algorithms, the choice β = 1 is for finding local maxima (when the function is convex); otherwise, the choice β = −1 is for finding local minima (when the function is concave). For the tolerance on being positive or negative definite, τ = 10−6 is taken and a convergence error = 10−15 is selected in [259].

658

Tensor Analysis

Algorithm 10.8

Generalized eigenproblem adaptive power method [259]

[m,n] ˆ 0 /ˆ ˆ 0 is an initial guess. given: A ∈ S[m,n] , B ∈ S+ and x0 ← x x0 , where x repeat m , Bxm−2 , Axm−1 , Bxm−1 , Axm 1. Compute Axm−2 k and Bxk . k k k k m m 2. λk ← Axk /(Bxk ). 3. Hk ← H(xk ). 4. αk ← β max{0, (τ − λmin (βHk ))/m}.   ˆ k+1 ← β Axkm−1 − λk Bxm−1 5. x + (αk + λk )Bxm k xk . k ˆ k+1 /ˆ 6. xk+1 ← x xk+1 . 7. exit if |λk+1 − λk | ≤ . return k ← k + 1. output: λ ← λk+1 .

Algorithm 10.9

Z-eigenpair adaptive power method [259]

ˆ 0 /ˆ ˆ 0 is an initial guess. given: A ∈ S[m,n] and x0 ← x x0 , where x repeat 1. Compute Axm−2 , Axm−1 and Axm k . k k m 2. λk ← Axk . (m−2) 3. Hk ← m(m − 1)Axk . 4. αk ← β max{0, (τ − λmin (βHk ))/m}.   ˆ k+1 ← β Axkm−1 + αk xk . 5. x ˆ k+1 /ˆ xk+1 . 6. xk+1 ← x 7. exit if |λk+1 − λk | ≤ . return k ← k + 1. output: λ ← λk+1 .

10.7.4 Orthogonal Decomposition of Symmetric Tensors Denote the mth-order outer product of a vector u ∈ Rn by u⊗m = u ◦ u ◦ · · · ◦ u (m times). When m = 2, u⊗2 = u ◦ u = uuT . Using the eigenvalue decomposition, every real-valued symmetric matrix M ∈ [2,n] S can be written as n n   λi ui uTi = λi u⊗2 M= i , i=1

i=1

where u1 , . . . , un are orthonormal. A real-valued mth-order n-dimensional symmetric tensor A ∈ S[m,n] is called orthogonally decomposable if r  λi u⊗m , (10.7.37) A= i i=1

10.8 Preprocessing and Postprocessing

659

where u1 , . . . , ur are orthonormal. The smallest r for which such a decomposition exists is known as the (symmetric) rank of A. If A ∈ S[m,n] is an orthogonally decomposable tensor, i.e., A = u1 , . . . , ur ∈ Rn are orthonormal then [415] Aum−1 = j

r 

r 

λi u⊗m and i

i=1

λi (ui ◦ uj )m−1 ui = λj uj

i=1

for all j = 1, . . . , r. Therefore, u1 , . . . , ur are the Z-eigenvectors of A, with corresponding Z-eigenvalues λ1 , . . . , λr . Note that A and u1 , . . . , ur are required to be real in order to force λ1 , . . . , λr to be real as well. DEFINITION 10.20 [415] A unit vector u = vk ∈ Rn is a robust eigenvector of A ∈ S[m,n] if there exists > 0 such that starting from one random vector v0 , the repeated iteration vk+1 ←

Avkm−1

Avkm−1

(10.7.38)

converges, i.e., vk+1 − vk < . Algorithm 10.10 uses the repeated iteration (10.7.38) starting with random vectors v0 ∈ Rn to compute robust eigenvectors. Algorithm 10.10

Tensor power method [259]

input: An orthogonally decomposable tensor A. initialization: i = 1. repeat until A = 0. 1. Choose one random vector v0 ∈ Rn . 2. Let ui = vk be the result of repeated iteration of (10.7.38) starting with v0 . 3. Solve the equation Aum−1 = λi ui to get the eigenvalue λi . i

4. Set A = A − λi u⊗m . i 5. i ← i + 1.

output: u1 , . . . , ur and λ1 , . . . , λr .

10.8 Preprocessing and Postprocessing In the previous sections, the theory and methods were introduced for multi-way data analysis. As with two-way data processing, multi-way data analysis also requires preprocessing and postprocessing [201], [36], [62], [245].

660

Tensor Analysis

10.8.1 Centering and Scaling of Multi-Way Data In matrix analysis, signal processing, pattern recognition, wireless communications etc., preprocessing most commonly involves the zero-mean normalization (i.e., centering) of data. Preprocessing in tensor analysis is more complicated than in matrix analysis, as it requires not only data centering, but also data scaling. The main purpose of the centering is to remove the “direct current” component in the original data, which is the fixed component, keeping only the random components, which contain all the useful information. For the third-order tensor X ∈ RI×J×K , its mode-n centering, for n = 1, 2, 3, is respectively given by 1 x , I i=1 ijk

(10.8.1)

J 1 = x , J j=1 ijk

(10.8.2)

K 1  xijk . K

(10.8.3)

I

1 xcent = xijk − x ¯:jk , ijk

2 xcent ijk

= xijk − x ¯i:k ,

3 = xijk − x ¯ij: , xcent ijk

x ¯:jk = x ¯i:k

x ¯ij: =

k=1

If the centering is carried out on two modes then cent (1,2)

= xijk − x ¯::k ,

x ¯::k =

cent (2,3)

= xijk − x ¯i:: ,

x ¯i:: =

xijk

xijk

J I 1  x , IJ i=1 j=1 ijk

(10.8.4)

K J 1  xijk , JK j=1

(10.8.5)

K I 1  xijk . IK i=1

(10.8.6)

k=1

cent (3,1)

xijk

= xijk − x ¯:j: ,

x ¯:j: =

k=1

In addition, the tensor data must be scaled, i.e., every piece of data is divided by some fixed factor:  K J   x ijk scal 1 ! xijk = , si = x2ijk i = 1, . . . , I, (10.8.7) si j=1 k=1  I  K  xijk 2 = , sj = ! x2ijk j = 1, . . . , J, (10.8.8) xscal ijk sj i=1 k=1  J I   x ijk 3 xscal = (10.8.9) , sk = ! x2ijk k = 1, . . . , K. ijk sk i=1 j=1 It should be noted that it is usual only to center the columns of the unfolding

10.8 Preprocessing and Postprocessing

661

matrices of a third-order tensor, and to scale the rows [62]. Therefore, we have for Kiers matricization, ⎧ I×JK 2 scal 1 ⎪ = [X::1 , . . . , X::K ] centering xcent ⎪ ijk , scaling xijk , ⎨X 3 scal 2 XJ×KI = [X1:: , . . . , XI:: ] centering xcent ijk , scaling xijk , ⎪ ⎪ ⎩XK×IJ = [X , . . . , X ] centering xcent 1 , scaling xscal 3 ; :1: :J: ijk ijk for LMV matricization, ⎧ I×JK 3 scal 1 ⎪ = [XT:1: , . . . , XT:J: ] centering xcent ⎪ ijk , scaling xijk , ⎨X 1 scal 2 XJ×KI = [XT::1 , . . . , XT::K ] centering xcent ijk , scaling xijk , ⎪ ⎪ ⎩XK×IJ = [XT , . . . , XT ] centering xcent 2 , scaling xscal 3 ; 1:: I:: ijk ijk for Kolda matricization, ⎧ 2 scal 1 ⎪ ⎪XI×JK = [X::1 , . . . , X::K ] centering xcent ijk , scaling xijk , ⎨ 1 scal 2 XJ×KI = [XT::1 , . . . , XT::K ] centering xcent ijk , scaling xijk , ⎪ ⎪ ⎩XK×IJ = [XT , . . . , XT ] centering xcent 1 , scaling xscal 3 . :1: :J: ijk ijk

10.8.2 Compression of Data Array When the dimension of a higher-order array is large (such as when In is in the hundreds or thousands), the low-convergence problem of the ALS for a CP decomposition is usually serious. In these cases, the data array has to be compressed: the original N th-order tensor X ∈ RI1 ×···×IN is compressed into an N th-order core tensor G ∈ RJ1 ×···×JN with smaller dimensions, then the ALS algorithm is run on the CP decomposition of the core tensor and finally the mode matrices of the CP decomposition of the original tensor are reconstructed from the mode matrices of the CP decomposition of the core tensor. As an example, consider the CP decomposition of a third-order tensor X ∈ RI1 ×I2 ×I3 : X(1) = A(C  B)T ,

X(2) = B(A  C)T ,

X(3) = C(B  A)T .

Assume that some dimension (say I3 ) is very large. Because the dimensions of the matricization X(n) , n = 1, 2, 3, are very large, an ALS algorithm for computing the factor matrices A ∈ RI1 ×R , B ∈ RI2 ×R , C ∈ RI3 ×R directly would face a troublesome amount of computation and slow convergence. Choose three orthogonal matrices U ∈ RI1 ×J1 , V ∈ RI2 ×J2 , W ∈ RI3 ×J3 such that A = UP,

B = VQ,

C = WR.

Then, the identification of the three mode matrices A, B, C of the CP decomposition of the third-order tensor is divided into two subproblems, that of identifying an

662

Tensor Analysis

orthogonal matrix triple (U, V, W) and that of identifying a nonorthogonal matrix triple (P, Q, R). Using the relation between the Khatri–Rao product and the Kronecker product, (WR)  (VQ) = (W ⊗ V)(R  Q), it is easy to see that T

X(1) = A(C  B)T = UP [(WR)  (VQ)] = UP(R  Q)T (W ⊗ V)T . If we let G(1) = P(R  Q)T then X(1) can be rewritten as X(1) = UG(1) (W  V)T .

(10.8.10)

Similarly, it can be shown that X(2) = VG(2) (U  W)T ,

(10.8.11)

X(3) = WG(3) (V  U) ,

(10.8.12)

T

where G(2) = Q(P  R)T ,

G(3) = R(Q  P)T .

Equations (10.8.10)–(10.8.12) show that: (1) G ∈ RJ1 ×J2 ×J3 is the core tensor of the Tucker decomposition of the third-order tensor X , and G(n) , n = 1, 2, 3, are the matricization results for the core tensor; (2) the matrix triple (P, Q, R) is the factor matrix triple of the CP decomposition of the core tensor G. The above considerations result in the data array compression algorithm shown in Algorithm 10.11. The following are some remarks on this data array compression algorithm. Remark 1 The essence of data array compression is that the CP decomposition of the original tensor X ∈ RI1 ×I2 ×I3 with large dimension is compressed into the CP decomposition of a core tensor G ∈ RJ1 ×J2 ×J3 with small dimension. Remark 2 The compression algorithm needs no iteration; its computation is simple, and thus a convergence problem does not exist. Remark 3 If the factor matrices B and C found by the compression algorithm are used as the initial matrices to run the ALS algorithm for the CP decomposition of the original third-order tensor X ∈ RI1 ×I2 ×I3 , then the ALS algorithm requires only a few iterations to achieve convergence. Therefore, data array compression can effectively accelerate the CP decomposition of a higher-dimensional tensor. Remark 4 The compression algorithm given in Algorithm 10.11 is based on the Kiers matricization. If in the unfolding matrices and the mode-n matrices appropriate replacements are made, Algorithm 10.11 can be used for the LMV or Kolda matricizations. After finding the factor (i.e., loading) matrices A, B, C via the ALS algorithm, it

10.8 Preprocessing and Postprocessing Algorithm 10.11

663

Data array compression algorithm [62]

input: X ∈ RI1 ×I2 ×I3 . initialize: X(1) , X(2) , X(3) . 1. Compute the SVD of the unfolding matrices [U, S, T] = SVD(X(1) , J1 ), [V, S, T] = SVD(X(2) , J2 ), [W, S, T] = SVD(X(3) , J3 ), where Ji are the numbers of leading singular values of X(i) , i = 1, 2, 3. 2. Compute the mode-n matrices of the core tensor, G(1) = UT X(1) (W ⊗ V), G(2) = VT X(2) (U ⊗ W), G(3) = WT X(3) (V ⊗ U). 3. Apply Algorithm 10.5 to make the CP decompositions G(1) = P(R  Q)T , G(2) = Q(P  R)T , G(3) = R(Q  P)T , where P ∈ RJ1 ×R , Q ∈ RJ2 ×R , R ∈ RJ3 ×R . 4. Find the factor matrices of the CP decomposition of the original tensor: A = UP, B = VQ, C = WR. output: A ∈ RI1 ×J1 , B ∈ RI2 ×J2 , C ∈ RI3 ×J3 .

is necessary to make a quality evaluation of these loading matrices. This evaluation is called leverage analysis [62]. Because the loading matrices in CP decomposition are nonorthogonal, one needs to compute a leverage vector for every loading matrix. The leverage vector of loading matrix U is defined as the vector consisting of the diagonal entries of the projection matrix of U, namely [62] v = diag(U(UT U)−1 UT ).

(10.8.13)

Letting U = A, B, C in sequence, one obtains the leverage vectors of three loading matrices. The values of the entries of a leverage vector are between 0 and 1, i.e., 0 ≤ vi ≤ 1. If the value of some leverage entry is larger then the leverage of the sample value used is larger; otherwise, the leverage is smaller. If some leverage entry is very large, then the sample data may perhaps contain some outlier; the model thus obtained would not be appropriate, and one would need to eliminate this outlier and restart the ALS algorithm to make a new tensor analysis.

664

Tensor Analysis

10.9 Nonnegative Tensor Decomposition Algorithms As stated in Chapter 6, nonnegative matrix factorization (NMF) is more useful than principal component analysis (PCA) or independent component analysis (ICA) in much modern data analysis. In this section, we consider the generalization of the NMF to a multi-way array (tensor). The study of nonnegative tensor decomposition (NTD) was first proposed by researchers studying nonnegatively constrained PARAFAC in chemometrics, [63], [66], [366], [368]. The nonnegative tensor decomposition problem is: given an N th-order nonnegative tensor X ∈ RI+1 ×···×IN , decompose it into NTD 1:

X ≈ G ×1 A(1) ×2 · · · ×N A(N ) with G ≥ 0, A(1) ≥ 0, . . . , A(N ) ≥ 0,

NTD 2:

(10.9.1)

X ≈ I ×1 A(1) ×2 · · · ×N A(N ) with A(1) ≥ 0, . . . , A(N ) ≥ 0,

(10.9.2)

where the expressions G ≥ 0 and A ≥ 0 denote that all the entries of the tensor G and of the matrix A respectively are nonnegative. The NTD problem can be expressed as the minimization of the objective function: 22 12 2 2 X − G ×1 A(1) ×2 · · · ×N A(N ) 2 , (10.9.3) NTD 1: min 2 F G,A(1) ,...,A(N ) 2 22 12 2 2 X − I ×1 A(1) ×2 · · · ×N A(N ) 2 . NTD 2: min (10.9.4) 2 F A(1) ,...,A(N ) 2 The basic idea of NTD is that the NTD problem is rewritten in the basic form of NMF: X = AS, A ≥ 0, S ≥ 0. In NTD there are two classes of common algorithms: multiplication updating algorithms and ALS updating algorithms.

10.9.1 Multiplication Algorithm The key to designing a multiplication algorithm for NTD is rewriting the Tucker decomposition or the CP decomposition of a nonnegative tensor in the standard NMF form X = AS. 1. Multiplication Algorithm for Nonnegative Tensor Decomposition Let X = [[xi1 ···iN ]] ∈ RI+1 ×···×IN be a nonnegative tensor, and consider its approximate Tucker decomposition X ≈ G ×1 A(1) ×2 A(2) ×3 · · · ×N A(N ) ,

(10.9.5)

where xi1 ···iN ≥ 0 and A(n) = [aij ] ∈ RI+n ×Jn , n = 1, . . . , N are N nonnegative (n)

factor matrices; G = [[gj1 ···jN ]] ∈ RJ+1 ×···×JN is the nonnegative core tensor, i.e.,

10.9 Nonnegative Tensor Decomposition Algorithms

665

(n)

aij ≥ 0, ∀ i = 1, . . . , In , j = 1, . . . , Jn , n = 1, . . . , N and gj1 ···jN ≥ 0, ∀ jn = 1, . . . , Jn , n = 1, . . . , N . This Tucker decomposition with a nonnegative core tensor and all nonnegative factor matrices is known as a nonnegative Tucker decomposition. In the Tucker decomposition of a general tensor, all the columns of every factor matrix are required to be orthogonal, but this requirement is no longer reasonable in a nonnegative Tucker decomposition. This is the essential difference between nonnegative Tucker decomposition and higher-order SVD. Using the unfolding matrices of the nonnegative tensor X , an NTD can be also expressed in the matricization form (n)T

X(n) = A(n) G(n) A⊗

J ×(J1 ···Jn−1 Jn+1 ···JN )

In ×L here X(n) ∈ R+ , G(n) ∈ R+n

A

(n)



(n) A⊗

RI+n ×Jn

;

(10.9.6) (n)

L×(J1 ···Jn−1 Jn+1 ···JN )

, A⊗ ∈ R +

,

and

⎧ (n−1) ⎪ ⊗ · · · ⊗ A(1) ⊗ A(N ) ⊗ · · · ⊗ A(n+1) ⎪ ⎨A = A(n+1) ⊗ · · · ⊗ A(N ) ⊗ A(1) ⊗ · · · ⊗ A(n−1) ⎪ ⎪ ⎩A(N ) ⊗ · · · ⊗ A(n+1) ⊗ A(n−1) ⊗ · · · ⊗ A(1)

(Kiers matricization) (LMV matricization)

(Kolda matricization) (10.9.7) denotes the Kronecker product of n − 1 factor matrices, excepting for the mode-n matrix. Defining the cost function J(A(n) , G(n) ) =

22 12 2 (n)T 2 2X(n) − A(n) G(n) A⊗ 2 , 2 F

(10.9.8)

and writing its matrix differential as standard forms dJ(A(n) , G(n) ) = tr(WdA(n) ) and dJ(A(n) , G(n) ) = tr(VdG(n) ), respectively, then ∂J(A(n) , G(n) ) (n) (n)T (n) = −X(n) A⊗ GT(n) + A(n) G(n) A⊗ A⊗ GT(n) , ∂A(n) ∂J(A(n) , G(n) ) (n) (n)T (n) = −A(n)T X(n) A⊗ + A(n)T A(n) G(n) A⊗ A⊗ . ∂G(n)

(10.9.9) (10.9.10)

Hence, we get the following gradient algorithms: ) * A(n) ← A(n) + ηA X(n) M(n) − A(n) MT(n) M(n) , ) * (n) (n)T (n) G(n) ← G(n) + ηG A(n)T X(n) A⊗ − A(n)T A(n) G(n) A⊗ A⊗ , (n)

where M(n) = A⊗ GT(n) .

666

Tensor Analysis

Taking step sizes ηij = 

a(n) (i, j)  , A(n) MT(n) M(n) ij

ηjk =
100 and X − Z ×1 V1T ×2 · · · ×N VN

return t ← t + 1. output: factor matrices V1 , . . . , VN , core tensor Z, the completed tensor X .

10.10.2 Smooth PARAFAC Tensor Completion The state-of-the art tensor completion methods can be categorized into three types: low-rank nuclear norm, low-rank Tucker decomposition and low-rank CP decomposition with optional smoothness constraints [533]. It has recently been realized (see e.g., [408], [196]) that smoothness is quite an important factor for visual data completion. Given an incomplete input tensor D ∈ RI1 ×I2 ×···×IN , let Ω denote the indices of the available elements of D. Consider a smooth PARAFAC tensor completion (SPC), formulated as the optimization problem [533]: + min

g,U(1) ,...,U(N )

R N  2 2p 12 gr2  (n) 2 2X − Z 22 + ρ 2L(n) ur(n) 2p F 2 2 n=1 r=1

0 (10.10.16)

10.10 Tensor Completion

675

subject to Z=

R 

(2) (N ) gr u(1) r ◦ ur ◦ · · · ◦ ur ,

r=1

XΩ = DΩ , XΩ = ZΩ , u(n) r 2 = 1, ∀ r ∈ {1, . . . , R}, ∀ n ∈ {1, . . . , N }, (n)

(n)

where Z is a completed output tensor, U(n) = [u1 , . . . , uR ], n = 1, . . . , N , are the factor matrices, ρ = [ρ1 , . . . , ρR ]T is the smoothness parameter vector and the constrained conditions XΩ = DΩ and XΩ = ZΩ stand for + D(i1 , . . . , iN ), (i1 , . . . , iN ) ∈ Ω, (10.10.17) X (i1 , . . . , iN ) = Z(i1 , . . . , iN ), otherwise. The parameters gr2 in (10.10.16) are adaptively updated, depending on the penalty levels of the smoothness constraint terms. This adaptively enforces the different levels of smoothness for the different components (factors). For image completion, the SPC model (10.10.16) decomposes an image adaptively into a strong smooth background and a weaker smooth foreground. Because the low-frequency components are generally stronger than the high-frequency components in the Fourier domain of the image data, different penalty levels are used for individual rank-1 components, depending on their scales gr [533]. Let L(n) ∈ R(In −1)×In be a smoothness constraint matrix defined typically as ⎡ ⎤ 1 −1 0 ⎢ ⎥ 1 −1 ⎢ ⎥ (10.10.18) L(n) = ⎢ ⎥. .. .. ⎣ ⎦ . . 0

1

−1

Then, the second term in (10.10.16) is seen to be a penalty term which enforces the (n) smoothness of the individual feature vectors ur via n −1  (n) 2 (n) (n) 2p I  ur (i) − ur(n) (i + 1)p . 2 L ur 2 = p

(10.10.19)

i=1

Two common choices of the parameter p in (10.10.19) correspond to two different smoothness constraints [533]: • Total variation (TV) Here p = 1 and is thus constrained to be the sum of the differences between neighboring entries of the individual feature vectors. • Quadratic variation (QV) Here p = 2 and is thus constrained to equal the sum of the squared differences between neighboring entries of the individual feature vectors. In image recovery problems, such as denoising and restoration, it is generally considered that the TV constraint is better than QV.

676

Tensor Analysis

Define the function 2 12 (N ) 22 2Yr − gr u(1) Fp(r,n) (u(n) r )= r ◦ · · · ◦ ur F 2 N 22 g 2  (m) 2 2 , + r ρ 2L(m) u(m) r p 2 m=1

(10.10.20)

for r = 1, . . . , R, n = 1, . . . , N . Let SDR denote the signal-distortion ratio. Algorithm 10.16 summarizes a lowrank smooth PARAFAC tensor completion (SPC) algorithm. Algorithm 10.16 has the following advantages and characteristics. (1) The factor number R can be adaptively determined. (2) When p = 1, the SPC algorithm is a tensor completion algorithm with the TV constraint. When p = 2 is used, the SPC algorithm is a tensor completion algorithm with the QV constraint. (3) The SPC algorithm focuses upon “good approximation” rather than “minimum rank decomposition”, since estimation of the number of factors for the smooth PARAFAC decomposition model is difficult, and its accuracy does not need to be very high provided that the reconstructed tensor is sufficiently well approximated in practice. It should be noted that the SPC algorithm does not guarantee global convergence since tensor decomposition is in general a nonconvex optimization problem.

10.11 Software The available software sources for tensor decompositions and tensor operations include the following (from [103], [238]). (1) The MATLAB Tensor Toolbox version 2.5 [21] offers classes for tensor operations and has several tensor decomposition algorithms. (2) The TDALAB [546] and TENSORBOX [388] provide a user friendly interface and advanced algorithms for CPD, nonnegative TKD and MWCA. (3) The Tensorlab toolbox [450] builds upon a complex optimization framework and offers numerical algorithms for computing the CPD, BTD, and TKD and includes the structured data fusion of tensors and complex optimization tools. (4) The N-way toolbox [15] includes (constrained) CPD, TKD and PLS in the context of chemometrics applications; many of these methods can handle constraints (e.g., nonnegativity and orthogonality) and missing elements. (5) The TT toolbox [361] includes functions for basic operations that are lowparametric representations of high-dimensional tensors. A Python implementation of the TT Toolbox is also available [362].

10.11 Software

677

SPC algorithm [533]

Algorithm 10.16

1. input: T , Ω, p, ρ, SDR, and ν. 2.  ← 10(−SDR/10) TΩ 2F . 3. XΩ ← TΩ , XΩ¯ ← average of TΩ . 4. Construct matrix L(n) , n = 1, . . . , N by (10.10.19). 5. R ← R + 1.

# (n) (n) -# (n) 6. initialize: ur ← ur #ur #2 , r = 1, . . . , R, n = 1, . . . , N . . (1) (2) (N ) / 7. gR ← XR , uR ◦ uR ◦ · · · ◦ uR .  (1) (2) (N ) 8. E = X − R . r=1 gr ur ◦ ur ◦ · · · ◦ ur 9. EΩ¯ ← 0. 10. t ← 0. 11. μt ← E2F . 12. repeat 13. for r = 1, . . . , R do (1)

(2)

(N )

Yr ← E + gr ur ◦ ur ◦ · · · ◦ ur

.

for n = 1, . . . , N do (n)

Use the ALS method to solve ur #

= ur

(n)

(ur ).

(n) ur

(n) -# #u(n) # . r 2

(n)

ur

(r,n)

← arg min Fp

end for / . (1) (2) (N ) Yr ,ur ◦ur ◦···◦ur . gr ← N p

1+

n=1

E ← Yr −

(n) ρ(n) L(n) ur

(1) gr ur



(2) ur

p

(N )

◦ · · · ◦ ur

and EΩ¯ ← 0.

end for 14. μt+1 ← E2F . 15. if

|μt − μt+1 | ≤ ν then |μt+1 − | R ← R + 1. # (n) (n) -# (n) initialize ur ← ur #ur #2 , r = 1, . . . , R, n = 1, . . . , N . . (1) (2) (N ) / gR ← E, uR ◦ uR ◦ · · · ◦ uR . (1)

(2)

(N )

E ← E − gR uR ◦ uR ◦ · · · ◦ uR . EΩ¯ ← 0. 16. end if 17. t ← t + 1. 18. exist if μt ≤ . R (1) (2) (N ) . r=1 gr ur ◦ ur ◦ · · · ◦ ur

19. Z ←

20. XΩ¯ ← ZΩ¯ . 21. return 22. output: X , Z.

678

Tensor Analysis

(6) The Hierarchical Tucker toolbox [261] provides the construction and manipulation of tensors. (7) The Tensor calculus library [147] provides tensor tools for scientific computing. (8) The CuBatch [183] is a MATLAB-based software for both tensor and classical data analysis, and includes validation tools providing graphical outputs. (9) The MATLAB CMTF Toolbox [4] provides constrained and unconstrained algorithms for the joint decomposition of tensors with different orders, using PARAFAC. (10) The ERPWAVELAB [333] is a MATLAB-based toolbox for the multi-channel time/frequency analysis of EEG and MEG by various tensor decompositions, including PARAFAC, shifted PARAFAC and Tucker. (11) The Tensor Toolbox in Python [50] includes the decomposition of tensors in tensor-train format1 and spectral tensor-train format. (12) The TensorReg Toolbox for MATLAB [546] provides sparse PARAFAC and Tucker regression functions. (13) The code developed for multiway analysis is also available from the Three-Mode Company [266]. Many algorithms developed by the authors in [238] can be found at http://www.research.cneuro.cu/categories/tensor, http://neurosignal.boun.edu.tr/software/tensor.

Exercises 10.1 Given that the front slice matrices of the third-order tensor X ∈ R3×4×2 are ⎡ ⎤ ⎡ ⎤ 1 4 7 10 15 18 21 24 X1 = ⎣2 5 8 11⎦ , X2 = ⎣16 19 22 25⎦ , 3 6 9 12 17 20 23 26   1 3 5 and find the front slice matrices Y1 and Y2 of Y = X ×1 U. let U = 2 4 6 10.2 Given a third-order tensor X with entries

1

X1,1,1 = 1,

X1,2,1 = 3,

X1,1,2 = 5,

X1,2,2 = 7,

X2,1,1 = 2,

X2,2,1 = 4,

X2,1,2 = 6,

X2,2,2 = 8,

Consider a given tensor B approximated by a tensor A with entries  (1) (2) (d) ai ...i = gα ,i ,α gα ,i ,α · · · gα 1

d

α0 ,...,αd−1 ,αd

0

1

1

1

2

2

d−1 ,id ,αd

.

This tensor decomposition can be represented graphically by a linear tensor network. This picture looks like a train with carriages and links between them, and that justifies the name tensor-train decomposition. See Oseledest O. V. Tensor-train decomposition. SIAM J. Sci. Comput., 33(5): 2295–2317, 2011.

Exercises

679

find the LMV and Kolda horizontal unfolding matrices 10.3 Given that the frontal slices of X ∈ R3×4×2 are ⎡ ⎤ ⎡ 1 −4 7 −10 13 −16 ⎣ ⎦ ⎣ X1 = 2 −5 8 −11 , X2 = 14 −17 3 −6 9 −12 15 −18 find the three mode-n 10.4 Given that the frontal ⎡ 1 X1 = ⎣ 2 3

of the tensor X . 19 20 21

⎤ −22 −23⎦ , −24

unfoldings of X and its vectorized version. slices of X ∈ R3×4×2 are ⎤ ⎡ ⎤ 4 7 10 13 16 19 22 5 8 11⎦ , X2 = ⎣14 17 20 23⎦ , 6 9 12 15 18 21 24

for a 2 × 3 matrix

 U=

1 2

3 4



5 6

find the frontal slices of the mode-1 product Y = X ×1 U ∈ R2×4×2 . 10.5 Prove that (I×JK)

xi,(k−1)J+j = xijk ⇒

X(I×JK) = [X::1 , . . . , X::K ] = AG(P ×QR) (C ⊗ B)T .

10.6 Let Xi:: = b1 cT1 ai1 + · · · + bR cTR aiR . Show that the CP decomposition of the horizonal unfolding matrix is given by X(J×KI) = [X1:: , . . . , XI:: ] = B(A  C)T and the CP decomposition of the longitudinal unfolding matrix is given by ⎤ ⎡ X1:: ⎥ ⎢ X(IJ×K) = ⎣ ... ⎦ = (A  B)CT . XI:: 10.7 Let X:j: = a1 cT1 bj1 + · · · + aR cTR bjR . Show that the CP decomposition of the horizonal unfolding matrix is X(K×IJ) = [X:1: , . . . , X:J: ] = C(B  A)T and the CP decomposition of the longitudinal unfolding matrix is ⎤ ⎡ X:1: ⎥ ⎢ X(JK×I) = ⎣ ... ⎦ = (B  C)AT , X:J: T where A = [a1 , . . . , aR ]T , B = [bjk ]J,R j=1,k and C = [c1 , . . . , cR ] .

680

Tensor Analysis

10.8 Let X::k = a1 bT1 ck1 + · · · + aR bTR ckR . Show that the CP decomposition of the horizonal unfolding matrix is X(J×KI) = [XT::1 , . . . , XT::K ] = B(C  A)T and the CP decomposition of the longitudinal unfolding matrix is ⎤ ⎡ X::1 ⎥ ⎢ X(KI×J) = ⎣ ... ⎦ = (C  A)BT . X::K 10.9 Use the definition of the (m − r)th-order tensor–vector products to prove the following results: (a) Axm = xT Axm−1 ; (b) Axm−1 = (Axm−2 )x. 10.10 Let x = [x1 , x2 , x3 ]T be a direction vector and W be a fourth-order threedimensional fully symmetric tensor, called the diffusion kurtosis (DK) tensor. Prove that the critical points of the maximization problem max Wx4 satisfy the following equations: +

subject to Dx2 = 1

Wx3 = λDx, Dx2 = 1.

Here D is a positive definite matrix. 10.11 Let f1 (x) = Axm and f2 (x) = x m 2 . Find the gradient vectors and the Hessian matrices of f1 (x) and f2 (x), respectively. 10.12 Find the gradient vector and the Hessian matrix of the function f (x) =

Axm . Bxm

10.13 Prove that Ixm−1 = x[m−1] , where I is an identity tensor. 10.14 Create a table for comparison of second-order matrix analysis and the higherorder matrix analysis (i.e., tensor analysis).

References

[1] Abatzoglou T. J., Mendel J. M., Harada G. A. The constrained total least squares technique and its applications to harmonic superresolution. IEEE Trans. Signal Processing, 39: 1070–1087, 1991. [2] Abbott D. The Biographical Dictionary of Sciences: Mathematicians. New York: P. Bedrick Books, 1986. [3] Abraham R., Marsden J. E., Ratiu T. Manifolds, Tensor Analysis, and Applications. New York: Addison-Wesley, 1983. [4] Acar E. The MATLAB CMTF Toolbox. 2014. [5] Acar E., Aykut-Bingo C., Bingo H., Bro R., Yener B. Multiway analysis of epilepsy tensors. Bioinformatics, 23: i10–i18, 2007. [6] Acar E., Camtepe S. A., Krishnamoorthy M., Yener B. Modeling and multiway analysis of chatroom tensors. In Proc. IEEE International Conference on Intelligence and Security Informatics. Springer, 256–268, 2005. [7] Acar E., Yener B. Unsupervised multiway data analysis: A literature survey. IEEE Trans. Knowledge and Data Engineering, 21(1): 6–20, 2009. [8] Acar R., Vogel C. R. Analysis of bounded variation penalty methods for illposed problems. Inverse Problems, 10: 1217–1229, 1994. [9] Adamyan V. M., Arov D. Z. A general solution of a problem in linear prediction of stationary processes. Theory Probab. Appl., 13: 294–407, 1968. [10] Adib A., Moreau E., Aboutajdine D. Source separation contrasts using a reference signal. IEEE Signal Processing Lett., 11(3): 312–315, 2004. [11] Afriat S. N. Orthogonal and oblique projectors and the characteristics of pairs of vector spaces. Math. Proc. Cambridge Philos. Soc., 53: 800–816, 1957. [12] Alexander S. T. Adaptive Signal Processing: Theory and Applications. New York: Springer, 1986. [13] Alter O., Brown P. O., Botstein D. Generalized singular value decomposition for comparative analysis of genome-scale expression data sets of two different organisms. Proc. Nat. Aca. Sci., USA, 100(6): 3351–3356, 2003. [14] Amari S., Nagaoka H. Methods of Information Geometry. New York: Oxford University Press, 2000. [15] Andersson C., Bro R. The N-way toolbox for MATLAB. [Online]. Chemomet. Intell. Lab. Syst., 52(1): 1–4, 2000. Available at http://www.models.life.ku.dk/nwaytoolbox/ (2000). [16] Andrews H., Hunt B. Digital Image Restoration. Engliwood Cliffs, NJ: Prentice-Hall, 1977. 681

682

References

[17] Aronszajn N. Theory of reproducing kernels. Trans. Amer. Math. Soc., 68: 800–816, 1950. [18] Auslender A. Asymptotic properties of the Fenchel dual functional and applications to decomposition problems. J. Optim. Theory Appl., 73(3): 427–449, 1992. [19] Autonne L. Sur les groupes lineaires, reelles et orthogonaus. Bull Soc. Math., France, 30: 121–133, 1902. [20] Aybat N. S., Iyengar G. A first-order augmented Lagrangian method for compressed sensing. SIAM J. Optim., 22(2): 429–459, 2012. [21] Bader B. W., Kolda T. G., et al. MATLAB Tensor Toolbox version 2.5. Available at http://www.sandia.gov/tgkolda/TensorToolbox/ (2012). [22] Banachiewicz T. Zur Berechungung der Determinanten, wie auch der Inverse, und zur darauf basierten Aufl¨osung der Systeme linearer Gleichungen. Acta Astronomica, S´er. C, 3: 41–67, 1937. [23] Bapat R. Nonnegative Matrices and Applications. Cambridge University Press, 1997. [24] Barbarossa S., Daddio E., Galati G. Comparison of optimum and linear prediction technique for clutter cancellation. Proc. IEE, Part F, 134: 277–282, 1987. [25] Barnett S. Matrices: Methods and Applications. Oxford: Clarendon Press, 1990. [26] Barzilai J., Borwein J. M. Two-point step size gradient methods. IMA J. Numer. Anal. 8: 141–148, 1988. [27] Basri R., Jacobs D. Lambertian reflectance and linear subspaces. IEEE Trans. Pattern Anal. Machine Intell., 25(2): 218–233, 2003. [28] Beal M., Jojic N., Attias H. A graphical model for audiovisual object tracking. IEEE Trans. Pattern Anal. Machine Intell., 25(7): 828–836, 2003. [29] Beck A., Teboulle M. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci., 2(1): 183–202, 2009. [30] Beck A., Elder Y. C. Structured total maximum likelihood: an alternative to structured total squares. SIAM J. Matrix Anal. Appl., 31(5): 2623–2649, 2010. [31] Behrens R. T., Scharf L. L. Signal processing applications of oblique projection operators. IEEE Trans. Signal Processing, 42(6): 1413–1424, 1994. [32] Bellman R. Introduction to Matrix Analysis, 2nd edn. McGraw-Hill, 1970. [33] Belochrani A., Abed-Merain K., Cardoso J. F., Moulines E. A blind source separation technique using second-order statistics. IEEE Trans. Signal Processing, 45(2): 434–444 1997. [34] Beltrami E. Sulle funzioni bilineari, Giomale di Mathematiche ad Uso Studenti Delle Universita. 11: 98–106, 1873. An English translation by D Boley is available as Technical Report 90–37, University of Minnesota, Department of Computer Science, 1990. [35] Berberian S. K. Linear Algebra. New York: Oxford University Press, 1992. [36] Berge J. M. F. T. Convergence of PARAFAC preprocessing procedures and the Deming–Stephan method of iterative proportional fitting. In Multiway Data Analysis (Coppi R., Bolasco S. eds.). Amsterdam: Elsevier, 53–63, 1989. [37] Berge J. M. F. T., Sidiropolous N. D. On uniqueness in CANDECOMP/ PARAFAC. Psychometrika, 67: 399–409, 2002.

References

683

[38] Berger C., Voltersen M., Eckardt R., Eberle J. Multi-modal and multitemporal data fusion: outcome of the 2012 GRSS data fusion contest. IEEE J. Selected Topics Appl. Earth Observat. Remote Sensing, 6(3): 1324–1340, 2013. [39] Berman A., Plemmons R. Nonnegative Matrices in the Mathematical Sciences. Philadephia: SIAM, 1994. [40] Berry M. W., Browne M., Langville A. N., Pauca V. P., Plemmons R. J. Algorithms and applications for approximate nonnegative matrix factorization. Computational Statistics & Data Analysis, 52: 155–173, 2007. [41] Bertsekas D. P. Multiplier methods: a survey. Automatica, 12: 133–145, 1976. [42] Bertsekas D. P. Nonlinear Programming, 2nd edn. Belmont: Athena Scientific, 1999. [43] Bertsekas D. P., Nedich A., Ozdaglar A. Convex Analysis and Optimization. Belmont: Athena Scientific, 2003. [44] Bertsekas D. P., Tseng P. Partial proximal minimization algorithms for convex programming. SIAM J. Optimization, 4(3): 551–572, 1994. [45] Bi H., Jiang C., Zhang B., Wang Z., Hong W. Radar change imaging with undersampled data based on matrix completion and Bayesian compressive sensing. IEEE Geosci. Remote Sensing Lett., 12(7): 1546–1550, 2015. [46] Bi H., Zhang B., Hong W., Zhou S. Matrix-completion-based airborne tomographic SAR inversion under missing data. IEEE Geosci. Remote Sensing Lett. 12(11): 2346–2350, 2015. [47] Bienvenu G., Kopp L. Princip`ede la goniomgraveetre passive adaptive. Proc. 7th Colloqucum GRESIT, Nice, France, 106/1–106/10, 1979. [48] Bien J., Tibshirani R. J. Sparse estimation of a covariance matrix. Biometrika, 98(4): 807–820, 2011. [49] Biessmann F., Plis S., Meinecke F. C., Eichele T., Mller K. R. Analysis of multimodal neuroimaging data. IEEE Reviews in Biomedical Engineering, 4: 26–58, 2011. [50] Bigoni D. Tensor toolbox, 2015. [51] Boot J. Computation of the generalized inverse of singular or rectangular matrices. Amer. Math. Monthly, 70: 302–303, 1963. [52] Boyd S., Parikh N., Chu E., Peleato B., Eckstein J. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3(1): 1–122, 2010. [53] Boyd S., Vandenberghe L. Convex Optimization. Cambridge, UK: Cambridge University Press, 2004. [54] Boyd S., Vandenberghe L. Subgradients. Notes for EE364b, Stanford University, Winter 2006–2007, April 13, 2008. [55] Brachat J., Common P., Mourrain B., Tsigaridas E. Symmetric tensor decomposition. Linear Algebra Appl., 433(11-12): 1851–1872, 2010. [56] Bramble J., Pasciak J. A preconditioning technique for indefinite systems resulting from mixed approximations of elliptic problems. Math. Comput., 50(181): 1–17, 1988. [57] Brandwood D. H. A complex gradient operator and its application in adaptive array theory. Proc. IEE, 130: 11–16, 1983. [58] Branham R. L. Total least squares in astronomy. In Recent Advances in Total Least Squares Techniques and Error-in-Variables Modeling (Van Huffel S. ed.). Philadelphia, PA: SIAM, 1997.

684

References

[59] Bregman L. M. The method of successive projection for finding a common point of convex sets. Soviet Math. Dokl., 6: 688–692, 1965. [60] Brewer J. W. Kronecker products and matrix calculus in system theory. IEEE Trans. Circuits Syst., 25: 772–781, 1978. [61] Bridges T. J., Morris P. J. Differential eigenvalue problems in which the parameters appear nonlinearly. J. Comput. Phys., 55: 437–460, 1984. [62] Bro R. PARAFAC: tutorial and applications. Chemometrics and Intelligent Laboratory Systems, 38: 149–171, 1997. [63] Bro R., de Jong S. A fast non-negatively constrained least squares algorithm. J. Chemometrics, 11(5): 393–401, 1997. [64] Bro R., Harshman R. A., Sidiropoulos N. D. Modeling multi-way data with linearly dependent loadings. Technical Report 2005-176, KVL, 2005. [65] Bro R. Multiway analysis in the food industry: models, algorithms and applications, Doctoral dissertation, University of Amsterdam, 1998. [66] Bro R., Sidiropoulos N. D. Least squares algorithms under unimodality and non-negativity constraints. J. Chemometrics, 12(4): 223–247, 1998. [67] Brockwell P. J., Davis R. A. Time Series: Theory and Methods. New York: Springer-Verlag, 1987. [68] Brookes M. Matrix Reference Manual 2004. Available at http://www.ee.ic.ac.uk/hp/staff/dmb/matrix/intro.html (2005). [69] Bruckstein A. M., Donoho D. L., Elad M. From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Review, 51(1): 34–81, 2009. [70] Brunet J. P., Tamayo P., Golub T. R., Mesirov J. P. Metagenes and molecular pattern discovery using matrix factorization. Proc. Nat. Acad. Sci. USA, 101(12): 4164–4169, 2004. [71] Bunse-Gerstner A. An analysis of the HR algorithm for computing the eigenvalues of a matrix. Linear Alg. Applic., 35: 155–173, 1981. [72] Byrd R. H., Hribar M. E., Nocedal J. An interior point algorithm for large scale nonlinear programming. SIAM J. Optimization, 1999, 9(4): 877–900. [73] Byrne C., Censor Y. Proximity function minimization using multiple Bregman projections, with applications to split feasibility and Kullback–Leibler distance minimization. Ann. Oper. Rese., 105: 77–98, 2001. [74] Cadzow J. A. Spectral estimation: an overdetermined rational model equation approach. Proc. IEEE, 70: 907–938, 1982. [75] Cai J. F., Cand`es E. J., Shen Z. A singular value thresholding algorithm for matrix completion. SIAM J. Optim., 20(4): 1956–1982, 2010. [76] Cai T. T., Wang L., Xu G. New bounds for restricted isometry constants. IEEE Trans. Inform. Theory, 56(9): 4388–4394, 2010. [77] Cand`es E. J., Li X., Ma Y., Wright J. Robust principal component analysis? J. ACM, 58(3): 1–37, 2011. [78] Cand`es E. J., Plan Y. Matrix completion with noise. Proc. IEEE, 98(6): 925– 936, 2010. [79] Cand`es E. J., Recht B. Exact matrix completion via convex optimization. Found. Comput. Math., 9: 717–772, 2009. [80] Cand`es E. J., Romberg J., Tao T. Stable signal recovery from incomplete and inaccurate information. Commun. Pure Appl. Math., 59: 1207–1233, 2005.

References

685

[81] Cand`es E. J., Tao T. Near optimal signal recovery from random projections: universal encoding strategies. IEEE Trans. Inform. Theory, 52(12): 5406– 5425, 2006. [82] Cao X., Wei X., Han Y., Lin D. Robust face clustering via tensor decomposition. IEEE Trans. Cybernetics, 45(11): 2546–2557, 2015. [83] Cardoso J. F. On the performance of orthogonal source separation algorithms. In Proc. Europ. Assoc. Signal Processing 94, VII, Edinburgh, U.K., 776C779, 1994. [84] Cardoso J. F., Souloumiac A. Blind beamforming for non-Gaussian signals. Proc. IEE, F, 40(6): 362–370, 1993. [85] Cardoso J. F., Souloumiac A. Jacobi angles for simultaneous diagonalization. SIAM J. Matrix Anal. Appl., 17(1): 161–164, 1996. [86] Carroll C. W. The created response surface technique for optimizing nonlinear restrained systems. Oper. Res., 9: 169–184, 1961. [87] Carroll J. D., Chang J. Analysis of individual differences in multidimensional scaling via an N -way generalization of Eckart–Young decomposition. Psychometrika, 35: 283–319, 1970. [88] Cattell R. B. Parallel proportional profiles and other principles for determining the choice of factors by rotation. Psychometrika, 9: 267–283, 1944. [89] Chan R. H., Ng M. K. Conjugate gradient methods for Toepilitz systems. SIAM Review, 38(3): 427–482, 1996. [90] Chandrasekaran V., Sanghavi S., Parrilo P. A., Wilisky A S. Rank-sparsity incoherence for matrix decomposition. SIAM J. Optim. 21(2): 572–596, 2011. [91] Chang C. I., Du Q. Estimation of number of spectrally distinct signal sources in hyperspectral imagery. IEEE Trans. Geosci. Remote Sensing, 42(3): 608– 619, 2004. [92] Chang K. C., Pearson K., Zhang T. On eigenvalue problems of real symmetric tensors. J. Math. Anal. Appl., 290(350): 416–422, 2009. [93] Chatelin F. Eigenvalues of Matrices. New York: Wiley, 1993. [94] Chen B., Petropulu A. P. Frequency domain blind MIMO system identification based on second- and higher order statistics. IEEE Trans. Signal Processing, 49(8): 1677–1688, 2001. [95] Chen S. S., Donoho D. L., Saunders M. A. Atomic decomposition by basis pursuit. SIAM J. Sci. Comput., 20(1): 33–61, 1998. [96] Chen S. S., Donoho D. L., Saunders M. A. Atomic decomposition by basis pursuit. SIAM Review, 43(1): 129–159, 2001. [97] Chen W., Chen M., Zhou J. Adaptively regularized constrained total leastsquares image restoration. IEEE Trans. Image Processing, 9(4): 588–596, 2000. [98] Chen Y. Incoherence-optimal matrix completion. IEEE Trans. Inform. Theory, 61(5): 2909–2913, 2015. [99] Chen Y. L., Hsu C. T., Liao H. Y. M. Simultaneous tensor decomposition and completion using factor priors. IEEE Trans. Pattern Anal. Machine Intell., 36(3): 577–591, 2014. [100] Chua L. O. Dynamic nonlinear networks: state-of-the-art. IEEE Trans. Circuits Syst., 27: 1024–1044, 1980. [101] Cichocki A., Cruces S., Amari S. I. Generalized alpha-beta divergences and their application to robust nonnegative matrix factorization. Entropy, 13: 134–170, 2011.

686

References

[102] Cichocki A., Lee H., Kim Y. D., Choi S. Non-negative matrix factorization with α-divergence. Pattern Recog. Lett., 29: 1433–1440, 2008. [103] Cichocki A., Mandic D. P., Phan A. H., Caiafa C. F., Zhou G., Zhao Q., Lathauwer L. D. Tensor decompositions for signal processing applications. IEEE Signal Processing Mag., 34(3): 145–163, 2015. [104] Cichocki A., Zdunek R., Amari S. I. Nonnegative matrix and tensor factorization. IEEE Signal Processing Mag., 25(1): 142–145, 2008. [105] Cichocki A., Zdunek R., Amari S. I. Csiszar’s divergences for nonnegative matrix factorization: family of new algorithms. In Lecture Notes in Computer Science; Springer: Charleston, SC, USA, 3889: 32–39, 2006. [106] Cichocki A., Zdunek R., Choi S., Plemmons R., Amari S. I. Novel multi-layer nonnegative tensor factorization with sparsity constraints. Springer LNCS, 4432: 271–280, 2007. [107] Cirrincione G., Cirrincione M., Herault J., et al. The MCA EXIN neuron for the minor component analysis. IEEE Trans. Neural Networks, 13(1): 160–187, 2002. [108] Clark J. V., Zhou N., Pister K. S. J. Modified nodal analysis for MEMS with multi-energy domains. In Proc. Inter. Conf. on Modeling and Simulation of Microsystems, Semiconductors, Sensors and Actuators. San Diego, USA, 2000. Available at www-bsac.EECS.Berkely.EDU/-cfm/publication.html [109] Combettes P. L., Pesquet J. C. Proximal splitting methods in signal processing. In Fixed-Point Algorithms for Inverse Problems in Science and Engineering, New York: Springer, 185–212, 2011. [110] Comon P., Golub G., Lim L. H., Mourrain B. Symmetric tensors and symmetric tensor rank. SM Technical Report 06–02, Stanford University, 2006. [111] Comon P., Golub G., Lim L. H., Mourrain B. Symmetric tensora and symmetric tensor rank. SIAM J. Matrix Anal. Appl., 30(3): 1254–1279, 2008. [112] Comon P., Moreau E. Blind MIMO equalization and joint-diagonalization criteria. Proc. 2001 IEEE Inter. Conf. on Acoustics, Speech, and Signal Processing (ICASSP ’01), 5: 2749–2752, 2001. [113] Correa N. M., Adall T., Li Y. O., Calhoun V. D. Canonical correlation analysis for data fusion and group inferences. IEEE Signal Processing Mag., 27(4): 39– 50, 2010. [114] Dai Y. H., Fletcher R. Projected Barzilai–Borwein methods for large-scale box-constrained quadratic programming. Num. Math., 100: 21–47, 2005. [115] Dai W., Milenkovic O. Subspace pursuit for compressive sensing signal reconstruction. IEEE Trans. Inform. Theory, 55(5): 2230–2249, 2009. [116] Davila C. E. A subspace approach to estimation of autoregressive parameters from noisy measurements, IEEE Trans. Signal Processing, 46: 531–534, 1998. [117] Davis G. A fast algorithm for inversion of block Toeplitz matrix. Signal Processing, 43: 3022–3025, 1995. [118] Davis G., Mallat S., Avellaneda M. Adaptive greedy approximation. J. Constr. Approx., 13(1): 57–98, 1997. [119] Decell Jr. H. P. An application of the Cayley–Hamilton theorem to generalized matrix inversion. SIAM Review, 7(4): 526–528, 1965. [120] Delsarte P., Genin Y. The split Levinson algorithm. IEEE Trans. Acoust., Speech, Signal Processing, 34: 471–478, 1986.

References

687

[121] Delsarte P., Genin Y. On the splitting of classical algorithms in linear prediction theory. IEEE Trans. Acoust., Speech, Signal Processing, 35: 645–653, 1987. [122] Dembo R. S., Steihaug T. Truncated-Newton algorithms for large-scale unconstrainted optimization. Math Progr., 26: 190–212, 1983. [123] Demoment G. Image reconstruction and restoration: overview of common estimation problems. IEEE Trans. Acoust., Speech, Signal Processing, 37(12): 2024–2036, 1989. [124] Dewester S., Dumains S., Landauer T., Furnas G., Harshman R. Indexing by latent semantic analysis. J. Soc. Inf. Sci., 41(6): 391–407, 1990. [125] Diamond R. A note on the rotational superposition problem. Acta Crystallogr. Sect. A, 44: 211–216, 1988. [126] Ding W., Wei Y. Generalized tensor eigenvalue problems. SIAM J. Matrix Anal. Appl., 36(3): 1073–1099, 2015. [127] Doclo S., Moonen M. GSVD-based optimal filtering for single and multimicrophone speech enhancement. IEEE Trans. Signal Processing, 50(9): 2230–2244, 2002. [128] Donoho D. L. Compressed sensing. IEEE Trans. Inform. Theory, 52(4): 1289– 1306, 2006. [129] Donoho D. L. For most large underdetermined systems of linear equations, the minimal 1 solution is also the sparsest solution. Commun. Pure Appl. Math., LIX: 797–829, 2006. [130] Donoho D. L., Elad M. Optimally sparse representations in general (nonorthogonal) dictionaries via 1 minimization. Proc. Nat. Acad. Sci. USA, 100(5): 2197–2202, 2003. [131] Donoho D. L., Huo X. Uncertainty principles and ideal atomic decomposition. IEEE Trans. Inform. Theory, 47(7): 2845–2862, 2001. [132] Donoho D. L., Tsaig Y. Fast solution of l1 -norm minimization problems when the solution may be sparse. IEEE Trans. Inform. Theory, 54(11): 4789–4812, 2008. [133] Donoho D. L., Tsaig T., Drori T., Starck J. -L. Sparse solution of underdetermined linear equations by stagewise orthogonal matching pursuit (StOMP). Stanford Univ., Palo Alto, CA, Stat. Dept. Tech. Rep. 2006-02, Mar. 2006. [134] Drakakis K., Rickard S., de Frein R., Cichocki A. Analysis of financial data using non-negative matrix factorization. International Mathematical Forum 3(38): 1853–1870, 2008. [135] Drmac Z. Accurate computation of the product-induced singular value decomposition with applications. SIAM J. Numer. Anal., 35(5): 1969–1994, 1998. [136] Drmac Z. A tangent algorithm for computing the generalized singular value decomposition. SIAM J. Numer. Anal., 35(5): 1804–1832, 1998. [137] Drmac Z. New accurate algorithms for singular value decomposition of matrix triplets. SIAM J. Matrix Anal. Appl., 21(3): 1026–1050, 2000. [138] Duda R. O., Hart P E. Pattern Classification and Scene Analysis. New York: Wiley, 1973. [139] Duncan W. J. Some devices for the solution of large sets of simultaneous linear equations. The London, Edinburgh, anf Dublin Philosophical Magazine and J. Science, Seventh Series, 35: 660–670, 1944. [140] Eckart C., Young G. The approximation of one matrix by another of lower rank. Psychometrica, 1: 211–218, 1936.

688

References

[141] Eckart C., Young G. A Principal axis transformation for non-Hermitian matrices. Null Amer. Math. Soc., 45: 118–121, 1939. [142] Edelman A., Arias T. A., Smith S. T. The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl., 20(2): 303–353, 1998. [143] Efron B., Hastie T., Johnstone I., Tibshirani R. Least angle regression. Ann. Statist., 32: 407–499, 2004. [144] Efroymson G., Steger A., Steeinberg S. A matrix eigenvalue problem. SIAM Review, 22(1): 99–100, 1980. [145] Eldar Y. C., Oppenheim A. V. MMSE whitening and subspace whitening. IEEE Trans. Inform. Theory, 49(7): 1846–1851, 2003. [146] Eldar Y. C., Opeenheim A. V. Orthogonal and projected orthogonal matched filter detection. Signal Processing, 84: 677–693, 2004. [147] Espig M., Schuster M., Killaitis A., Waldren N., et al. Tensor calculus library. Available at http://gitorious.org/tensorcalculus/ (2012). [148] Estienne F., Matthijs N., Massart D. L., Ricoux P., Leibovici D. Multi-way modelling of high-dimensionality electroencephalographic data. Chemometrics Intell. Lab. Syst., 58(1): 59–72, 2001. [149] Faddeev D. K., Faddeeva V. N. Computational Methods of Linear Algebra. San Francisco: W H Freedman Co., 1963. [150] Farina A., Golino G., Timmoneri L. Comparison between LS and TLS in adaptive processing for radar systems. IEE P-Radar Sonar Nav., 150(1): 2–6, 2003. [151] Fernando K. V., Hammarling S. J. A product induced singular value decomposition (PSVD) for two matrices and balanced relation. In Proc. Conf. on Linear Algebra in Signals, Systems and Controls, Society for Industrial and Applied Mathematics (SIAM). PA: Philadelphia, 128–140, 1988. [152] Fiacco A. V., McCormick G. P. Nonlinear Programming: Sequential Unconstrained Minimization Techniques. New York: Wiley, 1968; or Classics Appl. Math. 4, SIAM, PA: Philadelphia, Reprint of the 1968 original, 1990. [153] Field D. J. Relation between the statistics of natural images and the response properties of cortical cells. J. Opt. Soc. Amer. A, 4: 2370–2393, 1984. [154] Figueiredo M. A. T., Nowak R. D., Wright S. J. Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process., 1: 586–597, 2007. [155] Flanigan F. Complex Variables: Harmonic and Analytic Functions, 2nd edn. New York: Dover Publications, 1983. [156] Fletcher R. Conjugate gradient methods for indefinite systems. In Proc. Dundee Conf. on Num. Anal. New York: Springer-Verlag, 73–89, 1975. [157] Fletcher R. Practical Methods of Optimization, 2nd edn. New York: John Wiley & Sons, 1987. [158] Forsgren A., Gill P. E., Wright M. H. Interior methods for nonlinear optimization. SIAM Review, 44: 525–597, 2002. [159] Forsythe G. E., Golub G. H. On the stationary values of a second-degree polynomial on the unit sphere. J. Soc. Ind. Appl. Math., 13(4): 1050–1068, 1965. [160] Foucart S., Lai M. J. Sparsest solutions of underdetermined linear systems via lq -minimization for 0 < q ≤ 1. Appl. Comput. Harmonic Anal., 26(3): 395–407, 2009.

References

689

[161] Frangi A. F., Niessen W. J., Viergever M. A. Three-dimensional modeling for functional analysis of cardiac images: a review. IEEE Trans. Medical Imaging, 20(1): 2–5, 2001. [162] Frankel T. The Geometry of Physics: An Introduction (with corrections and additions). Cambridge: Cambridge University Press, 2001. [163] Friedman J., Hastie T., H¨ofling H., Tibshirani R. Pathwise coordinate optimization. Ann. Appl. Statist., 1: 302–332, 2007. [164] Friedlander M. P., Hatz K. Computing nonnegative tensor factorizations. Available at http://www.optimization-online.org/DBHTML/2006/10/1494.html (2006). ¨ [165] Frobenius G. Uber Matrizen aus nicht negativen Elementen. Berlin: S.-B. Preuss Acad. Wiss. 456–477, 1912. [166] Fuhrmann D. R. An algorithm for subspace computation with applications in signal processing. SIAM J. Matrix Anal. Appl., 9: 213–220, 1988. [167] Gabay D., Mercier B. A dual algorithm for the solution of nonlinear variational problems via finite element approximations. Comput. Math. Appl., 2: 17–40, 1976. [168] Galatsanou N. P., Katsaggelos A. K. Methods for choosing the regularization parameter and estimating the noise variance in image restoration and their relation. IEEE Trans. Image Processing, 1(3): 322–336, 1992. [169] Gander W., Golub G. H., Von Matt U. A constrained eigenvalue problem. Linear Algebra Appl., 114–115: 815–839, 1989. [170] Gandy S., Recht B., Yamada I. Tensor completion and low-n-rank tensor recovery via convex optimization. Inverse Prob., 27(2): 25 010–25 028, 2011. [171] Gantmacher F. R. The Theory of Matrices. London: Chelsea Publishing, 1977. [172] Gersho A., Gray R. M. Vector Quantization and Signal Compression. Boston: Kluwer Acad. Press, 1992. [173] Gilbert A. C., Muthukrishnan M., Strauss M. J. Approximation of functions over redundant dictionaries using coherence. Proc. 14th Annu. ACM-SIAM Symp. Discrete Algorithms, Jan. 2003. [174] Gillies A. W. On the classfication of matrix generalized inverse. SIAM Review, 12(4): 573–576, 1970. [175] Gleser L. J. Estimation in a multivariate “errors in variables” regression model: large sample results. Ann. Statist., 9: 24-44, 1981. [176] Glowinski R., Marrocco A. Sur l’approximation, par elements finis d’ordre un, et la resolution, par penalisation-dualit´e, d’une classe de problems de Dirichlet non lineares. Revue Fran¸caise d’Automatique, Informatique, et Recherche Op´erationelle, 9: 41–76, 1975. [177] Goldstein T., Osher S. The split Bregman method for L1-regularized problems. SIAM J. Imaging Sci., 2(2): 323–343, 2009. [178] Golub G. H., Van Loan C. F. An analysis of the total least squares problem. SIAM J. Numer. Anal., 17: 883–893, 1980. [179] Golub G. H., Van Loan C. F. Matrix Computation, 2nd edn. Baltimore: The John Hopkins University Press, 1989. [180] Gonzaga C. C., Karas E. W. Fine tuning Nesterovs steepest descent algorithm for differentiable convex programming. Math. Program., Ser.A, 138: 141–166, 2013.

690

References

[181] Gonzales E. F., Zhang Y. Accelerating the Lee–Seung algorithm for nonnegative matrix factorization. Technical report. Department of Computational and Applied Mathematics, Rice University, 2005. [182] Gorodnitsky I. F., Rao B. D. Sparse signal reconstruction from limited data using FOCUSS: a re-weighted norm minimization algorithm, IEEE Trans. Signal Process., 1997, 45: 600–616. [183] Gourvenec S., et al. CuBatch, 2005. Available at http://www.mathworks.com. [184] Grassmann H. G. Die Ausdehnungslehre. Berlin: Enslin, 1862. [185] Gray R. M. On the asymptotic eigenvalue distribution of Toeplitz matrices. IEEE Trans Inform. Theory, 18(6): 267–271, 1972. [186] Graybill F. A. Matrices with Applications in Statistics. Balmont CA: Wadsworth International Group, 1983. [187] Graybill F. A., Meyer C. D., Painter R. J. Note on the computation of the generalized inverse of a matrix. SIAM Review, 8(4): 522–524, 1966. [188] Greville T. N. E. Some applications of the pseudoinverse of a matrix. SIAM Review, 2: 15–22, 1960. [189] Griffiths J. W. Adaptive array processing: a tutorial. Proc. IEE, Part F, 130: 137–142, 1983. [190] Grippo L., Sciandrone M. On the convergence of the block nonlinear Gauss– Seidel method under convex constraints. Operations Research Letter, 26: 127– 136, 1999. [191] Guan N., Tao D., Lou Z., Yu B. NeNMF: an optimal gradient method for nonnegative matrix factorization. IEEE Trans. Signal Processing, 60(6): 2082– 2098, 2012. [192] Guttman L. Enlargement methods for computing the inverse matrix. Ann Math Statist, 17: 336–343, 1946. [193] Haghighat M., Abdel-Mottaleb M., Alhalabi W. Discriminant correlation analysis: real-time feature level fusion for multimodal biometric recognition. IEEE Trans. Inform. Forensics Security, 11(9): 1984–1996, 2016. [194] Hale E. T., Yin W., Zhang Y. Fixed-point continuation for 1 - minimization: methodology and convergence. SIAM J. Optim., 19(3): 1107–1130, 2008. [195] Halmos P. R. Finite Dimensional Vector Spaces. New York: Springer-Verlag, 1974. [196] Han X., Wu J., Wang L., Chen Y., Senhadji L., Shu H. Linear total variation approximate regularized nuclear norm optimization for matrix completion. Abstract Appl. Anal., ID 765782, 2014. [197] Hanchez Y., Dooren P. V. Elliptic and hyperbolic quadratic eigenvalue problems and associated distance problems. Linear Algebra Appl., 371: 31–44, 2003. [198] Harshman R. A. Foundation of the PARAFAC procedure: models and conditions for an “explanatory” multi-modal factor analysis. UCLA Working papers in Phonetics, 16: 1–84, 1970. [199] Harshman R. A. Parafac2: mathematical and technical notes. UCLA Working papers in Phonetics, 22: 30–44, 1972. [200] Harshman R. A., Hong S., Lundy M. E. Shifted factor analysis – Part i: models and properties. J. Chemometrics, 17(7): 363–378, 2003. [201] Harshman R. A., Lundy M. E. Data preprocessing and the extended PARAFAC model. In Research Methods for Multimode Data Analysis (Law

References

[202] [203] [204] [205] [206] [207] [208] [209] [210] [211] [212] [213] [214] [215] [216] [217] [218] [219] [220] [221] [222]

691

H. G., Snyder C. W., Hattie J. A., McDonald R. P. eds.). New York: Praeger, pp. 216–284, 1984. Hazan T., Polak T., Shashua A. Sparse image coding using a 3d nonnegative tensor factorization. Technical Report, The Hebrew University, 2005. He X., Cai D., Niyogi P. Tensor subspace analysis. Adv. Neural Informat. Process. Syst., 18: 499, 2006. Heeg R. S., Geurts B. J. Spatial instabilities of the incompressible attachmentline flow using sparse matrix Jacobi–Davidson techniques. Appl. Sci. Res., 59: 315–329, 1998. Helmke U., Moore J. B. Optimization and Dynamical Systems. London, UK: Springer-Verlag, 1994. Hendeson H. V., Searle S. R. On deriving the inverse of a sum of matrices. SIAM Review, 23: 53–60, 1981. Henderson H. V., Searle S. R. The vec-permutation matrix, the vec operator and Kronecker products: a review. Linear Multilinear Alg., 9: 271–288, 1981. Herzog R., Sachs E. Preconditioned conjugate gradient method for optimal control problems with control and state constraints. SIAM. J. Matrix Anal. and Appl., 31(5): 2291–2317, 2010. Hestenes M. R. Multiplier and gradient methods, J. Optim. Theory Appl., 4: 303–320, 1969. Hestenes M. R., Stiefel E. Methods of conjugate gradients for solving linear systems. J. Res. Nat. Bureau of Standards, 49: 409–436, 1952. Hindi H. A tutorial on convex optimization, in Proc. 2004 American Control Conf., Boston, Masachusetts June 30-July 2, 3252–3265, 2004. Hitchcock F. L. The expression of a tensor or a polyadic as a sum of products. J. Math. Phys., 6: 164–189, 1927. Hitchcock F. L. Multilple invariants and generalized rank of a p-way matrix or tensor. J. Math. Phys., 7: 39–79, 1927. Horn R. A., Johnson C. R. Matrix Analysis. Cambridge: Cambridge University Press, 1985. Horn R. A., Johnson C. R. Topics in Matrix Analysis. Cambridge: Cambridge University Press, 1991. Hotelling H. Some new methods in matrix calculation. Ann. Math. Statist, 14: 1–34, 1943. Hotelling H. Further points on matrix calculation and simultaneous equations. Ann. Math. Statist, 14: 440–441, 1943. Howland P., Jeon M., Park H. Structure preserving dimension reduction for clustered text data based on the generalized singular value decomposition. SIAM J. Matrix Anal. Appl., 25(1): 165–179, 2003. Howland P., Park H. Generalizing discriminant analysis using the generalized singular value decomposition. IEEE Trans. Pattern Anal. Machine Intell., 26(8): 995–1006, 2004. Hoyer P. O. Non-negative matrix factorization with sparseness constraints. J. Machine Learning Res., 5: 1457–1469, 2004. Hu Y., Zhang D., Ye J., Li X., He X. Fast and accurate matrix completion via truncated nuclear norm regularization. IEEE Trans. Pattern Anal. Machine Intell., 35(9): 2117–2130, 2013. Hu S., Huang Z. H., Ling C., Qi L. On determinants and eigenvalue theory of tensors. J. Symbolic Comput., 50: 508–531, 2013.

692

References

[223] Huang B. Detection of abrupt changes of total least squares models and application in fault detection. IEEE Trans. Control Syst. Technol., 9(2): 357–367, 2001. [224] Huang L. Linear Algebra in System and Control Theory (in Chinese). Beijing: Science Press, 1984. [225] Huber P. J. Robust estimation of a location parameter. Ann. Statist., 53: 73–101, 1964. [226] Huffel S. V., Vandewalle J. The Total Least Squares Problems: Computational Aspects and Analysis. Frontiers in Applied Mathemamatics, vol. 9, Philadelphia: SIAM, 1991. [227] Hyland D. C., Bernstein D. S. The optimal projection equations for model reduction and the relationships among the methods of Wilson, Skelton and Moore. IEEE Trans. Automatic Control, 30: 1201–1211, 1985. [228] Jain P. K., Ahmad K. Functional analysis, 2nd edn. New Age International, 1995. [229] Jain S. K., Gunawardena A. D. Linear Algebra: An Interactive Approach. New York: Thomson Learning, 2003. [230] Jennings A., McKeown J. J. Matrix Computations. New York: John Wiley & Sons, 1992. [231] Johnson D. H., Dudgeon D. E. Array Signal Processing: Concepts and Techniques. Englewood Cliffs, NJ: PTR Prentice Hall, 1993. [232] Johnson L. W., Riess R. D., Arnold J. T. Introduction to Linear Algebra, 5th edn. New York: Prentice-Hall, 2000. [233] Joho M., Mathis H. Joint diagonalization of correlation matrices by using Gradient methods with application to blind signal separation. In IEEE Proc. SAM, 273–277, 2002. [234] Joho M., Rahbar K. Joint diagonalization of correlation matrices by using Newton methods with application to blind signal separation. In IEEE Proc. SAM, 403–407, 2002. [235] Jolliffe I. Principal Component Analysis. New York: Springer-Verlag, 1986. [236] Jordan C. Memoire sur les formes bilineaires. J. Math. Pures Appl., Deuxieme Serie, 19: 35–54, 1874. [237] Kalogerias D. S., Petropulu A. P. Matrix completion in colocated MIMO radar: recoverability, bounds and theoretical guarantees. IEEE Trans. Signal Processing, 62(2): 309–321, 2014. [238] Karahan E., Rojas-Lopez P. A., Bringas-Vega M. L., Valdes-Hernandez P. A., and Valdes-Sosa P. A. Tensor analysis and fusion of multimodal brain images. Proc. IEEE, 103(9): 1531–1559, 2015. [239] Karmarkar N. A new polynomial-time algorithm for linear programming. Combinatorica, 4(4): 373–395, 1984. [240] Kato T. A Short Introduction to Perturbation Theory for Linear Operators. New York: Springer-Verlag, 1982. [241] Kay S. M. Modern Spectral Estimation: Theory and Applications. Englewood Cliffs, NJ: Prentice-Hall, 1988. [242] Kayalar S., Weinert H. L. Oblique projections: formulas, algorithms, error bounds. Math. Control, Signals, Syst., 2(1): 33–45, 1989. [243] Kelley C. T. Iterative methods for linear and nonlinear equations. Frontiers in Applied Mathematics, vol. 16, Philadelphia, PA: SIAM, 1995.

References

693

[244] Khatri C. G., Rao C. R. Solutions to some functional equations and their applications to characterization of probability distributions. Sankhya: The Indian J. Stat., Series A, 30: 167–180, 1968. [245] Kiers H. A. L. Towards a standardized notation and terminology in multiway analysis. J. Chemometrics, 14: 105–122, 2000. [246] Kim J., Park H. Fast nonnegative matrix factorization: an active-set-like method and comparisons. SIAM J. Sci. Comput., 33(6): 3261–3281, 2011. [247] Kim H., Park H. Sparse non-negative matrix factorizations via alternating non-negativity-constrained least squares for microarray data analysis. Bioinformatics, 23(12): 1495–1502, 2007. [248] Kim S. J., Koh K., Lustig M., Boyd S., Gorinevsky D. An interior-point method for large-scale 1 -regularized least squares. IEEE J. Selec. Topics Signal Processing, 1(4): 606–617, 2007. [249] Klein J. D., Dickinson B. W. A normalized ladder form of residual energy ratio algorithm for PARCOR estimation via projections. IEEE Trans. Automatic Control, 28: 943–952, 1983. [250] Klema V. C., Laub A. J. The singular value decomposition: its computation and some applications. IEEE Trans. Automatic Control, 25: 164–176, 1980. [251] Klemm R. Adaptive airborne MTI: an auxiliary channel approach. Proc. IEE, Part F, 134: 269–276, 1987. [252] Kofidis E., Regalia Ph. On the best rank-1 approximation of higher-order supersymmetric tensors. SIAM J. Matrix Anal. Appl. 23: 863–884, 2002. [253] Kolda T. G. Orthogonal tensor decompositions. SIAM J. Matrix Anal. Appl., 23(1): 243–255, 2001. [254] Kolda T. G. Multilinear operators for higher-order decompositions. Sandia Report SAND2006-2081, California, Apr. 2006. [255] Kolda T. G., Bader B. W., Kenny J. P. Higher-order web link analysis using multilinear algebra. Proc. 5th IEEE Inter. Conf. on Data Mining, 242–249, 2005. [256] Kolda T. G., Bader B. W. The tophits model for higher-order web link analysis. In Workshop on Link Analysis, Counterterrorism and Security, 2006. [257] Kolda T. G., Bader B. W. Tensor decompositions and applications. SIAM Review, 51(3): 455–500, 2009. [258] Kolda T. G., Mayo J. R. Shifted power method for computing tensor eigenpairs. SIAM J. Matrix Anal. Appl., 32(4): 1095–1124, 2011. [259] Kolda T. G., Mayo J. R. An adaptive shifted power method for computing generalized tensor eigenpairs. SIAM J. Matrix Anal. Appl., 35(4): 1563–1581, 2014. [260] Komzsik L. Implicit computational solution of generalized quadratic eigenvalue problems. Finite Elemnets in Analysis and Design, 37: 799–810, 2001. [261] Kressner D., Tobler C. htucker-A MATLAB toolbox for tensors in hierarchical Tucker format. MATHICSE, EPF Lausanne. Available at http://anchp.epfl.ch/htucker/ (2012). [262] Kreutz-Delgado K. Real vector derivatives and gradients. Available at http://dsp.ucsd.edu/ kreutz/PEI05.html (2005). [263] Kreutz-Delgado K. The complex gradient operator and the calculus. 2005. Available at http://dsp.ucsd.edu/ kreutz/PEI05.html (2005). [264] Kreyszig E. Advanced Engineering Mathematics, 7th edn. New York: John Wiley & Sons, Inc., 1993.

694

References

[265] Krishna H., Morgera S. D. The Levinson recurrence and fast algorithms for solving Toeplitz systems of linear equations. IEEE Trans. Acoust., Speech, Signal Processing, 35: 839–847, 1987. [266] Kroonenberg P. The three-mode company: a company devoted to creating three-mode software and promoting three-mode data analysis. Available at http://three-mode.leidenuniv.nl/ (2000). [267] Kruskal J. B. Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics. Linear Algebra Appl., 18: 95–138, 1977. [268] Kruskal J. B. Rank, decomposition, uniqueness for 3-way and N-way arrays, in Multiway Data Analysis (Coppi and Bolasco eds.), North-Holland: Elsevier Science Publishers B.V. 7–18, 1989. [269] Kumar R. A fast algorithm for solving a Toeplitz system of equations. IEEE Trans Acoust., Speech, Signal Processing, 33: 254–267, 1985. [270] Kumaresan R. Rank reduction techniques and burst error-correction decoding in real/complex fields. In Proc. 19h Asilomar Conf. Circuits Syst. Comput., CA: Pacific Grove, 1985. [271] Kumaresan R., Tufts D. W. Estimating the angle of arrival of multiple plane waves. IEEE Trans. Aerospace Electron Syst., 19: 134–139, 1983. [272] Lahat D., Adali T., Jutten C. Multimodal data fusion: an overview of methods, challenges, prospects. Proc. IEEE, 103(9): 1449–1477, 2015. [273] Lancaster P. Lambda-Matrices and Vibrating Systems. Oxford: Pergamon Press, 1966. [274] Lancaster P. Quadratic eigenvalue problems. Linear Alg. Appl., 150: 499–506, 1991. [275] Lancaster P., Tismenetsky M. The Theory of Matrices with Applications, 2nd edn. New York: Academic, 1985. [276] Landsberg J. M. Tensors: Geometry and Applications. Graduate Studres Mathematics, vol. 128, Providence RI: American Mathematical Society, 2011. [277] Langville A. N., Meyer C. D., Albright R., Cox J., Duling D. Algorithms, initializations, convergence for the nonnegative matrix factorization. SAS Technical Report, ArXiv: 1407.7299, 2014. [278] Lasdon L. Optimization Theory for Large Systems. New York: Macmillan, 1970. [279] Lathauwer L. D. A link between the canonical decomposition in multilinear algebra and simultaneous matrix diagonalization. SIAM J. Matrix Anal. Appl., 28: 642–666, 2006. [280] Lathauwer L. D. Decompositions of a higher-order tensor in block terms – Part II: definitions and uniqueness. SIAM J. Matrix Anal. Appl., 30(3): 1033– 1066, 2008. [281] Lathauwer L. D., Moor B. D., Vandewalle J. A multilinear singular value decomposition. SIAM J. Matrix Anal. Appl., 21: 1253–1278, 2000. [282] Lathauwer L. D., Nion D. Decompositions of a higher-order tensor in block terms – Part III: alternating least squares algorithms. SIAM J. Matrix Anal. Appl., 30(3): 1067–1083, 2008. [283] Laub A. J., Heath M. T., Paige C. C., Ward R. C. Computation of system balancing transformations and other applications of simultaneous diagonalization algorithms. IEEE Trans. Automatic Control, 32: 115–122, 1987. [284] Lauritzen S. Graphical Models. London: Oxford University Press, 1996.

References

695

[285] Lay D. C. Linear Algebra and Its Applications, 2nd edn. New York: AddisonWesley, 2000. [286] Lee D. D., Seung H. S. Learning the parts of objects by non-negative matrix factorization. Nature, 401: 788–791, 1999. [287] Lee D.D., Seung H. S. Algorithms for non-negative matrix factorization. Advances in Neural Information Proc. 13 (Proc. of NIPS 2000), MIT Press, 13: 556–562, 2001. [288] Lee K., Bresler Y. Admira: atomic decomposition for minimum rank approximation. IEEE Trans. Inform. Theory, 56(9): 4402–4416, 2010. [289] Leonard I. E. The matrix exponential. SIAM Review, 38(3): 507–512, 1996. [290] Letexier D., Bourennane S., Blanc-Talon J. Nonorthogonal tensor matricization for hyperspectral image filtering. IEEE Geosci. Remote Sensing. Lett., 5(1): 3-7, 2008. [291] Levinson N. The Wiener RMS (root-mean-square) error criterion in filter design and prediction, J. Math. Phys., 25: 261–278, 1947. [292] Lewis A. S. The mathematics of eigenvalue optimization. Math. Program., 97(1-2): 155–176, 2003. [293] Li N., Kindermannb S., Navasca C. Some convergence results on the regularized alternating least-squares method for tensor decomposition. Linear Algebra Appl., 438(2): 796–812, 2013. [294] Li G., Qi L., Yu G. The Z-eigenvalues of a symmetric tensor and its application to spectral hypergraph theory. Numer. Linear Algebra Appl., 20: 1001–1029, 2013. [295] Li X. L., Zhang X. D. Non-orthogonal approximate joint diagonalization free of degenerate solution. IEEE Trans. Signal Processing, 55(5): 1803–1814, 2007. [296] Lim L. H. Singular values and eigenvalues of tensors: a variational approach. In Proc. IEEE Inter. Workshop on Computational Advances in Multi-Sensor Adaptive Processing, vol. 1: 129–132, 2005. [297] Lin C. J. Projected gradient methods for nonnegative matrix factorization. Neural Comput., 19(10): 2756–2779, 2007. [298] Lin Z., Chen M., Ma Y. The augmented Lagrange multiplier method for exact recovery of corrupted low-rank matrices. Technical Report UILU-ENG09-2215, Nov. 2009. [299] Liu H., Daniel L., Wong N. Model reduction and simulation of nonlinear circuits via tensor decomposition. IEEE Trans. Computer-Aided Design of Integ. Circuits Syst., 34(7): 1059–1069, 2015. [300] Liu J., Musialski P., Wonka P., Ye J. Tensor completion for estimating missing values in visual data. IEEE Trans. Pattern Anal. Machine Intell., 35(1): 208– 220, 2013. [301] Liu S., Trenklerz G. Hadamard, Khatri–Rao, Kronecker and other matrix products. Inter. J. Inform. Syst., 4(1): 160–177, 2008. [302] Liu X., Sidiropoulos N. D. Cramer–Rao lower bounds for low-rank decomposition of multidimensional arrays, IEEE Trans. Signal Processing, 49(9): 2074–2086, 2001. [303] Lorch E. R. On a calculus of operators in reflexive vector spaces. Trans. Amer. Math. Soc., 45: 217–234, 1939.

696

References

[304] Lu H., Plataniotis K. N., Venetsanopoulos A. N. MPCA: multilinear principal component analysis of tensor objects. IEEE Trans. Neural Networks, 19(1): 18–39, 2008. [305] Luenberger D. An Introduction to Linear and Nonlinear Programming, 2nd edn. MA: Addison-Wesley, 1989. [306] Luo Y., Liu T., Tao D., Xu C. Multiview matrix completion for multilabel image classification. IEEE Trans. Image Processing, 24(8): 2355–2369, 2015. [307] L¨ utkepohl H. Handbook of Matrices. New York: John Wiley & Sons, 1996. [308] Lyantse V. E. Some properties of idempotent operators. Troret, i Prikl. Mat, 1: 16–22, 1958. [309] MacDuffee C. C. The Theory of Matrices. Berlin: Springer-Verlag, 1933. [310] Magnus J. R, Neudecker H. The commutation matrix: some properties and applications. Ann. Statist., 7: 381–394, 1979. [311] Magnus J. R., Neudecker H. Matrix Differential Calculus with Applications in Statistics and Econometrics, revised edn. Chichester: Wiley, 1999. [312] Mahalanobis P. C. On the generalised distance in statistics. Proc. of the Natl. Inst. Sci. India, 2(1): 49–55, 1936. [313] Mallat S. G., Zhang Z. Matching pursuits with time-frequency dictionaries. IEEE Trans. Signal Processing, 41(12): 3397–3415, 1993. [314] Manolakis D., Truslow E., Pieper M., Cooley T., Brueggeman M. Detection algorithms in hyperspectral imaging systems: an overview of practical algorithms. IEEE Signal Processing. Mag., 31(1): 24–33, 2014. [315] Marshall Jr. T. G. Coding of real-number sequences for error correction: a digital signal processing problem. IEEE J. Select. Areas Commun., 2(2): 381– 392, 1984. [316] Mateos G., Bazerque J. A., Giannakis G. B. Distributed sparse linear regression. IEEE Trans. Signal Processing, 58(10): 5262–5276, 2010. [317] Mathew G., Reddy V. Development and analysis of a neural network approach to Pisarenko’s harmonic retrieval method. IEEE Trans. Signal Processing, 42: 663–667, 1994. [318] Mathew G., Reddy V. Orthogonal eigensubspaces estimation using neural networks. IEEE Trans. Signal Processing, 42: 1803–1811, 1994. [319] Matthews K. R. Linear algebra notes. Department of Mathematics, University of Queensland, MP274, 1991. [320] McLeod K., Sermesant M., Beerbaum P., Pennec X. Spatio-temporal tensor decomposition of a polyaffine motion model for a better analysis of pathological left ventricular dynamics. IEEE Trans. Medical Imaging, 34(7): 1562–1575, 2015. [321] Meerbergen K. Locking and restarting quadratic eigenvalue solvers. SIAM J. Sci. Comput., 22(5): 1814–1839, 2001. [322] Meng S., Huang L. T., Wang W. Q. Tensor decomposition and PCA jointed algorithm for hyperspectral image denoising. IEEE Signal Processing Lett., 13(7): 897–901, 2016. [323] Mesarovic V. Z., Galatsanos N. P., Katsaggelos K. Regularized constrained total least squares image restoration. IEEE Trans. Image Processing, 4(8): 1096–1108, 1995. [324] Meyer C. Matrix Analysis and Applied Linear Algebra. Philadephia: SIAM, 2000.

References

697

[325] Michalewicz Z., Dasgupta D., Le Riche R., Schoenauer M. Evolutionary algorithms for constrained engineering problems. Computers & Industrial Eng. J., 30: 851–870, 1996. [326] Micka O. J., Weiss A. J. Estimating frequencies of exponentials in noise using joint diagonalization. IEEE Trans. Signal Processing, 47(2): 341–348, 1999. [327] Million E. The Hadamard product. Available at http://buzzard.ups.edu/courses/2007spring/projects/million-paper.pdf (2007). [328] Mirsky L. Symmetric gauge functions and unitarily invariant norms. Quart J. Math. Oxford, 11: 50–59, 1960. [329] Mohanty N. Random Signal Estimation and Identification. New York: Van Nostrand Reinhold, 1986. [330] Moore E. H. General analysis, Part 1. Mem. Amer. Philos. Soc., 1: 1, 1935. [331] Moreau J. J. Fonctions convexes duales et points proximaux dans un espace Hilbertien. Rep. Paris Acad. Sci., Series A, 255: 2897–2899, 1962. [332] Moreau E. A generalization of joint-diagonalization criteria for source separation. IEEE Trans. Signal Processing, 49(3): 530–541, 2001. [333] Morup M. ERPWAVELAB, 2006. [334] Morup M., Hansen L. K., Herrmann C. S., Parnas J., Arnfred S. M. Parallel factor analysis as an exploratory tool for wavelet transformed event-related EEG. NeuroImage, 29: 938–947, 2006. [335] Murray F. J. On complementary manifofolds and projections in Lp and lp . Trans. Amer. Math. Soc, 43: 138–152, 1937. [336] Nasrabadi N. M. Hyperspectral target detection: an overview of current and future challenges. IEEE Signal Processing Magazine, 31(1); 34–44, 2014. [337] Natarajan B K. Sparse approximate solutions to linear systems. SIAM J. Comput., 24: 227–234, 1995. [338] Navasca C., Lathauwer L. D., Kindermann S. Swamp reducing technique for tensor decomposition. In Proc. 16th European Signal Processing Conf., Lausanne, Switzerland, August 25–29, 2008. [339] Neagoe V. E. Inversion of the Van der Monde Matrix. IEEE Signal Processing Letters, 3: 119–120, 1996. [340] Needell D., Vershynin R. Uniform uncertainty principle and signal recovery via regularized orthogonal matching pursuit. Found. Comput. Math., 9(3): 317–334, 2009. [341] Needell D., Vershynin R. Signal recovery from incomplete and inaccurate measurements via regularized orthogonal matching pursuit. IEEE J. Sel. Topics Signal Processing, 4(2): 310–316, 2009. [342] Needell D., Tropp J. A. CoSaMP: iterative signal recovery from incomplete and inaccurate samples. Appl. Comput. Harmonic Anal., 26(3): 301–321, 2009. [343] Nesterov Y. A method for solving a convex programming problem with rate of convergence O(1/k 2 ). Soviet Math. Doklady, 269(3): 543–547, 1983. [344] Nesterov Y. Introductory Lectures on Convex Optimization: A Basic Course. Boston MA: Kluwer Academic, 2004. [345] Nesterov Y. Smooth minimization of nonsmooth functions. Math. Program. 103(1): 127–152, 2005.

698

References

[346] Nesterov Y., Nemirovsky A. A general approach to polynomial-time algorithms design for convex programming. Report of the Central Economical and Mathematical Institute, USSR Academy of Sciences, Moscow, 1988. [347] Neumaier A. Solving ill-conditioned and singular linear systems: a tutorial on regularization. SIAM Review, 40(3): 636–666, 1998. [348] Ng L., Solo V. Error-in-variables modeling in optical fow estimation. IEEE Trans. Image Processing, 10(10): 1528–1540, 2001. [349] Ni G., Qi L., Bai M. Geometric measure of entanglement and U-eigenvalues of tensors. SIAM J. Matrix Anal. Appl., 35(1): 73–87, 2014. [350] Nievergelt Y. Total least squares: state-of-the-art regression in numerical analysis. SIAM Review, 36(2): 258–264, 1994. [351] Nionand D., Sidiropoulos N. D. Tensor algebra and multidimensional harmonic retrieval in signal processing for MIMO radar. IEEE Trans. Signal Processing, 58(11): 5693–5705, 2010. [352] Noble B., Danniel J. W. Applied Linear Algebra, 3rd edn. Englewood Cliffs, NJ: Prentice-Hall, 1988. [353] Nocedal J., Wright S. J. Numerical Optimization. New York: Springer-Verlag, 1999. [354] Nour-Omid B., Parlett B. N., Ericsson T., Jensen P. S. How to implement the spectral transformation. Math. Comput., 48: 663–673, 1987. [355] Noutsos D. Perron Frobenius theory and some extensions. Department of Mathematics, University of Ioannina, May 2008. [356] Ohlsson H., Ljung L., Boyd S. Segmentation of ARX-models using sum-ofnorms regularization. Automatica, 46: 1107–1111, 2010. [357] Ohsmann M. Fast cosine transform of Toeplitz matrices, algorithm and applications. IEEE Trans. Signal Processing, 41: 3057–3061, 1993. [358] Oja E. The nonlinear PCA learning rule in independent component analysis. Neurocomputing, 17: 25–45, 1997. [359] Olson L., Vandini T. Eigenproblems from finite element analysis of fluid– structure interactions. Comput. Struct., 33: 679–687, 1989. [360] Ortega J. M., Rheinboldt W. C. Iterative Solution of Nonlinear Equations in Several Variables. New York/London: Academic Press, 1970. [361] Oseledets I. TT-toolbox 2.2. Available at http://github.com/oseledets/TT-Toolbox/ (2012). [362] Oseledets I., Saluev T., Savostyanov D. V., Dolgov S. V. ttpy: python implementation of the tensor train, 2014. [363] Osborne M., Presnell B., Turlach B. A new approach to variable selection in least squares problems. IMA J. Numer. Anal., 20: 389–403, 2000. [364] Osher S., Burger M., Goldfarb D., Xu J., Yin W. An iterative regularization method for total variation-based image restoration. Multiscale Model. Simul., 4(2): 460–489, 2005. [365] Ottersten B., Asztely D., Kristensson M., Parkvall S. A statistical approach to subspace based estimation with applications in telecommunications. In Recent Advances in Total Least Squares Techniques and Error-in-Variables Modeling (Van Huffel S. ed.), Philadelphia, PA: SIAM, 1997. [366] Paatero P. A weighted non-negative least squares algorithm for three-way PARAFAC factor analysis. Chemometr. Intell. Lab. Syst., 38: 223–242, 1997.

References

699

[367] Paatero P., Tapper U. Positive matrix factorization: a non-negative factor model with optimal utilization of error estimates of data values. Environmetrics, 5: 111–126, 1994. [368] Paatero P., Tapper U. Least squares formulation of robust non-negative factor analysis. Chemometrics Intell. Lab, 37: 23–35, 1997. [369] Paige C. C. Computing the generalized singular value decomposition. SIAM J. Sci. Statist. Comput., 7: 1126–1146, 1986. [370] Paige C. C., Saunders N. A. Towards a generalized singular value decomposition. SIAM J. Numer. Anal., 18: 269–284, 1981. [371] Pajunnen P., Karhunen J. Least-squares methods for blind source Separation based on nonlinear PCA. Int. J. of Neural Syst., 8: 601–612, 1998. [372] Papoulis A. Probability, Random Variables and Stochastic Processes. New York: McGraw-Hill, 1991. [373] Parikh N., Bord S. Proximal algorithms. Found. Trends Optim., 1(3): 123– 231, 2013. [374] Parlett B. N. The Rayleigh quotient iteration and some generalizations for nonormal matrices. Math. Comput., 28(127): 679–693, 1974. [375] Parlett B. N. The Symmetric Eigenvalue Problem. Englewood Cliffs, NJ: Prentice-Hall, 1980. [376] Parra L., Spence C., Sajda P., Ziehe A., Muller K. Unmixing hyperspectral data. In Advances in Neural Information Processing Systems, vol. 12. Cambridge, MA: MIT Press, 942–948, 2000. [377] Pati Y. C., Rezaiifar R., Krishnaprasad P. S. Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition. In Proc. 27th Ann. Asilomar Conf. Signals Syst. Comput., vol. 1, 40–44, 1993. [378] Pauca V. P., Piper J., Plemmons R. Nonnegative matrix factorization for spectral data analysis. Linear Algebra Appl., 416(1): 29–47, 2006. [379] Pauca V. P., Shahnaz F., Berry M. W., Plemmons R. J. Text mining using non- hnegative matrix factorizations. In Proc. 4th SIAM Inter. Conf. on Data Mining, Lake Buena Vista, Florida, April 22-24, 2004. [380] Pavon M. New results on the interpolation problem for continue-time stationary-increments processes. SIAM J. Control Optim., 22: 133–142, 1984. [381] Pearson K. On lines and planes of closest fit to points in space. Phil. Mag, 559–572, 1901. [382] Pease M. C. Methods of Matrix Algebra. New York: Academic Press, 1965. [383] Peng C. Y., Zhang X. D. On recursive oblique projectors. IEEE Signal Processing Lett., 12(6): 433–436, 2005. [384] Penrose R. A. A generalized inverse for matrices. Proc. Cambridge Philos. Soc., 51: 406–413, 1955. [385] Perron O. Zur Theorie der Matrices. Math. Ann., 64(2): 248–263, 1907. [386] Petersen K. B., Petersen M. S. The Matrix Cookbook. Available at http://matrixcookbook.com (2008). [387] Pham D. T. Joint approximate diagonalization of positive defnite matrices. SIAM J. Matrix Anal. Appl., 22(4): 1136–1152, 2001. [388] Phan A. H., Tichavsky P., Cichocki A. TENSORBOX: a MATLAB package for tensor decomposition, LABSP, RIKEN, Japan. Available at http://www.bsp.brain.riken.jp/phan/tensorbox.php/ (2012).

700

References

[389] Phan A. H., Tichavsky P., Cichocki A. Low complexity damped Gauss– Newton algorithms for CANDECOMP/PARAFAC. SIAM J. Matrix Anal. Appl., 34: 126–147, 2013. [390] Phillip A., Regalia P. A., Mitra S. Kronecker propucts, unitary matrices and signal processing applications. SIAM Review, 31(4): 586–613, 1989. [391] Piegorsch W. W., Casella G. The early use of matrix diagonal increments in statistical problems. SIAM Review, 31: 428–434, 1989. [392] Piegorsch W. W., Casella G. Erratum: inverting a sum of matrices. SIAM Review, 32: 470, 1990. [393] Pintelon R., Guillaume P., Vandersteen G., Rolain Y. Analyzes, development and applications of TLS algorithms in frequency domain system identification. In Recent Advances in Total Least Squares Techniques and Error-in-Variable Modeling (Van Huffel S. ed.), Philadelphia, PA: SIAM, 1997. [394] Pisarenko V. F. The retrieval of harmonics from a covariance function. Geophysics, J. Roy. Astron. Soc., 33: 347–366, 1973. [395] Piziak R., Odell P. L. Full rank factorization of matrices. Math. Mag., 72(3): 193–202, 1999. [396] Polyak B. T. Introduction to Optimization. New York: Optimization Software Inc., 1987. [397] Poularikas A. D. The Handbook of Formulas and Tables for Signal Processing. New York: CRC Press, Springer, IEEE Press, 1999. [398] Powell M. J. D. A method for nonlinear constraints in minimization problems. In Optimization (Fletcher R. ed.), New York: Academic Press, 283–298, 1969. [399] Powell M. J. D. On search directions for minimization algorithms. Math. Progr., 4: 193–201, 1973. [400] Price C. The matrix pseudoinverse and minimal variance estimates. SIAM Review, 6: 115–120, 1964. [401] Pringle R. M., Rayner A. A. Generalized Inverse of Matrices with Applications to Statistics. London: Griffin 1971. [402] Puig A. T., Wiesel A., Fleury G., Hero A. O. Multidimensional shrinkagethresholding operator and group LASSO penalties. IEEE Signal Processing Lett., 18(6): 343–346, 2011. [403] Qi L. Eigenvalues of a real supersymmetric tensor. J. Symb. Comput. 40: 1302–1324, 2005. [404] Qi L., Teo K. L. Multivariate polynomial minimization and its application in signal processing. J. Global Optim., 26: 419–433, 2003. [405] Qi L., Wang Y., Wu E. X. D-eigenvalues of diffusion kurtosis tensors, J. Comput. Appl. Math., 221: 150–157, 2008. [406] Qi L., Xu C., Xu Y. Nonnegative tensor factorization, completely positive tensors, hierarchical elimination algorithm. SIAM J. Matrix Anal. Appl., 35(4): 1227–1241, 2014. [407] Rado R. Note on generalized inverse of matrices. Proc. Cambridge Philos. Soc., 52: 600–601, 1956. [408] Ram I., Elad M., Cohen I. Image processing using smooth ordering of its patches. IEEE Trans. Image Process., 22(7): 2764–2774, 2013. [409] Rao C. R. Estimation of heteroscedastic variances in linear models. J. Amer. Statist. Assoc., 65: 161–172, 1970. [410] Rao C. R., Mitra S. K. Generalized Inverse of Matrices. New York: John Wiley & Sons, 1971.

References

701

[411] Rayleigh L. The Theory of Sound, 2nd edn. New York: Macmillian, 1937. [412] Recht B., Fazel M., Parrilo P. A. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Review, 52(3): 471–501, 2010. [413] Regalia P. A., Mitra S. K. Kronecker products, unitary matrices and signal processing applications. SIAM Review, 31(4): 586–613, 1989. [414] Riba J., Goldberg J., Vazquez G. Robust beamforming for interference rejection in mobile communications. IEEE Trans. Signal Processing, 45(1): 271– 275, 1997. [415] Robeva E. Orthogonal decomposition of symmetric tensors. SIAM J. Matrix Anal. Appl., 37(1): 86–102, 2016. [416] Roos C. A full-Newton step O(n) infeasible interior-point algorithm for linear optimization. SIAM J. Optim., 16(1): 1110–1136, 2006. [417] Roos C., Terlaky T., Vial J. Ph. Theory and Algorithms for Linear Optimization: An Interior-Point Approach. Chichester, UK: John Wiley & Sons, 1997. [418] Roth V. The Generalized LASSO. IEEE Trans. Neural Networks, 15(1): 16– 28, 2004. [419] Roy R., Kailath T. ESPRIT – Estimation of signal parameters via rotational invariance techniques. IEEE Trans. Acoust., Speech, Signal Processing, 37: 297–301, 1989. [420] Rudin L., Osher S., Fatemi E. Nonlinear total variation based noise removal algorithms. Physica D, 60: 259–268, 1992. [421] Saad Y. The Lanczos biothogonalization algorithm and other oblique projection methods for solving large unsymmetric systems. SIAM J. Numer. Anal., 19: 485-506, 1982. [422] Saad Y. Numerical Methods for Large Eigenvalue Problems. New York: Manchester University Press, 1992. [423] Salehi H. On the alternating projections theorem and bivariate stationary stichastic processes. Trans. Amer. Math. Soc., 128: 121–134, 1967. [424] Samson C. A unified treatment of fast algorithms for identification. Inter. J. Control, 35: 909–934, 1982. [425] Scales L. E. Introduction to Non-linear Optimization. London: Macmillan, 1985. [426] Schmidt R. O. Multiple emitter location and signal parameter estimation. Proc RADC Spectral Estimation Workshop, NY: Rome, 243–258, 1979. [427] Schmidt R. O. Multiple emitter location and signal parameter estimation. IEEE Trans. Antenna Propagat., 34: 276–280, 1986. [428] Schott J. R. Matrix Analysis for Statistics. Wiley: New York, 1997. [429] Schoukens J., Pintelon R., Vandersteen G., Guillaume P. Frequency-domain system identification using nonparametric noise models estimated from a small number of data sets. Automatica, 33(6): 1073–1086, 1997. [430] Schutz B. Geometrical Methods of Mathematical Physics. Cambridge: Cambridge University Press, 1980. [431] Schultz T., Seidel H. P. Estimating crossing fibers: a tensor decomposition approach. IEEE Trans. Visualization Comput. Graphics, 14: 1635–1642, 2008. [432] Scutari G., Palomar D. P., Facchinei F., Pang J. S. Convex optimization, game theory, variational inequality theory. IEEE Signal Processing Mag., 27(3): 35– 49, 2010.

702

References

[433] Searle S. R. Matrix Algebra Useful for Statististics. New York: John Wiley & Sons, 1982. [434] Shahnaz F., Berry M. W., Pauca V. P., Plemmons R. J. Document clustering using nonnegative matrix factorization. Inf. Processing & Management, 42(2): 373–386, 2006. [435] Shashua A., Levin A. Linear image coding for regression and classification using the tensor-rank principle. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, 2001. [436] Shashua A., Zass R., Hazan T. Multi-way clustering using super-symmetric nonnegative tensor factorization. In Proc. European Conf. on Computer Vision, Graz, Austria, May 2006. [437] Shavitt I., Bender C. F., Pipano A., Hosteny R. P. The iterative calculation of several of the lowest or highest eigenvalues and corresponding eigenvectors of very large symmetric matrices. J. Comput. Phys., 11: 90–108, 1973. [438] Sherman J., Morrison W. J. Adjustment of an inverse matrix corresponding to changes in the elements of a given column or a given row of the original matrix (abstract). Ann. Math. Statist., 20: 621, 1949. [439] Sherman J., Morrison W. J. Adjustment of an inverse matrix corresponding to a change in one element of a given matrix. Ann. Math. Statist., 21: 124–127, 1950. [440] Shewchuk J. R. An introduction to the conjugate gradient method without the agonizing pain. Available at http://quake-papers/painless-conjugategradient-pics.ps (1994). [441] Shivappa S. T., Trivedi M. M., Rao B. D. Audiovisual information fusion in human-computer interfaces and intelligent environments: a survey. Proc. IEEE, 98(10): 1692–1715, 2010. [442] Sidiropoulos N. D., Bro R. On the uniqueness of multilinear decomposition of N-way arrays. J. Chemometrics, 14: 229–239, 2000. [443] Sidiropoulos N. D., Bro R. On communication diversity for blind identifiability and the uniqueness of low-rank decomposition of N-way arrays. In Proc. Int. Conf. Acoust. Speech Signal Processing, vol. 5: 2449–2452, 2000. [444] Sidiropoulos N. D., Bro R., Giannakis G. B. Parallel factor analysis in sensor array processing. IEEE Trans. Signal Processing, 48(8): 2377–2388, 2000. [445] Sidiropoulos N. D., Liu H. Identifiability results for blind beamforming in incoherent multipath with small delay spread. IEEE Trans. Signal Processing, 49(1): 228–236, 2001. [446] Sidiropoulos N. D., Papalexakis E. E., Faloutsos C. Parallel randomly compressed cubes: a scalable distributed architecture for big tensor decomposition. IEEE Signal Processing Mag., 33(9): 57–70, 2014. [447] Silva V., Lim L. H. Tensor rank and the ill-posedness of the best low-rank approximation problem, SIAM J. Matrix Anal. Appl., 30(3): 1084–1127, 2008. [448] Simon J. C. Patterns and Operators: The Foundations and Data Representation. North Oxford Academic Publishers, 1986. [449] Solomentsev E. D. Euclidean space. Encyclopedia of Mathematics. Springer, 2011. [450] Sorber L., Barel M. V., Lathauwer D. L. Tensorlab v2.0. Available at http://www.tensorlab.net/ (2014).

References

703

[451] Sorensen M., Lathauwer L. D. Blind signal separation via tensor decomposition with Vandermonde factor: canonical polyadic decomposition. IEEE Trans. Signal Processing, 61(22): 5507–5519, 2013. [452] Spivak M. A Comprehensive Introduction to Differential Geometry, 2nd edn. (five volumes). New York: Publish or Perish Press, 1979. [453] Stein D. W. J., Beaven S. G., Hoff L. E., Winter E. M., Schaum A. P., Stocker A. D. Anomaly detection from hyperspectral imagery. IEEE Signal Processing Mag., 19(1): 58–69, 2002. [454] Stewart G. W. On the early history of the singular value decomposition. SIAM Review, 35(4): 551–566, 1993. [455] Stewart G. W., Sun J. G. Matrix Perturbation Theory. New York: Academic Press, 1990. [456] Stiefel E. Richtungsfelder und ferparallelismus in n-dimensionalem mannig faltigkeiten. Commentarii Math Helvetici, 8: 305–353, 1935. [457] Stoica P., Sorelius J., Cedervall M., S¨oderstr¨ om T. Error-in-variables modeling: an instrumental variable approach. In Recent Advances in Total Least Squares Techniques and Error-in-Variables Modeling (Van Huffel S. ed.). Philadelphia, PA: SIAM, 1997. [458] Sun J., Zeng H., Liu H., Lu Y., Chen Z. Cubesvd: a novel approach to personalized web search. In Proc. 14th Inter. Conf. on World Wide Web, 652–662, 2005. [459] Sun Y., Gao J., Hong X., Mishra B., Yin B. Heterogeneous tensor decomposition for clustering via manifold optimization. IEEE Trans. Pattern Anal. Machine Intell., 38(3): 476–489, 2016. [460] Syau Y. R. A note on convex functions. Inter. J. Math. & Math. Sci. 22(3): 525–534, 1999. [461] Takeuchi K., Yanai H., Mukherjee B. N. The Foundations of Multivariante Analysis. New York: Wiley, 1982. [462] Tibshirani R. Regression shrinkage and selection via the lasso. J. R. Statist. Soc. B, 58: 267–288, 1996. [463] Tichavsky P., Phan A. H., Koldovsky Z. Cramer–Rao-induced bounds for CANDECOMP/PARAFAC tensor decomposition. IEEE Trans. Signal Processing, 61(8): 1986–1997, 2013. [464] Tikhonov A. Solution of incorrectly formulated problems and the regularization method, Soviet Math. Dokl., 4: 1035–1038, 1963. [465] Tikhonov A., Arsenin V. Solution of Ill-Posed Problems. New York: Wiley, 1977. [466] Tisseur F., Meerbergen K. Quadratic eigenvalue problem. SIAM Review, 43(2): 235–286, 2001. [467] Toeplitz O. Zur Theorie der quadratischen und bilinearen Formen von unendlichvielen Ver¨anderlichen. I Teil: Theorie der L-Formen, Math Annal, 70: 351–376, 1911. [468] Tomasi G., Bro R. A comparison of algorithms for fitting the PARAFAC model. Computat. Statist. Data Anal., 50(7): 1700–1734, 2006. [469] Tou J. T., Gonzalez R. C. Pattern Recognition Principles. London: AddisonWesley Publishing Comp, 1974. [470] Tropp J. A. Greed is good: algorithmic results for sparse approximation. IEEE Trans. Inform. Theory, 50(10): 2231–2242, 2004.

704

References

[471] Tropp J. A. Just relax: convex programming methods for identifying sparse signals in noise. IEEE Trans. Inform. Theory, 52(3): 1030–1051, 2006. [472] Tropp J. A., Wright S. J. Computational methods for sparse solution of linear inverse problems. Proc. IEEE, 98(6): 948–958, 2010. [473] Tsallis C. Possible generalization of Boltzmann–Gibbs statistics. J. Statist. Phys., 52: 479–487, 1988. [474] Tsallis C. The nonadditive entropy Sq and its applications in physics and elsewhere: some remarks. Entropy, 13: 1765–1804, 2011. [475] Tucker L. R. Implications of factor analysis of three-way matrices for measurement of change. In Problems in Measuring Change (Harris C. W. ed.), Wisconsin: University of Wisconsin Press, 122–137, 1963. [476] Tucker L. R. The extension of factor analysis to three-dimensional matrices. In Contributions to Mathematical Psychology (Gulliksen H., Frederiksen N. eds.), New York: Holt, Rinehart & Winston, 109–127, 1964. [477] Tucker L. R. Some mathematical notes on three-mode factor analysis. Psychometrika, 31: 279–311, 1966. [478] Turk M. Multimodal interaction: a review. Pattern Recog. Lett., 36(1): 189– 195, 2014. [479] van der Kloot W. A., Kroonenberg P. M. External analysis with three-mode principal component models, Psychometrika, 50: 479-494, 1985. [480] van der Veen A. J. Joint diagonalization via subspace fitting techniques. Proc. 2001 IEEE Inter. Conf. on Acoust., Speech, Signal Processing, vol. 5: 2773– 2776, 2001. [481] Van Huffel S. (ed.). Recent Advances in Total Least Squares Techniques and Error-in-Variables Modeling. Philadelphia, PA: SIAM, 1997. [482] Van Huffel S. TLS applications in biomedical signal processing. In Recent Advances in Total Least Squares Techniques and Error-in-Variables Modeling (Van Huffel S. ed.). Philadelphia, PA: SIAM, 1997. [483] Van Loan C. F. Generalizing the singular value decomposition. SIAM J. Numer. Anal., 13: 76–83, 1976. [484] Van Loan C. F. Matrix computations and signal processing. In Selected Topics in Signal Processing (Haykin S. ed.). Englewood Cliffs: Prentice-Hall, 1989. [485] van Overschee P., De Moor B. Subspace Identification for Linear Systems. Boston, MA: Kluwer, 1996. [486] Van Huffel S., Vandewalle J. Analysis and properties of the generalized total least squares problem Ax = b when some or all columns in A are subject to error. SIAM J. Matrix Anal. Appl., 10: 294–315, 1989. [487] Vandaele P., Moonen M. Two deterministic blind channel estimation algorithms based on oblique projections. Signal Processing, 80: 481–495, 2000. [488] Vandenberghe L. Lecture Notes for EE236C (Spring 2011-12), UCLA. [489] Vanderbei R. J. An interior-point algorithm for nonconvex nonlinear programming. Available at http://orfe.princeton.edu/rvdb/pdf/talks/level3/nl.pdf (2000) [490] Vanderbei R. J., Shanno D. F. An interior-point algorithm for nonconvex nonlinear programming. Comput. Optim. Appl., 13(1-3): 231–252, 1999. [491] Vasilescu M. A. O., Terzopoulos D. Multilinear analysis of image ensembles: TensorFaces. In Proc. European Conf. on Computer Vision, Copenhagen, Denmark, 447–460, 2002.

References

705

[492] Vasilescu M. A. O., Terzopoulos D. Multilinear image analysis for facial recognition. In Proc. Inter. Conf. on Pattern Recognition, Quebec City, Canada, 2002. [493] Veen A. V. D. Algebraic methods for deterministic blind beamforming. Proc. IEEE, 86: 1987–2008, 1998. [494] Veganzones M. A., Cohen J. E., Farias R. C., Chanussot J., Comon P. Nonnegative tensor CP decomposition of hyperspectral data. IEEE Trans. Geosci. Remote Sensing, 54(5): 2577–2588, 2016. [495] Viberg M., Ottersten B. Sensor array processing based on subspace fitting. IEEE Trans. Signal Processing, 39: 1110–1121, 1991. [496] Vollgraf R., Obermayer K. Quadratic optimization for simultaneous matrix diagonalization. IEEE Trans. Signal Processing, 54(9): 3270–3278, 2006. [497] W¨achter A., Biegler L. T. On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming. Math. Progr. Ser. A, 106(1): 25–57, 2006. [498] Wang H. Coordinate descent algorithm for covariance graphical lasso. Stat. Comput., 24: 521–529, 2014. [499] Wang H., Ahuja N. Compact representation of multidimensional data using tensor rank-one decomposition. In Proc. Inter. Conf. on Pattern Recognition, Vol. 1, 44–47, 2004. [500] Wang H., Amini A. Cardiac motion and deformation recovery from MRI: a review. IEEE Trans. Medical Imaging, 31(2): 487–503, 2012. [501] Wang W., Sanei S., Chambers J. A. Penalty function-based joint diagonalization approach for convolutive blind separation of nonstationary sources. IEEE Trans. Signal Processing, 53(5): 1654–1669, 2005. [502] Watkins D. S. Understanding the QR algorithm. SIAM Review, 24(4): 427– 440, 1982. [503] Watson G. A. Characterization of the subdifferential of some matrix norms, Linear Algebra Appl., 170: 33–45, 1992. [504] Wax M., Sheinvald J. A least squares approach to joint diagonalization. IEEE Signal Processing Lett., 4(2): 52–53, 1997. [505] Weiss A. J., Friedlander B. Array processing using joint diagonalization. Signal Processing, 50(3): 205–222, 1996. [506] Welling M., Weber M. Positive tensor factorization. Pattern Recognition Lett., 22: 1255–1261, 2001. [507] Whiteford H. A., Degenhardt L.., et al. Global burden of disease attributable to mental and substance use disorders: findings from the Global Burden of Disease Study 2010. Lancet, 382(9904): 1575–1586, 2013. [508] Wiesel A., Eldar Y. C., Beck A. Maximum likelihood estimation in linear models with Gaussian model matrix, IEEE Signal Processing Lett., 13: 292– 295, 2006. [509] Wiesel A., Eldar Y. C., Yeredor A. Linear regression with Gaussian model uncertainty: algorithms and bounds, IEEE Trans. Signal Processing, 56: 2194– 2205, 2008. [510] Wilkinson J. H. The Algerbaic Eigenvalue Problem. Oxford, UK: Clarendon Press, 1965. [511] Wirtinger W. Zur formalen Theorie der Funktionen von mehr Komplexen vromnderlichen. Math. Ann., 97: 357–375, 1927.

706

References

[512] Wolf J. K. Redundancy, the discrete Fourier transform, and impulse noise cancellation. IEEE Trans. Commun, 31: 458–461, 1983. [513] Woodbury M. A. Inverting modified matrices. Memorandum Report 42, Statistical Research Group, Princeton, 1950. [514] Wright J., Ganesh A., Rao S., Peng Y., Ma Y. Robust principal component analysis: exact recovery of corrupted low-rank matrices via convex optimization. In Proc. Adv. Neural Inf. Processing Syst., 87(4): 20:3–20:56, 2009. [515] Xu H., Caramanis C., Mannor S. Robust regression and Lasso. IEEE Trans. Inform. Theory, 56(7): 3561–3574, 2010. [516] Xu G., Cho Y., Kailath T. Application of fast subspace decomposition to signal processing and communication problems. IEEE Trans. Signal Processing, 42: 1453–1461, 1994. [517] Xu G., Kailath T. A fast algorithm for signal subspace decomposition and its performance analysis. In Proc. Inter. Conf. of Acoust., Speech and Signal Processing, Toronto, Canada, 3069–3072, 1991. [518] Xu G., Kailath T. Fast subspace decomposition. IEEE Trans. Signal Processing, 42: 539–551, 1994. [519] Xu G., Kailath T. Fast estimation of principal eigenspace using Lanczos algorithm. SIAM J. Matrix Anal. Appl. 15(3): 974–994, 1994. [520] Xu L., Oja E., Suen C. Modified Hebbian learning for curve and surface fitting. Neural Networks, 5: 441–457, 1992. [521] Yan H., Paynabar K., Shi J. Image-based process monitoring using low-rank tensor decomposition. IEEE Trans. Automation Sci. Eng., 12(1): 216–227, 2015. [522] Yang B. Projection approximation subspace tracking. IEEE Trans. Signal Processing, 43: 95–107, 1995. [523] Yang B. An extension of the PASTd algorithm to both rank and subspace tracking. IEEE Signal Processing Lett., 2(9): 179–182, 1995. [524] Yang J., Yuan X. Linearized augmented Lagrangian and alternating direction methods for nuclear norm minimization. Math. Comput., 82: 301–329, 2013. [525] Yang J. F., Kaveh M. Adaptive eigensubspace algorithms for direction or frequency estimation and tracking. IEEE Trans. Acoust., Speech, Signal Processing, 36: 241–251, 1988. [526] Yang X., Sarkar T. K., Arvas E. A survey of conjugate gradient algorithms for solution of extreme eigen-problems of a symmetric matrix. IEEE Trans. Acoust., Speech, Signal Processing, 37: 1550–1556, 1989. [527] Yang Y., Yang Q. Further results for Perron–Frobenius theorem for nonnegative tensors. SIAM J. Matrix Anal. Appl., 31: 2517–2530, 2010. [528] Yang W., Gao Y., Shi Y., Cao L. MRM-Lasso: a sparse multiview feature selection method via low-rank analysis. IEEE Trans. Neural Networks Learning Syst., 26(11): 2801-2815, 2015. [529] Yeniay O., Ankara B. Penalty function methods for constrained optimization with genetic algorithms. Math. Comput. Appl., 10(1): 45–56, 2005. [530] Yeredor A. Non-orthogonal joint diagonalization in the least squares sense with application in blind source separation. IEEE Trans. Signal Processing, 50(7): 1545–1553, 2002. [531] Yeredor A. Time-delay estimation in mixtures. In Proc. 2003 IEEE Inter. Conf. on Acoustics, Speech, and Signal Processing, 5: 237–240, 2003.

References

707

[532] Yin W., Osher S., Goldfarb D., Darbon J. Bregman iterative algorithms for l1 -minimization with applications to compressed sensing. SIAM J. Imaging Sci., 1(1): 143–168, 2008. [533] Yokota T., Zhao Q., Cichocki A. Smooth PARAFAC decomposition for tensor completion. IEEE Trans. Signal Processing, 64(20): 5423–5436, 2016. [534] Youla D. C. Generalized image restoration by the method of altenating projections. IEEE Trans. Circuits Syst., 25: 694–702, 1978. [535] Yu X., Tong L. Joint channel and symbol estimation by oblique projections. IEEE Trans. Signal Processing, 49(12): 3074–3083, 2001. [536] Yu Y. L. Nesterov’s optimal gradient method. 2009. Available at http://www.webdocs.cs. ualberta.ca/yaoliang/mytalks/NS.pdf (2009). [537] Yuan M., Lin Y. Model selection and estimation in regression with grouped variables. J. Royal Statist. Soc., Series B, 68(1): 49–67, 2006. [538] Zass R., Shashua A. Nonnegative sparse PCA. In Proc. Conf. on Neural Information Processing Systems, Vancuver, Canada, 2006. [539] Zdunek R., Cichocki A. Nonnegative matrix factorization with constrained second-order optimization. Signal Processing, 87(8): 1904–1916, 2007. [540] Zha H. The restricted singular value decomposition of matrix triplets. SIAM J. Matrix Anal. Appl., 12: 172–194, 1991. [541] Zhang X., Wen G., Dai W. A tensor decomposition-based anomaly detection algorithm for hyperspectral image. IEEE Trans. Geosci. Remote Sensing, 54(10): 5801–5820, 2016. [542] Zhang X. D. Numerical computations of left and right pseudo inverse matrices (in Chinese). Kexue Tongbao, 7(2): 126, 1982. [543] Zhang X. D., Liang Y. C. Prefiltering-based ESPRIT for estimating parameters of sinusoids in non-Gaussian ARMA noise. IEEE Trans. Signal Processing, 43: 349–353, 1995. [544] Zhang Z. Y. Nonnegative matrix factorization: models, algorithms and applications. Intell. Syst. Ref. Library, 24(6): 99–134, 2012. [545] Zhou G., Cichocki A. TDALAB: tensor decomposition laboratory, LABSP, Wako-shi, Japan. Available at http://bsp.brain.riken.jp/TDALAB/ (2013). [546] Zhou H. TensorReg toolbox for Matlab, 2013.

Index

A-conjugacy, 210 A-invariance, 515 A-orthogonality, 210 AB divergence, 362 absolute minimum point, 184 AC–DC algorithm, 501 alternating columns (AC) phase, 501 diagonal centers (DC) phase, 501 active constraint, 403 active set, 403 adaptive convexity parameter, 222 additivity, 364 adjoint, 95 affine, 202 combination, 202 function, 201 image, 201 algebraic multiplicity, 416 algebraic vector, 5 alpha-divergence (α divergence), 362 alternating constrained least squares (ACLS) method, 375 direction multiplier method (ADMM), 248 scaled ADMM, 251 least squares (ALS) method, 324 nonnegative least square (ANLS) algorithm, 374 regularization nonnegative least squares (ARNLS), 375 annihilation polynomial, 447 anti-regularized method, 322 anti-Tikhonov regularization method, 322 APG stopping criterion (APGstop), 394 approximate joint diagonalization, 294 approximation, 184 approximation solution sequence, 221 Armijo condition, 253

Aronszajn synthetic formula, 571 array, 590 two-way, 590 multi-way, 590 one-way, 590 associative law of addition, 23 associative law of scalar multiplication, 23 atoms, 84 augmented Lagrange multiplier method, 241 Lagrangian function, 241 matrix, 13 auxiliary optimization problem, 238 axiom on addition, 23 on scalar multiplication, 23 backtracking line search, 253 backward identity matrix, 100 iteration, 234 prediction error vector, 565 prediction value vector, 566 balanced matrix, 436 reflexivity, 436 symmetry, 436 transitivity, 436 balanced reduction, 436 Banach space, 35 band matrix, 107 barrier function, 239 exponential, 239 Fiacco–McCormick logarithmic, 239 inverse, 239 logarithmic, 239 power-function, 239 barrier method, 240 barrier parameter, 239

708

Index Barzilai–Borwein gradient projection (BBGP) method, 396 basis, 519 basis pursuit (BP), 383 basis pursuit de-noising, 224, 384, 404 basis vector, 6 best linear mean square estimate, 554 beta-divergence, 363 between-class scatter matrix, 295 biconjugate gradient algorithm, 211 bi-Lanczos iteration, 546 bilinear analysis, 625 model, 625 binary Kronecker product, 117 bit-reversed index vector, 118 black-box optimization, 205 blind matrix equation, 315 source separation (BSS) problem, 495 block inversion lemma, 63 matrix, 6 Boltzmann–Gibbs entropy, 364 bound-constrained quadratic program (BCQP) problem, 394 bounded-variation function, 404 norm, 404 Bregman distance, 363, 405 iteration, 405 iterative algorithm, 406 Broyden–Fletcher–Goldfarb–Shanno (BFGS) method, 252 canonical form, 423 inner product, 27 capacitance matrix, 60 Cartesian product, 22 Cauchy–Riemann condition, 158 equation, 158 inequality, 29 Cayley–Hamilton theorem, 447 center path, 262 centering, 660 condition, 262 matrix, 111 centrality direction, 265 chain rule, 134

characteristic equation, 52 matrix, 415 characteristic polynomial, 416 root, 416 circle phenomenon, 327 ciphertext, 86 class discrimination space, 481 classical Fiacco–McCormick merit function, 267 closed ball, 201 neighborhood, 185 closure for addition, 23 for scalar multiplication, 23 clustering, 32 code book, 357 vector, 357 coding matrix, 86 cofactor, 49 cogradient matrix, 133 operator, 133, 163 vector, 133, 163 coherent, 43 column space, 517 span, 517 stacking, 79 combinatorial optimization problem, 224 optimization solution, 228 commutative law of addition, 23 companion form, 488 first, 488 second, 488 companion matrix, 510 complementary slackness, 262 complete orthogonal basis, 84 vector space, 35 complex analytic function, 158 chain rule, 161 cogradient operator, 164 conjugate cogradient operator, 164 conjugate gradient operator, 164 conjugate gradient vector, 164 conjugate partial derivative, 160 differential rule, 161 differentiable, 158 gradient, 157

709

710 gradient matrix, 167 gradient operator, 164 gradient vector, 164 Hessian matrix, 175 full, 176 identification, 177 main, 176 part, 176 matrix differential, 163 matrix equation solving, 19 partial derivative, 160 complexity, 215 composite system identification, 297 compression mapping, 386 compression ratio, 461 compression sampling matching pursuit, 392 condition number, 273 conic combination, 202 conjugacy, 210 conjugate cogradient operator, 163 cogradient vector, 163 direction algorithm, 211 gradient, 157 gradient algorithm, 211 gradient matrix, 167 identification, 169 gradient vector, 164 Jacobian matrix, 167 identification, 169 symmetry, 27 consistent equation, 56 constant vector, 5 constrained convex optimization problem, 237 minimization problem, 237 nonnegative matrix factorization (CNMF), 374 TLS problem, 345 convergence rate, 215 cubical, 216 limit, 215 linear, 215 local, 216 logarithmical, 216 Q, 215 quadratical, 215 quotient, 215 sublinear, 215 superlinear, 215 α-order Q, 215 convex combination, 202

Index cone, 201 hull, 201 function, 202 strictly, 202 strongly, 202 optimization, 200 relaxation, 383 set, 200 convexity parameter, 203 coordinate, 513 coordinate system, 513 correlation coefficient, 43 cost function, 183, 198 coupled optimization problems, 324 covariant operator, 133 critical point, 184, 325 cross-correlation matrix, 42 cross-covariance matrix, 42 CuBatch, 678 cubical tensor, 592 curvature, 197 curvature direction, 197 negative curvature, 197 positive curvature, 197 zero curvature, 197 CP decomposition, 615, 628 alternating least squares algorithm, 638 regularized, 641 data array compression algorithm, 663 data centering, 660 compression, 461 fusion, 642 least squares (DLS) method, 320 least squares solution, 321 scaling, 660 data-driven approach, 643 Davidon–Fletcher–Powell (DFP) method, 252 Decell theorem, 447 decision variable, 198 decomposed tensor set, 613 definition domain, 199 deformed exponential, 366 logarithm, 366 degenerate solutions, 501 derogatory matrix, 417 descent direction, 206 method, 206 step, 206

Index determinant, 49 determinant divisor, 437 determinant rank, 437 diagonal function, 498 matrix function, 499 vector function, 498 matrix, 5 main diagonal, 5 diagonal elements, 5 method, 49 diagonalization theorem, 426 diagonalizing matrix, 497 dictionary matrix, 85 differential, 141 dimension of tensor mode, 519 dimension reduction method, 296 dimensionality reduction, 461 direct sum, 71 decomposition, 572 of subspaces, 513 direct product, 75 discrete cosine transform (DCT), 125 Fourier transform (DFT), 114 Kauhunen–Loeve (KL) expansion, 460 Karhunen–Loeve transform, 458 disjoint, 513 dissimilarity, 33 distributed encoding, 461 distributed optimization problems, 248 divergence, 361 diversity, 642 dual ascent method, 237 feasibility, 249 gap, 244 normal matrix, 267 objective function, 243 problem, 242 residual, 250 variable, 242 vector, scaled, 250 Duncan–Guttman inversion formula, 61 dyadic decomposition, 275 dynamic scene reconstruction, 303 echelon matrix, 14 effective rank, 281 eigenanalysis, 413 eigenpair, 415 eigenpolynomial, 416 eigenspace, 515

eigensystem, 463 eigenvalue, 52, 413 defective, 417 degenerated, 417 multiple, 416 semi-single, 417 single, 416 eigenvalue decomposition, 414 eigenvalue–eigenvector equation, 414 eigenvalue inequality, 74 eigenvalue multiplicity, 415 eigenvector, 52, 413 left, 419 right, 419 properties, 420 elementary divisor, 445 operations, 13 row matrices, 435 row transformation, 13 elementwise product, 73 encoding matrix, 86 region, 357 variable matrix, 360 energy, 85 entrywise norms, 39 equality constraint, 198 equivalent matrix pencil, 466 spanning set, 512 subspaces, 533 systems of equations, 13 ERPWAVELAB, 678 ESPRIT, 468 basic algorithm 1, 470 basic algorithm 2, 473 TLS-ESPRIT, 470 essentially equal matrices, 496 Euclidean ball, 209 distance, 35 length, 35 space, 35 exchange matrix, 102 exchangeable matrices, 128 expectation-maximization (EM) algorithm, 367 explanatory variables, 85 explicit constraint, 199 extensive entropy, 364 extracting vector, 566 extremum, 186

711

712

Index

extremum point, 186 first-order necessary condition, 187 face recognition, 86, 297 factor loading, 625 matrix, 626 vector, 626 factor priors, 671 factor score, 625 matrix, 626 vector, 626 FAL stopping criterion FALstop, 394 fast approximate joint diagonalization algorithm, 502 Fourier transform (FFT) algorithm, 119 iterative soft-thresholding algorithm (FISTA), 236 subspace decomposition algorithm, 545 feature extraction, 461 feasible point, 199 feasible point algorithm, 262 feasible set, 199, 238 feasibility direction, 265 Fejer theorem, 73 field, 22 complex number, 22 rational number, 22 real number, 22 first-order augmented Lagrangian (FAL) algorithm, 394 black-box optimization, 205 differential, 141 primal-dual interior point method, 262 Fisher inequality, 51 measure, 480 forgetting factor, 540 formal partial derivative, 159 forward iteration, 234 prediction error vector, 565 problem, 26 Fourier matrix, 115 Frobenius norm ratio, 282 frontal slice, 594 frontal slice matrix, 594 full column rank matrix, 58 full-rank matrix, 58 decomposition, 69 polynomial matrix, 435

full relaxation, 208 full row rank matrix, 58 full singular value decomposition, 276 function vector, 5 Gauss elimination method, 16 for matrix inversion, 19 Gauss–Jordan elimination method, 16 Gauss–Markov theorem, 317 Gauss–Seidel (GS) method, 324 proximate point version, 327 Gaussian random vector, 44 characteristic function, 44 complex, 44 joint probability density function, 44 properties, 45 general solution, 17 generalized Bezout theorem, 487 characteristic equation, 464 characteristic polynomial, 464 complex partial derivative, 162 eigenpair, 464 eigenproblem adaptive power method, 657 eigensystem, 463 eigenvalue, 463 eigenvalue decomposition, 463 eigenvalue problem, 463 eigenvector, 463 left, 465 right, 465 inverse matrix, 65 normalized, 67 self-reflexive, 67 weak, 67 permutation matrix, 102 Rayleigh quotient, 478 Rayleigh–Ritz ratio, 478 singular value, 290 nontrivial pairs, 290 trivial pairs, 290 singular value decomposition, 289 GSVD algorithm 1, 293 GSVD algorithm 2, 293 singular vectors, 292 singular-vector matrix, 292 tensor eigenpair, 657 total least squares, 335 geometric multiplicity, 417 vector, 4

Index global approach, 671 maximum, 186 maximum point, 186 minimum, 184 minimum point, 184 positioning, 304 prior, 671 Goldstein–Armijo rule, 208 GPSR-BB algorithm, 397 gradient computation, 133 chain rule, 134 linear rule, 134 product rule, 134 quotient rule, 134 descent algorithm, 197 flow direction, 133 matrix, 132 matrix operator, 132 projection method, 208 vector, 132 vector operator, 132 grandmother cell coding, 357 graphical modeling, 297 Grassmann manifold, 534 GS method, 324 proximal point versions, 327 PGS method, 327 greedy algorithm, 391 group Lasso problem, 398 g matrix, 102 Hadamard inequality, 51, 96 matrix, 121 product, 72 eigenvalue inequality, 74 Oppenheim inequality, 74 rank inequality, 74 product theorem, 73 half-space, 209 harmonic function, 159 superresolution problem, 347 heavy ball method, 220 Hellinger distance, 362 Hermite normalized form, 14 Hermitian matrix, 8, 95 anti-, 95 centro-, 95 Toeplitz, 124 Hessenberg matrix, lower,107 Hessian

713

matrix, 150 identification, 153 hierarchical Tucker toolbox, 678 higher-order singular value decomposition, 616 orthogonal iteration (HOOI) algorithm, 624 SVD algorithm, 624 matrix analysis, 590 Hilbert space, 36 holomorphic complex matrix functions, 161 function, 158 homogeneity constraint, 532 homogeneous linear equations, 518 homotopy algorithm, 403 horizontal slice, 594 slice matrix, 594 unfolding, 597 Kiers method, 598 Kolda method, 600 LMV method, 599 matrix, 597 Hotelling transformation, 458 Householder matrix, 106 hyperbolic symmetry, 106 hyperdeterminants, 651 hypermatrix, n-dimensional, 592 hypernormal matrix, 106 idempotent matrix, 96 identifiable equation, 317 identity transformation, 413 ill-conditioned equation, 274 inactive constraint, 403 inconsistent equation, 56 indefinite matrix, 48 independence assumption, 134 rule, 165 independently identical distribution, 44 index vector, 116 indicator function, 231 inequality constraint, 198 inertia, 420 inexact line search, 253 infeasible points, 199 infimum, 199, 243 infinite-dimensional vector subspace, 513 inner product, 27 conjugate symmetry, 27 linearity in the first argument, 27 nonnegativity, 27

714 inner product space, 36 input regeneration, 413 interior-point condition, 262 interlacing eigenvalue theorem, 422 for singular values, 277 interpretability, 357 invariant factors, 438 subspace, 532 inverse affine image, 201 discrete Fourier transform (IDFT), 116 matrix, 10 problem, 26 ill-conditioned, 349 singular, 349 involutory matrix, 100 property, 100 irreducible matrix, 353 isolated local extremal point, 186 Itakura–Saito divergence, 363 iteration matrix, 210 iterative soft thresholding method, 236 Tikhonov regularization, 322 Jacobian determinant, 142 matrix, 142 matrix identification, 141 operator, 130 Jensen inequality, 202 joint diagonalization, 496 approximate, 497 exact, 497 joint diagonalizer, 497 joint-factor variation, 671 Jordan blocks, 426 canonical form, 426 representation, 431 Jordan–Wielandt theorem, 421 J-orthogonal matrix, 106 Karhunen–Loeve expansion, 460 transformation, 458 Karush–Kuhn–Tucker (KKT) conditions, 245 Khatri–Rao product, 78 associative law, 78 commutative law, 78 distributive law, 78

Index KKT point, 245 Kronecker product, 75 columnwise, 78 generalized Kronecker product, 77 left, 75 right, 75 properties, 76-77 Krylov matrix, 210 subspace, 210 subspace method, 210 Kruskal rank, 636 K-sparse, 86 K-sparse approximation, 382 Kullback–Leibler (KL) divergence, 363 Lagrange multiplier method, 237 vector, 237 Lagrangian dual method, 244 Lanczos algorithm, 544 basis, 544 Laplace equation, 159 LARS algorithm, 398 contraction function, 386 selection function, 386 latent semantic index, 298 variables, 643 lateral slice, 594 lateral slice matrix, 594 leading entry, 14 leading-1 entry, 14 leading-1 polynomial, 447 learning algorithm, 198 step, 198 least absolute shrinkage and selection operator (Lasso) algorithm, 385 distributed Lasso, 387 generalized Lasso, 387 group Lasso, 387 MRM Lasso, 387 least squares (LS) method, 316 left inverse matrix, 63 pseudo inverse matrix, 64 order recursion, 64 quadratic eigenvectors, 485 singular vector, 276 singular-vector matrix, 276 solvent, 487

Index leverage analysis, 663 vector, 663 limit point, 325 linear combination, 11 dependence, 11 independence, 11 inverse transform, 10 inverse transform matrix, 10 manifold, 532 mapping, 25 programming problem, 261 regression, 385 search, 253 transformation, 25 transform matrix, 10 linearized Bregman iterative algorithm, 408 λ-matrix, 488 Lipschitz constant, 217 continuous function, 217 L-Lipschitz continuous function, 217 local approach, 671 maximum, 186 maximum point, 186 minimum, 185 minimum point, 185 second-order necessary condition, 187 longitudinal unfolding, 597 Keirs method, 601 Kolda method, 602 LMV method, 602 matrix, 597 lossless compression, 288 lossy compression, 288 low-rank sparse matrix decomposition, 298 low-rank TLS solution, 333 lower unbounded, 199, 243 LQ decomposition, 581 0 -norm minimization, 382 1 -norm ball, 209 1 -norm minimization, 383 Mahalanobis distance, 33 manifold, 517 mapping, 25 codomain, 25 domain, 25 final set, 25 image, 25

initial set, 25 injective, 25 inverse, 25 one-to-one, 25 original image, 25 range, 25, 517 surjective, 25 matching pursuit, 391 MATLAB CMTF toolbox, 678 matricizing of column vector, 597 of row vector, 82 matrix, 4 addition, 8 commutative law, 8 associative law, 8 autocorrelation, 7 block, 6 broad, 5 completion, 302 complex conjugate transpose, 8 complex, 4 conjugate, 8 cross-correlation, 7 differential, 305 equation, 4 function, 11 derivative, 12 exponential, 12 integral, 13 logarithm, 12 triangle, 12 Hermitian transpose, 8 ill-conditioned, 273 inversion, 19 augmented lemma, 60 block lemma, 61 Gauss elimination method, 19 lemma, 60 Sherman–Morrison formula, 60 Woodbury formula, 60 λ-matrix, 439, 485 nonregular, 485 regular, 485 norm, 39 absolute column sum, 38 absolute row sum, 38 entrywise form, 39 Frobenius, 39 induced, 37 Mahalanobis, 39 max, 39 Minkowski p-norm, 38 p -norm, 38

715

716 nuclear, 40 Schatten p-norm, 40 spectral norm, see matrix norm, nuclear, 38 triangle inequality, 28 pencil (matrix pair), 464 performance indexes, 58 positive definiteness criterion, 96 power, 431 product, 9 associative law, 9 left distributive law, 9 right distributive law, 9 real, 4 square, 5 submatrix, 6 subtraction, 8 symmetric, 8 transformation, 26 transpose, 8 unit, 5 well-conditioned, 273 zero, 5 maximization problem, 183 maximum ascent rate, 133 descent rate, 133 Z-eigenvalue function, 653 maximum likelihood solution, 320 maximin problem, 243 measure, 32 nonnegativity and positiveness, 32 symmetry, 32 triangle inequality, 32 merit function, 267 MIMO radar, 303 minimal polynomial, 447 minimax problem, 242, 386 minimization problem, 183 minimum norm solution, 330 0 solution, 85 1 solution, 389 2 solution, 85 Minkowski inequality, 51 minor component, 463 minor component analysis, 463 mixture matrix, 102 modality, 642 mode-n centering, 660 product, 611 mode-N vectors, 593 model-driven approach, 643 monic polynomial, 447

Index monotone function, 203 strictly monotone function, 203 strongly monotone function, 203 Moore–Penrose conditions, 66 Moore–Penrose inverse, 66 recursive algorithms, 71 KL decomposition method, 69 Moreau decomposition, 229 multi-way array data, 589 multidimensional array data, 589 multi-label image classification, 305 multilinear principal component analysis, 647 multimodality, 642 multiple-input–multiple-output (MIMO), 494 multiple signal classification (MUSIC), 530 multiplicity, 415 mutual coherence, 387 μ-center, 262 N -way toolbox, 676 nearest neighbour, 33 nearest neighbour classification, 33 negative curvature direction, 197 negative definite matrix, 48 semi-definite matrix, 48 neighborhood, 189 Nesterov first optimal gradient algorithm, 221 third optimal gradient algorithm 223 Newton method, 207 complex, 254 modified, 252 quasi-, 252 truncated, 252 Newton algorithm, 253 infeasible start, 256 feasible start, 255 Newton descent direction, 207 Newton search direction, 254 Newton step, 207 complex Newton step, 254 Newton–Raphson method, 207 Neyman χ2 distance, 363 nilpotent matrix, 97 noise subspace, 526 noise subspace method, 528 nonactive constraint, 403 nonconvex set, 200 nonextensive entropy, 364 nonsmooth convex optimizations, 232

Index nonnegative blind matrix equation, 353 CP decomposition, 666 alternating least squares algorithm, 669 multiplication algorithm, 666 matrix, 353 matrix approximation, 360 matrix factorization, 360 multiplication algorithm, 366 Nesterov optimal gradient algorithm, 373 projected gradient algorithm, 372 orthant, 201 quadrant, 201 tensor, 652 Tucker decomposition, 665 alternating least squares algorithm, 669 multiplication updating algorithm, 666 vector, 353 nonnegativity, 27 nonnegativity constraint, 355 nonnormal λ-matrix, 485 nonsingular matrix, 11, 50 polynomial, 435 nonstationary iterative methods, 210 nontrivial solution, 59 normal complex partial derivative, 162 equation, 457 matrix, 105 vector, 338 normalized Euclidean distance, 34 singular values, 281 normed vector space, 35 null space, 518 nullity, 518 numerical stability, 272 object–concept distance, 35 objective function, 183 oblique projector, 574 geometric interpretation, 576 physical meaning, 577 observation data space, 526 off function, 498 off matrix, 498 one-way array data, 589 ordinary least squares (OLS) method, 316 open ball, 201

717

neighborhood, 185 Oppenheim inequality, 96 optimal dual value, 244 least square approximate solution, 333 K-sparse approximation, 383 primal value, 242 solution, 199 unbiased estimator, 317 optimality direction, 265 optimization variable, 198 vector, 183 ordered n-ples, 22 ordered pair, 22 ordering indetermination, 635 original cost function, 242 feasibility, 249 optimum point, 244 problem, 242 residual, 250 variable, 242 original–dual interior-point method, 264 orthogonal, 43 approximate joint diagonalization, 498 complement subspace, 514 decomposition, 551 equivalence, 105 geometric interpretation, 32 group, 535 matching pursuit, 391 matching pursuit algorithm, 392 matrix, 104 physical meaning, 32 projection, 552 projection matrix, 561 updating formulas, 563 projector, 557 random vector orthogonal, 31 set, 104 subspace, 513 system, 554 orthogonality, 32 constraint, 532 geometric interpretation, 32 mathematical definition, 32 physical meaning, 32 random vector orthogonality, 31 orthogonally decomposable tensor, 658 orthonormal input matrix, 385 set, 104 system, 554

718

Index

outer product, 7 overcomplete atoms, 85 over-determined equation, 56 matrix equation, 315 partial derivative vector, 130 PAST via deflation (PASTd) algorithm, 541 pattern classification, 32 vector, 356 Pearson χ2 distance, 362 penalty factors, 238 penalty function, 238 exterior, 239 interior, 239 penalty function method, 238 exterior, 240 interior, 240 penalty parameter vector, 241 permutation matrix, 98 Perron–Frobenius theorem, 355 perturbed complementarity condition, 262 PGS method, 327 physical vector, 4 Pisarenko harmonic decomposition, 455 pivot column, 14 position, 14 plaintext, 86 point-spread function (PSF), 348 pointwise maximal function, 226 polarization identity, 28 polynomial matrix, 434 inverse, 435 rank, 435 positive definite matrix, 48 positive matrix, 353 factorization, 360 positive-part operator, 396 positive semi-definite cone, 201, 209 matrix, 48 positive vector, 353 power property, 424 preconditioned conjugate gradient algorithm, 213 iteration, 212 predictor, 385 preprocessor, 212 primal cost function, 242 primal–dual interior point method, 262

primitive matrix, 353 principal component, 462 principal component analysis (PCA), 461 principal minor, 49 product singular value decomposition, 283 projected gradient method, 208 projection, 208, 552 approximation subspace tracking (PAST) algorithm, 540 matrix, 558 first-order partial derivative, 561 second-order partial derivative, 561 updating formula, 563 operator, 556 theorem, 553 projector, 556 prox-function, 223 proximate gradient method, 233 mapping, 228 pseudo additivity, 364 Pythagorean theorem, 32, 40 Q-convergence rate, 215 q-distribution, 365 q-exponential, 365 q-Gaussian distribution, 365 q-logarithm, 365 QR decomposition, 274 quadratic characteristic equation, 485 characteristic polynomial, 485 eigentriple, 485 eigenvalue problem, 485 elliptic quadratic eigenvalue problem, 486 hyperbolic quadratic eigenvalue problem, 486 eigenvalue problem solving, 486 decomposition method, 487 linearized algorithm, 488 linearized method, 488 eigenvalues, 485 form, 47 matrix equation, 487 matrix polynomial, 487 programming (QP) problem, 384 variation (QV), 675 quadratically constrained linear program (QCLP) problem, 383 quantization coefficient matrix, 357 vectors, 357

Index quasi-convex function, 203, 326 strictly quasi-convex function, 203 strongly quasi-convex function, 203 quotient, 450 quotient singular value decomposition, 292 radar imaging, 303 random matrix, 342 random matrix equation, 342 random vector, real, 5 range, 25, 517 rank, 519 rank deficient matrix, 58 rank equality, 58 rank inequality, 58, 74 rank-1 updating algorithm, 252 ranking and collaborative filtering, 304 Rayleigh quotient, 474 boundedness, 474 homogeneity, 474 iteration, 475 minimum residual, 474 orthogonality, 474 problem solving, 476 conjugate gradient algorithm, 477 gradient algorithm, 476 shift invariance, 474 Rayleigh sequence, 475 scale invariance, 475 shift invariance, 476 unitary similarity, 476 Rayleigh–Ritz approximation, 544 ratio, 474 (RR) value, 543 (RR) vector, 543 theorem, 474 rectangular set, 209 recursive least squares (RLS) algorithm, 540 reduced row echelon form matrix, 14 reducible matrix, 353 reflection matrix, 100 regular matrix, 353 λ-matrix, 485 tensor pair, 656 regularization method, 322 orthogonal matching pursuit, 392 parameter, 322 path, 323

regularized Gauss-Seidel method, 327 least squares, 321 method, 322 relative interior domain, 247 points, 247 relatively feasible interior set, 238 relaxation method, 322 sequence, 183 remainder, 450 remote sensing, 304 resolvent, 231 response vector, 85 restricted isometry constant (RIC), 388 property (RIP), 388 resultant, 651 Richardson iteration, 210 right inverse matrix, 63 pseudo inverse matrix, 64 order recursion, 64 quadratic eigenvectors, 485 singular vector, 276 singular-vector matrix, 276 solvent, 487 Ritz pairs, 543 robust linear regression, 386 principal component analysis, 300 rotation operator, 468 row equivalent, 14 partial derivative operator, 130 space, 517 span, 517 vectorization, 79 Rudin–Osher–Fatemi de-noising model, 405 saddle point, 188 sample density, 305 scalar multiplication association rule, 23 distribution rule, 23 unit rule, 23 scale uncertainty, 635 scaling, 660 Schur inequality, 55 Schur product, 72 search direction, 206

719

720

Index

searching point sequence, 221 second-order cone, 209 (matrix) differential rule, 153 primal-dual interior point method, 264 select matrix, 103 selection function, 386 self-adjoint matrix, 95 semi-orthogonal matrix, 104, 532 set, 20 difference, 21 intersection, 21 null, 21 proper subset, 21 relative complement, 21 subset, 21 sum, 21 superset, 21 union, 21 Shannon entropy, 364 Sherman–Morrison formula, 60 shift matrix, 101 short matrix, 533 semi-orthogonal, 534 shrinkage operator, 230 signal decoding, 461 encoding, 461 subspace, 527 subspace method, 528 signature, 420 signum multifunction, 390 similar matrices, 423 reflexivity, 423 symmetry, 423 transitivity, 423 similarity, 32 similarity invariance, 54 similarity transformation, 423 simultaneous tensor decomposition and completion (STDC), 671 single singular value, 275 singleton, 225 singular matrix, 11 singular tensor pair, 656 singular value, 275 decomposition, 275 shrinkage operator, 299 thresholding (SVT), 298 thresholding for matrix completion, 308 slack variable, 247 slack variable vector, 247 Slater condition, 247 Slater theorem, 247

slice matrices, 594 frontal slice, 594 horizontal slice, 594 lateral slice, 594 slope, 338 slope parameter, 338 Smith normal form, 436 smooth function, 218 PARAFAC tensor completion, 674 smooth convex optimization problem, 205 soft threshold value, 229 soft thresholding, 298 operation, 298 operator, 229 spanning set, 511 spanning set theorem, 512 sparse approximation problem, 85 coding problem, 382 decomposition, 85 matrix, 84 representation problem, 85 vector, 84 sparseness, 378 sparsest representation, 85 solution, 382 sparsity, 382 sparsity constraint, 356 SPC algorithm, 676 spectrum, 420 spectral radius, 420 split Bregman iterative algorithm, 409 split optimization problem, 228 square root matrix, 109 squared Euclidean distance, 362 stagewise orthogonal matching pursuit, 392 standard Euclidean structure, 35 incoherence condition, 306 standardized Hadamard matrix, 121 orthonormal Hadamard matrix, 121 stationary iterative methods, 210 point, 184 statistical fidelity, 357 independence, 43 uncorrelation, 43 steepest descent direction, 207

Index method, 197, 206 Stiefel manifold, 536 strict absolute minimum, 184 global maximum, 186 global maximum point, 186 global minimum, 185 global minimum point, 185 local minimum, 185 local minimum point, 185 strictly convex function, 202 strictly quasi-convex function, 326 strictly square integrable function, 514 strong duality, 244 strongly convex function, 202 strongly local minimum point, 186 strongly monotone, 203 Sturmian separation theorem, 509 SVD-TLS algorithm, 333 subdifferentiable, 225 subdifferential, 225 subgradient vector, 225 suboptimal point, 244 ε-suboptimal dual point, 244 ε-suboptimal original point, 244 ε-suboptimal solution, 205 subspace dimension, 513 generator, 511 nest, 515 pursuit algorithm, 392 whitening matrix, 532 sum-of-norms regularization, 398 summing vector, 110 super plane, 209 super plane fitting, 341 superdiagonal line, 593 supersymmetric tensor, 592 supremum, 242 swamp, 327 symmetric tensor, 650, 592 tall-skinny matrix, 533 semi-orthogonal matrix, 534 tangent algorithm, 466 space, 466 Tanimoto measure, 34 target signal, 85 Taylor series expansion, 187 TDALAB, 676

tensor, 590 characteristic polynomial, 652 core, 616 decomposed, 612 eigenvalue, 651 B-eigenvalue, 656 D-eigenvalue, 654 E-eigenvalue, 653 H-eigenvalue, 652 Z-eigenvalue, 653 U-eigenvalue, 654 US-eigenvalue, 655 eigenvector, 651 B-eigenvector, 656 D-eigenvector, 654 E-eigenvector, 653 H-eigenvector, 652 Z-eigenvector, 653 U-eigenvector, 654 US-eigenvector, 655 epilepsy, 591 face, 591 fibers, 593 column, 593 depth, 593 horizontal, 593 row, 593 tube, 593 fiber, 593 flattening, 597 first-order, 590 Frobenius norm, 606 inner product, 606 matricization, 597 matricizing, 597, 598 mode-n rank, 613 norm, 606 nth-order, 590 outer product, 607 pair, 656 pencil, 656 positive definite, 650 positive semi-definite, 650 product, 75 rank, 614 rank decomposition, 614 reconstruction, 605 reshaping, 605 second-order, 590 supersymmetric, 592 tensorization, 605 third-order, 592 toolbox, 676 toolbox in Python, 678

721

722 train decomposition, 678 unfolding, 597 unit, 593 unit third-order, 592 zero-order, 590 tensor analysis, 589 tensor–vector products, 650 tensor–vectorization, 597 TENSORBOX, 676 Tensorlab toolbox, 676 tensorization, 605 TensorReg toolbox, 678 term-document matrix, 295 thin singular value decomposition, 276 third-order SVD, 617 Tikhonov regularization, 322 regularizing solution, 322 limit characteristic, 323 linearity, 323 regularization, 323 time-shifting operation, 564 time-shifting operator, 564 Toeplitz matrix, 123 Hermitian, 124 fast DCT algorithm, 125 skew Hermitian, 124 symmetric, 124 Toeplitz system of linear equations, 124 total differential, 141 least squares, 328 least square (TLS) algorithm, 331 least squares fitting, 338 least squares solution, 329 maximum likelihood (TML) method, 342 variance, 675 trace, 54 equality, 54 inequality, 55 transformation matrix, 26 triangular matrix, 107 lower, 108 strict lower, 108 strict upper, 108 unit, 108 unit upper, 108 upper, 108 tridiagonal matrix, 107 tri-Lanczos iteration, 545 tripotent matrix, 98 trivial solution, 59

Index subspace, 512 truncated nuclear norm, 308 truncated nuclear norm minimization, 308 accelerated proximal gradient line (APGL) search algorithm, 310 ADMM algorithm, 311 truncated singular value decomposition, 276 Tsallis entropy, 364 logarithm, 365 mathematical statistics, 365 TT toolbox, 676 Tucker decomposition, 615 alternating least squares algorithm, 623 multiplication algorithm, 666 Tucker1 decomposition, 621 Tucker2 decomposition, 621 Tucker3 decomposition, 621 Tucker operator, 615 Tucker product, 608 Tucker mode-1 product, 608 Tucker mode-2 product, 608 Tucker mode-3 product, 608 Type-I elementary row matrix, 435 Type-I elementary row operations, 13 Type-II elementary row matrix, 435 Type-II elementary row operations, 13 Type-III elementary row matrix, 435 Type-III elementary row operations, 13 unconstrained minimization problem, 195 first-order necessary condition, 187 second-order necessary condition, 187 second-order sufficient condition, 188 unbiased estimator, 317 uncertainty set, 386 under bandwidth, 107 under-determined equation, 56 sparse matrix equation, 315 unidentifiable, 317 unfixed iterative method, 210 unfold principal component analysis, 646 unipotent matrix, 97 unique representation theorem, 513 unit ball, 200 lower triangular matrix, 108 third-order tensor, 593 upper triangular matrix, 108 unitarily equivalent, 105 unitary matrix, 104 para-unitary matrix, 104

Index unitary transformation, 105 unity law, 23 untraceable, 382 upper bandwidth, 107 upper Hessenberg matrix, 107 utility function, 183 value function, 183 Vandermonde matrix, 112 factor matrices, 672 inverse matrix, 113 vectors, 3 algebraic, 5 angle, 30 basis, 6 column, 3 constant, 5 dual residual, 250 Euclidean distance, 30 Euclidean length, 29 function, 5 inner product (dot product), 7, 30 geometric, 4 norm, 30 normalized, 30 outer product (cross product), 7, 607 physical, 4 random, 5 autocorrelation matrix, 41 autocovariance matrix, 41 inner product, 31 mean vector, 41 norm, 31 row, 4 scaled dual vector, 250 zero, 5 -neighborhood, 29 vector addition, 6 associative law, 6 commutative law, 6 vector multiplication, 6 vector norm, 28 Cauchy–Schwartz inequality, 29 Euclidean norm, 29 H¨ older norm, 29 homogeneity, 28 0 -norm, 29 1 -norm, 29 p -norm, 29 ∞ -norm, 29 nonnegativity, 28 norm axioms, 28

parallelogram law, 28 positivity, 28 triangle inequality, 28 unitary invariant norm, 30 vector quantization, 357 vector quantization method, 357 vector quantization problem, 357 vector space, 23 complex, 24 real, 24 vector subspaces, 24 vectorization, 79 video surveillance, 297 violated constraint, 242, 403 weak duality, 244 weighted least squares solution, 410 well-defined, 327 well-determined equation, 56 Weyl theorem, 422 whitening matrix, 531 winner-take-all, 357 Wirtinger partial derivative, 159 within-class scatter matrix, 295 within-factor variation, 671 Woodbury formula, 60 zero-dimension array data, 589 zero matrix, 5

723