Introduction to Unconstrained Optimization with R [1 ed.] 978-981-15-0894-3

Table of contents :
Preface......Page 6
Contents......Page 8
About the Authors......Page 11
Nomenclature......Page 12
1.1 Historical Note on Unconstrained Optimization......Page 14
1.2 History of blueR......Page 17
1.3 History of Algorithm......Page 18
1.4 Motivation to Use blueR in Optimization......Page 19
2.1 Vectors and Matrices......Page 21
2.2 Eigenvalues and Eigenvectors......Page 27
2.3 Neighborhoods......Page 32
2.4 Algorithm......Page 35
2.5 Taylor's Theorem......Page 38
2.6 Quadratic Functions......Page 39
2.7 Exercises......Page 43
3.1 Introduction......Page 46
3.2 Basics of blueR Programming......Page 47
3.3 Decision-Making and Loop Statements......Page 56
3.4 Graphics......Page 62
3.5 Exercises......Page 65
4.1 Introduction......Page 67
4.2 Directional Derivative......Page 74
4.3 First-Order Necessary Condition......Page 77
4.4 Second-Order Necessary Condition......Page 83
4.5 Second-Order Sufficient Condition for Interior Case......Page 88
4.6 Exercises......Page 92
5.1 Introduction......Page 95
5.2 Interval Halving Search Method......Page 96
5.3 Fibonacci Search Method......Page 101
5.4 Golden Section Search Method......Page 111
5.5 Quadratic Interpolation Search Method......Page 117
5.6 Bisection Method......Page 122
5.7 Newton–Raphson Method......Page 126
5.8 Secant Method......Page 129
5.9 Case Study: Producing Micro Electronics for Lithography......Page 132
5.10 Exercises......Page 139
6.1 Introduction......Page 141
6.2 Basics of Steepest Descent Method......Page 142
6.3 Steepest Descent Method for Quadratic Functions......Page 146
6.4 Convergence Analysis of Steepest Descent Algorithm......Page 156
6.5 Case Study: Portfolio Selection Problem......Page 170
6.6 Exercises......Page 181
7.1 Introduction......Page 184
7.2 Newton's Method for Multiple Unknowns......Page 185
7.3 Convergence Analysis of Newton's Method......Page 192
7.4 Modified Newton's Method......Page 197
7.5 Levenberg–Marquardt Method......Page 203
7.6 Case Study: Optimal Design of a Pi-Electric Circuit......Page 212
7.7 Exercises......Page 217
8.2 Basics of Conjugate Direction......Page 219
8.3 Convergence Analysis of Conjugate Direction Method......Page 225
8.4 Method of Conjugate Gradient......Page 230
8.5 Method of Conjugate Gradient for General Functions......Page 239
8.6 Case Study: Solving System of Linear Equations in Electrical Engineering......Page 247
8.7 Exercises......Page 250
9.2 Basics of Quasi-Newton Methods......Page 253
9.3 Approximating the Inverse Hessian......Page 255
9.4 The Rank One Correction Formula......Page 258
9.5 Convergence Analysis of Rank One Correction Formula......Page 266
9.6 Davidon Fletcher Powell Method......Page 268
9.7 Convergence Analysis of DFP Method......Page 276
9.8 Broyden–Fletcher–Goldfarb–Shanno Method......Page 281
9.9 Case Study: Heat Conduction Problem......Page 292
9.10 Exercises......Page 295
Appendix Solutions......Page 298
BookmarkTitle:......Page 305
Index......Page 308

Citation preview

Shashi Kant Mishra Bhagwat Ram

Introduction to Unconstrained Optimization with R

Introduction to Unconstrained Optimization with R

Shashi Kant Mishra Bhagwat Ram •

Introduction to Unconstrained Optimization with R

123

Shashi Kant Mishra Department of Mathematics Banaras Hindu University Varanasi, Uttar Pradesh, India

Bhagwat Ram Department of Mathematics Banaras Hindu University Varanasi, Uttar Pradesh, India

ISBN 978-981-15-0893-6 ISBN 978-981-15-0894-3 https://doi.org/10.1007/978-981-15-0894-3

(eBook)

© Springer Nature Singapore Pte Ltd. 2019 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd. The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore

To Viaan, my grandson with Love —Shashi Kant Mishra

Preface

This book is based on the lectures given to master’s students in the Department of Mathematics, Institute of Science, Banaras Hindu University, Varanasi, India. Though there are numerous books on unconstrained optimization dealing with methods given in this book, there is no book on unconstrained optimization with to the best of our knowledge. We have presented usual methods, their algorithm, and convergence analysis and tried to provide proofs as simple as possible so that any reader with a background of calculus and linear algebra can easily follow them. We have provided manually solved examples for each method, examples solved using codes, case studies, and developed program for each method. This book contains 16 algorithms that are illustrated by 115 examples and 71 figures and 61 problems in the form of exercises given at the end of each chapter. We have developed 24 functions on the lines of 17 algorithms to minimize nonlinear objective functions. A reader will be able to use these methods and programs in real-life situations with a large number of variables after reading this book. tool is a free open-source computing environment which works on several platforms such as Windows, Linux, and macOS. In recent years, there has been an increasing interest in using software to perform the data analysis. We assume that the reader has no prior experience in optimization and scripts. We are in the age of digital information—a power in which software has played a very important role. Unconstrained optimization is applied in science, engineering, social science, finance, and agriculture. As is free open source, this book will be useful for several audience groups. The book is organized as follows: it has nine chapters; the first chapter presents the historical background of unconstrained optimization and . Chapter 2 recalls basic definitions, notations, and preliminaries from algebra, advance calculus, general properties of algorithm, and concept of convergence that are frequently used throughout the book. In Chap. 3, the foundation of is presented, which includes the installation of , looping, decision-making statement, user-defined functions, matrices, packages, and drawing graphs. Chapter 4 provides the first- and second-order necessary conditions for a point to be a local minimizer and a special

vii

viii

Preface

case of the second-order sufficient conditions. Chapter 5 presents several methods for one-dimensional optimization, which are referred as line search methods such as interval halving, Fibonacci search, golden section, quadratic interpolation, bisection method, Newton–Raphson method, and Secant method. The steepest descent method, its algorithm, and convergence analysis are given in Chap. 6. Further, codes are developed to minimize quadratic functions and examples are given. Chapter 7 is concerned about Newton’s method, modified Newton’s method, Levenberg–Marquardt method, and their algorithms, and convergence analysis is presented. Codes are also developed. Chapter 8 focuses on the conjugate direction method, basic conjugate direction, and then conjugate gradient algorithm and its convergence analyses; the script is developed to solve the quadratic and general nonlinear functions and examples are also provided. In Chap. 9, Quasi-Newton methods such as Rank one, DFP, BFGS, and their algorithms, properties, convergence analyses, programs, and examples are provided. Varanasi, India November 2019

Shashi Kant Mishra Bhagwat Ram

Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Historical Note on Unconstrained Optimization 1.2 History of . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 History of Algorithm . . . . . . . . . . . . . . . . . . . 1.4 Motivation to Use in Optimization . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

1 1 4 5 6

2 Mathematical Foundations . . . . . . 2.1 Vectors and Matrices . . . . . . 2.2 Eigenvalues and Eigenvectors 2.3 Neighborhoods . . . . . . . . . . . 2.4 Algorithm . . . . . . . . . . . . . . 2.5 Taylor’s Theorem . . . . . . . . . 2.6 Quadratic Functions . . . . . . . 2.7 Exercises . . . . . . . . . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

9 9 15 20 23 26 27 31

3 Basics of ......................... 3.1 Introduction . . . . . . . . . . . . . . . . . . . . 3.2 Basics of Programming . . . . . . . . . . 3.3 Decision-Making and Loop Statements 3.4 Graphics . . . . . . . . . . . . . . . . . . . . . . 3.5 Exercises . . . . . . . . . . . . . . . . . . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

35 35 36 45 51 54

4 First-Order and Second-Order Necessary Conditions . . 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Directional Derivative . . . . . . . . . . . . . . . . . . . . . . 4.3 First-Order Necessary Condition . . . . . . . . . . . . . . 4.4 Second-Order Necessary Condition . . . . . . . . . . . . 4.5 Second-Order Sufficient Condition for Interior Case 4.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

57 57 64 67 73 78 82

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

ix

x

5 One-Dimensional Optimization Methods . . . . . . . 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Interval Halving Search Method . . . . . . . . . 5.3 Fibonacci Search Method . . . . . . . . . . . . . . 5.4 Golden Section Search Method . . . . . . . . . . 5.5 Quadratic Interpolation Search Method . . . . 5.6 Bisection Method . . . . . . . . . . . . . . . . . . . . 5.7 Newton–Raphson Method . . . . . . . . . . . . . . 5.8 Secant Method . . . . . . . . . . . . . . . . . . . . . . 5.9 Case Study: Producing Micro Electronics for 5.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . .

Contents

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

85 85 86 91 101 107 112 116 119 122 129

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

131 131 132 136 146 160 171

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

175 175 176 183 188 194 203 208

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

211 211 211 217 222 231

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . . Lithography . ...........

6 Steepest Descent Method . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Basics of Steepest Descent Method . . . . . . . . . . . . . . 6.3 Steepest Descent Method for Quadratic Functions . . . 6.4 Convergence Analysis of Steepest Descent Algorithm . 6.5 Case Study: Portfolio Selection Problem . . . . . . . . . . 6.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

7 Newton’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Newton’s Method for Multiple Unknowns . . . . . . . . 7.3 Convergence Analysis of Newton’s Method . . . . . . . 7.4 Modified Newton’s Method . . . . . . . . . . . . . . . . . . . 7.5 Levenberg–Marquardt Method . . . . . . . . . . . . . . . . . 7.6 Case Study: Optimal Design of a Pi-Electric Circuit . 7.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

8 Conjugate Gradient Methods . . . . . . . . . . . . . . . . . . . . . . 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Basics of Conjugate Direction . . . . . . . . . . . . . . . . . . 8.3 Convergence Analysis of Conjugate Direction Method 8.4 Method of Conjugate Gradient . . . . . . . . . . . . . . . . . 8.5 Method of Conjugate Gradient for General Functions . 8.6 Case Study: Solving System of Linear Equations in Electrical Engineering . . . . . . . . . . . . . . . . . . . . . . 8.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . .

. . . . . .

. . . . . . . . 239 . . . . . . . . 242

9 Quasi-Newton Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Basics of Quasi-Newton Methods . . . . . . . . . . . . . . . . . 9.3 Approximating the Inverse Hessian . . . . . . . . . . . . . . . . 9.4 The Rank One Correction Formula . . . . . . . . . . . . . . . . 9.5 Convergence Analysis of Rank One Correction Formula .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

245 245 245 247 250 258

Contents

9.6 9.7 9.8 9.9 9.10

xi

Davidon Fletcher Powell Method . . . . . . . . . Convergence Analysis of DFP Method . . . . Broyden–Fletcher–Goldfarb–Shanno Method Case Study: Heat Conduction Problem . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

260 268 273 284 287

Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303

About the Authors

Shashi Kant Mishra Ph.D., D.Sc. is Professor at the Department of Mathematics, Institute of Science, Banaras Hindu University, Varanasi, India. With over 20 years of teaching experience, he has authored six books, including textbooks and monographs, and has been on the editorial boards of several respected international journals. He has guest edited special issues of the Journal of Global Optimization and Optimization Letters (both Springer Nature) and Optimization (Taylor & Francis). A DST Fast Track Fellow (2001–2002), Prof. Mishra has published over 150 papers and supervised 15 Ph.D. students. He has visited around 15 institutes/ universities in countries such as France, Canada, Italy, Spain, Japan, Taiwan, China, Singapore, Vietnam, and Kuwait. Bhagwat Ram is Senior Research Fellow at the DST Centre for Interdisciplinary Mathematical Sciences, Institute of Science, Banaras Hindu University, Varanasi. He holds an M.Sc. in Computer Science, and co-authored the book Introduction to Linear Programming with MATLAB, with Prof. Shashi Kant Mishra. He is currently developing generalized gradient methods to solve unconstrained optimization problems and instructing graduate students in their MATLAB practicals at the Centre for Interdisciplinary Mathematical Sciences at the Banaras Hindu University. He received an international travel grant from the Council of Scientific Industrial and Research, Government of India, to attend a summer school on linear programming at New South Wales University, Australia, in January 2019.

xiii

Nomenclature

Rn T x

f. . .g f ðxÞ; f x f ðx Þ C1 C2 ;  j:j ½ a; b  R det A B A1 I xT y k xk fxk g f 0 ðxÞ @f ðxÞ @xi

rf ðxÞ . FðxÞ arg min f

n-dimensional Euclidean (real) space transpose 2 3of a vector or matrix x1 6 x2 7 6 7 x ¼ 6 .. 7 4 . 5 xn set objective function local minimizer minimum function value set of continuous differentiable functions set of continuous and twice differentiable function subset of absolute value closed interval between the real numbers a and b set of real numbers determinant of matrix A Euclidean closed unit ball Inverse of matrix A identity matrix scalar product of the vectors x and y Euclidean norm of x sequence in Rn derivative of f at x partial derivative of f with respect to xi at x gradient of f at x comment Hessian matrix of f at x set of points in which f reaches its minimum value

xv

xvi

Nomenclature

9 lim

exists limit as i tends to infinity

Q

Hessian software

i!1

Chapter 1

Introduction

In our daily life, we always choose the best possible solutions for several problems. We encounter the problems of maximum and minimum. In mathematics, the study of maximum and minimum problems began a very long time ago. There has been increasing interest in the problem of minimizing functions of n variables numerically. There were no uniform ways to find the maxima or minima of problems. The first general methods of investigation and solution of extremal problems were created about 300 years ago, at the time of the formation of mathematical analysis. Then, it became clear that certain special optimization problems play a crucial role in the natural sciences. Specifically, it was found that many laws of nature can be derived from optimization methods. We shall learn some historical development of unconstrained optimization.

1.1 Historical Note on Unconstrained Optimization It is not clear that when the term optimization has been coined in mathematics, but the existence of optimization methods can be seen from the days of Newton. In 1664, Isaac Newton (1643–1727) discussed the numerical solution of the cubic polynomial equations. The method developed by Newton did not compute the successive approximations of the polynomial. In 1690, Joseph Raphson (1648–1715) developed a method, now known as the Newton–Raphson method, for approximating the roots of an equation. Isaac Newton had developed a very similar formula in his “Method of Fluxions” book, written in 1671, but this work was not published until 1736, nearly 50 years after Raphson gave the simplified version of this method and it was considered superior. The use of the Hessian in computing the direction vector was not available in those years, thus attention was only focused on numerical computation (Fig. 1.1).

© Springer Nature Singapore Pte Ltd. 2019 S. K. Mishra and B. Ram, Introduction to Unconstrained Optimization with R, https://doi.org/10.1007/978-981-15-0894-3_1

1

2

1 Introduction

Fig. 1.1 I. Newton (1643–1727)

Unconstrained optimization deals with the problem of minimizing or maximizing nonlinear functions in the absence of any restrictions. Augustin-Louis Cauchy (1789–1857) in 1847 discovered the steepest descent method to accept negative gradient vector in solving a system of nonlinear equations. This method is also called a gradient method. Further, the idea of Cauchy had been studied in detail by several mathematicians to solve a large system of linear equations. There after, the gradient, method was adopted to solve the optimization problems. By the mid-twentieth century, very fast digital computers came to solve complex optimization problems and initiated further research on novel techniques. Kenneth Levenberg (1919–1973) developed the Levenberg algorithm in 1944 while working at the Frankford Army Arsenal, United States. It was again rediscovered in 1963 by Donald W. Marquardt (1929–1997), while working as a statistician at DuPont, USA, which was known as a Levenberg–Marquardt algorithm. The Levenberg–Marquardt method can be thought of as a combination of the steepest descent and Newton method. When the current solution is far from the correct solution, the method behaves like a steepest descent method, which guarantees to converge. When the current solution is close to the correct solution, it becomes a Newton’s method. The Fibonacci search and Golden Section Search Method were discovered by J.C. Kiefer in 1953 who was an American Statistician. In 1952, Magnus Rudolph Hestenes (1906–1991) and Eduard L. Stiefel (1909– 1978) developed the conjugate gradient method for the solution of a set of simultaneous linear equations with the availability of a symmetric positive definite matrix. The major developments in the area of numerical methods of unconstrained optimization

1.1 Historical Note on Unconstrained Optimization

3

Fig. 1.2 A.L. Cauchy (1789–1857)

problems were made in the United Kingdom during the 1960s. The classical Newton method draws the attention because of its faster convergence rather than the steepest descent method. But, the main drawback of this scheme is that it fails to produce a descent direction if the Hessian is not positive definite. To overcome this drawback, several Newton-like schemes appeared. William Copper Davidon (1927–2013) developed the rank two formula in 1959. Then after, Roger Fletcher (1939–2016), who was a British Mathematician and professor at the University of Dundee U.K., modified the rank two method in 1963. He was a Fellow of the Society for Industrial and Applied Mathematics (SIAM) and was elected as a Fellow of the Royal Society in 2003. Again the rank two method was subsequently modified by Michael J.D. Powell (1936–2015) in 1963. It paved the path to Quasi-Newton methods. Therefore, the rank two method was known as the Davidon-Fletcher-Powell (DFP) method. The calculation of the Hessian was computationally expensive. However, efforts were made to find a way to produce the Hessian more economically. The critical insights from which the current Quasi-Newton methods came, was proposed by Charles George Broyden (1933–2011) in 1969, who used the information from the current iteration for computing the new Hessian. In 1970, an alternative update formula was suggested independently by R. Fletcher, Donald Goldfarb (1941–), and David F. Shanno (1938–). This method is now called the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm. The concept of duality, or complementarity was used to develop this algorithm (Fig. 1.3).

4

1 Introduction

Fig. 1.3 E.L. Stiefel (1909–1978)

1.2 History of R S was created by John Chambers in 1976 at Bell Labs. John McKinley Chambers is the creator of the S programming language and core member of the R programming language project. He was awarded the 1998 ACM Software System Award for developing S. He donated his prize money to the American Statistical Association to endow an award for novel statistical software. He received a B.Sc. from the University of Toronto, Canada and M.A. and Ph.D. degrees in Statistics from Harvard University, United States. Chambers is a fellow of the American Statistical Association and the Institute of Mathematical Statistics. R is an implementation of the S programming language combined with lexical scoping semantics motivated by S scheme. Lexical scoping is a property of the program text. R was created by Ross Ihaka (1954–) and Robert Gentleman (1959–) at the University of Auckland, New Zealand, and is currently developed by the R Development Core Team, of which J. Chambers is a member. R is named partly after the first names of the first two R authors and partly as a play on the name of S. The project was conceived in 1992, with an initial version released in 1995 and a stable beta version in 2000 (Fig. 1.4).

1.2 History of R

5

Fig. 1.4 R.C. Gentleman (1959–)

Fig. 1.5 G.R. Ihaka (1954–)

R is a free software under the terms of the Free software Foundations’ GNU General Public License. It compiles and runs on UNIX, Windows, etc. (Fig. 1.5).

1.3 History of Algorithm The word algorithm comes from the name of the ninth-century Persian Muslim Mathematician Abu Abdullah Muhammad ibn Musa al-Khwarizmi. The word algorithm originally referred only to the rules of performing arithmetic using Hindu–Arabic numerals but evolved via European Latin translation of al-Khwarizmi’s name into

6

1 Introduction

Fig. 1.6 al-Khwarizmi (c780–c850)

algorithm by the eighteenth century. The use of the word evolved to include all definite procedures for solving problems or performing tasks. In 1939, J. Barkley Rosser (1907–1989) defined the algorithm in an effective mathematical method. It is the sense of a method in which each step is precisely determined and produced, the answer in a finite number of steps. Algorithms find their place in computer programs. The code written in any computer language solves the complex problem in a definite number of steps (Fig. 1.6).

1.4 Motivation to Use R in Optimization R is freely available under GNU General Public License, and pre-compiled binary versions statistical software, which are provided for various operating systems. Although R has a facility of command-line interface. There are several graphical front ends such as RStudio and Integrated Development Environments. The R language is widely used among statisticians and data miners for developing statistical software, for data analysis purposes; and studies of scholarly literature databases

1.4 Motivation to Use R in Optimization

7

show that R’s popularity has increased substantially in recent years. As of November 2019, R ranks 16th in the TIOBE index, a measure of the popularity of programming languages. This language provides specific features for using and managing datasets. Because of its numerical computations, R is a good environment to minimize the objective functions. If we talk about any other software such as MATLAB, then we must note that the cost of MATLAB software is very high and a normal student cannot afford it, only independent learners can. Finally, we have R software which provides a complete environment free of charges, and any optimization algorithm can be implemented on this platform.

Chapter 2

Mathematical Foundations

In this chapter, we recall notions and results from calculus and linear algebra that will be needed in the sequel of this book.

2.1 Vectors and Matrices We present a column vector of n entries denoted as ⎡ ⎤ x1 ⎢ x2 ⎥ ⎢ ⎥ x = ⎢ . ⎥, ⎣ .. ⎦

(2.1)

xn where xi is called the ith entry of the vector x. If we take the transpose of the vector x, then we get the following row vector:  x T = x1 x2 . . . xn .

(2.2)

The set of real numbers is denoted by R and the set of column vectors with real n n entries is denoted ⎡ entries ⎡ ⎤by R . The ⎤ of x ∈ R are denoted as x1 , x2 , . . . , xn . Two y1 x1 ⎢ y2 ⎥ ⎢ x2 ⎥ ⎢ ⎥ ⎢ ⎥ vectors x = ⎢ . ⎥ and y = ⎢ . ⎥ are equal if xi = yi , where i = 1, 2, . . . , n. The ⎣ .. ⎦ ⎣ .. ⎦ xn yn addition or subtraction of two vectors x and y denoted by x ± y is the vector as follows: © Springer Nature Singapore Pte Ltd. 2019 S. K. Mishra and B. Ram, Introduction to Unconstrained Optimization with R, https://doi.org/10.1007/978-981-15-0894-3_2

9

10

2 Mathematical Foundations

⎡

⎤ x1 ± y1 ⎢ x2 ± y2 ⎥ ⎢ ⎥ x ± y = ⎢ . ⎥. ⎣ .. ⎦ xn ± yn

Definition 2.1 (Vector Space) A non-empty set V is said to be a vector space over R if there exist maps + : V × V → V , defined by (x, y) → x + y, called addition, and . : R × V → V defined by (α, x) → α.x, called scalar multiplication satisfying the following properties: 1. 2. 3. 4. 5. 6. 7. 8.

Commutativity of addition: x + y = y + x. Associativity of addition: (x + y) + z = x + (y + z). Existence of additive identity: there exists 0 ∈ V such that x + 0 = x = 0 + x. Existence of additive inverse: for every x ∈ V , there exists an additive inverse y ∈ V such that x + y = 0 = y + x. This y is denoted by −x. α.(x + y) = α.x + α.y. (α + β).x = α.x + β.x. (αβ).x = x.(αβ). 1.x = x.

Definition 2.2 (Linearly Dependence) A finite, non-empty set of vectors {v1 , v2 , . . . , vk } in a vector space V are linearly dependent if there are scalars αi , i = 1, . . . , k, not all zeros, such that α1 v1 + α2 v2 + · · · + αk vk = 0.

(2.3)

Definition 2.3 (Linearly Independence) A finite, non-empty set of vectors {v1 , v2 , . . . , vk } in a vector space V is said to be linearly independent if there are scalars αi = 0, where i = 1, . . . , k, such that α1 v1 + α2 v2 + · · · + αk vk = 0. Theorem 2.1 Let v1 , v2 , . . . , vk be a set of vectors in a vector space V . Then {v1 , v2 , . . . , vk } is linearly dependent if and only if at least one of the vectors in the set can be expressed as a linear combination of the others. Proof If v1 , v2 , . . . , vk is linearly dependent, then according to Definition 2.2, there exist scalars α1 , α2 , . . . , αk , not all zeros, such that α1 v1 + α2 v2 + · · · + αk vk = 0. Suppose that αi = 0. Then, we can express vi as a linear combination of the other vectors as follows: vi = −

1 (α1 v1 + α2 v2 + · · · + αi−1 vi−1 + αi+1 vi+1 + · · · + αk vk ). αi

Conversely, suppose that one of the vectors, say, v j , can be expressed as a linear combination of the remaining vectors. That is,

2.1 Vectors and Matrices

11

v j = α1 v1 + α2 v2 + · · · + α j−1 v j−1 + α j+1 v j+1 + · · · + αk vk . Adding (−1)v j to both sides of this equation yields α1 v1 + α2 v2 + · · · + α j−1 v j−1 − v j + α j+1 v j+1 + · · · + αk vk = 0. Since the coefficient of v j is −1 = 0, thus the set of vectors {v1 , v2 , . . . , vk } is linearly dependent. This completes the proof.  ⎡ ⎤ ⎡ ⎤ 1 2 Example 2.1 If v1 = ⎣2⎦ and v2 = ⎣ 4 ⎦, then {v1 , v2 } is linearly dependent in R3 . 9 18 ⎡ ⎤ ⎡ ⎤ 2 1 Example 2.2 If v1 = ⎣−4⎦ and v2 = ⎣ 3 ⎦, then {v1 , v2 } is linearly independent 7 −8 in R3 . The absolute value of a real number a, denoted by |a|, is presented as

|a| =

a if a ≥ 0, −a if a < 0.

(2.4)

The following properties hold 1. 2. 3. 4. 5. 6.

|a| = | − a|, −|a| ≤ a ≤ |a|, |a + b| ≤ |a| + |b|, a| − |b ≤ |a| + |b|, |ab| = |a||b|, and |a| ≤ c and |b| ≤ d imply that |a + b| < c + d.

Definition 2.4 (Inner Product) The inner product is a real-valued function . , . : Rn × Rn → R satisfying the following properties: 1. Positive definiteness: x , x ≥ 0, for any x ∈ Rn and x , x = 0 if and only if x = 0. 2. Symmetry: x , y = y , x , for any x, y ∈ Rn . 3. Additivity: x + y , z = x , z + y , z , for any x, y, z ∈ Rn . 4. Homogeneity: αx , y = α x , y , for any α ∈ R and x, y ∈ Rn . The inner product called dot product can be presented as x , y =

n  i=1

xi yi = x T y, for all x, y ∈ Rn .

(2.5)

12

2 Mathematical Foundations

⎡

⎤ ⎡ ⎤ 1 2 Example 2.3 Consider two vectors x = ⎣−3⎦ and y = ⎣3⎦. The inner product is 2 1 given as ⎡ ⎤ 2  x , y = x T y = 1 −3 2 ⎣3⎦ = 2 − 9 + 2 = −5. 1 Theorem 2.2 (Cauchy–Schwarz inequality) If x and y are vectors in an inner product space V , then x , y 2 ≤ x , x y , y . Proof If x = 0, it is clear. Assume x = 0. For any scalar α, we have 0 ≤ αx + y , αx + y = x , x α 2 + 2 x , y α + y , y . This inequality implies that the polynomial x , x α 2 + 2 x , y α + y , y in α has either no real roots or a repeated real root. Therefore, its discriminant must be nonpositive: x , y 2 − x , x y , y ≤ 0, which implies the inequality. This completes the proof.



Definition 2.5 (Vector Norm) A vector norm . on Rn is a function . : Rn → R satisfying the following properties: 1. Non-negativity: x ≥ 0, for any x ∈ Rn , and x = 0 if and only if x = 0. 2. Positive homogeneity: αx = |α|x, for any α ∈ R and x ∈ Rn . 3. Triangle inequality: x + y ≤ x + y, for any x, y ∈ Rn . We present the proof of triangle inequality using the Cauchy–Schwarz inequality as follows: x + y2 = x2 + 2x T y + y2 . From the Cauchy–Schwarz inequality, x + y2 ≤ x2 + 2xy + y2 = (x + y)2 . Therefore, x + y ≤ x + y. Note that if x and y are orthogonal, that is, x T y = 0, then x + y2 = x2 + y2 . This is called the Pythagorean theorem for Rn .

2.1 Vectors and Matrices

13

One natural way to generate a norm on Rn is to take any inner product . , . on R and define the associated norm as follows: n

x =



x, x for all x ∈ Rn ,

(2.6)

which can be easily seen to be the norm. A p-norm or l p -norm is given as 1

x p = |x1 | p + |x2 | p + · · · + |xn | p p =

 1p  n p |xi | for all x ∈ Rn , p > 0. i=1

(2.7) Definition 2.6 (Orthogonal) A set of vectors {v1 , v2 , . . . , vk } are mutually orthogonal if every pair of vectors is orthogonal, that is, (vi )T v j = 0, for all i = j, i, j = 1, 2, . . . , k. ⎧⎡ ⎤ ⎡ ⎤ ⎡ ⎤⎫ 1 ⎨ 1 ⎬ √1 √ Example 2.4 The set of vectors ⎣ 0 ⎦ , ⎣ 2⎦ , ⎣− 2⎦ is mutually orthogo⎩ ⎭ −1 1 1 nal, that is, ⎡ ⎤  √1 1 0 −1 ⎣ 2⎦ = 0, 1 ⎡ ⎤ 1  √ 1 0 −1 ⎣− 2⎦ = 0, 1 ⎡ ⎤ 1  √ √ 1 2 1 ⎣− 2⎦ = 0. 1 A matrix is a rectangular array of numbers, commonly denoted by upper case letters A. A matrix with m rows and n columns is called an m × n matrix as given below: ⎤ ⎡ a11 a12 . . . a1n ⎢ a21 a22 . . . a2n ⎥ ⎥ ⎢ (2.8) A=⎢ . .. .. ⎥ . . ⎣ . . . ⎦ am1 am2 . . . amn The real number ai j located in the ith row and jth column is called the (i, j)th entry of matrix A.

14

2 Mathematical Foundations

Suppose that A, B, C are m × n matrices. O is zero matrix and α, β are scalars. Then, the following rules of matrix addition and scalar multiplication hold: 1. 2. 3. 4. 5. 6. 7.

Commutativity: A + B = B + A. Additive identity: A + O = O + A. Additive inverse: A + (−A) = O = (−A) + A. Associativity: A + (B + C) = (A + B) + C. Distributive property for matrices: α(A + B) = α A + β B. Distributive property for scalars: (α + β)A = α A + β A. Associativity property for scalars: (αβ)A = α(β A).

Transpose of Matrix Given a matrix A ∈ Rm×n , the transpose of A is n × m whose rows are the columns of A. The transpose of matrix A is denoted by A T . ⎡ ⎤   1 2 19 4 Example 2.5 For matrix A = ⎣9 3 ⎦, the transpose of A is A T = . 2 3 −9 4 −9 Consider two matrices A, B ∈ Rm×n and a scalar α ∈ R, we have the following algebraic rules for transposes: 1. 2. 3. 4.

(A T )T = A. (α A)T = α A T . (A + B)T = A T + B T . (AB)T = B T A T .

Symmetric Matrix A square matrix A ∈ Rn×n is said to be a symmetric matrix if A = A T . Rank of a Matrix ⎡

a11 ⎢ a21 ⎢ Given that A = ⎢ . ⎣ .. am1 ak as below:

a12 a22 .. .

... ...

am2 . . .

⎤ a1n a2n ⎥ ⎥ .. ⎥. Let us denote the kth column of A by . ⎦ amn ⎡

⎤ a1k ⎢ a2k ⎥ ⎢ ⎥ ak = ⎢ . ⎥ . ⎣ .. ⎦ amk

The maximal number of linearly independent rows (columns) of A is called the rank of the matrix A, denoted by rank(A).

2.1 Vectors and Matrices

15

⎡

⎤ 123 Example 2.6 Given that A = ⎣4 5 6⎦. Since the first two rows are linearly inde579 pendent, thus rank(A) = 2. Matrix Multiplication Two matrices A ∈ Rm×n and B ∈ Rs×t are multiplied if n = s, i.e., the number of columns of matrix A should be equal to the number of rows of matrix B and the product can be expressed as n   aik bk j , i = 1, . . . , m, and j = 1, . . . , t. AB = C = ci j m×t , where ci j = k=1

 Example 2.7 Consider two matrices A =

   1 2 1 and B = , then the product −3 4 2

of A and B is  AB =

1 2 −3 4

      1 1.1 + 2.2 5 = = . 2 −3.1 + 4.2 5

Identity Matrix The identity matrix In of order n is defined by In = [δi j ]n×n , where

δi j =

1, i = j, 0, i = j.

In other words, a square matrix in which all elements of the principal diagonal are ones and all other elements are zeros. ⎡ ⎤ ⎡ ⎤ 100 123 Example 2.8 For I3 = ⎣0 1 0⎦ and A = ⎣4 5 6⎦ , we have I3 A = AI3 = A. 001 789

2.2 Eigenvalues and Eigenvectors Let A be an n × n real square matrix. A nonzero vector x ∈ Rn is called an eigenvector (or characteristic vector) of A if there is a scalar λ ∈ R such that Ax = λx.

(2.9)

16

2 Mathematical Foundations

The scalar λ is called an eigenvalue (or characteristic value) of A and we say x belongs to λ. An eigenvalue λ and an eigenvector x are two unknowns in the system (λI − A)x = 0. To find these unknowns, first, we should determine an eigenvalue λ by using the fact that the equation (λI − A)x = 0 has a nontrivial solution x if and only if λ satisfies the following equation: det[λI − A] = 0.

(2.10)

We call the polynomial det[λI − A] is the characteristic polynomial of the matrix A, and the above equation is called the characteristic equation. Thus, the eigenvalues are just the roots of the characteristic equation det(λI − A) = 0. The eigenvectors of A can be determined by solving the homogeneous system (λI − A)x = 0 for each eigenvalue A. Theorem 2.3 Suppose that the characteristic equation det[λI − A] = 0 has n distinct roots λ1 , λ2 , . . . , λn . Then, there exist n linearly independent vectors v1 , v2 , . . . , vn such that Avi = λi vi , i = 1, 2, . . . , n. Proof The proof is out of the scope of the book. See page no. 34 of Chong et al. (2013). Theorem 2.4 All eigenvalues of a symmetric matrix are real. Proof Let Ax = λx, where x = 0. Taking the inner product of Ax with x, then Ax, x = λx, x = λ x, x .

(2.11)

¯ Ax, x = x, A T x = x, Ax = x, λx = λ x, x .

(2.12)

On the other hand,

The above follows from the definition of the inner product, we note that x, x is real and x, x > 0. Hence, λ x, x = λ¯ x, x , that is, ¯ (λ − λ) x, x = 0. Since x, x > 0, therefore, Thus, λ is real. This completes the proof.

¯ λ = λ. 

2.2 Eigenvalues and Eigenvectors

17

Theorem 2.5 Any real symmetric n × n matrix has a set of n eigenvectors that are mutually orthogonal. Proof We present the proof for n distinct eigenvalues. Suppose that Av1 = λ1 v1 , Av2 = λ2 v2 , where λ1 = λ2 . Then, Av1 , v2 = λ1 v1 , v2 = λ1 v1 , v2 .

(2.13)

Since A = A T , therefore Av1 , v2 = v1 , A T v2 = v1 , λ2 v2 = λ2 v1 , v2 .

(2.14)

From (2.13) and (2.14), we get λ1 v1 , v2 = λ2 v1 , v2 . Since λ1 = λ2 , therefore, v1 , v2 = 0. This completes the proof.   6 −1 Example 2.9 Find all eigenvalues and eigenvectors of matrix A = . 1 4



The characteristic equation of matrix A is 

         6 −1 10 6 −1 λ0 6 − λ −1 A − λI = −λ = − = . 1 4 01 1 4 0λ 1 4−λ The eigenvalues of matrix A is given by det (A − λI ) = 0,   6 − λ −1     1 4 − λ = 0, λ2 − 10λ + 25 = 0, (λ − 5)2 = 0. We get two eigenvalues λ1 = 5 and λ2 = 5. The eigenvector corresponding to eigenvalue λ = λ1 = λ2 = 5 is (A − λI )X = 0,       6 −1 10 x1 = 0, −4 x2 1 4 01    2 −1 x1 = 0. 1 0 x2

18

2 Mathematical Foundations

Thus, we get 2x1 − x2 = 0 and x1 = 0. The eigenvectors are

  0 . 0

n 1/2 √ Theorem 2.6 Let x = |x |2 = x , x . The matrix norm induced by √k=1 k this vector norm is A = λ1 , where λ1 is the largest eigenvalue of the matrix A T A. Proof The proof is out of the scope of the book. See page no. 37 of Chong et al. (2013).  Definition 2.7 (Rayleigh’s Inequality) If an n × n matrix Q = Q T > 0 is a real symmetric positive definite matrix, then λmin (Q)x2 ≤ x T Qx ≤ λmax (Q)x2 ,

(2.15)

where λmin (Q) denotes the smallest eigenvalue of Q and λmax (Q) denotes the largest eigenvalue of Q. For Q = Q T > 0, we also have λmin (Q −1 ) =

1 , λmax (Q)

(2.16)

λmax (Q −1 ) =

1 , λmin (Q)

(2.17)

and λmin (Q −1 )x2 ≤ x T Q −1 x ≤ λmax (Q −1 )x2 .

(2.18)

  31 and let the norm in R2 be given by Example 2.10 Consider the matrix A = 13    31 x = x12 + x22 . Then A T = A = . 13 

We find A AT =

31 13



   31 10 6 = . 13 6 10

The characteristics equation of matrix A A T is A A T − λI =

      10 6 10 10 − λ 6 −λ = 6 10 01 6 10 − λ.

The eigenvalues of above matrix is obtained as

2.2 Eigenvalues and Eigenvectors

19

det (A A T − λI ) = 0,   10 − λ 6     6 10 − λ = 0, (10 − λ)2 − 36 = 0, 100 + λ2 − 20λ − 36 = 0, λ2 − 20λ + 64 = 0, (λ − 16)(λ − 4) = 0. We get two eigenvalues λ1 = 16 and λ2 = 4. We have A = vector of A T A corresponding to λ1 = 16 is

√

16 = 4. The eigen-

(A − λ1 I )X = 0,       10 6 10 x1 = 0, − 16 x2 6 10 01    −6 6 x1 = 0. 6 −6 x2 Thus, we get x1 = x2 . We can have x1 =

√1 2

  1 . Note that 1

√          1 31 1  1 2 1  4 2 4 2     = √ 4 + 4 = √ = 4. =√  Ax1  =  √2 13 1  2 2 2 4  Since A = A T , we also have A = max1≤i≤n |λi (A)|, where λ1 (A), . . . , λn (A) are the eigenvalues of A. A sequence of real numbers is a function whose domain is the set of natural numbers 1, 2, . . . , k, . . . and whose range is contained in R. Thus, a sequence of real numbers can be viewed as a set of numbers {x1 , x2 , . . . , xk , . . . }, which is often denoted as {xk }. A number x ∗ ∈ R is called the limit of the sequence {x(k) }, if for any positive ε there is a number K such that for all k > K , |xk − x ∗ | < ε, that is, xk lies between x ∗ − ε and x ∗ + ε for all k > K . We write x ∗ = lim xk , or xk → x ∗ . k→∞

A sequence that has a limit is called a convergent sequence. The notion of a sequence can be extended to sequences with elements in Rn . A sequence in Rn is a function whose domain is the set of natural numbers 1, 2, . . . , k, . . . and whose range is contained in Rn . We use the notation {x (1) , x (2) , . . . } or x (k) for sequences in Rn . We say x ∗ ∈ Rn is called the limit of the sequence {x (k) } if for any positive ε there is a number K such that for all k > K , x (k) − x ∗  < ε. If x (k) is convergent, we write

20

2 Mathematical Foundations

Fig. 2.1 Graph of Himmelblau’s function

x2

z

x1

0 −6

x2

4

Fig. 2.2 Level sets of Himmelblau’s function

−6 −4 −2

0

x1

2

4

6

x ∗ = lim x (k) , or x (k) → x ∗ . k→∞

Definition 2.8 (Level Set) The level set of a function f : Rn → R at level c is the set of points S = {x : f (x) = c}. For f : R2 → R, we are interested in S when it is a curve. For f : R3 → R, the sets S most often considered are surfaces. Example 2.11 Consider the following real-valued function on R2 : f (x) =

(x12

+ y − 11) + (x1 + 2

x22

  x − 7) , x = 1 x2 2

The above function is called Himmelblau’s function. A plot of the function f (x) is shown in Fig. 2.1. The level sets of f (x) at levels 2, 2.5, 3 are depicted in Fig. 2.2.

2.3 Neighborhoods A neighborhood of a given point x ∈ Rn is the set {y ∈ Rn : y − x < ε},

(2.19)

2.3 Neighborhoods Fig. 2.3 Example of neighborhoods of a point in Rn

21

B(x, )

x

where ε is some positive number. The neighborhood is also called the ball with radius ε and center x. See Fig. 2.3.   x 2 In the plane R , a neighborhood of x = 1 consists of all the points inside of a x2 ⎡ ⎤ x1 disk centered at x. In R3 , a neighborhood of x = ⎣x2 ⎦ consists of all points inside x3 of a sphere centered at x. Interior Point A point x in a set  ⊆ Rn is an interior point of the set  if there exists an ε > 0 for which B(x, ε) ⊆ . In other words, a point x is said to be an interior point of the set S if the set S contains some neighborhood of x, that is, if all the points within some neighborhood of x are also in S. The set of all the interior points of S is called the interior of S. Example 2.12 Let S = [1, 3] . Then 25 is an interior point of S but neither 1 or 3 are interior points of S because [1, 3] is not a neighborhood of 1 and 3. Boundary Point A point x is said to be a boundary point of the set S if every neighborhood of x contains a point in S and a point not in S. Note that the boundary point of S may or may not be an element of S. The set of all boundary points of S is called the boundary of S. See Fig. 2.4.

Open Set A set S is said to be open if it contains a neighborhood of each of its points, that is, if each of its points is an interior point or equivalently if S contains no boundary

22

2 Mathematical Foundations

Fig. 2.4 x is an interior point and y is a boundary point

S B(x, )

x

y Fig. 2.5 S1 is open set and S2 is close set

x2

4

2

S1

S2

1

x1 1

points. In Fig. 2.5, S1 =

2

3

4

5

   x1 : 1 < x1 < 2, 1 < x2 < 2 is an open set. x2

Closed Set A set S is said to be closed if it contains its boundary . We can if  say that a set is closed  x1 : 3 ≤ x1 ≤ 4, 1≤x2 ≤2 and only if its complement is open. In Fig. 2.6, S2 = x2 is a closed set. A set that is contained in a ball of finite radius is said to be bounded. A set is compact if it is both closed and bounded. Compact sets are important in optimization problems. Definition 2.9 (Continuous Function) A function f : Rn → R is said to be contin¯ < δ which uous at x¯ ∈ Rn if, for any ε > 0, there exists δ > 0 such that x − x implies | f (x) − f (x)| ¯ < ε.

2.3 Neighborhoods

23

If f is continuous at every point in an open set  ⊂ Rn , then f is said to be continuous on . A continuous function f : Rn → R is said to be continuously differentiable at x ∈ Rn , if ∂∂xfi (x), where i = 1, . . . , n, exists and is continuous. If f is continuously differentiable at every point of an open set  ⊂ Rn , then f is said to be continuously differentiable on  and denoted by f ∈ C 1 . A continuously differentiable function f : Rn → R is called twice continuously 2 differentiable at x ∈ Rn if ∂ x∂i ∂fx j (x), where i, j = 1, . . . , n, exists and is continuous. If f is twice continuously differentiable at every point of an open set  ⊂ Rn , then f is said to be twice continuously differentiable on  and denoted by f ∈ C 2 .

2.4 Algorithm An algorithm is a collection of sequential instructions that is used to solve computational problems. The method involved in the algorithm is either sequential or iterative. The instructions used in the algorithm should be well-defined so that these can be implemented as a computer program. A computer program is a collection of sequential instructions written by a computer programmer in any programming language that performs a specific task when executed by a computer. Optimization algorithms execute a sequence of iterations to solve an optimization problem. The algorithm starts with an initial guess point and creates a sequence of points x (k) that converges to an optimal point to satisfy certain optimality conditions. The rate at which the iterations approach the optimal point is called the convergence rate of the algorithm. The algorithm converges faster if it takes a short time to obtain the optimal solution. Stopping Criteria A stopping criterion is a useful tool to stop an iterative algorithm. In an optimization algorithm, the process starts by choosing an initial guess point and generates the next point with the help of any line search methods (discussed in Chap. 4) that minimizes the function at each iteration. It is necessary to use the stopping criteria in the iterative algorithm to terminate the search process in order to minimize nonlinear functions. The following stopping criteria are responsible to stop the algorithm: 1. We may compute the absolute difference as | f (x (k+1) ) − f (x (k) )| between two function values at x (k) and x (k+1) , respectively for every two successive iterations, and if the difference is less than some prespecified threshold value ε, then the algorithm stops which can be expressed as follows: | f (x (k+1) ) − f (x (k) )| < ε, where ε can be called as prespecified optimization tolerance value.

(2.20)

24

2 Mathematical Foundations

2. We may compute the norm of the difference x (k+1) − x (k)  between two points x (k) and x (k+1) for every two successive iterates. The algorithm stops if the norm is less than a prespecified threshold value ε which can be expressed as x (k+1) − x (k) < ε.

(2.21)

3. We may compute the gradient of function as ∇ f (x (k) ) at a point x (k) that should be less than the threshold value to terminate the iterative algorithm. This can be given as ∇ f (x (k) ) < ε.

(2.22)

4. We may check the following r elative value to terminate the iterative algorithm: | f (x (k+1) ) − f (x (k) )| < ε. | f (x (k) )|

(2.23)

Remark 2.1 The stopping criteria discussed above are relevant to all iterative algorithms presented in this book. In unconstrained optimization methods, we present algorithms that are simply based on the following three steps: 1. Initialize the current guess point. 2. Generate a new point from the current guess point. 3. If the stopping criterion is not satisfied and the progress is being made, use the new guess as the current guess and repeat the process from step 2. Theorem 2.7 A convergent sequence has only one limit point. Proof We prove this result by contradiction. Suppose that a sequence {x (k) } has two different limits, say x1 and x2 . Then, we can have x1 − x2  > 0. Let ε = 21 x1 − x2 . From the definition of a limit, there exist K 1 and K 2 such that for k > K 1 , we have x k − x1  < ε,

(2.24)

x k − x2  < ε.

(2.25)

and for k > K 2 , we have

Let K = max(K 1 , K 2 ). Then, if k > K , we have x (k) − x1  < ε and x (k) − x2  < ε. Adding (2.24) and (2.25) to yield x (k) − x1  + x (k) − x2  < 2ε.

(2.26)

2.4 Algorithm

25

Applying the triangle inequality to give −x1 + x2  = x (k) − x1 − x (k) + x2  = (x (k) − x1 ) − (x (k) − x2 ) ≤ x (k) − x1  + x (k) − x2 . (2.27) From (2.26) and (2.27), we get −x1 + x2  = x1 − x2  < 2ε. However, the above contradicts the assumption that x1 − x2  = 2ε. This completes the proof.



Theorem 2.8 Every convergent sequence is bounded. Proof Let x (k) be a convergence sequence with limit point x ∗ . Then by the definitions of the limit point, there exists a natural number n ∈ k. such that for all k > K , we have x (k) − x ∗  < ε. Set ε = 1 in the above inequality, we get x (k) − x ∗  < 1.

(2.28)

From the property of triangle inequality, we know that x (k)  − x ∗  ≤ x (k) − x ∗ .

(2.29)

From (2.28) and (2.29), we have x (k)  − x ∗  < 1, that is, x (k)  < x ∗  + 1, ∀ k > K .  Let B = max x (1) , x (2) , . . . , x (k) , x ∗  + 1 . We get B ≥ x (k) , ∀ k. It means that the sequence {x (k) } is bounded. This completes the proof.



Big O and little o notations Let g be a real-valued function defined in some neighborhood of 0 ∈ Rn , with g(x) = 0 if x = 0. Let f :  → Rm be defined in a domain  ⊂ Rn that includes 0. Then, we write

26

2 Mathematical Foundations

1. f (x) = O(g(x)) to mean that the quotient

 f (x) g(x)

is bounded near 0, that is, there

exist numbers K > 0 and δ > 0 such that if x < δ, x ∈ , then 2. f (x) = o(g(x)) to mean that

 f (x) |g(x)|

≤ K.

 f (x) = 0. x→0,x∈ |g(x)| lim

The symbol O(g(x)) read as “big-oh of g(x)” is used to represent a function that is bounded by a scaled  version of g in a neighborhood of 0. For example, x = x3 = O(x 2 ). On the other hand, o(g(x)) read as “little-oh of O(x) and 3x 2 + 2x 4 g(x)” represents a function that goes to zero “faster” than g(x) that   in3the sense x o(g(x)) = o(x), lim = 0. Examples of such functions are x 3 = o(x), 3x 2 + 2x 4 x→0 |g(x)| 4 3 and x = o(x ).

2.5 Taylor’s Theorem Taylor’s theorem is named after the scientist Brook Taylor (1685–1731) who in 1715 was the first one to show that any differentiable function can be approximated through an expansion of its derivatives with known coefficients. The Taylor series is a tool for approximating a function f near a specified point x0 . This approximation obtained is a polynomial which is easy to manipulate. It has the following uses: 1. It allows estimating the value of the function near the given point when the function is difficult to evaluate directly. 2. The derivatives of the approximation can be used to estimate the derivatives of the original function. 3. It is used to derive many algorithms for minimizing or maximizing the function. Theorem 2.9 (Taylor’s Theorem) Let f ∈ C n+1 ([α, β]) and x0 ∈ [α, β]. Then, for each x ∈ (α, β), there exists ξ = ξ(x) that lies between x0 and x such that f (x) = Pn (x) + Rn (x), where n  f (k) (x0 ) (x − x0 )k , Pn (x) = k! k=0

and Rn (x) =

f (n+1) (ξ ) (x − x0 )n+1 . (n + 1)!

The polynomial Pn (x) is called the Taylor polynomial of degree n. Proof The proof is beyond the scope of this book. See page no. 72 of Chong et al. (2013). 

2.5 Taylor’s Theorem

27

We use Taylor’s Theorem for multivariable functions in the following forms: 1. If f : Rn → R is a first-order continuously differentiable function, then ¯ f (x) = f (x) ¯ + ∇ f (x) ˜ T (x − x),

(2.30)

where x˜ is a point on the line segment connecting x¯ and x. 2. If f : Rn → R is a second-order continuously differentiable function, then 1 ¯ + (x − x) ˜ − x), ¯ f (x) = f (x) ¯ + ∇ f (x) ¯ T (x − x) ¯ T ∇ 2 f (x)(x 2

(2.31)

where x˜ is a point on the line segment between x¯ and x. The error terms are often represented as big O and little o notations as ¯ + O(x − x ¯ 2 ); f (x) = f (x) ¯ + ∇ f (x) ¯ T (x − x)

(2.32)

f (x) = f (x) ¯ + ∇ f (x) ¯ (x − x) ¯ + o(x − x). ¯

(2.33)

T

3. If f : Rn → R is third-order continuously differentiable function, then 1 ¯ T ∇ 2 f (x)(x ¯ + (x − x) ¯ − x) ¯ + R2 (x), f (x) = f (x) ¯ + ∇ f (x) ¯ T (x − x) 2 (2.34) ¯ 2 ). where the error term R2 (x) can be expressed as R2 (x) = o(x − x

2.6 Quadratic Functions Quadratic functions are widely used in optimization. A homogeneous polynomial of second degree of any number of variables is called a quadratic function. For example, f (x1 , x2 ) = 3x12 + 6x1 x2 − 8x22 is a quadratic function of two variables and f (x1 , x2 , x3 ) = 4x12 + 3x22 + 4x32 − 3x1 x2 + 5x3 x1 is a quadratic function of three variables. In general, f (x1 , x2 , . . . , xn ) =

n  n 

ai j xi x j

(2.35)

i=1 j=1

is a quadratic function in n-variables. In the above polynomial, the coefficients of x12 , x22 , . . . , xn2 are a11 , a22 , . . . , ann , respectively, and the coefficient of a product term xi x j where i = j is ai j + a ji . If

28

2 Mathematical Foundations

⎡

a11 ⎢a21 ⎢ Q=⎢ . ⎣ ..

⎡ ⎤ ⎤ x1 a1n ⎢ x2 ⎥ a2n ⎥ ⎢ ⎥ ⎥ .. ⎥ , and x = ⎢ .. ⎥ , ⎣.⎦ . ⎦ . . . ann xn

a12 . . . a22 . . . .. .

an1 an2

then a quadratic of function f : Rn → R is given as f (x) = x T Qx,

(2.36)

where Q ia an n × n real matrix and x ∈ Rn . Without loss of generality, we assume that Q is a real symmetric matrix, that is, Q = Q T . Remark 2.2 Note that if Q is not a symmetric matrix, then we replace Q with the symmetric matrix Q 0 as follows: Q 0 = Q 0T =

1 (Q + Q T ). 2

(2.37)

Therefore, we get the quadratic function as 1 1 f (x) = x T Qx = x T Q 0 x = x T ( Q + Q T )x. 2 2

(2.38)

Example 2.13 Find the matrix of the quadratic function: f (x1 , x2 , x3 ) = a11 x12 + a22 x22 + a33 x32 + 2a12 x1 x2 + 2a23 x2 x3 + 2a31 x3 x1 . Since the matrix of a quadratic form is symmetric, therefore a11 = a21 , a23 = a32 , and a31 = a13 . Thus, f (x1 , x2 , x3 ) = a11 x12 + a22 x22 + a33 x32 + a12 x1 x2 + a21 x1 x2 + a23 x2 x3 + a32 x3 x2 + a31 x3 x1 + a13 x3 x1 . Thus, ⎡ ⎤⎡ ⎤ x1 a11 a12 a13 f (x1 , x2 , x3 ) = x1 x2 x3 ⎣a21 a22 a23 ⎦ ⎣x2 ⎦ . a31 a32 a33 x3 

⎡ ⎤ ⎡ ⎤ a11 a12 a13 x1 It is equivalent as f (x) = x T Qx, where x = ⎣x2 ⎦, and Q = ⎣a21 a22 a23 ⎦ . x3 a31 a32 a33 Example 2.14 Write the following quadratic function in matrix notation: f (x1 , x2 , x3 ) = x12 + 2x22 + 3x32 + 4x1 x2 + 5x2 x3 + 6x3 x1 .

2.6 Quadratic Functions

29

We have a11 = 1, a22 = 2, a33 = 3, a12 = a21 = 2, a23 = a32 = 25 , and a31 = a13 = 3. Therefore, ⎡ ⎤⎡ ⎤ x1 1 2 35 f (x1 , x2 , x3 ) = x1 x2 x3 ⎣2 2 2 ⎦ ⎣x2 ⎦ . x3 3 25 3 

Definition 2.10 (Positive Definiteness for Quadratic Function) The quadratic form x T Qx, where x ∈ Rn is said to be 1. 2. 3. 4.

Positive semidefinite if x T Qx ≥ 0, for all x. Positive definite if x T Qx > 0, for all nonzero vector x. Negative semidefinite if x T Qx ≤ 0, for all x. Negative definite, if x T Qx < 0, for all nonzero vector x.

Definition 2.11 (Positive Definiteness for Symmetric matrix) If the quadratic function x T Qx is 1. 2. 3. 4.

Positive semidefinite, then matrix Q is to be positive semidefinite. Positive definite, then matrix Q is said to be positive definite. Negative semidefinite, then matrix Q is said to be negative semidefinite. Negative definite, then matrix Q is to be negative definite.

Remark 2.3 If there exist x and y such that x T Qx is positive and y T Qy is negative, then Q is called indefinite. Example 2.15 For which real numbers k is the quadratic form f (x) = kx12 − 6x1 x2 + kx22 positive definite? To determine the definiteness, we need to consider the matrix   k −3 A= −3 k whose characteristic polynomial is det (A − λI ) = (k − λ)2 − 9 = λ2 − 2kλ + (k 2 − 9), that is, det (A − λI ) = (λ − (k + 3))(λ − (k − 3)) = 0. We find that λ1 = k + 3 and λ2 = k − 3. In order for f to be positive definite, both of these eigenvalues must be positive, and in particular we must have λ2 > 0. Thus, k > 3 is a necessary and sufficient condition for f to be a positive definite quadratic form.

30

2 Mathematical Foundations

Sylvester’s Criteria for a Positive Definite Matrix Let Q be an n × n real symmetric matrix such as ⎡

a11 ⎢a21 ⎢ Q=⎢ . ⎣ ..

a12 . . . a22 . . . .. .

⎤ a1n a2n ⎥ ⎥ .. ⎥ . . ⎦

an1 an2 . . . ann

Minors of the above matrix Q are the determinants of the matrices obtained by successively removing rows and columns from Q. The leading principal minors are detQ and the minors are obtained by successively removing the last row and the last columns. That is, the leading principal minors are

1 = a11 ,   a a 

2 = a11 a12  , 21 22   a11 a12 a13   

3 = a21 a22 a23  , a31 a32 a33  .. .   a11 a12 . . . a1n    a21 a22 . . . a2n  

n =  .. .. ..  . .   . . an1 a2 . . . ann  Note that if all principal minors of Q are positive, then Q is positive definite. Theorem 2.10 A quadratic function f (x) = x T Qx, where Q = Q T , is positive definite if and only if the leading principal minors of Q are positive. Proof The proof is beyond the scope of this book. See page no. 32 of Chong et al. (2013).  Example ⎡ 2.16 ⎤Apply Sylvester’s test to check the positive definiteness of the matrix 1 −2 3 A = ⎣4 1 2⎦ . We have 3 −1 2

1 = 1 > 0,   1 −2  = 1 + 8 = 9 > 0,

2 =  4 1   1 −2 3  

3 = 4 1 2 = 1(2 + 2) + 2(8 − 6) + 3(−4 − 3) = 4 + 4 − 21 = −13. 3 −1 2

2.6 Quadratic Functions

31

Since 3 < 0, therefore, A is not positive definite. Remark 2.4 Note that if Q is not symmetric, Sylvester’s criterion cannot be used to check positive definiteness of the quadratic function f (x) = x T Qx.   1 0 T . Example 2.17 Consider f (x) = x Qx, where Q = −6 1 The leading principal minors of Q are

1 = 1 > 0, and 2 = detQ = 1 > 0.   1 However, if x = , then 1  x T Qx = 1 1



1 0 −6 1

     1 1 = −5 1 = −4 < 0. 1 1

Thus, the associated quadratic function is not positive definite. Note that  x Qx = x T

T

       1 T 1 0 1 0 1 −6 1 −3 T x= x + x=x x = x T Q 0 x. −6 1 −6 1 0 1 −3 1 2

The leading principal minors of Q 0 are 1 = 1 > 0 and 2 = detQ 0 = −8 < 0, as expected.

2.7 Exercises 1. Determine whether the given vectors are linearly dependent or linearly independent in R4 . a. v1 = (1, 2, −3, 4), v2 = (3, −1, 2, 1), and v3 = (1, −5, 8, −7). b. v1 = (2, 5, 2, −3), v2 = (3, 6, 5, 2), v3 =(4, 5, 14, 14), and v4 = (5, 10, 8, 4). 2. In this problem, the matrices A and B are given as 

 32 1 A= , 0 4 −1 If the sum is defined, then find a. A + B, b. A T + B, c. A T + B T .



 0 1 1 B= . 2 −1 4

32

2 Mathematical Foundations

⎡ ⎤ ⎡ ⎤ 1 3 3. Find the inner product of x = ⎣2⎦ and y = ⎣−2⎦ . 3 3 4. Compute the p-norm for p = 1, ∞ of the matrices A and B as given below: ⎡ ⎤ 12 3 a. A = ⎣2 4 7 ⎦ , 3 6 10 ⎡ ⎤ 9 −4 2 b. B = ⎣3 −5 4 ⎦ . 8 1 −6 ⎡ ⎤ 1 −1 1 −1 ⎢−1 1 −1 1 ⎥ ⎥ 5. Find the rank of the matrix A = ⎢ ⎣ 1 −1 1 −1⎦ . −1 1 −1 1   2 2 . 6. Determine the eigenvalues of the matrix A = 5 −1   12 7. Find the λmin in the Rayleigh inequality for the matrix A = . 21 " !1 8. Show that the set n : n ∈ N is not open. 9. Show that interior and boundary points of a set are limit points of the set. 10. Are the following functions continuous? a. f (x) = x1 , b. f (x) = ln x. 11. Determine positive definiteness of f (x) = x12 + 2x1 x2 + x22 . 12. Determine the definiteness of f (x1 , x2 ) = x12 + 2x1 x2 + x22 . 13. 14. 15. 16.

17. 18.

⎤ 146 Determine the definiteness of the symmetric 3 × 3 matrix A = ⎣4 2 1⎦ . 616 Determine the definiteness of the quadratic function f (x1 , x2 , x3 ) = 3x12 + 6x1 x3 + x22 − 4x2 x3 + 8x32 . using the chain rule for the function z(x, y) = x 2 y + x + y with Evaluate dz dt x(t) = log t, and y(t) = t 2 . 4 Write down the Taylor series expansion  of the function f (x1 , x2 ) = x1 + 1 2x12 x22 + x24 about the given point x0 = . Neglect terms of order three or 1 higher. Determine linear approximation of the function f (x) = x + x1 − 5 about the point x0 = 2. Obtain a quadratic approximation for the function f (x) = 3x13 + x22 + x12 x23 + 4x1 x2 + 2

  1 at point x + δ if x = . 1

⎡

2.7 Exercises

33

19. Write an algorithm to multiply two matrices. 20. Write the following quadratic forms in matrix notation: a. f (x1 , x2 , x3 , x4 ) = x1 x2 + x2 x3 + x3 x1 + x1 x4 + x2 x4 + x3 x4 . b. f (x1 , x2 , x3 , x4 ) = x12 − 2x22 + 4x32 − 4x42 − 2x1 x2 + 3x1 x4 +4x2 x3 −5x3 x4 . 21. Write the function f (x) = 6x12 + 3x1 x2 + 2x22 in matrix form.

Chapter 3

Basics of R

3.1 Introduction The R language was developed as a statistical programming language for analyzing, modeling, and visualizing the data. The R is very popular among the data scientists to handle the large volume of data. It is a well-developed, simple and effective programming language which supports conditionals, loops, user-defined functions, and input/output functions, etc. The R is an open-source program and is freely available whereas the RStudio is an integrated development environment (IDE) which is treated as a front-end for giving us a Console, a Scripting Window, a Graphics Window, and an R Workspace, etc. In this book, we write our programs in the R Scripts and see the output on the R Console, and display the graphs in the Graphics Window provided by RStudio. We have assumed that readers are using Windows operating systems. Therefore, we follow the following steps to install R and RStudio in the computer: 1. Download R from http://cran.us.r-project.org. 2. Click on Download R for Windows. Click on base. Click on Download R 3.3.2 for Windows (or a new version that appears). 3. Install R software. Leave all default settings in the installation options. 4. Download RStudio Desktop for windows from http://rstudio.org/download/ desktop (it should be called something like RStudio 1.0.136–Windows 7/8/10). Choose default installation options. The R is a case sensitive programming language. For example, ’A’ and ’a’ are different variables. Note that the keyword version provides detailed information about the running version of R software. The R is an interpreted programming language in which statements of the R program are executed line by line similar to other interpreted languages such as Python or Ruby rather than compiling the whole source code as done in C++, Java, etc. These are the following steps to open and run R program:

© Springer Nature Singapore Pte Ltd. 2019 S. K. Mishra and B. Ram, Introduction to Unconstrained Optimization with R, https://doi.org/10.1007/978-981-15-0894-3_3

35

36

3 Basics of R

1. Double Click on RStudio Desktop Icon or Open Program from Start menu of Window 2. Click on File → Open Script 3. Select the Program you want to run, it will appear in an R Script 4. Right Click Select All (or Press Ctrl-A) 5. Right Click Run Line or Selection (or Press Ctrl-Enter) 6. Output will appear on the R Console Window To know the version of R, one can simply write the following script on the R Console: > print(version) platform arch os system status major minor year month day svn rev version.string nickname

i386-w64-mingw32 i386 mingw32 i386, mingw32 3 4.3 2017 11 30 73796 language R version 3.4.3 (2017-11-30) Kite-Eating Tree

R

3.2 Basics of R Programming R Console has a command line interface, which accepts simple commands to display the output. The symbol > used in the R is called a prompt which indicates that R is ready for another command. R provides arithmetic operators such as addition (+), subtraction (-), multiplication (*), division (/), and exponentiation (ˆ) to perform arithmetic operations. If we type any arithmetic expression and press the enter key of the keyboard, then the R evaluates the expression and shows the output. > 3+7 [1] 10 > x x + 3 [1] 5

3.2 Basics of R Programming

37

We do not need to terminate the expressions or lines by a semicolon. In R, everything works as an object, which is nothing but an entity. The expression x 1/0 [1] Inf > 0/0 [1] NaN > Inf/NaN [1] NaN > log(Inf) [1] Inf > NA+Inf [1] NA

Comments Comments are helpful to explain the operations of R scripts. The comments given in an R program are ignored by the interpreter while executing the program. They do not affect the performance of codes developed by programmers. The pound sign # is used to display sentences as comments in the R script. For example, # R programs are developed in book by Prof. S.K. Mishra > 1 + 1 # Addition of two numeric values [1] 2

38

3 Basics of R

Installation of Package Sometimes we need additional functions beyond those offered by the R library. In order to install an extension package, we should invoke the install.packages function on the R Console and follow the instructions. For example, install.packages("ggplot2")

This package offers a powerful graphics language for creating elegant and complex plots.

Functions In practice, most of the interactions we have with R is through functions. A function is a group of related statements to perform a specific task. The main objective of creating function is to reduce the lines of codes and make the program as simple as possible. There are two types of function: predefined functions, provided by the packages, and user defined functions, containing application logic. The syntax for calling a function in R is similar to most other programming languages. A builtin function is a predefined functions, which are directly executable by passing the appropriate values. Programmers can not modify it. We present built-in function sum() to add numbers. > sum(1:10) [1] 55

Data Structures in R Data structure can be defined as the specific form of organizing and storing the data. These data structures are also called objects. R supports the following basic types of data structure: • • • •

vector matrix list data frame

Vector Vector is a basic data structure which contains similar types of element. These elements can be logical, integer, double, character, etc. A function c() is used to create vectors. Note that function c() concatenates the elements to create a vector. > x x [1] 2 -1 0 4

3.2 Basics of R Programming

If we want to add

1 2

39

to each element of vector x, then we write as:

> x+1/2 [1]

1.5 -1.5

4.5

0.5

Sometimes, we need to extract some elements from the given vector. We can do it in the following way: > x > x [1] > y > y [1]

length(x) [1] 5

We can use typeof() to check the type of data stored in the vector. > typeof(x) [1] "double"

The function seq() generates any arithmetic progression as below: > seq(from=2, to=5, by=0.3) [1] 2.0 2.3 2.6 2.9 3.2 3.5 3.8 4.1 4.4 4.7 5.0

Example 3.1 Write R script to convert the Celsius temperatures (C=[75.3, 77.3, 71.8]) to Fahrenheit temperatures (F).

> C=c(75.3, 77.3, 71.8) > F=(9/5)*C+32 > F [1] 167.54 171.14 161.24

40

3 Basics of R

Matrix Matrix is a data structure which contains similar type of elements in a tabular format. We can perform arithmetic operations on some elements of the matrix or the whole matrix. A matrix is created using function matrix() as follows: > + + + + + >

mat A = c(2, 5, -2, 1) > A [1] 2 5 -2 1 > A = matrix(A, nrow = 2, byrow = FALSE) > A [,1] [,2] [1,] 2 -2 [2,] 5 1 > B = c(1, 5, 3, -2) > B [1] 1 5 3 -2 > B = matrix(B, nrow = 2, byrow = FALSE) > B [,1] [,2] [1,] 1 3 [2,] 5 -2 > C = A%*%B > C [,1] [,2] [1,] -8 10 [2,] 10 13

3.2 Basics of R Programming

41

We can find the transpose of matrix using function t() as follows: > A = c(2, 5, -2, 1) > A = matrix(A, nrow = 2, byrow = FALSE) > A [,1] [,2] [1,] 2 -2 [2,] 5 1 > T = t(A) > T [,1] [,2] [1,] 2 5 [2,] -2 1

Lists This data structure includes data of different types. It is similar to a vector but a vector contains data of similar type. But, the list contains data of heterogeneous type. A list is created using list() as: > a=list(’a’, 1, 2, ’Mathematics’)

We can combine two lists as: > l1 l2 l3=c(l1, l2) > l3 l1$num [1] 1 2 3 4 5 l1$char [1] "a" "b" "c" l2$language [1] "R" l2$day [1] "Mon"

"Python" "MATLAB"

"Tue"

"Wed"

"Thus"

Data Frame A frame used for storing data tables is called Data Frame. It is a list of vectors of equal length. For example, the following variable df is a data frame containing three vectors u, p, t.

42

3 Basics of R > > > >

u ncol(df) [1] 3

These are the following built-in numerical functions given in Table 3.1.

Table 3.1 Numerical functions Function abs(x) sqrt(x) ceiling(x) floor(x) trunc(x) round(x, digits=n) cos(x), sin(x), tan(x) log(x) log10(x) exp(x)

Description returns absolute value of x returns square root of x ceiling(5.475) is 5 floor(5.475) is 5 trunc(5.99) is 5 round(3.475, digits=2) is 3.48 cos(0) is 1 natural logarithm returns value logx to base 10 returns value of e x

3.2 Basics of R Programming

43

Solve Command We can use predefined function solve() to find the inverse of matrix as follows: > A = c(2, 5, -2, 1) > A = matrix(A, nrow = 2, byrow = FALSE) > inv=solve(A) > inv [,1] [,2] [1,] 0.08333333 0.1666667 [2,] -0.41666667 0.1666667

Cat Command The function cat() is used to take the list of variables, convert them to a text form and concatenate the output. > > > >

A="Prof." B="S." C="K." D="Mishra"

> cat(A, B, C, D) Prof. S. K. Mishra

Print Command The function print() is used to display the contents of a single variable as follows: > A="Professor" > print(A) [1] "Professor"

Format Command The function format() converts the R object to a string. It allows to increase or decrease the size of string as follows: > a=exp(1) > a [1] 2.718282 > b=format(a,digits=2) > b [1] "2.7" > b=format(a,digits=10) > b [1] "2.718281828"

44

3 Basics of R

In some situation, we need to write our own function because we have to accomplish a particular task and no built-in function exists. We can create user-defined functions R in accordance to the requirement of the problem and can be used like the built-in functions. The syntax of user defined function is given as follows: func_name func_var(x) [1] -37

To quit our R session, we type > q()

Logical Operators Logical operators work on scalar and vector inputs and return either TRUE or FALSE value. These operators are useful when the user needs to check more than one con-

3.2 Basics of R Programming

45

dition at a time. Logical operators are generally used in decision-making statements. Various logical operators such as & (AND), || (OR), and ! (NOT) are used in R. > 1 & 2 [1] TRUE > 1 & 0 [1] FALSE > !0 [1] TRUE > !1 [1] FALSE > a = c(1,2,3) > b = c(0,1,7) > print(a||b) [1] TRUE > a && b [1] FALSE

3.3 Decision-Making and Loop Statements Decision-making statements require to specify one or more conditions to be evaluated by the program, if the conditions are determined to be true, then statements are executed. On the other hand, there may be a situation when we need to execute a block of code several times. At that time we use loop statements that allow us to execute a statement or group of statements multiple times. We first study the following decision-making statement: If…Else Statement The if…else statement checks a condition, and if the condition is evaluated to be a True, then if part of statement(s) is executed , but if the condition is evaluated to be a False , then else part of statement(s) is executed. Note that if and else parts are mutually exclusive. The syntax of if…else statement is: i f ( test_expression ) { statement ( s ) } else { statement ( s ) }

46

3 Basics of R

Example 3.4 Write R function to check greater number, out of the two numbers. 1 2 3 4 5 6 7

Greaterb) cat (a , " is bigger . ") else cat (b, " is bigger . ") }

Output > Greater(1,2) 2 is bigger.

Loop Statements In order to complete a task or perform any mathematical calculation, the user needs to execute a single statement or group of statements multiple times. To achieve this, every programming language uses the loop concept. A loop statement reduces the lines of code of the program. R also has the following types of the loop: • for • while For Loop for() loop is used to repeat a specific block of code. A for() loop is used to iterate over a vector in R programming. Syntax of for() loop is given as follows: for ( val in sequence) { statement ( s ) }

Here, sequence is a vector and val takes on each of its value during the loop. In each iteration, statement is evaluated. Example 3.5 Write R function to display even number and total even numbers from a given vector. 1 even x=c(10, 15, 20, 25, 30) > y=c(1.1, 2.7, 3.9, 4.1, 5.5) > plot(x, y, type = ’l’)

See Fig. 3.3. This is an another example of graph. x 0 such that f (x ∗ ) < f (x), for all x ∈ Ω\{x ∗ } and x − x ∗ < ε.

(4.3)

Definition 4.5 (Global Minimizer) Let f : Ω ⊆ Rn → R be a real-valued function. Then, a point x ∗ ∈ Ω is a global minimizer of f over Ω, if f (x ∗ ) ≤ f (x), for all x ∈ Ω\{x ∗ }.

(4.4)

Definition 4.6 (Strictly Global Minimizer) Let f : Ω ⊆ Rn → R be a real-valued function. Then, a point x ∗ ∈ Ω is a global minimizer of f over Ω, if f (x ∗ ) < f (x), for all x ∈ Ω\{x ∗ }.

(4.5)

Figure 4.1 shows various types of minimizers. Example 4.1 The function f (x) = (x − 3)2 has a strict local minimizer at x = 3. See Fig. 4.2.

4.1 Introduction

59

Fig. 4.2 f (x) = (x − 3)2

4

f (x)

3 2 1 x 1

2

3

4

Definition 4.7 (Gradient Vector) Let a function f : Ω → R be defined on a set Ω ⊆ Rn , then first-order derivative of f (x) is given as D f (x) =



∂f (x) ∂∂xf2 (x) ∂ x1

...

∂f (x) ∂ xn



,

where ∂∂xfi is the partial derivative of f with respect to xi . The transpose of D f (x) is called the gradient of f (x) given as follows: ⎡ ∂f

(x)

⎤

∂x ⎢ ∂ f1 ⎥ ⎢ ∂ x2 (x)⎥ ⎢ ⎥

∇ f (x) = (D f (x))T = ⎢ ⎣

.. ⎥ . . ⎦ ∂f (x) ∂ xn

(4.6)

⎡ ⎤ 1 Example 4.2 Find the gradient of the function f (x) = 2x12 x2 + x33 at x = ⎣1⎦ . 2 Th gradient of the function f (x) is ⎡

⎤ 4x1 x2 ∇ f (x) = ⎣ 2x12 ⎦ . 3x32 ⎡ ⎤ ⎡ ⎤ 1 4 The gradient of f (x) at x = ⎣1⎦ is ∇ f (x) = ⎣ 2 ⎦ . 2 12

60

4 First-Order and Second-Order Necessary Conditions

The R function Grad_Vec(x) for finding the gradient of the objective function at a point x is given below in Function 4.1 R Function 4.1 Grad_Vec.R 1 # Description : Gradient of function 2 # Theory : See Definition 4.7 3 # Input : x : Starting Point 4 # Output : Grad_Vec : Gradient value of function 5 # Execute on R Console: Grad_Vec(x) 6 Grad_Vec 0 such that x + αd ∈ Ω for all α ∈ [0, α0 ], then we can have A(x + αd) = b, that is, Ax + α Ad = b.

(4.8)

64

4 First-Order and Second-Order Necessary Conditions

Since Ax = b.

(4.9)

From (4.8) and (4.9), we get α Ad = 0, ∀ α, that is, Ad = 0. Conversely, suppose that Ad = 0. We have to show that d is a feasible direction. For that we must show that for all α ∈ [0, α0 ] for some α0 > 0, we have x + αd ∈ Ω, that is, A(x + αd) = b, which can be simplified as Ax + α Ad = b. Since Ad = 0, we get

Ax = b.

This yields x + αd ∈ Ω. Hence, d is a feasible direction. 3 Example 4.7⎡ Find ⎤ the condition under which a direction d ∈ R is a feasible direc2

tion at x = ⎣3⎦ in the domain of the function f (x1 , x2 , x3 ) = 7 − x12 − x22 − x32 . 6

The domain of the given function is X = {(x1 , x2 , x3 ) ∈ R3 |7 − x12 − x22 − x32 ≥ 0}. / X . Thus, there does not exist any feaBut 7 − (2)2 − (3)2 −⎡(6)⎤2 < 0. That is, x ∈ 2 sible direction at x = ⎣3⎦ . 6

4.2 Directional Derivative Let f : Rn → R be a real-valued function and d ∈ Rn be a feasible direction at ∂f at x is x ∈ Ω. Then, the directional derivative of f in the direction d, denoted as ∂d the real-valued function given as follows:

4.2 Directional Derivative

65

∂f f (x + αd) − f (x) (x) = lim+ . α→0 ∂d α

(4.10)

∂f If d= 1, then ∂d is the rate of increase of f at x in the direction d. If x and d are given, then f (x + αd) is a function of α only. Using chain rule of differentiation, we can express as

d ∂f (x) = f (x + αd)|α=0 ∂d dα

⎡ ⎤ d1  ⎢d2 ⎥ ⎥ ⎢ = [∇ f (x)]T d = ∂∂xf1 (x) ∂∂xf2 (x) . . . ∂∂xfn (x) ⎢ . ⎥ . ⎣.⎦ dn = ∇ f (x), d = d ∇ f (x). T

Proposition 4.1 Let f : Rn → R be a differentiable function at x ∈ Rn and d ∈ Rn be a unit vector, then ∂f f (x + αd) − f (x) (x) = lim+ = d T ∇ f (x). α→0 ∂d α

(4.11)

Proof Applying Taylor’s Theorem on f (x + αd), 

n

∂ f (x) f (x) + α di + o(αd) − f (x) ∂ xi j=1 

n ∂ f (x) 1 α di + o(αd) = α ∂ xi j=1

1 f (x + αd) − f (x) = α α

=

n j=1

di

∂ f (x) o(αd) . + ∂ xi α

Taking the limit of both sides as α → 0+ , we have ∂ f (x) f (x + αd) − f (x) o(αd) = lim+ . di + lim+ α→0 α→0 α ∂ xi α j=1 n

lim+

α→0

Since lim+ α→0

o(αd) α

= 0, thus f (x + αd) − f (x) ∂ f (x) = di = d T ∇ f (x). α ∂ x i j=1 n

lim+

α→0

66

4 First-Order and Second-Order Necessary Conditions

That is,

∂f (x) = d T ∇ f (x). ∂d 

This completes the proof.

Remark 4.1 If d is a unit vector, that is, d = 1, then ∇ f (x), d is the rate of increase of f at the point x in the direction d. Example 4.8 Let f : R3 → R⎡be f ⎤ (x) = x1 x2 x3 . Compute the directional derivative 1/2 of f (x) in the direction d = ⎣ 1/2 ⎦. √ 1/ 2 The directional derivative of f (x) in the direction d is  T ∂f (x) = ∇ f (x) d ∂d =



⎡

1 ⎤ 2 x2 x3 x1 x3 x1 x2 ⎣ 21 ⎦ √1 2



x1 x3 x1 x2 x2 x3 + + √ . 2 2 2

This is rate of increase of f at x in direction d as d= 41 + =

1 4

+

1 2

= 1.

⎡ ⎤ 1 Example 4.9 Find the gradient of f (x) = x12 + cos(x1 x2 ) + x3 at x = ⎣0⎦, and 1 ⎡ √⎤ 1/ 3 √ directional derivative in the direction d = ⎣1/ 3⎦ using R. √ 1/ 3 The R function func_var() is used to write the above objective function as func_var G = Grad_Vec(x = c(1,0,1)) > G [1] -16 0 4 > G = matrix(G,nrow = 1, ncol = 3) > G [1,]

[,1] [,2] [,3] -16 0 4

We perform the⎡following operations to find the directional derivative of f (x) in the √ ⎤ 1/ 3 √ direction d = ⎣1/ 3⎦ as follows: √ 1/ 3 > d = c(1/sqrt(3), 1/sqrt(3), 1/sqrt(3)) > d [1] 0.5773503 0.5773503 0.5773503 > d = matrix(d,nrow = 3, ncol = 1) > d [,1] [1,] 0.5773503 [2,] 0.5773503 [3,] 0.5773503 > Dir_Deriv = G%*%d > Dir_Deriv [,1] [1,] -6.928201

4.3 First-Order Necessary Condition We are prepared to present the first-order necessary condition for local minimizer of a function. Theorem 4.1 (First-order necessary condition for a minimum (FONC)) Let Ω be a subset of Rn and f ∈ C 1 be a real-valued function on Ω. If x ∗ is a local minimizer of f over Ω, then for any feasible direction d at x ∗ , we have

68

4 First-Order and Second-Order Necessary Conditions

d T ∇ f (x ∗ ) ≥ 0. Proof If d is a feasible direction at x ∗ ∈ Ω then from Definition 4.9, there exists α0 > 0 such that x(α) = x ∗ + αd ∈ Ω for all α ∈ [0, α0 ].

(4.12)

x(0) = x ∗ .

(4.13)

φ(α) = f (x(α)).

(4.14)

Note that

We define the composite function

Applying Taylor’s Theorem to φ(α) and at α = 0, we get φ(α) = φ(0) + αφ  (0) + O(α), that is, φ(α) − φ(0) = αφ  (0) + O(α)

(4.15)

Using (4.12)–(4.14) in (4.15), we get 

f (x ∗ + αd) − f (x ∗ ) = αφ (0) + O(α).

φ  (α) =

d d f (x(α)) = f (x ∗ + αd) dα dα = d T ∇ f (x ∗ + αd).

At α = 0, we have x ∗ = x(0), then φ  (0) = d T ∇ f (x(0)). Therefore, f (x ∗ + αd) − f (x ∗ ) = αd T ∇ f (x ∗ ) + O(α), where α ≥ 0.

(4.16)

Since x ∗ is a local minimizer of f over Ω, therefore f (x ∗ ) ≤ f (x ∗ + αd) for sufficiently small α > 0. From (4.16) and (4.17), we get αd T ∇ f (x(0)) = f (x ∗ + αd) − f (x ∗ ) ≥ 0,

(4.17)

4.3 First-Order Necessary Condition

69

therefore, αd T ∇ f (x(0)) ≥ 0, α ∈ [0, α0 ] then d T ∇ f (x(0)) ≥ 0.

(4.18)

Using (4.13) in (4.18), we get d T ∇ f (x ∗ ) ≥ 0. 

This completes the proof.

Using directional derivatives, an alternative proof of Theorem 4.1 is given as follows: We can express (FONC) as ∂f ∗ (x ) ≥ 0, for all feasible direction d. ∂d We know that x ∗ is a local minimizer of f over Ω. Then, for any feasible direction d, there exists α0 > 0 such that α ∈ (0, α0 ), we have f (x ∗ ) ≤ f (x ∗ + αd), that is,

Therefore,

f (x ∗ + αd) − f (x ∗ ) ≥ 0. f (x ∗ + αd) − f (x ∗ ) ≥ 0, ∀ α ∈ (0, α0 ) α

Taking the limit as α → 0+ on both sides, we get lim+

α→0

that is,

which is equivalent to

This completes the proof.

f (x ∗ + αd) − f (x ∗ ) ≥ 0, α ∂f ∗ (x ) ≥ 0, ∂d d T ∇ f (x ∗ ) ≥ 0. 

Remark 4.2 If x ∗ is an interior point of Ω, then every direction is a feasible direction. See Fig. 4.4.

70

4 First-Order and Second-Order Necessary Conditions

Fig. 4.4 Opposite directions

x∗ d • −d

For such a special case of interest, the above theorem immediately gives us the following corollary. Corollary 4.1 (Interior Case) Let Ω ba a subset of Rn and f ∈ C 1 be a real-valued function on Ω. If x ∗ is a local minimizer of f over Ω and if x ∗ is an interior point of Ω, then ∇ f (x ∗ ) = 0. Proof Since x ∗ is a local minimizer of f over Ω, then for any feasible direction, we have (4.19) d T ∇ f (x ∗ ) ≥ 0. Since x ∗ is an interior point of Ω, then every direction is a feasible direction, therefore − d T ∇ f (x ∗ ) ≥ 0.

(4.20)

From (4.19) and (4.20), we get d T ∇ f (x ∗ ) = 0. Therefore, ∇ f (x ∗ ) = 0 for all d ∈ Rn . This completes the proof.  2 2 2 Example  4.10 Let f : R → R be defined by f (x) = x1 + x2 + 6x2 + 3, where x x = 1 and x1 ≥ 0, x2 ≥ 0. Check the first-order necessary condition (FONC) to x2 be satisfied for the local minimizer at the following points:     1 2x1 . . The gradient of f (x) is ∇ f (x) = 1. xa = 2x2 + 6 3 The gradient of f at xa is   2 ∇ f (xa ) = . 12

4.3 First-Order Necessary Condition

71

    1 is an interior point of Ω = xa : x1 ≥ 0, x2 ≥ 0 . 3   0 Thus, the FONC requires ∇ f (xa ) = . But, the FONC is violated at the point 0 xa for a local minimizer (Figs. 4.5, 4.6).   0 2. xb = . The gradient of f at xb is 4 Note that point xa =

 ∇ f (xa ) =

 0 . 14

  0 Note that xb = is a boundary point of Ω. The FONC requires d T ∇ f (xb ) ≥ 4   d 0. If d = 1 , then d2     0 d1 d2 = 14d2 . 14 For d to be feasible at xb , we need d1 ≥ 0, but d2 can take any value in R. 1 Suppose that d = , then −1     0 d T ∇ f (x b ) = 1 −1 = −14 < 0. 14 Thus, the FONC is violated at the point xb for a local minimizer.   2 3. xc = . The gradient of f at xc is 0   4 ∇ f (xc ) = 6 and xc is a boundary point of Ω. Therefore, FONC requires d T ∇ f (xc ) ≥ 0 and    4  d1 d2 = 4d1 + 6d2 ≥ 0. 6 For d to be  feasible, we need d2 ≥ 0 and d1 can take any value in R. −5 If d = , then 1     4 = −20 + 6 = −14 < 0. d T ∇ f (xc ) = −5 1 6

72

4 First-Order and Second-Order Necessary Conditions

Fig. 4.5 Feasible direction d at point xa

f(x) 4 • xa =

  1 3

2

x 1 Fig. 4.6 Feasible direction d at xb

xb =

2

3

4

1  2 •d= 1 −1

3

  f(x) 0 4• 4 2 x

−1 −2

Thus, the FONC is violated at the point xc for a local minimizer. 0 4. xd = . 0 The gradient of f at xd is   0 ∇ f (xd ) = . 6 Therefore,

   0  d ∇ f (xd ) = d1 d2 = 6d2 . 6 T

For d to be feasible, we need d1 ≥ 0 and d2 ≥ 0. Therefore, xd satisfies the FONC for a local minimizer (Fig. 4.7).

4.4 Second-Order Necessary Condition

73

Fig. 4.7 Feasible direction d at point xc

3

d=

  −5 1

2

•

1

f(x)

xc =

−6

−4

−2

  2 0 x

• 2

4

4.4 Second-Order Necessary Condition We now present a second-order necessary condition for a local minimizer. Theorem 4.2 (Second-order necessary condition for a minimum (SONC)) Let Ω be a non-empty subset of Rn , f : Ω → R be C 2 function on Ω, x ∗ be a local minimizer of f over Ω, and d be a feasible direction at x ∗ . If d T ∇ f (x ∗ ) = 0, then d T F(x ∗ )d ≥ 0, where F is the Hessian of f . Proof Given that d is a feasible direction at x ∗ . From Definition 4.9, we have x(α) = x ∗ + αd for sufficiently small α ≥ 0,

(4.21)

and define composite function φ(α) = f (x ∗ + αd).

(4.22)

φ(0) = f (x ∗ ).

(4.23)

Set α = 0, we get

Differentiating (4.22) w.r.t. α, we get 

φ (α) = d T ∇ f (x ∗ + αd). Set α = 0 in (4.24) to get



φ (0) = d T ∇ f (x ∗ ).

(4.24)

74

4 First-Order and Second-Order Necessary Conditions

Differentiating (4.24) w.r.t. α, we get 

φ (α) = d T F(x ∗ + αd)d.

(4.25)

Set α = 0 in (4.25), then we get 

φ (0) = d T F(x ∗ )d. Using (4.21) in (4.22) to get φ(α) = f (x(α)).

(4.26)

Applying Taylor’s Theorem to φ(α) at α = 0 

φ(α) = φ(0) + αφ (0) +

α 2  φ (0) + o(α 2 ). 2!

(4.27)

Using (4.22) and (4.23) in (4.27), we get 

f (x ∗ + αd) − f (x ∗ ) = αφ (0) +

α 2  φ (0) + O(α 2 ). 2!



Since φ (0) = d T ∇ f (x ∗ ) = 0, then f (x ∗ + αd) − f (x ∗ ) =

α2 T d F(x ∗ )d + O(α 2 ). 2

(4.28)

Since x ∗ is local minimizer of f , therefore f (x ∗ + αd) − f (x ∗ ) ≥ 0.

(4.29)

From (4.28) and (4.29), we get α2 T d F(x ∗ )d ≥ 0. 2 For all α ≥ 0 and feasible direction d = 0 at x ∗ , we obtain d T F(x ∗ )d ≥ 0. This completes the proof.



Remark 4.3 Let Ω be a non-empty subset of Rn , f : Ω → R be C 2 − function on Ω; x ∗ ∈ Ω be a local minimizer of f over Ω, and d be a feasible direction at x ∗ . If d T ∇ f (x ∗ ) = 0, then Hessian of f is positive semidefinite. That is,

4.4 Second-Order Necessary Condition

75

−100

f(x) 0 50 100

Fig. 4.8 Graph of 2x 3 + 1

−4

−2

0 x

2

4

d T F(x ∗ )d ≥ 0 ∀ d ∈ Rn . Example 4.11 Consider a function f : R → R as f (x) = 2x 3 + 1. Check FONC and SONC for minimizer. Note that f  (x) = 6x 2 , and f  (x) = 12x. At x = 0, we have f  (0) = 0 and f (0) = 0. The point x = 0 satisfies both FONC and SONC. However, x = 0 is not a minimizer, which can be seen in Fig. 4.8. 

Example 4.12 Let f : R2 →Rwhere f (x) = −x12 + x22 . Apply FONC and SONC 0 to check that the point x ∗ = is a minimizer . 0   −2x1 . The gradient of f at x ∗ is The gradient of f (x) is ∇ f (x) = 2x2   0 ∇ f (x ) = . 0 ∗

Therefore, FONC is satisfied at x ∗ . The Hessian of f at x ∗ is F(x ∗ ) =

  −2 0 . 0 2

The Hessian is indefinite because for some d1 ∈ R2 we have d1T F(x ∗ )d1 < 0. If   1 d1 = then 0        −2 0 1   1 10 = −2 0 = −2 < 0, 0 2 0 0 and for some d2 ∈ R , we have 2

d2T

  0 F(x )d2 > 0. If d2 = then 1 ∗

       −2 0 0   0 01 = 02 = 2 > 0. 0 2 1 1

76

4 First-Order and Second-Order Necessary Conditions

Fig. 4.9 Graph of −x12 + x22

  0 . It concludes that x ∗ 0 is not a minimizer of f (x) over R2 . The graph of −x12 + x22 is shown in Fig. 4.9. This is indefinite. Therefore, SONC is not satisfied at x ∗ =

Example 4.13 Consider the following problem: minimize x13 − x12 x2 + 2x22 subject to x1 ≥ 0, x2 ≥ 0. Does the point x ∗ =

  0 satisfy the first- and second-order necessary condition for 0

a minimizer?

  2 3x1 − 2x1 x2 . The gradient of f at x ∗ The gradient of f is ∇ f (x) = −x12 + 4x2 ∇ f (x ∗ ) =

  0 . 0

Therefore, FONC is satisfied at x ∗ . The Hessian of f is   6x1 − 2x2 −2x1 F(x) = . −2x1 4 The Hessian of f at x ∗ is F(x ∗ ) =

    00 1 . For the d = , we have 04 0

     00 1 d T F(x ∗ )d = 1 0 = 0. 04 0 Therefore, SONC is satisfied. The graph of this problem is shown in Fig. 4.10.

4.4 Second-Order Necessary Condition

77

Fig. 4.10 Graph of x13 − x12 x2 + 2x22

Fig. 4.11 Graph of 1 3 1 3 3 x 1 + 3 x 2 − 16x 1 − 4x 2

Example 4.14 Find the minimizers and maximizers of the function f (x) = 13 x13 + 1 3 x − 16x1 − 4x2 . 3 2 The graph of f is shown in Fig. 4.11. The gradient of f is 

 x12 − 16 ∇ f (x) = . x22 − 4   x If x = 1 is an interior point, then FONC requires ∇ f (x ∗ ) = 0, that is, x2 ∗



   x12 − 16 0 = . x22 − 4 0

Then, we get x12 − 16 = 0 =⇒ x1 = ±4 and x22 − 4 = 0 =⇒ x2 = ±2.

78

4 First-Order and Second-Order Necessary Conditions

The points are xa = at x ∗ is

        4 −4 4 −4 , xb = , xc = , xd = . The Hessian of f 2 2 −2 −2   2x1 0 ∗ F(x ) = . 0 2x2 |F(x ∗ )| = 4x1 x2

1. 2. 3. 4.

If x ∗ If x ∗ If x ∗ If x ∗

= = = =

xa , then |F(xa )| = 32 > 0. Therefore, the point xa is minimizer. xb , then |F(xb )| = −32 < 0. Therefore, the point xb is maximizer. xc , then |F(xc )| = −32 < 0. Therefore, the point xc is maximizer. xd , then |F(xd )| = 32 > 0. Therefore, the point xd is minimizer.

4.5 Second-Order Sufficient Condition for Interior Case We now present the following theorem, which gives sufficient condition to find the local minimizer for the interior case. Theorem 4.3 (Second-order sufficient conditions (SOSC)) Case of Interior. Let f ∈ C 2 be defined on a region Ω ⊆ Rn in which x ∗ is an interior point. Suppose that 1. ∇ f (x ∗ ) = 0; 2. F(x ∗ ) > 0 (positive definiteness). Then, x ∗ is a strict local minimizer of f . Proof We know that the function f is twice continuously differentiable, then its Hessian at x ∗ is ⎡ ∂2 f ⎤ ∂2 f ∂2 f . . . 2 ∂ xn ∂ x1 ⎢ ∂∂2x1f ∂ x∂22∂fx1 ⎥ ∂2 f ⎥ ⎢ . . . 2 ∂ xn ∂ x2 ⎥ ⎢ ∂ x1 ∂ x2 ∂ x2 ∗ . F(x ) = ⎢ . .. . ⎥ ⎢ . ⎥ . . . . .. ⎦ ⎣ . 2 ∂2 f ∂2 f . . . ∂∂ x 2f . ∂ x1 ∂ xn ∂ x2 ∂ xn n

Since F(x ∗ ) = (F(x ∗ ))T , that is, F is symmetric, and from assumption 2, F(x ∗ ) is positive definite, then from the Rayleigh’s inequality, we have λmin (F(x ∗ ))d2 ≤ d T F(x ∗ )d ≤ λmax (F(x ∗ ))d2 .

(4.30)

Since F(x ∗ ) is positive definite, each eigenvalue is positive, and if d = 0 then d T F(x ∗ )d ≥ λmin (F(x ∗ ))d2 > 0.

(4.31)

4.5 Second-Order Sufficient Condition for Interior Case

79

Fig. 4.12 Graph of 2x12 + x22 + 2

From Taylor’s Theorem, and assumption 1 of this theorem, we have 1 f (x ∗ + αd) − f (x ∗ ) = α 2 d T F(x ∗ )d + o(α2 ). 2

(4.32)

Set α = 1 in (4.32) to get f (x ∗ + d) − f (x ∗ ) =

1 T d F(x ∗ )d + o(1). 2

(4.33)

Using (4.31) in (4.33), then we get f (x ∗ + d) − f (x ∗ ) > 0, that is, f (x ∗ ) < f (x ∗ + d). Therefore, x ∗ is a strict local minimizer of f over Ω. This completes the proof.    0 Example 4.15 Show that the point x ∗ = is a strict local minimizer of the function 0 f (x1 , x2 ) = 2x12 + x22 + 2, where x1 ≥ 0, x2 ≥ 0.  The gradient of f is ∇ f (x) =

 4x1 . 2x2

The gradient of f at x ∗ is ∇ f (x ∗ ) =

  0 . 0

80

4 First-Order and Second-Order Necessary Conditions

  0 is an interior point, therefore FONC is satisfied at x ∗ . The Hessian 0 of f at x ∗ is   40 F(x) = . 02

Since x ∗ =

Since the Hessian F is positive definite, therefore SONC   is also satisfied. Because 0 SOSC is satisfied, therefore we conclude that x ∗ = is a strict local minimizer of 0 2 f over R and graph is shown in Fig. 4.12. Example 4.16 Evaluate α ∈ R for which the function f (x) = −2α 2 x12 + x1 x22 + 4x14  will have a strict local minimizer at x ∗ =

α/2

0

 .

The gradient and Hessian of f are: ∇ f (x) =

  −4α 2 x1 + x22 + 16x13 , 2x1 x2 

and F(x) =

 −4α 2 + 48x12 2x2 . 2x2 2x1

The Hessian of f at x ∗ is 

  2  −4α 2 + 12α 2 0 8α 0 F(x ) = = . 0 α 0 α ∗

A sufficient condition for x ∗ to be strict local minimizer is ∇ f (x ∗ ) = 0, and detF(x ∗ ) > 0, which is possible if 8α 3 > 0, that is, α > 0. Example 4.17 Rao, Singiresu (1996) A direct current generator developing an open circuit voltage of 240 Volt has an internal resistance of 10 ohm. Find load resistance for which the power delivered by generator will be maximum. Find the maximum power delivered as well. We need the following notations to derive the formula for the electric circuit given in Fig. 4.13. V = An open circuit voltage, Rl = Load resistance, P = Power delivered by circuit,

4.5 Second-Order Sufficient Condition for Interior Case

81

Fig. 4.13 Illustration Example 4.17

− Ri Rl

V

− G + I

Ri = Internal resistance, I = Current in the circuit. The current in the given electric circuit is I =

V . Rl + Ri

(4.34)

The power dissipated by the internal resistance Ri is P = I 2 Ri .

(4.35)

Using (4.34) in (4.35), we get P=

V 2 Ri . (Rl + Ri )2

(4.36)

The value of Rl that dissipates the maximum power is obtained by differentiating P with respect to Ri , and equating to zero. (Rl + Ri )2 V 2 − 2V 2 Ri (Rl + Ri ) dP = = 0, d Ri (Rl + Ri )4 that is, V 2 (Rl + Ri )(Rl − Ri ) = 0. (Rl + Ri )4 Therefore, V 2 (Rl − Ri ) = 0,

82

4 First-Order and Second-Order Necessary Conditions

which leads to Rl = Ri . d2 P d 2  (Ri −Rl )V 2 [(Ri + Rl )3 (−V 2 ) − 3(Ri + Rl )2 (Rl − Ri )V 2 ] = = . 3 (Ri + Rl )6 d Ri2 d Ri2 (Rl +Ri ) Since

d2 P d R2

is negative at Rl = Ri . Thus P is maximum, and maximum power Pmax is Pmax =

V2 . 4Rl

We know that V = 240 Volt, and Rl = Ri = 10 ohm. Thus, the load resistance is 2 = 1440 watt. Rl = 10 ohm, the maximum power is obtained as Pmax = (240) 4×10

4.6 Exercises 2 2 1.  Find  the gradient and Hessian of the function f (x) = x1 x2 − 4x1 + 3x2 at x = 1 using R. 4 2. Find ⎡ the gradient and Hessian of the function f (x) = 3x12 x3 + 2x3 − 4x3 x22 at ⎤ 3 x = ⎣ 1 ⎦ using R. −1 2 2 2 3. Find the directional ⎡ ⎤derivative of the function ⎡ ⎤f (x) = x1 x2 + x2 x3 − x1 x2 x3 at 1 1 the point x = ⎣ 1 ⎦ in the direction d = ⎣2⎦ . −1 3 4. Consider the function f : R2 → R given below

 f (x) = x T

   12 3 x + xT + 7. 43 5

  1 a. Find the gradient and Hessian of f at the point x = . 1   1 b. Find the directional derivative of f at the point x = . 1

4.6 Exercises

83

5. Compute the gradient and Hessian of  the  Rosenbrock function f (x) = 100(x2 − 1 x1 )2 + (1 − x1 )2 . Show that x ∗ = is the only local minimizer of this func1 tion, and the Hessian at that point is positive definite. a. Find the feasible directions. b. Check if the second-order necessary condition is satisfied.       0 4 4 , xb = , xc = are possible maximizers of the problem 6. xa = 3 0 3 minimize x12 + x22 − 6x1 − 12x2 + 26 subject to

x1 ≥ 0, ≥ 0.

a. Find the feasible directions. b. Check if the second-order necessary condition is satisfied. 7. Check first-order and second-order necessary conditions for the  function f (x) = 1 −(x1 − 1)2 − (x2 − 2)2 to be local minimizer at point x = . 2 5 8. Determine the maximum and minimum values of f (x) = 4x − 25x 4 + 40x 3 + 10.   2 9. Is point xa = is a minimizer of the function f (x) = 41 [x12 + 4x22 − 4(3x1 + 4 8x2 ) + 100]?  4 10. Point xa = is a possible minimizer of the problem −1 15 x2 subject to x1 , x2 ≥ 0. minimize x1 −

a. Find the feasible direction. b. Check if the second-order necessary condition is satisfied. 11. Find the extreme points of thefunction f  (x) = x13+x23 + 2x12 + 4x22 + 6. 1 0 1 12. One of the points xa = , xb = , xc = minimizes the function −1 0 1 f (x) = 100(x2 − x12 ) + (1 − x1 )2 . By using appropriate tests, identify the minimizer. 13. Check the existence of extrema for the function f (x) = x12 + x1 x2 + 2x  2 + 4. −1/8 2 2 and the 14. Given that f (x1 , x2 ) = 4x1 + 2x2 − 4x1 x2 + x1 , the point xa = 0   −1/5 direction d = 2 . /5 a. Is d a descent direction?

84

4 First-Order and Second-Order Necessary Conditions

b. Denoting f (α) = f (xa + αd). Find f  (1). 15. Determine the minima/maxima of the following quadratic function f (x) = 2x12 + 3x32 − 5x1 x2 . 16. Determine the minimizers/maximizers of the following function f (x) = x12 − 4x1 x22 + x24 − x25 . 17. Determine the minimizers of the function f (x) = x12 + 2x22 + 5x32 − 3x1 x2 − 4x2 x3 − 3x3 . 18. Distinguish between the local and global extrema of the following objective function: f (x) = 2x13 + x22 + x12 x22 + 4x1 x2 + 3. 19. Find a point at which function f (x) = 8x1 + 12x2 + 2x12 − 3x22 is minimum. 20. Find the minimum of the function f (x) = 10x 6 − 48x 5 + 15x 4 + 200x 3 − 120x 2 − 480x + 100.

Chapter 5

One-Dimensional Optimization Methods

5.1 Introduction In this chapter, we discuss the minimization problems in the following form minimize f (x),

x ∈ R,

where function f : R → R is a nonlinear function of one variable. The objective is to find the value of x for which f (x) is minimum. We will consider minimization problems with the understanding that minimizing f (x) is same as maximizing − f (x). One-dimensional optimization methods are simple and easy to understand. A clear understanding of these methods will be helpful to learn the complex algorithms for solving multidimensional unconstrained optimization problems. A function f : [a, b] → R is a unimodal function if there exists x ∗ ∈ [a, b] such that f is decreasing or nonincreasing for x ∈ [a, x ∗ ] and increasing or nondecreasing for x ∈ [x ∗ , b]. At outset, the function f is referred to as the objective function which is continuous and unimodal and the function has only one minimizer on the given interval [a, b] where a is a lower limit and b is an upper limit. This interval is called an initial level of uncertainty. In this chapter, we study the following methods to find the minimizer of unimodal functions: • • • • • • •

Interval halving search method, Fibonacci search method, Golden Section search method, Quadratic interpolation method, Bisection method, Newton–Raphson method, Secant method.

Note that the first three methods work with the comparison of function values, while quadratic interpolation method interpolates the objective function f on [a, b] by a © Springer Nature Singapore Pte Ltd. 2019 S. K. Mishra and B. Ram, Introduction to Unconstrained Optimization with R, https://doi.org/10.1007/978-981-15-0894-3_5

85

86

5 One-Dimensional Optimization Methods

Fig. 5.1 A unimodal function with function values at two distinct point

f (x)

• a

• x1

• x2

• b

x

polynomial that has the same function value or derivative value at a number of points on [a, b], and the rest three methods require derivative information. Suppose a unimodal function f (x) is defined on the interval [a, b] as depicted in Fig. 5.1 and we want to find a point x ∗ at which the given function is minimized, then we consider another two points x1 and x2 which lies in the interval [a, b] such that a < x1 < x2 < b, and hold the following properties to reduce the interval of uncertainty: 1. If f (x1 ) > f (x2 ) then the minimum x ∗ does not lie in [a, x1 ]. 2. If f (x1 ) < f (x2 ) then the minimum x ∗ lies either in the interval [a, x1 ] or in [x1 , x2 ], but does not lie in [x2 , b]. 3. If f (x1 ) = f (x2 ) then the minimum x ∗ lies in [x1 , x2 ] but does not lie in [a, x1 ] and [x2 , b].

5.2 Interval Halving Search Method The interval-halving method deletes the exactly one-half of the current interval at every stage. We find function value at three different points. These three points divide the search space into four regions with an equal region (See Fig. 5.2). We have chosen 2 . All three points x1 , x2 , and α in the interval [a, b] such that x1 < x2 and α = x1 +x 2 points are equidistant from each other and also equidistant from the boundaries a and b by the same amount If f (x1 ) < f (α), then the minimum point x ∗ can not lie beyond α. Therefore, we reduce the interval from [a, b] to [a, α]. The point α being the middle of the search space, this elimination reduces the search space to 50% of the original space. On the other hand, if f (x1 ) > f (α), then the minimum can not lie in [a, x1 ]. The point x1 being one-fourth point in the search space, therefore, the reduction is only 25%. We compare function values of α and x2 to eliminate further 25% of the search space. This process continues until a small enough interval

5.2 Interval Halving Search Method

87

Fig. 5.2 Three points x1 , α, and x2 used in the interval halving method

a

x1

x2

b

is found. We conclude that exactly one-half of the search space is retained at each iteration of the algorithm, therefore, this method is called the interval halving method. We present the Interval Halving Search algorithm. Algorithm 5.1 (Interval Halving Search Algorithm) 1: Choose a, b ∈ R, ε  a and b are lower and upper bound, ε is a tolerance value 2: Compute α ← a+b  mid point of a and b 2 3: k ← 0  k is number of iterations of method 4: for k to maxiter do  maxiter is maximum number of iterations to perform 5: Set L ← b − a  length of level of uncertainty [a, b] 6: Set x1 ← a + L/4, and x2 ← b − L/4. 7: Compute f 1 ← f (x1 ), f 2 ← f (x2 ), f α ← f (α) 8: if f 1 < f α then  compare f 1 with f α 9: Set b ← α, α ← x1 , and f α ← f 1 10: else 11: if f 2 < f α then  compare f 2 with f α 12: Set a ← α, α ← x2 , and f α ← f 2 . 13: else 14: Set a ← x1 and b ← x2 15: end if 16: end if 17: if |L| < ε then  stopping criterion 18: Converged. Print x ∗ ← α and f (x ∗ ) ← f (α)  best solution 19: end if 20: end for

At every iteration, the unimodal function is evaluated on two new points, and the interval reduces to half of that at the previous iteration. Therefore, the interval reduces to about  n/2 1 L0 2

88

5 One-Dimensional Optimization Methods

after n function evaluations. Here, L 0 is the length of the interval [a, b]. The function evaluations required to achieve the desired accuracy ε can be computed by solving the following:  n/2 1 (b − a) = ε. 2

(5.1)

Example 5.1 Apply the interval halving search method to find the minimum of f (x) = x(x − 1) over the interval [0, 3]. Given that a = 0, b = 3, and choose ε = 10−3 . The point α is the mid point of the = 0+3 = 1.5, and the function value at α is f (α) = search interval. Thus α = a+b 2 2 f (1.5) = 0.75. The initial interval length is L 0 = L = b − a = 3 − 0 = 3. We now set L0 =0+ 4 L0 =3− x2 = b − 4 x1 = a +

3 = 0.75, 4 3 = 3 − 0.75 = 2.25, 4

f (x1 ) = −0.1875, f (x2 ) = 2.8125.

We compare the function values and observe that f (x1 ) < f (α), therefore, we set following points as b = α = 1.5,

α = x1 = 0.75.

The function value at α = 0.75 is f (α) = −0.1875. We get the length of new interval as L = b − a = 1.5 − 0 = 1.5, which is exactly half of that in the original interval (L 0 = 3). Since L ≮ ε, therefore we update the values of x1 and x2 as 1.5 L =0+ = 0.375, 4 4 1.5 L = 1.125, x2 = b − = 1.5 − 4 4 x1 = a +

f (x1 ) = −0.234, f (x2 ) = 0.1406.

We compare the function values and observe that f (x1 ) < f (α), therefore, we set the following points as b = α = 0.75

α = x1 = 0.375

The function value at α = 0.375 is f (α) = −0.234. We get the length of new interval as L = b − a = 0.75 − 0 = 0.75 Since L ≮ ε, therefore, we again compute the new values of x1 and x2 as

5.2 Interval Halving Search Method

89

0.75 L =0+ = 0.1875, 4 4 L 0.75 x2 = b − = 0.75 − = 0.5625, 4 4 x1 = a +

f (x1 ) = −0.152, f (x2 ) = −0.2460.

We compare the function values and observe that f (x1 ) ≮ f (α) but f (x2 ) < f (α). We set the following updated points as a = α = 0.375

α = x2 = 0.5625

We continue this process until an L smaller than a specified value(ε) is obtained. We shall get the minimum point x ∗ = 0.499 and the minimum value of the unimodal function f (x) = −0.25 in 11 iterations. The R function Interval_Halving_Search(a, b, epsilon, maxiter) for the interval halving search algorithm is given below in Function 5.1. R Function 5.1 Interval_Halving_Method.R 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

23 24 25 26 27 28 29

# # # # # # # # #

Description : Interval Halving Search Method Theory : See S e c t i o n 5.2 and A l g o r i t h m 5.1 Input : a : Lower bound b : Upper bound epsilon : Tolerance value maxiter : M a x i m u m n u m b e r of i t e r a t i o n s O u t p u t : xm : Minimum point fxm : Minimum function value E x e c u t e on R C o n s o l e : I n t e r v a l _ H a l v i n g _ S e a r c h ( a , b , 1 e -7 , 100) I n t e r v a l _ H a l v i n g _ S e a r c h L20 then  comparison is made to ensure x1 lies to the left of x2 12: Set x1 ← b − L ∗j , x2 ← a + L ∗j . 13: else 14: Set x1 ← a + L ∗j , x2 ← b − L ∗j 15: end if 16: Compute f 1 ← f (x1 ), and f 2 ← f (x2 ) 17: if f 1 < f 2 then  compare f 1 with f 2 18: Set b ← x2 F j 19: Compute L ∗j ← Fn−(n−j−2) × L0  new interval 20: else 21: if f 2 < f 1 then  compare f 2 with f 1 22: Set a ← x1 . F j 23: L ∗j ← Fn−(n−j−2) × L0.  new interval 24: else 25: Set a ← x1 and b ← x2 . F j 26: Compute L ∗j ← Fn−(n−j−2) × (b − a)  new interval 27: end if 28: end if 29: end for 30: if f 1 ≤ f 2 then  stopping criterion 31: Print x ∗ ← x1 , f (x ∗ ) ← f (x1 )  best solution 32: else 33: Print x ∗ ← x2 , f (x ∗ ) ← f (x2 )  best solution 34: end if

Example 5.3 Apply the Fibonacci search method to minimize f (x) = x 2 + 2x on the interval [−3 , 4]. Obtain the optimal value within 5% exact value. We have Length of final interval of uncertainty 5 ≤ , 2 × Length of initial interval of uncertainty 100 that is,

Ln 1 ≤ L 0, 2 20

96

5 One-Dimensional Optimization Methods

which implies Ln ≤

L0 . 10

We know that Ln 1 1 = ≤ , L0 Fn 10 that is, Fn ≥ 10. We have the following sequence of Fibonacci numbers: F0 = 1,

F1 = 1,

F2 = 2,

F3 = 3,

F4 = 5,

F5 = 8,

F6 = 13.

Thus, n ≥ 6 will satisfy the above inequality. Therefore, six number of experiments are to be performed to obtain the desire percentage of accuracy. We get L 0 = 4 − (−3) = 7, and n = 6. To achieve the values of x1 and x2 , we first find L ∗2 =

Fn−2 F4 5 × 7 = 2.6923. L0 = L0 = Fn F6 13

We obtain x1 = a + L ∗2 = −3 + 2.6923 = −0.3077, x2 = b − L ∗2 = 4 − 2.6923 = 1.3077 with f (x1 ) = −0.5207 and f (x2 ) = 4.32. Since f (x1 ) < f (x2 ), therefore we discard the interval (x2 , 4]. The new level of uncertainty is L 2 = [−3, x2 ] = [−3, 1.3077] which is shown in Fig. 5.5. We find L 2 = L 0 − L ∗2 = 7 − 2.26923 = 4.3077. We can also find L 2 as L2 =

Fn−1 F5 8 L0 = ×7= × 7 = 4.3077, Fn F6 13

which gives same value. To obtain x3 , we first find

5.3 Fibonacci Search Method

97

Fig. 5.5 Discard (x2 , 4] and obtain [−3, x2 ]

f (b) = 24 f (a) = 3

f (x2 ) = .43255 L2∗

xxxxxxxxx

x1 = −.3077 x2 = 1.3077 L0

a = −3

b=4

f (x1 ) = −0.5207 Fig. 5.6 Discard (x1 , x2 ] and obtain [a, x1 ]

f (a) = 3 f (x2 ) = .4325 L3∗

xxxxxxxx a = −3 x3 = −.8521 x1 = −.3077 x2 = 1.3077 L2 f (x3 ) = −0.8521

Fn−3 F3 3 × 7 = 1.6154. L0 = ×7= Fn F6 13

L ∗3 = Thus,

f (x1 ) = −.5207

x3 = −3 + L ∗3 = −3 + 1.6154 = −1.3846,

with f (x3 ) = −0.8521 and given that f (x1 ) = −0.3077. Since f (x3 ) < f (x1 ), therefore we discard (x1 , x2 ]. and obtain new interval L 3 = [−3, −0.3077] which is depicted in Fig. 5.6. We find L 3 = L 2 − L ∗3 = 4.3077 − 1.6154 = 2.6923. To obtain x4 , we find L ∗4 =

Fn−4 F2 L0 = × 7 = 1.0769. Fn F6

Thus, we compute x4 as: x4 = −3 + L ∗4 = −1.9231, with f (x4 ) = −0.1479, and we already have f (x3 ) = −.8521. Since f (x3 ) < f (x4 ), thus we discard [−3, x4 ) and new interval is L 4 = [−1.9231, −0.3077] which is shown in Fig. 5.7. We now find

98

5 One-Dimensional Optimization Methods

Fig. 5.7 Discard [−3, x4 )

f (a) = 3 L4∗ xxxxxxx a = −3 x4 = −1.9231 x3 = −1.3896 x1 = −.3077 L3 f (x4 ) = −.1479 f (x1 ) = −.5207 f (x3 ) = −.8521

L ∗5 =

Fn−5 1 × L0 = × 7 = 0.5384, Fn 13

so that we can compute x5 = −0.3077 + 0.5385 = −0.8462, with corresponding value f (x5 ) = −0.9763, and we already have f (x3 ) = −0.8521. Since f (x5 ) < f (x3 ), thus discard [x4 , x3 ), and obtain new interval L 5 = [−1.3846, −0.3077] which is depicted in Fig. 5.8. To find x6 , we have L ∗6 = We compute

Fn−6 × L 0 = 0.5385. Fn

x6 = x3 + L ∗6 = −0.8461,

with corresponding value f (x6 ) = −0.9761, and we have f (x5 ) = −0.8462. Since f (x5 ) < f (x6 ), thus discard (x6 , x1 ] and new interval is L 6 = [x3 , x6 ] = [−1.3846, −0.8461] which is depicted in Fig. 5.9. The middle point of L 6 is the minimum point x ∗ = 0.4385 and the minimum function value is f (x ∗ ) = 1.0692.

Fig. 5.8 Discard [−3, x4 )

L5∗ xxxxxxxxxxx x4 x3 L4 f (x4 ) = −0.1479

x5

x1 f (x1 ) = −0.5207

f (x3 ) = −0.8521 f (x5 ) = −0.9763

Fig. 5.9 Discard [−3, x4 )

L6∗ x3

x5

xxxxxxxx x6 x1

5.3 Fibonacci Search Method

Note 5.1 Note that

Ln L0

=

0.5385 7

99

= 0.0769 =

1 Fn

=

1 F6

=

1 . 13

The R function Fibonacci_Search(a, b, n) for Fibonacci search algorithm is given below in Function 5.2 R Function 5.2 Fibonacci_Search.R 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34

35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

# Description : Fibonacci Search Method # Theory : See S e c t i o n 5.3 and A l g o r i t h m 5.2 # Function Called : func _ var ( x ) # Input : a,b : L o w e r b o u n d and U p p e r b o u n d # n : M a x i m u m n u m b e r of e x p e r i m e n t # O u t p u t : xm : minimum point # fm : Minimum function value # E x e c u t e on R C o n s o l e : F i b o n a c c i _ S e a r c h (a , b , n ) F i b o n a c c i _ S e a r c h L0 / 2) { anew =b - L2 bnew = a + L2 } else { if ( L2 fx1 ) 57 { 58 b = bnew 59 # C o m p u t e r new i n t e r v a l 60 L2 = f [N - j ] * L0 / f [N - j +2] 61 } 62 else 63 { 64 if ( fx2 < fx1 ) 65 { 66 a = anew 67 L2 = f [N - j ] * L0 / f [N -( j -2) ] 68 } 69 else 70 { 71 if ( fx2 == fx1 ) 72 { 73 b = bnew 74 L2 = f [N - j ] * ( b - a ) / f [N -( j -2) ] 75 } 76 } 77 } 78 cat (j -1 , ’\ t ’ , f o r m a t ( a , d i g i t s =4 , n s m a l l =4) , ’\ t ’ , f o r m a t (b , d i g i t s =4 , n s m a l l =4) , ’\ t ’ , f o r m a t ( fx1 , d i g i t s =4 , n s m a l l =4) , ’ \t’, f o r m a t ( fx2 , d i g i t s =4 , n s m a l l =4) , ’\ n ’ ) 79 } 80 cat ( ’ \ n M i n i m u m p o i n t : ’ , f o r m a t ( a , d i g i t s =4 , n s m a l l =4) ) 81 cat ( ’ \ n M i n i m u m f u n c t i o n v a l u e : ’ , f o r m a t ( func _ var ( a ) , d i g i t s =4 , n s m a l l =4) ) 82 }

Example 5.4 Apply the Fibonacci Search algorithm to minimize the function f (x) = x + esin x on interval [− π2 , π2 ] using R. The R function func_var(x) is used to write the above function as: func_var Fibonacci_Search(a = -pi/2, b = pi/2, n = 23) k 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

a -1.5708 -1.5708 -1.5708 -1.5708 -1.5708 -1.5708 -1.5708 -1.5708 -1.5708 -1.5708 -1.5708 -1.5708 -1.5708 -1.5708 -1.5708 -1.5708 -1.5708 -1.5708 -1.5708 -1.5708 -1.5708 -1.5708 -1.5708 -1.5708

b 1.5708 0.3708 -0.3708 -0.8292 -1.1124 -1.2875 -1.3957 -1.4626 -1.5039 -1.5295 -1.5453 -1.5550 -1.5610 -1.5648 -1.5671 -1.5685 -1.5694 -1.5699 -1.5703 -1.5705 -1.5706 -1.5707 -1.5707 -1.5708

f(x1) 0.3252 0.3252 -0.3508 -0.7046 -0.9047 -1.0222 -1.0926 -1.1352 -1.1613 -1.1773 -1.1871 -1.1931 -1.1969 -1.1992 -1.2006 -1.2015 -1.2020 -1.2024 -1.2026 -1.2027 -1.2028 -1.2028 -1.2028 -1.2029

f(x2) 1.8076 1.8076 0.3252 -0.3508 -0.7046 -0.9047 -1.0222 -1.0926 -1.1352 -1.1613 -1.1773 -1.1871 -1.1931 -1.1969 -1.1992 -1.2006 -1.2015 -1.2020 -1.2024 -1.2026 -1.2027 -1.2028 -1.2028 -1.2029

Minimum point: -1.5708 Minimum function value: -1.2029

See Fig. 5.10.

5.4 Golden Section Search Method Golden Section Search method is the improved version of the Fibonacci search method. The major difficulty in the Fibonacci search method is that the Fibonacci numbers have to be calculated and stored. The proportion of the eliminated region is different at every iteration. To overcome these problems, the Golden section search method is used. In this method, we choose the intermediate points in such a way that the reduction in the range is symmetric in the sense that

102

5 One-Dimensional Optimization Methods

Fig. 5.11 Evaluating objective function at two intermediate points

x1 − a a

b − x2 x1

x2

b

1 , (5.11) 2 which can be depicted in Fig. 5.11. We then evaluate f (x) at two intermediate points x1 and x2 . If f (x1 ) < f (x2 ), then the minimizer x ∗ must lie in the range [a, x2 ]. If on the other hand, f (x1 ) ≥ f (x2 ), then the minimizer is located in the range [x1 , b]. Starting with the reduced range of uncertainty, we can repeat the process and similarly find two new points, say x3 , and x4 using the same value of r < 21 . However, we want to minimize the number of function evaluations while reducing the width of the uncertainty interval. Suppose for example, if we have f (x1 ) < f (x2 ), then we get the minimizer x ∗ lies in interval [a, x2 ]. Since x1 coincides with x4 . Thus, only new evaluations of f at x3 would be necessary. Without loss of generality, imagine that the original range [a, b] is of unit length. Then to find one new evaluation of f it is enough to choose r such that x1 − a = b − x2 = r (b − a), where r
f (x2 ), thus the uncertainty interval is reduced to [0.7639 , 2]. We set a = x1 = 0.7639, b = 2, x1 = x2 = 1.236066, x2 = ra + (1 − r )b = 0.3819 × 0.7639 + (1 − .3819) × 2 = 1.527864, f (x1 ) = f (x2 ) = −1.180339, f (x2 ) = −1.249224. Since f (x1 ) > f (x2 ), thus the uncertainty interval is reduced to [1.236066 , 2]. We set a = x1 = 1.236066, b = 2, x1 = x2 = 1.527864, x2 = ra + (1 − r )b = 0.3819 × 1.236066 + (1 − 0.3819) × 2 = 1.708202, f (x1 ) = f (x2 ) = −1.249224, f (x2 ) = −1.206652. Since f (x1 ) < f (x2 ), thus the uncertainty interval is reduced to [1.236066, 1.708202]. We set b = x2 = 1.708202, x2 = x1 = 1.527864, f (x2 ) = f (x1 ) = −1.249224, x1 = ra + (1 − r )b = 0.3819 × 1.236066 + (1 − 0.3819) × 1.708202 = 1.416407 f (x1 ) = −1.243012.

Since f (x1 ) > f (x2 ), thus the uncertainty interval is reduced to [1.416407, 1.708202]. We set

5.4 Golden Section Search Method

105

a = x1 = 1.416407, b = 1.708202, x1 = x2 = 1.527864, f (x1 ) = f (x2 ) = −1.249224, x2 = ra + (1 − r )b = 0.3819 × 1.236066 + (1 − 0.3819) × 2 = 1.596746, f (x2 ) = −1.24064. Since f (x1 ) < f (x2 ), thus the uncertainty interval is reduced to [1.416407 , 1.596746]. We set b = x2 = 1.596746, x2 = x1 = 1.527864, f (x2 ) = f (x1 ) = −1.249224, x1 = ra + (1 − r )b = 0.3819 × 1.416407 + (1 − 0.3819) × 1.596746 = 1.48529, f (x1 ) = −1.249784.

Since f (x1 ) < f (x2 ), then value x that minimizes f is located in the interval [1.416407 , 1.527864]. We get minimum point 1.4992 and minimum function value −1.2500 in 15 iterations. The R function Golden_Section_Search(a, b, epsilon, maxiter) for the Golden Section Search algorithm is given below in Function 5.3. R Function 5.3 Golden_Section_Search.R 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

21 22 23 24

# Description : Golden Section Search Method # Theory : See S e c t i o n 5.4 and A l g o r i t h m 5.3 # Function Called : func _ var ( x ) # Input : a : Lower bound # b : Upper bound # epsilon : Tolerance Value # maxiter : M a x i m u m n u m b e r of i t e r t i o n s # O u t p u t : xm : minimum point # fxm : minimum function value # E x e c u t e on R C o n s o l e : G o l d e n _ S e c t i o n _ s e a r c h ( a ,b ,1 e -7 ,100) G o l d e n _ S e c t i o n _ S e a r c h Meanof(B) [1] 0.0307

170

6 Steepest Descent Method

We find the variance of the return of stock of the asset Indian Oil Corporation Ltd. > Varianceof(B) [1] 0.0047

We find the covariance between the return of stock of the Reliance Industries Ltd., and of the Indian Oil Corporation Ltd. > Covarianceof(A,B) [1] -0.00025

We use variance–covariance matrix of (6.56) and quadratic function of (6.58) to minimize the risk through the following R function func_var(): fun_var 0, then q(x) achieves a minimum at x (0) and x can be obtained as x = x (0) − (F(x (0) ))−1 ∇ f (x (0) ).

(7.5)

For general current point, the Newton’s method is given as follows: x (k+1) = x (k) − (F(x (k) ))−1 ∇ f (x (k) ). For simplicity, if we use the notation g (k) = ∇ f (x (k) ), then we get the following recursive formula for Newton’s method: x (k+1) = x (k) − (F(x (k) ))−1 g (k) .

(7.6)

7.2 Newton’s Method for Multiple Unknowns The optimization problems of multiple unknowns have great practical relevance. For example, any thermal system has more than two components. Newton’s method is based on the information of the first-order and second-order derivative of nonlinear functions. The second-order derivative of the function is known as a Hessian. If the Hessian matrix is positive semidefinite, then there will be a descent direction. If the initial starting point is far away from the optimum solution, then the search direction may not always be a descent. Therefore, we should always require a restart with a different starting point to avoid such difficulty. Although, Newton’s method is well known for converging in a single iteration for the quadratic function. The algorithm for Newton’s method is described in Algorithm 7.1.

7.2 Newton’s Method for Multiple Unknowns

177

Algorithm 7.1 (Newton’s Algorithm) 1: Choose x (0) ∈ Rn , ε  x (0) is a starting point, ε is a tolerance value 2: Set k ← 0.  k is number of iterations of method 3: for k to maxiter do  maxiter is maximum number of iterations to perform 4: Compute ∇ f (x (k) )  gradient of f at x (k) (k) −1 5: Compute (F(x )) .  inverse of Hessian of f at x (k) 6: Update x (k+1) ← x (k) − (F(x (k) ))−1 ∇ f (x (k) ).  a new point 7: if  f (x (k+1) ) − f (x (k) ) < ε or ∇ f (x (k) ) < ε then  stopping criteria 8: Converged. Print x ∗ ← x (k+1) and f (x ∗ ) ← f (x (k+1) )  best solution 9: end if 10: end for

Example 7.1 Apply Newton’s method to minimize the function f (x) = 8x12 − 4x1 x2 + 5x22 , from a starting point x (0) =

  5 . 2

The gradient and Hessian of f are  16x1 − 4x2 , −4x1 + 10x2   16 −4 F(x) = . −4 10 

∇ f (x) =

The gradient and Hessian of f at x (0) are ∇ f (x (0) ) =



   80 − 8 72 = , −20 + 20 0

and F(x (0) ) =



 16 −4 . −4 10

This is a real symmetric positive definite because |F(x (0) )| = 144 > 0. The inverse of the Hessian of f at x (0) is (F(x A new point is generated as

(0)

))

−1

  1 10 4 = . 144 4 16

178

7 Newton’s Method

     1 10 4 72 5 − 0 2 144 4 16     1 720 5 = − 2 144 288     5 5 = − 2 2   0 = . 0

x (1) = x (0) − (F(x (0) ))−1 ∇ f (x (0) ) =

Since ∇ f (x (1) ) =

  0 , 0 ∗

thus the algorithm stops. The minimum value is x = x function value is f (x ∗ ) = 0.

(1)

  0 = , and minimum 0

Example 7.2 Apply Newton’s method to find the minimizer of the function f (x1 , x2 , x3 ) = (x1 − 4)2 + (x2 − 3)2 + 4(x3 + 5)2 . with a starting point x (0)

⎡ ⎤ 0 = ⎣0⎦ . The gradient and Hessian of f are 0 ⎡

⎤ 2x1 − 8 ∇ f (x) = ⎣ 2x2 − 6 ⎦ , 8x3 + 40 ⎡ ⎤ 200 F(x) = ⎣0 2 0⎦ . 008 The gradient and Hessian of f at x (0) are ⎡

⎤ −8 ∇ f (x (0) ) = ⎣−6⎦ , 40 and

⎡

⎤ 200 F(x (0) ) = ⎣0 2 0⎦ . 008

7.2 Newton’s Method for Multiple Unknowns

179

This is a real symmetric positive definite because |F(x (0) )| = 32 > 0. The inverse of the Hessian of f at x (0) as ⎡

(F(x (0) ))−1

⎤ 0 0 = ⎣ 0 1/2 0 ⎦ . 0 0 1/8 1/2

A new point is generated as x (1)

⎤⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ 1/2 0 0 −8 4 0 = x (0) − (F(x (0) ))−1 ∇ f (x (0) ) = ⎣0⎦ − ⎣ 0 1/2 0 ⎦ ⎣−6⎦ = ⎣ 3 ⎦ . 40 −5 0 0 0 1/8

(1) ∗ (1) Since ⎡ ⎤∇ f (x ) = 0, thus the algorithm stops. The minimum value is x = x = 4 ⎣ 3 ⎦, and minimum value of the function is f (x ∗ ) = 0. −5 The R function Newton(x, epsilon, maxiter) for the Newton’s method is given below in Function 7.1.

R Function 7.1 Newton.R 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

# Description : Newton ’ s m e t h o d for m u l t i p l e u n k n o w n s # Theory : See S e c t i o n 7.2 and A l g o r i t h m 7.1 # Function Called : func _ var ( ) , Grad _ Vec ( ) , H e s s i a n ( ) # Input : x : Starting point # epsilon : Tolerance Value # maxiter : N u m b e r of i t e r t i o n s p e r f o r m e d # Output : x : Minimum point # fm : Minimum function value # E x e c u t e on R C o n s o l e : N e w t o n ( x , 1 e -7 , 1 0 0 ) N e w t o n Newton(x = c(1, -1, 1), epsilon = 1e-7, maxiter = 100) Initial function value: -1.0000 k 0 1 2 3 4 5

1.0000 0.7104 0.5926 0.5715 0.5709 0.5709

x -1.0000 -0.9543 -0.9417 -0.9396 -0.9396 -0.9396

1.0000 0.8323 0.7782 0.7685 0.7682 0.7682

f(x) -1.0000 -1.8367 -1.9101 -1.9118 -1.9118 -1.9118

||gradient|| 5.0000 1.1538 0.1528 0.004425 4.1e-06 2.613e-12

Minimum point: 0.5709 -0.9396 0.7682 Minimum function value: -1.9118

Since the starting point is closer to the minimum point, therefore the function value reduces monotonically, and eventually converges to the minimum in five iterations. Let us call again the function Newton(x, epsilon, maxiter) with x = c(0, 1, 0), epsilon = 1e-7 and maxiter = 100 to display the following output on R Console: > Newton(x=c(0, 1, 0), epsilon = 1e-7, maxiter = 100) Initial function value: 1.0000 k 0 1 2 3

0.0000 -1.0000 -0.7213 -0.2312

x 1.0000 0.7000 0.5526 0.1996

0.0000 1.0000 0.6787 0.4366

f(x) 1.0000 0.7501 0.4686 0.1169

||gradient|| 3.3166 1.3345 0.5723 0.9776

7.2 Newton’s Method for Multiple Unknowns 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

-1.2368 -0.8878 -0.6011 -0.4721 -0.8519 -0.5833 7.4466 4.9162 3.2124 2.0888 1.3584 0.9070 0.6663 0.5818 0.5710 0.5709 0.5709

0.2315 0.6271 0.2598 0.6203 0.3956 0.6384 0.9978 0.7744 0.7693 0.8711 0.5795 0.6435 -3.0519 -2.7169 -2.5627 6.1803 -1.8051 2.7540 -1.3559 1.6637 -1.1039 1.1577 -0.9897 0.9230 -0.9503 0.8119 -0.9406 0.7732 -0.9396 0.7683 -0.9396 0.7682 -0.9396 0.7682

181 1.4523 0.6275 0.3954 0.8704 0.6049 0.4101 3626.6756 686.1827 130.8410 22.6392 2.1626 -1.3935 -1.8780 -1.9114 -1.9118 -1.9118 -1.9118

3.8386 1.2732 0.5386 2.6133 0.8313 0.4715 1830.0136 531.8338 157.9117 46.3743 13.3000 3.5397 0.7440 0.07586 0.001147 2.758e-07 3.14e-13

Minimum point: 0.5709 -0.9396 0.7682 Minimum function value: -1.9118

Note that if the starting point is far away from the minimum point, then in some iteration, the search direction is not a descent as the function value increases instead of monotonically decreasing. However, the algorithm converges to the minimum point in 20 iterations. Example 7.3 Apply the Newton’s method to minimize   the function f (x1 , x2 ) = 1 2 2 (0) x1 + 2x2 − 4x1 − 2x1 x2 with a starting point x = using R. 3 The R function func_var(x) is written for the given function as func_var Newton(x = c(1,3), epsilon = 1e-7, maxiter = 100) Initial function value: 9.0000 k 0

x 1.0000

3.0000

f(x) 9.0000

||gradient|| 12.8062

182

7 Newton’s Method

1 2 −1

x2

3

Fig. 7.1 Graph of x12 + 2x22 − 4x1 − 2x1 x2

−1

0

1

2

3

4

5

x1

1

4.0000

2.0000

-8.0000

1.79e-09

Minimum point: 4.0000 2.0000 Minimum function value: -8.0000

See Fig. 7.1. Example 7.4 Apply the Newton’s method to minimize the following function: f (x) = x14 + x12 (1 − 2x2 ) − 2x2 + 2x22 − 2x3 + x32 + x34 + 2x1 (−2 + x4 ) − 4x4 − 2x32 x4 + 2x42 ⎡

with a starting point x (0)

⎤ 3 = ⎣−1⎦ using R. 2

The R function func_var() is written for the given function as func_var Newton(x = c(2,2,1,1), epsilon = 1e-7, maxiter = 100) Initial function value: 0.0000 k 0

x 2.0000 2.0000 1.0000 1.0000

f(x) 0.0000

||gradient|| 18.2209

7.2 Newton’s Method for Multiple Unknowns 1 2 3 4 5 6

1.4932 1.1803 1.0372 1.0021 1.0000 1.0000

1.4863 1.1476 1.0277 1.0015 1.0000 1.0000

0.8356 0.9373 0.9874 0.9993 1.0000 1.0000

0.5890 0.8439 0.9676 0.9982 1.0000 1.0000

183 0.0000 -5.1655 -5.9122 -5.9967 -6.0000 -6.0000

4.6323 1.2234 0.2136 0.01177 4.169e-05 5.11e-10

Minimum point: 1.0000 1.0000 1.0000 1.0000 Minimum function value: -6.0000

7.3 Convergence Analysis of Newton’s Method In the previous section, we have seen the examples where Newton’s method gives the minimizer of quadratic functions in one iteration only. The convergence analysis of Newton’s method is very straightforward when the function f is quadratic. The Newton’s method reaches the minimum point x ∗ , such that ∇ f (x ∗ ) = 0, in just one step starting from any initial point x (0) . To see this, consider the following quadratic function: f (x) =

1 T x Qx − x T b, 2

(7.7)

where Q = Q T > 0. We have g(x) = ∇ f (x) = Qx − b, and F(x) = Q. Thus, for the given initial point x (0) , we use Newton’s method (7.6) to get the following: x (1) = x (0) − (F(x (0) ))−1 g(x (0) ) Since (F(x (0) ))−1 = Q −1 , and g(x (0) ) = Qx (0) − b, then x (1) = x (0) − Q −1 [Qx (0) − b] = x (0) − Q −1 Qx (0) + Q −1 b = x (0) − x (0) + Q −1 b = Q −1 b. Since Qx ∗ = b, therefore

184

7 Newton’s Method

x (1) = x ∗ .

(7.8)

Let {x (k) } be the Newton’s method sequence for minimizing a function f : Rn → R. We show that {x (k) } converges to the minimizer x ∗ with an order of convergence at least 2. Theorem 7.1 Suppose that f ∈ C 3 and x ∗ ∈ Rn , which is not a minimum, is a point such that ∇ f (x ∗ ) = 0 and F(x ∗ ) is invertible. Then, for x (0) , sufficiently close to x ∗ , Newton’s method is well defined for all k, and converges to x ∗ with an order of convergence at least 2. Proof Consider the Taylor expansion of ∇ f (x) about x = x (0) as ∇ f (x) = ∇ f (x (0) ) + F(x (0) )(x − x (0) ) + O((x − x (0) )2 ), that is, ∇ f (x) − ∇ f (x (0) ) − F(x (0) )(x − x (0) ) = O((x − x (0) )2 ).

(7.9)

Therefore, ∇ f (x) − ∇ f (x (0) ) − F(x (0) )(x − x (0) ) ≤ c1 x − x (0) 2

(7.10)

We know that if A : Rr → Rn×n be an n × n matrix-valued function that is continuous at ξ0 and if (A(ξ0 ))−1 exists, then (A(ξ ))−1 exists for ξ sufficiently close to ξ0 and (A(.))−1 is continuous at ξ0 . Since F is continuous at x ∗ and F(x ∗ ) is invertible, ∗ −1 then (F(x))−1 is continuous for x sufficiently close to x , that is, (F(x)) is con∗ tinuous  on x : ||x − x || ≤  for ε > 0. Since it is continuous, then it is bounded on x : ||x − x ∗ || ≤ ε for ε > 0. That is, (F(x ∗ ))−1  ≤ c2 for some c2 > 0.

(7.11)

Suppose that x (0) ∈ {x : x − x ∗ } ≤ ε, then putting x = x ∗ in (7.10), we get ∇ f (x ∗ ) − ∇ f (x (0) ) − F(x (0) )(x ∗ − x (0) ) ≤ c1 x ∗ − x (0) 2 . Since ∇ f (x ∗ ) = 0, then −∇ f (x (0) ) − F(x (0) )(x ∗ − x (0) )≤ c1 x ∗ − x (0) 2 , that is, F(x (0) )(x (0) − x ∗ ) − ∇ f (x (0) )≤ c1 x (0) − x ∗ 2 . Recall Newton’s iteration formula for the first iterate:

(7.12)

7.3 Convergence Analysis of Newton’s Method

185

x (1) = x (0) − (F(x (0) ))−1 ∇ f (x (0) ).

(7.13)

Subtracting x ∗ from both sides of (7.13), we get x (1) − x ∗ = x (0) − x ∗ − (F(x (0) ))−1 ∇ f (x (0) ), that is, x (1) − x ∗  = x (0) − x ∗ − (F(x (0) ))−1 ∇ f (x (0) )  = (F(x (0) ))−1 F(x (0) )(x (0) − x ∗ ) − ∇ f (x (0) ) || ≤ (F(x (0) ))−1 F(x (0) )(x (0) − x ∗ ) − ∇ f (x (0) ). Thus, x (1) − x ∗  ≤ (F(x (0) ))−1 F(x (0) )(x (0) − x ∗ ) − ∇ f (x (0) ).

(7.14)

Using (7.11) and (7.12) in (7.14) to get x (1) − x ∗  ≤ c1 c2 x (0) − x ∗ 2 .

(7.15)

Since x (0) ∈ {x : x − x ∗  ≤ ε}, and suppose that x (0) is such that x (0) − x ∗ ≤ where α ∈ (0, 1], then x (1) − x ∗ ≤ c1 c2

α , c1 c2

α x (0) − x ∗ , c1 c2

that is, x (1) − x ∗ ≤ αx (0) − x ∗ . Similarly, we get x (k+1) − x ∗ ≤ c1 c2 x (k) − x ∗ 2 , that is, ||x (k+1) − x ∗ ||≤ α||x (k) − x ∗ ||. We get

lim x (k) − x ∗ = 0.

k→∞

 Therefore, the sequence x (k) converges to x ∗ . This completes the proof.



186

7 Newton’s Method

Remark 7.1 1. The order of convergence is at least 2 because x (k+1) − x ∗ ≤ c1 c2 x (k) − x ∗ 2 = O(x (k) − x ∗ 2 ). 2. Newton’s method has superior convergence properties if the starting point x (0) is sufficiently close to the solution x ∗ . Note that the above method is not guaranteed to converge to the minimizer if we start far away from the solution. Of course, it may not be a descent method as we see in the following example. Example 7.5 Consider the function f (x) = − x 2 +x1 2 +1 . The minimum function f 1 2   0 ∗ ∗ with f (x ) = −1. has a minimizer at x = 0 The gradient and Hessian of function f (x) are  ∇ f (x) = 2

x1/(x 2 +x 2 +1)2

1 2 x2 (x 2 +x 2 +1)2 1 2

/

 ,

and F(x) =

  2 −3x12 + x22 + 1 −4x1 x2 . −4x1 x2 x12 − 3x22 + 1 (x12 + x22 + 1)3 

 20 Hence, we get F(x ) ≈ in a small neighborhood of x ∗ , If we take x (0) suf02  (0)  x ficiently close to x ∗ , then x (k) will converge to the minimizer. Let x (0) = 1 , 0 where x1 1. Then, the Hessian at x (0) is given as ∗

F(x

(0)

)=

−

6 (x1(0) )4

0

0 −6 (x1(0) )4

 ,

which is not positive definite. Therefore, x (1) = x (0) − (F(x (0) ))−1 g (0) =

4

x (0) 3 1 0

 .

Thus, the iterated sequence {x (1) , x (2) , . . . } diverges away from the origin. The following result modifies the Newton’s method in a simple and logical way to hold the descent property, where the Hessian is assumed as positive definite.

7.3 Convergence Analysis of Newton’s Method

187

 Theorem 7.2 Let x (k) be a sequence generated by the Newton’s method for minimizing a given objective function f (x). If the Hessian F(x (k) ) > 0 and g (k) = ∇ f (x (k) ) = 0, then the search direction d (k) = x (k+1) − x (k) = −(F(x (k) ))−1 g (k) from x (k) to x (k+1) is a descent direction for f in the sense that there exists an α¯ > 0 such that for all α ∈ (0, α), ¯ f (x (k) + αd (k) ) < f (x (k) ). Proof Let φ(α) = f (x (k) + αd (k) ). Then, by chain rule, we get 

φ (α) = (∇ f (x (k) + αd (k) ))T d (k) .

(7.16)

Set α = 0 in (7.16) then 

φ (0) = (∇ f (x (k) ))T d (k) . Since d (k) = −(F(x (k) ))−1 g (k) , then 

φ (0) = −(∇ f (x (k) ))T (F(x (k) ))−1 g (k) .

(7.17)

Since (∇ f (x (k) ))T = (g (k) )T , then 

φ (0) = −(g (k) )T (F(x (k) ))−1 g (k) .

(7.18)

Since (F(x (k) ))−1 > 0 and g (k) = 0, then 

φ (0) = −(g (k) )T (F(x (k) ))−1 g (k) < 0.

(7.19)

Therefore, φ is decreasing at “0”. It means that there exists α¯ > 0 such that for all α ∈ (0, α), ¯ we have φ(α) < φ(0), which leads to ¯ f (x (k) + αd (k) ) < f (x (k) ), ∀ α ∈ (0, α). This completes the proof.

(7.20) 

188

7 Newton’s Method

7.4 Modified Newton’s Method Newton’s method has superior convergence properties provided the starting point is sufficiently close to a minimizer. However, the method is not guaranteed to converge to the solution if we start far away from it. The method may not be descent method that is it is possible that f (x (k+1) ) ≥ f (x (k) ). Fortunately, it is possible to modify the Newton’s method such that descent property holds. The modified Newton’s method is similar to Newton’s method with a modification that a unidirectional search is performed in the search direction of the Newton’s method. Note that Theorem 7.2 motivates the following modified Newton’s method: x (k+1) = x (k) − αk (F(x (k) ))−1 g (k) , where

αk = arg minα≥0 ( f (x (k) ) − α(F(x (k) ))−1 g (k) ).

It means that we compute the step length αk using any line search methods in the direction −(F(x k ))−1 g k at every iteration. From Theorem 7.2, we conclude that the modified Newton’s method has the descent property, that is f (x (k+1) ) < f (x (k) ), whenever g (k) = 0 and (F(x (k) ))−1 > 0. On the basis of above theory, the modified Newton’s method is presented below in the form of algorithm. Algorithm 7.2 (Modified Newton’s Algorithm) 1: Choose x (0) ∈ Rn , ε  x (0) is a starting point and ε is tolerance value 2: Set k ← 0  k is number of iterations of method 3: for k to maxiter do  maxiter is maximum number of iterations to perform 4: Compute g (k) ← ∇ f (x (k) )  gradient of f at x (k) 5: Compute (F(x (k) ))−1  inverse of Hessian of f at x (k) 6: Compute d (k) ← −(F(x (k) ))−1 ∇ f (x (k) )  descent direction 7: Find αk ← arg minα≥0 ( f (x (k) ) − α(F(x (k) ))−1 g (k) )  use Golden_Line() or (6.23) 8: Update x (k+1) ← x (k) + αk d (k)  a new point 9: if ( f (x (k+1) ) − f (x (k) )) < ε or ∇ f (x (k) ) < ε then  stopping criterion 10: Converged. Print x ∗ ← x (k+1) and f (x ∗ ) ← f (x (k+1) )  best solution 11: end if 12: end for

7.4 Modified Newton’s Method

189

Example 7.6 Use modified Newton’s method to minimize the function f (x) = x1 − x2 + 2x12 + 2x1 x2 + x22 with a starting point x

(0)

  0 = . 0

The gradient of f is

 1 + 4x1 + 2x2 . ∇ f (x) = −1 + 2x1 + 2x2 

The gradient and Hessian of f at x (0) are g

(0)

= ∇ f (x

(0)

and Q = F(x

(0)



 1 )= , −1 

 42 )= . 22

This is a real symmetric positive definite because |F(x (0) )| = 4 > 0. Therefore, the inverse of the Hessian of f is (F(x

(0)

))

−1

  1 2 −2 = . 4 −2 4

The step length is computed as

α0 =

(g (0) )T g (0) (g (0) )T Qg (0)

 

1 1 −1 −1 2 2    =   = = 1. =

42

2 1 1 1 −1 20 2 2 −1 −1

A new point generated for modified Newton’s method is x

(1)

=x

(0)

− α0 (F(x

(0)

−1

)) ∇ f (x

(0)

     1 2 −2 0 1 )= −1× −2 4 0 −1 4     1 4 0 = − × −6 0 4   −1 . = 3 /2.

The gradient of f (x) at x (1) is ∇ f (x

(1)



   1−4+3 0 )= = . −1 − 2 + 3 0

190

7 Newton’s Method

Since ∇ f (x (1) ) = 0, therefore the modified Newton algorithm stops. The minimum  −1 , and the minimum value of the objective function is point is x ∗ = x (1) = 3 /2 f (x ∗ ) = (−1) −

 3   3 2 3 + = −1.25. + 2(−1)2 + 2(−1) 2 2 2

The R function Modified_Newton(x, epsilon, maxiter) for the modified Newton algorithm is given below in Function 7.2. R Function 7.2 Modified_Newton.R 1 # Description : Modified Newton Method 2 # Theory : See S e c t i o n 7.4 and A l g o r i t h m 7.2 3 # Function Called : func _ var ( ) , G o l d e n _ Line ( ) , H e s s i a n ( ) 4 # Input : x : Starting point 5 # epsilon : Tolerance Value 6 # maxiter : M a x i m u m of n u m b e r of i t e r t i o n s to perform 7 # O u t p u t : xm : Minimum point 8 # fm : M i n i m u m v a l u e of f u n c t i o n at xm 9 # E x e c u t e on R C o n s o l e : M o d i f i e d _ N e w t o n ( x , 1 e -7 , 1 0 0 ) 10 M o d i f i e d _ N e w t o n Modified_Newton(x=c(0.5,0), epsilon = 1e-7,maxiter = 100) Initial function value: -0.4444 k 0 1

x 0.5000 0.0005909

0.0000 0.0000000

f(x) -0.4444 -0.5000

Minimum point: 0.0005909 0.0000000 Minimum function value: -0.5000

||gradient|| 0.1975 0.0002955

192

7 Newton’s Method

Example 7.8 Apply the modified Newton’s method to minimize the Freudenstein and Roth test function (Moré et al. 1981) f (x) = (−13 + x1 + ((5 − x2 )x2 − 2)x2 )2 + (−29 + x1 + ((x2 + 1)x2 − 14)x2 )2 

with a starting point x

(0)

 1 = using R. −3

We use R function func_var(x) to write above objective function func_var 1 for all α0 > 1, thus the value of f (x) cannot be minimized. We conclude that the modified Newton’s method fails to get minimizer of this function. In the next section, we describe Levenberg–Marquardt Method which is the modification of Newton’s method to overcome problems as stated above.

194

7 Newton’s Method

7.5 Levenberg–Marquardt Method The Levenberg–Marquardt method is a composition of steepest descent and Newton’s method that converges to a solution point. We have previously noted that the steepest descent method typically provides a good reduction in the objective function when starting point x (0) is far away from the solution x ∗ . On the other hand, the Newton’s method generates the descent directions when the initial point x (0) is sufficiently close to the solution. It is not known to us that the chosen initial point is far away from the solution or close to the solution for any objective function, and if the Hessian F(x (k) ) is not positive definite, then also the search direction d (k) = −F(x (k) )−1 g (k) is not a descent direction. We present that if the Hessian is not positive definite, it can still be modified so that the iteration resulting from this modification has the descent property. Consider the matrix Mk = F(x (k) ) + μk In , where In is the n × n identity matrix and μk is scalar. If we denote the eigenvalues of F by λi , where i = 1, . . . , n then the eigenvalues of Mk are given by λi + μk , where i = 1, . . . , n. If vi is the eigenvector of F corresponding to the eigenvalue λi , then Mk vi = (F(x (k) ) + μk In )vi = λi vi + μk vi = (λi + μk )vi . Thus, if we choose μk > |λmin (F(x (k) ))|, where λmin (F(x (k) )) is the minimum eigenvalue of F(x (k) ), then all eigenvalues of Mk are positive, so Mk is positive definite matrix. To make sure that the descent property holds, we can use the direction −Mk−1 g (k) instead of the direction −(F(x (k) ))−1 g (k) used in Newton’s method. Including the step length, we obtain the following iteration: x (k+1) = x (k) − αk Mk−1 g (k) ,

(7.21)

where αk = arg minα≥0 f (x k − α Mk−1 g (k) ). This method is referred to as the Levenberg–Marquardt method. To see this, we write down the derivative of the function φk (α) = f (x k − α Mk−1 g k ). We have φk (0) = −(g (k) )T (Mk )−1 g (k) < 0. Since Mk−1 is positive definite. The Levenberg–Marquardt method is in some sense intermediate between the steepest descent and Newton’s methods. If μk = 0, then

7.5 Levenberg–Marquardt Method

195

it coincides with Newton’s method. If μk = 0, then Mk = C In for some very large C > 0, so Mk−1 = ε In for some small ε = C1 > 0, and the iteration is x (k+1) = x (k) − αk εg (k) . Thus, we obtain an approximation of the steepest descent iteration in this case. The iterative procedure of Levenberg–Marquardt algorithm is summarized as below. Algorithm 7.3 (Levenberg–Marquardt Algorithm) 1: Choose x (0) ∈ Rn , ε  x (0) is a starting point, ε is a tolerance value 2: Set k ← 0  k is number of iterations of method 3: Set μk ← 103  value of scalar μk is taken large 4: for k to maxiter do  maxiter is maximum number of iterations to perform 5: Compute g (k) ← ∇ f (x (k) )  gradient of f at x (k) 6: Compute F(x (k) )  Hessian of f at x k (k) 7: Compute Mk ← F(x ) + μk In  positive definite matrix 8: Compute d (k) ← −Mk−1 ∇g (k)  descent direction  Use Golden_Line() or (6.23) 9: Compute αk ← arg minα≥0 f (x k − α Mk−1 g (k) ) 10: Update x (k+1) ← x (k) + αk d (k) .  a new point 11: if f (x (k+1) ) < f (x (k) ) then 12: Set μk+1 ← μ2k  approaches to Newton’s method 13: else 14: Set μk+1 ← 2μk  approaches to steepest descent 15: end if 16: if ( f (x (k+1) ) − f (x (k) )) < ε or ∇ f (x (k) ) < ε then  stopping criteria 17: Converged. Print x ∗ ← x (k+1) and f (x ∗ ) ← f (x (k+1) ).  best solution 18: end if 19: end for

Example 7.10 Apply the Levenberg–Marquardt method to minimize the following function: f (x) = x1 − x2 + 3x12 + x1 x2 + 2x22 with starting point x

(0)

  1 = , and μ0 = 103 . 1

The value of function f at x (0) is f (x (0) ) = 6. The gradient of f is given as  1 + 6x1 + x2 . ∇ f (x) = −1 + x1 + 4x2 

196

7 Newton’s Method

The gradient and Hessian of f at x (0) are g

(0)

= ∇ f (x

(0)

  8 )= , 4

and F(x

(0)

  61 )= . 14

We find the step length

α0 =

(g (0) )T g (0) (g (0) )T Qg (0)

  8 84 4 5 80    = = . = 61 8 512 32 84 14 4

We now compute a new point as x (1) = x (0) − α0 (F(x (0) ) + μ0 I2 )−1 ∇ f (x (0) )     −1     5 1 8 0.9998 1 6 + 103 = − × = . × 1 4 0.9998 1 4 + 103 32 The value of function f (x) at x = x (1) is given as f (x (1) ) = 5.9976. Since f (x (1) ) < f (x (0) ), then set μ1 =

1000 μ0 = = 500, 2 2

and proceed to the next iteration. The gradient of f (x) at x = x (1) is as follows: g (1) = ∇ f (x (1) ) =



 7.9986 . 3.9990

The Hessian will remain same. We again find the step length

α1 =

(g (1) )T g (1) (g (1) )T Qg (1)

 

7.9986 7.9986 3.9990 3.9990    = 0.1562. =

6 1 7.9986 7.9986 3.9990 1 4 3.9990

7.5 Levenberg–Marquardt Method

197

We now compute a new point as x (2) = x (1) − α1 (F(x (1) ) + μ1 I2 )−1 ∇ f (x (1) )       −1   0.9998 7.9986 506 1 0.9973 = × − 0.1562 × = . 0.9998 3.9990 1 504 0.9986 The value of function f at x = x (2) is f (x (2) ) = 5.97283. Since f (x (2) ) < f (x (1) ), then set μ2 =

500 μ1 = = 250, 2 2

and proceed to the next iteration. The iteration process is to be continued until it satisfies the convergence criterion. point and minimum value of the  The minimum  −0.2174 function are obtained as x ∗ = and f (x ∗ ) = −0.2609 in five iterations, 0.3044 if we take tolerance value ε = 10−7 . The R function Levenberg_Marquardt(x, mu, epsilon, maxiter) for the Levenberg–Marquardt method is given below in Function 7.3 R Function 7.3 Levenberg_Marquardt.R 1 # Description : I m p l e m e n t s the Levenberg - M a r q u a r d t Algorithm 2 # Theory : See S e c t i o n 7.5 and A l g o r i t h m 7.3 3 # Function Called : func _ var ( ) , Grad _ Vec ( ) , H e s s i a n ( ) 4 # Input : x : Starting Point 5 # epsilon : Optimization Tolerance Value 6 # maxiter : N u m b e r of i t e r t i o n p e r f o r m e d 7 # mu : Large value 8 # O u t p u t : xm : minimum point 9 # fm : m i n i m u m f u n c t i o n v a l u e at xm 10 # E x e c u t e on R C o n s o l e : L e v e n b e r g _ M a r q u a r d t ( x , 1 e3 , 1e -7 , 100) 11 L e v e n b e r g _ M a r q u a r d t Levenberg_Marquardt(x=c(1,-1,1),mu=1e3,epsilon=1e-7, maxiter=100) Initial function value: -1.0000 k 0 1 2 3 4 5 6 7 8 9 10 11 12

1.0000 0.9951 0.9856 0.9679 0.9366 0.8867 0.8180 0.7398 0.6685 0.6174 0.5888 0.5761 0.571

x -1.0000 -1.0000 -1.0000 -0.9998 -0.9994 -0.9979 -0.9938 -0.9853 -0.9724 -0.9586 -0.9481 -0.9423 -0.9401

1.0000 1.0000 1.0000 0.9998 0.9993 0.9976 0.9921 0.9776 0.9464 0.8953 0.8365 0.7928 0.7735

f(x) -1.0000 -1.0243 -1.0701 -1.1515 -1.2821 -1.4584 -1.6433 -1.7835 -1.8598 -1.8935 -1.9072 -1.9112 -1.9117

mu ||gradient|| 1000.00 5.0000 500.000 4.9120 250.000 4.7441 125.000 4.4376 62.5000 3.9207 31.2500 3.1559 15.6250 2.2212 7.8125 1.3245 3.9062 0.6692 1.9531 0.3064 0.9766 0.1327 0.4883 0.0451 0.2441 0.0096

7.5 Levenberg–Marquardt Method 13 14

0.5710 0.5709

-0.9396 -0.9396

199

0.7688 0.7682

-1.9118 -1.9118

0.1221 0.06104

0.0011 7.586e-05

Minimum point: 0.5709 -0.9396 0.7682 Minimum function value: -1.9118

Note that if the function value decreases in the iteration, the value of μk also decreases by some factor, which indicates the less weightage on the steepest descent direction. The Levenberg–Marquardt algorithm converges to the minimum point in 13 iterations. Example 7.11 Minimize f (x) = x14 + x1 x2 + (1 + x2 )2 with starting point x (0) =   0 . 0 We have seen in Example 7.9 that the modified Newton’s method fails to get the minimizer of this function, because the Hessian of f at x (0) is not positive definite, that is,   01 (0) F(x ) = . 12 But, we can solve this with the help of Levenberg–Marquardt method. The R function func_var(x) is used for the above objective function as func_varLevenberg_Marquardt(x = c(0,0), mu = le3, epsilon = le-7, maxiter=100) Initial function value: 1.0000 k 0 1 2 3 4 5 6

x 0.0000 1.996e-06 1.394e-05 6.939e-05 0.0003045 0.001257 0.004976

0.0000 -1.996e-03 -5.972e-03 -1.386e-02 -0.0293937 -0.059509 -0.116230

f(x) 1.0000 0.9960 0.9881 0.9725 0.9421 0.8844 0.7805

mu ||gradient|| 1000.0000 2.0000 500.0000 1.9960 250.0000 1.9881 125.0000 1.9724 62.5000 1.9417 31.250 1.8832 15.6250 1.7763

7 Newton’s Method

−2

x2

Fig. 7.3 Graph of x14 + x1 x2 + (1 + x2 )2

0 1 2

200

−1.0

−0.5

0.0

0.5

1.0

x1

7 8 9 10 11 12 13 14

0.0189 0.06802 0.2270 0.5843 0.6660 0.6896 0.6951 0.6958

-0.2176 -0.38399 -0.6310 -0.9655 -1.2124 -1.3188 -1.3444 -1.3477

0.6081 0.3534 -0.004428 -0.4464 -0.5656 -0.5817 -0.5824 -0.5824

7.8125 3.9062 1.9531 0.9766 0.4883 0.2441 0.1221 0.06104

1.5986 1.3552 1.1280 0.6745 0.2431 0.05243 0.006342 0.0004111

Minimum point: 0.6959 -1.3479 Minimum function value: -0.5824

See Fig. 7.3 Example 7.12 Minimize the function f (x1 , x2 ) = 2x12 + 15x22 − 3x1 x2 − 2x1 − 6x2 − 1 5 using Levenberg–Marquardt method with starting point x (0) = (Fig. 7.3). 3 The R function func_var() for above objective function is given as func_var Levenberg_Marquardt(x = c(1,5), mu = 1e3, epsilon = 1e-7, maxiter = 100) Initial function value: 325.0000 k 0

x 1.0000

5.0000

f(x) 325.0000

mu ||gradient|| 1000.0000 141.5980

7.5 Levenberg–Marquardt Method

−1

1

x2

3

5

Fig. 7.4 Graph of 2x12 + 15x22 − 3x1 x2 − 2x1 − 6x2 − 5

201

−1

0

1

2

3

4

5

x1

1 2 3 4 5 6 7 8 9 10 11 12 13

1.0125 1.0359 1.0764 1.1377 1.2077 1.2453 1.1981 1.0593 0.8901 0.7682 0.7165 0.7043 0.7028

4.8631 4.6051 4.1446 3.4032 2.4035 1.3852 0.6847 0.3842 0.3000 0.2782 0.2719 0.2705 0.2703

305.8266 271.2311 214.5809 137.0020 59.0226 10.9069 -4.0626 -6.1865 -6.4467 -6.5055 -6.5132 -6.5135 -6.5135

500.0000 250.0000 125.0000 62.5000 31.2500 15.6250 7.8125 3.9062 1.9531 0.9766 0.4883 0.2441 0.1221

137.4299 129.5708 115.5541 92.9978 62.6342 1.8421 10.9708 2.5860 0.7380 0.2419 0.05087 0.00599 0.0003747

Minimum point: 0.7027 0.2703 Minimum function value: -6.5135

See Fig. 7.4 Example 7.13 Use the Levenberg–Marquardt method to minimize the following Beale function (Rao and Singiresu 1996): f (x1 , x2 ) = (1.5 − x1 + x1 x2 )2 + (2.5 − x1 + x1 x22 )2 + (2.625 − x1 + x1 x23 )2 with a starting point x (0) =

  1 . The R function func_var() for above objective 2

function is given as func_var Levenberg_Marquardt(x = c(1,3), lambda = 1e3, epsilon = 1e-7, maxiter = 100) Initial function value: 941.8906 k 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

x 1.0000 1.5838 1.5608 1.5898 1.6720 1.8062 1.9824 2.1720 2.3391 2.4814 2.6222 2.7605 2.8794 2.9640 3.0084 3.0228 3.0251

3.0000 2.0188 1.6900 1.3779 1.0837 0.8126 0.5760 0.3966 0.3045 0.3050 0.3516 0.3985 0.4345 0.4579 0.4694 0.4730 0.4736

f(x) 941.8906 262.0504 109.7069 46.8290 19.9767 8.1036 2.8594 0.8309 0.2826 0.1585 0.09453 0.05891 0.04376 0.03916 0.03834 0.03828 0.03828

Minimum point: 3.0253 0.4737 Minimum function value: 0.03828

See Fig. 7.5

Lambda ||gradient|| 1000.0000 2368.2850 500.0000 697.4529 250.0000 308.2470 125.0000 137.5518 62.5000 62.0903 31.2500 28.2645 15.6250 12.5940 7.8125 4.9876 3.9062 1.5192 1.9531 0.5597 0.9766 0.3272 0.4883 0.1792 0.2441 0.09066 0.1221 0.03879 0.06104 0.01056 0.03052 0.001345 0.01526 8.061e-05

7.5 Levenberg–Marquardt Method

203

The major advantages of the Levenberg–Marquardt method are its simplicity, the descent property, and excellent convergence rate provided the starting point x (0) is nearby the solution x ∗ , and absence of line search for computing step length αk . The major disadvantages are the requirement of calculating the Hessian Q and solve the set of linear equations using (7.21).

7.6 Case Study: Optimal Design of a Pi-Electric Circuit The circuit in Fig. 7.6 represents the interstage coupling of a tuned amplifier, see (Pierre 1986). This circuit is used as an output network of automatically tuned transmitters in which the tuning capacitor C1 and the loading capacitor C2 are adjusted with semi-independent control loops. We have the following objectives for the given electric circuit: 1. To obtain 180◦ phase shift between I1 ( jw) and the voltage across C1 with given carrier frequency ω. 2. To obtain a certain level of power output to an antenna system. The objective is to find values of C1 and C2 so that maximum power is delivered to R2 . Let the power output across R2 be denoted by P0 in the following form: P0 =

|V0 ( jw)|2 . R2

(7.22)

Let τ1 = R1 C1 and τ2 = R2 C2 . From circuit analysis, we have V0 (s) =

s 3 (Lτ1 τ1 )

+

s 2 L(τ1

−R1 R2 I1 (s) , + τ2 ) + s(L + R1 τ2 + R2 τ1 ) + R1 + R2

(7.23)

which gives

Fig. 7.6 Pi-electric circuit

L

I1 ( j )

R1

C1

C2

R2

V0

204

7 Newton’s Method

|V0 ( jw)| = 

I1 R1 Re2 + Im2

,

where I1 ≡ |I1 ( jw)|, R1 w2 Lτ1 w2 Lτ2 − − , R2 R2 R2 wL R1 wτ2 w3 Lτ1 τ2 Im{V0 ( jw)} = + + wτ1 − . R2 R2 R2

Re{V0 ( jw)} = 1 +

To find P0 , one has to maximize V0 or minimize f , such that f = [Re{Vo (jw)}]2 + [Im{Vo (jw)}]2 .

(7.24)

with the following dimensionless parameters, R1 , R2 wL b= , R2 x1 = wτ1 , a=

x2 = wτ2 , the function to be minimized becomes f (x1 , x2 ) = (1 + a − bx1 − bx2 )2 + (b + ax2 + x1 − bx1 x2 )2 .

(7.25)

The values of a and b must be specified prior to apply any method to minimize the above function. If a = 10 and b = 1, the objective function becomes f (x1 , x2 ) = (11 − x1 − x2 )2 + (1 + 10x2 + x1 − x1 x2 )2 .

(7.26)

We say (7.26) as electric-Pi circuit (EPC) function. This gives the approximate measure of performance for the circuit diagram shown in Fig. 7.6. The R function func_var(x) is written for EPC function as func_var Newton(x = c(5,-3), epsilon = 1e-7, maxiter = 100) Initial function value: 162.0000 k 0 1 2 3 4 5 6 7

x 5.0000 5.9825 5.6822 7.1997 6.9823 6.9995 7.0000 7.0000

-3.0000 -2.0568 -1.0888 -1.8190 -2.0248 -2.0000 -2.0000 -2.0000

f(x) 162.0000 51.6844 44.9686 41.2230 40.0164 40.0000 40.0000 40.0000

||gradient|| 140.5845 32.8654 6.2470 8.7877 1.0939 0.01128 1.164e-06 3.553e-12

Minimum point: 7.0000 -2.0000 Minimum function value: 40.0000

See Fig. 7.7 On executing function Modified_Newton(x, epsilon, maxiter) with x = c(-5, 3), epsilon = 1e-7, and maxiter = 100 on R Console, the following output is displayed for the EPC objective function:

7 Newton’s Method

−2 −4

−3

x2

−1

0

206

4

5

6

7

8

x1 Fig. 7.8 Graph of (11 − x1 − x2 )2 + (1 + 10x2 + x1 − x1 x2 )2 using Modified_Newton()

> Modified_Newton(x=c(5, -3), epsilon = 1e-7,maxiter = 100) Initial function value: 1850.0000 k 0 1 2 3 4 5 6

x -5.0000 8.5835 4.6686 6.6162 6.6948 6.9960 6.9999

3.0000 0.6079 -0.5222 -1.1987 -1.7839 -1.9857 -1.9999

f(x) 1850.0000 112.3615 55.2926 43.8377 40.3122 40.0013 40.0000

||gradient|| 1218.8995 26.3715 17.7487 13.6845 2.1830 0.2691 0.0007897

Minimum point: 6.9999 -1.9999 Minimum function value: 40.0000

See Fig. 7.8 On executing function Levenberg_Marquardt(x, mu, epsilon, maxiter) with x=c(5, -3), mu = 1e3, epsilon=1e-7, and maxiter=100 on R Console, the following output is displayed for the EPC function:

207

−2 −4

−3

x2

−1

0

7.6 Case Study: Optimal Design of a Pi-Electric Circuit

4

5

6

7

8

x1 Fig. 7.9 Graph of (11 − x1 − x2 )2 + (1 + 10x2 + x1 − x1 x2 )2 using Levenberg_Marquardt()

Initial function value: 162.0000 k 0 1 2 3 4 5 6 7 8 9 10 11 12 13

x 5.0000 5.0814 5.2170 5.4167 5.6668 5.9341 6.1946 6.4458 6.6764 6.8535 6.9536 6.9908 6.9990 6.9999

-3.0000 -2.9020 -2.7375 -2.4939 -2.1916 -1.8963 -1.6977 -1.6520 -1.7335 -1.8588 -1.9516 -1.9903 -1.9990 -1.9999

f(x) 162.0000 144.9198 119.8825 90.3767 64.6298 49.0752 42.8285 40.9952 40.3555 40.0828 40.0091 40.0004 40.0000 40.0000

mu 1000.0000 500.0000 250.0000 125.0000 62.5000 31.2500 15.6250 7.8125 3.9062 1.9531 0.9766 0.4883 0.2441 0.1221

||gradient|| 140.5845 127.6877 107.4400 80.4864 51.5992 27.3647 11.6953 4.2426 1.7568 0.8165 0.2778 0.05495 0.00588 0.0003359

The minimum point: 7.0000 -2.0000 The minimum function value: 40.0000

See Fig. 7.9 The EPC function is an appropriate measure of performance for the above circuit diagram. The power P0 is maximized when f = −40.

208

7 Newton’s Method

7.7 Exercises 1. Apply the Newton’s method to minimize the function f (x) = 3x1 + x2 +  0 6x1 x2 − 2x12 + 2x22 with a starting point x (0) = using R. 0 2 4 2. Apply the Newton’s method ⎡ to⎤minimize f (x) = (x1 − x2 ) + (x1 − 2x3 ) with −2.6 a starting point x (0) = ⎣ 2.2 ⎦ using R. 2 )2 + 3. Apply the Newton’s method to minimize the function f (x) = (x1 − 10x2⎡ ⎤ 1 ⎢2⎥ ⎥ 4.5(x3 − x4 )2 + (x2 − 2x3 )4 + 10(x1 − x4 )4 with a starting point x (0) = ⎢ ⎣3⎦ 4 using R. 4 4. Apply the Newton’s Method to minimize the function f (x) = 5x14 + 5.7x  2− 1 6x12 + 2x1 x2 + 5x22 + 15x1 − 7x2 + 13 starting from the point x (0) = using 1 R. 2 x1 5. Apply the Newton’s method to minimize   the function f (x1 , x2 ) = x1 e + 1 x22 e x2 + 2 with a starting point x (0) = using R. 1 x2

6. Apply the Newton’s method to minimize the function f (x) = 7 + 21 + x1 x2 +   2 6x2 from a starting point x = using R. 2 7. Let {d (k) } be the sequence of search directions generated by Newton’s method for minimizing f (x) = x13 + x1 x2 + x2 . What will be the starting point x (0) for which the search direction on d (0) from x (0) to x (1) is a descent direction? 8. Apply the Newton’s method to minimize the Cubetest function f (x) =  −1.2 100(x13 − x2 )2 + (1 − x1 )2 with a starting point x (0) = using R. 1 9. Apply modified Newton’s method to minimize the Powell’s quadratic func10x2 ⎤ )2 + 5(x3 − x4 )2 + (x2 − 2x3 )4 + tion (Fletcher et al. 1987) f (x) = (x1 + ⎡ 3 ⎢−1⎥ 4 (0) ⎢ 10(x1 − x4 ) with starting point x = ⎣ ⎥ using R. 0⎦ 1 10. Apply modified Newton’s method to minimize the Shalow function   (Pierre 1986) −2 using R. f (x) = (x12 − x2 )2 + (1 − x1 )2 with starting point x (0) = −2

7.7 Exercises

209

11. Apply modified Newton’s method to minimize the function f (x) = (x1 + 2 2 2 2 2 (0) 10x ⎡ ⎤1 x2 + 100x1 x2 x3 ) + (1 − x1 + 50x2 − 100x3 ) with starting point x = 1 ⎣2⎦ using R. 3 12. Apply modified Newton’s method to minimize the Wood’s function (More et al. 1981) f (x) = 100(x2 − x12 )2 + (1 − x1 )2 + 90(x4 − x32 )2 + (1 − x3 )2 + 10(x2 + x4 − 2)2 + ⎡ ⎤ −3 ⎢−1⎥ 2 (0) ⎥ 0.1(x2 − x4 ) with starting point x = ⎢ ⎣−3⎦ using R. −1 13. Apply modified Newton’s method to minimize the NONSCOMP CUTE function ⎡ ⎤ 3 f (x) = (x1 − 1)2 + 4(x2 − x12 )2 + 4(x3 − x22 )2 with starting point x (0) = ⎣3⎦ 3 using R. 14. Apply the Levenberg–Marquardt method to minimize the Tridiagonal Double Borded function f (x) = (x1 − 1)2 + (x1 − 21 x12 + 21 x22 )2 + (x2 − 21 x22 + 21 x32 )2 ⎡ ⎤ 1 with a starting point x (0) = ⎣1⎦ using R. 1 15. Apply the Levenberg–Marquardt method to minimize the Himmelblau func2 2 2 2 tion (Rao  1996) f (x) = (x1 + x2 − 11) + (x1 + x2 − 7) with starting point 0 x (0) = using R. 0 16. Apply the Levenberg–Marquardt method to minimize the Brown’s badly scaled 6 2 −6 2 function (Moré et al. 1981) f (x)  = (x1 − 10 ) + (x2 − 2 × 10 ) + (x1 x2 − 1 2)2 with starting point x (0) = using R. 1 17. Apply the Levenberg–Marquardt method to minimize the Rosenbrock’s parabolic valley function (Moré et al. 1981) f (x) = 100(x2 − x1 )2 + (1 − x1 )2  −1.2 with starting point x (0) = using R. 1.0 2 18. Apply the Levenberg–Marquardt method to minimize   the function f (x) = x1 + 0 x22 − 3x1 − x1 x2 + 2 with starting point x (0) = using R. 0 19. Apply the Levenberg–Marquardt method to minimize   the function f (x) = −3 2 2 (0) using R. 3x1 + 3x2 − 3x1 + 4x2 with starting point x = 1 20. Apply the Levenberg–Marquardt method to minimize the function f (x) = 2 2 2 with starting point (x1 − x⎡ 2 +⎤x 3 ) + (−x 1 + x 2 + x 3 ) + (x 1 + x 2 − x 3 ) 0 x (0) = ⎣0⎦ using R. 0

Chapter 8

Conjugate Gradient Methods

8.1 Introduction Our interest in the conjugate gradient methods is twofold. First, they are among the most useful techniques to solve a large system of linear equations. Second, they can be adopted to solve large nonlinear optimization problems. In the previous chapters, we studied two important methods for finding a minimum point of real-valued functions of n real variables, namely, the steepest descent method and Newton’s method. The steepest descent method is easy to apply. However, convergence is often very slow. On the other hand, Newton’s algorithm normally has rapid convergence but involves considerable computation at each step. Recall Newton’s method, which involves the computation of Hessian of the function at every iteration. It is always required to reserve space for storing n × n Hessian to run this algorithm. Also, Newton’s method does not choose n suitable directions for n number of variables of the function. If the inverse of Hessian is not available, then Newton’s method fails to find the minimum point. These drawbacks are the central theme in the development of an important class of minimization algorithms, what so-called the conjugate direction algorithm. It uses the history of the previous iteration for creating new search directions. The conjugate direction method acts as an intermediate method between the steepest descent method and Newton’s method.

8.2 Basics of Conjugate Direction We study the conjugate direction algorithm for minimizing the quadratic function of n variables. Consider the quadratic function in the following form: f (x) =

1 T x Qx − x T b, 2

© Springer Nature Singapore Pte Ltd. 2019 S. K. Mishra and B. Ram, Introduction to Unconstrained Optimization with R, https://doi.org/10.1007/978-981-15-0894-3_8

(8.1)

211

212

8 Conjugate Gradient Methods

where Q = Q T > 0 is n × n real symmetric and positive definite matrix, that is, x T Qx > 0 for all x ∈ Rn or equivalently all the eigenvalues of the matrix Q is positive, and b ∈ Rn . The point x ∗ is the minimizer of the function f (x) at which the gradient of the function is equal to zero. We can express it as ∇ f (x ∗ ) = Qx ∗ − b = 0, or Qx ∗ = b.

(8.2)

This shows that the same point x ∗ that minimizes f (x) also serves as the solution to the linear system of equations. Assume that the set of search directions exists in advance and such directions are called Q−conjugate directions. These Q−conjugate directions follow the property of orthogonality. Definition 8.1 (Q−Conjugate Directions) Let Q be a real symmetric n × n matrix such that Q = Q T > 0. Then, the directions d (0) , d (1) , d (2) , . . . , d (m) are Q−conjugate directions if, for all i = j, we have (d (i) )T Qd ( j) = 0. We can formulate the basic conjugate direction algorithm as follows: choose an initial point x (0) , and the Q−conjugate directions d (0) , d (1) , . . . , d (n−1) are given for minimizing the objective function (8.1). Then, we start from k ≥ 0, to compute g (k) = ∇ f (x (k) ) = Qx (k) − b,

(8.3)

x (k+1) = x (k) + αk d (k) ,

(8.4)

(g (k) )T d (k) . (d (k) )T Qd (k)

(8.5)

and

where αk = −

Remark 8.1 The conjugate direction methods perform better than the method of steepest descent but not as well as the Newton’s method. We now show that the Q−conjugate directions d (0) , d (1) , . . . , d (k) form a set of linearly independent vectors. Lemma 8.1 Let Q = Q T > 0 be a symmetric positive definite n × n matrix. If the directions d (0) , d (1) , . . . , d (k) ∈ Rn , where k ≤ n − 1 are nonzero and Q−conjugate, then they are linearly independent.

8.2 Basics of Conjugate Direction

213

Proof Let α0 , α1 , . . . , αk be scalars such that α0 d (0) + α1 d (1) + · · · + αk d (k) = 0.

(8.6)

In order to show that d (0)) , d (1) , . . . , d (k) are linearly independent, we need to show that α0 = α1 = · · · = αk = 0. Premultiplying (8.6) by (d ( j) )T Q, where 0 ≤ j ≤ k, we get α0 (d ( j) )T Qd (0) + α1 (d ( j) )T Qd (1) + · · · + α j−1 (d ( j) )T Qd ( j−1) + α j (d ( j) )T Qd ( j) + α j+1 (d ( j) )T Qd ( j+1) + · · · + αk (d ( j) )T Qd (k) = 0. Since (d ( j) )T Qd (i) = 0 for all i = j, then we obtain α j (d ( j) )T Qd ( j) = 0.

(8.7)

Since Q = Q T > 0 and d ( j) = 0, where 0 ≤ j ≤ k, it leads to α j = 0, that is, α0 = 0, α1 = 0, . . . , αk = 0. Thus, search directions d (0) , d (1) , . . . , d (k) are linearly independent. This completes the proof.  ⎡ ⎤ 301 Example 8.1 Let Q = ⎣0 3 1⎦ . We construct a set of Q−conjugate directions 112 (0) (1) (2) d , d , and d . Note that Q = Q T > 0. We see that all its leading minors are positive, that is, ∇1 = 3 > 0,   30 ∇2 = det = 9 > 0, 03 and ∇3 = det Q = 12 > 0.

214

8 Conjugate Gradient Methods

Thus, Q is a positive definite matrix. Our objective is to construct a set of Q−conjugate ⎡ ⎤ ⎡ (1) ⎤ ⎡ (2) ⎤ d1 d1 1 (1) (0) (1) (2) (0) (1) (2) ⎣ ⎦ ⎣ ⎣ ⎦ vectors d , d , and d . Let d = 0 , d = d2 , and d = d2(2) ⎦. We 0 d3(1) d3(2) (0) T (1) have to show (d ) Qd = 0. (d (0) )T Qd (1)

⎡ ⎤ ⎡ (1) ⎤ ⎡ (1) ⎤ d1  301  d1(1) (1) = 1 0 0 ⎣0 3 1⎦ ⎣d2 ⎦ = 3 0 1 ⎣d2 ⎦ = 3d1(1) + d3(1) . 112 d3(1) d3(1) ⎡

⎤ 1 If d1(1) = 1, then d3(1) = −3 and d (1) = ⎣ 0 ⎦ , and thus (d (0) )T Qd (1) = 0. To find −3 the third direction d (2) , we need to show (d (0) )T Qd (2) = 0 and (d (1) )T Qd (2) = 0. We calculate (d (0) )T Qd (2)

⎡ ⎤ ⎡ (2) ⎤ ⎡ 2⎤ d1 d1  301  = 1 0 0 ⎣0 3 1⎦ ⎣d2(2) ⎦ = 3 0 1 ⎣d22 ⎦ = 3d12 + d32 . 112 d32 d3(2)

That is, 3d1(2) + d3(2) = 0.

(8.8)

If (d (1) )T Qd (2) = 0, then ⎡ (2) ⎤ ⎤ ⎡ (2) ⎤ d1 d 3 0 1  ⎢ 1 ⎥ ⎢ ⎥  = 1 0 −3 ⎣0 3 1⎦ ⎣d2(2) ⎦ = 3 − 3 −3 1 − 6 ⎣d2(2) ⎦ = −3d2(2) − 5d3(2) , 112 d3(2) d3(2) ⎡

(d (1) )T Qd (2)

that is, −3d2(2) − 5d3(2) = 0.

(8.9)

Take d1(2) = λ, that is, d3(2) = −3λ and d2(2) = 5λ. We have ⎡

d (2)

in particular. Thus, d (0) vectors.

⎤ ⎡ ⎤ λ 1 = ⎣−3λ⎦ = ⎣−3⎦ 5λ 5

⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 1 1 1 = ⎣0⎦ , d (1) = ⎣ 0 ⎦, and d (2) = ⎣−3⎦ are Q−conjugate 0 −3 −5

8.2 Basics of Conjugate Direction

215

The basic idea is to create a set of n linearly independent search directions and perform a series of unidirectional searches along each of these search directions, starting each time from the previous best point. The basic problem before us is to develop computational procedures for constructing conjugate vectors. If we choose the conjugate vectors d (k) carefully, then we may not need all of them to obtain a good approximation to the solution x ∗ . So, we want to regard the conjugate direction method as an iterative method. This also allows us to approximately solve the systems where n is so large that the direct method would take too much time. We present the following conjugate direction algorithm on the basis of the above theory. Algorithm 8.1 (Conjugate Direction Algorithm) 1: 2: 3: 4: 5: 6:

Choose x (0) ∈ Rn , ε Choose d (0) , d (1) , . . . , d (n−1) Set k ← 0 for k to n − 1 do Compute g (k) ← ∇ f (x (k) ) (k) T d (k) Compute αk ← − (d(g(k) )T) Qd (k)

 x (0) is a starting point, ε is a tolerance value  Q−conjugate directions  k is number of iterations of method  gradient of f at x (k)

7: Update x (k+1) ← x (k) + αk d (k) . 8: if ( f (x (k+1) ) − f (x (k) )) < ε or ∇ f (x (k+1) ) < ε then 9: Converged. Print x ∗ ← x (k+1) and f (x ∗ ) ← f (x (k+1) ) 10: end if 11: end for

 a new point  stopping criteria  best solution

Example 8.2 Apply the conjugate direction method to minimize the function   f (x) = 1 x12 + 2x22 + x1 − x2 + 1 along with the Q−conjugate directions d (0) = , d (1) = 0     0 0 , and a starting point x (0) = . 1 0 We write the function f (x) in the matrix form as f (x) = where

1 T x Qx − x T b + c, 2



   20 −1 Q= , b= , and c = 1. 04 1

The gradient of f at x (0) is g (0) = Qx (0) − b = and



20 04

      0 −1 1 − = , 0 1 −1

216

8 Conjugate Gradient Methods

(g (0) )T d (0) (d (0) )T Qd (0)    1 1 −1 0    =−  20 1 10 04 0 1 =− . 2

α0 = −

Therefore, x

(1)

=x

(0)

+ α0 d

(0)

      1 1 −1/2 0 . = = − 0 0 2 0

To find x (2) , we need to compute the following quantities: g

(1)

= Qx

(1)



20 −b = 04

          −1 −1/2 −1 −1 0 − = − = , 0 1 0 1 −1

and

α1 = −

(g (1) )T d (1) (d (1) )T Qd (1)

   0 0 −1 1    =−  20 0 01 04 1 1 −1  = . =−  0 4 04 1

Therefore, x (2) = x (1) + α1 d (1) =

      1 0 −1/2 −1/2 . + = 1 0 /4 4 1

The gradient of f at x (2) is g (2) = Qx (2) − b =



20 04



         −1 −1 −1/2 −1 0 − = − = . 1/4 1 1 1 0

Thus, the conjugate direction algorithm stops as the gradient is zero. Since f is a quadratic function of two variables, therefore it converges to the minimum point  −1/2 (2) ∗ and the minimum value of the objective function f (x ∗ ) = 58 in x =x = 1 /4 two iterations.

8.2 Basics of Conjugate Direction

217

Remark 8.2 The conjugate direction algorithm requires Q−conjugate directions to minimize any nonlinear function, which is very cumbersome and not very useful from a practical point of view, therefore, we are not providing the R function for this method.

8.3 Convergence Analysis of Conjugate Direction Method Theorem 8.1 For any starting point x (0) , the basic conjugate direction algorithm converges to a unique point x ∗ , that is, Qx ∗ = b in exactly n steps, that is, x (n) = x ∗ . Proof Note that

x ∗ − x (0) ∈ Rn ,

and since d (0) , d (1) , . . . , d (n−1) are linearly independent, then there exist constants βi , where i = 0, 1, 2, . . . , n − 1 such that x ∗ − x (0) = β0 d (0) + β1 d (1) + · · · + βn−1 d (n−1) .

(8.10)

Premultiplying (8.10) by (d (k) )T Q, where 0 ≤ k ≤ n − 1, we get (d (k) )T Q(x ∗ − x (0) ) = β0 (d (k) )T Qd (0) + β1 (d (k) )T Qd (1) + · · · + βk−1 (d (k) )T Qd (k−1) + βk (d (k) )T Qd (k) + βk+1 (d (k) )T Qd (k+1) + · · · + βn−1 (d (k) )T Qd (n−1) .

Since (d (k) )T Qd (i) = 0 f or i = k, therefore, (d (k) )T Q(x ∗ − x (0) ) = βk (d (k) )T Qd (k) . That is, βk =

(d (k) )T Q(x ∗ − x (0) ) . (d (k) )T Qd (k)

Since x (k) = x (k−1) + αk−1 d (k−1) , = x (k−2) + αk−2 d (k−2) + αk−1 d (k−1) , .. . so on, therefore

(8.11)

218

8 Conjugate Gradient Methods

x (k) = x (0) + α0 d (0) + α1 d (1) + · · · + αk−1 d (k−1) , or x (k) − x (0) = α0 d (0) + α1 d (1) + · · · + αk−1 d (k−1) . We can write

(8.12)

x ∗ − x (0) = x ∗ − x (k) + x (k) − x (0) ,

that is, x ∗ − x (0) = (x ∗ − x (k) ) + (x (k) − x (0) ).

(8.13)

Premultiplying (8.13) by (d (k) )T Q, we get (d (k) )T Q(x ∗ − x (0) ) = (d (k) )T Q(x ∗ − x (k) ) + (d (k) )T Q(x (k) − x (0) ). The above equality along with (8.12) gives (d (k) )T Q(x ∗ − x (0) ) = (d (k) )T Q(x ∗ − x (k) ) + (d (k) )T Q[α0 d (0) + α1 d (1) + · · · + αk−1 d (k−1) ].

(8.14)

s

Since d (i) are Q−conjugate, therefore (d (k) )T Q[α0 d (0) + α1 d (1) + · · · + αk−1 d (k−1) ] = 0.

(8.15)

Using (8.15) in (8.14), we get (d (k) )T Q(x ∗ − x (0) ) = (d (k) )T Qx ∗ − (d (k) )T Qx (k) . Since Qx ∗ = b, then (d (k) )T Q(x ∗ − x (0) ) = (d (k) )T b − (d (k) )T Qx (k) , that is, (d (k) )T Q(x ∗ − x (0) ) = (d (k) )T [b − Qx (k) ].

(8.16)

Since b − Qx (k) = −g (k) , then (d (k) )T Q(x ∗ − x (0) ) = −(d (k) )T g (k) . From (8.11) and (8.17), we have

(8.17)

8.3 Convergence Analysis of Conjugate Direction Method

βk = −

219

(d (k) )T g (k) . (d (k) )T Qd (k)

(8.18)

(d (k) )T g (k) . (d (k) )T Qd (k)

(8.19)

We can also write (8.18) for αk as αk = − From (8.10) and (8.19), we get x ∗ − x (0) = α0 d (0) + α1 d (1) + · · · + αn−1 d (n−1) , that is, x ∗ = x (0) + α0 d (0) + α1 d (1) + · · · + αn−1 d (n−1) . Using (8.12) in the above equality, we get x ∗ = x (0) + x (n) − x (0) . Thus, x ∗ = x (n) . 

This completes the proof.

Note that the Q−conjugate directions are available in advance to find the minimum point of a quadratic function of n variables in n steps. The following result holds stronger condition for conjugate direction method. Lemma 8.2 In the conjugate direction algorithm, (g (k+1) )T d (i) = 0 for all k, where 0 ≤ k ≤ n − 1 and 0 ≤ i ≤ k. Proof Note that Q(x (k+1) − x (k) ) = Qx (k+1) − Qx (k) = (Qx (k+1) − b) − (Qx (k) − b). Since x (k+1) = x (k) + αk d (k) , Qx (k+1) − b = g (k+1) , and Qx (k) − b = g (k) , we have Q(x (k) + αk d (k) − x (k) ) = g (k+1) − g (k) , that is, αk Qd (k) = g (k+1) − g (k) , which implies

220

8 Conjugate Gradient Methods

g (k+1) = g (k) + αk Qd (k) .

(8.20)

We prove this Lemma by mathematical induction. The result is true for k = 0 : (g (1) )T d (0) = (Qx (1) − b)T d (0) = (Q(x (0) + α0 d (0) ) − b)T d (0) = (Qx (0) + α0 Qd (0) − b)T d (0) , that is, (g (1) )T d (0) = (x (0) )T Qd (0) + α0 (d (0) )T Qd (0) − b T d (0) . (0) T

(8.21)

(0)

d Since α0 = − (d(g(0) )T) Qd (0) , therefore

(g (1) )T d (0) = (x (0) )T Qd (0) −

(g (0) )T d (0) (d (0) )T Qd (0) − b T d (0) , (d (0) )T Qd (0)

that is, (g (1) )T d (0) = (Qx (0) − b)T d (0) − (g (0) )T d (0) .

(8.22)

Since Qx (0) − b = g (0) , then (g (1) )T d (0) = (g (0) )T d (0) − (g (0) )T d (0) , that is, (g (1) )T d (0) = 0.

(8.23)

Assume that the result is true for k − 1, that is, (g (k) )T d (i) = 0, where 0 ≤ i ≤ k − 1. Then, we have to prove that the result is true for k, that is, (g (k+1) )T d (i) = 0, where 0 ≤ i ≤ k. For 0 < i < k, we know that g (k+1) = g (k) + αk Qd (k) .

(8.24)

Taking transpose on both sides and post-multiplying by d (i) in (8.24), we get (g (k+1) )T d (i) = (g (k) )T d (i) + αk (d (k) )T Qd (i) . By Q−conjugancy, (d (k) )T Qd (i) = 0 for k = i, and by mathematical induction hypothesis, (g (k) )T d (i) = 0. Therefore,

8.3 Convergence Analysis of Conjugate Direction Method

221

(g (k+1) )T d (i) = 0, ∀ 0 < i < k. For i = k, we have

(g (k+1) )T d (k) = (Qx (k+1) − b)T d (k) ,

that is, (g (k+1) )T d (k) = ((x (k+1) )T Q − b T )d (k) = (x (k+1) )T Qd (k) − b T d (k) . Since (x (k+1) )T = (x (k) + αk d (k) )T , then (g (k+1) )T d (k) = (x (k) + αk d (k) )T Qd (k) − b T d (k) . (k) T

(8.25)

(k)

d Since αk = − (d(g(k) )T) Qd (k) , then

(g (k) )T d (k) (k) T (g (k+1) )T d (k) = x (k) − (k) T d Qd (k) − b T d (k) , (d ) Qd (k) that is, (g (k) )T d (k) (d (k) )T Qd (k) − b T d (k) (d (k) )T Qd (k) = (Qx (k) − b)T d (k) − (g (k) )T d (k) .

(g (k+1) )T d (k) = (x (k) )T Qd (k) −

Since Qx (k) − b = g (k) , we have (g (k+1) )T d (k) = (g (k) )T d (k) − (g (k) )T d (k) , that is, (g (k+1) )T d (k) = 0. Therefore, by mathematical induction, for all 0 ≤ k ≤ n − 1 and 0 ≤ i ≤ k, (g (k+1) )T d (i) = 0. This completes the proof.



The conjugate direction method is very effective. However, we need to specify the Q−conjugate directions to use this algorithm. Fortunately, there is a method to generate Q−conjugate directions as we perform iterations. In the next section, we discuss such an algorithm that incorporates the generation of Q−conjugate directions.

222

8 Conjugate Gradient Methods

8.4 Method of Conjugate Gradient The conjugate gradient method was introduced by Hestenes and Stiefel in 1952. They presented the conjugate gradient parameter to solve the system of linear equations with a real symmetric positive definite matrix. The conjugate gradient method is the modification of the steepest descent method. The conjugate gradient method uses conjugate directions to minimize a quadratic function, but the steepest descent method uses local gradients to minimize the quadratic function. The steepest descent method may not be efficient because it gets the zig-zag pattern and repeats the same search directions many times, but the conjugate gradient method does not repeat any previous search direction and converges in n iterations for n variables of quadratic functions. In contrast, no such property exists with the steepest descent method. Therefore, due to its simplicity and low memory management, the conjugate gradient method is recognized as one of the best methods in the field of unconstrained optimization. Consider the quadratic function f (x) =

1 T x Qx − x T b, 2

(8.26)

where Q = Q T > 0 is an n × n real symmetric positive definite matrix, and x, b ∈ Rn . A solution to problem f (x) always exists, and it is necessarily unique. We can find the minimizer of the quadratic function of n variables by performing n searches along mutually conjugate directions. Our first search direction from the starting point x (0) is in the direction of the steepest descent, that is, d (0) = −g (0) . Thus, x (1) is generated as x (1) = x (0) + α0 d (0) , where α0 = arg min f (x (0) + αd (0) ) = − α≥0

(8.27) (g (0) )T d (0) (d (0) )T Qd (0)

is the step length. In the next step, we search a direction d (1) which is Q−conjugate to d (0) . We choose d (1) as a linear combination of g (1) and d (0) that can be expressed as d (1) = −g (1) + β0 d (0) . In general, at (k + 1)th step, we have x (k+1) = x (k) + αk d (k) ,

(8.28)

8.4 Method of Conjugate Gradient

223

where αk = −

(g (k) )T d (k) , (d (k) )T Qd (k)

(8.29)

and we choose d (k+1) as a linear combination of g (k+1) and d (k) , which can be presented as d (k+1) = −g (k+1) + βk d (k) , k = 0, 1, 2, . . . .

(8.30)

The coefficient βk , where k = 1, 2, . . . are chosen in such a way that d (k+1) is Q−conjugate to d (0) , d (1) , . . . , d (k) . This is accomplished by choosing βk as follows: βk =

(g (k+1) )T Qd (k) . (d (k) )T Qd (k)

(8.31)

Conjugate gradient algorithm has a significant advantage due to low storage capacity over most of the other algorithms covered in this book, but the method of the steepest descent is an exceptional case. The conjugate gradient algorithm does not use prespecified conjugate directions, rather computes the directions as the algorithm progresses. Note that at each stage, the direction is calculated as a linear combination of the previous direction and the current gradient, in such a way that all the directions are mutually Q−conjugate. Therefore, this method is termed as the conjugate gradient method. On the basis of the above theory, we present the conjugate gradient algorithm. Algorithm 8.2 (Conjugate Gradient Algorithm) 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13:

Choose x (0) ∈ Rn , ε  x (0) is a starting point, ε is a tolerance value Set k ← 0  k is number of iterations of method Compute g (k) ← ∇ f (x (k) )  gradient of f at x (k) (k) (k) Set d ← −g  descent direction Compute Q  Hessian of f at x (k) for k to maxiter do  maxiter is maximum number of iterations to perform (k) T d (k) Compute αk ← arg minα≥0 f (x (k) + αd k )  use Golden_Line() or αk ← − (d(g(k) )T) Qd (k)

Update x (k+1) ← x (k) + αk d (k)  a new point if ( f (x (k+1) ) − f (x (k) )) < ε or ∇ f (x (k) ) < ε then  stopping criteria Converged. Print x ∗ ← x (k+1) and f (x ∗ ) ← f (x (k+1) )  best solution end if Compute g (k+1) ← ∇ f (x (k+1) ) (k+1) T Qd (k)  βk is chosen such that d (k) is Q−conjugate with d (k+1) Compute βk ← (g(d (k) )T) Qd (k)

14: Compute d (k+1) ← −g (k+1) + βk d (k) 15: end for

 a new search direction

224

8 Conjugate Gradient Methods

Example 8.3 Apply the conjugate gradient method to minimize the function f (x) = x1 − x2 + 2x12 + 2x1 x2 + x22 with a starting point x

(0)

  0 = . 0

The gradient and Hessian of f are  1 + 4x1 + 2x2 , ∇ f (x) = −1 + 2x1 + 2x2 

and

 F(x) =

 42 . 22

The gradient and Hessian of f at x (0) are given as   1 (0) (0) g = ∇ f (x ) = , −1 and Q = F(x

(0)



 42 )= . 22

The descent direction d (0) is computed as d (0) = −g (0) =

  −1 . 1

Since f is a quadratic function, thus the step length α0 is computed as    −1 1 −1 1 (g (0) )T d (0) −2 2    = −   = = 1. = − α0 = − (0) T (0)  4 2 −1  −1 (d ) Qd 2 −1 1 −2 0 22 1 1 Therefore, x (1) = x (0) + α0 d (0) =

      0 −1 −1 +1× = . 0 1 1

The gradient of f at x (1) is g (1) = ∇ f (x (1) ) =



   1−4+2 −1 = . −1 − 2 + 2 −1

8.4 Method of Conjugate Gradient

β0 =

(g (1) )T Qd (0) (d (0) )T Qd (0)

225

  4 −3 0 2  =  4 −1 1 2

     −1 −1 −6 −4 1 1 2   =   = = 1.  2 2 −1 −1 −2 0 2 1 1 2 2

We now compute d

(1)

= −g

(1)

+ β0 d

(0)



         −1 −1 1 −1 0 =− +1 = + = . −1 1 1 −1 2

Therefore,

α1 = −

(g (1) )T d (1) (d (1) )T Qd (1)

   0 −1 −1 2 1 2 2    = −  = = , =−  42 0  0 8 4 02 44 22 2 2

and x (2) = x (1) + α1 d (1) =

        1 0 −1 −1 −1 = 3 . = + /2 1 1 + 1/2 4 2

Note that g (2) = ∇ f (x (2) ) =



   1−4+3 0 = . −1 − 2 + 3 0

Therefore, the conjugate gradient algorithm stops as the gradient is zero. Since f is a −1 quadratic function of two variables, thus we get minimum point x (2) = x ∗ = 3 /2 5 ∗ and minimum value of the objective function f (x ) = − 4 in two iterations. The R function Conjugate_Gradient(x, epsilon, maxiter) for the conjugate gradient algorithm is given below in Function 8.1. R Function 8.1 Conjugate_Gradient.R 1 2 3 4 5 6 7 8 9 10 11 12 13

# Description : Conjugate Gradient Method # Theory : See S e c t i o n 8.4 and A l g o r i t h m 8.2 # Function Called : func _ var ( ) , G o l d e n _ Line ( ) , H e s s i a n ( ) # Input : x : S t a r t i n g Point # epsilon : T o l e r a n c e Value # maxiter : M a x i m u m N u m b e r of i t e r t i o n p e r f o r m e d # O u t p u t : xm : minimum point # fm : m i n i m u m f u n c t i o n value # E x e c u t e on R C o n s o l e : C o n j u g a t e _ G r a d i e n t (x , 1 e -7 , 100) C o n j u g a t e _ G r a d i e n t Conjugate_Gradient(x=c(1,-1,1),epsilon=1e-7,maxiter=100) Initial function value: -1.0000 k 0 1 2 3 4

1.0000 0.6070 0.5862 0.5717 0.5708

x -1.0000 -1.0000 -0.9161 -0.9402 -0.9396

1.0000 1.0000 0.9161 0.7691 0.7684

f(x) -1.0000 -1.8546 -1.8865 -1.9118 -1.9118

||gradient|| 5.0000 0.5558 0.4166 0.007012 0.0005239

Minimum point: 0.5708 -0.9396 0.7684 Minimum function value: -1.9118

We observe that the conjugate gradient algorithm converges to the minimum point in four iterations. Example 8.4 Apply conjugate gradient method to minimize the function f (x1 , x2 ) = x12 + 2x22 − 4x1 − 2x1 x2 with a starting point x

(0)

  1 = using R. 1

The R function func_var(x) is used to write the above objective in the R script as func_var Conjugate_Gradient(x=c(1,1), epsilon = 1e-7,maxiter = 100)

228

8 Conjugate Gradient Methods

−2

−1

0

x2

1

2

3

Fig. 8.1 Graph of x12 + 2x22 − 4x1 − 2x1 x2

−1

0

1

2

3

4

x1

Initial function value: -1.0000 k 0 1 2

1.0000 1.4933 3.6359

x 1.0000 0.3094 1.0908

f(x) -1.0000 -4.6507 -7.2727

||gradient|| 8.6023 2.3855 0.0006943

The minimum point: 3.6359 1.0908 The minimum function value: -7.2727

Proposition 8.1 In the conjugate gradient algorithm, the directions d (0) , d (1) , . . . , d (n−1) are Q−conjugate. Proof We use mathematical induction to prove this proposition. We first show that (d (0) )T Qd (1) = 0. Note that (d (0) )T Qd (1) = (d (0) )T Q[−g (1) + β0 d (0) ]. Since β0 =

(g (1) )T Qd (0) , (d (0) )T Qd (0)

then

 (g (1) )T Qd (0) (0)  (d (0) )T Qd (1) = (d (0) )T Q − g (1) + (0) T d (d ) Qd (0)   (g (1) )T Qd (0) (0) T (0) , = −(d (0) )T Qg (1) + (d ) Qd (d (0) )T Qd (0) that is,

(8.32)

8.4 Method of Conjugate Gradient

229

(d (0) )T Qd (1) = −(d (0) )T Qg (1) + (g (1) )T Qd (0) = −(d (0) )T Qg (1) + (d (0) )T Qg (1) = 0. Assume that the result is true for k < n − 1. Therefore, (d (k) )T Qd ( j) = 0, ∀ j = 1, . . . , k − 1, that is, d (0) , d (1) , . . . , d (k) are Q−conjugate. We shall prove the result for k + 1, that is, (d (k+1) )T Qd ( j) = 0, ∀ j = 0, 1, . . . , k. For that, we have to show (g (k+1) )T g ( j) = 0, ∀ j = 0, 1, . . . , k. For a fixed j ∈ {0, 1, . . . , k}, we have d ( j) = −g ( j) + β j−1 d ( j−1) . Therefore,   (g (k+1) )T d ( j) = (g (k+1) )T − g ( j) + β j−1 d ( j−1) = −(g (k+1) )T g ( j) + β j−1 (g (k+1) )T d ( j−1) .

From Lemma 8.2, we obtain (g (k+1) )T d (i) = 0, ∀ k, 0 ≤ k ≤ n − 1, 0 ≤ i ≤ k.

(8.33)

(g (k+1) )T g ( j) = 0, ∀ j = 0, 1, . . . , k.

(8.34)

Thus,

We first take j < k (d (k+1) )T Qd ( j) = [−g (k+1) + βk d (k) ]T Qd ( j) = −(g (k+1) )T Qd ( j) + βk (d (k) )T Qd ( j) . From mathematical induction, we have (d (k) )T Qd ( j) = 0, therefore, (d (k+1) )T Qd ( j) = −(g (k+1) )T Qd ( j) . Note that

(8.35)

230

8 Conjugate Gradient Methods

    Q x (k+1) − x (k) = Qx (k+1) − b − Qx (k) − b , that is,

  Q x (k) + αk d (k) − x (k) = g (k+1) − g (k) .

Then, we get

αk Qd (k) = g (k+1) − g (k) ,

which implies g (k+1) = g (k) + αk Qd (k) . Since g ( j+1) = g ( j) + α j Qd ( j) , therefore, Qd ( j) =

g ( j+1) − g ( j) . αj

Using (8.36) in (8.35) we get (d (k+1) )T Qd ( j) = −(g (k+1) )T

 g ( j+1) − g ( j) 

αj   1 (g (k+1) )T g ( j+1) − (g (k+1) )T g ( j) =− αj

Since (g (k+1) )T g ( j+1) = 0 and (g (k+1) )T g ( j) = 0, then (d (k+1) )T Qd ( j) = 0. We now take for j = k : (d (k+1) )T Qd (k) = [−g (k+1) + βk d (k) ]T Qd (k) (g (k+1) )T Qd (k) (k) T (d ) Qd (k) (d (k) )T Qd (k) = −(g (k+1) )T Qd (k) + (g (k+1) )T Qd (k) . = −(g (k+1) )T Qd (k) +

Thus,

(d (k+1) )T Qd (k) = 0.

(8.36)

8.4 Method of Conjugate Gradient

231



This completes the proof.

The disadvantage of the conjugate gradient algorithm is that Hessian must be reevaluated and stored at every iteration. To overcome this problem, we present several formulas that do not include the Hessian in the next section.

8.5 Method of Conjugate Gradient for General Functions We have seen that the conjugate gradient algorithm is itself a conjugate direction method. Therefore, the minimizer of the positive definite quadratic objective function of n variables requires n steps. The algorithm also works for general nonlinear functions by interpreting 1 f (x) = x T Qx − x T b 2 as a second-order Taylor series expansion. Such functions behave approximately as quadratically near the solution. For a quadratic function, the Hessian remains constant while finding the minimizer. However, for general nonlinear objective functions, the matrix is required to re-evaluate at every iteration while running the algorithm, which becomes computationally very expensive. To overcome this problem, we present an efficient implementation of the conjugate gradient algorithm that eliminates the Hessian Q at every iteration. Note that the Hessian Q appears in the computation of scalars αk and βk . But, we know that αk = arg min f (x (k) + αd (k) ), α≥0

which is a closed-form formula that can naturally be replaced by any line search methods. Thus, we need to eliminate Hessian Q from the formula of βk only. Then, the algorithm will only depend on the function and gradient values at each iteration. We now present three well-known modifications. 1. Hestenes–Stiefel Formula: Recall βk =

(g (k+1) )T Qd (k) . (d (k) )T Qd (k)

(8.37)

We have the following iterative method: x (k+1) = x (k) + αk d (k) . Premultiplying (8.38) by Q on both sides, we get Qx (k+1) = Qx (k) + Qαk d (k) , that is,

(8.38)

232

8 Conjugate Gradient Methods

Qx (k+1) − b = Qx (k) − b + Qαk d (k) ,

(8.39)

Since Qx (k+1) − b = g (k+1) and Qx (k) − b = g (k) , then g (k+1) = g (k) + Qαk d (k) ,

(8.40)

that is, Qd (k) =

g (k+1) − g (k) . αk

(8.41)

Using (8.41) in (8.37), we obtain  (k+1) (k)  (g (k+1) )T g α−g k  (k+1) (k)  , βk = g −g (k) T (d ) αk that is,   (g (k+1) )T g (k+1) − g (k)   . βk = (d (k) )T g (k+1) − g (k)

(8.42)

This is called Hestenes–Stiefel Formula. 2. Polak–Ribiere Formula: We start from the Hestenes–Stiefel Formula, that is,   (g (k+1) )T g (k+1) − g (k) βk = (k) T (k+1) . (d ) g − (d (k) )T g (k) From Lemma 8.2, we have (g (k+1) )T d (i) = 0 for all k, 0 ≤ k ≤ n − 1, 0 ≤ i ≤ k, thus (d (k) )T g (k+1) = 0.

(8.43)

Using (8.43) in Hestenes–Stiefel Formula, we get

βk =

  (g (k+1) )T g (k+1) − g (k) −(d (k) )T g (k)

.

(8.44)

We know that d (k) = −g (k) + βk−1 d (k−1) .

(8.45)

8.5 Method of Conjugate Gradient for General Functions

233

Premultiplying (8.45) by (g (k) )T on both sides, we get   (g (k) )T d (k) = (g (k) )T − g (k) + βk−1 d (k−1) . Thus, (g (k) )T d (k) = −(g (k) )T g (k) + βk−1 (g (k) )T d (k−1) .

(8.46)

Since from Lemma 8.2, therefore, (g (k) )T d (k−1) = 0, and thus, (g (k) )T d (k) = −(g (k) )T g (k) , that is, −(d (k) )T g (k) = (g (k) )T g (k)

(8.47)

Using (8.47) in (8.44), we get

βk =

  (g (k+1) )T g (k+1) − g (k) (g (k) )T g (k)

.

(8.48)

This is called Polak–Ribiere Formula. 3. Fletcher–Reeves Formula: We multiply out the numerator of Polak–Ribiere Formula as βk =

(g (k+1) )T g (k+1) − (g (k+1) )T g (k) . (g (k) )T g (k)

(8.49)

We know that d (k) = −g (k) + βk−1 d (k−1) . Premultiplying by (g (k+1) )T on both sides of the above equality, we get   (g (k+1) )T d (k) = (g (k+1) )T − g (k) + βk−1 d (k−1) , that is, (g (k+1) )T d (k) = −(g (k+1) )T g (k) + βk−1 (g (k+1) )T d (k−1) , Since from Lemma 8.2, we have (g (k+1) )T d (k) = 0, and (g (k+1) )T d (k−1) = 0, then we get

234

8 Conjugate Gradient Methods

(g (k+1) )T g k = 0.

(8.50)

Using (8.50) in (8.49), we get βk =

(g (k+1) )T g (k+1) . (g (k) )T g (k)

(8.51)

This is called the Fletcher–Reeves Formula. The Fletcher–Reeves method is a variation of the conjugate gradient method. Its main feature is step length αk for k = 0, 1, 2, . . . which are determined by minimizing f (x (k) + αk d (k) ) with respect to α using a line search, as in the case of the steepest descent or the Newton’s method. The main difference between this method and the steepest descent or Newton method is that d (k) is the conjugate direction with respect to d (k−1) , d (k−2) , . . . , d (0) rather than the steepest descent or Newton direction. An implementation of Fletcher–Reeves algorithm is presented here. Algorithm 8.3 (Fletcher–Reeves Algorithm) 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12:

Choose x (0) ∈ Rn , ε  x (0) is a starting point, ε is a tolerance value Set k ← 0  k is number of iterations of method Compute g (k) ← ∇ f (x (k) )  gradient of f at x (k) Set d (k) ← −g (k)  descent direction for k to maxiter do  maxiter is maximum number of iterations to perform (k) T d (k) Compute αk ← arg minα≥0 f (x (k) + αd k )  use Golden_Line() or αk ← − (d(g(k) )T) Qd (k) Update x (k+1) ← x (k) + αk d (k) . if ( f (x (k+1) ) − f (x (k) )) < ε or ∇ f (x (k) ) < ε. then Converged. Print x ∗ ← x (k+1) and f (x ∗ ) ← f (x (k+1) ). end if Compute g (k+1) ← ∇ f (x (k+1) ). (k+1) T (k+1) Compute βk ← (g (g(k) ))T gg(k) .

13: Compute d (k+1) ← −g (k+1) + βk d (k) . 14: end for

 a new point  stopping criteria  best solution

 Fletcher–Reeves formula  a new search direction

Example 8.5 Apply the Fletcher–Reeves method to minimize the function f (x) = (2x1 − x2 )2 + (x2 + 1)2  with a starting point x

(0)

=

5/2

2

 .

The gradient and Hessian of f are

8.5 Method of Conjugate Gradient for General Functions

235

 4(2x1 − x2 ) , 4x2 − 4x1 + 2   8 −4 F(x) = . −4 4 

∇ f (x) =

The gradient and Hessian of f at x (0) are   12 g (0) = ∇ f (x (0) ) = , 0   8 −4 . Q = F(x (0) ) = −4 4 The search direction is d (0) = −g (0) =

  −12 . 0

Since the objective function f is quadratic, thus the step length α0 is computed as   −12 12 0 0 1 −144 144    =−   = = . =−  8 −4 −12  −12 1152 8 −12 0 −96 48 −4 4 0 0 

α0 = −

(g (0) )T d (0) (d (0) )T Qd (0)

Therefore, x (1) = x (0) + α0 d (0) =



5/2

2

 +

    1 −12 1 = . 2 8 0

The gradient of f at x (1) is g (1) = ∇ f (x (1) ) =

β0 =

(g (1) )T g (1) (g (0) )T g (0)

   0 06 6 1 36  = = . =  12 144 4 12 0 0

d (1) = −g (1) + β0 d (0) = − Therefore,

  0 . 6

      1 −12 −3 0 = . + −6 6 4 0

236

8 Conjugate Gradient Methods

  −3 0 6 −6 1 (g (1) )T d (1) 36 36    = −   = = , α1 = − (1) T =− (1)   −3 72 2 8 −4 −3 (d ) Qd −3 −6 0 −12 −4 4 −6 −6 

and x (2) = x (1) + α1 d (1) =

      1 −3 −1/2 1 . = + −1 2 2 −6

Note that g (2) = ∇ f (x (2) ) =

  0 . 0

Thus, the Fletcher–Reeves algorithm stops as the gradient is zero. Since f is a ∗ (2) quadratic   function of two variables, therefore we get the minimum point x = x = −1/2 , and the minimum value of the objective function f (x ∗ ) = 0 in two iterations. −1 The R function Fletcher_Reeves(x, epsilon, maxiter) for the Fletcher–Reeves algorithm is given below in Function 8.2. R Function 8.2 Fletcher_Reeves.R 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

# Description : Fletcher - R e e v e s M e t h o d # Theory : See S e c t i o n 8.5 and A l g o r i t h m 8.3 # Function Called : func _ var ( ) , Grad _ Vec ( ) , G o l d e n _ Line ( ) # Input : x : S t a r t i n g point # epsilon : T o l e r a n c e value # maxiter : M a x i m u m n u m b e r of i t e r t i o n s p e r f o r m e d # O u t p u t : xm : Minimum point # fm : m i n i m u m v a l u e of f u n c t i o n at xm # E x e c u t e on R C o n s o l e : F l e t c h e r _ R e e v e s ( x , 1 e -7 , 1 0 0 ) F l e t c h e r _ R e e v e s 0. It is true only when Hk is positive definite. For this, we have already presented the proof of the following proposition. Proposition 9.1 Let f ∈ C 1 , x (k) ∈ Rn , g (k) = ∇ f (x (k) ) = 0, and Hk is an n × n real symmetric positive definite. If x (k+1) = x (k) − αk Hk g (k) , where αk = arg minα≥0 f (x (k) − α Hk g (k) ), then αk > 0, and f (x (k+1) ) < f (x (k) ).



9.2 Basics of Quasi-Newton Methods

247

We use objective function and its gradient to construct an approximation for the inverse of the Hessian matrix. In the next section, we present the method to choose Hk such that iteration may be carried out without any evaluation of the Hessian and solution of set of linear equations.

9.3 Approximating the Inverse Hessian Let H0 , H1 , H2 , . . . be successive approximation of the inverse F(x (k) )−1 . The Hessian F(x) of objective function f is constant and independent of x. Consider a quadratic objective function with Hessian F(x) = Q for all x, where Q = Q T . Then, g (k+1) − g (k) = Q(x (k+1) − x (k) ). Set Δg (k) = g (k+1) − g (k) and Δx (k) = x (k+1) − x (k) . Therefore, Δg (k) = QΔx (k) , that is, Δx (k) = Q −1 Δg (k) . We can also write the above equality in the following form: Q −1 Δg (i) = Δx (i) ,

0 ≤ i ≤ k.

(9.6)

We also impose the requirement that the approximation Hk+1 satisfy Hk+1 Δg (i) = Δx (i) ,

0 ≤ i ≤ k.

(9.7)

If n steps are involved, then moving in n directions Δx (0) , Δx (1) , . . . , Δx (n−1) gives Hn Δg (0) = Δx (0) , Hn Δg (1) = Δx (1) , .. . Hn Δg (n−1) = Δx (n−1) . We can represent the set of above equations as     Hn Δg (0) , Δg (1) , . . . , Δg (n−1) = Δx (0) , Δx (1) , . . . , Δx (n−1) .

(9.8)

248

9 Quasi-Newton Methods

Since Δg (i) = QΔx (i) , then     Q Δx (0) , Δx (1) , . . . , Δx (n−1) = Δg (0) , Δg (1) , . . . , Δg (n−1) , that is,     Q −1 Δg (0) , Δg (1) , . . . , Δg (n−1) = Δx (0) , Δx (1) , . . . , Δx (n−1) .   If Δg (0) , Δg (1) , . . . , Δg (n−1) is nonsingular, then Q −1 is determined uniquely after n steps through  −1  Q −1 = Hn = Δx (0) , Δx (1) , . . . , Δx (n−1) Δg (0) , Δg (1) , . . . , Δg (n−1) . Under these circumstances, we conclude that if Hn satisfies Hn Δg (i) = Δx (i) , where 0 ≤ i ≤ n − 1, then the algorithm x (k+1) = x (k) − αk Hk g (k) ,

(9.9)

where αk = arg minα≥0 f (x (k) − α Hk g (k) ), guarantees to solve the quadratic functions of n variables in n + 1 steps because of the updating next point as x (n+1) = x (n) − αn Hn g (n) which is equivalent to Newton’s algorithm. The above discussion reflects the basic idea behind the Quasi-Newton algorithms. In fact, we present a result which shows that such algorithms find the minimizer of quadratic functions of n variables in at most n iterations. Although, the Quasi-Newton algorithms have the following form: d (k) = −Hk g (k) , αk = arg min f (x (k) + αd (k) ), α≥0

and x (k+1) = x k + αk d (k) .

(9.10)

The matrices H0 , H1 , . . . are symmetric. For the quadratic functions, these matrices should satisfy Hk+1 Δg (i) = Δx (i) , 0 ≤ i ≤ k,

9.3 Approximating the Inverse Hessian

249

where Δx (i) = x (i+1) − x (i) = αi d (i) and Δg (i) = g (i+1) − g (i) = QΔx (i) . Note that the Quasi-Newton methods are also conjugate direction methods which can be seen in the following result. Theorem 9.1 Consider a Quasi-Newton algorithm applied to a quadratic function with Hessian Q = Q T such that Hk+1 Δg (i) = Δx (i) , 0 ≤ i ≤ k, 0 ≤ k ≤ n − 1, where Hk+1 = (Hk+1 )T . If αi = 0, where 0 ≤ i ≤ k, then directions d (0) , d (1) , . . . , d (k+1) are Q−conjugate. Proof We proceed by mathematical induction, we begin with k = 0 : Since d (1) = −(F(x (1) ))−1 g (1) and (d (1) )T = −(g (1) )T H1 ), then (d (1) )T Qd (0) = −(g (1) )T H1 Qd (0) . Since d (0) =

x (1) −x (0) α0

=

Δx (0) , α0

then

(d (1) )T Qd 0 = −(g (1) )T H1 Q

Δx (0) . α0

Since QΔx (0) = Δg (0) , then (d (1) )T Qd (0) = −(g (1) )T H1

Δg (0) . α0

(9.11)

Since H1 Δg (0) = Δx (0) , then (d (1) )T Qd (0) = −(g (1) )T

Δx (0) . α0

(d (1) )T Qd (0) = −(g (1) )T d (0) .

(9.12) (9.13)

Since α0 > 0 is minimizer of φ(α) = f (x (0) + αd (0) ), therefore (d (1) )T Qd (0) = 0. Assume that the result is true for k − 1, where k < n − 1, that is, d (0) , d (1) , . . . , d (k−1) are Q−conjugate. We now prove the result is true for k, that is, (d (k+1) )T Qd (i) = 0,

0 ≤ i ≤ k.

250

9 Quasi-Newton Methods

Note that (d (k+1) )T Qd (i) = −(Hk+1 g (k+1) )T Qd (i) = −(g (k+1) )T Hk+1 Qd (i) = −(g (k+1) )T Hk+1 Q

Δx (i) , αi

that is, (d (k+1) )T Qd (i) = −(g (k+1) )T Hk+1

Δg (i) αi

Δx (i) αi (k+1) T (i) = −(g ) d . = −(g (k+1) )T

Since d (0) , . . . , d (k) are Q−conjugate by assumption then from Lemma 8.2, we have (g (k+1) )T d (i) = 0, thus (d (k+1) )T Qd (i) = 0. 

This completes the proof.

From this theorem, we conclude that a Quasi-Newton methods solve a quadratic of n variables in at most n steps. Remark 9.1 Note that the equations that the matrices Hk are required to satisfy do not determine those matrices uniquely. So, we have some freedom. In the following section, we compute Hk+1 by adding a correction quantity to Hk .

9.4 The Rank One Correction Formula The rank one correction formula is sometimes called the single rank symmetric (SRF) algorithm. In the rank one correction formula, we add ak z(k) ((z)k )T with Hk , where ak ∈ R, and z(k) ∈ Rn . Therefore, Hk+1 = Hk + ak z(k) (z(k) )T .

(9.14)

Note that z(k) (z(k) )T is a real symmetric matrix. The product z(k) (z(k) )T is sometimes referred to as the dyadic product or outer product. We call rank one correction formula because the rank of matrix z(k) (z(k) )T is one. Since z(k) ∈ Rn , then we can have

9.4 The Rank One Correction Formula

251

z(k)

⎡ ⎤ z1 ⎢z2 ⎥ ⎢ ⎥ = ⎢ . ⎥. ⎣ .. ⎦ zn

Taking product of z(k) and (z(k) )T , we get ⎡ 2 ⎡ ⎤ z1 z1 z2 z1 ⎢z1 z2 z22 ⎢z2 ⎥

 ⎥ ⎢ ⎢ z(k) (z(k) )T = ⎢ . ⎥ z1 z2 . . . zn = ⎢ . .. ⎣ .. ⎣ .. ⎦ . zn z1 zn z2 zn Applying row elimination method, that is R2 − above matrix in the following form: ⎡

z21 z1 z2 ⎢0 0 ⎢ z(k) (z(k) )T = ⎢ . . ⎣ .. .. 0 0

R 1 z2 ,..., z1

... ... .. .

... ... .. .

⎤ z1 zn z2 zn ⎥ ⎥ .. ⎥ . . ⎦

. . . z2n

Rn −

R 1 zn , z1

to reduce the

⎤ z1 zn 0 ⎥ ⎥ .. ⎥ . . ⎦

... 0

Since number of nonzero rows in above row echelon matrix is 1, thus rank of above matrix is 1. Note that if Hk is symmetric, then Hk+1 is also symmetric. In order to find the value of Hk+1 , we need to compute ak , and z(k) for given values of Hk , Δg (k) , Δx (k) . In the previous section, we have already seen the following: Hk+1 Δg (i) = Δx (i) , i = 0, 1, . . . , k.

(9.15)

First, consider Hk+1 Δg (k) = Δx (k) for given Hk , Δg (k) , Δx (k) . Let us find ak , and z(k) . Using (9.14) in (9.15), we get

that is,

Hk + ak z(k) (z(k) )T Δg (k) = Δx (k) ,

(9.16)

Hk Δg (k) + ak z(k) (z(k) )T Δg (k) = Δx (k) .

We get

ak z(k) (z(k) )T Δg (k) = Δx (k) − Hk Δg (k) . Since (z(k) )T Δg (k) is scalar, then

(9.17)

252

9 Quasi-Newton Methods

z(k) =

Δx (k) − Hk Δg (k) . ak (z(k) )T Δg (k)

(9.18)

We can also have ak z(k) (z(k) )T = ak



Δx (k) − Hk Δg (k) ak (z(k) )T Δg (k)



Δx (k) − Hk Δg (k) ak (z(k) )T Δg (k)

T ,

that is,  T  (k) Δx − Hk Δg (k) Δx (k) − Hk Δg (k) ak z (z ) = .

2 ak (z(k) )T Δg (k) (k)

(k) T

(9.19)

Using (9.19) in (9.14), we obtain Hk+1

 T  (k) Δx − Hk Δg (k) Δx (k) − Hk Δg (k) = Hk + .

2 ak (z(k) )T Δg (k)

(9.20)

Note that   Δx (k) − Hk Δg (k) = ak (z(k) )T Δg (k) z(k) .

(9.21)

Premultiplying (Δg k )T from both sides of (9.21) to get   (Δg (k) )T (Δx (k) − Hk Δg (k) ) = ak (z(k) )T Δg (k) (Δg (k) )T z(k) . Since (Δg (k) )T z(k) is scalar, then  2 (Δg (k) )T (Δx (k) − Hk Δg (k) ) = ak ((z)k )T Δg (k) .

(9.22)

From (9.20) and (9.22), we get Hk+1

 T  (k) Δx − Hk Δg (k) Δx (k) − Hk Δg (k)   = Hk + . (Δg (k) )T Δx (k) − Hk Δg (k)

(9.23)

This is called rank one correction formula. A rank one algorithm is presented to solve unconstrained minimization problem. The symmetric rank one method is a Quasi-Newton method to update the second derivative (Hessian) based on the derivatives (gradients) calculated at two points. Note that it is a generalization to the secant method for multidimensional problems. This update maintains the symmetry of the matrix but does not guarantee that the

9.4 The Rank One Correction Formula

253

update to be positive definite. We summarize the above development in the form of the following algorithm. Algorithm 9.1 (Rank One Correction Algorithm) 1: 2: 3: 4: 5: 6: 7:

Choose x (0) ∈ Rn , ε

x (0) is a starting point, ε is tolerance value Set k ← 0

k is number of iterations of the method Set matrix H0 ← In

In is an identity matrix, n is number of variables of function for k to maxiter do

maxiter is number of iteration to perform Compute g (k) ← ∇ f (x (k) )

gradient of f at x (k) Compute d (k) ← −Hk g (k)

search direction (k) T d (k) Compute αk ← arg minα≥0 f (x (k) + αd k ) use Golden_Line() or αk ← − (d(g(k) )T) Qd (k) Update x (k+1) ← x (k) + αk d (k)  T  Δx (k) −Hk Δg (k) Δx (k) −Hk Δg (k)   Compute Hk+1 ← Hk + (k) T (k) (k)

8: 9:

(Δg

)

Δx

−Hk Δg f (x (k) ) < ε

10: if  f (x (k+1) ) − f (x (k) ) < ε or ∇ then 11: Converged. Print x ∗ ← x (k+1) and f (x ∗ ) ← f (x (k+1) ) 12: end if 13: end for

a new point

approximate inverse Hessian

stopping criteria

best solution

Example 9.1 Apply the rank one correction formula   to minimize the function 2 1 2 2 (0) f (x) = 2 x1 + x2 + 4 with a starting point x = and H0 = I2 (2 × 2 identity 1 matrix). The gradient and Hessian of function f are  x1 , 2x2

 ∇ f (x) = 

and Q=

 10 . 02

The gradient and Hessian of f at x (0) are g (0) = ∇ f (x (0) ) = and Q = F(x (0) ) =

  2 , 2



 10 . 02 

 10 The Hessian Q is a positive definite. Given that H0 = , which is also a positive 01 (0) definite. The search direction at x is

254

9 Quasi-Newton Methods

d (0) = −H0 g (0) = −



10 01

    2 −2 = . 2 −2

Since f is quadratic, then the step length is computed as    −2 22 −2 2    = . =−

 1 0 −2 3 −2 −2 0 2 −2

α0 = −

(g (0) )T d (0) (d (0) )T Qd (0)

We compute next point as x (1) = x (0) + α0 d (0) =

  2       2 −2 2 − 43 2 = 31 . = + 1 1 − 43 −3 3 −2

To compute the value of H1 , we need to get the value of the following quantities:     2 −2 −4/3 = −4/3 3 −2   2/3 g (1) = ∇ f (x (1) ) = −2/3   2     2 −4/3 , Δg (0) = g (1) − g (0) = 3 2 − = −8/3 2 −3   −4/3 , H0 Δg (0) = Δg (0) = −8/3       −4/3 0 −4/3 − = 4 , Δx (0) − H0 Δg (0) = −4/3 −8/3 /3  

 0 32 (Δg (0) )T (Δx (0) − H0 Δg (0) ) = −4/3 −8/3 4 = − . /3 9 Δx (0) = α0 d (0) =

Therefore,  T  (0) Δx − H0 Δg (0) Δx (0) − H0 Δg (0)   H1 = H0 + (Δg (0) )T Δx (0) − H0 Δg (0)   0 4    0 /3 4/3 10 = + 01 −32/9           9 0 0 10 0 0 1 0 10 = − = . = − 01 0 1/2 0 1/2 01 32 0 16/9 The search direction is computed as

9.4 The Rank One Correction Formula

d (1) = −H1 g (1) = −



255

1 0 0 1/2



2/3



 =−

−2/3

2/3



−1/3

=

  −2/3 . 1/3

The step length is again computed as    −2/3 1/3    =−

 1 0 −2/3 2 1 − /3 /3 1/3 02 

−4/9 −2/9   =−

 −2/3 −2/3 2/3 1 /3 6 − /9 =− 4/9 + 2/9 6/9 = = 1. 6/9

α1 = −

(g (1) )T d (1) (d (1) )T Qd (1)

2/3

−2/3

The new point x (2) is computed as: x

(2)

=x

(1)

+ α1 d

(1)

 =

2/3



−1/3

    −2/3 0 + 1 = . /3 0

Since the gradient of f at x (2) is g (2) = ∇ f (x (2) ) =

  0 , 0

therefore, the rank one correction algorithm stops. Since  the  objective function f is 0 quadratic, thus we get the minimum point x ∗ = x (2) = , and the minimum value 0 ∗ of the function f (x ) = 4 in two iterations only. We present the following remarks on the basis of above the example. Remark 9.2 1. Rank one correction algorithm solves a quadratic function of two variables in two iterates only. 2. The directions d (0) and d (1) are Q−conjugate because (d

(0) T

) Qd

(1)

    

 −2/3

 1 0 −2/3 4 4 = −2 −4 = − = 0. = −2 −2 1/3 1/3 02 3 3

The R function Rank_One_Correction(x, epsilon, maxiter) for the rank one correction algorithm is given below in Function 9.1.

256

9 Quasi-Newton Methods

R Function 9.1 Rank_One_Correction.R 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54

# Description : Rank One C o r r e c t i o n M e t h o d # Theory : See S e c t i o n 9.4 and A l g o r i t h m 9.1 # Function Called : func _ var ( ) , Grad _ Vec ( ) , G o l d e n _ Line ( ) # Input : x : S t a r t i n g point # epsilon : T o l e r a n c e value # maxiter : N u m b e r of i t e r t i o n s p e r f o r m e d # O u t p u t : xm : Minimum point # fm : M i n i m u m f u n c t i o n value # E x e c u t e on R C o n s o l e : Rank _ One _ C o r r e c t i o n ( x , 1 e -7 , 100) Rank _ One _ C o r r e c t i o n 0, for all x = 0. For this, we have to write the following quadratic form:  T   T (k) Hk Δg (k) Δx (k) Δx (k) T Hk Δg x Hk+1 x = x Hk x + x  x−x x T  T Δx (k) Δg (k) Δg (k) Hk Δg (k) 2  T   T  T x Δx (k) x Hk Δg (k) Δg (k) Hk x T = x Hk x +  − . T  T Δx (k) Δg (k) Δg (k) Hk Δg (k) T

T

T

Therefore, 2 2  T x T Δx (k) x Hk Δg (k) x Hk+1 x = x Hk x +  − . T T Δx (k) Δg (k) Δg (k) Hk Δg (k) 

T

T

Set

1

a := Hk2 x,

1

1

(9.30)

1

b := Hk2 Δg (k) , where Hk = Hk2 Hk2 .

1

Since Hk > 0, and Hk2 is well defined, then 1 1  1 T  21  Hk x = a T a, x T Hk x = x T Hk2 Hk2 x = Hk2 x 1

1

x T Hk Δg (k) = x T Hk2 Hk2 Δg (k) = a T b, and  (k) T T 1 1 T  1    1 Δg Hk Δg (k) = Δg (k) Hk2 Hk2 Δg (k) = Hk2 Δg (k) Hk2 Δg (k) = b T b. Therefore, (9.30) is reduced in the following form:

9.7 Convergence Analysis of DFP Method

271

2  T 2 x T Δx (k) a b x Hk+1 x = a a +  − T T b b Δx (k) Δg (k)   T  2 2 x Δx (k) a, b 2 − = a +  T b2 Δx (k) Δg (k) 2  2  T a, b x Δx k 2 = a − + , T b2 Δx (k) Δg (k) 

T

T

that is, 2  2  T a2 b2 − a, b x Δx (k) x Hk+1 x = + T b2 Δx (k) Δg (k) T

(9.31)

Note that   (k) T (k)  (k) T  (k+1) g Δx Δg = Δx − g (k) T T   = Δx (k) g (k+1) − Δx (k) g (k)  T T  = αk d (k) g (k+1) − Δx (k) g (k)  T Since d (k) g (k+1) = 0, then T  (k) T (k)  Δx Δg = − Δx (k) g (k) Since Δx (k) = αk d (k) = −αk Hk g (k) , then  (k) T (k)  T Δx Δg = αk g (k) Hk g (k)

(9.32)

Using (9.32) in (9.31), we obtain 2  2  T a2 b2 − a, b x Δx (k) +  T x Hk+1 x = b2 αk g (k) Hk g (k) T

(9.33)

In (9.33), both terms of RHS are nonnegative. The first term is nonnegative due to the Cauchy–Schwarz inequality x, y ≤ xy, ∀ x, y ∈ Rn equality holds if x = αy, where α ∈ R. The second term is nonnegative due to the fact that Hk is positive definite and αk > 0.

272

9 Quasi-Newton Methods

Therefore, in order to show that x T Hk+1 x > 0 for x = 0, we need to show that the first term is zero only if a and b are proportional, that is, if a = βb for some β ∈ R. Thus, to complete the proof, it is enough to show that if a = βb, then

2 x T Δx (k) > 0.  T αk g (k) Hk g (k) 

Note that 1 1 1  Hk2 x = a = βb = β Hk2 Δg (k) = Hk2 βΔg (k) ,

which yields

x = βΔg (k) .

Using value of x in the second term of right-hand side of (9.33), we get 2 T 2  x T Δx (k) βΔg (k) Δx (k) =  T  T αk g (k) Hk g (k) αk g (k) Hk g (k)  (k) T 2 Δx (k) 2 Δg =β  T αk g (k) Hk g (k)   (k) T 2 Hk g (k) 2 αk g =β  T αk g (k) Hk g (k)  T = β 2 αk g (k) Hk g (k) > 0. 

(9.34)

(9.35)

Thus, from (9.33), for all x = 0, we get x T Hk+1 x > 0. This completes the proof.



Remark 9.6 We did not consider the case that the second term of RHS is zero in (9.33). Suppose  T 2 x Δx (k) = 0,  T αk g (k) Hk g (k)

9.7 Convergence Analysis of DFP Method

273

that is, only Δx (k) = 0 if and only if x (k+1) = x (k) , that is, x (k) = x ∗ already. That is, g (k) = 0, and αk > 0. If αk < 0, then d (k) is not negative, which is wrong. The DFP method is the conjugate direction method. It plays an important role in the theoretical analysis and numerical computations. However, further studies indicate that the DFP method is numerically unstable, and sometimes produces numerically singular Hessian approximations. Therefore, the other famous Quasi-Newton update is presented in the next section, which overcomes from the drawbacks of the DFP method and performs better than the DFP update.

9.8 Broyden–Fletcher–Goldfarb–Shanno Method It is possible to update the Hessian rather than its inverse at every iteration. Such popular update formula was suggested independently in 1970 by Broyden, Fletcher, Goldfarb, and Shanno that had proven to be most effective in many applications. We will see its one application in the form of case study to solve the problem of heat conduction. This method is known as the BFGS algorithm. The BFGS method replaces the conjugate gradient technique for solving the nonlinear functions. In order to derive the BFGS update, we use the concept of duality, or complementarity, see (Fletcher 1970). Recall the updating formulas for the approximation of the inverse of the Hessian matrix which were based on the following equations: Hk+1 Δg (i) = Δx (i) , where 0 ≤ i ≤ k,

(9.36)

which is derived from Δg (i) = QΔx (i) , where 0 ≤ i ≤ k. We then formulated update formulas for the approximations to the inverse of the Hessian, that is, Q −1 . We have an alternative to approximate Q instead of approximating Q −1 . For doing this, let Bk be our estimate of Q k at the kth step. We need Bk+1 to satisfy the following set of equations: Δg (i) = Bk+1 Δx (i) , where 0 ≤ i ≤ k.

(9.37)

Note that (9.36) and (9.37) are similar, the only difference is that Δx (i) and Δg (i) are interchanged. Thus, given any update formula for Hk , a corresponding update formula for Bk can be found by interchanging the roles of Bk and Hk and of Δg (k) and Δx (k) . In particular, the BFGS update for Bk corresponding to the DFP update for Hk . Formulas related in this way are said to be dual or complementary. Recall the DFP update for the approximation Hk of the inverse Hessian from (9.28) as DF P Hk+1 = Hk +

(Δx (k) )(Δx (k) )T (Hk Δg (k) )(Hk Δg (k) )T − (Δx (k) )T Δg (k) (Δg (k) )T Hk Δg (k)

(9.38)

274

9 Quasi-Newton Methods

We apply the complementarity concept in the above equation. Therefore, we obtain an update equation for the approximation Bk of the Hessian as Bk+1 = Bk +

Δg (k) (Δg (k) )T Bk Δx (k) (Δx (k) )T Bk − . (Δg (k) )T Δg (k) (Δx (k) )T Bk Δx (k)

(9.39)

This is called the BFGS update of Bk . We wish to obtain the BFGS update for the approximation of the inverse Hessian. Therefore, we take the inverse of Bk+1 as BFGS Hk+1 = (Bk+1 )−1 =

 Bk +

Δg (k) (Δg (k) )T Bk Δx (k) (Δx (k) )T Bk − (Δg (k) )T Δx (k) (Δx (k) )T Bk Δx (k)

−1 . (9.40)

BFGS To compute Hk+1 , we invert the RHS of (9.40) by using the following lemma:

Lemma 9.1 Let A be a nonsingular matrix. Let u and v be column vectors such that 1 + v T A−1 u = 0. Then, A + uv T is nonsingular, and (A + uv T )−1 = A−1 −

(A−1 u)(v T A−1 ) . 1 + v T A−1 u

Proof Proof is out of the scope of this book. See page 123 of (Householder 1975).  Applying Lemma 9.1 twice to Bk+1 of (9.40) to get  T  (k) T  Hk Δg (k) Δx (k) Δx (k) Δg = Hk + 1 +  T  T Δg (k) Δx (k) Δx (k) Δg (k)  T   T T Hk Δg (k) Δx (k) + Hk Δg (k) Δx (k) −  T Δg (k) Δx (k) 

BFGS Hk+1

(9.41)

This represents the BFGS formula for updating Hk . The BFGS algorithm saves the effort to invert the Hessian matrix at each iteration. It also preserves the symmetry and positive definiteness of matrix at each iteration. Note that the inverse Hessian is updated by the sum of two symmetric rank one matrices at every step in the DFP algorithm, but BFGS algorithm estimates the Hessian Hk , instead of its inverse. The following algorithm is presented for BFGS.

9.8 Broyden–Fletcher–Goldfarb–Shanno Method

275

Algorithm 9.3 (BFGS Algorithm) 1: 2: 3: 4: 5: 6: 7:

Choose x (0) ∈ Rn , ε

x (0) is a starting point, ε is tolerance value k←0

k is number of iterations of method Set H0 ← In .

In is an identity matrix, n is number of variables of function. for k to maxiter do

maxiter is maximum number of iterations to perform. Compute g (k) ← ∇ f (x (k) )

gradient of f at x k Compute d (k) ← −Hk g (k) .

search direction (k) T d (k) Compute αk ← arg minα≥0 f (x (k) + αd k ) use Golden_Line() or αk ← − (d(g(k) )T) Qd (k)

Update x (k+1) ← x (k) + αk d (k) . T  T    Δg (k) Hk Δg (k) Δx (k) Δx (k) 9: Find Hk+1 ← Hk + 1 +  − T  T

8:

Δg (k)

Δx (k)

Δx (k)



Hk Δg (k) Δx (k)

T 



Δg (k)

Δg (k)

approximate inverse Hessian 10: if ( f (x (k+1) ) − f (x (k) )) < ε or ∇ f (x (k+1) ) < ε then 11: Converged. Print x ∗ ← x (k+1) and f (x ∗ ) ← f (x (k+1) ). 12: end if 13: end for

a new point  T T

+ Hk Δg (k) Δx (k)

T

Δx (k)

stopping criteria

best solution

We conclude our discussion of the BFGS algorithm with the following numerical example. Example 9.6 Apply the BFGS method to minimize the function     1 T 2 −3 T 0 x−x f (x) = x −3 5 1 2 with H0 = I2 (2 × 2 identity matrix), and the starting point x (0) =

  0 . Verify that 0

H2 = Q −1 . We know that   d (0) = −H0 g (0) = −g (0) = − Qx (0) − b = −



2 −3 −3 5

      0 0 0 + = . 0 1 1

Since objective function is a quadratic, thus we use the following formula to compute the step length α0 as 

 (0) T

d (0) g α0 = −  T d (0) Qd (0)

 

 0 01 1 1 1   = .   = =

 2 −3 0

 0 5 01 −3 5 −3 5 1 1

Therefore, x (1) = x (0) + α0 d (0) =

      1 0 0 0 = 1 . + /5 0 5 1

276

9 Quasi-Newton Methods

To compute H1 = H1B F G S , we need the following quantities: Δx

(0)

=x

(1)

−x

(0)

 0 = 1 . /5 

g (1) = Qx (1) − b      2 −3 0 0 = − 1/5 −3 5 1     3 − /5 0 = − 1 1   −3/5 . = 0 Therefore, Δg (0) = g (1) − g (0) =

      0 −3/5 −3/5 − = . 0 −1 1

    (0) T  1 0 −3/5

(0) 3 Δg H0 Δg = − /5 1 1 01    −3/5

= −3/5 1 1 9 +1 = 25 34 = . 25 

 T Δx (0) Δx (0) =



   0 0 0 1  0 / 5 = . 1/5 0 1/25

   (0) T (0)

 −3/5 1 Δx Δg = 0 1/5 = . 1 5    (0) T  0

1 (0) 3 Δg Δx = − /5 1 1 = . /5 5

9.8 Broyden–Fletcher–Goldfarb–Shanno Method

277

     T  T −3/5 1  0 −3/25 0 /5 = . H0 Δg (0) Δx (0) = Δg (0) Δx (0) = 1 0 1/5 Therefore,  (0) T  T  T   T T   H0 Δg (0) Δx (0) Δx (0) Δg H0 Δg (0) Δx (0) + H0 Δg (0) Δx (0) H1 = H0 + 1 +  − T  T  T Δg (0) Δx (0) Δx (0) Δg (0) Δg (0) Δx (0)        

34/25 1 0 0 0 −3/25 0 0 10 + 1+ ×5× − + = 1 1/5 0 1/25 −3/25 1/5 01 0 1/5 5

We evaluate      0 −3/25 10 0 0 − 5 + 39 −3/25 2/5 01 0 1/25       0 −3/5 10 0 0 − = + −3/5 2 01 0 39/25     10 0 3/5 = + 3 /5 −11/25 01   1 3/5 . = 3 14 /5 /25 

H1 =

We now have d (1) = −H1 g (1) = −



1 3/5 3/5 14/25



     3/5 −3/5 −3/5 =− 9 = 9 . 0 − /25 /25

The step length α1 is computed as follows:

T g (1) d (1) α1 = − T d (1) Qd (1)    3/5

3 − − /5 0 9 /25    =

 2 −3 3/5 3/5 9/25 9/25 −3 5 −9/25 =− = 5. 9/125 Therefore, x

(2)

=x

(1)

+ α1 d

(1)

         3/5 0 3 3 0 = 1 + 9 = = 1 +5 9 . /5 /25 /5 /5 2 

278

9 Quasi-Newton Methods

  0 , thus we do not need to get x (3) . The BFGS algorithm stops. 0 Since the objective function f is quadratic, thus we get the minimum point x ∗ = x (2) = 23 , and the minimum value of the objective function f (x ∗ ) = −1 To verify that H2 = Q −1 , we must compute the following quantities:

Note that g (2) =

      3 3 0 = 9 − 1 /5 /5 2       3/5 0 −3/5 = = . − 0 0 0

Δx (1) = x (2) − x (1) = Δg (1) = g (2) − g (1)

     (1)  (1) T 3 9  9 27/5 3 /5 = 27 81 . Δx = 9 Δx /5 /5 /25     

T 3/5  1 3/5  3/5



9 Δg (1) H1 Δg (1) = 3/5 0 3 14 , = 3/5 9/25 = /5 /25 0 0 25  

T  3

(1) (1) 3 Δg Δx = /5 0 9 = /5  

T  3/5

Δx (1) Δg (1) = 3 9/5 = 0

9 5 9 . 5

   

T  1 3/5  3/5

3/5

9/5 27/25   (1) 9 9 3 /5 = 9 H1 Δg . Δx = 3 14 3 5 = 27 81 /5 /25 0 /25 /25 /125 (1)

Thus,  T  T   T T  (1) T   H1 Δg (1) Δx (1) Δx (1) Δg H1 Δg (1) Δx (1) + H1 Δg (1) Δx (1) H2 = H1 + 1 +  − T  T  T Δg (1) Δx (1) Δx (1) Δg (1) Δg (1) Δx (1)          9/25 9/5 27/25 9/5 27/25 1 1 3/5 9 27/5 1 = 3 14 − + 1+ + 27/5 81/25 9/5 27/25 81/125 27/25 81/125 9/5 9/5 /5 /25       2 9 27/5 5 18/5 54/25 1 3/5 + − = 3 14 /5 /25 3 27/5 81/25 9 54/25 162/125       6 18/5 2 6/5 1 3/5 + 18 54 − 6 18 = 3 14 /5 /25 /5 /25 /5 /25     3/5 + 18/5 − 6/5 7−2 53 , = 3 = 18 6 54 18 14 32 /5 + /5 − /5 /25 + /25 − /25

and

9.8 Broyden–Fletcher–Goldfarb–Shanno Method

Q −1 =

279

  53 . 32

Note that indeed H2 Q = Q H2 = I2 , and hence H2 = Q −1 . For nonquadratic problems, Quasi-Newton algorithms will not usually converge in n steps. As in the case of the conjugate gradient methods, here too, some modifications may be necessary to deal with nonquadratic problems. For example, we may reinitialize the direction vector to the negative gradient after every few iterations, and continue until the algorithm satisfies the stopping criteria. The R function BFGS(x, epsilon, maxiter) for the BFGS algorithm is given below in Function 9.3. R Function 9.3 BFGS.R 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

33 34 35 36 37

# Description : BFGS Method # Theory : See S e c t i o n 9.8 and A l g o r i t h m 9.3 # Function Called : func _ var ( ) , Grad _ Vec ( ) , G o l d e n _ Line ( ) # Input : x : S t a r t i n g point # epsilon : T o l e r a n c e Value # maxiter : N u m b e r of i t e r t i o n s p e r f o r m e d # O u t p u t : xm : Minimum point # fm : M i n i m u m f u n c t i o n value # E x e c u t e on R c o n s o l e : BFGS (x , 1 e -7 , 100) BFGS BFGS(x=c(-3,4), epsilon = 1e-7, maiter = 100) Initial function value: 2516.0000 k 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

x -3.0000 4.0000 -2.0344 4.1607 -3.9878 15.9282 -3.9881 15.9056 -3.7005 13.6119 -3.4937 12.0880 -2.5770 6.5980 -2.5873 6.6819 -2.3312 5.3650 -2.0874 4.2655 -0.5279 0.2686 -0.5295 0.2762 -0.34375 0.08114 -0.17229 -0.01911 1.0094 1.0190 1.0094 1.0190 1.0002 1.0004 0.9999 0.9998 1.0006 1.0011 1.0000 1.0001

f(x) 2516.0000 9.2554 24.9434 24.8813 22.7666 21.5843 12.9783 12.8830 11.5781 10.3758 2.3448 2.3412 1.9427 1.6124 8.99e-05 8.94e-05 1.056e-06 2.005e-08 9.469e-07 5.607e-08

|| gradient || 6090.6550 12.4741 31.1610 9.0091 131.7390 175.4054 52.0235 19.8061 72.6927 84.8792 5.5779 4.0314 10.7393 11.3057 0.03438 0.008812 0.04555 0.005445 0.03555 0.01082

minimum point: 1.0000 1.0001 minimum function value: 5.607e-08

  1 The Rosenbrock function is minimized at x = in 19 iterations, whereas the DFP 1 algorithm mimimizes the Rosenbrock function in 34 iterations with same starting ∗

9.8 Broyden–Fletcher–Goldfarb–Shanno Method

283

  −3 . We conclude that the BFGS algorithm is better than the DFP 4 algorithm in number of iterations. point x (0) =

Example 9.8 Apply the BFGS algorithm to minimize the NONDQUAR test function ⎡ ⎤ 1 f (x) = (x1 − x2 )2 + (x1 x2 + x3 )4 + (x2 + x3 )2 with a starting point x (0) = ⎣−1⎦ 1 using R. The R function func_var(x) is used to write the above objective in the R script as func_var BFGS(x = c(1, -1, 1), epsilon = 1e-7, maxiter = 100) Initial function value: 4.0000 k 0 1 2 3 4 5 6 7 8 9 10

1.0000 0.3902 -0.1574 -0.1573 -0.1488 -0.08624 -0.07675 -0.04135 -0.03422 -0.02910 -0.001653

x -1.0000 -0.3902 -0.1239 -0.1569 -0.1389 -0.08971 -0.07686 -0.03977 -0.03418 -0.02922 -0.001623

1.0000 1.0000 0.1566 0.1607 0.1289 0.09306 0.07687 0.03822 0.03412 0.02934 0.001621

f(x) || gradient || 4.0000 5.6569 1.4974 3.7571 0.003157 0.1708 0.001195 0.03371 0.000698 0.02269 0.0001265 0.01265 4.694e-05 0.002313 7.431e-06 0.004264 1.556e-06 0.0001104 8.593e-07 0.000423 9.048e-10 8.12e-05

minimum point: -0.001653 -0.001623 minimum function value: 9.048e-10

0.001621

284

9 Quasi-Newton Methods

9.9 Case Study: Heat Conduction Problem Consider a rectangular flat plate with dimensions 5 × 4 units with an internal generation of heat, see (Surkan and Wu 1968). Let the thermal conductivity k be constant, while the generation of heat per unit area f is nonlinear function of the temperature T . Our objective is to find the temperature of the slab so that the temperature along the perimeter of the slab is zero. Therefore, it is desired to calculate the temperature of the slab as a function of poisson. Poisson’s equation relates the temperature within this region is given by 

∂2T ∂2T + k ∂x2 ∂ y2

 + f (T ) = 0,

(9.42)

where, in this case, k = 2, and 3 1 f (T ) = 20 − T + T 2 . 2 20

(9.43)

An approximation to this partial differential equation is afforded by replacing the derivatives with central differences. These differences were established on a regular square mesh. While there are 12 mesh points, by symmetry only four of these have different temperatures. Thus, four equations are obtained. 3 2(T2 + T3 − 4T1 ) = −20 + T1 − 2 3 2(T3 + T1 + T4 − 4T3 ) = −20 + T3 − 2 3 2(2T1 + T4 − 4T2 ) = −20 + T2 − 2 3 2(2T3 + T2 − 3T4 ) = −20 + T4 − 2

1 2 T , 20 1 1 2 T , 20 3 1 2 T , 20 2 1 2 T . 20 4

(9.44) (9.45) (9.46) (9.47)

These equations, arranged in powers of T1 , are T12 − 190T1 + 40(T2 + T3 + 10) = 0, − 150T3 + 400 + T4 = 0, 40 2 T − 190T2 + 400 + T4 = 0, 2T1 + 2 40 T42 − 150T4 + 40T2 + 80T3 + 400 = 0. T1 +

T32

(9.48) (9.49) (9.50) (9.51)

The objective function f is constructed by summing the square of each of the functions associated with each nonlinear equations. Rearranged in the final form for finding the required first and second derivative expressions essential to the method,

9.9 Case Study: Heat Conduction Problem

285

this function becomes f (T1 , T2 , T3 , T4 , H1 , H2 , H3 , H4 , H5 , H6 ) = Z 1 + Z 2 + Z 3 + Z 4 ,

(9.52)

where Z 1 = Z 52 ,

(9.53)

Z 62 , Z 72 , Z 82 ,

(9.54)

Z2 = Z3 = Z4 =

 1 2 T + H1 T1 + H2 (T2 + T3 + H3 ) , 20 1   (T 2 + H4 T3 ) + H5 + T4 , Z 6 = 2 T1 + 3 H2   (T 2 + H1 T2 ) + H5 + T4 , Z 7 = 2 H6 T1 + 2 H2   1 T42 + H4 T4 + H2 T2 + H2 H6 T3 + H2 H5 . Z8 = 20 Z5 =

(9.55) (9.56) (9.57) (9.58) (9.59) (9.60)

We obtain our objective function as follows: 2 1  2 T1 + H1 T1 + H2 (T2 + T3 + H3 ) 400 2 2   (T32 + H4 T3 ) (T 2 + H1 T2 ) + 4 T1 + + H5 + T4 + 4 H6 T1 + 2 + H5 + T4 H2 H2 2 1  2 T4 + H4 T4 + H2 T2 + H2 H6 T3 + H2 H5 . + (9.61) 400

f (T1 , T2 , T3 , T4 , H1 , H2 , H3 , H4 ) =

If H1 = −190, H2 = 40, H3 = 10, H4 = −150, H5 = 10, H6 = 2, and let T1 = x1, T2 = x2 , T3 = x3 , T4 = x4 , then   x2 2 f = 2(x2 + x3 − 4x1 ) + 20 − 1.5x1 + 1 20     2 x 2 x2 2 + 2(x1 − 3x3 + x4 ) + 20 − 1.5x3 + 3 + 2(2x1 + x4 − 4x2 ) + 20 − 1.5x2 + 2 20 20 2  2 x + 2(x2 + 2x3 − 3x4 ) + 20 − 1.5x4 + 4 . (9.62) 20

We say (9.62) as heat conduction problem (HCP) function. The R function func_var(x) is used to write the HCP function in the R script as

286

9 Quasi-Newton Methods

func_var Rank_One_Correction(x=c(4.7, 6.1, 6.5, 8.0), 1e-7, 100) Initial function value: 7.8916 k 0 1 2 3 4 5

4.7000 4.8312 4.8272 4.8417 4.8520 4.8520

6.1000 5.9979 6.0272 6.0403 6.0545 6.0545

x 6.5000 6.3990 6.3711 6.3782 6.4042 6.4042

8.0000 8.0839 8.1017 8.1045 8.1382 8.1383

f(x) ||gradient|| 7.8916 0.2118 0.1672 0.04441 0.02976 0.02098 0.01736 0.04605 2.521e-06 0.0001302 1.194e-08 1.27e-05

Minimum point: 4.8520 6.0545 6.4042 8.1383 Minimum function value: 6.402e-08

On executing the R function DFP(x, epsilon, maxiter) with x=c(4.7, 6.1, 6.5, 8.0), epsilon=1e-7 and maxiter=100, the following output is displayed on the R Console for the given HCP function. > DFP(x=c(4.7, 6.1, 6.5, 8.0), 1e-7, 100) Initial function value: 7.8916 k 0 1 2 3 4 5

4.7000 4.8312 4.8272 4.8417 4.8519 4.8520

6.1000 5.9979 6.0272 6.0403 6.0545 6.0545

x 6.5000 6.3990 6.3711 6.3782 6.4042 6.4042

8.0000 8.0839 8.1017 8.1045 8.1381 8.1383

f(x) ||gradient|| 7.8916 72.9096 0.1672 6.2076 0.02976 1.2037 0.01736 0.9784 2.704e-06 0.0410 1.282e-08 0.002823

9.9 Case Study: Heat Conduction Problem

287

Minimum point: 4.8520 6.0545 6.4042 8.1383 Minimum function value: 1.282e-08

On executing the R function BFGS(x, epsilon, maxiter) with x=c(4.7, 6.1, 6.5, 8.0), epsilon=1e-7, and maxiter=100, the following output is displayed on the R Console for the given HCF function. > BFGS(x=c(4.7, 6.1, 6.5, 8.0), 1e-7, 100) Initial function value: 7.8916 k 0 1 2 3 4 5

x 4.7000 4.8312 4.8272 4.8417 4.8519 4.8520

6.1000 5.9979 6.0272 6.0403 6.0544 6.0545

6.5000 6.3990 6.3711 6.3782 6.4041 6.4042

8.0000 8.0839 8.1016 8.1045 8.1381 8.1383

f(x) || 7.8916 0.1672 0.02976 0.01736 2.909e-06 1.379e-08

gradient || 72.9096 6.2076 1.2033 0.9762 0.04105 0.002825

Minimum point: 4.8520 6.0545 6.4042 8.1383 Minimum function value: 1.379e-08

Thus, we obtained the temperature of slab as follows: T1 = 4.851746, T2 = 6.054155, T3 = 6.403995, T4 = 8.137709, and temperature along the slab is obtained as f (T1 , T2 , T3 , T4 ) = 0.

9.10 Exercises 1. Apply the rank one correction formula to minimize the function   0 2 (0) x2 − x1 + x2 + 7 with a starting point x = using R. 0 2. Apply the rank one correction formula to minimize the function   2 x22 + x1 x2 + x1 + 2x2 with a starting point x (0) = using R. 1 3. Apply the rank one correction formula to minimize the function 7 2 x + 29 x32 + 2x1 x2 + 3x2 x3 + x3 x1 − 9x1 + 8x3 + 3 with a 2 2 ⎡ ⎤ 1 x (0) = ⎣1⎦ using R. 1

f (x) = 21 x12 + f (x) = 23 x12 + f (x) = 25 x12 + starting point

288

9 Quasi-Newton Methods

4. Apply the rank one correction formula to minimize the function f (x) = x22

x14 2

+

+ x1 x2 − x1 + x2 using R starting from the following two different points:   1 (0) ; a. x = 1   3/2 b. x (0) = 3 . /2 3

5. Apply the rank one correction formula to minimize  the  function f (x) = 0.5(1 − 2 2 2 2 (0) x1 ) + (x2 − x1 ) with a starting point x = using R. −1 6. Apply the rank one correction formula to minimize the function f (x) = (x12 + x2 − 11)2 + (x22 + x1 − 7)2 with the following starting points:   1 (0) ; a. x = 1   3/2 b. x (0) = 7 . /5 7. Apply the DFP algorithm to minimize the function f (x) = x12 + 2x22 + x1 + 7   0 with a starting a point x (0) = . 0 8. Apply the DFP method to minimize the function f (x) = x1 − x2 with the following starting points:   0 a. x (0) = , 0   3/2 . b. x (0) = 1

x14 2

+

x22 4

− 2x1 x2 +

2 9. Apply the DFP algorithm to minimize ⎡ the ⎤ function f (x) = (x1 − x2 ) + (x2 − −2.6 x3 )4 from a starting point x (0) = ⎣ 2 ⎦ using R. 2 10. Apply the DFP algorithm to minimize f (x) = 100(x2 − x12 )2 +  the function  −1.2 (1 − x1 )2 with a starting point x (0) = using R. 1.0 11. Apply the DFP method to minimize the Himmelblaufunction f (x) = (x12 +  0 x2 − 11)2 + (x1 + x22 − 7)2 with starting point x (0) = using R. 0 12. Apply the DFP algorithm to minimize the Powell’s function f (x) = (x1 + 10x2 )⎤2 + 5(x3 − x4 )2 + (x2 − 2x3 )4 + 10(x1 − x4 )4 with a starting point x (0) = ⎡ 3 ⎢−1⎥ ⎢ ⎥ using R. ⎣0⎦ 1

9.10 Exercises

289

4  13. Apply the DFP algorithm to minimize the quartic function f (x) = i=1 ⎡ ⎤ 1 ⎢1⎥

i−1 4  ⎥ 10 xi + xi3 + 101−i xi2 with a starting point x (0) = ⎢ ⎣1⎦ using R. 1 2 14. Apply the BFGS algorithm to minimize the function   f (x) = (x1 + 2x2 − 7) + −1 (2x1 + x2 − 5)2 with a starting point x (0) = . 2 15. Apply the BFGS method minimize the function f (x) = (x1 + x2 )4 + x22 with  to  −3 a starting point x (0) = using R. 2 2 2 2 16. Apply the BFGS method to minimize the function   f (x) = (x1 − x2 − 9) + −3 (x1 + 2x2 − 3)2 with a starting point x (0) = using R. 2 2 2 17. Apply the BFGS method to minimize the function  f (x) = 5x1 + x2 + 4x1 x2 − 0.5 14x1 − 6x2 + 15 with a starting point x (0) = using R. 0.5  18. Apply the BFGS method to minimize the function f (x) = x12 + x22 + 21 x1 +   2  4 1 x2 + 21 x1 + x2 with a starting point x (0) = using R. 2 19. Apply the BFGS method to minimize the Beale function f (x) = (1.5 − x1 (1 − 2 2 3 2 x2 )2 ) +⎡(2.25 ⎤ − x1 (1 − x2 ) ) + (2.625 − x1 (1 − x2 ) ) with a starting point 1 x (0) = ⎣2⎦ using R. 3 20. Apply the BFGS method to minimize the Valley function (YUAN  3  1991) f (x) = π using R. 100(x2 − sin(x1 ))2 + 0.25x12 with a starting point x (0) = 2 −1

Solutions

Chapter 2 1. a. Linear dependent, b. Linearly independent 2. a. Possible, b. Not possible, c. Possible. 3. 8 4. 1. A1 = 20, and A∞ = 19, 2. B1 = 20, B∞ = 15. 5. Rank of Matrix is 1. 6. Eigenvalues are −3 and 4. 7. λmin = −1. 10. a. Yes, except x = 0. b. Yes, for x > 0. Chapter 4

    4 82 , and F(x) = . 25 26 ⎡ ⎤ ⎡ ⎤ 18 −6 0 18 2. ∇ f (x) = ⎣ 8 ⎦ , and F(x) = ⎣ 0 8 −8⎦. 25 18 −8 0 ⎡ ⎤ 0 3. a. ∇ f (x) = ⎣2⎦ , and direction derivative is = 4. 0 b. Directional derivative is 36. c. Strictlyminimizer.    24 9 4. ∇ f (x) = , F(x) = 68 19 1. ∇ f (x) =

© Springer Nature Singapore Pte Ltd. 2019 S. K. Mishra and B. Ram, Introduction to Unconstrained Optimization with R, https://doi.org/10.1007/978-981-15-0894-3

291

292

Solutions

8. Inflection point.

  0 11. f (x) is minimum at x = . 0 13. No unique extrema. Chapter 5 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20.

x∗ x∗ x∗ x∗ x∗ x∗ x∗ x∗ x∗ x∗ x∗ x∗ x∗ x∗ x∗ x∗ x∗ x∗ x∗ x∗

= 2.823108, and f (x ∗ ) = 23.9098. = 6, and f (x ∗ ) = −103. = 0.4502, and f (x ∗ ) = −0.2325. = 1.0225, and f (x ∗ ) = 0. = 0, and f (x ∗ ) = 2. = 1.1855, and f (x ∗ ) = 12.6515. = −10.0000, and f (x ∗ ) = −1580.6667. = 1.3508, and f (x ∗ ) = 3.0836. = 0.5724, and f (x ∗ ) = 0.7509. = −0.5599, and f (x ∗ ) = −0.03171. = 4.9411, and f (x ∗ ) = −12613.1004. = −0.7035, and f (x ∗ ) = 1.8327. = 0.7809, and f (x ∗ ) = −24.3696. = 0.7101, and f (x ∗ ) = −0.03202. = −5.7751, and f (x ∗ ) = 41165.6144. = 1.0845, and f (x ∗ ) = 1.9601. = −1.6574 , and f (x ∗ ) = 3.8517. = 0.1078, and f (x ∗ ) = 0.8988. = 2.5981, and f (x ∗ ) = −11.6913. = −0.6503, and f (x ∗ ) = −2.1870

Chapter 6   0 ∗ 1. x = , and f (x ∗ ) = 0. 0   −0.8 ∗ 2. x = , and f (x ∗ ) = −0.7. 0.6 3. f (x ∗ ) = 0.   2 4. x ∗ = , and f (x ∗ ) = −4. 2   0.9994 5. x ∗ = , and f (x ∗ ) = 10.0000. 1.0012 ⎡ ⎤ 0 6. x ∗ = ⎣0⎦ , and f (x ∗ ) = 0. 0

Solutions

293



 0.02694 , and f (x ∗ ) = 0.00653. 0.05387 ⎡ ⎤ 4 = ⎣3⎦ , and f (x ∗ ) = −50.5. 1   0 = , and f (x ∗ ) = 0. 0   0 = , and f (x ∗ ) = 0. 0   0.9373 = , and f (x ∗ ) = −2.3437. −0.6248   5 = , and f (x ∗ ) = −22. 6 ⎡ ⎤ 1 = ⎣1⎦ , and f (x ∗ ) = 0. 1   −0.1332 = , and f (x ∗ ) = 0.9333. 3.3328   1.6593 = , and f (x ∗ ) = 3.0837. 1.6593 ⎡ ⎤ 4.0001 = ⎣ 3.0000 ⎦ , and f (x ∗ ) = 0. −4.0001   0 = , and f (x ∗ ) = 0. 0   0.7502 = , and f (x ∗ ) = −0.75. 0.7502   3.3999 = , and f (x ∗ ) = 0. −1.8000   1 = , and f (x ∗ ) = 0. 1

7. x ∗ = 8. x ∗ 9. x ∗ 10. x ∗ 11. x ∗ 12. x ∗ 13. x ∗ 14. x ∗ 15. x ∗ 16. x ∗ 17. x ∗ 18. x ∗ 19. x ∗ 20. x ∗

Chapter 7   0.1154 1. x ∗ = , and f (x ∗ ) = −0.03846. −0.4231 ⎡ ⎤ 8.1462 2. x ∗ = ⎣8.1462⎦ , and f (x ∗ ) = 0. 4.0739

294

3. x ∗

4. x ∗ 5. x ∗ 6. x ∗ 7. x ∗

8. x ∗

9. x ∗ 10. x ∗

11. x ∗

12. x ∗

13. x ∗ 14. x ∗ 15. x ∗ 16. x ∗ 17. x ∗

Solutions

⎡ ⎤ 0 ⎢0⎥ ∗ ⎥ =⎢ ⎣0⎦ , and f (x ) = 0. 0   −1.1422 = , and f (x ∗ ) = −6.5229. 0.5497   0 = , and f (x ∗ ) = 2. 0   −6 = , and f (x ∗ ) = 25. 6   1 = , and f (x ∗ ) = 0. 1 ⎡ ⎤ 0 ⎢0⎥ ∗ ⎥ =⎢ ⎣0⎦ , and f (x ) = 0. 0   1 = , and f (x ∗ ) = 0. 1 ⎡ ⎤ 0 = ⎣−0.16⎦ , and f (x ∗ ) = 0. −0.15 ⎡ ⎤ 1 ⎢1⎥ ⎢ =⎣ ⎥ , and f (x ∗ ) = 0. 1⎦ 1 ⎡ ⎤ 1 = ⎣1⎦ , and f (x ∗ ) = 0. 1 ⎡ ⎤ 1.0002445 = ⎣−0.0003654⎦ , and f (x ∗ ) = 0.25. −0.0333070   3 = , and f (x ∗ ) = 0. 2   1e + 06 = , and f (x ∗ ) = 0. 2e − 06   1 = , and f (x ∗ ) = 0. 1   2 = , and f (x ∗ ) = −1. 1

Solutions

295



 0.4999 , and f (x ∗ ) = −2.0833. −0.6666 ⎡ ⎤ 0 19. x ∗ = ⎣0⎦ , and f (x ∗ ) = 0. 0 18. x ∗ =

Chapter 8 1. d (1) 3. Yes. 4. d (1)

⎡ ⎤ ⎡ ⎤ 1 2 = ⎣0⎦ , and d (2) = ⎣6⎦ . 0 5 ⎤ ⎡ ⎤ 1 1 = ⎣−2⎦ , and d (2) = ⎣−2⎦. 0 −3

5. Yes.

  0 , and f (x ∗ ) = 0. 0   2 = , and f (x ∗ ) = 0. 1 ⎡ ⎤ 0 x ∗ = ⎣0⎦ , and f (x ∗ ) = 1. 0

 = 0.9994 − 3.0071 − 2.0057 , and f (x ∗ ) = −0.5.   1 = , and f (x ∗ ) = 0. 0 ⎡ ⎤ −0.1666 ⎢−0.5834⎥ ∗ ⎥ =⎢ ⎣−0.1807⎦ , and f (x ) = 0. −0.1807   1 = , and f (x ∗ ) = 5. 1 ⎡ ⎤ 0 = ⎣ 1 ⎦ , and f (x ∗ ) = 0. −2 ⎡ ⎤ 0 ⎢0⎥ ∗ ⎥ =⎢ ⎣0⎦ , and f (x ) = 0. 0   0.138 = , and f (x ∗ ) = 0.4893. −0.252

6. x ∗ = 7. x ∗

8. b. 9. x ∗ 10. x ∗

11. x ∗

12. x ∗ 13. x ∗

14. x ∗

15. x ∗

⎡

296

Solutions

  1 , and f (x ∗ ) = 0. 1   −0.714 = , and f (x ∗ ) = −0.5714. 1.571   −2.3881 = , and f (x ∗ ) = 0. −5.2972   0.5 = , and f (x ∗ ) = −2.0833. −0.667   2 = , and f (x ∗ ) = −10. −2

16. x ∗ = 17. x ∗ 18. x ∗ 19. x ∗ 20. x ∗

Chapter 9   1 ∗ 1. x = , and f (x ∗ ) = 6.25. −0.5

 2. x ∗ = 0 − 1 , and f (x ∗ ) = −1. ⎡ ⎤ 2.0668 3. x ∗ = ⎣−0.1297⎦ , and f (x ∗ ) = −10.6025. −1.0754   1.3064 ∗ 4. x = , and f (x ∗ ) = −3.8396. −3.4599   1 ∗ 5. x = , and f (x ∗ ) = 0. 1   −2.8051 ∗ 6. a. x = , and f (x ∗ ) = 0. 3.1313   3.5844 b. x ∗ = , and f (x ∗ ) = 0. −1.8481   −0.5 7. x ∗ = , and f (x ∗ ) = 6.75. 0   0.9779 ∗ 8. x = , and f (x ∗ ) = −1.7463. 1.435 ⎡ ⎤ 0.4664 9. x ∗ = ⎣0.4667⎦ , and f (x ∗ ) = 0. 0.4669   1 , and f (x ∗ ) = 0. 10. x ∗ = 1   −3.33915 11. x ∗ = , and f (x ∗ ) = −10.3279. −0.07487

Solutions

12. x ∗

13. x ∗

14. x ∗ 15. x ∗ 16. x ∗ 17. x ∗ 18. x ∗ 19. x ∗ 20. x ∗

⎡ ⎤ 0 ⎢0⎥ ∗ ⎥ =⎢ ⎣0⎦ , and f (x ) = 0. 0 ⎡ ⎤ 0 ⎢0⎥ ∗ ⎥ =⎢ ⎣0⎦ , and f (x ) = 0. 0   1 = , and f (x ∗ ) = 0. 3   0 = , and f (x ∗ ) = 0. 0   −5 = , and f (x ∗ ) = 0. 4   1 = , and f (x ∗ ) = 5. 1   0 = , and f (x ∗ ) = 0. 0 ⎡ ⎤ 2.8453 = ⎣0.3521⎦ , and f (x ∗ ) = 0.2565. 3.0   0 = , and f (x ∗ ) = 0. 0

297

Bibliography

Agarwal, R. P., & Sen, S. K. (2014). Creators of mathematical and computational sciences. New York: Springer Cham Heidelberg. Apostol, T. M. (1974). Mathematical analysis (2nd ed.). Reading, MA: Addison-Wesley Publishing Co. Aragón, F. J., Goberna, M. A., López, M. A., & Rodríguez, M. M. L. (2019). Nonlinear optimization. Springer Nature, Switzerland: Springer Undergraduate Texts in Mathematics and Technology. Arora, J. S. (1989). Introduction to optimum design. New York: McGraw-Hill Book Co. Arora, R. (2015). Optimization: Algorithms and applications. USA: CRC Press. Attia, J. O. (1999). Electronics and circuit analysis using MATLAB. USA: CRC Press. Avriel, M. (1976). Nonlinear programming: Analysis and methods. Mineola, New York: Dover Publications. Inc. Bakr, M. (2013). Nonlinear optimization in electrical engineering with applications in MATLAB. UK: The Institution of Engineering and Technology. Bartle, R. G. (1976). The elements of real analysis (2nd ed.). New York: John Wiley & Sons. Belegundu, A. D., & Chandrupatla, T. (2011). R, Optimization concepts and applications in engineering. United States of America: Cambridge University Press. Broyden, C. G. (1970). The convergence of a class of double-rank minimization algorithms 1. General Considerations, IMA Journal of Applied Mathematics, 06, 76–90. Cauchy, A. (1847). Méthode générale pour la résolution des systéms d’équations simultanées. Comptes Rendus de I’Académie des Sciences, Paris, 25, 536–538. Chong, E. K. P., & Z˙ ak, S. H. (2013). An introduction to optimization (4th ed.). John Wiley & Sons Inc. Christon, N. (2008). Enhancing the teaching of statistics: Portfolio theory, an application of statistics in finance. Journal of Statistics Education, 16, https://doi.org/10.1080/10691898.2008. 11889570. Conn, A. R. (1991). Convergence of quasi-Newton matrices generated by the symmetric rank one update. Mathematical Programming, 50, 177–195. Dennis, J. E., & Schnabel, R. B. (1983). Numerical methods for unconstrained optimization and nonlinear equations. USA; Prentice-Hall. Drummonda, L. M. G., & Svaiter, B. F. (2005). A steepest descent method for vector optimization. Journal of Computational and Applied Mathematics, 175, 395–414. Edgar Thomas, F., & Himmelblau David, M. (1988). Optimization of chemical process. New York: McGraw-Hill. Fletcher, R. (1978). Practical methods of optimization (2nd ed.). Chichester: John Wiley & Sons. © Springer Nature Singapore Pte Ltd. 2019 S. K. Mishra and B. Ram, Introduction to Unconstrained Optimization with R, https://doi.org/10.1007/978-981-15-0894-3

299

300

Bibliography

Fletcher, R. (1970). A new approach to variable metric algorithms. The Computer Journal, 13, 317–322. Fletcher, R., & Powell, M. J. D. (1963). A rapidly convergent descent method for minimization. The Computer Journal, 6, 163–168. Fletcher, R., & Reeves, C. M. (1964). Function minimization by conjugate gradients. The Computer Journal, 7, 149–154. Fliege, J., & Svaiter, B. F. (2000). Steepest descent method for multicriteria optimization. Mathematical Methods of Operations Research, 51, 479–494. Goldfarb, D. (1970). A family of variable-metric methods derived by variational means. Mathematics of Computation, 24, 23–26. Golub, G. H., & O’leary, D. P., (1989). Some histroy of the conjugate gradient and lanczos algorithm: 1948–1976. SIAM, 31, 50–102. Griva, I., Nash, S. G., & Sofer, A. (2009). Linear and nonlinear optimization. USA: SIAM. Hendrix, E. M. T., & G.-Tóth, B. (2010). Introduction to nonlinear and global optimization. In Springer undergraduate texts in mathematics and technology. New York: Springer. Hestenes, M. R., & Stiefel, E. (1952). Methods of conjugate gradients for solving linear systems. Journal of Research of the National Bureau of Standards, 49, 409–436. Householder, A. S. (1975). The theory of matrices in numerical analysis. New York: Dover Publications. http://finance.yahoo.com . Johansson, E. M., Dowla, F. U., & Goodman, D. M. (1992). Backpropagation learning for multilayer feed-forward neural networks using the conjugate gradient method. International Journal of Neural Systems, 2, 291–301. Levenberg, K. (1944). A method for the solution of certain non-linear problems in least squares. Quarterly of Applied Mathematics, 2, 164–168. Luenberger, D. G. (1984). Linear and nonlinear programming. MA: Addison-Wesley. Markowitz, H. M. (1959). Portfolio selection, efficient diversification of investments, I. John wiley & Sons. Markowitz, H. M. (1999). The early history of Portfolio Theory: 1600–1960. Financial Analysts Journal, 55, 5–16. Mohammad, F., & Hassan, B. (2019). Optimization in electrical engineering. Switzerland: Springer Nature. Moré, J. J., Garbow, B. S., & Hillstrom, K. E. (1981). Testing unconstrained optimization software. ACM Transactions on Mathematical Software, 7(1), 17–41. Norris, D. O., & Gerken, J. D. (1977). A rank-one algorithm for unconstrained function minimization. Journal of Optimization Theory and Applications, 21, 261–275. Pardalos, P. M., Sandstrom, M., & Zopounidis, C. (1994). On the use of optimization models for Portfolio selection: A review and some computational results. Computational Economics, 7, 227–244. Pearson, J. D. (1969). Variable metric methods of minimization. The Computer Journal, 12, 171– 178. Pierre, D. A. (1986). Optimization theory with applications (2nd ed.). USA: Dover Publications. Polak, E., & Ribiere, G. (1969). Note sur la convergence de méthodes de directions conjuguées. ESAIM: Mathematical Modelling and Numerical Analysis, 3, 35–43. Powell, M. J. D. (1964). An efficient method for finding the minimum of a solution of several variables without calculating derivatives. The Computer Journal, 7, 155–162. Powell, M. J. D. (1986). How bad are the BFGS and DFP methods when the objective function is quadratic ? Mathematical Programming, 34, 34–47. Rao, S. S. (1996). Engineering optimization theory and practice. New Age International (P) Ltd. Ruddin, R. (1976). Principles of mathematical analysis (3rd ed.). New York: McGraw-Hill Book Co.

Bibliography

301

Salleh, Z., & Alhawarat, A. (2016). An efficient modification of the Hestenes-Stiefel nonlinear conjugate gradient method with restart property. Journal of Inequalities and Applications,110. https://doi.org/10.1186/s13660-016-1049-5. Shanno, D. F. (1970). Conditioning of quasi-Newton methods for function minimization. Mathematics of Computation, 24, 647–656. Surkan, A. J., & Wu, C. L. (1968). Solution of reaction and Heat Flow Problems by Nonlinear Estimation. The Canadian Journal of Chemical Engineering, 46, 229–232. Williams, J. B. (1938). Theory of investment value. Harvard University Press. Yuan, YA-X. (1991). A modified BFGS algorithm for unconstrained optimization. IMA Journal of Numerical Analysis, 11. Zlobec, S. (1983). Private communication.

Index

Symbols BFGS(x, epsilon, maxiter), 279 Bisection( ), 114 Grad_Vec(x), 60

B BFGS, 271

C Closed set, 22 Conjugate direction, 211 Conjugate gradient, 222 Conjugate_Gradient(), 236 Conjugate_Gradient(x, epsilon, maxiter), 225 Constrained optimization problem, 57 Convegence analysis of conjugate direction method, 217 Convergence analysis of DFP method, 268 Convergence analysis of Newton’s method, 183 Convergence analysis of rank one correction formula, 258 Convergence analysis of steepest descent algorithm, 146 Convergence sequence, 25

Fibonacci_Search(), 99 Fibonacci search algorithm, 95 Fibonacci search method, 91 First-order necessary condition for a minimum, 67 G Global minimizer, 58 Golden_Line(), 138 Golden_Section_Search(), 105 Golden section search algorithm, 103 Golden section search method, 101 H Heat conduction problem, 284 Hessian(x), 61 I Inner product, 11 Interior case, 70 Interval_Halving_Method(), 89 Interval halving search method, 86

D Davidon Fletcher and Powell (DFP), 260 Directional derivative, 64

L Linear equations in electrical engineering, 239 Linearly dependent, 10 Linearly independent, 10 Local minimizer, 58 Loop Statements, 46

F Feasible direction, 63

M Modified_Newton(), 190

© Springer Nature Singapore Pte Ltd. 2019 S. K. Mishra and B. Ram, Introduction to Unconstrained Optimization with R, https://doi.org/10.1007/978-981-15-0894-3

303

304 Modified Newton’s method, 188 N Neighborhoods, 20 Newton–Raphson algorithm, 116 Newton–Raphson method, 116 Newton’s method, 175 O Open set, 21 Optimal design of a Pi-Circuit, 203 Optimizing portfolio, 160 Orthogonal, 13 P Positive definiteness, 29 Producing microelectronics for lithography, 122 Q Quadratic form, 27 Quadratic_interpolation(), 109 Quadratic_Interpolation_Search(), 109 Quadratic interpolation search method, 107 Quasi-Newton methods, 245 R Rank_One_Correction(), 255 Rank one correction formula, 250 Rayleigh’s inequality, 18 S Secant(), 120

Index Secant algorithm, 119 Secant method, 119 Second-order necessary condition, 73 Second-order necessary condition for a minimum , 73 Second-order sufficient condition for interior case, 78 Steepest_Descent(), 141 Steepest descent algorithm, 139 Steepest descent method, 131 Stopping criterion, 23 Strictly global minimizer, 58 Strictly local minimizer, 58 Sylvester’s criteria, 30 Symmetric matrix, 14

T Taylor’s Theorem, 26

U Unconstrained optimization problem, 57 Unimodal function, 86

V Vector norm, 12 Vector space, 10

W While loop, 47

Z Zlobec’s function, 142, 143, 180, 191, 198, 227, 237, 257, 265, 280