An Easy Path to Convex Analysis and Applications [1 ed.] 1627052372, 9781627052375, 9781627052382, 1627052380

Convex optimization has an increasing impact on many areas of mathematics, applied sciences, and practical applications.

619 33 1MB

English Pages 218 Year 2014

Report DMCA / Copyright

DOWNLOAD FILE

Polecaj historie

An Easy Path to Convex Analysis and Applications [1 ed.]
 1627052372, 9781627052375, 9781627052382, 1627052380

Table of contents :
Content: PrefaceAcknowledgmentsList of SymbolsConvex Sets and FunctionsSubdifferential CalculusRemarkable Consequences of ConvexityApplications to Optimization and Location ProblemsSolutions and Hints for ExercisesBibliographyAuthors' BiographiesIndex

Citation preview

An Easy Path to Convex Analysis and Applications

Synthesis Lectures on Mathematics and Statistics Editor Steven G. Krantz, Washington University, St. Louis

An Easy Path to Convex Analysis and Applications Boris S. Mordukhovich and Nguyen Mau Nam 2014

Applications of Affine and Weyl Geometry Eduardo García-Río, Peter Gilkey, Stana Nikčević, and Ramón Vázquez-Lorenzo 2013

Essentials of Applied Mathematics for Engineers and Scientists, Second Edition Robert G. Watts 2012

Chaotic Maps: Dynamics, Fractals, and Rapid Fluctuations Goong Chen and Yu Huang 2011

Matrices in Engineering Problems Marvin J. Tobias 2011

e Integral: A Crux for Analysis Steven G. Krantz 2011

Statistics is Easy! Second Edition Dennis Shasha and Manda Wilson 2010

Lectures on Financial Mathematics: Discrete Asset Pricing Greg Anderson and Alec N. Kercheval 2010

iii

Jordan Canonical Form: eory and Practice Steven H. Weintraub 2009

e Geometry of Walker Manifolds Miguel Brozos-Vázquez, Eduardo García-Río, Peter Gilkey, Stana Nikčević, and Ramón Vázquez-Lorenzo 2009

An Introduction to Multivariable Mathematics Leon Simon 2008

Jordan Canonical Form: Application to Differential Equations Steven H. Weintraub 2008

Statistics is Easy! Dennis Shasha and Manda Wilson 2008

A Gyrovector Space Approach to Hyperbolic Geometry Abraham Albert Ungar 2008

Copyright © 2014 by Morgan & Claypool

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means—electronic, mechanical, photocopy, recording, or any other except for brief quotations in printed reviews, without the prior permission of the publisher.

An Easy Path to Convex Analysis and Applications Boris S. Mordukhovich and Nguyen Mau Nam www.morganclaypool.com

ISBN: 9781627052375 ISBN: 9781627052382

paperback ebook

DOI 10.2200/S00554ED1V01Y201312MAS014

A Publication in the Morgan & Claypool Publishers series SYNTHESIS LECTURES ON MATHEMATICS AND STATISTICS Lecture #14 Series Editor: Steven G. Krantz, Washington University, St. Louis Series ISSN Synthesis Lectures on Mathematics and Statistics Print 1938-1743 Electronic 1938-1751

An Easy Path to Convex Analysis and Applications

Boris S. Mordukhovich Wayne State University

Nguyen Mau Nam Portland State University

SYNTHESIS LECTURES ON MATHEMATICS AND STATISTICS #14

M &C

Morgan & cLaypool publishers

ABSTRACT Convex optimization has an increasing impact on many areas of mathematics, applied sciences, and practical applications. It is now being taught at many universities and being used by researchers of different fields. As convex analysis is the mathematical foundation for convex optimization, having deep knowledge of convex analysis helps students and researchers apply its tools more effectively. e main goal of this book is to provide an easy access to the most fundamental parts of convex analysis and its applications to optimization. Modern techniques of variational analysis are employed to clarify and simplify some basic proofs in convex analysis and build the theory of generalized differentiation for convex functions and sets in finite dimensions. We also present new applications of convex analysis to location problems in connection with many interesting geometric problems such as the Fermat-Torricelli problem, the Heron problem, the Sylvester problem, and their generalizations. Of course, we do not expect to touch every aspect of convex analysis, but the book consists of sufficient material for a first course on this subject. It can also serve as supplemental reading material for a course on convex optimization and applications.

KEYWORDS Affine set, Carathéodory theorem, convex function, convex set, directional derivative, distance function, Fenchel conjugate, Fermat-Torricelli problem, generalized differentiation, Helly theorem, minimal time function, Nash equilibrium, normal cone, Radon theorem, optimal value function, optimization, smallest enclosing ball problem, set-valued mapping, subdifferential, subgradient, subgradient algorithm, support function, Weiszfeld algorithm

vii

e first author dedicates this book to the loving memory of his father S M (1924–1993), a kind man and a brave warrior. e second author dedicates this book to the memory of his parents.

ix

Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii List of Symbols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv

1

Convex Sets and Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 1.2 1.3 1.4 1.5 1.6

2

Subdifferential Calculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11

3

Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Convex Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Convex Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Relative Interiors of Convex Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 e Distance Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Exercises for Chapter 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Convex Separation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Normals to Convex Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Lipschitz Continuity of Convex Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Subgradients of Convex Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Basic Calculus Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 Subgradients of Optimal Value Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 Subgradients of Support Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Fenchel Conjugates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 Directional Derivatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 Subgradients of Supremum Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 Exercises for Chapter 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

Remarkable Consequences of Convexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 3.1 3.2 3.3

Characterizations of Differentiability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 Carathéodory eorem and Farkas Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Radon eorem and Helly eorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

x

3.4 3.5 3.6 3.7 3.8 3.9 3.10

4

Tangents to Convex Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Mean Value eorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 Horizon Cones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 Minimal Time Functions and Minkowski Gauge . . . . . . . . . . . . . . . . . . . . . . . 111 Subgradients of Minimal Time Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 Nash Equilibrium . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 Exercises for Chapter 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

Applications to Optimization and Location Problems . . . . . . . . . . . . . . . . . . . 129 4.1 4.2 4.3 4.4 4.5 4.6 4.7

Lower Semicontinuity and Existence of Minimizers . . . . . . . . . . . . . . . . . . . . 129 Optimality Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 Subgradient Methods in Convex Optimization . . . . . . . . . . . . . . . . . . . . . . . . 140 e Fermat-Torricelli Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 A Generalized Fermat-Torricelli Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 A Generalized Sylvester Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 Exercises for Chapter 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

Solutions and Hints for Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 Authors’ Biographies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

xi

Preface Some geometric properties of convex sets and, to a lesser extent, of convex functions had been studied before the 1960s by many outstanding mathematicians, first of all by Hermann Minkowski and Werner Fenchel. At the beginning of the 1960s convex analysis was greatly developed in the works of R. Tyrrell Rockafellar and Jean-Jacques Moreau who initiated a systematic study of this new field. As a fundamental part of variational analysis, convex analysis contains a generalized differentiation theory that can be used to study a large class of mathematical models with no differentiability assumptions imposed on their initial data. e importance of convex analysis for many applications in which convex optimization is the first to name has been well recognized. e presence of convexity makes it possible not only to comprehensively investigate qualitative properties of optimal solutions and derive necessary and sufficient conditions for optimality but also to develop effective numerical algorithms to solve convex optimization problems, even with nondifferentiable data. Convex analysis and optimization have an increasing impact on many areas of mathematics and applications including control systems, estimation and signal processing, communications and networks, electronic circuit design, data analysis and modeling, statistics, economics and finance, etc. ere are several fundamental books devoted to different aspects of convex analysis and optimization. Among them we mention “Convex Analysis” by Rockafellar [26], “Convex Analysis and Minimization Algorithms” (in two volumes) by Hiriart-Urruty and Lemaréchal [8] and its abridge version [9], “Convex Analysis and Nonlinear Optimization” by Borwein and Lewis [4], “Introductory Lectures on Convex Optimization” by Nesterov [21], and “Convex Optimization” by Boyd and Vandenberghe [3] as well as other books listed in the bibliography below. In this big picture of convex analysis and optimization, our book serves as a bridge for students and researchers who have just started using convex analysis to reach deeper topics in the field. We give detailed proofs for most of the results presented in the book and also include many figures and exercises for better understanding the material. e powerful geometric approach developed in modern variational analysis is adopted and simplified in the convex case in order to provide the reader with an easy path to access generalized differentiation of convex objects in finite dimensions. In this way, the book also serves as a starting point for the interested reader to continue the study of nonconvex variational analysis and applications. It can be of interest from this viewpoint to experts in convex and variational analysis. Finally, the application part of this book not only concerns the classical topics of convex optimization related to optimality conditions and subgradient algorithms but also presents some recent while easily understandable qualitative and numerical results for important location problems.

xii

PREFACE

e book consists of four chapters and is organized as follows. In Chapter 1 we study fundamental properties of convex sets and functions while paying particular attention to classes of convex functions important in optimization. Chapter 2 is mainly devoted to developing basic calculus rules for normals to convex sets and subgradients of convex functions that are in the mainstream of convex theory. Chapter 3 concerns some additional topics of convex analysis that are largely used in applications. Chapter 4 is fully devoted to applications of basic results of convex analysis to problems of convex optimization and selected location problems considered from both qualitative and numerical viewpoints. Finally, we present at the end of the book Solutions and Hints to selected exercises. Exercises are given at the end of each chapter while figures and examples are provided throughout the whole text. e list of references contains books and selected papers, which are closely related to the topics considered in the book and may be helpful to the reader for advanced studies of convex analysis, its applications, and further extensions. Since only elementary knowledge in linear algebra and basic calculus is required, this book can be used as a textbook for both undergraduate and graduate level courses in convex analysis and its applications. In fact, the authors have used these lecture notes for teaching such classes in their universities as well as while visiting some other schools. We hope that the book will make convex analysis more accessible to large groups of undergraduate and graduate students, researchers in different disciplines, and practitioners. Boris S. Mordukhovich and Nguyen Mau Nam December 2013

xiii

Acknowledgments e first author acknowledges continued support by grants from the National Science Foundation and the second author acknowledges support by the Simons Foundation. Boris S. Mordukhovich and Nguyen Mau Nam December 2013

xv

List of Symbols R R D . 1; 1 RC R> N co ˝ aff ˝ int ˝ ri ˝ span ˝ cone ˝ K˝ dim ˝ ˝ bd ˝ IB IB.xI N r/ dom f epi f gph F LŒa; b d.xI ˝/ ˘.xI ˝/ hx; yi A N.xI N ˝/ ker A D  F .x; N y/ N T .xI N ˝/ F1 T˝F L.x; / F

the real numbers the extended real line the nonnegative real numbers the positive real numbers the positive integers convex hull of ˝ affine hull of ˝ interior of ˝ relative interior of ˝ linear subspace generated by ˝ cone generated by ˝ convex cone generated by ˝ dimension of ˝ closure of ˝ boundary of ˝ closed unit ball closed ball with center xN and radius r domain of f epigraph of f graph of mapping F line connecting a and b distance from x to ˝ projection of x to ˝ inner product of x and y adjoint/transpose of linear mapping/matrix A normal cone to ˝ at xN kernel of linear mapping A coderivative to F at .x; N y/ N tangent cone to ˝ at xN horizon/asymptotic cone of F minimal time function defined by constant dynamic F and target set ˝ Lagrangian Minkowski gauge of F

xvi

LIST OF SYMBOLS

˘F .I ˝/ GF .x; N t/

generalized projection defined by the minimal time function generalized ball defined by dynamic F with center xN and radius t

1

CHAPTER

1

Convex Sets and Functions is chapter presents definitions, examples, and basic properties of convex sets and functions in the Euclidean space Rn and also contains some related material.

1.1

PRELIMINARIES

We start with reviewing classical notions and properties of the Euclidean space Rn . e proofs of the results presented in this section can be found in standard books on advanced calculus and linear algebra. Let us denote by Rn the set of all n tuples of real numbers x D .x1 ; : : : ; xn /. en Rn is a linear space with the following operations: .x1 ; : : : ; xn / C .y1 ; : : : ; yn / D .x1 C y1 ; : : : ; xn C yn /; .x1 ; : : : ; xn / D .x1 ; : : : ; xn /;

where .x1 ; : : : ; xn /; .y1 ; : : : ; yn / 2 Rn and  2 R. e zero element of Rn and the number zero of R are often denoted by the same notation 0 if no confusion arises. For any x D .x1 ; : : : ; xn / 2 Rn , we identify it with the column vector x D Œx1 ; : : : ; xn T , where the symbol “T ” stands for vector transposition. Given x D .x1 ; : : : ; xn / 2 Rn and y D .y1 ; : : : ; yn / 2 Rn , the inner product of x and y is defined by hx; yi WD

n X

x i yi :

iD1

e following proposition lists some important properties of the inner product in Rn . Proposition 1.1

For x; y; z 2 Rn and  2 R, we have:

(i) hx; xi  0, and hx; xi D 0 if and only if x D 0. (ii) hx; yi D hy; xi. (iii) hx; yi D hx; yi. (iv) hx; y C zi D hx; yi C hx; zi.

e Euclidean norm of x D .x1 ; : : : ; xn / 2 Rn is defined by q kxk WD x12 C : : : C xn2 :

2

1. CONVEX SETS AND FUNCTIONS

It follows directly from the definition that kxk D Proposition 1.2

p hx; xi.

For any x; y 2 Rn and  2 R, we have:

(i) kxk  0, and kxk D 0 if and only if x D 0. (ii) kxk D jj  kxk. (iii) kx C yk  kxk C kyk .the triangle inequality/. (iv) jhx; yij  kxk  kyk .the Cauchy-Schwarz inequality/.

Using the Euclidean norm allows us to introduce the balls in Rn , which can be used to define other topological notions in Rn .

e   centered at xN with radius r  0 and the    of Rn are defined, respectively, by ˇ ˇ ˚ ˚ IB.xI N r/ WD x 2 Rn ˇ kx xk N  r and IB WD x 2 Rn ˇ kxk  1 : Definition 1.3

It is easy to see that IB D IB.0I 1/ and IB.xI N r/ D xN C rIB . Definition 1.4

Let ˝  Rn . en xN is an   of ˝ if there is ı > 0 such that IB.xI N ı/  ˝:

e set of all interior points of ˝ is denoted by int ˝ . Moreover, ˝ is said to be  if every point of ˝ is its interior point. We get that ˝ is open if and only if for every xN 2 ˝ there is ı > 0 such that IB.xI N ı/  ˝ . n It is obvious that the empty set ; and the whole space R are open. Furthermore, any open ball B.xI N r/ WD fx 2 Rn j kx xk N < rg centered at xN with radius r is open. Definition 1.5

A set ˝  Rn is  if its complement ˝ c D Rn n˝ is open in Rn .

It follows that the empty set, the whole space, and any ball IB.xI N r/ are closed in Rn . Proposition 1.6

(i) e union of any collection of open sets in Rn is open.

(ii) e intersection of any finite collection of open sets in Rn is open. (iii) e intersection of any collection of closed sets in Rn is closed. (iv) e union of any finite collection of closed sets in Rn is closed. Definition 1.7 Let fxk g be a sequence in Rn . We say that fxk g  to xN if kxk xk N ! 0 as k ! 1. In this case we write

lim xk D x: N

k!1

1.1. PRELIMINARIES

is notion allows us to define the following important topological concepts for sets. Definition 1.8

Let ˝ be a nonempty subset of Rn . en:

(i) e  of ˝ , denoted by ˝ or cl ˝ , is the collection of limits of all convergent sequences belonging to ˝ . (ii) e  of ˝ , denoted by bd ˝ , is the set ˝ n int ˝ .

We can see that the closure of ˝ is the intersection of all closed sets containing ˝ and that the interior of ˝ is the union of all open sets contained in ˝ . It follows from the definition that xN 2 ˝ if and only if for any ı > 0 we have IB.xI N ı/ \ ˝ ¤ ;. Furthermore, xN 2 bd ˝ if and only if for any ı > 0 the closed ball IB.xI N ˝/ intersects both sets ˝ and its complement ˝ c .

Let fxk g be a sequence in Rn and let fk` g be a strictly increasing sequence of positive integers. en the new sequence fxk` g is called a  of fxk g.

Definition 1.9

We say that a set ˝ is bounded if it is contained in a ball centered at the origin with some radius r > 0, i.e., ˝  IB.0I r/. us a sequence fxk g is bounded if there is r > 0 with kxk k  r for all k 2 N :

e following important result is known as the Bolzano-Weierstrass theorem. eorem 1.10

Any bounded sequence in Rn contains a convergent subsequence.

e next concept plays a very significant role in analysis and optimization.

We say that a set ˝ is  in Rn if every sequence in ˝ contains a subsequence converging to some point in ˝ .

Definition 1.11

e following result is a consequence of the Bolzano-Weierstrass theorem. eorem 1.12

A subset ˝ of Rn is compact if and only if it is closed and bounded.

For subsets ˝ , ˝1 , and ˝2 of Rn and for  2 R, we define the operations: ˇ ˚ ˚ ˇ ˝1 C ˝2 WD x C y ˇ x 2 ˝1 ; y 2 ˝2 ; ˝ WD x ˇ x 2 ˝ : e next proposition can be proved easily. Proposition 1.13

Let ˝1 and ˝2 be two subsets of Rn .

(i) If ˝1 is open or ˝2 is open, then ˝1 C ˝2 is open. (ii) If ˝1 is closed and ˝2 is compact, then ˝2 C ˝2 is closed.

3

4

1. CONVEX SETS AND FUNCTIONS

Recall now the notions of bounds for subsets of the real line. Definition 1.14

have

Let D be a subset of the real line. A number m 2 R is a   of D if we x  m for all x 2 D:

If the set D has a lower bound, then it is  . Similarly, a number M 2 R is an   of D if x  M for all x 2 D; and D is   if it has an upper bound. Furthermore, we say that the set D is  if it is simultaneously bounded below and above. Now we are ready to define the concepts of infimum and supremum of sets.

Let D  R be nonempty and bounded below. e infimum of D , denoted by inf D , is the greatest lower bound of D . When D is nonempty and bounded above, its supremum, denoted by sup D , is the least upper bound of D . If D is not bounded below .resp. above/, then we set inf D WD 1 .resp. sup D WD 1/. We also use the convention that inf ; WD 1 and sup ; WD 1. Definition 1.15

e following fundamental axiom ensures that these notions are well-defined. Completeness Axiom. For every nonempty subset D of R that is bounded above, the least upper bound of D exists as a real number. Using the Completeness Axiom, it is easy to see that if a nonempty set is bounded below, then its greatest lower bound exists as a real number. roughout the book we consider for convenience extended-real-valued functions, which take values in R WD . 1; 1. e usual conventions of extended arithmetics are that a C 1 D 1 for any a 2 R, 1 C 1 D 1, and t  1 D 1 for t > 0. Definition 1.16 Let f W ˝ ! R be an extended-real-valued function and let xN 2 ˝ with f .x/ N < 1. en f is  at xN if for any  > 0 there is ı > 0 such that

jf .x/

f .x/j N <  whenever kx

xk N < ı; x 2 ˝:

We say that f is continuous on ˝ if it is continuous at every point of ˝ . It is obvious from the definition that if f W ˝ ! R is continuous at xN (with f .x/ N < 1), then it is finite on the intersection of ˝ and a ball centered at xN with some radius r > 0. FurtherN < 1) if and only if for every sequence fxk g in ˝ more, f W ˝ ! R is continuous at xN (with f .x/ converging to xN the sequence ff .xk /g converges to f .x/ N .

1.2. CONVEX SETS

Definition 1.17 Let f W ˝ ! R and let xN 2 ˝ with f .x/ N < 1. We say that f has a   at xN relative to ˝ if there is ı > 0 such that

f .x/  f .x/ N for all x 2 IB.xI N ı/ \ ˝:

We also say that f has a /  at xN relative to ˝ if f .x/  f .x/ N for all x 2 ˝:

e notions of local and global maxima can be defined similarly. Finally in this section, we formulate a fundamental result of mathematical analysis and optimization known as the Weierstrass existence theorem. eorem 1.18 Let f W ˝ ! R be a continuous function, where ˝ is a nonempty, compact subset of Rn . en there exist xN 2 ˝ and uN 2 ˝ such that

ˇ ˇ ˚ ˚ N D sup f .x/ ˇ x 2 ˝ : f .x/ N D inf f .x/ ˇ x 2 ˝ and f .u/

In Section 4.1 we present some “unilateral” versions of eorem 1.18.

1.2

CONVEX SETS

We start the study of convexity with sets and then proceed to functions. Geometric ideas play an underlying role in convex analysis, its extensions, and applications. us we implement the geometric approach in this book. Given two elements a and b in Rn , define the interval/line segment ˇ ˚ Œa; b WD a C .1 /b ˇ  2 Œ0; 1 : Note that if a D b , then this interval reduces to a singleton Œa; b D fag. Definition 1.19 A subset ˝ of Rn is  if Œa; b  ˝ whenever a; b 2 ˝ . Equivalently, ˝ is convex if a C .1 /b 2 ˝ for all a; b 2 ˝ and  2 Œ0; 1.

P Pm Given !1 ; : : : ; !m 2 Rn , the element x D m iD1 i !i , where iD1 i D 1 and i  0 for some m 2 N , is called a convex combination of !1 ; : : : ; !m .

Proposition 1.20

elements.

A subset ˝ of Rn is convex if and only if it contains all convex combinations of its

5

6

1. CONVEX SETS AND FUNCTIONS

Figure 1.1: Convex set and nonconvex set.

Proof. e sufficient condition is trivial. To justify the necessity, we show by induction that any P convex combination x D m iD1 i !i of elements in ˝ is an element of ˝ . is conclusion follows directly from the definition for m D 1; 2. Fix now a positive integer m  2 and suppose that every convex combination of k 2 N elements from ˝ , where k  m, belongs to ˝ . Form the convex combination mC1 mC1 X X y WD i !i ; i D 1; i  0 iD1

iD1

and observe that if mC1 D 1, then 1 D 2 D : : : D m D 0, so y D !mC1 2 ˝ . In the case where mC1 < 1 we get the representations m X iD1

i D 1

mC1 and

m X iD1

1

i D 1; mC1

which imply in turn the inclusion z WD

m X iD1

1

i !i 2 ˝: mC1

It yields therefore the relationships y D .1

mC1 /

m X iD1

1

i !i C mC1 !mC1 D .1 mC1

and thus completes the proof of the proposition.

mC1 /z C mC1 !mC1 2 ˝ 

Let ˝1 be a convex subset of Rn and let ˝2 be a convex subset of Rp . en the Cartesian product ˝1  ˝2 is a convex subset of Rn  Rp . Proposition 1.21

1.2. CONVEX SETS

Proof. Fix a D .a1 ; a2 /; b D .b1 ; b2 / 2 ˝1  ˝2 , and  2 .0; 1/. en we have a1 ; b1 2 ˝1 and a2 ; b2 2 ˝2 . e convexity of ˝1 and ˝2 gives us ai C .1

/bi 2 ˝i for i D 1; 2;

which implies therefore that a C .1

/b D a1 C .1

/b1 ; a2 C .1

 /b2 2 ˝1  ˝2 :

us the Cartesian product ˝1  ˝2 is convex.



Let us continue with the definition of affine mappings. Definition 1.22 A mapping B W Rn ! Rp is  if there is a linear mapping A W Rn ! Rp and an element b 2 Rp such that B.x/ D A.x/ C b for all x 2 Rn .

Every linear mapping is affine. Moreover, B W Rn ! Rp is affine if and only if B.x C .1

/y/ D B.x/ C .1

/B.y/ for all x; y 2 Rn and  2 R :

Now we show that set convexity is preserved under affine operations. Proposition 1.23 Let B W Rn ! Rp be an affine mapping. Suppose that ˝ is a convex subset of Rn and  is a convex subset of Rp . en B.˝/ is a convex subset of Rp and B 1 ./ is a convex subset of Rn .

Proof. Fix any a; b 2 B.˝/ and  2 .0; 1/. en a D B.x/ and b D B.y/ for x; y 2 ˝ . Since ˝ is convex, we have x C .1 /y 2 ˝ . en a C .1

/b D B.x/ C .1

/B.y/ D B.x C .1

/y/ 2 B.˝/;

which justifies the convexity of the image B.˝/. Taking now any x; y 2 B 1 ./ and  2 .0; 1/, we get B.x/; B.y/ 2  . is gives us  B.x/ C .1 /B.y/ D B x C .1 /y 2  by the convexity of  . us we have x C .1 the inverse image B 1 ./.

/y 2 B

1

./, which verifies the convexity of 

Let ˝1 ; ˝2  Rn be convex and let  2 R. en both sets ˝1 C ˝2 and ˝1 are also convex in Rn . Proposition 1.24

Proof. It follows directly from the definitions.



7

8

1. CONVEX SETS AND FUNCTIONS

Next we proceed with intersections of convex sets. Proposition 1.25 subset of Rn .

Let f˝˛ g˛2I be a collection of convex subsets of Rn . en

T

˛2I

˝˛ is also a convex

T Proof. Taking any a; b 2 ˛2I ˝˛ , we get that a; b 2 ˝˛ for all ˛ 2 I . e convexity of each T ˝˛ ensures that a C .1 /b 2 ˝˛ for any  2 .0; 1/. us a C .1 /b 2 ˛2I ˝˛ and T the intersection ˛2I ˝˛ is convex.  Definition 1.26

Let ˝ be a subset of Rn . e   of ˝ is defined by o \ n ˇˇ C ˇ C is convex and ˝  C : co ˝ WD

e next proposition follows directly from the definition and Proposition 1.25.

Figure 1.2: Nonconvex set and its convex hull.

Proposition 1.27

e convex hull co ˝ is the smallest convex set containing ˝ .

Proof. e convexity of the set co ˝  ˝ follows from Proposition 1.25. On the other hand, for any convex set C that contains ˝ we clearly have co ˝  C , which verifies the conclusion.  Proposition 1.28

For any convex subset ˝ of Rn , its convex hull admits the representation co ˝ D

m nX iD1

m ˇ X o ˇ i ai ˇ i D 1; i  0; ai 2 ˝; m 2 N : iD1

1.2. CONVEX SETS

Proof. Denoting by C the right-hand side of the representation to prove, we obviously have ˝  C . Let us check that the set C is convex. Take any a; b 2 C and get a WD

p X

˛i ai ;

iD1

Pp

where ai ; bj 2 ˝ , ˛i ; ˇj  0 with iD1 ˛i D for every number  2 .0; 1/, we have a C .1

b WD

/b D

p X iD1

Pq

j D1

q X

ˇj bj ;

j D1

ˇj D 1, and p; q 2 N . It follows easily that

˛i ai C

q X

.1

/ˇj bj :

j D1

en the resulting equality p X iD1

˛i C

q X

.1

j D1

/ˇj D 

p X iD1

˛i C .1

/

q X j D1

ˇj D 1

ensures that a C .1 /b 2 C , and thus co ˝  C by the definition of co ˝ . Fix now any a D Pm iD1 i ai 2 C with ai 2 ˝  co ˝ for i D 1; : : : ; m. Since the set co ˝ is convex, we conclude by Proposition 1.20 that a 2 co ˝ and thus co ˝ D C .  Proposition 1.29

e interior int ˝ and closure ˝ of a convex set ˝  Rn are also convex.

Proof. Fix a; b 2 int ˝ and  2 .0; 1/. en find an open set V such that a 2 V  ˝ and so a C .1

/b 2 V C .1

/b  ˝:

Since V C .1 /b is open, we get a C .1 /b 2 int ˝ , and thus the set int ˝ is convex. To verify the convexity of ˝ , we fix a; b 2 ˝ and  2 .0; 1/ and then find sequences fak g and fbk g in ˝ converging to a and b , respectively. By the convexity of ˝ , the sequence fak C .1 /bk g lies entirely in ˝ and converges to a C .1 /b . is ensures the inclusion a C .1 /b 2 ˝ and thus justifies the convexity of the closure ˝ .  To proceed further, for any a; b 2 Rn , define the interval ˇ ˚ Œa; b/ WD Œa; b n fbg D a C .1 /b ˇ  2 .0; 1 :

We can also define the intervals .a; b and .a; b/ in a similar way. Lemma 1.30 For a convex set ˝  Rn with nonempty interior, take any a 2 int ˝ and b 2 ˝ . en Œa; b/  int ˝ .

9

10

1. CONVEX SETS AND FUNCTIONS

Proof. Since b 2 ˝ , for any  > 0, we have b 2 ˝ C IB . Take now  2 .0; 1 and let x WD 2  a C .1 /b . Choosing  > 0 such that a C  IB  ˝ gives us  x C IB D a C .1 /b  C IB   a C .1 / ˝ C IB C IB D a C .1 /˝ C .1 /IB C IB h 2  i  aC IB C .1 /˝   ˝ C .1 /˝  ˝:

is shows that x 2 int ˝ and thus verifies the inclusion Œa; b/  int ˝ .



Now we establish relationships between taking the interior and closure of convex sets. Proposition 1.31

(i) int ˝ D ˝

Let ˝  Rn be a convex set with nonempty interior. en we have: and

.ii/ int ˝ D int ˝ .

Proof. (i) Obviously, int ˝  ˝ . For any b 2 ˝ and a 2 int ˝ , define the sequence fxk g by xk WD

 1 aC 1 k

1 b; k 2 N: k

Lemma 1.30 ensures that xk 2 int ˝ . Since xk ! b , we have b 2 int ˝ and thus verify (i). (ii) Since the inclusion int ˝  int ˝ is obvious, it remains to prove the opposite inclusion int ˝  int ˝ . To proceed, fix any b 2 int ˝ and a 2 int ˝ . If  > 0 is sufficiently small, then c WD b C .b

a/ 2 ˝ , and hence b D

 1 aC c 2 .a; c/  int ˝; 1C 1C

which verifies that int ˝  int ˝ and thus completes the proof.

1.3



CONVEX FUNCTIONS

is section collects basic facts about general (extended-real-valued) convex functions including their analytic and geometric characterizations, important properties as well as their specifications for particular subclasses. We also define convex set-valued mappings and use them to study a remarkable class of optimal value functions employed in what follows. Definition 1.32 Let f W ˝ ! R be an extended-real-valued function defined on a convex set ˝  Rn . en the function f is  on ˝ if  f x C .1 /y  f .x/ C .1 /f .y/ for all x; y 2 ˝ and  2 .0; 1/: (1.1)

1.3. CONVEX FUNCTIONS

11

If the inequality in (1.1) is strict for x ¤ y , then f is   on ˝ . Given a function f W ˝ ! R, the extension of f to Rn is defined by ( e.x/ WD f .x/ if x 2 ˝; f 1 otherwise: e is convex on Rn . Furthermore, Obviously, if f is convex on ˝ , where ˝ is a convex set, then f if f W Rn ! R is a convex function, then it is also convex on every convex subset of Rn . is allows to consider without loss of generality extended-real-valued convex functions on the whole space Rn .

Figure 1.3: Convex function and nonconvex function.

Definition 1.33

e  and  of f W Rn ! R are defined, respectively, by ˇ ˚ dom f WD x 2 Rn ˇ f .x/ < 1 and

ˇ ˇ ˚ ˚ epi f WD .x; t/ 2 Rn  R ˇ x 2 Rn ; t  f .x/ D .x; t / 2 Rn  R ˇ x 2 dom f; t  f .x/ :

Let us illustrate the convexity of functions by examples. Example 1.34

e following functions are convex:

(i) f .x/ WD ha; xi C b for x 2 Rn , where a 2 Rn and b 2 R. (ii) g.x/ WD kxk for x 2 Rn . (iii) h.x/ WD x 2 for x 2 R.

12

1. CONVEX SETS AND FUNCTIONS

Indeed, the function f in (i) is convex since  f x C .1 /y D ha; x C .1 /yi C b D ha; xi C .1 /ha; yi C b D .ha; xi C b/ C .1 /.ha; yi C b/ D f .x/ C .1 /f .y/ for all x; y 2 Rn and  2 .0; 1/: e function g in (ii) is convex since for x; y 2 Rn and  2 .0; 1/, we have  g x C .1 /y D kx C .1 /yk  kxk C .1 /kyk D g.x/ C .1

/g.y/;

which follows from the triangle inequality and the fact that k˛uk D j˛j  kuk whenever ˛ 2 R and u 2 Rn . e convexity of the simplest quadratic function h in (iii) follows from a more general result for the quadratic function on Rn given in the next example. Example 1.35 Let A be an n  n symmetric matrix. It is called positive semidefinite if hAu; ui  0 for all u 2 Rn . Let us check that A is positive semidefinite if and only if the function f W Rn ! R

defined by

1 hAx; xi; x 2 Rn ; 2 is convex. Indeed, a direct calculation shows that for any x; y 2 Rn and  2 .0; 1/ we have f .x/ WD

f .x/ C .1

/f .y/

f x C .1

 1 /y D .1 2

/hA.x

y/; x

yi:

(1.2)

If the matrix A is positive semidefinite, then hA.x y/; x yi  0, so the function f is convex by (1.2). Conversely, assuming the convexity of f and using equality (1.2) for x D u and y D 0 verify that A is positive semidefinite. e following characterization of convexity is known as the Jensen inequality. eorem 1.36 A function f W Rn ! R is convex if and only if for any numbers i  0 as i D P n 1; : : : ; m with m iD1 i D 1 and for any elements xi 2 R , i D 1; : : : ; m, it holds that

f

m X iD1

m  X i xi  i f .xi /:

(1.3)

iD1

Proof. Since (1.3) immediately implies the convexity of f , we only need to prove that any convex function f satisfies the Jensen inequality (1.3). Arguing by induction and taking into account that for m D 1 inequality (1.3) holds trivially and for m D 2 inequality (1.3) holds by the definition of convexity, we suppose that it holds for an integer m WD k with k  2. Fix numbers i  0, P n i D 1; : : : ; k C 1, with kC1 iD1 i D 1 and elements xi 2 R , i D 1; : : : ; k C 1. We obviously have the relationship k X i D 1 kC1 : iD1

1.3. CONVEX FUNCTIONS

13

If kC1 D 1, then i D 0 for all i D 1; : : : ; k and (1.3) obviously holds for m WD k C 1 in this case. Supposing now that 0  kC1 < 1, we get k X iD1

1

i D1 kC1

and by direct calculations based on convexity arrive at f

 kC1 X

i xi

iD1



h D f .1  .1 D .1  .1 D

kC1 X

kC1 /

Pk

iD1

i xi

C kC1 xkC1

i

1 kC1 P  k x  iD1 i i kC1 /f C kC1 f .xkC1 / 1 kC1 k X  i kC1 /f xi C kC1 f .xkC1 / 1 kC1 iD1

kC1 /

k X iD1

i f .xi / C kC1 f .xkC1 / 1 kC1

i f .xi /:

iD1

is justifies inequality (1.3) and completes the proof of the theorem.



e next theorem gives a geometric characterization of the function convexity via the convexity of the associated epigraphical set. A function f W Rn ! R is convex if and only if its epigraph epi f is a convex subset of the product space Rn  R.

eorem 1.37

Proof. Assuming that f is convex, fix pairs .x1 ; t1 /; .x2 ; t2 / 2 epi f and a number  2 .0; 1/. en we have f .xi /  ti for i D 1; 2. us the convexity of f ensures that  f x1 C .1 /x2  f .x1 / C .1 /f .x2 /  t1 C .1 /t2 :

is implies therefore that .x1 ; t1 / C .1

/.x2 ; t2 / D .x1 C .1

/x2 ; t1 C .1

/t2 / 2 epi f;

and thus the epigraph epi f is a convex subset of Rn  R. Conversely, suppose that the set epi f is convex and fix x1 ; x2 2 dom f and a number  2 .0; 1/. en .x1 ; f .x1 //; .x2 ; f .x2 // 2 epi f . is tells us that    x1 C .1 /x2 ; f .x1 / C .1 /f .x2 / D  x1 ; f .x1 / C .1 / x2 ; f .x2 / 2 epi f

14

1. CONVEX SETS AND FUNCTIONS

and thus implies the inequality f x1 C .1

 /x2  f .x1 / C .1

/f .x2 /;

which justifies the convexity of the function f .



Figure 1.4: Epigraphs of convex function and nonconvex function.

Now we show that convexity is preserved under some important operations.

Let fi W Rn ! R be convex functions for all i D 1; : : : ; m. en the following functions are convex as well: Proposition 1.38

(i) e multiplication by scalars f for any  > 0. P (ii) e sum function m iD1 fi . (iii) e maximum function max1im fi . Proof. e convexity of f in (i) follows directly from the definition. It is sufficient to prove (ii) and (iii) for m D 2, since the general cases immediately follow by induction.

(ii) Fix any x; y 2 Rn and  2 .0; 1/. en we have     f1 C f2 x C .1 /y D f1 x C .1 /y C f2 x C .1 /y  f1 .x/ C .1 /f1 .y/ C f2 .x/ C .1 /f2 .y/ D .f1 C f2 /.x/ C .1 /.f1 C f2 /.y/;

which verifies the convexity of the sum function f1 C f2 .

(iii) Denote g WD maxff1 ; f2 g and get for any x; y 2 Rn and  2 .0; 1/ that  fi x C .1 /y  fi .x/ C .1 /fi .y/  g.x/ C .1 /g.y/

for i D 1; 2. is directly implies that  ˚ g x C .1 /y D max f1 x C .1

 /y ; f2 x C .1

/y



 g.x/ C .1

which shows that the maximum function g.x/ D maxff1 .x/; f2 .x/g is convex.

/g.y/; 

1.3. CONVEX FUNCTIONS

15

e next result concerns the preservation of convexity under function compositions.

Let f W Rn ! R be convex and let ' W R ! R be nondecreasing and convex on a convex set containing the range of the function f . en the composition ' ı f is convex. Proposition 1.39

Proof. Take any x1 ; x2 2 Rn and  2 .0; 1/. en we have by the convexity of f that  f x1 C .1 /x2  f .x1 / C .1 /f .x2 /:

Since ' is nondecreasing and it is also convex, it follows that    ' ı f x1 C .1 /x2 D ' f x1 C .1 /x2   ' f .x1 / C .1 /f .x2 /   ' f .x1 / C .1 /' f .x2 / D .' ı f /.x1 / C .1 /.' ı f /.x2 /; which verifies the convexity of the composition ' ı f .



Now we consider the composition of a convex function and an affine mapping.

Let B W Rn ! Rp be an affine mapping and let f W Rp ! R be a convex function. en the composition f ı B is convex.

Proposition 1.40

Proof. Taking any x; y 2 Rn and  2 .0; 1/, we have    f ı B x C .1 /y D f .B.x C C .1 /B.y/  .1 /y// D f B.x/   f B.x/ C .1 /f B.y/ D .f ı B/.x/ C .1 /.f ı B/.y/

and thus justify the convexity of the composition f ı B .



e following simple consequence of Proposition 1.40 is useful in applications. Corollary 1.41 Let f W Rn ! R be a convex function. For any x; N d 2 Rn , the function 'W R ! R defined by '.t/ WD f .xN C td / is convex as well. Conversely, if for every x; N d 2 Rn the function ' defined above is convex, then f is also convex.

Proof. Since B.t/ D xN C td is an affine mapping, the convexity of ' immediately follows from Proposition 1.40. To prove the converse implication, take any x1 ; x2 2 Rn ,  2 .0; 1/ and let xN WD x2 , d WD x1 x2 . Since '.t/ D f .xN C td / is convex, we have    f x1 C .1 /x2 D f x2 C .x1 x2 / D './ D ' .1/ C .1 /.0/  '.1/ C .1 /'.0/ D f .x1 / C .1 /f .x2 /;

which verifies the convexity of the function f .



16

1. CONVEX SETS AND FUNCTIONS

e next proposition is trivial while useful in what follows. Proposition 1.42 Let f W Rn  Rp ! R be convex. For .x; N y/ N 2 Rn  Rp , the functions '.y/ WD f .x; N y/ and .x/ WD f .x; y/ N are also convex.

Now we present an important extension of Proposition 1.38(iii). Proposition 1.43 Let fi W Rn ! R for i 2 I be a collection of convex functions with a nonempty index set I . en the supremum function f .x/ WD supi2I fi .x/ is convex.

Proof. Fix x1 ; x2 2 Rn and  2 .0; 1/. For every i 2 I , we have  fi x1 C .1 /x2  fi .x1 / C .1 /fi .x2 /  f .x1 / C .1

/f .x2 /;

which implies in turn that f x1 C .1

 /x2 D sup fi x1 C .1 i2I

 /x2  f .x1 / C .1

/f .x2 /:

is justifies the convexity of the supremum function.



Our next intention is to characterize convexity of smooth functions of one variable. To proceed, we begin with the following lemma. Lemma 1.44 Given a convex function f W R ! R, assume that its domain is an open interval I . For any a; b 2 I and a < x < b , we have the inequalities

f .x/ x

f .a/ f .b/  a b

f .a/ f .b/  a b

a 2 .0; 1/. en a   a D f a C t.b a/ D f tb C .1

Proof. Fix a; b; x as above and form the numbers t WD f .x/ D f a C .x

  x a/ D f a C b

a b a

is gives us the inequalities f .x/  tf .b/ C .1 f .x/

f .a/  tf .b/ C .1

t/f .a/

x b

 t/a :

t/f .a/ and

 f .a/ D t f .b/

 x f .a/ D b

which can be equivalently written as f .x/ x

f .x/ : x

f .b/ f .a/  a b

f .a/ : a

a f .b/ a

 f .a/ ;

1.3. CONVEX FUNCTIONS

17

Similarly, we have the estimate f .x/

f .b/  tf .b/ C .1

t/f .a/

f .b/ D .t

 1/ f .b/

 x f .a/ D b

b f .b/ a

 f .a/ ;

which finally implies that f .b/ f .a/ f .b/  b a b and thus completes the proof of the lemma.

f .x/ x 

eorem 1.45 Suppose that f W R ! R is differentiable on its domain, which is an open interval I . en f is convex if and only if the derivative f 0 is nondecreasing on I .

Proof. Suppose that f is convex and fix a < b with a; b 2 I . By Lemma 1.44, we have f .x/ x

f .b/ f .a/  a b

f .a/ a

for any x 2 .a; b/. is implies by the derivative definition that f .b/ b

f 0 .a/ 

f .a/ : a

Similarly, we arrive at the estimate f .b/ b

f .a/  f 0 .b/ a

and conclude that f 0 .a/  f 0 .b/, i.e., f 0 is a nondecreasing function. To prove the converse, suppose that f 0 is nondecreasing on I and fix x1 < x2 with x1 ; x2 2 I and t 2 .0; 1/. en x1 < x t < x2 for x t WD tx1 C .1 t/x2 : By the mean value theorem, we find c1 ; c2 such that x1 < c1 < x t < c2 < x2 and f .x t / f .x t /

f .x1 / D f 0 .c1 /.x t f .x2 / D f 0 .c2 /.x t

x1 / D f 0 .c1 /.1 t /.x2 x1 /; x2 / D f 0 .c2 /t.x1 x2 /:

is can be equivalently rewritten as tf .x t / tf .x1 / D f 0 .c1 /t.1 t /.x2 x1 /; .1 t/f .x t / .1 t/f .x2 / D f 0 .c2 /t.1 t/.x1

x2 /:

Since f 0 .c1 /  f 0 .c2 /, adding these equalities yields f .x t /  tf .x1 / C .1

which justifies the convexity of the function f .

t/f .x2 /; 

18

1. CONVEX SETS AND FUNCTIONS

Corollary 1.46 Let f W R ! R be twice differentiable on its domain, which is an open interval I . en f is convex if and only if f 00 .x/  0 for all x 2 I .

Proof. Since f 00 .x/  0 for all x 2 I if and only if the derivative function f 0 is nondecreasing on this interval. en the conclusion follows directly from eorem 1.45. 

Example 1.47

Consider the function 8 0; otherwise:

2 > 0 for all x belonging to the domain of f , which x3 is I D .0; 1/. us this function is convex on R by Corollary 1.46.

To verify its convexity, we get that f 00 .x/ D

Next we define the notion of set-valued mappings (or multifunctions), which plays an important role in modern convex analysis, its extensions, and applications. Definition 1.48 We say that F is a -  between Rn and Rp and denote it by ! Rp if F .x/ is a subset of Rp for every x 2 Rn . e  and  of F are defined, F W Rn !

respectively, by

ˇ ˇ ˚ ˚ dom F WD x 2 Rn ˇ F .x/ ¤ ; and gph F WD .x; y/ 2 Rn  Rp ˇ y 2 F .x/ :

Any single-valued mapping F W Rn ! Rp is a particular set-valued mapping where the set F .x/ is a singleton for every x 2 Rn . It is essential in the following definition that the mapping F is set-valued. ! Rp and let ' W Rn  Rp ! R. e   or  Let F W Rn !  associated with F and ' is defined by ˇ ˚ .x/ WD inf '.x; y/ ˇ y 2 F .x/ ; x 2 Rn : (1.4) Definition 1.49

roughout this section we assume that .x/ > 1 for every x 2 Rn . ! Rp is of Assume that 'W Rn  Rp ! R is a convex function and that F W Rn ! convex graph. en the optimal value function  in (1.4) is convex. Proposition 1.50

1.3. CONVEX FUNCTIONS

19

Figure 1.5: Graph of set-valued mapping.

Proof. Take x1 ; x2 2 dom ,  2 .0; 1/. For any  > 0, find yi 2 F .xi / such that '.xi ; yi / < .xi / C  for i D 1; 2:

It directly implies the inequalities '.x1 ; y1 / < .x1 / C ;

.1

/'.x2 ; y2 / < .1

/.x2 / C .1

/:

Summing up these inequalities and employing the convexity of ' yield  ' x1 C .1 /x2 ; y1 C .1 /y2  '.x1 ; y1 / C .1 /'.x2 ; y2 / < .x1 / C .1 /.x2 / C : Furthermore, the convexity of gph F gives us  x1 C .1 /x2 ; y1 C .1 /y2 D .x1 ; y1 / C .1

/.x2 ; y2 / 2 gph F;

and therefore y1 C .1 /y2 2 F .x1 C .1 /x2 /. is implies that    x1 C .1 /x2  ' x1 C .1 /x2 ; y1 C .1 /y2 < .x1 / C .1 Letting finally  ! 0 ensures the convexity of the optimal value function .

/.x2 / C : 

Using Proposition 1.50, we can verify convexity in many situations. For instance, given two convex functions fi W Rn ! R, i D 1; 2, let '.x; y/ WD f1 .x/ C y and F .x/ WD Œf2 .x/; 1/. en the function ' is convex and set gph F D epi f2 is convex as well, and hence we justify the convexity of the sum .x/ D inf '.x; y/ D f1 .x/ C f2 .x/: y2F .x/

20

1. CONVEX SETS AND FUNCTIONS

Another example concerns compositions. Let f W Rp ! R be convex and let B W Rn ! Rp be affine. Define '.x; y/ WD f .y/ and F .x/ WD fB.x/g. Observe that ' is convex while F is of convex graph. us we have the convex composition  .x/ D inf '.x; y/ D f B.x/ ; x 2 Rn : y2F .x/

e examples presented above recover the results obtained previously by direct proofs. Now we establish via Proposition 1.50 the convexity of three new classes of functions.

Let ' W Rp ! R be convex and let B W Rp ! Rn be affine. Consider the set! Rp , define valued inverse image mapping B 1 W Rn ! ˇ ˚ f .x/ WD inf '.y/ ˇ y 2 B 1 .x/ ; x 2 Rn ;

Proposition 1.51

and suppose that f .x/ > 1 for all x 2 Rn . en f is a convex function. Proof. Let '.x; y/  '.y/ and F .x/ WD B

1

.x/. en the set ˇ ˚ gph F D .u; v/ 2 Rn  Rp ˇ B.v/ D u

is obviously convex. Since we have the representation f .x/ D

inf '.y/;

y2F .x/

x 2 Rn ;

the convexity of f follows directly from Proposition 1.50.

Proposition 1.52



For convex functions f1 ; f2 W Rn ! R, define the   ˇ ˚ .f1 ˚ f2 /.x/ WD inf f1 .x1 / C f2 .x2 / ˇ x1 C x2 D x

and suppose that .f1 ˚ f2 /.x/ > 1 for all x 2 Rn . en f1 ˚ f2 is also convex. Proof. Define ' W Rn  Rn ! R by '.x1 ; x2 / WD f1 .x1 / C f2 .x2 / and B W Rn  Rn ! Rn by B.x1 ; x2 / WD x1 C x2 . We have ˇ ˚ inf '.x1 ; x2 / ˇ .x1 ; x2 / 2 B 1 .x/ D .f1 ˚ f2 /.x/ for all x 2 Rn ;

which implies the convexity of .f1 ˚ f2 / by Proposition 1.51.



1.4. RELATIVE INTERIORS OF CONVEX SETS

21

p

Definition 1.53 A function g W R ! R is called   if     xi  yi for all i D 1; : : : ; p H) g.x1 ; : : : ; xp /  g.y1 ; : : : ; yp / :

Now we are ready to present the final consequence of Proposition 1.50 in this section that involves the composition. Proposition 1.54 Define h W Rn ! Rp by h.x/ WD .f1 .x/; : : : ; fp .x//, where fi W Rn ! R for i D 1; : : : ; p are convex functions. Suppose that g W Rp ! R is convex and nondecreasing componentwise. en the composition g ı h W Rn ! R is a convex function.

! Rp be a set-valued mapping defined by Proof. Let F W Rn ! F .x/ WD Œf1 .x/; 1/  Œf2 .x/; 1/  : : :  Œfp .x/; 1/:

en the graph of F is represented by ˇ ˚ gph F D .x; t1 ; : : : ; tp / 2 Rn  Rp ˇ ti  fi .x/ : Since all fi are convex, the set gph F is convex as well. Define further ' W Rn  Rp ! R by '.x; y/ WD g.y/ and observe, since g is increasing componentwise, that ˇ  ˚ inf '.x; y/ ˇ y 2 F .x/ D g f1 .x/; : : : ; fp .x/ D .g ı h/.x/; which ensures the convexity of the composition g ı h by Proposition 1.50.

1.4



RELATIVE INTERIORS OF CONVEX SETS

We begin this section with the definition and properties of affine sets. Given two elements a and b in Rn , the line connecting them is ˇ ˚ LŒa; b WD a C .1 /b ˇ  2 R : Note that if a D b , then LŒa; b D fag. Definition 1.55

A subset ˝ of Rn is  if for any a; b 2 ˝ we have LŒa; b  ˝ .

For instance, any point, line, and plane in R3 are affine sets. e empty set and the whole space are always affine. It follows from the definition that the intersection of any collection of affine sets is affine. is leads us to the construction of the affine hull of a set. Definition 1.56

e   of a set ˝  Rn is \˚ ˇ aff ˝ WD C ˇ C is affine and ˝  C :

22

1. CONVEX SETS AND FUNCTIONS

An element x in Rn of the form

xD

m X

i !i with

iD1

m X iD1

i D 1;

m 2 N;

is called an affine combination of !1 ; : : : ; !m . e proof of the next proposition is straightforward and thus is omitted. Proposition 1.57

e following assertions hold:

(i) A set ˝ is affine if and only if ˝ contains all affine combinations of its elements. (ii) Let ˝ , ˝1 , and ˝2 be affine subsets of Rn . en the sum ˝1 C ˝2 and the scalar product ˝ for any  2 R are also affine subsets of Rn . (iii) Let B W Rn ! Rp be an affine mapping. If ˝ is an affine subset of Rn and  is an affine subset of Rp , then the image B.˝/ is an affine subset of Rp and the inverse image B 1 ./ is an affine subset of Rn . (iv) Given ˝  Rn , its affine hull is the smallest affine set containing ˝ . We have

aff ˝ D

m nX iD1

m ˇ X o ˇ i D 1; !i 2 ˝; m 2 N : i !i ˇ iD1

(v) A set ˝ is a .linear/ subspace if and only if ˝ is an affine set containing the origin.

Next we consider relationships between affine sets and (linear) subspaces. Lemma 1.58 ! 2 ˝.

A nonempty subset ˝ of Rn is affine if and only if ˝

! is a subspace of Rn for any

Proof. Suppose that a nonempty set ˝  Rn is affine. It follows from Proposition 1.57(v) that ˝ ! is a subspace for any ! 2 ˝ . Conversely, fix ! 2 ˝ and suppose that ˝ ! is a subspace denoted by L. en the set ˝ D ! C L is obviously affine. 

e preceding lemma leads to the following notion. Definition 1.59

An affine set ˝ is  to a subspace L if ˝ D ! C L for some ! 2 ˝ .

e next proposition justifies the form of the parallel subspace. Proposition 1.60 Let ˝ be a nonempty, affine subset of Rn . en it is parallel to the unique subspace L of Rn given by L D ˝ ˝ .

1.4. RELATIVE INTERIORS OF CONVEX SETS

23

Proof. Given a nonempty, affine set ˝ , fix ! 2 ˝ and come up to the linear subspace L WD ˝ ! parallel to ˝ . To justify the uniqueness of such L, take any !1 ; !2 2 ˝ and any subspaces L1 ; L2  Rn such that ˝ D !1 C L1 D !2 C L2 . en L1 D !2 !1 C L2 . Since 0 2 L1 , we have !1 !2 2 L2 . is implies that !2 !1 2 L2 and thus L1 D !2 !1 C L2  L2 . Similarly, we get L2  L1 , which justifies that L1 D L2 . It remains to verify the representation L D ˝ ˝ . Let ˝ D ! C L with the unique subspace L and some ! 2 ˝ . en L D ˝ !  ˝ ˝ . Fix any x D u ! with u; ! 2 ˝ and observe that ˝ ! is a subspace parallel to ˝ . Hence ˝ ! D L by the uniqueness of L proved above. is ensures that x 2 ˝ ! D L and thus ˝ ˝  L. 

e uniqueness of the parallel subspace shows that the next notion is well defined.

e      ; ¤ ˝  Rn is the dimension of the linear subspace parallel to ˝ . Furthermore, the      ; ¤ ˝  Rn is the dimension of its affine hull aff ˝ .

Definition 1.61

To proceed further, we need yet another definition important in what follows. Definition 1.62

e elements v0 ; : : : ; vm in Rn , m  1, are   if m hX iD0

i vi D 0;

m X iD0

i   i D 0 H) i D 0 for all i D 0; : : : ; m :

It is easy to observe the following relationship with the linear independence. Proposition 1.63 e elements v0 ; : : : ; vm in Rn are affinely independent if and only if the elements v1 v0 ; : : : ; vm v0 are linearly independent.

Proof. Suppose that v0 ; : : : ; vm are affinely independent and consider the system m X iD1

i .vi

v0 / D 0; i.e., 0 v0 C

m X iD1

i vi D 0;

Pm Pm where 0 WD iD1 i . Since the elements v0 ; : : : ; vm are affinely independent and iD0 i D 0, we have that i D 0 for all i D 1; : : : ; m. us v1 v0 ; : : : ; vm v0 are linearly independent. e proof of the converse statement is straightforward. 

Recall that the span of some set C , span C , is the linear subspace generated by C . Lemma 1.64 Let ˝ WD afffv0 ; : : : ; vm g, where vi 2 Rn for all i D 0; : : : ; m. en the span of the set fv1 v0 ; : : : ; vm v0 g is the subspace parallel to ˝ .

24

1. CONVEX SETS AND FUNCTIONS

Proof. Denote by L the subspace parallel to ˝ . en ˝ v0 D L and therefore vi all i D 1; : : : ; m. is gives ˇ ˚ span vi v0 ˇ i D 1; : : : ; m  L:

v0 2 L for

To prove the converse inclusion, fix any v 2 L and get v C v0 2 ˝ . us we have v C v0 D

m X

m X

i vi ;

iD0

iD0

i D 1:

is implies the relationship vD

m X

i .vi

iD1

˚ v0 / 2 span vi

ˇ v0 ˇ i D 1; : : : ; m ;

which justifies the converse inclusion and hence completes the proof.



e proof of the next proposition is rather straightforward. Proposition 1.65 e elements v0 ; : : : ; vm are affinely independent in Rn if and only if its affine hull ˝ WD afffv0 ; : : : ; vm g is m-dimensional.

Proof. Suppose that v0 ; : : : ; vm are affinely independent. en Lemma 1.64 tells us that the subspace L WD spanfvi v0 j i D 1; : : : ; mg is parallel to ˝ . e linear independence of v1 v0 ; : : : ; vm v0 by Proposition 1.63 means that the subspace L is m-dimensional and so is ˝ . e proof of the converse statement is also straightforward. 

Affinely independent systems lead us to the construction of simplices. Definition 1.66

Let v0 ; : : : ; vm be affinely independent in Rn . en the set ˚ ˇ m WD co vi ˇ i D 0; : : : ; m

is called an m- in Rn with the vertices vi , i D 0; : : : ; m. An important role of simplex vertices is revealed by the following proposition.

Consider an m-simplex m with vertices vi for i D 0; : : : ; m. For every v 2 m , there is a unique element .0 ; : : : ; m / 2 RmC1 such that C Proposition 1.67

vD

m X iD0

i vi ;

m X iD0

i D 1:

1.4. RELATIVE INTERIORS OF CONVEX SETS

Proof. Let .0 ; : : : ; m / 2

RmC1 C

vD

m X iD0

and .0 ; : : : ; m / 2

i vi D

m X

i vi ;

iD0

mC1 RC

m X iD0

25

satisfy

i D

m X iD0

i D 1:

is immediately implies the equalities m X .i iD0

m X .i

i /vi D 0;

iD0

i / D 0:

Since v0 ; : : : ; vm are affinely independent, we have i D i for i D 0; : : : ; m.



Now we are ready to define a major notion of relative interiors of convex sets. Definition 1.68 Let ˝ be a convex set. We say that an element v 2 ˝ belongs to the   ri ˝ of ˝ if there exists  > 0 such that IB.vI / \ aff ˝  ˝ .

We begin the study of relative interiors with the following lemma. Lemma 1.69

Any linear mapping A W Rn ! Rp is continuous.

Proof. Let fe1 ; : : : ; en g be the standard orthonormal basis of Rn and let vi WD A.ei /, i D 1; : : : ; n. For any x 2 Rn with x D .x1 ; : : : ; xn /, we have n n n X  X X A.x/ D A x i ei D xi A.ei / D xi vi : iD1

iD1

iD1

en the triangle inequality and the Cauchy-Schwarz inequality give us v v u n u n n X uX uX t 2 kA.x/k  jxi jkvi k  jxi j t kvi k2 D M kxk; iD1

where M WD

qP n

iD1

iD1

iD1

kvi k2 . It follows furthermore that

kA.x/

A.y/k D kA.x

y/k  M kx

yk for all x; y 2 Rn ;

which implies the continuity of the mapping A. e next proposition plays an essential role in what follows. Proposition 1.70

Let m be an m-simplex in Rn with some m  1. en ri m ¤ ;.



26

1. CONVEX SETS AND FUNCTIONS

Proof. Consider the vertices v0 ; : : : ; vm of the simplex m and denote m

v WD

1 X vi : mC1 iD0

We prove the proposition by showing that v 2 ri m . Define ˇ ˚ L WD span vi v0 ˇ i D 1; : : : ; m and observe that L is the m-dimensional subspace of Rn parallel to aff m D afffv0 ; : : : ; vm g. It is easy to see that for every x 2 L there is a unique collection .0 ; : : : ; m / 2 RmC1 with xD

m X iD0

i vi ;

m X iD0

i D 0:

Consider the mapping A W L ! RmC1 , which maps x to the corresponding coefficients .0 ; : : : ; m / 2 RmC1 as above. en A is linear, and hence it is continuous by Lemma 1.69. Since A.0/ D 0, we can choose ı > 0 such that kA.u/k


0, and that f is subadditive if f .x C y/  f .x/ C f .y/ for all x; y 2 Rn . Show that a positively homogeneous function is convex if and only if it is subadditive.

Exercise 1.15 Given a nonempty set ˝  Rn , define

[ ˚ ˇ K˝ WD x ˇ   0; x 2 ˝ D ˝: 0

(i) Show that K˝ is a cone. (ii) Show that K˝ is the smallest cone containing ˝ . (iii) Show that if ˝ is convex, then the cone K˝ is convex as well.

Exercise 1.16 Let ' W Rn  Rp ! R be a convex function and let K be a nonempty, convex

subset of Rp . Suppose that for each x 2 Rn the function '.x; / is bounded below on K . Verify that the function on Rn defined by f .x/ WD inff'.x; y/ j y 2 Kg is convex.

Exercise 1.17 Let f W Rn ! R be a convex function.

(i) Show that for every ˛ 2 R the level set fx 2 Rn j f .x/  ˛g is convex. (ii) Let ˝  R be a convex set. Is it true in general that the inverse image f subset of Rn ?

1

.˝/ is a convex

1.6. EXERCISES FOR CHAPTER 1 nC1

Exercise 1.18 Let C be a convex subset of R

35

.

(i) For x 2 Rn , define F .x/ WD f 2 R j .x; / 2 C g. Show that F is a set-valued mapping with convex graph and give an explicit formula for its graph. (ii) For x 2 Rn , define the function ˇ ˚ fC .x/ WD inf  2 R ˇ .x; / 2 C : (1.9)

Find an explicit formula for fC when C is the closed unit ball of R2 . (iii) For the function fC defined in (ii), show that if fC .x/ > 1 for all x 2 Rn , then fC is a convex function.

Exercise 1.19 Let f W Rn ! R be bounded from below. Define its convexification .co f /.x/ WD inf

m nX iD1

m m ˇ o X X ˇ i f .xi / ˇ i  0; i D 1; i xi D x; m 2 N : iD1

(1.10)

iD1

Verify that (1.10) is convex with co f D fC , C WD co .epi f /, and fC defined in (1.9).

Exercise 1.20 We say that f W Rn ! R is quasiconvex if f x C .1

˚ /y/  max f .x/; f .y/ for all x; y 2 Rn ;  2 .0; 1/:

(i) Show that a function f is quasiconvex if and only if for any ˛ 2 R the level set fx 2 Rn j f .x/  ˛g is a convex set. (ii) Show that any convex function f W Rn ! R is quasiconvex. Give an example demonstrating that the converse is not true.

Exercise 1.21 Let a be a nonzero element in Rn and let b 2 R. Show that ˇ ˚ ˝ WD x 2 Rn ˇ ha; xi D b

is an affine set with dim ˝ D 1.

Exercise 1.22 Let the set fv1 ; : : : ; vm g consist of affinely independent elements and let v … aff fv1 ; : : : ; vm g. Show that v1 ; : : : ; vm ; v are affinely independent. Exercise 1.23 Suppose that ˝ is a convex subset of Rn with dim ˝ D m, m  1. Let the set fv1 ; : : : ; vm g  ˝ consist of affinely independent elements and let ˚ m WD co v1 ; : : : ; vm :

˚ Show that aff ˝ D aff m D aff v1 ; : : : ; vm .

36

1. CONVEX SETS AND FUNCTIONS

Exercise 1.24 Let ˝ be a nonempty, convex subset of Rn . (i) Show that aff ˝ D aff ˝ . (ii) Show that for any xN 2 ri ˝ and x 2 ˝ there exists t > 0 such that xN C t.xN (iii) Prove Proposition 1.73(ii).

x/ 2 ˝ .

Exercise 1.25 Let ˝1 ; ˝2  Rn be convex sets with ri ˝1 \ ri ˝2 ¤ ;. Show that: (i) ˝1 \ ˝2 D ˝ 1 \ ˝ 2 . (ii) ri .˝1 \ ˝2 / D ri ˝1 \ ri˝2 .

Exercise 1.26 Let ˝1 ; ˝2  Rn be convex sets with ˝ 1 D ˝ 2 . Show that ri ˝1 D ri ˝2 . Exercise 1.27 (i) Let B W Rn ! Rp be an affine mapping and let ˝ be a convex subset of Rn . Prove the equality

B.ri ˝/ D ri B.˝/:

(ii) Let ˝1 and ˝2 be convex subsets of Rn . Show that ri .˝1

˝2 / D ri ˝1

ri ˝2 .

Exercise 1.28 Let f W Rn ! R be a convex function. Show that: (i) aff .epi f / D ˚aff .domˇ f /  R : (ii) ri .epi f / D .x; / ˇ x 2 ri .dom f /;  > f .x/ :

Exercise 1.29 Find the explicit formulas for the distance function d.xI ˝/ and the Euclidean projection ˘.xI ˝/ in the following cases: n (i) ˝ is the ˇ of R . ˚ closed unit ball 2ˇ (ii) ˝ WD .x1 ; x2 / 2 R jx1 j C jx2 j  1 . (iii) ˝ WD Œ 1; 1  Œ 1; 1.

Exercise 1.30 Find the formulas for the projection ˘.xI ˝/ in the following cases:

ˇ ˚ (i) ˝ WD ˚x 2 Rn ˇˇha; xi D b , where 0 ¤ a 2 Rn and b 2 R : (ii) ˝ WD x 2 Rn ˇ Ax D b , where A is an m  n matrix with rank A D m and b 2 Rm . (iii) ˝ is the nonnegative orthant ˝ WD RnC .

Exercise 1.31 For ai  bi with i D 1; : : : ; n, define

ˇ ˚ ˝ WD x D .x1 ; : : : ; xn / 2 Rn ˇ ai  x  bi for all i D 1; : : : ; n :

Show that the projection ˘.xI ˝/ has the following representation: ˇ ˚ ˘.xI ˝/ D u D .u1 ; : : : ; un / 2 Rn ˇ ui D maxfai ; minfb; xi gg for i 2 f1; : : : ; ng :

37

CHAPTER

2

Subdifferential Calculus In this chapter we present basic results of generalized differentiation theory for convex functions and sets. A geometric approach of variational analysis based on convex separation for extremal systems of sets allows us to derive general calculus rules for normals to sets and subgradients of functions. We also include into this chapter some related results on Lipschitz continuity of convex functions and convex duality. Note that some extended versions of the presented calculus results can be found in exercises for this chapter, with hints to their proofs given at the end of the book.

2.1

CONVEX SEPARATION

We start this section with convex separation theorems, which play a fundamental role in convex analysis and particularly in deriving calculus rules by the geometric approach. e first result concerns the strict separation of a nonempty, closed, convex set and an out-of-set point.

Let ˝ be a nonempty, closed, convex subset of Rn and let xN … ˝ . en there is a nonzero element v 2 Rn such that ˇ ˚ N (2.1) sup hv; xi ˇ x 2 ˝ < hv; xi: Proposition 2.1

Proof. Denote wN WD ˘.xI N ˝/ and let v WD xN hv; x

wi N D hxN

w; N x

wN ¤ 0. Applying Proposition 1.78 gives us wi N  0 for any x 2 ˝:

It follows furthermore that hv; x

.xN

which implies in turn that hv; xi  hv; xi N

v/i D hv; x

wi N  0;

kvk2 and thus verifies (2.1).



In the setting of Proposition 2.1 we choose a number b 2 R such that ˇ ˚ sup hv; xi ˇ x 2 ˝ < b < hv; xi N and define the function A.x/ WD hv; xi for which A.x/ N > b and A.x/ < b for all x 2 ˝ . en we say that xN and ˝ are strictly separated by the hyperplane L WD fx 2 Rn j A.x/ D bg. e next theorem extends the result of Proposition 2.1 to the case of two nonempty, closed, convex sets at least one of which is bounded.

38

2. SUBDIFFERENTIAL CALCULUS

eorem 2.2 Let ˝1 and ˝2 be nonempty, closed, convex subsets of Rn with ˝1 \ ˝2 D ;. If ˝1 is bounded or ˝2 is bounded, then there is a nonzero element v 2 Rn such that

ˇ ˇ ˚ ˚ sup hv; xi ˇ x 2 ˝1 < inf hv; yi ˇ y 2 ˝2 :

Proof. Denote ˝ WD ˝1 ˝2 . en ˝ is a nonempty, closed, convex set and 0 … ˝ . Applying Proposition 2.1 to ˝ and xN D 0, we have ˇ ˚

WD sup hv; xi ˇ x 2 ˝ < 0 D hv; xi N with some 0 ¤ v 2 Rn :

For any x 2 ˝1 and y 2 ˝2 , we have hv; x yi  , and so hv; xi  C hv; yi. erefore, ˇ ˇ ˇ ˚ ˚ ˚ sup hv; xi ˇ x 2 ˝1  C inf hv; yi ˇ y 2 ˝2 < inf hv; yi ˇ y 2 ˝2 ; which completes the proof of the theorem.



Figure 2.1: Illustration of convex separation.

Next we show that two nonempty, closed, convex sets with empty intersection in Rn can be separated, while not strictly in general, if both sets are unbounded.

Let ˝1 and ˝2 be nonempty, closed, convex subsets of Rn such that ˝1 \ ˝2 D ;. en they are  in the sense that there is a nonzero element v 2 Rn with ˇ ˇ ˚ ˚ sup hv; xi ˇ x 2 ˝1  inf hv; yi ˇ y 2 ˝2 : (2.2) Corollary 2.3

2.1. CONVEX SEPARATION

39

Proof. For every fixed k 2 N , we define the set k WD ˝2 \ IB.0I k/ and observe that it is a closed, bounded, convex set, which is also nonempty if k is sufficiently large. Employing eorem 2.2 allows us to find 0 ¤ uk 2 Rn such that huk ; xi < huk ; yi for all x 2 ˝1 ; y 2 k : uk and assume without loss of generality that vk ! v ¤ 0 as k ! 1. Fix furkuk k ther x 2 ˝1 and y 2 ˝2 and take k0 2 N such that y 2 k for all k  k0 . en

Denote vk WD

hvk ; xi < hvk ; yi whenever k  k0 :

Letting k ! 1, we have hv; xi  hv; yi, which justifies the claim of the corollary.



Now our intention is to establish a general separation theorem for convex sets that may not be closed. To proceed, we verify first the following useful result.

Figure 2.2: Geometric illustration of Lemma 2.4.

Lemma 2.4 Let ˝ be a nonempty, convex subset of Rn such that 0 2 ˝ n ˝ . en there is a sequence xk ! 0 as k ! 1 with xk … ˝ for all k 2 N .

Proof. Recall that ri ˝ ¤ ; by eorem 1.72(i). Fix x0 2 ri ˝ and show that tx0 … ˝ for all t > 0; see Figure 2.2. Indeed, suppose the contrary that tx0 2 ˝ and get by eorem 1.72(ii) that t 1 0D x0 C . tx0 / 2 ri ˝  ˝; 1Ct 1Ct x0 gives us xk … ˝ for every k and xk ! 0.  a contradiction. Letting then xk WD k

40

2. SUBDIFFERENTIAL CALCULUS

Note that Lemma 2.4 may not hold for nonconvex sets. As an example, consider the set ˝ WD Rn nf0g for which 0 2 ˝ n ˝ while the conclusion of the Lemma 2.4 fails. Let ˝1 and ˝2 be nonempty, convex subsets of Rn with ˝1 \ ˝2 D ;. en they are separated in the sense of (2.2) with some v ¤ 0. eorem 2.5

Proof. Consider the set ˝ WD ˝1 the following two cases:

˝2 for which 0 … ˝ . We split the proof of the theorem into

Case 1: 0 … ˝ . In this case we apply Proposition 2.1 and find v ¤ 0 such that hv; zi < 0 for all z 2 ˝:

is gives us hv; xi < hv; yi whenever x 2 ˝1 and y 2 ˝2 , which justifies the statement.

Case 2: 0 2 ˝ . In this case we employ Lemma 2.4 and find a sequence xk ! 0 with xk … ˝ . en Proposition 2.1 produces a sequence fvk g of nonzero elements such that hvk ; zi < hvk ; xk i for all z 2 ˝:

Without loss of generality, we get (dividing by kvk k and extracting a convergent subsequence) that vk ! v and kvk k D kvk D 1. is gives us by letting k ! 1 that hv; zi  0 for all z 2 ˝:

To complete the proof, we just repeat the arguments of Case 1.



Figure 2.3: Extremal system of sets.

eorem 2.5 ensures the separation of convex sets, which do not intersect. To proceed now with separation of convex sets which may have common points as in Figure 2.3, we introduce the notion of set extremality borrowed from variational analysis that deals with both convex and nonconvex sets while emphasizing intrinsic optimality/extremality structures of problems under consideration; see, e.g., [14] for more details.

2.1. CONVEX SEPARATION

41

n

Given two nonempty, convex sets ˝1 ; ˝2  R , we say that the set system f˝1 ; ˝2 g is  if for any  > 0 there is a 2 Rn such that kak <  and

Definition 2.6

.˝1

a/ \ ˝2 D ;:

e next proposition gives us an important example of extremal systems of sets. Proposition 2.7 Let ˝ be a nonempty, convex subset of Rn and let xN 2 ˝ be a boundary point of ˝ . en the sets ˝ and fxg N form an extremal system.

Proof. Since xN 2 bd ˝ , for any number  > 0 we get IB.xI N =2/ \ ˝ c ¤ ;. us choosing b 2 IB.xI N =2/ \ ˝ c and denoting a WD b xN yield kak <  and .˝ a/ \ fxg N D ;. 

e following result is a separation theorem for extremal systems of convex sets. It admits a far-going extension to nonconvex settings, which is known as the extremal principle and is at the heart of variational analysis and its applications [14]. It is worth mentioning that, in contrast to general settings of variational analysis, we do not impose here any closedness assumptions on the sets in question. is will allow us to derive in the subsequent sections of this chapter various calculus rules for normals to convex sets and subgradients of convex functions by using a simple variational device without imposing closedness and lower semicontinuity assumptions conventional in variational analysis. Let ˝1 and ˝2 be two nonempty, convex subsets of Rn . en ˝1 and ˝2 form an extremal system if and only if there is a nonzero element v 2 Rn which separates the sets ˝1 and ˝2 in the sense of (2.2). eorem 2.8

Proof. Suppose that ˝1 and ˝2 form an extremal system of convex sets in Rn . en there is a sequence fak g in Rn with ak ! 0 and .˝1

ak / \ ˝2 D ;;

k 2N:

For every k 2 N , apply eorem 2.5 to the convex sets ˝1 nonzero elements uk 2 Rn satisfying huk ; x

e normalization vk WD

ak and ˝2 and find a sequence of

ak i  huk ; yi for all x 2 ˝1 and y 2 ˝2 ;

k 2N:

uk with kvk k D 1 gives us kuk k

hvk ; x

ak i  hvk ; yi for all x 2 ˝1 and y 2 ˝2 :

(2.3)

us we find v 2 Rn with kvk D 1 such that vk ! v as k ! 1 along a subsequence. Passing to the limit in (2.3) verifies the validity of (2.2).

42

2. SUBDIFFERENTIAL CALCULUS

For the converse implication, suppose that there is a nonzero element v 2 Rn which separates the sets ˝1 and ˝2 in the sense of (2.2). en it is not hard to see that  1  ˝1 v \ ˝2 D ; for every k 2 N : k us ˝1 and ˝2 form an extremal system.

2.2



NORMALS TO CONVEX SETS

In this section we apply the separation results obtained above (particularly, the extremal version in eorem 2.8) to the study of generalized normals to convex sets and derive basic rules of the normal cone calculus. Definition 2.9

Let ˝  Rn be a convex set with xN 2 ˝ . e   to ˝ at xN is ˇ ˚ N  0 for all x 2 ˝ : N.xI N ˝/ WD v 2 Rn ˇ hv; x xi

(2.4)

By convention, we put N.xI N ˝/ WD ; for xN … ˝ .

Figure 2.4: Epigraph of the absolute value function and the normal cone to it at the origin.

We first list some elementary properties of the normal cone defined above. Proposition 2.10

Let xN 2 ˝ for a convex subset ˝ of Rn . en we have:

(i) N.xI N ˝/ is a closed, convex cone containing the origin. (ii) If xN 2 int ˝ , then N.xI N ˝/ D f0g. Proof. To verify (i), fix vi 2 N.xI N ˝/ and i  0 for i D 1; 2. We get by the definition that 0 2 N.xI N ˝/ and hvi ; x xi N  0 for all x 2 ˝ implying h1 v1 C 2 v2 ; x

xi N  0 whenever x 2 ˝:

2.2. NORMALS TO CONVEX SETS

43

us 1 v1 C 2 v2 2 N.xI N ˝/, and hence N.xI N ˝/ is a convex cone. To check its closedness, fix a sequence fvk g  N.xI N ˝/ that converges to v . en passing to the limit in hvk ; x

xi N  0 for all x 2 ˝ as k ! 1

yields hv; x xi N  0 whenever x 2 ˝ , and so v 2 N.xI N ˝/. is completes the proof of (i). To check (ii), pick v 2 N.xI N ˝/ with xN 2 int ˝ and find ı > 0 such that xN C ıIB  ˝ . us for every e 2 IB we have the inequality hv; xN C ıe

xi N  0;

which shows that ıhv; ei  0, and hence hv; ei  0. Choose t > 0 so small that t v 2 IB . en hv; tvi  0 and therefore tkvk2 D 0, i.e., v D 0.  We start calculus rules with deriving rather simple while very useful results. Proposition 2.11 (i) Let ˝1 and ˝2 be nonempty, convex subsets of Rn and Rp , respectively. For .xN 1 ; xN 2 / 2 ˝1  ˝2 , we have

 N .xN 1 ; xN 2 /I ˝1  ˝2 D N.xN 1 I ˝1 /  N.xN 2 I ˝2 /:

(ii) Let ˝1 and ˝2 be convex subsets of Rn with xN i 2 ˝i for i D 1; 2. en N.xN 1 C xN 2 I ˝1 C ˝2 / D N.xN 1 I ˝1 / \ N.xN 2 I ˝2 /:

Proof. To verify (i), fix .v1 ; v2 / 2 N..xN 1 ; xN 2 /I ˝1  ˝2 / and get by the definition that h.v1 ; v2 /; .x1 ; x2 /

.xN 1 ; xN 2 /i D hv1 ; x1

xN 1 i C hv2 ; x2

xN 2 i  0

(2.5)

whenever .x1 ; x2 / 2 ˝1  ˝2 . Putting x2 WD xN 2 in (2.5) gives us hv1 ; x1

xN 1 i  0 for all x1 2 ˝1 ;

which means that v1 2 N.xN 1 I ˝1 /. Similarly, we obtain v2 2 N.xN 2 I ˝2 / and thus justify the inclusion “” in (2.5). e opposite inclusion is obvious. Let us now verify (ii). Fix v 2 N.xN 1 C xN 2 I ˝1 C ˝2 / and get by the definition that hv; x1 C x2

.xN 1 C xN 2 /i  0 whenever x1 2 ˝1 ; x2 2 ˝2 :

Putting there x1 WD xN 1 and x2 WD xN 2 gives us v 2 N.xN 1 I ˝1 / \ N.xN 2 I ˝2 /. e opposite inclusion in (ii) is also straightforward. 

44

2. SUBDIFFERENTIAL CALCULUS

Consider further a linear mapping A W Rn ! Rp and associate it with a p  n matrix as usual. e adjoint mapping A W Rp ! Rn is defined by hAx; yi D hx; A yi for all x 2 Rn ; y 2 Rp ;

which corresponds to the matrix transposition. e following calculus result describes normals to solution sets for systems of linear equations. Proposition 2.12 Let BW Rn ! Rp be defined by B.x/ WD Ax C b , where A W Rn ! Rp is a linear mapping and b 2 Rp . Given c 2 R, consider the solution set ˇ ˚ ˝ WD x 2 Rn ˇ Bx D c :

For any xN 2 ˝ with B.x/ N D c , we have

ˇ ˚ N.xI N ˝/ D im A D v 2 Rn ˇ v D A y; y 2 Rp :

(2.6)

Proof. Take v 2 N.xI N ˝/ and get hv; x xi N  0 for all x such that Ax C b D c . Fixing any u 2 ker A, we see that A.xN u/ D AxN D c b , i.e., A.xN u/ C b D c , which yields hv; ui  0 whenever u 2 ker A. Since u 2 ker A, it follows that hv; ui D 0 for all u 2 ker A. Arguing by contradiction, suppose that there is no y 2 Rp with v D A y . us v … W WD A Rp , where the set W  Rn is obviously nonempty, closed, and convex. Employing Proposition 2.1 gives us a nonzero element uN 2 Rn satisfying ˇ ˚ N vi; sup hu; N wi ˇ w 2 W < hu;

and implying that 0 < hu; N vi since 0 2 W . Furthermore thu; N A yi D hu; N A .ty/i < hu; N vi for all t 2 R; y 2 Rp :

Hence hu; N A yi D 0, which shows that hAu; N yi D 0 whenever y 2 Rp , i.e., AuN D 0. us uN 2 ker A while hv; ui N > 0. is contradiction justifies the inclusion “” in (2.6). To verify the opposite inclusion in (2.6), fix any v 2 Rn with v D A y for some y 2 Rp . For any x 2 ˝ , we have hv; x

xi N D hA y; x

xi N D hy; Ax

Axi N D 0;

which means that v 2 N.xI N ˝/ and thus completes the proof of the proposition.



e following proposition allows us to characterize the separation of convex (generally nonclosed) sets in terms of their normal cones.

Let ˝1 and ˝2 be convex subsets of Rn with xN 2 ˝1 \ ˝2 . en the following assertions are equivalent: Proposition 2.13

2.2. NORMALS TO CONVEX SETS

45

n

(i) f˝1 ; ˝2 g forms an extremal system of two convex sets in R . (ii) e sets ˝1 and in the sense of (2.2) with some v ¤ 0.  ˝2 are separated  (iii) N.xI N ˝1 / \ N.xI N ˝2 / ¤ f0g. Proof. e equivalence of (i) and (ii) follows from eorem 2.8. Suppose that (ii) is satisfied. Since xN 2 ˝2 , we have ˇ ˚ sup hv; xi ˇ x 2 ˝1  hv; xi; N

and thus hv; x xi N  0 for all x 2 ˝1 , i.e., v 2 N.xI N ˝1 /. Similarly, we can verify the inclusion v 2 N.xI N ˝2 /, and hence arrive at (iii).   Now suppose that (iii) is satisfied and let 0 ¤ v 2 N.xI N ˝1 / \ N.xI N ˝2 / . en (2.2) follows directly from the definition of the normal cone. Indeed, for any x 2 ˝1 and y 2 ˝2 one has hv; x xi N  0 and h v; y xi N  0: is implies hv; x

xi N  0  hv; y

xi N , and hence hv; xi  hv; yi.



e following corollary provides a normal cone characterization of boundary points.

Let ˝ be a nonempty, convex subset of Rn and let xN 2 ˝ . en xN 2 bd ˝ if and only if the normal cone to ˝ is nontrivial, i.e., N.xI N ˝/ ¤ f0g.

Corollary 2.14

Proof. e “only if ” part follows from Proposition 2.10(ii). To justify the “if ” part, take xN 2 bd ˝ and consider the sets ˝1 WD ˝ and ˝2 WD fxg N , which form an extremal system by Proposition 2.7. Note that N.xI N ˝2 / D Rn . us the claimed result is a direct consequence of Proposition 2.13 since N.xI N ˝1 / \ Œ N.xI N ˝2 / D N.xI N ˝1 / ¤ f0g. 

Now we are ready to establish a major result of the normal cone calculus for convex sets providing a relationship between normals to set intersections and normals to each set in the intersection. Its proof is also based on convex separation in extremal systems of sets. Note that this approach allows us to capture general convex sets, which may not be closed. eorem 2.15 Let ˝1 and ˝2 be convex subsets of Rn with xN 2 ˝1 \ ˝2 . en for any v 2 N.xI N ˝1 \ ˝2 / there exist   0 and vi 2 N.xI N ˝i /, i D 1; 2, such that

v D v1 C v2 ; .; v1 / ¤ .0; 0/:

Proof. Fix v 2 N.xI N ˝1 \ ˝2 / and get by the definition that hv; x

xi N  0 for all x 2 ˝1 \ ˝2 :

(2.7)

46

2. SUBDIFFERENTIAL CALCULUS

Denote 1 WD ˝1  Œ0; 1/ and 2 WD f.x; / j x 2 ˝2 ;   hv; x xig N . ese convex sets form an extremal system since we obviously have .x; N 0/ 2 1 \ 2 and .0; a// D ; whenever a > 0:

1 \ .2

By eorem 2.8, find 0 ¤ .w; / 2 Rn  R such that hw; xi C 1  hw; yi C 2 for all .x; 1 / 2 1 ; .y; 2 / 2 2 :

(2.8)

It is easy to see that  0. Indeed, assuming > 0 and taking into account that .x; N k/ 2 1 when k > 0 and .x; N 0/ 2 2 , we get hw; xi N C k  hw; xi; N

which leads to a contradiction. ere are two possible cases to consider.

Case 1: D 0. In this case we have w ¤ 0 and hw; xi  hw; yi for all x 2 ˝1 ; y 2 ˝2 : N ˝1 / \ Œ N.xI N ˝2 /, and thus en following the proof of Proposition 2.13 gives us w 2 N.xI (2.7) holds for  D 0, v1 D w , and v2 D w .

Case 2: < 0. In this case we let  WD x 2 ˝1 , and .x; N 0/ 2 2 , that

> 0 and deduce from (2.8), by using .x; 0/ 2 1 when

hw; xi  hw; xi N for all x 2 ˝1 : w 2 N.xI N ˝1 /. To proceed further, we get from (2.8),  due to .x; N 0/ 2 1 and .y; ˛/ 2 2 for all y 2 ˝2 with ˛ WD hv; y xi N , that D E D E w; xN  w; y C hv; y xi N whenever y 2 ˝2 :

is implies that w 2 N.xI N ˝1 /, and hence

Dividing both sides by gives us Dw E Dw E ; xN  ; y C hv; y

is clearly implies the inequality Dw C v; y

xi N whenever y 2 ˝2 :

E xN  0 for all y 2 ˝2 ;

w w w Cv D C v 2 N.xI N ˝2 /. Letting finally v1 WD 2 N.xI N ˝1 / and v2 WD

  v 2 N.xI N ˝2 / gives us v D v1 C v2 , which ensures the validity of (2.7) with  D 1.

and thus

w C  

2.2. NORMALS TO CONVEX SETS

47

Figure 2.5: Normal cone to set intersection.

Observe that representation (2.7) does not provide a calculus rule for representing normals to set intersections unless  ¤ 0. However, it is easy to derive such an intersection rule directly from eorem 2.15 while imposing the following basic qualification condition, which plays a crucial role in the subsequent calculus results; see Figure 2.5.

Let ˝1 and ˝2 be convex subsets of Rn with xN 2 ˝1 \ ˝2 . Assume that the basic qualification condition (BQC) is satisfied:   N.xI N ˝1 / \ N.xI N ˝2 / D f0g: (2.9)

Corollary 2.16

en we have the normal cone intersection rule (2.10)

N.xI N ˝1 \ ˝2 / D N.xI N ˝1 / C N.xI N ˝2 /:

Proof. For any v 2 N.xI N ˝1 \ ˝2 /, we find by eorem 2.15 a number   0 and elements vi 2 N.xI N ˝i / as i D 1; 2 such that the conditions in (2.7) hold. Assuming that  D 0 in (2.7) gives us that 0 ¤ v1 D v2 2 N.xI N ˝1 / \ Œ N.xI N ˝2 /, which contradicts BQC (2.9). us  > 0 and v D v1 = C v2 = 2 N.xI N ˝1 / C N.xI N ˝2 /. Hence the inclusion “” in (2.10) is satisfied. e opposite inclusion in (2.10) can be proved easily using the definition of the normal cone. Indeed, fix any v 2 N.xI N ˝1 / C N.xI N ˝2 / with v D v1 C v2 , where vi 2 N.xI N ˝i / for i D 1; 2. For any x 2 ˝1 \ ˝2 , one has hv; x

xi N D hv1 C v2 ; x

xi N D hv1 ; x

xi N C hv2 ; x

xi N  0;

which implies v 2 N.xI N ˝1 \ ˝2 / and completes the proof of the corollary.



48

2. SUBDIFFERENTIAL CALCULUS

e following example shows that the intersection rule (2.10) does not hold generally without the validity of the basic qualification condition (2.9). Example 2.17 Let ˝1 WD f.x; / 2 R2 j   x 2 g and let ˝2 WD f.x; / 2 R2 j   x 2 g. en for xN D .0; 0/ we have N.xI N ˝1 \ ˝2 / D R2 while N.xI N ˝1 / C N.xI N ˝2 / D f0g  R.

Next we present an easily verifiable condition ensuring the validity of BQC (2.9). More general results related to relative interior are presented in Exercise 2.13 and its solution. Corollary 2.18 Let ˝1 ; ˝2  Rn be such that int ˝1 \ ˝2 ¤ ; or int ˝2 \ ˝1 ¤ ;. en for any xN 2 ˝1 \ ˝2 both BQC (2.9) and intersection rule (2.10) are satisfied.

Proof. Consider for definiteness the case where int ˝1 \ ˝2 ¤ ;. Let us fix xN 2 ˝1 \ ˝2 and show that BQC (2.9) holds, which ensures the intersection rule (2.10) by Corollary 2.16. Arguing by contradiction, suppose the opposite   N.xI N ˝1 / \ N.xI N ˝2 / ¤ f0g

and then find by Proposition 2.13 a nonzero element v 2 Rn that separates the sets ˝1 and ˝2 as in (2.2). Fixing now uN 2 int ˝1 \ ˝2 gives us a number ı > 0 such that uN C ıIB  ˝1 . It ensures by (2.2) that hv; xi  hv; ui N for all x 2 uN C ıIB: is implies in turn the inequality

hv; ui N C ıhv; ei  hv; ui N whenever e 2 IB v and tells us that hv; ei  0 for all e 2 IB . Choosing finally e WD , we arrive at a contradiction kvk and thus complete the proof of this corollary. 

Employing induction arguments allows us to derive a counterpart of Corollary 2.16 (and, similarly, of Corollary 2.18) for finitely many sets. T Corollary 2.19 Let ˝i  Rn for i D 1; : : : ; m be convex sets and let xN 2 m iD1 ˝i . Assume the validity of the following qualification condition: m hX iD1

i   vi D 0; vi 2 N.xI N ˝i / H) vi D 0 for all i D 1; : : : ; m :

en we have the intersection formula m m  \  X N xI N ˝i D N.xI N ˝i /: iD1

iD1

2.3. LIPSCHITZ CONTINUITY OF CONVEX FUNCTIONS

2.3

49

LIPSCHITZ CONTINUITY OF CONVEX FUNCTIONS

In this section we study the fundamental notion of Lipschitz continuity for convex functions. is notion can be treated as “continuity at a linear rate” being highly important for many aspects of convex and variational analysis and their applications. Definition 2.20 A function f W Rn ! R is L  on a set ˝  dom f if there is a constant `  0 such that

jf .x/

f .y/j  `kx

yk for all x; y 2 ˝:

(2.11)

We say that f is  L  around xN 2 dom f if there are constants `  0 and ı > 0 such that (2.11) holds with ˝ D IB.xI N ı/. Here we give several verifiable conditions that ensure Lipschitz continuity and characterize its localized version for general convex functions. One of the most effective characterizations of local Lipschitz continuity, holding for nonconvex functions as well, is given via the following notion of the singular (or horizon) subdifferential, which was largely underinvestigated in convex analysis and was properly recognized only in the framework of variational analysis; see [14, 27]. Note that, in contrast to the (usual) subdifferential of convex analysis defined in the next section, the singular subdifferential does not reduce to the classical derivative for differentiable functions.

Let f W Rn ! R be a convex function and let xN 2 dom f . e   of the function f at xN is defined by ˇ   ˚ N f .x/ N I epi f : (2.12) @1 f .x/ N WD v 2 Rn ˇ .v; 0/ 2 N .x; Definition 2.21

e following example illustrates the calculation of @1 f .x/ N based on definition (2.12). Example 2.22

Consider the function f .x/ WD

( 0 1

if jxj  1;

otherwise:

en epi f D Œ 1; 1  Œ0; 1/ and for xN D 1 we have  N .x; N f .x//I N epi f D Œ0; 1/  . 1; 0; which shows that @1 f .x/ N D Œ0; 1/; see Figure 2.6. e next proposition significantly simplifies the calculation of @1 f .x/ N . Proposition 2.23

For a convex function f W Rn ! R with xN 2 dom f , we have @1 f .x/ N D N.xI N dom f /:

50

2. SUBDIFFERENTIAL CALCULUS

Figure 2.6: Singular subgradients.

Proof. Fix any v 2 @1 f .x/ N with x 2 dom f and observe that .x; f .x// 2 epi f . en using (2.12) and the normal cone construction (2.4) give us  hv; x xi N D hv; x xi N C 0 f .x/ f .x/ N  0;

which shows that v 2 N.xI N dom f /. Conversely, suppose that v 2 N.xI N dom f / and fix any .x; / 2 epi f . en we have f .x/  , and so x 2 dom f . us hv; x

xi N C 0.

f .x// N D hv; x

xi N  0;

which implies that .v; 0/ 2 N..x; N f .x//I N epi f /, i.e., v 2 @1 f .x/ N .



To proceed with the study of Lipschitz continuity of convex functions, we need the following two lemmas, which are of their own interest. Lemma 2.24

Let fei j i D 1; : : : ; ng be the standard orthonormal basis of Rn . Denote ˇ ˚ A WD xN ˙ ei ˇ i D 1; : : : ; n ;  > 0:

en the following properties hold: (i) xN C ei 2 co A for j j   and i D 1; : : : ; n. (ii) IB.xI N =n/  co A. Proof. (i) For j j   , find t 2 Œ0; 1 with D t . / C .1 xN C ei D t.xN

ei / C .1

t/ . en xN ˙ ei 2 A gives us

t /.xN C ei / 2 co A:

2.3. LIPSCHITZ CONTINUITY OF CONVEX FUNCTIONS

51

(ii) For x 2 IB.xI N =n/, we have x D xN C .=n/u with kuk  1. Represent u via fei g by uD

where ji j 

qP n

iD1

n X

i ei ;

iD1

2i D kuk  1 for every i . is gives us

x D xN C

 X i X 1  xN C i ei : u D xN C ei D n n n n

n

iD1

iD1

Denoting i WD i yields j i j   . It follows from (i) that xN C i ei D xN C i ei 2 co A, and thus x 2 co A since it is a convex combination of elements in co A.  Lemma 2.25 If a convex function f W Rn ! R is bounded above on IB.xI N ı/ for some element xN 2 dom f and number ı > 0, then f is bounded on IB.xI N ı/.

Proof. Denote m WD f .x/ N and take M > 0 with f .x/  M for all x 2 IB.xI N ı/. Picking any u 2 IB.xI N ı/, consider the element x WD 2xN u. en x 2 IB.xI N ı/ and m D f .x/ N Df

which shows that f .u/  2m

x C u 1 1  f .x/ C f .u/; 2 2 2

f .x/  2m

M and thus f is bounded on IB.xI N ı/.



e next major result shows that the local boundedness of a convex function implies its local Lipschitz continuity. eorem 2.26 Let f W Rn ! R be convex with xN 2 dom f . If f is bounded above on IB.xI N ı/ for some ı > 0, then f is Lipschitz continuous on IB.xI N ı=2/.

Proof. Fix x; y 2 IB.xI N ı=2/ with x ¤ y and consider the element u WD x C

Since e WD ˛ WD

ı 2kx

x kx yk

ı 2kx

yk

 x

 y :

y ı ı ı 2 IB , we have u D x C e 2 xN C IB C IB  xN C ıIB . Denoting further yk 2 2 2

gives us u D x C ˛.x xD

y/, and thus 1 ˛ uC y: ˛C1 ˛C1

52

2. SUBDIFFERENTIAL CALCULUS

It follows from the convexity of f that f .x/ 

˛ 1 f .u/ C f .y/; ˛C1 ˛C1

which implies in turn the inequalities f .x/

f .y/ 

1 .f .u/ ˛C1

f .y//  2M

1 2kx yk 4M D 2M  kx ˛C1 ı C 2kx yk ı

yk

with M WD supfjf .x/j j x 2 IB.xI N ı/g < 1 by Lemma 2.25. Interchanging the role of x and y above, we arrive at the estimate jf .x/

f .y/j 

4M kx ı

yk

and thus verify the Lipschitz continuity of f on IB.xI N ı=2/.



e following two consequences of eorem 2.26 ensure the automatic local Lipschitz continuity of a convex function on appropriate sets.

Any extended-real-valued convex function f W Rn ! R is locally Lipschitz continuous on in the interior of its domain int.dom f /. Corollary 2.27

Proof. Pick xN 2 int.dom f / and choose  > 0 such that xN ˙ ei 2 dom f for every i . Considering the set A from Lemma 2.24, we get IB.xI N =n/  co A by assertion (ii) of this lemma. Denote M WD maxff .a/ j a 2 Ag < 1 over the finite set A. Using the representation xD

m X iD1

i ai with i  0

m X iD1

i D 1; ai 2 A

for any x 2 IB.xI N =n/ shows that f .x/ 

m X iD1

i f .ai / 

m X iD1

i M D M;

and so f is bounded above on IB.xI N =n/. en eorem 2.26 tells us that f is Lipschitz continuous on IB.xI N =2n/ and thus it is locally Lipschitz continuous on int .dom f /.  It immediately follows from Corollary 2.27 that any finite convex function on Rn is always locally Lipschitz continuous on the whole space. Corollary 2.28

If f W Rn ! R is convex, then it is locally Lipschitz continuous on Rn .

2.4. SUBGRADIENTS OF CONVEX FUNCTIONS

53

e final result of this section provides several characterizations of the local Lipschitz continuity of extended-real-valued convex functions. eorem 2.29

Let f W Rn ! R be convex with xN 2 dom f . e following are equivalent:

(i) f is continuous at xN . (ii) xN 2 int.dom f /. (iii) f is locally Lipschitz continuous around xN . (iv) @1 f .x/ N D f0g. Proof. To verify (i)H)(ii), by the continuity of f at xN , for any  > 0 find ı > 0 such that jf .x/

f .x/j N <  whenever x 2 IB.xI N ı/:

Since f .x/ N < 1, this implies that IB.xI N ı/  dom f , and so xN 2 int.dom f /. Implication (ii)H)(iii) follows directly from Corollary 2.27 while (iii)H)(i) is obvious. us conditions (i), (ii), and (iii) are equivalent. To show next that (ii)H)(iv), take xN 2 int.dom f / for which N.xI N dom f / D f0g. en Proposition 2.23 yields @1 f .x/ N D f0g. To verify finally (iv)H)(ii), we argue by contradiction and suppose that xN … int.dom f /. en xN 2 bd.dom f / and it follows from Corollary 2.14 that N.xI N dom f / ¤ f0g, which contradicts the condition @1 f .x/ N D f0g by Proposition 2.23. 

2.4

SUBGRADIENTS OF CONVEX FUNCTIONS

In this section we define and start studying the concept of subdifferential (or the collection of subgradients) for convex extended-real-valued functions. is notion of generalized derivative is one of the most fundamental in analysis, with a variety of important applications. e revolutionary idea behind the subdifferential concept, which distinguishes it from other notions of generalized derivatives in mathematics, is its set-valuedness as the indication of nonsmoothness of a function around the reference individual point. It not only provides various advantages and flexibility but also creates certain difficulties in developing its calculus and applications. e major techniques of subdifferential analysis revolve around convex separation of sets.

Let f W Rn ! R be a convex function and let xN 2 dom f . An element v 2 Rn is called a  of f at xN if Definition 2.30

hv; x

xi N  f .x/

f .x/ N for all x 2 Rn :

(2.13)

e collection of all the subgradients of f at xN is called the  of the function at this point and is denoted by @f .x/ N .

54

2. SUBDIFFERENTIAL CALCULUS

Figure 2.7: Subdifferential of the absolute value function at the origin.

Note that it suffices to take only x 2 dom f in the subgradient definition (2.13). Observe also that @f .x/ N ¤ ; under more general conditions; see, e.g., Proposition 2.47 as well as other results in this direction given in [26]. e next proposition shows that the subdifferential @f .x/ N can be equivalently defined geometrically via the normal cone to the epigraph of f at .x; N f .x// N . Proposition 2.31

For any convex function f W Rn ! R and any xN 2 dom f , we have ˇ  ˚ N f .x//I N epi f : @f .x/ N D v 2 Rn ˇ .v; 1/ 2 N .x;

(2.14)

Proof. Fix v 2 @f .x/ N and .x; / 2 epi f . Since   f .x/, we deduce from (2.13) that ˝ .v; 1/; .x; /

˛ .x; N f .x// N D hv; x  hv; x

xi N xi N

. f .x// N .f .x/ f .x// N  0:

is readily implies that .v; 1/ 2 N..x; N f .x//I N epi f /. To verify the opposite inclusion in (2.14), take an arbitrary element v 2 Rn such that .v; 1/ 2 N..x; N f .x//I N epi f /. For any x 2 dom f , we have .x; f .x// 2 epi f . us hv; x

xi N

.f .x/

˝ f .x// N D .v; 1/; .x; f .x//

which shows that v 2 @f .x/ N and so justifies representation (2.14).

˛ .x; N f .x// N  0;



Along with representing the subdifferential via the normal cone in (2.14), there is the following simple way to express the normal cone to a set via the subdifferential (as well as the singular subdifferential (2.12)) of the extended-real-indicator function.

2.4. SUBGRADIENTS OF CONVEX FUNCTIONS

Example 2.32

55

n

Given a nonempty, convex set ˝  R , define its indicator function by ( 0 if x 2 ˝; f .x/ D ı.xI ˝/ WD 1 otherwise:

(2.15)

en epi f D ˝  Œ0; 1/, and we have by Proposition 2.11(i) that  N .x; N f .x//I N epi f D N.xI N ˝/  . 1; 0; which readily implies the relationships @f .x/ N D @1 f .x/ N D N.xI N ˝/:

(2.16)

A remarkable feature of convex functions is that every local minimizer for a convex function gives also the global/absolute minimum to the function.

Let f W Rn ! R convex and let xN 2 dom f be a local minimizer of f . en f attains its global minimum at this point. Proposition 2.33

Proof. Since xN is a local minimizer of f , there is ı > 0 such that f .u/  f .x/ N for all u 2 IB.xI N ı/:

Fix x 2 Rn and construct a sequence of xk WD .1 k 1 /xN C k 1 x as k 2 N . us we have xk 2 IB.xI N ı/ when k is sufficiently large. It follows from the convexity of f that f .x/ N  f .xk /  .1

which readily implies that k

1

f .x/ N k

1

k

1

/f .x/ N Ck

1

f .x/;

f .x/, and hence f .x/ N  f .x/ for all x 2 Rn .



Next we recall the classical notion of derivative for functions on Rn , which we understand throughout the book in the sense of Fréchet unless otherwise stated; see Section 3.1.

We say that f W Rn ! R is .Fréchet/  at xN 2 int.dom f / if there exists an element v 2 Rn such that

Definition 2.34

lim

x!xN

f .x/

f .x/ N hv; x kx xk N

xi N

D 0:

In this case the element v is uniquely defined and is denoted by rf .x/ N WD v . e Fermat stationary rule of classical analysis tells us that if f W Rn ! R is differentiable at xN and attains its local minimum at this point, then rf .x/ N D 0. e following proposition

56

2. SUBDIFFERENTIAL CALCULUS

gives a subdifferential counterpart of this result for general convex functions. Its proof is a direct consequence of the definitions and Proposition 2.33. Proposition 2.35 Let f W Rn ! R be convex and let xN 2 dom f . en f attains its local/global minimum at xN if and only if 0 2 @f .x/ N .

Proof. Suppose that f attains its global minimum at xN . en

f .x/ N  f .x/ for all x 2 Rn ;

which can be rewritten as 0 D h0; x

f .x/ N for all x 2 Rn :

xi N  f .x/

e definition of the subdifferential shows that this is equivalent to 0 2 @f .x/ N .



Now we show that the subdifferential (2.13) is indeed a singleton for differentiable functions reducing to the classical derivative/gradient at the reference point and clarifying the notion of differentiability in the case of convex functions. Proposition 2.36 Let f W Rn ! R be convex and differentiable at xN 2 int.dom f /. en we have @f .x/ N D frf .x/g N and

hrf .x/; N x

xi N  f .x/

f .x/ N for all x 2 Rn :

(2.17)

Proof. It follows from the differentiability of f at xN that for any  > 0 there is ı > 0 with kx

xk N  f .x/

f .x/ N

hrf .x/; N x

xi N  kx

xk N whenever kx

xk N < ı:

(2.18)

Consider further the convex function '.x/ WD f .x/

f .x/ N

hrf .x/; N x

xi N C kx

xk; N

x 2 Rn ;

and observe that '.x/  '.x/ N D 0 for all x 2 IB.xI N ı/. e convexity of ' ensures that '.x/  '.x/ N for all x 2 Rn . us hrf .x/; N x

xi N  f .x/

f .x/ N C kx

xk N whenever x 2 Rn ;

which yields (2.17) by letting  # 0. It follows from (2.17) that rf .x/ N 2 @f .x/ N . Picking now v 2 @f .x/ N , we get hv; x

xi N  f .x/

f .x/: N

en the second part of (2.18) gives us that hv

rf .x/; N x

xi N  kx

xk N whenever kx

xk N < ı:

Finally, we observe that kv rf .x/k N   , which yields v D rf .x/ N since  > 0 was chosen arbitrarily. us @f .x/ N D frf .x/g N . 

2.4. SUBGRADIENTS OF CONVEX FUNCTIONS

57

n

Corollary 2.37 Let f W R ! R be a strictly convex function. en the differentiability of f at xN 2 int.dom f / implies the strict inequality

hrf .x/; N x

xi N < f .x/

f .x/ N whenever x ¤ x: N

(2.19)

Proof. Since f is convex, we get from Proposition 2.36 that hrf .x/; N u

xi N  f .u/

f .x/ N for all u 2 Rn :

Fix x ¤ xN and let u WD .x C x/=2 N . It follows from the above that D E  x C xN  x C xN 1 1 xN  f f .x/ N < f .x/ C f .x/ N rf .x/; N 2 2 2 2

f .x/: N

us we arrive at the strict inequality (2.19) and complete the proof.



A simple while important example of a nonsmooth convex function is the norm function on Rn . Here is the calculation of its subdifferential. Example 2.38

Let p.x/ WD kxk be the Euclidean norm function on Rn . en we have 8 0 such that khk2  f .xN C h/

f .x/ N

hrf .x/; N hi

1 hAh; hi  khk2 for all khk  ı: 2

(2.22)

60

2. SUBDIFFERENTIAL CALCULUS

By Proposition 2.36, we readily have f .xN C h/

f .x/ N

hrf .x/; N hi  0:

Combining the above inequalities ensures that 1 hAh; hi whenever khk  ı: (2.23) 2 u satisfies khk  ı and, being Observe further that for any 0 ¤ u 2 Rn the element h WD ı kuk substituted into (2.23), gives us the estimate 1 D u u E ı 2  ı 2 A ; : 2 kuk kuk khk2 

It shows therefore (since the case where u D 0 is trivial) that 2kuk2  hAu; ui whenever u 2 Rn ;

which implies by letting  # 0 that 0  hAu; ui for all u 2 Rn .



Now we are ready to justify the aforementioned second-order characterization of convexity for twice continuously differentiable functions. eorem 2.42 Let f W Rn ! R be a function twice continuously differentiable on its domain D , which is an open convex subset of Rn . en f is convex if and only if r 2 f .x/ N is positive semidefinite for every xN 2 D .

Proof. Taking Lemma 2.41 into account, we only need to verify that if r 2 f .x/ N is positive semidefinite for every xN 2 D , then f is convex. To proceed, for any x1 ; x2 2 D define x t WD tx1 C .1 t /x2 and consider the function  '.t/ WD f tx1 C .1 t/x2 tf .x1 / .1 t /f .x2 /; t 2 R :

It is clear that ' is well defined on an open interval I containing .0; 1/. en ' 0 .t/ D hrf .x t /; x1 x2 i f .x1 / C f .x2 / and ' 00 .t / D hr 2 f .x t /.x1 x2 /; x1 x2 i  0 for every t 2 I since r 2 f .x t / is positive semidefinite. It follows from Corollary 1.46 that ' is convex on I . Since '.0/ D '.1/ D 0, for any t 2 .0; 1/ we have  '.t/ D ' t.1/ C .1 t /0  t '.1/ C .1 t /'.0/ D 0; which implies in turn the inequality f tx1 C .1

 t/x2  tf .x1 / C .1

t/f .x2 /;

is justifies that the function f is convex on its domain and thus on Rn .



2.5. BASIC CALCULUS RULES

2.5

61

BASIC CALCULUS RULES

Here we derive basic calculus rules for the convex subdifferential (2.13) and its singular counterpart (2.12). First observe the obvious subdifferential rule for scalar multiplication. Proposition 2.43

For a convex function f W Rn ! R with xN 2 dom f , we have

@.˛f /.x/ N D ˛@f .x/ N and @1 .˛f /.x/ N D ˛@1 f .x/ N whenever ˛ > 0:

Let us now establish the fundamental subdifferential sum rules for both constructions (2.13) and (2.12) that play a crucial role in convex analysis. We give a geometric proof for this result reducing it to the intersection rule for normals to sets, which in turn is based on convex separation of extremal set systems. Observe that our variational approach does not require any closedness/lower semicontinuity assumptions on the functions in question. Let fi W Rn ! R for i D 1; 2 be convex functions and let xN 2 dom f1 \ dom f2 . Impose the singular subdifferential qualification condition   @1 f1 .x/ N \ @1 f2 .x/ N D f0g: (2.24)

eorem 2.44

en we have the subdifferential sum rules @.f1 C f2 /.x/ N D @f1 .x/ N C @f2 .x/; N

@1 .f1 C f2 /.x/ N D @1 f1 .x/ N C @1 f2 .x/: N

(2.25)

Proof. For definiteness we verify the first rule in (2.25); the proof of the second result is similar. Fix an arbitrary subgradient v 2 @.f1 C f2 /.x/ N and define the sets ˝1 WD f.x; 1 ; 2 / 2 Rn  R  R j 1  f1 .x/g; ˝2 WD f.x; 1 ; 2 / 2 Rn  R  R j 2  f2 .x/g:

Let us first verify the inclusion  .v; 1; 1/ 2 N .x; N f1 .x/; N f2 .x//I N ˝1 \ ˝2 :

(2.26)

For any .x; 1 ; 2 / 2 ˝1 \ ˝2 , we have 1  f1 .x/ and 2  f2 .x/. en hv; x xi N C . 1/.1 f1 .x// N C . 1/.2 f2 .x// N  hv; x xi N f1 .x/ C f2 .x/ f1 .x/ N f2 .x/ N  0;

where the last estimate holds due to v 2 @.f1 C f2 /.x/ N . us (2.26) is satisfied. To proceed with employing the intersection rule in (2.26), let us first check that the assumed qualification condition (2.24) yields    N .x; N f1 .x/; N f2 .x//I N ˝1 \ N .x; N f1 .x/; N f2 .x//I N ˝2 D f0g; (2.27)

62

2. SUBDIFFERENTIAL CALCULUS

which is the basic qualification condition (2.9) needed for the application of Corollary 2.16 in the setting of (2.26). Indeed, we have from ˝1 D epi f1  R that   N .x; N f1 .x/; N f2 .x//I N ˝1 D N .x; N f1 .x//I N epi f1  f0g and observe similarly the representation ˇ  ˚  N .x; N f1 .x/; N f2 .x//I N ˝2 D .v; 0; / 2 Rn  R  R ˇ .v; / 2 N .x; N f2 .x//I N epi f2 : Further, fix any element   .v; 1 ; 2 / 2 N .x; N f1 .x/; N f2 .x//I N ˝1 \

 N .x; N f1 .x/; N f2 .x//I N ˝2 :

en 1 D 2 D 0, and hence

  .v; 0/ 2 N .x; N f1 .x//I N epi f1 and . v; 0/ 2 N .x; N f2 .x//I N epi f2 :

It follows by the definition of the singular subdifferential that  v 2 @1 f1 .x/ N \ @1 f2 .x/ N D f0g: It yields .v; 1 ; 2 / D .0; 0; 0/ 2 Rn  R  R. Applying Corollary 2.16 in (2.26) shows that    N .x; N f1 .x/; N f2 .x//I N ˝1 \ ˝2 D N .x; N f1 .x/; N f2 .x//I N ˝1 C N .x; N f1 .x/; N f2 .x//I N ˝2 ; which implies in turn that .v; 1; 1/ D .v1 ; 1 ; 0/ C .v2 ; 0; 2 /

with .v1 ; 1 / 2 N..x; N f1 .x//I N epi f1 / and .v2 ; 2 / 2 N..x; N f1 .x//I N epi f2 /. en we get 1 D

2 D 1 and v D v1 C v2 , where vi 2 @fi .x/ N for i D 1; 2. is verifies the inclusion “” in the first subdifferential sum rule of (2.25). e opposite inclusion therein follows easily from the definition of the subdifferential. Indeed, any v 2 @f1 .x/ N C @f2 .x/ N can be represented as v D v1 C v2 , where vi 2 @fi .x/ N for i D 1; 2. en we have hv; x

xi N D hv1 ; x  f1 .x/

xi N C hv2 ; x xi N f1 .x/ N C f2 .x/ f2 .x/ N D .f1 C f1 /.x/

.f1 C f2 /.x/ N

for all x 2 Rn , which readily implies that v 2 @.f1 C f2 /.x/ N and therefore completes the proof of the theorem.  Note that by Proposition 2.23 the qualification condition (2.24) can be written as   N.xI N dom f1 / \ N.xI N dom f2 / D f0g:

2.5. BASIC CALCULUS RULES

63

Let us present some useful consequences of eorem 2.44. e sum rule for the (basic) subdifferential (2.13) in the following corollary is known as the Moreau-Rockafellar theorem. Corollary 2.45 Let fi W Rn ! R for i D 1; 2 be convex functions such that there exists u 2 dom f1 \ dom f2 for which f1 is continuous at u or f2 is continuous at u. en

(2.28)

@.f1 C f2 /.x/ D @f1 .x/ C @f2 .x/

whenever x 2 dom f1 \ dom f2 . Consequently, if both functions fi are finite-valued on Rn , then the sum rule (2.28) holds for all x 2 Rn . Proof. Suppose for definiteness that f1 is continuous at u 2 dom f1 \ dom f2 , which implies that u 2 int.dom f1 / \ dom f2 . en the proof of Corollary 2.18 shows that   N.xI dom f1 / \ N.xI dom f2 / D f0g whenever x 2 dom f1 \ dom f2 :

us the conclusion follows directly from eorem 2.44.



e next corollary for sums of finitely many functions can be derived from eorem 2.44 by induction, where the validity of the qualification condition in the next step of induction follows from the singular subdifferential sum rule in (2.25).

Consider finitely many convex functions fi W Rn ! R for i D 1; : : : ; m. e following assertions hold: T (i) Let xN 2 m iD1 dom fi and let the implication

Corollary 2.46

m hX iD1

i   vi D 0; vi 2 @1 fi .x/ N H) vi D 0 for all i D 1; : : : ; m

(2.29)

hold. en we have the subdifferential sum rules @

m X iD1

m  X fi .x/ N D @fi .x/; N iD1

@

1

m X iD1

m  X fi .x/ N D @fi .x/: N

Tm

iD1

(ii) Suppose that there is u 2 iD1 dom fi such that all (except possibly one) functions fi are continuous at u. en we have the equality @

m X iD1

for any common point of the domains x 2

m  X fi .x/ D @fi .x/ iD1

Tm

iD1

dom fi .

64

2. SUBDIFFERENTIAL CALCULUS

e next result not only ensures the subdifferentiability of convex functions, but also justifies the compactness of its subdifferential at interior points of the domain.

For any convex function f W Rn ! R and any xN 2 int.dom f /, we have that the subgradient set @f .x/ N is nonempty and compact in Rn . Proposition 2.47

Proof. Let us first verify the boundedness of the subdifferential. It follows from Corollary 2.27 that f is locally Lipschitz continuous around xN with some constant `, i.e., there exists a positive number  such that jf .x/

f .y/j  `kx

yk for all x; y 2 IB.xI N /:

us for any subgradient v 2 @f .x/ N and any element h 2 Rn with khk  ı we have hv; hi  f .xN C h/

f .x/ N  `khk:

is implies the bound kvk  `. e closedness of @f .x/ N follows directly from the definition, and so we get the compactness of the subgradient set. It remains to verify that @f .x/ N ¤ ;. Since .x; N f .x// N 2 bd .epi f /, we deduce from N f .x//I N epi f / ¤ f0g. Take such an element .0; 0/ ¤ .v; / 2 Corollary 2.14 that N..x; N..x; N f .x//I N epi f / and get by the definition that hv; x

xi N

.

f .x// N  0 for all .x; / 2 epi f:

Using this inequality with x WD xN and  D f .x/ N C 1 yields  0. If D 0, then v 2 @1 f .x/ N D f0g by eorem 2.29, which is a contradiction. us > 0 and .v= ; 1/ 2 N..x; N f .x//I N epi f /. Hence v= 2 @f .x/ N , which completes the proof of the proposition.  Given f W Rn ! R and any ˛ 2 R, define the level set ˇ ˚ ˝˛ WD x 2 Rn ˇ f .x/  ˛ : Proposition 2.48 Let f W Rn ! R be convex and let xN 2 ˝˛ with f .x/ N D ˛ for some ˛ 2 R. If 0 … @f .x/ N , then the normal cone to the level set ˝˛ at xN is calculated by

N.xI N ˝˛ / D @1 f .x/ N [ R> @f .x/; N

where R> denotes the set of all positive real numbers.

(2.30)

2.5. BASIC CALCULUS RULES n

nC1

Proof. To proceed, consider the set  WD R f˛g  R

65

and observe that

˝˛  f˛g D epi f \ :

Let us check the relationship   N .x; N ˛/I epi f \

N .x; N ˛/I 



D f0g;

(2.31)

i.e., the qualification condition (2.9) holds for the sets epi f and  . Indeed, take .v; / from the set on left-hand side of (2.31) and deduce from the structure of  that v D 0 and that   0. If we suppose that  > 0, then  1 .v; / D .0; 1/ 2 N .x; N ˛/I epi f ; 

and so 0 2 @f .x/ N , which contradicts the assumption of this proposition. us we have .v; / D .0; 0/, and then applying the intersection rule of Corollary 2.16 gives us   N.x; N ˝˛ /  R D N .x; N ˛/I ˝˛  f˛g D N .x; N ˛/I epi f C .f0g  R/: Fix now any v 2 N.xI N ˝˛ / and get .v; 0/ 2 N.x; N ˝˛ /  R. is implies the existence of   0 such that .v; / 2 N..x; N ˛/I epi f / and .v; 0/ D .v; / C .0; /. If  D 0, then v 2 @1 f .x/ N . In the case when  > 0 we obtain v 2 @f .x/ N and thus verify the inclusion “” in (2.30). Let us finally prove the opposite inclusion in (2.30). Since ˝˛  dom f , we get @1 f .x/ N D N.xI N dom f /  N.xI N ˝˛ /:

Take any v 2 R> @f .x/ N and find  > 0 such that v 2 @f .x/ N . Taking any x 2 ˝˛ gives us f .x/  ˛ D f .x/ N . erefore, we have h

1

v; x

xi N  f .x/

f .x/ N  0;

which implies that v 2 N.xI N ˝˛ / and thus completes the proof.



Corollary 2.49 In the setting of Proposition 2.48 suppose that f is continuous at xN and that 0 … @f .x/ N . en we have the representation

N.xI N ˝˛ / D RC @f .x/ N D

[

˛0

˛@f .x/; N

˛ 2 R:

Proof. Since f is continuous at xN , it follows from eorem 2.29 that @1 f .x/ N D f0g. en the  result of this corollary follows directly from Proposition 2.48.

66

2. SUBDIFFERENTIAL CALCULUS

We now obtain some other calculus rules for subdifferentiation of convex function. For brevity, let us only give results for the basic subdifferential (2.13) leaving their singular subdifferential counterparts as exercises; see Section 2.11. To proceed with deriving chain rules, we present first the following geometric lemma. Lemma 2.50 Let B W Rn ! Rp be an affine mapping given by B.x/ D A.x/ C b , where A W Rn ! Rp is a linear mapping and b 2 Rp . For any .x; N y/ N 2 gph B , we have ˇ  ˚ N .x; N y/I N gph B D .u; v/ 2 Rn  Rp ˇ u D A v :

Proof. It is clear that the set gph B is convex and .u; v/ 2 N..x; N y/ N gph B/ if and only if hu; x

xi N C hv; B.x/

B.x/i N  0 whenever x 2 Rn :

(2.32)

It follows directly from the definitions that hu; x

xi N C hv; B.x/

B.x/i N D hu; x xi N C hv; A.x/ A.x/i N  D hu; x xi N C hA v; x xi N D hu C A v; x xi; N

which implies the equivalence of (2.32) to hu C A v; x

xi N  0, and so to u D

A v .



eorem 2.51 Let f W Rp ! R be a convex function and let B W Rn ! Rp be as in Lemma 2.50 with B.x/ N 2 dom f for some xN 2 Rp . Denote yN WD B.x/ N and assume that

ker A \ @1 f .y/ N D f0g:

(2.33)

en we have the subdifferential chain rule ˇ  ˚ @.f ı B/.x/ N D A @f .y/ N D A v ˇ v 2 @f .y/ N :

(2.34)

Proof. Fix v 2 @.f ı B/.x/ N and form the subsets of Rn  Rp  R by ˝1 WD gph B  R and ˝2 WD Rn epi f:

It follows from the definition of the subdifferential and the normal cone that .v; 0; 1/ 2 N..x; N y; N z/I N ˝1 \ ˝2 /, where zN WD f .y/ N . Indeed, fix any .x; y; / 2 ˝1 \ ˝2 . en y D B.x/ and   f .y/, and so   f .B.x//. us hv; x

xi N C 0.y

y/ N C . 1/.

zN /  hv; x

xi N

Œf .B.x//

f .B.x// N  0:

2.5. BASIC CALCULUS RULES

67

Employing the intersection rule from Corollary 2.19 under (2.33) gives us   .v; 0; 1/ 2 N .x; N y; N z/I N ˝1 C N .x; N y; N z/I N ˝2 ; which reads that .v; 0; 1/ D .v; w; 0/ C .0; w; 1/ with .v; w/ 2 N..x; N y/I N gph B/ and .w; 1/ 2 N..y; N z/I N epi f /. en we get v D A w and w 2 @f .y/; N

which implies in turn that v 2 A .@f .y// N and thus verifies the inclusion “” in (2.34). e opposite inclusion follows directly from the definition of the subdifferential.  In the corollary below we present some conditions that guarantee that the qualification (2.33) in eorem 2.51 is satisfied. Corollary 2.52 In the setting of eorem 2.51 suppose that either A is surjective or f is continuous at the point yN 2 Rn ; the latter assumption is automatic when f is finite-valued. en we have the

subdifferential chain rule (2.34).

Proof. Let us check that the qualification condition (2.33) holds under the assumptions of this corollary. e surjectivity of A gives us ker A D f0g, and in the second case we have yN 2 int.dom f /, so @1 f .y/ N D f0g. e validity of (2.33) is obvious. 

e last consequence of eorem 2.51 presents a chain rule for normals to sets.

Let ˝  Rp be a convex set and let B W Rn ! Rp be an affine mapping as in Lemma 2.50 satisfying yN WD B.x/ N 2 ˝ . en we have

Corollary 2.53

N.xI N B

1

.˝// D A .N.yI N ˝//

under the validity of the qualification condition ker A \ N.yI N ˝/ D f0g: Proof. is follows from eorem 2.51 with f .x/ D ı.xI ˝/.



Given fi W Rn ! R for i D 1; : : : ; m, consider the maximum function f .x/ WD max fi .x/; iD1;:::;m

and for xN 2 Rn define the set

x 2 Rn ;

ˇ ˚ I.x/ N WD i 2 f1; : : : ; mg ˇ fi .x/ N D f .x/ N :

(2.35)

68

2. SUBDIFFERENTIAL CALCULUS

In the proposition below we assume that each function fi for i D 1; : : : ; m is continuous at the reference point. A more general version follows afterward. Proposition 2.54 Let fi W Rn ! R, i D 1; : : : ; m, be convex functions. Take any point xN 2 Tm N . en we have the maximum rule iD1 dom fi and assume that each fi is continuous at x [ @ max fi /.x/ N D co @fi .x/: N i2I.x/ N

Proof. Let f .x/ be the maximum function defined in (2.35). We obviously have

epi f D

m \

epi fi :

iD1

Since fi .x/ N < f .x/ N DW ˛N for any i … I.x/ N , there exist a neighborhood U of xN and ı > 0 such that fi .x/ < ˛ whenever .x; ˛/ 2 U  .˛N ı; ˛N C ı/. It follows that .x; N ˛/ N 2 int epi fi , and so N..x; N ˛/I N epi fi / D f.0; 0/g for such indices i . Let us show that the qualification condition from Corollary 2.19  is satisfied for the sets ˝i WD epi fi , i D 1; : : : ; m, at .x; N ˛/ N . Fix .vi ; i / 2 N .x; N ˛/I N epi fi with m X iD1

.vi ; i / D 0:

en .vi ; i / D .0; 0/ for i … I.x/ N , and hence X .vi ; i / D 0: i2I.x/ N

Note that fi .x/ N D f .x/ N D ˛N for i 2 I.x/ N . e proof of Proposition 2.47 tells us that i  0 P 1 for i 2 I.x/ N . us the equality i2I.x/ N D f0g for N i D 0 yields i D 0, and so vi 2 @ fi .x/ every i 2 I.x/ N by eorem 2.29. is verifies that .vi ; i / D .0; 0/ for i D 1; : : : ; m. It follows furthermore that m X  X   N .x; N f .x//I N epi f D N .x; N ˛/I N epi fi D N .x; N fi .x//I N epi fi : iD1

i2I.x/ N

For any v 2 @f .x/ N , we have .v; 1/ 2 N..x; N f .x//I N epi f /, which allows us to find .vi ; i / 2 N..x; N fi .x//I N epi fi / for i 2 I.x/ N such that X .vi ; i /: .v; 1/ D P

P

i2I.x/ N

1 is yields i2I.x/ N D f0g. Hence we N i D 1 and v D i2I.x/ N vi . If i D 0, then vi 2 @ fi .x/ assume without loss of generality that i > 0 for every i 2 I.x/ N . Since N..x; N fi .x//I N epi fi / is a

2.5. BASIC CALCULUS RULES

69

cone, it gives us .vi =i ; 1/ 2 N..x; N fi .x//I N epi fi /, and so vi 2 i @fi .x/ N . We get the represenP P tation v D i2I.x/ N and i2I.x/ N i ui , where ui 2 @fi .x/ N i D 1. us [ v 2 co @fi .x/: N i2I.x/ N

Note that the opposite inclusion follows directly from @fi .x/ N  @f .x/ N for all i 2 I.x/; N

which in turn follows from the definition. Indeed, for any i 2 I.x/ N and v 2 @fi .x/ N we have hv; x

xi N  fi .x/

fi .x/ N D fi .x/

f .x/ N  f .x/

f .x/ N whenever x 2 Rn ;

which implies that v 2 @f .x/ N and thus completes the proof.



To proceed with the general case, define m X ˇ ˚ i D 1; i fi .x/ N .x/ N WD .1 ; : : : ; m / ˇ i  0; iD1

 f .x/ N D0

and observe from the definition of .x/ N hat X ˇ ˚ .x/ N D .1 ; : : : ; m / ˇ i  0; i D 1; i D 0 for i … I.x/ N : i2I.x/ N

eorem 2.55 Let fi W Rn ! R be convex functions for i D 1; : : : ; m. Take any point xN 2 Tm N for any i … I.x/ N and that the qualiiD1 dom fi and assume that each fi is continuous at x

fication condition h X

i2I.x/ N

i   vi D 0; vi 2 @1 fi .x/ N H) vi D 0 for all i 2 I.x/ N

holds. en we have the maximum rule ˇ o [n X  ˇ @ max fi .x/ N D i ı @fi .x/ N ˇ .1 ; : : : ; m / 2 .x/ N ; i2I.x/ N

where i ı @fi .x/ N WD i @fi .x/ N if i > 0, and i ı @fi .x/ N WD @1 fi .x/ N if i D 0. Proof. Let f .x/ be the maximum function from (2.35). Repeating the first part of the proof of N f .x//I N epi fi / D f.0; 0/g for i … I.x/ N . e imposed qualificaProposition 2.54 tells us that N..x; tion condition allows us to employ the intersection rule from Corollary 2.19 to get m X   X  N .x; N fi .x//I N epi fi : N .x; N f .x//I N epi f D N .x; N f .x//I N epi fi D iD1

i2I.x/ N

(2.36)

70

2. SUBDIFFERENTIAL CALCULUS

For any v 2 @f .x/ N , we have .v; 1/ 2 N..x; N f .x//I N epi f /, and thus find by using (2.36) such pairs .vi ; i / 2 N..x; N fi .x//I N epi fi / for i 2 I.x/ N that X .v; 1/ D .vi ; i /: i2I.x/ N

P P is shows that i2I.x/ N and i  0 for all i 2 N i D 1 and v D i2I.x/ N vi with vi 2 i ı @fi .x/ I.x/ N . erefore, we arrive at ˇ o [n X ˇ v2 i ı @fi .x/ N ˇ .1 ; : : : ; m / 2 .x/ N ; i2I.x/ N

which justifies the inclusion “” in the maximum rule. e opposite inclusion can also be verified easily by using representations (2.36). 

2.6

SUBGRADIENTS OF OPTIMAL VALUE FUNCTIONS

is section is mainly devoted to the study of the class of optimal value/marginal functions defined in (1.4). To proceed, we first introduce and examine the notion of coderivatives for set-valued mappings, which is borrowed from variational analysis and plays a crucial role in many aspects of the theory and applications of set-valued mappings; in particular, those related to subdifferentiation of optimal value functions considered below. ! Rp be a set-valued mapping with convex graph and let .x; Definition 2.56 Let F W Rn ! N y/ N 2 p ! n  gph F . e  of F at .x; N y/ N is the set-valued mapping D F .x; N y/W N R ! R with the

values

ˇ  ˚ N y/I N gph F ; D  F .x; N y/.v/ N WD u 2 Rn ˇ .u; v/ 2 N .x;

v 2 Rp :

(2.37)

It follows from (2.37) and definition (2.4) of the normal cone to convex sets that the coderivative values of convex-graph mappings can be calculated by ˇ n  o ˇ D  F .x; N y/.v/ N D u 2 Rn ˇ hu; xi N hv; yi N D max hu; xi hv; yi : .x;y/2gph F

Now we present explicit calculations of the coderivative in some particular settings. Proposition 2.57 Let F W Rn ! Rp be a single-valued mapping given by F .x/ WD Ax C b , where A W Rn ! Rp is a linear mapping, and where b 2 Rp . For any pair .x; N y/ N such that yN D AxN C b ,

we have the representation

D  F .x; N y/.v/ N D fA vg whenever v 2 Rp :

2.6. SUBGRADIENTS OF OPTIMAL VALUE FUNCTIONS

71

Proof. Lemma 2.50 tells us that ˇ ˚ N .x; N y/I N gph F / D .u; v/ 2 Rn  Rp ˇ u D

A v :

us u 2 D  F .x; N y/.v/ N if and only if u D A v .



Proposition 2.58 Given a convex function f W Rn ! R with xN 2 dom f , define the set-valued ! R by mapping F W Rn !   F .x/ WD f .x/; 1 :

Denoting yN WD f .x/ N , we have

8 N ˆ 0; if  D 0; if  < 0:

In particular, if f is finite-valued, then D  F .x; N y/./ N D @f .x/ N for all   0. Proof. It follows from the definition of F that ˇ ˇ ˚ ˚ gph F D .x; / 2 Rn  R ˇ  2 F .x/ D .x; / 2 Rn  R ˇ   f .x/ D epi f:

us the inclusion v 2 D  F .x; N y/./ N is equivalent to .v; / 2 N..x; N y/I N epi f /. If  > 0, then .v=; 1/ 2 N..x; N y/I N epi f /, which implies v= 2 @f .x/ N , and hence v 2 @f .x/ N . e formula in the case where  D 0 follows from the definition of the singular subdifferential. If  < 0, then D  F .x; N y/./ N D ; because   0 whenever .v; / 2 N..x; N y/I N epi f / by the proof of Proposition 2.47. To verify the last part of the proposition, observe that in this case D  F .x; N y/.0/ N D 1 @ f .x/ N D f0g by Proposition 2.23, so we arrive at the conclusion.  e next simple example is very natural and useful in what follows. ! Rp by F .x/ D K for every x 2 Rn , where K  Rp Define F W Rn ! is a nonempty, convex set. en gph F D Rn K , and Proposition 2.11(i) tells us that N..x; N y/I N gph F / D f0g  N.yI N K/ for any .x; N y/ N 2 Rn K . us ( f0g if v 2 N.yI N K/;  D F .x; N y/.v/ N D ; otherwise: Example 2.59

In particular, for the case where K D Rp we have ( f0g if v D 0; D  F .x; N y/.v/ N D ; otherwise:

72

2. SUBDIFFERENTIAL CALCULUS

Now we derive a formula for calculating the subdifferential of the optimal value/marginal function (1.4) given in the form ˇ ˚ .x/ WD inf '.x; y/ ˇ y 2 F .x/

via the coderivative of the set-valued mapping F and the subdifferential of the function ' . First we derive the following useful estimate. Lemma 2.60 Let ./ be the optimal value function (1.4) generated by a convex-graph mapping ! Rp and a convex function ' W Rn  Rp ! R. Suppose that .x/ > 1 for all x 2 Rn , F W Rn ! fix some xN 2 dom , and consider the solution set ˇ ˚ S.x/ N WD yN 2 F .x/ N ˇ .x/ N D '.x; N y/ N :

If S.x/ N ¤ ;, then for any yN 2 S.x/ N we have [   u C D  F .x; N y/.v/ N  @.x/: N

(2.38)

.u;v/2@'.x; N y/ N

Proof. Pick w from the left-hand side of (2.38) and find .u; v/ 2 @'.x; N y/ N with w

It gives us .w

u 2 D  F .x; N y/.v/: N

u; v/ 2 N..x; N y/I N gph F / and thus hw

u; x

xi N

hv; y

yi N  0 for all .x; y/ 2 gph F;

which shows that whenever y 2 F .x/ we have hw; x

xi N  hu; x

xi N C hv; y

yi N  '.x; y/

'.x; N y/ N D '.x; y/

.x/: N

is implies in turn the estimate hw; x

xi N 

inf '.x; y/

y2F .x/

.x/ N D .x/

.x/; N

which justifies the inclusion w 2 @.x/ N , and hence completes the proof of (2.38).



eorem 2.61 Consider the optimal value function (1.4) under the assumptions of Lemma 2.60. For any yN 2 S.x/ N , we have the equality [   @.x/ N D u C D  F .x; N y/.v/ N (2.39) .u;v/2@'.x; N y/ N

provided the validity of the qualification condition   @1 '.x; N y/ N \ N..x; N y/I N gph F / D f.0; 0/g; which holds, in particular, when ' is continuous at .x; N y/ N .

(2.40)

2.6. SUBGRADIENTS OF OPTIMAL VALUE FUNCTIONS

73

Proof. Taking Lemma 2.60 into account, it is only required to verify the inclusion “” in (2.39). To proceed, pick w 2 @.x/ N and yN 2 S.x/ N . For any x 2 Rn , we have hw; x

xi N  .x/ .x/ N D .x/ '.x; N y/ N  '.x; y/ '.x; N y/ N for all y 2 F .x/:

is implies that, whenever .x; y/ 2 Rn  Rp , the following inequality holds:   hw; x xi N C h0; y yi N  '.x; y/ C ı .x; y/I gph F / '.x; N y/ N C ı .x; N y/I N gph F : Denote further f .x; y/ WD '.x; y/ C ı..x; y/I gph F / and deduce from the subdifferential sum rule in eorem 2.44 under the qualification condition (2.40) that the inclusion  .w; 0/ 2 @f .x; N y/ N D @'.x; N y/ N C N .x; N y/I N gph F is satisfied. is shows that .w; 0/ D .u1 ; v1 / C .u2 ; v2 / with .u1 ; v1 / 2 @'.x; N y/ N and .u2 ; v2 / 2 N..x; N y/I N gph F /. is yields v2 D v1 , so .u2 ; v1 / 2 N..x; N y/I N gph F /. Finally, we get u2 2 D  F .x; N y/.v N 1 / and therefore w D u1 C u2 2 u1 C D  F .x; N y/.v N 1 /;

which completes the proof of the theorem.



Let us present several important consequences of eorem 2.61. e first one concerns the following chain rule for convex compositions.

Let f W Rn ! R be a convex function and let ' W R ! R be a nondecreasing convex function. Take xN 2 Rn and denote yN WD f .x/ N assuming that yN 2 dom ' . en [ @.' ı f /.x/ N D @f .x/ N

Corollary 2.62

2@'.y/ N

under the assumption that either @1 '.y/ N D f0g or 0 … @f .x/ N . Proof. e composition ' ı f is a convex function by Proposition 1.39. Define the set-valued mapping F .x/ WD Œf .x/; 1/ and observe that .' ı f /.x/ D

inf '.y/

y2F .x/

since ' is nondecreasing. Note each of the assumptions @1 '.y/ N D f0g and 0 … @f .x/ N ensure the validity of the qualification condition (2.40). It follows from eorem 2.61 that [ D  F .x; N y/./: N @.' ı f /.x/ N D 2@'.y/ N

74

2. SUBDIFFERENTIAL CALCULUS

Taking again into account that ' is nondecreasing yields   0 for every  2 @'.y/ N . is tells us by Proposition 2.58 that [ [ D  F .x; N y/./ N D @f .x/; N @.' ı f /.x/ N D 2@'.y/ N

2@'.y/ N

which verifies the claimed chain rule of this corollary.



e next corollary addresses calculating subgradients of the optimal value functions over parameter-independent constraints. Corollary 2.63 Let ' W Rn  Rp ! R, where the function '.x; / is bounded below on Rp for every x 2 Rn . Define ˇ ˇ ˚ ˚ .x/ WD inf '.x; y/ ˇ y 2 Rp ; S.x/ WD y 2 Rp ˇ '.x; y/ D .x/ :

For xN 2 dom  and for every yN 2 S.x/ N , we have ˇ ˚ N y/ N : @.x/ N D u 2 Rn ˇ .u; 0/ 2 @'.x; Proof. Follows directly from eorem 2.61 for F .x/  Rp and Example 2.59. Note that the qualification condition (2.40) is satisfied since N..x; N y/I N gph F / D f.0; 0/g. 

e next corollary specifies the general result for problems with affine constraints.

Let B W Rp ! Rn be given by B.y/ WD Ay C b via a linear mapping AW Rp ! Rn and an element b 2 Rn and let ' W Rp ! R be a convex function bounded below on the inverse image set B 1 .x/ for all x 2 Rn . Define ˇ ˇ ˚ ˚ .x/ WD inf '.y/ ˇ B.y/ D xg; S.x/ WD y 2 B 1 .x/ ˇ '.y/ D .x/ Corollary 2.64

and fix xN 2 Rn with .x/ N < 1 and S.x/ N ¤ ;. For any yN 2 S.x/ N , we have  @.x/ N D .A / 1 @'.y/ N : Proof. e setting of this corollary corresponds to eorem 2.61 with F .x/ D B '.x; y/  '.y/. en we have ˇ  ˚ N .x; N y/I N gph F D .u; v/ 2 Rn  Rp ˇ A u D v ;

which therefore implies the coderivative representation ˇ ˚ D  F .x; N y/.v/ N D u 2 Rn ˇ A u D v D .A / 1 .v/;

v 2 Rp :

1

.x/ and

2.7. SUBGRADIENTS OF SUPPORT FUNCTIONS

75

It is easy to see that the qualification condition (2.40) is automatic in this case. is yields [   @.x/ N D D  F .x; N y/.v/ N D .A / 1 .@'.y//; N v2@'.y/ N

and thus we complete the proof.



Finally in this section, we present a useful application of Corollary 2.64 to deriving a calculus rule for subdifferentiation of infimal convolutions important in convex analysis. Corollary 2.65

Given convex functions f1 ; f2 W Rn ! R, consider their infimal convolution ˇ ˚ .f1 ˚ f2 /.x/ WD inf f1 .x1 / C f2 .x2 / ˇ x1 C x2 D x ; x 2 Rn ;

and suppose that .f1 ˚ f2 /.x/ > 1 for all x 2 Rn . Fix xN 2 dom .f1 ˚ f2 / and xN 1 ; xN 2 2 Rn satisfying xN D xN 1 C xN 2 and .f1 ˚ f2 /.x/ N D f1 .xN 1 / C f2 .xN 2 /. en we have @.f1 ˚ f2 /.x/ N D @f1 .xN 1 / \ @f2 .xN 2 /:

Proof. Define 'W Rn  Rn ! R and AW Rn  Rn ! Rn by '.x1 ; x2 / WD f1 .x1 / C f2 .x2 /;

A.x1 ; x2 / WD x1 C x2 :

It is easy to see that A v D .v; v/ for any v 2 Rn and that @'.xN 1 ; xN 2 / D .@f1 .xN 1 /; @f2 .xN 2 // for .xN 1 ; xN 2 / 2 Rn  Rn . en v 2 @.f1 ˚ f2 /.x/ N if and only if we have A v D .v; v/ 2 @'.xN 1 ; xN 2 /, which means that v 2 @f1 .xN 1 / \ @f2 .xN 2 /. us the claimed calculus result follows from Corollary 2.64. 

2.7

SUBGRADIENTS OF SUPPORT FUNCTIONS

is short section is devoted to the study of subgradient properties for an important class of nonsmooth convex functions known as support functions of convex sets. Definition 2.66 Given a nonempty .not necessarily convex/ subset ˝ of Rn , the   of ˝ is defined by ˇ ˚ ˝ .x/ WD sup hx; !i ˇ ! 2 ˝ : (2.41)

First we present some properties of (2.41) that can be deduced from the definition. Proposition 2.67

e following assertions hold:

(i) For any nonempty subset ˝ of Rn , the support function ˝ is subadditive, positively homogeneous, and hence convex.

76

2. SUBDIFFERENTIAL CALCULUS

(ii) For any nonempty subsets ˝1 ; ˝2  Rn and any x 2 Rn , we have ˚ ˝1 C˝2 .x/ D ˝1 .x/ C ˝2 .x/; ˝1 [˝2 .x/ D max ˝1 .x/; ˝2 .x/ :

e next result calculates subgradients of support functions via convex separation. eorem 2.68

Let ˝ be a nonempty, closed, convex subset of Rn and let ˇ ˚ S.x/ WD p 2 ˝ ˇ hx; pi D ˝ .x/ :

For any xN 2 dom ˝ , we have the subdifferential formula @˝ .x/ N D S.x/: N

Proof. Fix p 2 @˝ .x/ N and get from definition (2.13) that hp; x

xi N  ˝ .x/

˝ .x/ N whenever x 2 Rn :

(2.42)

Taking this into account, it follows for any u 2 Rn that hp; ui D hp; u C xN

xi N  ˝ .u C x/ N

˝ .x/ N  ˝ .u/ C ˝ .x/ N

˝ .x/ N D ˝ .u/: (2.43)

Assuming that p 62 ˝ , we use the separation result of Proposition 2.1 and find u 2 Rn with ˇ ˚ ˝ .u/ D sup hu; !i ˇ ! 2 ˝ < hp; ui; which contradicts (2.43) and so verifies p 2 ˝ . From (2.43) we also get hp; xi N  ˝ .x/ N . Letting x D 0 in (2.42) gives us ˝ .x/ N  hp; xi N . Hence hp; xi N D ˝ .x/ N and thus p 2 S.x/ N . Now fix p 2 S.x/ N and get from p 2 ˝ and hp; xi N D ˝ .x/ N that hp; x

xi N D hp; xi

hp; xi N  ˝ .x/

˝ .x/ N for all x 2 Rn :

is shows that p 2 @˝ .x/ N and thus completes the proof. Note that eorem 2.68 immediately implies that ˇ ˚ @˝ .0/ D S.0/ D ! 2 ˝ ˇ h0; !i D ˝ .0/ D 0 D ˝



(2.44)

for any nonempty, closed, convex set ˝  Rn . e last result of this section justifies a certain monotonicity property of support functions with respect to set inclusions. Proposition 2.69 Let ˝1 and ˝2 be a nonempty, closed, convex subsets of Rn . en the inclusion ˝1  ˝2 holds if and only if ˝1 .v/  ˝2 .v/ for all v 2 Rn .

2.8. FENCHEL CONJUGATES

77

n

Proof. Taking ˝1  ˝2 as in the proposition, for any v 2 R we have ˇ ˇ ˚ ˚ ˝1 .v/ D sup hv; xi ˇ x 2 ˝1  sup hv; xi ˇ x 2 ˝2 D ˝2 .v/:

Conversely, suppose that ˝1 .v/  ˝2 .v/ whenever v 2 Rn . Since ˝1 .0/ D ˝2 .0/ D 0, it follows from definition (2.13) of the subdifferential that @˝1 .0/  @˝2 .0/;

which yields ˝1  ˝2 by formula (2.44).

2.8



FENCHEL CONJUGATES

Many important issues of convex analysis and its applications (in particular, to optimization) are based on duality. e following notion plays a crucial role in duality considerations. Definition 2.70 Given a function f W Rn ! R .not necessarily convex/, its F  f  W Rn ! Œ 1; 1 is ˇ ˇ ˚ ˚ (2.45) f  .v/ WD sup hv; xi f .x/ ˇ x 2 Rn D sup hv; xi f .x/ ˇ x 2 dom f :

Note that f  .v/ D 1 is allowed in (2.45) while f  .v/ > 1 for all v 2 Rn if dom f ¤ ;. It follows directly from the definitions that for any nonempty set ˝  Rn the conjugate to its indicator function is the support function of ˝ : ˇ ˚  (2.46) ı˝ .v/ D sup hv; xi ˇ x 2 ˝ D ˝ .v/; v 2 ˝: e next two propositions can be easily verified.

Let f W Rn ! R be a function, not necessarily convex, with dom f ¤ ;. en its Fenchel conjugate f  is convex on Rn .

Proposition 2.71

Proof. Function (2.45) is convex as the supremum of a family of affine functions.



Proposition 2.72 Let f; g W Rn ! R be such that f .x/  g.x/ for all x 2 Rn . en we have f  .v/  g  .v/ for all v 2 Rn .

Proof. For any fixed v 2 Rn , it follows from (2.45) that hv; xi

f .x/  hv; xi

g.x/;

x 2 Rn :

is readily implies the relationships ˇ ˚ ˚ f  .v/ D sup hv; xi f .x/ ˇ x 2 Rn  sup hv; xi for all v 2 Rn , and therefore f   g  on Rn .

ˇ g.x/ ˇ x 2 Rn D g  .v/ 

78

2. SUBDIFFERENTIAL CALCULUS

Figure 2.8: Fenchel conjugate.

e following two examples illustrate the calculation of conjugate functions. Example 2.73

(i) Given a 2 Rn and b 2 R, consider the affine function f .x/ WD ha; xi C b;

x 2 Rn :

en it can be seen directly from the definition that ( b if v D a;  f .v/ D 1 otherwise: (ii) Given any p > 1, consider the power function 8 p 0, we see that function v .x/ WD p 1 x p vx is convex and differentiable on

2.8. FENCHEL CONJUGATES 0 v .x/

p 1

.0; 1/ with Dx its minimum at x D v 1=.p

1/

0 v .x/

v . us D 0 if and only if x D v , and so . erefore, the conjugate function is calculated by

 f  .v/ D 1

Taking q from q

1

D1

p

1=.p 1/

1

1  p=.p v p

1/

;

v

79

attains

v 2 Rn :

, we express the conjugate function as 8 0

which verifies the results claimed in the proposition.



Corollary 2.82 If f W Rn ! R is a convex function, then f 0 .xI N d / is a real number for any xN 2 n int.dom f / and d 2 R .

Proof. It follows from eorem 2.29 that f is locally Lipschitz continuous around xN , i.e., there is `  0 such that ˇ f .xN C td / f .x/ N ˇˇ `t kd k ˇ D `kd k for all small t > 0; ˇ ˇ t t

which shows that jf 0 .xI N d /j  `kd k < 1.



To establish relationships between directional derivatives and subgradients of general convex functions, we need the following useful observation. Lemma 2.83

For any convex function f W Rn ! R and xN 2 dom f , we have f 0 .xI N d /  f .xN C d /

f .x/ N whenever d 2 Rn :

Proof. Using Lemma 2.80, we have for the function ' therein that '.t/  '.1/ D f .xN C d /

f .x/ N for all t 2 .0; 1/;

which justifies the claimed property due to f 0 .xI N d / D inf t>0 '.t /  '.1/. eorem 2.84

Let f W Rn ! R be convex with xN 2 dom f . e following are equivalent:

(i) v 2 @f .x/ N . (ii) hv; d i  f 0 .xI N d / for all d 2 Rn . (iii) f 0 .xI N d /  hv; d i  f 0 .xI N d / for all d 2 Rn .



84

2. SUBDIFFERENTIAL CALCULUS

Proof. Picking any v 2 @f .x/ N and t > 0, we get hv; td i  f .xN C td /

f .x/ N whenever d 2 Rn ;

which verifies the implication (i)H)(ii) by taking the limit as t ! 0C . Assuming now that assertion (ii) holds, we get by Lemma 2.83 that hv; d i  f 0 .xI N d /  f .xN C d /

f .x/ N for all d 2 Rn :

It ensures by definition (2.13) that v 2 @f .x/ N , and thus assertions (i) and (ii) are equivalent. It is obvious that (iii) yields (ii). Conversely, if (ii) is satisfied, then for d 2 Rn we have hv; d i  f 0 .xI N d /, and thus f 0 .xI N d/ D

f 0 .xI N d /  hv; d i for any d 2 Rn :

is justifies the validity of (iii) and completes the proof of the theorem.



Let us next list some properties of (2.49) as a function of the direction. Proposition 2.85 For any convex function f W Rn ! R with xN 2 dom f , we define the directional function .d / WD f 0 .xI N d /, which satisfies the following properties:

(i) .0/ D 0. (ii) .d1 C d2 /  .d1 / C .d2 / for all d1 ; d2 2 Rn . (iii) .˛d / D ˛ .d / whenever d 2 Rn and ˛ > 0. (iv) If furthermore xN 2 int.dom f /, then is finite on Rn . Proof. It is straightforward to deduce properties (i)–(iii) directly from definition (2.49). For instance, (ii) is satisfied due to the relationships f .xN C t.d1 C d2 // f .x/ N t xN C 2td1 C xN C 2t d2 / f .x/ N f. 2 D lim t t!0C f .xN C 2t d1 / f .x/ N f .xN C 2t d2 /  lim C lim 2t 2t t!0C t!0C

.d1 C d2 / D lim

t!0C

f .x/ N

D

.d1 / C

.d2 /:

us it remains to check that .d / is finite for every d 2 Rn when xN 2 int.dom f /. To proceed, choose ˛ > 0 so small that xN C ˛d 2 dom f . It follows from Lemma 2.83 that .˛d / D f 0 .xI N ˛d /  f .xN C ˛d /

Employing (iii) gives us 0D

which implies that

f .x/ N < 1:

.d / < 1. Further, we have from (i) and (ii) that  .0/ D d C . d /  .d / C . d /; d 2 Rn ;

.d / 

. d /. is yields

.d / >

1 and so verifies (iv).



2.9. DIRECTIONAL DERIVATIVES

85

To derive the second major result of this section, yet another lemma is needed.

We have the following relationships between a convex function f W Rn ! R and the directional function defined in Proposition 2.85:

Lemma 2.86

(i) @f .x/ N D @ .0/.  (ii) .v/ D ı˝ .v/ for all v 2 Rn , where ˝ WD @ .0/. Proof. It follows from eorem 2.84 that v 2 @f .x/ N if and only if hv; d

0i D hv; d i  f 0 .xI N d/ D

.d / D

.d /

.0/;

d 2 Rn :

is is equivalent to v 2 @ .0/, and hence (i) holds. To justify (ii), let us first show that  .v/ D 0 for all v 2 ˝ D @ .0/. Indeed, we have ˇ ˚  .0/ D 0: .v/ D sup hv; d i .d / ˇ d 2 Rn  hv; 0i Picking now any v 2 @ .0/ gives us hv; d i D hv; d

0i 

.d /

.0/ D

.d /;

d 2 Rn ;

which implies therefore that 

˚ .v/ D sup hv; d i

ˇ .d / ˇ d 2 Rn  0

and so ensures the validity of  .v/ D 0 for any v 2 @ .0/. It remains to verify that  .v/ D 1 if v … @ .0/. For such an element v , find d0 2 Rn with hv; d0 i > .d0 /. Since is positively homogeneous by Proposition 2.85, it follows that ˇ ˚  .t d0 // .v/ D sup hv; d i .d / ˇ d 2 Rn  sup.hv; t d0 i t>0

D sup t.hv; d0 i t>0

.d0 // D 1;

which completes the proof of the lemma.



Now we are ready establish a major relationship between the directional derivative and the subdifferential of an arbitrary convex function. eorem 2.87

Given a convex function f W Rn ! R and a point xN 2 int.dom f /, we have ˇ ˚ f 0 .xI N d / D max hv; d i ˇ v 2 @f .x/ N for any d 2 Rn :

86

2. SUBDIFFERENTIAL CALCULUS

Proof. It follows from Proposition 2.76 that f 0 .xI N d/ D

.d / D



.d /;

d 2 Rn ;

by the properties of the function .d / D f 0 .xI N d / from Proposition 2.85. Note that is a finite convex function in this setting. Employing now Lemma 2.86 tells us that  .v/ D ı˝ .v/, where ˝ D @ .0/ D @f .x/ N . Hence we have by (2.46) that ˇ ˚   .d / D ı˝ .d / D sup hv; d i ˇ v 2 ˝ : Since the set ˝ D @f .x/ N is compact by Proposition 2.47, we complete the proof.



2.10 SUBGRADIENTS OF SUPREMUM FUNCTIONS Let T be a nonempty subset of Rp and let g W T  Rn ! R. For convenience, we also use the notation g t .x/ WD g.t; x/. e supremum function f W Rn ! R for g t over T is f .x/ WD sup g.t; x/ D sup g t .x/; t 2T

t2T

x 2 Rn :

(2.50)

If the supremum in (2.50) is attained (this happens, in particular, when the index set T is compact and g.; x/ is continuous), then (2.50) reduces to the maximum function, which can be written in form (2.35) when T is a finite set. e main goal of this section is to calculate the subdifferential (2.13) of (2.50) when the functions g t are convex. Note that in this case the supremum function (2.50) is also convex by Proposition 1.43. In what follows we assume without mentioning it that the functions g t W Rn ! R are convex for all t 2 T . For any point xN 2 Rn , define the active index set ˇ ˚ N D f .x/ N ; (2.51) S.x/ N WD t 2 T ˇ g t .x/ which may be empty if the supremum in (2.50) is not attained for x D xN . We first present a simple lower subdifferential estimate for (2.66). Proposition 2.88 Let dom f ¤ ; for the supremum function (2.50) over an arbitrary index set T . For any xN 2 dom f , we have the inclusion [ clco @g t .x/ N  @f .x/: N (2.52) t 2S.x/ N

Proof. Inclusion (2.52) obviously holds if S.x/ N D ;. Supposing now that S.x/ N ¤ ;, fix t 2 S.x/ N and v 2 @g t .x/ N . en we get g t .x/ N D f .x/ N and therefore hv; x

xi N  g t .x/

g t .x/ N D g t .x/

f .x/ N  f .x/

f .x/; N

which shows that v 2 @f .x/ N . Since the subgradient set @f .x/ N is a closed and convex, we conclude  that inclusion (2.52) holds in this case.

2.10. SUBGRADIENTS OF SUPREMUM FUNCTIONS

87

Further properties of the supremum/maximum function (2.50) involve the following useful extensions of continuity. Its lower version will be studied and applied in Chapter 4. Definition 2.89 Let ˝ be a nonempty subset of Rn . A function f W ˝ ! R is    xN 2 ˝ if for every sequence fxk g in ˝ converging to xN it holds

lim sup f .xk /  f .x/: N k!1

We say that f is    ˝ if it is upper semicontinuous at every point of this set. Proposition 2.90 Let T be a nonempty, compact subset of Rp and let g.I x/ be upper semicontinuous on T for every x 2 Rn . en f in (2.50) is a finite-valued convex function.

Proof. We need to check that f .x/ < 1 for every x 2 Rn . Suppose by contradiction that f .x/ N D n 1 for some xN 2 R . en there exists a sequence ftk g  T such that g.tk ; x/ N ! 1. Since T is compact, assume without loss of generality that tk ! tN 2 T . By the upper semicontinuity of g.; x/ N on T we have 1 D lim sup g.tk ; x/ N  g.tN; x/ N < 1; k!1

which is a contradiction that completes the proof of the proposition.



Proposition 2.91 Let the index set ; ¤ T  Rp be compact and let g.I x/ be upper semicontinuous for every x 2 Rn . en the active index set S.x/ N  T is nonempty and compact for every xN 2 Rn . Furthermore, we have the compactness in Rn of the convex hull

C WD co

[ t2S.x/ N

@g t .x/: N

Proof. Fix xN 2 Rn and get from Proposition 2.90 that f .x/ N < 1 for the supremum function (2.50). Consider an arbitrary sequence ftk g  T with g.tk ; x/ N ! f .x/ N as k ! 1. By the comN N pactness of T we find t 2 T such that tk ! t along some subsequence. It follows from the assumed upper semicontinuity of g that f .x/ N D lim sup g.tk ; x/ N  g.tN; x/ N  f .x/; N k!1

which ensures that tN 2 S.x/ N by the construction of S in (2.51), and thus S.x/ N ¤ ;. Since g.t; x/ N  f .x/ N , for any t 2 T we get the representation ˇ ˚ S.x/ N D t 2 T ˇ g.t; x/ N  f .x/ N

88

2. SUBDIFFERENTIAL CALCULUS

implying the closedness of S.x/ N (and hence its compactness since S.x/ N  T ), which is an immediate consequence of the upper semicontinuity of g.I x/ N . It remains to show that the subgradient set [ Q WD @g t .x/ N t 2S.x/ N

is compact in Rn ; this immediately implies the claimed compactness of C D co Q, since the convex hull of a compact set is compact (show it, or see Corollary 3.7 in the next chapter). To proceed with verifying the compactness of Q, we recall first that Q  @f .x/ N by Proposition 2.88. is implies that the set Q is bounded in Rn , since the subgradient set of the finite-valued convex function is compact by Proposition 2.47. To check the closedness of Q, take a sequence fvk g  Q converging to some vN and for each k 2 N and find tk 2 S.x/ N such that vk 2 @g tk .x/ N . Since S.x/ N is compact, we may assume that tk ! tN 2 S.x/ N . en g.tk ; x/ N D g.tN; x/ N D f .x/ N and hvk ; x

xi N  g.tk ; x/

g.tk ; x/ N D g.tk ; x/

g.tN; x/ N for all x 2 Rn

by definition (2.13). is gives us the relationships hv; N x

xi N D lim suphvk ; x k!1

xi N  lim sup g.tk ; x/ k!1

g.tN; x/ N  g.tN; x/

which verify that vN 2 @gtN.x/ N  Q and thus complete the proof.

g.tN; x/ N D gtN .x/

gtN .x/; N 

e next result is of its own interest while being crucial for the main subdifferential result of this section. eorem 2.92

In the setting of Proposition 2.91 we have f 0 .xI N v/  C .v/ for all xN 2 Rn ; v 2 Rn

via the support function of the set C defined therein. Proof. It follows from Proposition 2.81 and Proposition 2.82 that 1 < f 0 .xI N v/ D inf>0 './ < 1, where the function ' is defined in Lemma 2.80 and is proved to be nondecreasing on .0; 1/. us there is a strictly decreasing sequence k # 0 as k ! 1 with f .xN C k v/ k k!1

f 0 .xI N v/ D lim

f .x/ N

;

k 2N:

For every k , we select tk 2 T such that f .xN C k v/ D g.tk ; xN C k v/. e compactness of T allows us to find tN 2 T such that tk ! tN 2 T along some subsequence. Let us show that tN 2 S.x/ N . Since g.tk ; / is convex and k < 1 for large k , we have  g.tk ; xN C k v/ D g tk ; k .xN C v/ C .1 k /xN  k g.tk ; xN C v/ C .1 k /g.tk ; x/; N (2.53)

2.10. SUBGRADIENTS OF SUPREMUM FUNCTIONS

89

which implies by the continuity of f (any finite-valued convex function is continuous) and the upper semicontinuity of g.I x/ that f .x/ N D lim f .xN C k v/ D lim g.tk ; xN C k v/  g.tN; x/: N k!1

k!1

is yields f .x/ N D g.tN; x/ N , and so tN 2 S.x/ N . Moreover, for any  > 0 and large k , we get g.tk ; xN C v/  C  with WD g.tN; xN C v/

by the upper semicontinuity of g . It follows from (2.53) that g.tk ; x/ N 

g.tk ; xN C k v/ k . C / ! f .x/; N 1 k

which tells us that limk!1 g.tk ; x/ N D f .x/ N due to f .x/ N  lim inf g.tk ; x/ N  lim sup g.tk ; x/ N  g.tN; x/ N D f .x/: N k!1

k!1

Fix further any  > 0 and, taking into account that k <  for all k sufficiently large, deduce from Lemma 2.80 that f .xN C k v/ k

f .x/ N

g.tk ; xN C k v/ g.tN; x/ N k g.tk ; xN C k v/ g.tk ; x/ N  k g.tk ; xN C v/ g.tk ; x/ N   D

is ensures in turn the estimates lim sup k!1

f .xN C k v/ k

f .x/ N

g.tk ; xN C v/ g.tk ; x/ N  k!1 g.tN; xN C v/ g.tN; x/ N   gtN .xN C v/ gtN .x/ N D ;   lim sup

which implies the relationships f 0 .xI N v/  inf

>0

gtN .xN C v/ 

gtN .x/ N

D gt0N .xI N v/:

Using finally eorem 2.87 and the inclusion @gtN .x/ N  C from Proposition 2.88 give us f 0 .xI N v/  gt0N .xI N v/ D

sup hw; vi  sup hw; vi D C .v/;

w2@gtN .x/ N

and justify therefore the statement of the theorem.

w2C



90

2. SUBDIFFERENTIAL CALCULUS

Now we are ready to establish the main result of this section. Let f be defined in (2.50), where the index set ; ¤ T  Rp is compact, and where the function g.t; / is convex on Rn for every t 2 T . Suppose in addition that the function g.; x/ is upper semicontinuous on T for every x 2 Rn . en we have [ @f .x/ N D co @g t .x/: N eorem 2.93

t 2S.x/ N

Proof. Taking any w 2 @f .x/ N , we deduce from eorem 2.87 and eorem 2.92 that hw; vi  f 0 .x; N v/  C .v/ for all v 2 Rn ;

where C is defined in Proposition 2.91. us w 2 @C .0/ D C by formula (2.44) for the set ˝ D C , which is a nonempty, closed, convex subset of Rn . e opposite inclusion follows from Proposition 2.88. 

2.11 EXERCISES FOR CHAPTER 2 Exercise 2.1 (i) Let ˝  Rn be a nonempty, closed, convex cone and let xN … ˝ . Show that there exists a nonzero element v 2 Rn such that hv; xi  0 for all x 2 ˝ and hv; xi N > 0. n (ii) Let ˝ be a subspace of R and let xN … ˝ . Show that there exists a nonzero element v 2 Rn such that hv; xi D 0 for all x 2 ˝ and hv; xi N > 0.

Exercise 2.2 Let ˝1 and ˝2 be two nonempty, convex subsets of Rn satisfying the condition ri ˝1 \ ri ˝2 D ;. Show that there is a nonzero element v 2 Rn which separates the sets ˝1 and ˝2 in the sense of (2.2).

Exercise 2.3 Two nonempty, convex subsets of Rn are called properly separated if there exists a nonzero element v 2 Rn such that ˇ ˇ ˚ ˚ sup hv; xi ˇ x 2 ˝1  inf hv; yi ˇ y 2 ˝2 ; ˇ ˇ ˚ ˚ inf hv; xi ˇ x 2 ˝1 < sup hv; yi ˇ y 2 ˝2 :

Prove that ˝1 and ˝2 are properly separated if and only if ri ˝1 \ ri ˝2 D ;.

Exercise 2.4 Let f W Rn ! R be a convex function attaining its minimum at some point xN 2

dom f . Show that the sets f˝1 ; ˝2 g with ˝1 WD epi f and ˝2 WD Rn ff .x/g N form an extremal system.

2.11. EXERCISES FOR CHAPTER 2

91

n

Exercise 2.5 Let ˝i for i D 1; 2 be nonempty, convex subsets of R with at least one point in common and satisfy

int ˝1 ¤ ; and int ˝1 \ ˝2 D ;:

Show that there are elements v1 ; v2 in Rn , not equal to zero simultaneously, and real numbers ˛1 ; ˛2 such that hvi ; xi  ˛i for all x 2 ˝i ; i D 1; 2; v1 C v2 D 0; and ˛1 C ˛2 D 0: ˇ

Exercise 2.6 Let ˝ WD fx 2 Rn ˇ x  0g. Show that

ˇ ˚ N D 0 for any xN 2 ˝: N.xI N ˝/ D v 2 Rn ˇ v  0; hv; xi

Exercise 2.7 Let f W Rn ! R be a convex function differentiable at xN . Show that  ˚ ˇ N .x; N f .x//I N epi f D  rf .x/; N 1 ˇ0 :

Exercise 2.8 Use induction to prove Corollary 2.19. Exercise 2.9 Calculate the subdifferentials and the singular subdifferentials of the following convex functions at every point of their domains: (i) f .x/ D jx 1j C jx 2j; x 2 R. (ii) f .x/ D e(jxj ; x 2 R. x 2 1 if jxj  1; (iii) f .x/ D 1 otherwise on R : ( p 1 x 2 if jxj  1; (iv) f .x/ D 1 otherwise on R : 2 (v) f .x1 ; x2 / D jx1 j C˚ jx2 j; .x 1 ; x2 / 2 R . (vi) f .x1 ; x2 / D max jx1 j; jx2 j ; .x1 ; x2 / 2 R2 . ˚ (vii) f .x/ D max ha; xi b; 0 ; x 2 Rn , where a 2 Rn and b 2 R.

Exercise 2.10 Let f W R ! R be a nondecreasing convex function and let xN 2 R. Show that v  0 for v 2 @f .x/ N .

Exercise 2.11 Use induction to prove Corollary 2.46. Exercise 2.12 Let f W Rn ! R be a convex function. Give an alternative proof for the part of

Proposition 2.47 stating that @f .x/ N ¤ ; for any element xN 2 int.dom f /. Does the conclusion still hold if xN 2 ri.dom f /?

92

2. SUBDIFFERENTIAL CALCULUS

Exercise 2.13 Let ˝1 and ˝2 be nonempty, convex subsets of Rn . Suppose that ri ˝1 \ ri ˝2 ¤

;. Prove that

N.xI N ˝1 \ ˝2 / D N.xI N ˝1 / C N.xI N ˝2 / for all xN 2 ˝1 \ ˝2 :

Exercise 2.14 Let f1 ; f2 W Rn ! R be convex functions satisfying the condition ri .dom f1 / \ ri .dom f2 / ¤ ;: Prove that for all xN 2 dom f1 \ dom f2 we have @.f1 C f2 /.x/ N D @f1 .x/ N C @f2 .x/; N

@1 .f1 C f2 /.x/ N D @1 f1 .x/ N C @1 f2 .x/: N

Exercise 2.15 In the setting of eorem 2.51 show that N..x; N y; N z/I N ˝1 / \ Œ N..x; N y; N zN /I ˝2 / D f0g

under the qualification condition (2.33).

Exercise 2.16 Calculate the singular subdifferential of the composition f ı B in the setting of eorem 2.51.

Exercise 2.17 Consider the composition g ı h in the setting of Proposition 1.54 and take any xN 2 dom .g ı h/. Prove that @.g ı h/.x/ N D

p nX iD1

ˇ o ˇ N vi 2 @fi .x/; N i D 1; : : : ; p i vi ˇ .1 ; : : : ; p / 2 @g.y/;

under the validity of the qualification condition h

02

p X iD1

i   i @fi .x/; N .1 ; : : : ; p / 2 @1 g.y/ N H) i D 0 for i D 1; : : : ; p :

Exercise 2.18 Calculate the singular subdifferential of the optimal value functions in the setting of eorem 2.61.

Exercise 2.19 Calculate the support functions for the following sets:

ˇ ˚ P (i) ˝1 WD ˚x D .x1 ; : : : ; xn / 2 Rn ˇˇkxk1 WD niD1 jxi j  1 : (ii) ˝2 WD x D .x1 ; : : : ; xn / 2 Rn ˇ kxk1 WD maxniD1 jxi j  1 :

2.11. EXERCISES FOR CHAPTER 2

93

Exercise 2.20 Find the Fenchel conjugate for each of the following functions on R: (i) f .x/ D ex .

ln.x/ x > 0; 1 x  0: 2 (iii) f .x/ D ax C bx C c , where a > 0. (iv) f .x/ D ıIB .x/, where IB D Œ 1; 1. (v) f .x/ D jxj.

(ii) f .x/ D

 Exercise 2.21 Let ˝ be a nonempty subset of Rn . Find the biconjugate function ı˝ .

Exercise 2.22 Prove the following statements for the listed functions on Rn :

v , where  > 0.    (ii) .f C / .v/ D f .v/ , where  2 R. (iii) fu .v/ D f  .v/ C hu; vi, where u 2 Rn and fu .x/ WD f .x

(i) .f / .v/ D f 

u/.

Exercise 2.23 Let f W Rn ! R be a convex and positively homogeneous function. Show that

f  .v/ D ı˝ .v/ for all v 2 Rn , where ˝ WD @f .0/.

Exercise 2.24 Let f W Rn ! R be a function with dom f D ;. Show that f  .x/ D f .x/ for all x 2 Rn .

Exercise 2.25 Let f W R ! R be convex. Prove that @f .x/ D Œf 0 .x/; fC0 .x/, where f 0 .x/ and fC0 .x/ stand for the left and right derivative of f at x , respectively.

Exercise 2.26 Define the function f W R ! R by f .x/ WD

( p 1 1

x2

if jxj  1; otherwise:

Show that f is a convex function and calculate its directional derivatives f 0 . 1I d / and f 0 .1I d / in the direction d D 1.

Exercise 2.27 Let f W Rn ! R be convex and let xN 2 dom f . For an element d 2 Rn , consider the function ' defined in Lemma 2.80 and show that ' is nondecreasing on . 1; 0/.

Exercise 2.28 Let f W Rn ! R be convex and let xN 2 dom f . Prove the following:

(i) f 0 .xI N d /  f 0 .xI N d / for all d 2 Rn . (ii) f .x/ N f .xN d /  f 0 .xI N d /  f 0 .xI N d /  f .xN C d /

Exercise 2.29 Prove the assertions of Proposition 2.67.

f .x/ N , d 2 Rn .

94

2. SUBDIFFERENTIAL CALCULUS

Exercise 2.30 Let ˝  Rn be a nonempty, compact set. Using eorem 2.93, calculate the subdifferential of the support function @˝ .0/.

Exercise 2.31 Let ˝ be a nonempty, compact subset of Rn . Define the function ˚ ˝ .x/ WD sup kx

ˇ !k ˇ ! 2 ˝g;

x 2 Rn :

Prove that ˝ is a convex function and find its subdifferential @˝ .x/ at any x 2 Rn .

Exercise 2.32 Using the representation ˇ ˚ kxk D sup hv; xi ˇ kvk  1 N of the norm function p.x/ WD kxk at any and eorem 2.93, calculate the subdifferential @p.x/ point xN 2 Rn .

Exercise 2.33 Let d.xI ˝/ be the distance function (1.5) associated with some nonempty, closed, convex subset ˝ of Rn . (i) Prove the representation ˚ d.xI ˝/ D sup hx; vi

ˇ ˝ .v/ ˇ kvk  1

via the support function of ˝ . (ii) Calculate the subdifferential @d.xI N ˝/ at any point xN 2 Rn .

95

CHAPTER

3

Remarkable Consequences of Convexity is chapter is devoted to remarkable topics of convex analysis that, together with subdifferential calculus and related results of Chapter 2, constitute the major achievements of this discipline most important for many applications—first of all to convex optimization.

3.1

CHARACTERIZATIONS OF DIFFERENTIABILITY

In this section we define another classical notion of differentiability, the Gâteaux derivative, and show that for convex functions it can be characterized, together with the Fréchet one, via a singleelement subdifferential of convex analysis. Definition 3.1 Let f W Rn ! R be finite around xN . We say that f is G  at xN if there is v 2 Rn such that for any d 2 Rn we have

lim

t !0

f .xN C td /

f .x/ N t

thv; d i

D 0:

en we call v the G  of f at xN and denote by fG0 .x/ N . It follows from the definition and the equality f 0 .xI N d / D f 0 .xI N d / that fG0 .x/ N D v if 0 and only if the directional derivative f .xI N d / exists and is linear with respect to d with f 0 .xI N d/ D hv; d i for all d 2 Rn ; see Exercise 3.1 and its solution. Moreover, if f is Gâteaux differentiable at xN , then f .xN C td / f .x/ N hfG0 .x/; N d i D lim for all d 2 Rn : t !0 t

Let f W Rn ! R be a function (not necessarily convex) which is locally Lipschitz continuous around xN 2 dom f . en the following assertions are equivalent:

Proposition 3.2

(i) f is Fréchet differentiable at xN . (ii) f is Gâteaux differentiable at xN .

96

3. REMARKABLE CONSEQUENCES OF CONVEXITY

Proof. Let f be Fréchet differentiable at xN and let v WD rf .x/ N . For a fixed 0 ¤ d 2 Rn , taking into account that kxN C td xk N D jtj  kd k ! 0 as t ! 0, we have

lim

t!0

f .xN C td /

f .x/ N t

thv; d i

D kd k lim

t !0

f .xN C t d /

f .x/ N tkd k

hv; t d i

D 0;

which implies the GaO teaux differentiablity of f at xN with fG0 .x/ N D rf .x/ N . Note that the local Lipschitz continuity of f around xN is not required in this implication. Assuming now that f is GaO teaux differentiable at xN with v WD fG0 .x/ N , let us show that lim

h!0

f .xN C h/

f .x/ N khk

hv; hi

D 0:

Suppose by contradiction that it does not hold. en we can find 0 > 0 and a sequence of hk ! 0 with hk ¤ 0 as k 2 N such that ˇ ˇf .xN C hk / f .x/ N hv; hk ij  0 : (3.1) khk k hk . en tk ! 0, and we can assume without loss of generality khk k that dk ! d with kd k D 1. is gives us the estimate ˇ ˇf .xN C tk dk / f .x/ N hv; tk dk ij 0 kdk k  : tk

Denote tk WD khk k and dk WD

Let `  0 be a Lipschitz constant of f around xN . e triangle inequality ensures that ˇ ˇf .xN C tk dk / f .x/ N hv; tk dk ij 0 kdk k  tk ˇ ˇ ˇf .xN C tk dk / f .xN C tk d /j C jhv; tk d i hv; tk dk ij ˇf .xN C tk d / f .x/ N hv; tk d ij  C tk tk ˇ ˇf .xN C tk d / f .x/ N hv; tk d ij  `kdk d k C kvk  kdk d k C ! 0 as k ! 1: tk is contradicts (3.1) since 0 kdk k ! 0 , so f is Fréchet differentiable at xN .



Now we are ready to derive a nice subdifferential characterization of differentiability. Let f W Rn ! R be a convex function with xN 2 int(domf /. e following assertions are equivalent:

eorem 3.3

(i) f is Fréchet differentiable at xN . (ii) f is Gâteaux differentiable at xN . (iii) @f .x/ N is a singleton.

3.2. CARATHÉODORY THEOREM AND FARKAS LEMMA

97

Proof. It follows from eorem 2.29 that f is locally Lipschitz continuous around xN , and hence (i) and (ii) are equivalent by Proposition 3.2. Suppose now that (ii) is satisfied and let v WD fG0 .x/ N . en f 0 .xI N d / D f 0 .xI N d / D hv; d i for all d 2 Rn . Observe that @f .x/ N ¤ ; since xN 2 int(domf /. Fix any w 2 @f .x/ N . It follows from eorem 2.84 that f 0 .xI N d /  hw; d i  f 0 .xI N d / whenever d 2 Rn ;

and thus hw; d i D hv; d i, so hw v; d i D 0 for all d . Plugging there d D w v gives us w D v and thus justifies that @f .x/ N is a singleton. Suppose finally that (iii) is satisfied with @f .x/ N D fvg and then show that (ii) holds. By eorem 2.87 we have the representation f 0 .xI N d/ D

sup hw; d i D hv; d i;

w2@f .x/ N

for all d 2 Rn . is implies that f is GaO teaux and hence Fréchet differentiable at xN .

3.2



CARATHÉODORY THEOREM AND FARKAS LEMMA

We begin with preliminary results on conic and convex hulls of sets that lead us to one of the most remarkable results of convex geometry established by Constantin Carathéodory in 1911. Recall that a set ˝ is called a cone if x 2 ˝ for all   0 and x 2 ˝ . It follows from the definition that 0 2 ˝ if ˝ is a nonempty cone. Given a nonempty set ˝  Rn , we say that K˝ is the convex cone generated by ˝ , or the convex conic hull of ˝ , if it is the intersection of all the convex cones containing ˝ .

Figure 3.1: Convex cone generated by a set.

98

3. REMARKABLE CONSEQUENCES OF CONVEXITY

Let ˝ be a nonempty subset of Rn . en the convex cone generated by ˝ admits the following representation: Proposition 3.4

K˝ D

m nX iD1

ˇ o ˇ i ai ˇ i  0; ai 2 ˝; m 2 N :

(3.2)

If, in particular, the set ˝ is convex, then K˝ D RC ˝ . Proof. Denote by C the set on the right-hand side of (3.2) and observe that it is a convex cone which contains ˝ . It remains to show that C  K for any convex cone K containing ˝ . To P see it, fix such a cone and form the combination x D m iD1 i ai with any i  0, ai 2 ˝ , and m 2 N . If i D 0 for all i , then x D 0 2 K . Otherwise, suppose that i > 0 for some i and denote P  WD m iD1 i > 0. is tells us that m X i  xD ai 2 K;  iD1

which yields C  K , so K˝ D C . e last assertion of the proposition is obvious. Proposition 3.5



For any nonempty set ˝  Rn and any x 2 K˝ n f0g, we have

xD

m X iD1

i ai with i > 0; ai 2 ˝ as i D 1; : : : ; m and m  n:

P Proof. Let x 2 K˝ n f0g. It follows from Proposition 3.4 that x D m iD1 i ai with some i > 0 and ai 2 ˝ as for i D 1; : : : ; m and m 2 N . If the elements a1 ; : : : ; am are linearly dependent, then there are numbers i 2 R, not all zeros, such that m X iD1

i ai D 0:

We can assume that I WD fi D 1; : : : ; m j i > 0g ¤ ; and get for any  > 0 that xD

m X iD1

i ai D

m X iD1

i a i



m X iD1

m   X

i ai D i iD1

n ˇ o i i ˇ Let  WD min ˇ i 2 I D 0 , where i0 2 I , and denote ˇi WD i

i

i0 en we have the relationships xD

m X iD1

  i ai :  i for i D 1; : : : ; m.

ˇi ai with ˇi0 D 0 and ˇi  0 for any i D 1; : : : ; m:

3.2. CARATHÉODORY THEOREM AND FARKAS LEMMA

99

Continuing this procedure, after a finite number of steps we represent x as a positive linear combination of linearly independent elements faj j j 2 J g, where J  f1; : : : ; mg. us the index set J cannot contain more than n elements.  e following remarkable result is known as the Carathéodory theorem. It relates convex hulls of sets with the dimension of the space; see an illustration in Figure 3.2.

Figure 3.2: Carathéodory theorem.

Let ˝ be a nonempty subset of Rn . en every x 2 co ˝ can be represented as a convex combination of no more than n C 1 elements of the set ˝ . eorem 3.6

Proof. Denoting B WD f1g  ˝  RnC1 , it is easy to observe that co B D f1g  co ˝ and that co B  KB . Take now any x 2 co ˝ . en .1; x/ 2 co B and by Proposition 3.5 find i  0 and .1; ai / 2 B for i D 0; : : : ; m with m  n such that .1; x/ D

is implies that x D

Pm

iD0

i ai with

Pm

iD0

m X

i .1; ai /:

iD0

i D 1, i  0, and m  n.



e next useful assertion is a direct consequence of eorem 3.6. Corollary 3.7

Suppose that a subset ˝ of Rn is compact. en the set co ˝ is also compact.

Now we present two propositions, which lead us to another remarkable result known as the Farkas lemma; see eorem 3.10.

100

3. REMARKABLE CONSEQUENCES OF CONVEXITY

Proposition 3.8 Let a1 ; : : : ; am be linearly dependent elements in Rn , where ai ¤ 0 for all i D 1; : : : ; m. Define the sets ˚ ˚ ˝ WD a1 ; : : : ; am and ˝i WD ˝ n ai ; i D 1; : : : ; m:

en the convex cones generated by these sets are related as K˝ D

m [

(3.3)

K˝ i :

iD1

Proof. Obviously, [m iD1 K˝i  K˝ . To prove the converse inclusion, pick x 2 K˝ and get xD

m X iD1

i ai with i  0;

i D 1; : : : ; m:

If x D 0, then x 2 [m iD1 K˝i obviously. Otherwise, the linear dependence of ai allows us to find

i 2 R not all zeros such that m X

i ai D 0: iD1

Following the proof of Proposition 3.5, we represent x as xD

m X iD1;i ¤i0

i ai with some i0 2 f1; : : : ; mg:

is shows that x 2 [m iD1 K˝i and thus verifies that equality (3.3) holds.



Proposition 3.9 Let a1 ; : : : ; am be elements of Rn . en the convex cone K˝ generated by the set ˝ WD fa1 ; : : : ; am g is closed in Rn .

Proof. Assume first that the elements a1 ; : : : ; am are linearly independent and take any sequence fxk g  K˝ converging to x . By the construction of K˝ , find numbers ˛ki  0 for each i D 1; : : : ; m and k 2 N such that m X xk D ˛ki ai : iD1

m

Letting ˛k WD .˛k1 ; : : : ; ˛km / 2 R and arguing by contradiction, it is easy to check that the sequence f˛k g is bounded. is allows us to suppose without loss of generality that ˛k ! .˛1 ; : : : ; ˛m / as k ! 1 with ˛i  0 for all i D 1; : : : ; m. us we get by passing to the limit P that xk ! x D m iD1 ˛i ai 2 K˝ , which verifies the closedness of K˝ .

3.2. CARATHÉODORY THEOREM AND FARKAS LEMMA

101

Considering next arbitrary elements a1 ; : : : ; am , assume without loss of generality that ai ¤ 0 for all i D 1; : : : ; m. By the above, it remains to examine the case where a1 ; : : : ; am are linearly dependent. en by Proposition 3.8 we have the cone representation (3.3), where each set ˝i contains m 1 elements. If at least one of the sets ˝i consists of linearly dependent elements, we can represent the corresponding cone K˝i in a similar way via the sets of m 2 elements. is gives us after a finite number of steps that K˝ D

p [

KCj ;

j D1

where each set Cj  ˝ contains only linear independent elements, and so the cones KCj are closed. us the cone K˝ under consideration is closed as well.  Now we are ready to derive the fundamental result of convex analysis and optimization established by Gyula Farkas in 1894. Let ˝ WD fa1 ; : : : ; am g with a1 ; : : : ; am 2 Rn and let b 2 Rn . e following assertions are equivalent:

eorem 3.10

(i) b 2 K˝ . (ii) For any x 2 Rn , we have the implication     hai ; xi  0; i D 1; : : : ; m H) hb; xi  0 : Proof. To verify .i/ H) .ii/, take b 2 K˝ and find i  0 for i D 1; : : : ; m such that bD

m X

i ai :

iD1

en, given x 2 Rn , the inequalities hai ; xi  0 for all i D 1; : : : ; m imply that hb; xi D

m X iD1

i hai ; xi  0;

which is (ii). To prove the converse implication, suppose that b … K˝ and show that (ii) does not hold in this case. Indeed, since the set K˝ is closed and convex, the strict separation result of Proposition 2.1 yields the existence of xN 2 Rn satisfying ˇ ˚ sup hu; xi N ˇ u 2 K˝ g < hb; xi: N Since K˝ is a cone, we have 0 D h0; xi N < hb; xi N and t u 2 K˝ for all t > 0, u 2 K˝ . us ˇ ˚ t sup hu; xi N ˇ u 2 K˝ < hb; xi N whenever t > 0:

102

3. REMARKABLE CONSEQUENCES OF CONVEXITY

Dividing both sides of this inequality by t and letting t ! 1 give us ˇ ˚ sup hu; xi N ˇ u 2 K˝  0: It follows that hai ; xi N  0 for all i D 1; : : : ; m while hb; xi N > 0. is is a contradiction, which therefore completes the proof of the theorem. 

3.3

RADON THEOREM AND HELLY THEOREM

In this section we present a celebrated theorem of convex geometry discovered by Edward Helly in 1913. Let us start with the following simple lemma.

Consider arbitrary elements w1 ; : : : ; wm in Rn with m  n C 2. en these elements are affinely dependent. Lemma 3.11

Proof. Form the elements w2 w1 ; : : : ; wm w1 in Rn and observe that these elements are linearly dependent since m 1 > n. us the elements w1 ; : : : ; wm are affinely dependent by Proposition 1.63. 

We continue with a theorem by Johann Radon established in 1923 and used by himself to give a simple proof of the Helly theorem. eorem 3.12 Consider the elements w1 ; : : : ; wm of Rn with m  n C 2 and let I WD f1; : : : ; mg. en there exist two nonempty, disjoint subsets I1 and I2 of I with I D I1 [ I2 such that for ˝1 WD fwi j i 2 I1 g, ˝2 WD fwi j i 2 I2 g one has

co ˝1 \ co ˝2 ¤ ;: Proof. Since m  n C 2, it follows from Lemma 3.11 that the elements w1 ; : : : ; wm are affinely dependent. us there are real numbers 1 ; : : : ; m , not all zeros, such that m X iD1

i wi D 0;

m X iD1

i D 0:

Consider the index sets I1 WD fi D 1; : : : ; m j i  0g and I2 WD fi D 1; : : : ; m j i < 0g. Since Pm P P  D 0, both sets I1 and I2 are nonempty and i2I1 i D i2I2 i . Denoting  WD PiD1 i  , we have the equalities i2I1 i X i2I1

i wi D

X i2I2

i wi and

X i X i wi D wi :  

i2I1

i2I2

3.4. TANGENTS TO CONVEX SETS

103

Define ˝1 WD fwi j i 2 I1 g, ˝2 WD fwi j i 2 I2 g and observe that co ˝1 \ co ˝2 ¤ ; since X i X i wi D wi 2 co ˝1 \ co ˝2 :  

i2I1

i2I2

is completes the proof of the Radon theorem.



Now we are ready to formulate and prove the fundamental Helly theorem. ˚ eorem 3.13 Let O WD ˝1 ; : : : ; ˝m be a collection of convex sets in Rn with m  n C 1. Suppose that the intersection of any subcollection of n C 1 sets from O is nonempty. en we T have m iD1 ˝i ¤ ;. Proof. Let us prove the theorem by using induction with respect to the number of elements in O. e conclusion of the theorem is obvious if m D n C 1. Suppose that the conclusion holds for ˚ any collection of m convex sets of Rn with m  n C 1. Let ˝1 ; : : : ; ˝m ; ˝mC1 be a collection of convex sets in Rn such that the intersection of any subcollection of n C 1 sets is nonempty. For i D 1; : : : ; m C 1, define mC1 \ i WD ˝j j D1;j ¤i

and observe that i  ˝j whenever j ¤ i and i; j D 1; : : : ; m C 1. It follows from the induction hypothesis that i ¤ ;, and so we can pick wi 2 i for every i D 1; : : : ; m C 1. Applying the above Radon theorem for the elements w1 ; : : : wmC1 , we find two nonempty, disjoint subsets index sets I1 and I2 of I WD f1; : : : ; m C 1g with I D I1 [ I2 such that for W1 WD fwi j i 2 I1 g and W2 WD fwi j i 2 I2 g one has co W1 \ co W2 ¤ ;. is allows us to pick an elemC1 ment w 2 co W1 \ co W2 . We finally show that w 2 \iD1 ˝i and thus arrive at the conclusion. Since i ¤ j for i 2 I1 and j 2 I2 , it follows that i  ˝j whenever i 2 I1 and j 2 I2 . Fix any i D 1; : : : ; m C 1 and consider the case where i 2 I1 . en wj 2 j  ˝i for every j 2 I2 . Hence w 2 co W2 D co fwj j j 2 I2 g  ˝i due the convexity of ˝i . It shows that w 2 ˝i for every i 2 I1 . Similarly, we can verify that w 2 ˝i for every i 2 I2 and thus complete the proof of the theorem. 

3.4

TANGENTS TO CONVEX SETS

In Chapter 2 we studied the normal cone to convex sets and learned that the very definition and major properties of it were related to convex separation. In this section we define the tangent cone to convex sets independently while observing that it can be constructed via duality from the normal cone to the set under consideration. We start with cones. Given a nonempty, convex cone C  Rn , the polar of C is ˇ ˚ C ı WD v 2 Rn ˇ hv; ci  0 for all c 2 C :

104

3. REMARKABLE CONSEQUENCES OF CONVEXITY

Proposition 3.14

For any nonempty, convex cone C  Rn , we have .C ı /ı D cl C:

Proof. e inclusion C  .C ı /ı follows directly from the definition, and hence cl C  .C ı /ı since .C ı /ı is a closed set. To verify the converse inclusion, take any c 2 .C ı /ı and suppose by contradiction that c … cl C . Since C is a cone, by convex separation from Proposition 2.1, find v ¤ 0 such that hv; xi  0 for all x 2 C and hv; ci > 0;

which yields v 2 C ı thus contradicts to the fact that c 2 .C ı /ı . Definition 3.15



Let ˝ be a nonempty, convex set. e   to ˝ at xN 2 ˝ is T .xI N ˝/ WD cl Kf˝

xg N

D cl RC .˝

x/: N

Figure 3.3: Tangent cone and normal cone.

Now we get the duality correspondence between normals and tangents to convex sets; see a demonstration in Figure 3.3. eorem 3.16

Let ˝ be a convex set and let xN 2 ˝ . en we have the equalities  ı  ı N.xI N ˝/ D T .xI N ˝/ and T .xI N ˝/ D N.xI N ˝/ :

(3.4)

3.4. TANGENTS TO CONVEX SETS

105

ı

Proof. To verify first the inclusion N.xI N ˝/  ŒT .xI N ˝/ , pick any v 2 N.xI N ˝/ and get by definition that hv; x xi N  0 for all x 2 ˝ . is implies that ˝ ˛ v; t.x x/ N  0 whenever t  0 and x 2 ˝:

Fix w 2 T .xI N ˝/ and find tk  0 and xk 2 ˝ with tk .xk hv; wi D lim hv; tk .xk k!1

x/ N ! w as k ! 1. en  ı x/i N  0 and thus v 2 T .xI N ˝/ :

To verify the opposite inclusion in (3.4), fix any v 2 ŒT .xI N ˝/ı and deduce from the relations hv; wi  0 as w 2 T .xI N ˝/ and x xN 2 T .xI N ˝/ as x 2 ˝ that hv; x

xi N  0 for all x 2 ˝:

is yields w 2 N.xI N ˝/ and thus completes the proof of the first equality in (3.4). e second equality in (3.4) follows from the first one due to  ı  ıı N.xI N ˝/ D T .xI N ˝/ D cl T .xI N ˝/ by Proposition 3.14 and the closedness of T .xI N ˝/ by Definition 3.15.



e next result, partly based on the Farkas lemma from eorem 3.10, provides effective representations of the tangent and normal cones to linear inequality systems. eorem 3.17

Let ai 2 Rn , let bi 2 R for i D 1; : : : ; m, and let ˇ ˚ ˝ WD x 2 Rn ˇ hai ; xi  bi :

Suppose that xN 2 ˝ and define the active index set I.x/ N WD fi D 1; : : : ; m j hai ; xi N D bi g. en we have the relationships ˇ ˚ T .xI N ˝/ D v 2 Rn ˇ hai ; vi  0 for all i 2 I.x/ N ; (3.5) ˚ ˇ N : (3.6) N.xI N ˝/ D cone ai ˇ i 2 I.x/ Here we use the convention that cone ; WD f0g. Proof. To justify (3.5), consider the convex cone ˇ ˚ K WD v 2 Rn ˇ hai ; vi  0 for all i 2 I.x/ N :

It is easy to see that for any x 2 ˝ , any i 2 I.x/ N , and any t  0 we have ˝ ˛  ai ; t.x x/ N D t hai ; xi hai ; xi N  t.bi bi / D 0;

106

3. REMARKABLE CONSEQUENCES OF CONVEXITY

and hence RC .˝

x/ N  K . us it follows from the closedness of K that T .xI N ˝/ D cl RC .˝

x/ N  K:

To verify the opposite inclusion in (3.5), observe that hai ; xi N < bi for i … I.x/ N . Using this and picking an arbitrary element v 2 K give us hai ; xN C tvi  bi for all i … I.x/ N and small t > 0:

Since hai ; vi  0 for all i 2 I.x/ N , it tells us that hai ; xN C t vi  bi as i D 1; : : : ; m yielding xN C tv 2 ˝; or v 2

1 ˝ t

 xN  T .xI N ˝/:

is justifies the tangent cone representation (3.5). Next we prove (3.6) for the normal cone. e first equality in (3.4) ensures that w 2 N.xI N ˝/ if and only if hw; vi  0 for all v 2 T .xI N ˝/:

en we see from (3.5) that w 2 N.xI N ˝/ if and only if     hai ; vi  0 for all i 2 I.x/ N H) hw; vi  0 whenever v 2 Rn . e Farkas lemma from eorem 3.10 tells us that w 2 N.xI N ˝/ if and only if w 2 conefai j i 2 I.x/g N , and thus (3.6) holds. 

3.5

MEAN VALUE THEOREMS

e mean value theorem for differentiable functions is one of the central results of classical real analysis. Now we derive its subdifferential counterparts for general convex functions. eorem 3.18 Let f W R ! R be convex, let a < b , and let the interval Œa; b belong to the domain of f , which is an open interval in R. en there is c 2 .a; b/ such that

f .b/ b

f .a/ 2 @f .c/: a

(3.7)

Proof. Define the real-valued function gW Œa; b ! R by g.x/ WD f .x/

h f .b/ b

f .a/ .x a

a/ C f .a/

i

for which g.a/ D g.b/. is implies, by the convexity and hence continuity of g on its open domain, that g has a local minimum at some c 2 .a; b/. e function i h f .b/ f .a/ .x a/ C f .a/ h.x/ WD b a

3.5. MEAN VALUE THEOREMS

107

Figure 3.4: Subdifferential mean value theorem.

is obviously differentiable at c , and hence ˚ n @h.c/ D h0 .c/ D

f .b/ b

f .a/ o : a

e subdifferential Fermat rule and the subdifferential sum rule from Chapter 2 yield n f .b/

0 2 @g.c/ D @f .c/

b

f .a/ o ; a

which implies (3.7) and completes the proof of the theorem.



To proceed further with deriving a counterpart of the mean value theorem for convex functions on Rn (eorem 3.20), we first present the following lemma. Lemma 3.19 Let f W Rn ! R be a convex function and let D be the domain of f , which is an open subset of Rn . Given a; b 2 D with a ¤ b , define the function 'W R ! R by

'.t / WD f tb C .1

 t /a ;

(3.8)

t 2 R:

en for any number t0 2 .0; 1/ we have ˇ ˚ @'.t0 / D hv; b ai ˇ v 2 @f .c0 / with c0 WD t0 a C .1

t0 /b:

Proof. Define the vector-valued functions AW R ! Rn and BW R ! Rn by A.t/ WD .b

a/t and B.t/ WD t b C .1

t/a D t .b

a/ C a D A.t / C a;

t 2 R:

108

3. REMARKABLE CONSEQUENCES OF CONVEXITY

It is clear that '.t/ D .f ı B/.t / and that the adjoint mapping to A is A v D hv; b By the subdifferential chain rule from eorem 2.51, we have ˇ ˚ @'.t0 / D A @f .c0 / D hv; b ai ˇ v 2 @f .c0 / ; which verifies the result claimed in the lemma.

ai, v 2 Rn .



Let f W Rn ! R be a convex function in the setting of Lemma 3.19. en there exists an element c 2 .a; b/ such that ˇ ˝ ˛ ˚ f .b/ f .a/ 2 @f .c/; b a WD hv; b ai ˇ v 2 @f .c/ :

eorem 3.20

Proof. Consider the function 'W R ! R from (3.8) for which '.0/ D f .a/ and '.1/ D f .b/. Applying first eorem 3.18 and then Lemma 3.19 to ' , we find t0 2 .0; 1/ such that ˇ ˚ f .b/ f .a/ D '.1/ '.0/ 2 @'.t0 / D hv; b ai ˇ v 2 @f .c/

with c D t0 a C .1

3.6

t0 /b . is justifies our statement.



HORIZON CONES

is section concerns behavior of unbounded convex sets at infinity. Definition 3.21 Given a nonempty, closed, convex subset F of Rn and a point x 2 F , the   of F at x is defined by the formula ˇ ˚ F1 .x/ WD d 2 Rn ˇ x C t d 2 F for all t > 0 :

Note that the horizon cone is also known in the literature as the asymptotic cone of F at the point in question. Another equivalent definition of F1 .x/ is given by F1 .x/ D

\F t>0

x t

;

which clearly implies that F1 .x/ is a closed, convex cone in Rn . e next proposition shows that F1 .x/ is the same for any x 2 F , and so it can be simply denoted by F1 . Proposition 3.22 Let F be a nonempty, closed, convex subset of Rn . en we have the equality F1 .x1 / D F1 .x2 / for any x1 ; x2 2 F .

3.6. HORIZON CONES

109

Figure 3.5: Horizon cone.

Proof. It suffices to verify that F1 .x1 /  F1 .x2 / whenever x1 ; x2 2 F . Taking any direction d 2 F1 .x1 / and any number t > 0, we show that x2 C t d 2 F . Consider the sequence    1 1 xk WD x1 C kt d C 1 x2 ; k 2 N : k k

en xk 2 F for every k because d 2 F1 .x1 / and F is convex. We also have xk ! x2 C td , and so x2 C td 2 F since F is closed. us d 2 F1 .x2 /.  Corollary 3.23

Let F be a closed, convex subset of Rn that contains the origin. en \ F1 D tF: t >0

Proof. It follows from the construction of F1 and Proposition 3.22 that F1 D F1 .0/ D

\F

t >0

0 t

D

\1 \ F D tF; t t>0 t>0

which therefore justifies the claim.



Let F be a nonempty, closed, convex subset of Rn . For a given element d 2 Rn , the following assertions are equivalent:

Proposition 3.24

(i) d 2 F1 . (ii) ere are sequences ftk g  Œ0; 1/ and ffk g  F such that tk ! 0 and tk fk ! d .

110

3. REMARKABLE CONSEQUENCES OF CONVEXITY

Proof. To verify (i)H)(ii), take d 2 F1 and fix xN 2 F . It follows from the definition that xN C kd 2 F for all k 2 N :

For each k 2 N , this allows us to find fk 2 F such that xN C kd D fk ; or equivalently,

1 1 xN C d D fk : k k

1 , we see that tk fk ! d as k ! 1. k To prove (ii)H)(i), suppose that there are sequences ftk g  Œ0; 1/ and ffk g  F with tk ! 0 and tk fk ! d . Fix x 2 F and verify that d 2 F1 by showing that

Letting tk WD

x C td 2 F for all t > 0:

Indeed, for any fixed t > 0 we have 0  t  tk < 1 when k is sufficiently large. us .1

t  tk /x C t  tk fk ! x C td as k ! 1:

It follows from the convexity of F that every element .1 t  tk /x C t  tk fk belongs to F . Hence x C td 2 F by the closedness, and thus we get d 2 F1 .  Finally in this section, we present the expected characterization of the set boundedness in terms of its horizon cone. Let F be a nonempty, closed, convex subset of Rn . en the set F is bounded if and only if its horizon cone is trivial, i.e., F1 D f0g. eorem 3.25

Proof. Suppose F is bounded and take any d 2 F1 . By Proposition 3.24, there exist sequences ftk g  Œ0; 1/ with tk ! 0 and ffk g  F such that tk fk ! d as k ! 1. It follows from the boundedness of F that tk fk ! 0, which shows that d D 0. To prove the converse implication, suppose by contradiction that F is unbounded while F1 D f0g. en there is a sequence fxk g  F with kxk k ! 1. is allows us to form the sequence of (unit) directions xk dk WD ; k 2N: kxk k

It ensures without loss of generality that dk ! d as k ! 1 with kd k D 1. Fix any x 2 F and observe that for all t > 0 and k 2 N sufficiently large we have   t t uk WD 1 xC xk 2 F kxk k kxk k

by the convexity of F . Furthermore, x C td 2 F since uk ! x C td and F is closed. us d 2 F1 , which is a contradiction that completes the proof of the theorem. 

3.7. MINIMAL TIME FUNCTIONS AND MINKOWSKI GAUGE

3.7

111

MINIMAL TIME FUNCTIONS AND MINKOWSKI GAUGE

e main object of this and next sections is a remarkable class of convex functions known as the minimal time function. ey are relatively new in convex analysis and highly important in applications. Some of their recent applications to location problems are given in Chapter 4. We also discuss relationships between minimal time functions and Minkowski gauge functions wellrecognized in convex analysis and optimization.

Let F be a nonempty, closed, convex subset of Rn and let ˝  Rn be a nonempty set, not necessarily convex. e    associated with the sets F and ˝ is defined by ˇ ˚ T˝F .x/ WD inf t  0 ˇ .x C tF / \ ˝ ¤ ; : (3.9)

Definition 3.26

Figure 3.6: Minimal time functions.

e name comes from the following interpretation: (3.9) signifies the minimal time needed for the point x to reach the target set ˝ along the constant dynamics F ; see Figure 3.6. Note that when F D IB , the closed unit ball in Rn , the minimal time function (3.9) reduces to the distance function d.xI ˝/ considered in Section 2.5. In the case where F D f0g, the minimal time function (3.9) is the indicator function of ˝ . If ˝ D f0g, we get T˝F .x/ D F . x/, where F is the Minkowski gauge defined later in this section. e minimal time function (3.9) also covers another interesting class of functions called the directional minimal time function, which corresponds to the case where F is a nonzero vector. In this section we study general properties of the minimal time function (3.9) used in what follows. ese properties are certainly of independent interest. Observe that T˝F .x/ D 0 if x 2 ˝ , and that T˝F .x/ is finite if there exists t  0 with .x C tF / \ ˝ ¤ ;;

which holds, in particular, when 0 2 int F . If no such t exists, then T˝F .x/ D 1. us the minimal time function (3.9) is an extended-real-valued function in general.

112

3. REMARKABLE CONSEQUENCES OF CONVEXITY

e sets F and ˝ in (3.9) are not necessarily bounded. e next theorem reveals behavior of in the case where ˝ is bounded, while F is not necessarily bounded. It is an exercise to formulate and prove related results for the case where F is bounded, while ˝ is not necessarily bounded.

T˝F .x/

Let F be a nonempty, closed, convex subset of Rn and let ˝  Rn be a nonempty, closed set. Assume that ˝ is bounded. en

eorem 3.27

T˝F .x/ D 0 implies x 2 ˝

F1 :

e converse holds if we assume additionally that 0 2 F . Proof. Suppose that T˝F .x/ D 0 and find tk ! 0, tk  0, such that .x C tk F / \ ˝ ¤ ;;

k 2N:

is gives us qk 2 ˝ and fk 2 F with x C tk fk D qk for each k . By the boundedness and closedness of ˝ , we can select a subsequence of fqk g that converges to some q 2 ˝ , and thus tk fk ! q x . It follows from Proposition 3.24 that q x 2 F1 , which justifies x 2 ˝ F1 . To prove the opposite implication, pick any x 2 ˝ F1 and find q 2 ˝ and d 2 F1 such that x D q d . Since 0 2 F and d 2 F1 , we get from Corollary 3.23 the inclusions k.q x/ D kd 2 F for all k 2 N . is shows that  1 1  q x 2 F and so x C F \ ˝ ¤ ;; k 2 N : k k 1 for all k 2 N , which holds only when T˝F .x/ D 0 and k thus completes the proof of the theorem. 

e latter ensures that 0  T˝F .x/ 

e following example illustrates the result of eorem 3.27. Example 3.28 Let F D R Œ 1; 1  R2 and let ˝ be the disk centered at .1; 0/ with radius r D 1. Using the representation of F1 in Corollary 3.23, we get F1 D R f0g and ˝ F1 D R Œ 1; 1. us T˝F .x/ D 0 if and only if x 2 R Œ 1; 1.

e next result characterizes the convexity of the minimal time function (3.9).

Let F be a nonempty, closed, convex subset of Rn . If ˝  Rn is nonempty and convex, then the minimal time function (3.9) is convex. If furthermore F is bounded and ˝ is closed, then the converse holds as well. Proposition 3.29

3.7. MINIMAL TIME FUNCTIONS AND MINKOWSKI GAUGE

Proof. Take any x1 ; x2 2

T˝F

113

dom T˝F

and let  2 .0; 1/. We now show that  x1 C .1 /x2  T˝F .x1 / C .1 /T˝F .x2 /

(3.10)

provided that both sets F and ˝ are convex. Denote i WD T˝F .xi / for i D 1; 2 and, given any  > 0, select numbers ti so that

i  ti < i C  and .xi C ti F / \ ˝ ¤ ;;

It follows from the convexity of F and ˝ that  x1 C .1 /x2 C t1 C .1 which implies in turn the estimates  T˝F x1 C .1 /x2  t1 C .1

i D 1; 2:

  /t2 F \ ˝ ¤ ;;

/t2  T˝F .x1 / C .1

/T˝F .x2 / C :

Since  is arbitrary, we arrive at (3.10) and thus verifies the convexity of T˝F . Conversely, suppose that T˝F is convex provided that F is bounded and that ˝ is closed. en the level set ˇ ˚ x 2 Rn ˇ T˝F .x/  0 is definitely convex. We can easily show set reduces to the target set ˝ , which justifies the convexity of the latter.  Definition 3.30 Let F be a nonempty, closed, convex subset of Rn . e M  associated with the set F is defined by ˇ ˚ (3.11) F .x/ WD inf t  0 ˇ x 2 tF ; x 2 Rn :

e following theorem summarizes the main properties of Minkowski gauge functions. Let F be a closed, convex set that contains the origin. e Minkowski gauge F is an extended-real-valued function, which is positively homogeneous and subadditive. Furthermore, F .x/ D 0 if and only if x 2 F1 .

eorem 3.31

Proof. We first prove that F is subadditive meaning that F .x1 C x2 /  F .x1 / C F .x2 / for all x1 ; x2 2 Rn :

(3.12)

is obviously holds if the right-hand side of (3.12) is equal to infinity, i.e., we have that x1 … dom F or x2 … dom F . Suppose now that x1 ; x2 2 dom F and fix  > 0. en there are numbers t1 ; t2  0 such that F .x1 /  t1 < F .x1 / C ; F .x2 /  t2 < F .x2 / C  and x1 2 t1 F; x2 2 t2 F:

114

3. REMARKABLE CONSEQUENCES OF CONVEXITY

It follows from the convexity of F that x1 C x2 2 t1 F C t2 F  .t1 C t2 /F , and so F .x1 C x2 /  t1 C t2 < F .x1 / C F .x2 / C 2;

which justifies (3.12) since  > 0 was chosen arbitrarily. To check the positive homogeneous property of F , we take any ˛ > 0 and conclude that ˇ n ˇ ˚ t o ˇ F .˛x/ D inf t  0 ˇ ˛x 2 tF D inf t  0 ˇ x 2 F ˛ ˇ nt t o ˇ D ˛ inf  0 ˇ x 2 F D ˛F .x/: ˛ ˛ It remains to verify the last statement of the theorem. Observe that F .x/ D T˝ F .x/ is a special case of the minimal time function with ˝ D f0g. Since 0 2 F , eorem 3.27 tells us that F .x/ D 0 if and only if x 2 ˝ . F /1 D F1 , which completes the proof.  We say for simplicity that 'W Rn ! R is ` Lipschitz continuous on Rn if it is Lipschitz continuous on Rn with Lipschitz constant `  0. e next result calculates a constant ` of Lipschitz continuity of the Minkowski gauge on Rn .

Let F be a closed, convex set that contains the origin as an interior point. en the Minkowski gauge F is `-Lipschitz continuous on Rn with the constant Proposition 3.32

` WD inf

o n1 ˇ ˇ ˇ IB.0I r/  F; r > 0 : r

(3.13)

In particular, we have F .x/  `kxk for all x 2 Rn . Proof. e condition 0 2 int F ensures that ` < 1 for the constant ` from (3.13). Take any r > 0 satisfying IB.0I r/  F and get from the definitions that ˇ ˇ ˚ ˚ ˚ kxk F .x/ D inf t  0 ˇ x 2 tF  inf t  0 ˇ x 2 tIB.0I r/ D inf t  0 j kxk  rt D ; r

which implies that F .x/  `kxk. en the subadditivity of F gives us that F .x/

F .y/  F .x

y/  `kx

yk for all x; y 2 Rn ;

and thus F is ` Lipschitz continuous on Rn with constant (3.13).



Next we relate the minimal time function to the Minkowski gauge. eorem 3.33

Let F be a nonempty, closed, convex set. en for any ˝ ¤ ; we have ˇ ˚ T˝F .x/ D inf F .! x/ ˇ ! 2 ˝ ; x 2 Rn :

(3.14)

3.7. MINIMAL TIME FUNCTIONS AND MINKOWSKI GAUGE

115

T˝F .x/

Proof. Consider first the case where D 1. In this case we have ˇ ˚ t  0 ˇ .x C tF / \ ˝ ¤ ; D ;;

which implies that ft  0 j ! x 2 tF g D ; for every ! 2 ˝ . us F .! x/ D 1 as ! 2 ˝ , and hence the right-hand side of (3.14) is also infinity. Considering now the case where T˝F .x/ < 1, fix any t  0 such that .x C tF / \ ˝ ¤ ;. is gives us f 2 F and ! 2 ˝ with x C tf D ! . us ! x 2 tF , and so F .! x/  t , which yields inf!2˝ F .! x/  t . It follows from definition (3.9) that inf F .!

!2˝

x/  T˝F .x/ < 1:

To verify the opposite inequality in (3.14), denote WD inf!2˝ F .! x/ and for any  > 0 find ! 2 ˝ with F .! x/ < C  . By definition of the Minkowski gauge (3.11), we get t  0 such that t < C  and ! x 2 tF . is gives us ! 2 x C tF and therefore

T˝F .x/  t < C : Since  was chosen arbitrarily, we arrive at

T˝F .x/  D inf F .! !2˝

x/;

which implies the remaining inequality in (3.14) and completes the proof.



As a consequence of the obtained result and Proposition 3.32, we establish now the ` Lipschitz continuity of minimal time function with the constant ` calculated in (3.13).

Let F be a closed, convex set that contains the origin as an interior point. en for any nonempty set ˝ the function T˝F is ` Lipschitz with constant (3.13). Corollary 3.34

Proof. It follows from the subadditivity of F and the result of Proposition 3.32 that F .!

x/  F .!

y/ C F .y

x/  F .!

y/ C `ky

xk for all ! 2 ˝

whenever x; y 2 Rn . en eorem 3.33 tells us that

T˝F .x/  T˝F .y/ C `kx

yk;

which implies the ` Lipschitz property of T˝F due to the arbitrary choice of x; y . Define next the enlargement of the target set ˝ via minimal time function by ˇ ˚ ˝r WD x 2 Rn ˇ T˝F .x/  r ; r > 0:



(3.15)

116

3. REMARKABLE CONSEQUENCES OF CONVEXITY

Let F be a nonempty, closed, convex subset of Rn and let ˝  Rn be a nonempty set. en for any nonempty set ˝ and any x … ˝r with T˝F .x/ < 1 we have

Proposition 3.35

T˝F .x/ D T˝Fr .x/ C r:

Proof. It follows from ˝  ˝r that T˝Fr .x/  T˝F .x/ < 1. Take any t  0 with .x C tF / \ ˝r ¤ ;

and find f1 2 F and u 2 ˝r such that x C tf1 D u. Since u 2 ˝r , we have T˝F .u/  r . For any  > 0, there is s  0 with s < r C  and .u C sF / \ ˝ ¤ ;. is implies the existence of ! 2 ˝ and f2 2 F such that u C sf2 D ! . us ! D u C sf2 D .x C tf1 / C sf2 2 x C .t C s/F

since F is convex. is gives us

T˝F .x/  t C s  t C r C  and so T˝F .x/  t C r by the arbitrary choice of  > 0. is allows us to conclude that

T˝F .x/  T˝Fr C r:

To verify the opposite inequality, denote WD T˝F .x/ and observe that r < by x … ˝r . For any  > 0, find t 2 Œ0; 1/, f 2 F , and ! 2 ˝ with  t < C  and x C tf D ! . Hence ! D x C tf D x C .t

r/f C rf 2 x C .t

r/f C rF and so T˝F .x C .t

which shows that x C .t r/f 2 ˝r . We also see that x C .t T˝Fr .x/  t r  r C  , which verifies that

r/f 2 x C .t

r/f /  r; r/F . us

r C T˝Fr  D T˝F .x/

since  > 0 was chosen arbitrarily. is completes the proof.



Proposition 3.36 Let F be a nonempty, closed, convex subset of Rn . en for any nonempty set ˝ , any x 2 dom T˝F , any t  0, and any f 2 F we have

T˝F .x

tf /  T˝F .x/ C t:

Proof. Fix  > 0 and select s  0 so that

T˝F .x/  s < T˝F .x/ C  and .x C sF / \ ˝ ¤ ;:

en .x

tf C tF C sF / \ ˝ ¤ ;, and hence .x

T˝F .x

tf C .t C s/F / \ ˝ ¤ ;. It shows that

tf /  t C s  t C T˝F .x/ C ;

which implies the conclusion of the proposition by letting  ! 0C .



3.8. SUBGRADIENTS OF MINIMAL TIME FUNCTIONS

3.8

117

SUBGRADIENTS OF MINIMAL TIME FUNCTIONS

Observing that the minimal time function (3.9) is intrinsically nonsmooth, we study here its generalized differentiation properties in the case where the target set ˝ is convex. In this case (since the constant dynamics F is always assumed convex) function (3.9) is convex as well by Proposition 3.29. e results below present precise formulas for calculating the subdifferential @T˝F .x/ N depending on the position of xN with respect to the set ˝ F1 . eorem 3.37 Let both sets 0 2 F  Rn and ˝  Rn be nonempty, closed, convex. For any xN 2 ˝ F1 , the subdifferential of T˝F at xN is calculated by

@T˝F .x/ N D N.xI N ˝

n

F1 / \ C  ;

(3.16)

where C  WD fv 2 R j F . v/  1g defined via (2.41). In particular, we have @T˝F .x/ N D N.xI N ˝/ \ C  for xN 2 ˝:

(3.17)

Proof. Fix any v 2 @T˝F .x/ N . Using (2.13) and the fact that T˝F .x/ D 0 for all x 2 ˝ F1 , we get v 2 N.xI N ˝ F1 /. Write xN 2 ˝ F1 as xN D !N d with !N 2 ˝ and d 2 F1 . Fix any f 2 F and define x t WD .xN C d / tf D !N tf as t > 0. en .x t C tF / \ ˝ ¤ ;, and so D d E hv; x t xi N  T˝F .x t /  t; or equivalently v; f  1 for all t > 0: t By letting t ! 1, it implies that hv; f i  1, and hence we get v 2 C  . is verifies the inclusion @T˝F .x/ N  N.xI N ˝ F1 / \ C  . N ˝ F1 / \ C  and any u 2 To prove the opposite inclusion in (3.16), fix any v 2 N.xI F dom T˝ . For any  > 0, there exist t 2 Œ0; 1/, ! 2 ˝ , and f 2 F such that

T˝F .u/  t < T˝F .u/ C  and u C tf D !:

Since ˝  ˝ hv; u

F1 , we have the relationships xi N D hv; !

xi N C thv; f i  t < T˝F .u/ C  D T˝F .u/

T˝F .x/ N C ;

which show that v 2 @T˝F .x/ N . In this way we get N.xI N ˝ F1 / \ C   @T˝F .x/ N and thus justify the subdifferential formula (3.16) in the general case where xN 2 ˝ F1 . It remains to verify the simplified representation (3.17) in the case where xN 2 ˝ . Since ˝  ˝ F1 , we obviously have the inclusion N.xI N ˝

F1 / \ C   N.xI N ˝/ \ C  :

To show the converse inclusion, pick v 2 N.xI N ˝/ \ C  and fix x 2 ˝ F1 . Taking ! 2 ˝ and d 2 F1 with x D ! d gives us t d 2 F since 0 2 F , so hv; t d i  1 for all t > 0. us hv; d i  0, which readily implies that hv; x

xi N D hv; !

In this way we arrive at v 2 N.xI N ˝

d

xi N D hv; !

xi N C hv; d i  0:

F1 / \ C  and thus complete the proof.



118

3. REMARKABLE CONSEQUENCES OF CONVEXITY

e next theorem provides the subdifferential calculation for minimal time functions in the cases complemented to those in eorem 3.37, i.e., when xN … ˝ F1 and xN … ˝ . Let both sets 0 2 F  Rn and ˝  Rn be nonempty, closed, convex and let xN 2 for the minimal time function (3.9). Suppose that ˝ is bounded. en for xN … ˝ F1

eorem 3.38

dom T˝F we have

@T˝F .x/ N D N.xI N ˝r / \ S  ;

(3.18)

where the enlargement ˝r is defined in (3.15), and where ˇ ˚ S  WD v 2 Rn ˇ F . v/ D 1 with r D T˝F .x/ N > 0: Proof. Pick v 2 @T˝F .x/ N . Observe from the definition that hv; x

xi N  T˝F .x/

T˝F .x/ N for all x 2 Rn :

(3.19)

en T˝F .x/  r D T˝F .x/ N , and hence hv; x xi N  0 for any x 2 ˝r . is yields v 2 N.xI N ˝r /. Using (3.19) for x WD xN f with f 2 F and Proposition 3.36 ensures that F . v/  1. Fix  2 .0; r/ and select t 2 R, f 2 F , and ! 2 ˝ so that r  t < r C  2 and ! D xN C tf:

We can write ! D xN C f C .t /f and then see that T˝F .xN C f /  t to x WD xN C f gives us the estimates hv; f i  T˝F .xN C f /

T˝F .x/ N t



r  2

 . Applying (3.19)

:

which imply in turn that 1

  h v; f i  F . v/:

Letting now  # 0 tells us that F . v/  1, i.e., F . v/ D 1 in this case. us v 2 S  , which proves the inclusion @T˝F .x/ N  N.xI N ˝r / \ S  . To verify the opposite inclusion in (3.18), take any v 2 N.xI N ˝r / with F . v/ D 1 and show that (3.19) holds. It follows from eorem 3.37 that v 2 @T˝Fr .x/ N , and so hv; x

xi N  T˝Fr .x/ for all x 2 Rn :

Fix x 2 Rn and deduce from Proposition 3.35 that T˝F .x/ r D T˝Fr .x/ in the case t WD T˝F .x/ > r , which ensures (3.19). Suppose now that t  r and for any  > 0 choose f 2 F such that hv; f i > 1  . en Proposition 3.36 tells us that T˝F .x .r t /f /  r , and so x .r t/f 2 ˝r . Since v 2 N.xI N ˝r /, one has hv; x .r t/f xi N  0, which yields  hv; x xi N  hv; f i.r t/  .1 /.t r/ D .1 / T˝F .x/ T˝F .x/ N : N . Since  > 0 was arbitrarily chosen, we get (3.19), and hence verify that v 2 @T˝F .x/



3.8. SUBGRADIENTS OF MINIMAL TIME FUNCTIONS

119

e next result addresses the same setting as in eorem 3.38 while presenting another formula for calculating the subdifferential of the minimal time function that involves the subdifferential of the Minkowski gauge and generalized projections to the target set. eorem 3.39 Let both sets 0 2 F  Rn and ˝  Rn be nonempty, closed, convex and let xN 2 dom T˝F . Suppose that ˝ is bounded and that xN … ˝ F1 . en we have   @T˝F .x/ N D @F .!N x/ N \ N.!I N ˝/ (3.20)

for any !N 2 ˘F .xI N ˝/ via the generalized projector ˇ ˚ ˘F .xI N ˝/ WD ! 2 ˝ ˇ T˝F .x/ N D F .!

x/ N :

(3.21)

Proof. Fixing any v 2 @T˝F .x/ N and !N 2 ˘F .xI N ˝/, we get T˝F .x/ N D F .!N hv; x

xi N  T˝F .x/

x/ N and

T˝F .x/ N for all x 2 Rn :

(3.22)

It is easy to check the following relationships that hold for any ! 2 ˝ : hv; !

!i N D hv; .!  F !

!N C x/ N xi N  T˝F .! .! !N C x/ N F .!N

!N C x/ N x/ N D 0:

T˝F .x/ N

xN and for any t 2 .0; 1/ and u 2 Rn get  e u/i  T˝F xN t .u e u/ T˝F .x/ N  F !N .xN t .u e u// F .!N x/ N D F .e u C t.u e u// F .e u/ D F tu C .1 t/e u F .e u/  tF .u/ C .1 t /F .e u/ F .e u/ D t F .u/ F .e u/

is shows that v 2 N.!I N ˝/. Denote now e u WD !N hv; t.u

by applying (3.22) with x WD xN h v; u

t .u

e u/. It follows that

e ui  F .u/

F .e u/ for all u 2 Rn :

us v 2 @F .e u/, and so v 2 @F .!N x/ N , which proves the inclusion “” in (3.20). To verify the opposite inclusion, write ˇ ˇ ˚ ˚ F .x/ D inf t  0 ˇ x 2 tF D inf t  0 ˇ . x C tF / \ O ¤ ; D TOF . x/ with O WD f0g. Since xN … ˝ F1 , for any !N 2 ˝ we have xN !N … O F1 D F1 . Furthermore, it follows from v 2 @F .!N x/ N by applying eorem 3.38 that v 2 S  . Hence it remains to justify the inclusion   @F .!N x/ N \ N.!I N ˝/  N.xI N ˝r / (3.23)

120

3. REMARKABLE CONSEQUENCES OF CONVEXITY

and then to apply eorem 3.38. To proceed with the proof of (3.23), pick any vector x 2 ˝r and a positive number  arbitrarily small. en there are t < r C  , f 2 F , and ! 2 ˝ such that ! D x C tf , which yields hv; x

Observe that hv; ! hv; !N

xi N D hv; ! tf xi N D th v; f i C hv; ! !i N C hv; !N xi N  t C hv; ! !i N C hv; !N xi N  T˝F .x/ N C  C hv; ! !i N C hv; !N xi: N

!i N  0 by v 2 N.!I N ˝/. Moreover, xi N D h v; 0

.!N

x/i N  F .0/

F .!N

x/ N D

T˝F .x/ N

since v 2 @F .!N x/ N . It follows that hv; x xi N   for all x 2 ˝r . Hence v 2 N.xI N ˝r / because  > 0 was chosen arbitrarily. is completes the proof of the theorem. 

3.9

NASH EQUILIBRIUM

is section gives a brief introduction to noncooperative game theory and presents a simple proof of the existence of Nash equilibrium as a consequence of convexity and Brouwer’s fixed point theorem. For simplicity, we only consider two-person games while more general situations can be treated similarly.

Let ˝1 and ˝2 be nonempty subsets of Rn and Rp , respectively. A   in the case of two players I and II consists of two strategy sets ˝i and two real-valued functions ui W ˝1  ˝2 ! R for i D 1; 2 called the  . en we refer to this game as f˝i ; ui g for i D 1; 2.

Definition 3.40

e key notion of Nash equilibrium was introduced by John Forbes Nash, Jr., in 1950 who proved the existence of such an equilibrium and was awarded the Nobel Memorial Prize in Economics in 1994. Definition 3.41 Given a noncooperative two-person game f˝i ; ui g, i D 1; 2, a N  is an element .xN 1 ; xN 2 / 2 ˝1  ˝2 satisfying the conditions

u1 .x1 ; xN 2 /  u1 .xN 1 ; xN 2 / for all x1 2 ˝1 ; u2 .xN 1 ; x2 /  u2 .xN 1 ; xN 2 / for all x2 2 ˝2 :

e conditions above mean that .xN 1 ; xN 2 / is a pair of best strategies that both players agree with in the sense that xN 1 is a best response of Player I when Player II chooses the strategy xN 2 , and xN 2 is a best response of Player II when Player I chooses the strategy xN 1 . Let us now consider two examples to illustrate this concept.

3.9. NASH EQUILIBRIUM

121

Example 3.42 In a two-person game suppose that each player can only choose either strategy A or B . If both players choose strategy A, they both get 4 points. If Player I chooses strategy A and Player II chooses strategy B , then Player I gets 1 point and Player II gets 3 points. If Player I chooses strategy B and Player II chooses strategy A, then Player I gets 3 points and Player II gets 1 point. If both choose strategy B , each player gets 2 points. e payoff function of each player

is represented in the payoff matrix below. Player II Player I

A

B

A

4; . 4

B

3; . 1

1; . 3

.

2; . 2

In this example, Nash equilibrium occurs when both players choose strategy A, and it also occurs when both players choose strategy B . Let us consider, for instance, the case where both players choose strategy B . In this case, given that Player II chooses strategy B , Player I also wants to keep strategy B because a change of strategy would lead to a reduction of his/her payoff from 2 to 1. Similarly, given that Player I chooses strategy B , Player II wants to keep strategy B because a change of strategy would also lead to a reduction of his/her payoff from 2 to 1. Let us consider another simple two-person game called matching pennies. Suppose that Player I and Player II each has a penny. Each player must secretly turn the penny to heads or tails and then reveal his/her choices simultaneously. Player I wins the game and gets Player II’s penny if both coins show the same face (heads-heads or tails-tails). In the other case where the coins show different faces (heads-tails or tails-heads), Player II wins the game and gets Player I’s penny. e payoff function of each player is represented in the payoff matrix below.

Example 3.43

Player II Player I

Heads 1; . 1

Heads

Tails .

1;. 1

Tails

1;. 1 1; . 1

In this game it is not hard to see that no Nash equilibrium exists. Denote by A the matrix that represents the payoff function of Player I, and denote by B the matrix that represents the payoff function of Player II: AD

1 1

1 1

and B D

1 1

1 1

122

3. REMARKABLE CONSEQUENCES OF CONVEXITY

Now suppose that Player I is playing with a mind reader who knows Player I’s choice of faces. If Player I decides to turn heads, then Player II knows about it and chooses to turn tails and wins the game. In the case where Player I decides to turn tails, Player II chooses to turn heads and again wins the game. us to have a fair game, he/she decides to randomize his/her strategy by, e.g., tossing the coin instead of putting the coin down. We describe the new game as follows. Player II Player I

Heads, q1 Heads, p1

1; . 1

Tails, q2 1;. 1

.

1;. 1

Tails, p2

1; . 1

In this new game, Player I uses a coin randomly with probability of coming up heads p1 and probability of coming up tails p2 , where p1 C p2 D 1. Similarly, Player II uses another coin randomly with probability of coming up heads q1 and probability of coming up tails q2 , where q1 C q2 D 1. e new strategies are called mixed strategies while the original ones are called pure strategies. Suppose that Player II uses mixed strategy fq1 ; q2 g. en Player I’s expected payoff for playing the pure strategy heads is uH .q1 ; q2 / D q1

q2 D 2q1

1:

Similarly, Player I’s expected payoff for playing the pure strategy tails is uT .q1 ; q2 / D

q1 C q2 D

2q1 C 1:

us Player I’s expected payoff for playing mixed strategy fp1 ; p2 g is u1 .p; q/ D p1 uH .q1 ; q2 / C p2 uT .q1 ; q2 / D p1 .q1

q2 / C p2 . q1 C q2 / D p T Aq;

where p D Œp1 ; p2 T and q D Œq1 ; q2 T . By the same arguments, if Player I chooses mixed strategy fp1 ; p2 g, then Player II’s expected payoff for playing mixed strategy fq1 ; q2 g is u2 .p; q/ D p T Bq D

u1 .p; q/:

For this new game, an element .p; N q/ N 2    is a Nash equilibrium if u1 .p; q/ N  u1 .p; N q/ N for all p 2 ; u2 .p; N q/  u2 .p; N q/ N for all q 2 ; ˇ ˚ where  WD .p1 ; p2 / ˇ p1  0; p2  0; p1 C p2 D 1 is a nonempty, compact, convex set.

Nash [23, 24] proved the existence of his equilibrium in the class of mixed strategies in a more general setting. His proof was based on Brouwer’s fixed point theorem with rather involved

3.9. NASH EQUILIBRIUM

123

arguments. Now we present, following Rockafellar [28], a much simpler proof of the Nash equilibrium theorem that also uses Brouwer’s fixed point theorem while applying in addition just elementary tools of convex analysis and optimization. Consider a two-person game f˝i ; ui g, where ˝1  Rn and ˝2  Rp are nonempty, compact, convex sets. Let the payoff functions ui W ˝1  ˝2 ! R be given by eorem 3.44

u1 .p; q/ WD p T Aq;

u2 .p; q/ WD p T Bq;

where A and B are n  p matrices. en this game admits a Nash equilibrium. Proof. It follows from the definition that an element .p; N q/ N 2 ˝1  ˝2 is a Nash equilibrium of the game under consideration if and only if u1 .p; q/ N  u1 .p; N q/ N for all p 2 ˝1 ; u2 .p; N q/  u2 .p; N q/ N for all q 2 ˝2 :

It is a simple exercise to prove (see also an elementary convex optimization result of Corollary 4.15 in Chapter 4) that this holds if and only if we have the normal cone inclusions rp u1 .p; N q/ N 2 N.pI N ˝1 /;

rq u2 .p; N q/ N 2 N.qI N ˝2 /:

(3.24)

By using the structures of ui and defining ˝ WD ˝1  ˝2 , conditions (3.24) can be expressed in one of the following two equivalent ways: AqN 2 N.pI N ˝1 / and B T pN 2 N.qI N ˝2 /I .Aq; N B T p/ N 2 N.pI N ˝1 /  N.qI N ˝2 / D N..p; N q/I N ˝1  ˝2 / D N..p; N q/I N ˝/:

By Proposition 1.78 on the Euclidean projection, this is equivalent also to .p; N q/ N D ˘..p; N q/ N C .Aq; N B T p/I N ˝/:

(3.25)

Defining now the mapping F W ˝ ! ˝ by F .p; q/ WD ˘ .p; q/ C .Aq; B T p/I ˝/ with ˝ D ˝1  ˝2

and employing another elementary projection property from Proposition 1.79 allow us to conclude that the mapping F is continuous and the set ˝ is nonempty, compact, and convex. By the N q/ N 2 ˝, classical Brouwer fixed point theorem (see, e.g., [33]), the mapping F has a fixed point .p;  which satisfies (3.25), and thus it is a Nash equilibrium of the game.

124

3. REMARKABLE CONSEQUENCES OF CONVEXITY

3.10 EXERCISES FOR CHAPTER 3 Exercise 3.1 Let f W Rn ! R be a function finite around xN . Show that f is Gâteaux dif-

ferentiable at xN with fG0 .x/ N D v if and only if the directional derivative f 0 .xI N d / exists and n 0 f .xI N d / D hv; d i for all d 2 R .

Exercise 3.2 Let ˝ be a nonempty, closed, convex subset of Rn . (i) Prove that the function f .x/ WD Œd.xI ˝/2 is differentiable on Rn and rf .x/ N D 2ŒxN

˘.xI N ˝/

for every xN 2 Rn . (ii) Prove that the function g.x/ WD d.xI ˝/ is differentiable on Rn n˝ .

Exercise 3.3 Let F be a nonempty, compact, convex subset of Rn and let A W Rn ! Rp be a linear mapping. Define

f .x/ WD sup hAx; ui u2F

 1 kuk2 : 2

Prove that f is convex and differentiable.

Exercise 3.4 Give an example of a function f W Rn ! R, which is Gâteuax differentiable but not Fréchet differentiable. Can such a function be convex?

Exercise 3.5 Let f W Rn ! R be a convex function and let xN 2 int.domf /. Show that if at xN exists for all i D 1; : : : ; n, then f is (Fréchet) differentiable at xN .

@f @xi

Exercise 3.6 Let f W Rn ! R be differentiable on Rn . Show that f is strictly convex if and only if it satisfies the condition hrf .x/; N x

xi N < f .x/

f .x/ N for all x 2 Rn and xN 2 Rn ; x ¤ x: N

Exercise 3.7 Let f W Rn ! R be twice continuously differentiable (C 2 ) on Rn . Prove that if

the Hessian r 2 f .x/ N is positive definite for all xN 2 Rn , then f is strictly convex. Give an example showing that the converse is not true in general.

Exercise 3.8 A convex function f W Rn ! R is called strongly convex with modulus c > 0 if for all x1 ; x2 2 Rn and  2 .0; 1/ we have f x1 C .1

 /x2  f .x1 / C .1

/f .x2 /

1 c.1 2

/kx1

x2 k2 :

3.10. EXERCISES FOR CHAPTER 3 2

125

n

Prove that the following are equivalent for any C convex function f W R ! R: (i) f strongly convex. c (ii) f k  k2 is convex. 2 (iii) hr 2 f .x/d; d i  ckd k2 for all x; d 2 Rn . ! Rn is said to be monotone if Exercise 3.9 A set-valued mapping F W Rn ! hv2

v1 ; x2

x1 i  0 whenever vi 2 F .xi /; i D 1; 2:

F is called strictly monotone if hv2

v1 ; x2

x1 i > 0 whenever vi 2 F .xi /; x1 ¤ x2 ; i D 1; 2:

It is called strongly monotone with modulus ` > 0 if hv2

v1 ; x2

x1 i  `kx1

x2 k2 whenever vi 2 F .xi /; i D 1; 2:

(i) Prove that for any convex function f W Rn ! R the mapping F .x/ WD @f .x/ is monotone. (ii) Furthermore, for any convex function f W Rn ! R the subdifferential mapping @f .x/ is strictly monotone if and only if f is strictly convex, and it is strongly monotone if and only if f is strongly convex.

Exercise 3.10 Prove Corollary 3.7. Exercise 3.11 Let A be an p  n matrix and let b 2 Rp . Use the Farkas lemma to show that the

system Ax D b , x  0 has a feasible solution x 2 Rn if and only if the system AT y  0, b T y > 0 has no feasible solution y 2 Rp .

Exercise 3.12 Deduce Carathéodory’s theorem from Radon’s theorem and its proof. Exercise 3.13 Use Carathéodory’s theorem to prove Helly’s theorem. Exercise 3.14 Let C be a subspace of Rn . Show that the polar of C is represented by ˇ ˚ C ı D C ? WD v 2 Rn ˇ hv; ci D 0 for all c 2 C :

Exercise 3.15 Let C be a nonempty, convex cone of Rn . Show that ˇ ˚ C ı D v 2 Rn ˇ hv; ci  1 for all c 2 C :

Exercise 3.16 Let ˝1 and ˝2 be convex sets with int ˝1 \ ˝2 ¤ ;. Show that T .xI N ˝1 \ ˝2 / D T .xI N ˝1 / \ T .xI N ˝2 / for any xN 2 ˝1 \ ˝2 :

126

3. REMARKABLE CONSEQUENCES OF CONVEXITY

Exercise 3.17 Let A be p  n matrix, let

ˇ ˚ ˝ WD x 2 Rn ˇ Ax D 0; x  0 ;

and let xN D .xN 1 ; : : : ; xN n / 2 ˝ . Verify the tangent and normal cone formulas: ˇ T .xI N ˝/ D fu D .u1 ; : : : ; un / 2 Rn ˇ Au D 0; ui  0 for any i with xN i D 0 ; ˇ ˚ N.xI N ˝/ D v 2 Rn ˇ v D A y

z; y 2 Rp ; z 2 Rn ; zi  0; hz; xi N D0 :

Exercise 3.18 Give an example of a closed, convex set ˝  R2 with a point xN 2 ˝ such that the cone RC .˝

x/ N is not closed.

Exercise 3.19 Let f W Rn ! R be a convex function such that there is `  0 satisfying @f .x/  `IB . Show that f is ` Lipschitz continuous on Rn .

Exercise 3.20 Let f W R ! R be convex. Prove that f is nondecreasing if and only if @f .x/  Œ0; 1/ for all x 2 R.

Exercise 3.21 Find the horizon cones of the following sets: (i) F WD f.x; y/ 2 R2 j y  x 2 g. (ii) F WD f.x; y/ 2 R2 j y  jxjg. (iii) F WD ˚f.x; y/ 2 R2 j x > 0; y  p 1=xg. 2 (iv) F WD .x; y/ 2 R j x > 0; y  x 2 C 1g.

Exercise 3.22 Let vi 2 Rn and bi > 0 for i D 1; : : : ; m. Consider the set ˇ ˚ F WD x 2 Rn ˇ hvi ; xi  bi for all i D 1; : : : ; mg:

Prove the following representation of the horizon cone and Minkowski gauge for F : ˇ ˚ F1 D x 2 Rn ˇ hvi ; xi  0 for all i D 1; : : : ; m ; n F .u/ D max 0;

hvi ; ui o : bi i2f1;:::;mg

max

Exercise 3.23 Let A; B be nonempty, closed, convex subsets of Rn . (i) Show that .A  B/1 D A1  B1 . (ii) Find conditions ensuring that .A C B/1 D A1 C B1 .

(3.26)

3.10. EXERCISES FOR CHAPTER 3

127

Exercise 3.24 Let F ¤ ; be a closed, bounded, convex set and let ˝ ¤ ; be a closed set in

Rn . Prove that T˝F .x/ D 0 if and only if x 2 ˝ . is means that the condition 0 2 F in eorem 3.27(i) is not required in this case.

Exercise 3.25 Let F ¤ ; be a closed, bounded, convex set and let ˝ ¤ ; be a closed, convex

set. Consider the sets C  and S  defined in eorem 3.37 and eorem 3.38, respectively. Prove the following: (i) If xN 2 ˝ , then (ii) If xN … ˝ , then (iii) If xN … ˝ , then

@T˝F .x/ N D N.xI N ˝/ \ C  : @T˝F .x/ N D N.xI N ˝r / \ S  ; r WD T˝F .x/ N > 0: @T˝F .x/ N D



@F .!N

 x/ N \ N.!I N ˝/

for any !N 2 ˘F .xI N ˝/

Exercise 3.26 Give a direct proof of characterizations (3.24) of Nash equilibrium. Exercise 3.27 In Example 3.43 with mixed strategies, find .p; N q/ N that yields a Nash equilibrium in this game.

129

CHAPTER

4

Applications to Optimization and Location Problems In this chapter we give some applications of the results of convex analysis from Chapters 1–3 to selected problems of convex optimization and facility location. Applications to optimization problems are mainly classical, being presented however in a simplified and unified way. On the other hand, applications to location problems are mostly based on recent journal publications and have not been systematically discussed from the viewpoint of convex analysis in monographs and textbooks.

4.1

LOWER SEMICONTINUITY AND EXISTENCE OF MINIMIZERS

We first discuss the notion of lower semicontinuity for extended-real-valued functions, which plays a crucial role in justifying the existence of absolute minimizers and related topics. Note that most of the results presented below do not require the convexity of f . Definition 4.1 An extended-real-valued function f W Rn ! R is   .l.s.c./  xN 2 Rn if for every ˛ 2 R with f .x/ N > ˛ there is ı > 0 such that

f .x/ > ˛ for all x 2 IB.xI N ı/:

(4.1)

We simply say that f is   if it is l.s.c. at every point of Rn . e following examples illustrate the validity or violation of lower semicontinuity. Example 4.2

(i) Define the function f W R ! R by ( x 2 if x ¤ 0; f .x/ WD 1 if x D 0:

It is easy to see that it is l.s.c. at xN D 0 and in fact on the whole real line. On the other hand, the replacement of f .0/ D 1 by f .0/ D 1 destroys the lower semicontinuity at xN D 0.

130

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

Figure 4.1: Which function is l.s.c?

(ii) Consider the indicator function of the set Œ 1; 1  R given by ( 0 if jxj  1; f .x/ WD 1 otherwise:

en f is a lower semicontinuous function on R. However, the small modification ( 0 if jxj < 1; g.x/ WD 1 otherwise violates the lower semicontinuity on R. In fact, g is not l.s.c. at x D ˙1 while it is lower semicontinuous at any other point of the real line. e next four propositions present useful characterizations of lower semicontinuity. Proposition 4.3 Let f W Rn ! R and let xN 2 dom f . en f is l.s.c. at xN if and only if for any  > 0 there is ı > 0 such that

 < f .x/ whenever x 2 IB.xI N ı/:

f .x/ N

Proof. Suppose that f is l.s.c. at xN . Since  WD f .x/ N f .x/ N

(4.2)

 < f .x/ N , there is ı > 0 with

 D  < f .x/ for all x 2 IB.xI N ı/:

Conversely, suppose that for any  > 0 there is ı > 0 such that (4.2) holds. Fix any  < f .x/ N and choose  > 0 for which  < f .x/ N  . en  < f .x/ N

 < f .x/ whenever x 2 IB.xI N ı/

for some ı > 0, which completes the proof.



4.1. LOWER SEMICONTINUITY AND EXISTENCE OF MINIMIZERS

131

n

Proposition 4.4 e function f W R ! R is l.s.c. if and only if for any number  2 R the level set fx 2 Rn j f .x/  g is closed.

Proof. Suppose that f is l.s.c. For any fix  2 R, denote ˇ ˚ ˝ WD x 2 Rn ˇ f .x/   :

If xN … ˝ , we have f .x/ N > . By definition, find ı > 0 such that f .x/ >  for all x 2 IB.xI N ı/;

which shows that IB.xI N ı/  ˝ c , the complement of ˝ . us ˝ c is open, and so ˝ is closed. To verify the converse implication, suppose that the set fx 2 Rn j f .x/  g is closed for any  2 R. Fix xN 2 Rn and  2 R with f .x/ N >  and then let ˇ ˚  WD x 2 Rn ˇ f .x/   : Since  is closed and xN …  , there is ı > 0 such that IB.xI N ı/   c . is implies f .x/ >  for all x 2 IB.xI N ı/

and thus completes the proof of the proposition. Proposition 4.5



e function f W Rn ! R is l.s.c. if and only if its epigraph is closed.

Proof. To verify the “only if ” part, fix .x; N / … epi f meaning  < f .x/ N . For any  > 0 with f .x/ N >  C  > , there is ı > 0 such that f .x/ >  C  whenever x 2 IB.xI N ı/. us IB.xI N ı/  .

;  C /  .epi f /c ;

and hence the set epi f is closed. Conversely, suppose that epi f is closed. Employing Proposition 4.4, it suffices to verify that for any  2 R the set ˝ WD fx 2 Rn j f .x/  g is closed. To see this, fix any sequence fxk g  ˝ that converges to xN . Since f .xk /  , we have .xk ; / 2 epi f for all k . It follows from .xk ; / ! .x; N / that .x; N / 2 epi f , and so f .x/ N  . us xN 2 ˝ , which justifies the closedness of the set ˝ . 

e function f W Rn ! R is l.s.c. at xN if and only if for any sequence xk ! xN we have lim infk!1 f .xk /  f .x/ N .

Proposition 4.6

132

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

Proof. If f is l.s.c. at xN , take a sequence of xk ! xN . For any  < f .x/ N , there exists ı > 0 N ı/, and so  < f .xk / for large k . It follows that   such that (4.1) holds. en xk 2 IB.xI lim infk!1 f .xk /, and hence f .x/ N  lim infk!1 f .xk / by the arbitrary choice of  < f .x/ N . To verify the converse, suppose by contradiction that f is not l.s.c. at xN . en there is  < 1 f .x/ N such that for every ı > 0 we have xı 2 IB.xI N ı/ with   f .xı /. Applying this to ık WD k gives us a sequence fxk g  Rn converging to xN with   f .xk /. It yields f .x/ N >   lim inf f .xk /; k!1

which is a contradiction that completes the proof.



Now we are ready to obtain a significant result on the lower semicontinuity of the minimal value function (3.9), which is useful in what follows. In the setting of eorem 3.27 with 0 2 F we have that the minimal time function T˝F is lower semicontinuous. eorem 4.7

Proof. Pick xN 2 Rn and fix an arbitrary sequence fxk g converging to xN as k ! 1. Based on Proposition 4.6, we check that

lim inf T˝F .xk /  T˝F .x/: N k!1

e above inequality obviously holds if lim infk!1 T˝F .xk / D 1; thus it remains to consider the case of lim infk!1 T˝F .xk / D 2 Œ0; 1/. Since it suffices to show that  T˝F .x/ N , we assume without loss of generality that T˝F .xk / ! . By the definition of the minimal time function, we find a sequence ftk g  Œ0; 1/ such that

T˝F .xk /  tk < T˝F .xk / C

1 and .xk C tk F / \ ˝ ¤ ; as k 2 N : k

For each k 2 N , fix fk 2 F and !k 2 ˝ with xk C tk fk D !k and suppose without loss of generality that !k ! ! 2 ˝ . Consider the two cases: > 0 and D 0. If > 0, then fk !

!

xN

2 F as k ! 1;

which implies that .xN C F / \ ˝ ¤ ;, and hence  T˝F .x/ N . Consider the other case where

D 0. In this case the sequence ftk g converges to 0, and so tk fk ! ! xN . It follows from Proposition 3.24 that ! xN 2 F1 , which yields xN 2 ˝ F1 . Employing eorem 3.27 tells us that T˝F .x/ N D 0 and also  T˝F .x/ N . is verifies the lower semicontinuity of T˝F in setting of e orem 3.27.

4.1. LOWER SEMICONTINUITY AND EXISTENCE OF MINIMIZERS

133

e following two propositions indicate two cases of preserving lower semicontinuity under important operations on functions.

Let fi W Rn ! R, i D 1; : : : ; m, be l.s.c. at some point xN . en the sum is l.s.c. at this point. Proposition 4.8

Pm

iD1

fi

Proof. It suffices to verify it for m D 2. Taking xk ! xN , we get

lim inf fi .xk /  f .x/ N for i D 1; 2; k!1

which immediately implies that   lim inf f1 .xk / C f2 .xk /  lim inf f1 .xk / C lim inf f2 .xk /  f1 .x/ N C f2 .x/; N k!1

k!1

k!1

and thus f1 C f2 is l.s.c. at xN by Proposition 4.6.



Let I be any nonempty index set and let fi W Rn ! R be l.s.c. for all i 2 I . en the supremum function f .x/ WD supi2I fi .x/ is also lower semicontinuous.

Proposition 4.9

Proof. For any number  2 R, we have directly from the definition that ˇ ˇ ˇ \˚ ˚ ˚ x 2 Rn ˇ fi .x/   : x 2 Rn ˇ f .x/   D x 2 Rn ˇ sup fi .x/   D i2I

i2I

us all the level sets of f are closed as intersections of closed sets. is insures the lower semicontinuity of f by Proposition 4.4.  e next result provides two interrelated unilateral versions of the classical Weierstrass existence theorem presented in eorem 1.18. e crucial difference is that now we assume the lower semicontinuity versus continuity while ensuring the existence of only minima, not together with maxima as in eorem 1.18. eorem 4.10

Let f W Rn ! R be a l.s.c. function. e following hold:

(i) e function f attains its absolute minimum on any nonempty, compact subset ˝  Rn that intersects its domain. (ii) Assume that infff .x/ j x 2 Rn g < 1 and that there is a real number  > infff .x/ j x 2 Rn g for which the level set fx 2 Rn j f .x/ < g of f is bounded. en f attains its absolute minimum on Rn at some point xN 2 dom f .

134

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

Proof. To verify (i), show first that f is bounded below on ˝ . Suppose on the contrary that for every k 2 N there is xk 2 ˝ such that f .xk /  k . Since ˝ is compact, we get without loss of generality that xk ! xN 2 ˝ . It follows from the lower semicontinuity that f .x/ N  lim inf f .xk / D k!1

1;

which is a contradiction showing that f is bounded below on ˝ . To justify now assertion (i), observe that WD infx2˝ f .x/ < 1. Take fxk g  ˝ such that f .xk / ! . By the compactness of ˝ , suppose that xk ! xN 2 ˝ as k ! 1. en f .x/ N  lim inf f .xk / D ; k!1

which shows that f .x/ N D , and so xN realizes the absolute minimum of f on ˝ . It remains to prove assertion (ii). It is not hard to show that f is bounded below, so WD infx2Rn f .x/ is a real number. Let fxk g  Rn for which f .xk / ! as k ! 1 and let k0 2 N be such that f .xk / <  for all k  k0 . Since the level set fx 2 Rn j f .x/ < g is bounded, we select a subsequence of fxk g (no relabeling) that converges to some xN . en f .x/ N  lim inf f .xk /  ; k!1

which shows that xN 2 dom f is a minimum point of f on Rn .



In contrast to all the results given above in this section, the next theorem requires convexity. It provides effective characterizations of the level set boundedness imposed in eorem 4.10(ii). Note that condition (iv) of the theorem below is known as coercivity. Let f W Rn ! R be convex and let infff .x/ j x 2 Rn g < 1. e following assertions are equivalent: eorem 4.11

(i) ere exists a real number ˇ > infff .x/ j x 2 Rn g for which the level set Lˇ WD fx 2 Rn j f .x/ < ˇg is bounded. (ii) All the level sets fx 2 Rn j f .x/ < g of f are bounded. (iii) limkxk!1 f .x/ D 1. f .x/ (iv) lim infkxk!1 > 0. kxk Proof. Let WD infff .x/ j x 2 Rn g, which is a real number or 1. We first verify (i)H)(ii). Suppose by contradiction that there is  2 R such that the level set L is not bounded. en there exists a sequence fxk g  L with kxk k ! 1. Fix ˛ 2 . ; ˇ/, xN 2 L˛ and define the convex combination   1  1 1  xk C 1 p xk xN D p x: N yk WD xN C p kxk k kxk k kxk k

4.2. OPTIMALITY CONDITIONS

135

By the convexity of f , we have the relationships f .yk /  p

1 kxk k

f .xk / C .1

p

1 kxk k

/f .x/ N 0 and a sequence fxk g such that kxk k ! 1 as k ! 1 and f .xk / < M for all k . But this obviously contradicts the boundedness of the level set fx 2 Rn j f .x/ < M g. Finally, we justify (iii)H)(i) leaving the proof of (iii)”(iv) as an exercise. Suppose that (iii) is satisfied and for M > , find ˛ such that f .x/  M whenever kxk > ˛ . en ˇ ˚ LM WD x 2 Rn ˇ f .x/ < M g  IB.0I ˛/; which shows that this set is bounded and thus verifies (i).

4.2



OPTIMALITY CONDITIONS

In this section we first derive necessary and sufficient optimality conditions for optimal solutions to the following set-constrained convex optimization problem: minimize f .x/ subject to x 2 ˝;

(4.3)

where ˝ is a nonempty, convex set and f W Rn ! R is a convex function. Definition 4.12 We say that xN 2 ˝ is a    to (4.3) if f .x/ N < 1 and there is > 0 such that f .x/  f .x/ N for all x 2 IB.xI N / \ ˝: (4.4)

If f .x/  f .x/ N for all x 2 ˝ , then xN 2 ˝ is a    to (4.3). e next result shows, in particular, that there is no difference between local and global solutions for problems of convex optimization. Proposition 4.13

e following are equivalent in the convex setting of (4.3):

(i) xN is a local optimal solution to the constrained problem (4.3). (ii) xN is a local optimal solution to the unconstrained problem

minimize g.x/ WD f .x/ C ı.xI ˝/ on Rn :

(4.5)

136

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

(iii) xN is a global optimal solution to the unconstrained problem (4.5). (iv) xN is a global optimal solution to the constrained problem (4.3). Proof. To verify (i)H)(ii), select > 0 such that (4.4) is satisfied. Since g.x/ D f .x/ for x 2 ˝ \ IB.xI N / and since g.x/ D 1 for x 2 IB.xI N / and x … ˝ , we have g.x/  g.x/ N for all x 2 IB.xI N /;

which means (ii). Implication (ii)H)(iii) is a consequence of Proposition 2.33. e verification of (iii)H)(iv) is similar to that of (i)H)(ii), and (vi)H)(i) is obvious.  In what follows we use the term “optimal solutions” for problems of convex optimization, without mentioning global or local. Observe that the objective function g of the unconstrained problem (4.5) associated with the constrained one (4.3) is always extended-real-valued when ˝ ¤ Rn . is reflects the “infinite penalty” for the constraint violation. e next theorem presents basic necessary and sufficient optimality conditions for optimal solutions to the constrained problem (4.3) with a general (convex) cost function. eorem 4.14 Consider the convex problem (4.3) with f W Rn ! R such that f .x/ N < 1 for some point xN 2 ˝ under the qualification condition   @1 f .x/ N \ N.xI N ˝/ D f0g; (4.6)

which automatically holds if f is continuous at xN . en the following are equivalent: (i) xN is an optimal solution to (4.3). (ii) 0 2 @f .x/ N C N.xI N ˝/, i.e., @f .x/ N \ Œ N.xI N ˝/ ¤ ;. (iii) f 0 .xI N d /  0 for all d 2 T .xI N ˝/. Proof. Observe first that @1 f .x/ N D f0g and thus (4.6) holds if f is continuous at xN . is follows from eorem 2.29. To verify now (i)H)(ii), we use the unconstrained form (4.5) of problem (4.3) due to Proposition 4.13. en Proposition 2.35 provides the optimal solution characterization 0 2 @g.x/ N . Applying the subdifferential sum rule from eorem 2.44 under the qualification condition (4.6) gives us the optimality condition in (ii). To show that (ii)H)(iii), we have by the tangent-normal duality (3.4) the equivalence   d 2 T .xI N ˝/ ” hv; d i  0 for all v 2 N.xI N ˝/ :

Fix an element w 2 @f .x/ N with w 2 N.xI N ˝/. en hw; d i  0 for all d 2 T .xI N ˝/ and we arrive at f 0 .xI N d /  hw; d i  0 by eorem 2.84, achieving (iii) in this way. It remains to justify implication (iii)H)(i). It follows from Lemma 2.83 that f .xN C d /

f .x/ N  f 0 .xI N d /  0 for all d 2 T .xI N ˝/:

4.2. OPTIMALITY CONDITIONS

 By definition of the tangent cone, we have T .xI N ˝/ D cl cone.˝ w xN 2 T .xI N ˝/ if w 2 ˝ . is yields f .w/

f .x/ N D f .w

xN C x/ N

137

 x/ N , which tells us that d WD

f .x/ N D f .xN C d /

f .x/ N  0;

which verifies (i) and completes the proof of the theorem.



Assume that in the framework of eorem 4.14 the cost function f W Rn ! R is differentiable at xN 2 dom f . en the following are equivalent: Corollary 4.15

(i) xN is an optimal solution to problem (4.3). (ii) rf .x/ N 2 N.xI N ˝/, which reduces to rf .x/ N D 0 if xN 2 int ˝ . Proof. Since the differentiability of the convex function f at xN implies its continuity and so LipN D frf .x/g N in this case by schitz continuity around this point by eorem 2.29. Since @f .x/ eorem 3.3, we deduce this corollary directly from eorem 4.14. 

Next we consider the following convex optimization problem containing functional inequality constraints given by finite convex functions, along with the set/geometric constraint as in (4.3). For simplicity, suppose that f is also finite on Rn . Problem .P / is: minimize f .x/ subject to gi .x/  0 for i D 1; : : : ; m; x 2 ˝: Define the Lagrange function for .P / involving only the cost and functional constraints by L.x; / WD f .x/ C

m X

i gi .x/

iD1

with i 2 R and the active index set given by I.x/ WD fi 2 f1; : : : ; mg j gi .x/ D 0g. eorem 4.16 Let xN be an optimal solution to .P /. en there are multipliers 0  0 and  D .1 ; : : : ; m / 2 Rm , not equal to zero simultaneously, such that i  0 as i D 0; : : : ; m,

0 2 0 @f .x/ N C

and i gi .x/ N D 0 for all i D 1; : : : ; m.

m X iD1

i @gi .x/ N C N.xI N ˝/;

138

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

Proof. Consider the finite convex function ˇ ˚ '.x/ WD max f .x/ f .x/; N gi .x/ ˇ i D 1; : : : ; m

and observe that xN solves the following problem with no functional constraints: minimize '.x/ subject to x 2 ˝: Since ' is finite and hence locally Lipschitz continuous, we get from eorem 4.14 that 0 2 @'.x/ N C N.xI N ˝/. e subdifferential maximum rule from Proposition 2.54 yields [   0 2 co @f .x/ N [Œ @gi .x/ N C N.xI N ˝/: i2I.x/ N

P is gives us 0  0 and i  0 for i 2 I.x/ N such that 0 C i2I.x/ N i D 1 and X 0 2 0 @f .x/ N C i @gi .x/ N C N.xI N ˝/:

(4.7)

i2I.x/ N

Letting i WD 0 for i … I.x/ N , we get i gi .x/ N D 0 for all i D 1; : : : ; m with .0 ; / ¤ 0.



Many applications including numerical algorithms to solve problem .P / require effective constraint qualification conditions that ensure 0 ¤ 0 in eorem 4.16. e following one is classical in convex optimization. Definition 4.17 S’   .CQ/ holds in .P / if there exists u 2 ˝ such that gi .u/ < 0 for all i D 1; : : : ; m.

e next theorem provides necessary and sufficient conditions for convex optimization in the qualified, normal, or Karush-Kuhn-Tucker .KKT/ form. Suppose that Slater’s CQ holds in .P /. en xN is an optimal solution to .P / if and only if there exist nonnegative Lagrange multipliers .1 ; : : : ; m / 2 Rm such that eorem 4.18

0 2 @f .x/ N C

m X iD1

i @gi .x/ N C N.xI N ˝/

(4.8)

and i gi .x/ N D 0 for all i D 1; : : : ; m. Proof. To verify the necessity part, we only need to show that 0 ¤ 0 in (4.7). Suppose on the contrary that 0 D 0 and find .1 ; : : : ; m / ¤ 0, vi 2 @gi .x/ N , and v 2 N.xI N ˝/ satisfying 0D

m X iD1

i vi C v:

4.2. OPTIMALITY CONDITIONS

139

is readily implies that 0D

m X iD1

i hvi ; x

xi N C hv; x

m X

xi N 

i gi .x/

iD1

m  X gi .x/ N D i gi .x/ for all x 2 ˝; iD1

P which contradicts since the Slater condition implies m iD1 i gi .u/ < 0. To check the sufficiency of (4.8), take v0 2 @f .x/ N , vi 2 @gi .x/ N , and v 2 N.xI N ˝/ such that P 0 D v0 C m  v C v with  g . x/ N D 0 and   0 for all i D 1; : : : ; m . en the optimality i i i iD1 i i of xN in .P / follows directly from the definitions of the subdifferential and normal cone. Indeed, for any x 2 ˝ with gi .x/  0 for i D 1; : : : ; m one has 0 D hi vi C v; x  D

m X iD1 m X iD1

i Œgi .x/

xi N D

m X iD1

i hvi ; x

gi .x/ N C f .x/

i gi .x/ C f .x/

xi N C hv; x

xi N

f .x/ N

f .x/ N  f .x/

f .x/; N

which completes the proof.



Finally in this section, we consider the convex optimization problem .Q/ with inequality and linear equality constraints minimize f .x/ subject to gi .x/  0 for i D 1; : : : ; m; Ax D b; where A is an p  n matrix and b 2 Rp . Corollary 4.19 Suppose that Slater’s CQ holds for problem .Q/ with the set ˝ WD fx 2 Rn j Ax b D 0g in Definition 4.17. en xN is an optimal solution to .Q/ if and only if there exist nonnegative multipliers 1 ; : : : ; m such that

0 2 @f .x/ N C

m X iD1

i @gi .x/ N C im A

and i gi .x/ N D 0 for all i D 1; : : : ; m. Proof. e result follows from eorem 4.16 and Proposition 2.12, where the normal cone to the set ˝ is calculated via the image of the adjoint operator A . 

140

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

e concluding result presents the classical version of the Lagrange multiplier rule for convex problems with differentiable data.

Let f and gi for i D 1; : : : ; m be differentiable at xN and let frgi .x/ N j i 2 I.x/g N be n linearly independent. en xN is an optimal solution to problem .P / with ˝ D R if and only if there are nonnegative multipliers 1 ; : : : ; m such that

Corollary 4.20

0 2 rf .x/ N C

m X iD1

i rgi .x/ N D Lx .x; N /

and i gi .x/ N D 0 for all i D 1; : : : ; m. Proof. In this case N.xI N ˝/ D f0g and it follows from (4.7) that 0 D 0 contradicts the linear independence of the gradient elements frgi .x/ N j i 2 I.x/g N . 

4.3

SUBGRADIENT METHODS IN CONVEX OPTIMIZATION

is section is devoted to algorithmic aspects of convex optimization and presents two versions of the classical subgradient method: one for unconstrained and the other for constrained optimization problems. First we study the unconstrained problem: minimize f .x/ subject to x 2 Rn ;

(4.9)

where f W Rn ! R is a convex function. Let f˛k g as k 2 N be a sequence of positive numbers. e subgradient algorithm generated by f˛k g is defined as follows. Given a starting point x1 2 Rn , consider the iterative procedure: xkC1 WD xk

˛k vk with some vk 2 @f .xk /;

k 2N:

(4.10)

In this section we assume that problem (4.9) admits an optimal solution and that f is convex and Lipschitz continuous on Rn . Our main goal is to find conditions that ensure the convergence of the algorithm. We split the proof of the main result into several propositions, which are of their own interest. e first one gives us an important subgradient property of convex Lipschitzian functions. Proposition 4.21

subgradient estimate

Let `  0 be a Lipschitz constant of f on Rn . For any x 2 Rn , we have the kvk  ` whenever v 2 @f .x/:

4.3. SUBGRADIENT METHODS IN CONVEX OPTIMIZATION

141

n

Proof. Fix x 2 R and take any subgradient v 2 @f .x/. It follows from the definition that hv; u

xi  f .u/

f .x/ for all u 2 Rn :

Combining this with the Lipschitz property of f gives us hv; u

xi  ` ku

u 2 Rn ;

xk ;

which yields in turn that hv; y C x

xk ; i.e. hv; yi  ` kyk ;

xi  ` ky C x

y 2 Rn :

Using y WD v , we arrive at the conclusion kvk  `.



Let `  0 be a Lipschitz constant of f on Rn . For the sequence fxk g generated by algorithm (4.10), we have  (4.11) kxkC1 xk2  kxk xk2 2˛k f .xk / f .x/ C ˛k2 `2 for all x 2 Rn ; k 2 N :

Proposition 4.22

Proof. Fix x 2 Rn , k 2 N and get from (4.10) that kxkC1

xk2 D kxk D kxk

˛k vk xk2 D kxk x ˛k vk k2 xk2 2˛k hvk ; xk xi C ˛k2 kvk k2 :

Since vk 2 @f .xk / and kvk k  ` by Proposition 4.21, it follows that hvk ; x

xk i  f .x/

f .xk /:

is gives us (4.11) and completes the proof of the proposition.



e next proposition provides an estimate of closeness of iterations to the optimal value of (4.9) for an arbitrary sequence f˛k g in the algorithm (4.10). Denote ˇ ˚ ˚ V WD min f .x/ ˇ x 2 Rn and Vk WD min f .x1 /; : : : ; f .xk / :

Let `  0 be a Lipschitz constant of f on Rn and let S be the set of optimal solutions to (4.9). For all k 2 N , we have

Proposition 4.23

0  Vk

V 

d.x1 I S/2 C `2 2

k P

iD1

˛i

k P

iD1

˛i2 :

(4.12)

142

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

Proof. Substituting any xN 2 S into estimate (4.11) gives us xk N 2  kxk

kxkC1

xk N 2

2˛k f .xk /

 f .x/ N C ˛k2 `2 :

N , we get Since Vk  f .xi / for all i D 1; : : : ; k and V D f .x/ kxkC1

xk N 2  kx1

xk N 2 xk N 2

 kx1

It follows from kxkC1

2

k X

2

k X

˛i f .xi /

iD1

k X  f .x/ N C `2 ˛i2 iD1

˛i .Vk

iD1

V / C `2

k X

˛i2 :

iD1

xk N 2  0 that 2

k X

˛i Vk

iD1

 V  kx1

xk N 2 C `2

k X

˛i2 :

iD1

Since Vk  V for every k , we deduce the estimate 0  Vk

V 

kx1

P xk N 2 C `2 kiD1 ˛i2 P 2 kiD1 ˛i

and hence arrive at (4.12) since xN 2 S was chosen arbitrarily.



e corollary below shows the number of steps in order to achieve an appropriate error estimate for the optimal value in the case of constant step sizes. Corollary 4.24 Suppose that ˛k D  for all k 2 N in the setting of Proposition 4.23. en there exists a positive constant C such that

0  Vk

V < `2  whenever k > C = 2 :

Proof. From (4.12) we readily get 0  Vk

V 

a C k`2  2 a `2  D C with a WD d.x1 I S /2 : 2k 2k 2

Since a=.2k/ C `2 =2 < `2  as k > a=.`2  2 /, the conclusion holds with C WD a=`2 .



As a direct consequence of Proposition 4.23, we get our first convergence result. Proposition 4.25

In the setting of Proposition 4.23 suppose in addition that 1 X kD1

˛k D 1 and

en we have the convergence Vk ! V as k ! 1.

lim ˛k D 0:

k!1

(4.13)

4.3. SUBGRADIENT METHODS IN CONVEX OPTIMIZATION

143

Proof. Given any  > 0, choose K 2 N such that ˛i ` <  for all i  K . For any k > K , using (4.12) ensures the estimates

0  Vk

V 

d.x1 I S /2 C `2 k P

2

K P

iD1

`2

˛i2 C

˛i

iDK

2

iD1



d.x1 I S/2 C `2 k P

2

K P

iD1

˛i2

 C

˛i

iD1



2

k P

˛i2

˛i

iD1 k P

˛i

iDK

2

k P

˛i

iD1

d.x1 I S/2 C `2 k P

k P

K P

iD1

˛i2

˛i

C =2:

iD1

e assumption

P1

kD1

˛k D 1 yields limk!1 0  lim inf.Vk k!1

Pk

iD1

˛i D 1, and hence

V /  lim sup.Vk k!1

Since  > 0 is chosen arbitrarily, one has limk!1 .Vk

V /  =2:

V / D 0 or Vk ! V as k ! 1.



e next result gives us important information about the value convergence of the subgradient algorithm (4.10). Proposition 4.26

In the setting of Proposition 4.25 we have lim inf f .xk / D V : k!1

Proof. Since V is the optimal value in (4.9), we get f .xk /  V for any k 2 N . is yields

lim inf f .xk /  V : k!1

To verify the opposite inequality, suppose on the contrary that lim infk!1 f .xk / > V and then find a small number  > 0 such that lim inf f .xk / k!1

2 > V D minn f .x/ D f .x/; N x2R

where xN is an optimal solution to problem (4.9). us lim inf f .xk / k!1

 > f .x/ N C :

144

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

In the case where lim infk!1 f .xk / < 1, by the definition of lim inf, there is k0 2 N such that for every k  k0 we have that f .xk / > lim inf f .xk / k!1

;

and hence f .xk / f .x/ N >  for every k  k0 . Note that this conclusion also holds if lim infk!1 f .xk / D 1. It follows from (4.11) with x D xN that  N 2  kxk xk N 2 2˛k f .xk / f .x/ N C ˛k2 `2 for every k  k0 : (4.14) kxkC1 xk Let k1 be such that ˛k `2 <  for all k  k1 and let kN1 WD maxfk1 ; k0 g. For any k > kN1 , it easily follows from estimate (4.14) that kxkC1

xk N 2  kxk

which contradicts the assumption

xk N 2

P1

kD1

˛k   : : :  xkN1

2

xN



k X

˛j ;

j DkN1

˛k D 1 in (4.13) and so completes the proof.



Now we are ready to justify the convergence of the subgradient algorithm (4.10). eorem 4.27

Suppose that the generating sequence f˛k g in (4.10) satisfies 1 X kD1

˛k D 1 and

1 X kD1

˛k2 < 1:

(4.15)

en the sequence fVk g converges to the optimal value V and the sequence of iterations fxk g in (4.10) converges to some optimal solution xN of (4.9). Proof. Since (4.15) implies (4.13), the sequence fVk g converges to V by Proposition 4.25. It remains to prove that fxk g converges to some optimal solution xN of problem (4.9). Let `  0 be a Lipschitz constant of f on Rn . Since the set S of optimal solutions to (4.9) is assumed to be nonempty, we pick any e x 2 S and substitute x D e x in estimate (4.11) of Proposition 4.22. It gives us  x k2  kxk e x k2 2˛k f .xk / V C ˛k2 `2 for all k 2 N : (4.16) kxkC1 e

Since f .xk /

V  0, inequality (4.16) yields kxkC1

e x k2  kxk

e x k2 C ˛k2 `2 :

en we get by induction that kxkC1

e x k2  kx1

e x k2 C `2

k X iD1

˛i2 :

(4.17)

4.3. SUBGRADIENT METHODS IN CONVEX OPTIMIZATION

e right-hand side of (4.17) is finite by our assumption that

P1

2 kD1 ˛k

145

< 1, and hence

e x k < 1:

sup kxkC1

k2N

is implies that the sequence fxk g is bounded. Furthermore, Proposition 4.26 tells us that lim inf f .xk / D V : k!1

Let fxkj g be a subsequence of fxk g such that lim f .xkj / D V :

(4.18)

j !1

en fxkj g is also bounded and contains a limiting point xN . Without loss of generality assume that xkj ! xN as j ! 1. We directly deduce from (4.18) and the continuity of f that xN is an optimal solution to (4.9). Fix any j 2 N and any k  kj . us we can rewrite (4.16) with e x D xN and see that kxkC1

2

xk N  kxk

2

xk N C

˛k2 `2

 : : :  xk

j

k X

2 2

xN C ` ˛i2 : iDkj

Taking first the limit as k ! 1 and then the limit as j ! 1 in the resulting inequality above gives us the estimate lim sup kxkC1 k!1

Since xkj ! xN and

P1

kD1

1 X

2 xN C `2 lim ˛i2 :

xk N 2  lim xkj

j !1

j !1

iDkj

˛k2 < 1, this implies that

lim sup kxkC1 k!1

xk N 2 D 0;

and thus justifies the claimed convergence of xk ! xN as k ! 1.



Finally in this section, we present a version of the subgradient algorithm for convex problems of constrained optimization given by minimize f .x/ subject to x 2 ˝;

(4.19)

where f W Rn ! R is a convex function that is Lipschitz continuous on Rn with some constant `  0, while—in contrast to (4.9)—we have now the constraint on x given by a nonempty, closed, convex set ˝  Rn . e following counterpart of the subgradient algorithm (4.10) for constrained problems is known as the projected subgradient method.

146

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

Given a sequence of positive numbers f˛k g and given a starting point x1 2 ˝ , the sequence of iterations fxk g is constructed by xkC1 WD ˘.xk

˛k vk I ˝/ with some vk 2 @f .xk /;

k 2 N;

(4.20)

where ˘.xI ˝/ stands for the (unique) Euclidean projection of x onto ˝ . e next proposition, which is a counterpart of Proposition 4.22 for algorithm (4.20), plays a major role in justifying convergence results for the projected subgradient method. Proposition 4.28

kxkC1

In the general setting of (4.20) we have the estimate  xk2  kxk xk2 2˛k f .xk / f .x/ C ˛k2 `2 for all x 2 ˝; k 2 N :

Proof. Define zkC1 WD xk kzkC1

˛k vk and fix any x 2 ˝ . It follows from Proposition 4.22 that  xk2  kxk xk2 2˛k f .xk / f .x/ C ˛k2 `2 for all k 2 N : (4.21)

Now by projecting zkC1 onto ˝ and using Proposition 1.79, we get kxkC1

xk D k˘.zkC1 I ˝/

˘.xI ˝/k  kzkC1

xk :

Combining this with estimate (4.21) gives us kxkC1

xk2  kxk

xk2

2˛k f .xk /

 f .x/ C ˛k2 `2 for all k 2 N;

which thus completes the proof of the proposition.



Based on this proposition and proceeding in the same way as for the subgradient algorithm (4.10) above, we can derive similar convergence results for the projected subgradient algorithm (4.20) under the assumptions (4.13) on the generating sequence f˛k g.

4.4

THE FERMAT-TORRICELLI PROBLEM

In 1643 Pierre de Fermat proposed to Evangelista Torricelli the following optimization problem: Given three points in the plane, find another point such that the sum of its distances to the given points is minimal. is problem was solved by Torricelli and was named the Fermat-Torricelli problem. A more general version of the Fermat-Torricelli problem asks for a point that minimizes the sum of the distances to a finite number of given points in Rn . is is one of the main problems in location science, which has been studied by many researchers. e first numerical algorithm for solving the general Fermat-Torricelli problem was introduced by Endre Weiszfeld in 1937. Assumptions that guarantee the convergence of the Weiszfeld algorithm were given by Harold Kuhn in 1972. Kuhn also pointed out an example in which the Weiszfeld algorithm fails to converge. Several new algorithms have been introduced recently to improve the Weiszfeld algorithm. e

4.4. THE FERMAT-TORRICELLI PROBLEM

147

goal of this section is to revisit the Fermat-Torricelli problem from both theoretical and numerical viewpoints by using techniques of convex analysis and optimization. Given points ai 2 Rn as i D 1; : : : ; m, define the (nonsmooth) convex function '.x/ WD

m X iD1

kx

(4.22)

ai k:

en the mathematical description of the Fermat-Torricelli problem is minimize '.x/ over all x 2 Rn : Proposition 4.29

(4.23)

e Fermat-Torricelli problem (4.23) has at least one solution.

Proof. e conclusion follows directly from eorem 4.10.



Proposition 4.30 Suppose that ai as i D 1; : : : ; m are not collinear, i.e., they do not belong to the same line. en the function ' in (4.22) is strictly convex, and hence the Fermat-Torricelli problem (4.23) has a unique solution.

Proof. Since each function 'i .x/ WD kx ai k as i D 1; : : : ; m is obviously convex, their sum P n 'D m iD1 'i is convex as well, i.e., for any x; y 2 R and  2 .0; 1/ we have  ' x C .1 /y  '.x/ C .1 /'.y/: (4.24)

Supposing by contradiction that ' is not strictly convex, find x; N yN 2 Rn with xN ¤ yN and  2 .0; 1/ such that (4.24) holds as equality. It follows that  'i xN C .1 /yN D 'i .x/ N C .1 /'i .y/ N for all i D 1; : : : ; m; which ensures therefore that k.xN

ai / C .1

/.yN

ai /k D k.xN

ai /k C k.1

If xN ¤ ai and yN ¤ ai , then there exists ti > 0 such that ti .xN xN

ai / with i WD

ai D i .yN

/.yN

ai /k;

ai / D .1 1

i D 1; : : : ; m: /.yN

ai /, and hence

 : ti 

Since xN ¤ yN , we have i ¤ 1. us ai D

1 1

i

xN

i 1

i

yN 2 L.x; N y/; N

where L.x; N y/ N signifies the line connecting xN and yN . Both cases where xN D ai and yN D ai give us ai 2 L.x; N y/ N . Hence ai 2 L.x; N y/ N for i D 1; : : : ; m, a contradiction. 

148

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

We proceed further with solving the Fermat-Torricelli problem for three points in Rn . Let us first present two elementary lemmas.

Let a1 and a2 be any two unit vectors in Rn . en we have a1 1 only if ha1 ; a2 i  . 2 Lemma 4.31

Proof. If a1

a2 2 IB if and

a2 2 IB , then a1 C a2 2 IB and ka1 C a2 k  1:

By squaring both sides, we get the inequality ka1 C a2 k2  1;

which subsequently implies that ka1 k2 C 2ha1 ; a2 i C ka2 k2 D 2 C 2ha1 ; a2 i  1:

erefore, ha1 ; a2 i 

1 . e proof of the converse implication is similar. 2



Let a1 ; a2 ; a3 2 Rn with ka1 k D ka2 k D ka3 k D 1. en a1 C a2 C a3 D 0 if and only if we have the equalities

Lemma 4.32

1 : 2

ha1 ; a2 i D ha2 ; a3 i D ha3 ; a1 i D

(4.25)

Proof. It follows from the condition a1 C a2 C a3 D 0 that ka1 k2 C ha1 ; a2 i C ha1 ; a3 i D 0;

(4.26)

ha2 ; a1 i C ka2 k2 C ha2 ; a3 i D 0;

(4.27)

ha3 ; a1 i C ha3 ; a2 i C ka3 k D 0:

(4.28)

2

Subtracting (4.28) from (4.26) gives us ha1 ; a2 i D ha2 ; a3 i and similarly ha2 ; a3 i D ha3 ; a1 i. Substituting these equalities into (4.26), (4.27), and (4.28) ensures the validity of (4.25). Conversely, assume that (4.25) holds. en we have ka1 C a2 C a3 k2 D

3 X iD1

kai k2 C

3 X i;j D1;i ¤j

hai ; aj i D 0;

which readily yields the equality a1 C a2 C a3 D 0 and thus completes the proof.



4.4. THE FERMAT-TORRICELLI PROBLEM

149

Now we use subdifferentiation of convex functions to solve the Fermat-Torricelli problem for three points in Rn . Given two nonzero vectors u; v 2 Rn , define cos.u; v/ WD

hu; vi kuk  kvk

and also consider, when xN ¤ ai , the three vectors vi D

xN kxN

ai ; i D 1; 2; 3: ai k

Geometrically, each vector vi is a unit one pointing in the direction from the vertex ai to xN . Observe that the classical (three-point) Fermat-Torricelli problem always has a unique solution even if the given points belong to the same line. Indeed, in the latter case the middle point solves the problem. e next theorem fully describes optimal solutions of the Fermat-Torricelli problem for three points in two distinct cases. eorem 4.33 Consider the Fermat-Torricelli problem (4.23) for the three points a1 ; a2 ; a3 2 Rn . e following assertions hold:

(i) Let xN … fa1 ; a2 ; a3 g. en xN is the solution of the problem if and only if

cos.v1 ; v2 / D cos.v2 ; v3 / D cos.v3 ; v1 / D 1=2: (ii) Let xN 2 fa1 ; a2 ; a3 g, say xN D a1 . en xN is the solution of the problem if and only if

cos.v2 ; v3 /  1=2: Proof. In case (i) function (4.22) is differentiable at xN . en xN solves (4.23) if and only if r'.x/ N D v1 C v2 C v3 D 0:

By Lemma 4.32 this holds if and only if hvi ; vj i D cos.vi ; vj / D

˚ 1=2 for any i; j 2 1; 2; 3 with i ¤ j:

To verify (ii), employ the subdifferential Fermat rule (2.35) and sum rule from Corollary 2.45, which tell us that xN D a1 solves the Fermat-Torricelli problem if and only if 0 2 @'.a1 / D v2 C v3 C IB:

Since v2 and v3 are unit vectors, it follows from Lemma 4.31 that hv2 ; v3 i D cos.v2 ; v3 / 

which completes the proof of the theorem.

1=2; 

150

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

Figure 4.2: Construction of the Torricelli point.

is example illustrates geometrically the solution construction for the FermatTorricelli problem in the plane. Consider (4.23) with the three given points A, B , and C on R2 as shown in Figure 4.2. If one of the angles of the triangle ABC is greater than or equal to 120ı , then the corresponding vertex is the solution to (4.23) by eorem 4.33(ii). Consider the case where none of the angles of the triangle is greater than or equal to 120ı . Construct the two equilateral triangles ABD and ACE and let S be the intersection of DC and BE as in the figure. Two quadrilaterals ADBC and ABCE are convex, and hence S lies inside the triangle ABC . It is clear that two triangles DAC and BAE are congruent. A rotation of 60ı about A maps the triangle DAC to the triangle BAE . e rotation maps CD to BE , so †DSB D 60ı . Let T be the image of S though this rotation. en T belongs to BE . It follows that †AS T D †ASE D 60ı . Moreover, †DSA D 60ı , and hence †BSA D 120ı . It is now clear that †AS C D 120ı and †BS C D 120ı . By eorem 4.33(i) the point S is the solution of the problem under consideration. Example 4.34

Next we present the Weiszfeld algorithm [32] to solve numerically the Fermat-Torricelli problem in the general case of (4.23) for m points in Rn that are not collinear. is is our standing assumption in the rest of this section. To proceed, observe that the gradient of the function ' from (4.22) is calculated by m X x r'.x/ D kx iD1

˚ ai for x … a1 ; : : : ; am : ai k

Solving the gradient equation r'.x/ D 0 gives us Pm

iD1

xD

Pm

iD1

ai kx kx

1

ai k ai k

DW f .x/:

(4.29)

4.4. THE FERMAT-TORRICELLI PROBLEM

151

n

We define f .x/ on the whole space R by letting f .x/ WD x for x 2 fa1 ; : : : ; am g. Weiszfeld algorithm consists of choosing an arbitrary starting point x1 2 Rn , taking f is from (4.29), and defining recurrently xkC1 WD f .xk / for k 2 N;

(4.30)

where f .x/ is called the algorithm mapping. In the rest of this section we present the precise conditions and the proof of convergence for Weiszfeld algorithm in the way suggested by Kuhn [10] who corrected some errors from [32]. In contrast to [10], we employ below subdifferential rules of convex analysis, which allow us to simplify the proof of convergence. e next proposition ensures decreasing the function value of ' after each iteration, i.e., the descent property of the algorithm. Proposition 4.35

If f .x/ ¤ x , then '.f .x// < '.x/.

Proof. e assumption of f .x/ ¤ x tells us that x is not a vertex from fa1 ; : : : ; am g. Observe also that the point f .x/ is a unique minimizer of the strictly convex function g.z/ WD

m X kz

a i k2 kx ai k

iD1

and that g.f .x// < g.x/ D '.x/ since f .x/ ¤ x . We have furthermore that m  X kf .x/ ai k2 g f .x/ D kx ai k iD1 m X .kx ai k C kf .x/ ai k D kx ai k iD1

D '.x/ C 2 '.f .x//



'.x/ C

kx

ai k/2

m X .kf .x/ iD1

ai k kx kx ai k

ai k/2

:

is clearly implies the inequality 

2' f .x/ C

m X .kf .x/ iD1

ai k kx kx ai k

ai k/2

< 2'.x/

and thus completes the proof of the proposition.



e following two propositions show how the algorithm mapping f behaves near a vertex that solves the Fermat-Torricelli problem. First we present a necessary and sufficient condition ensuring the optimality of a vertex in (4.23). Define Rj WD

m X iD1;i ¤j

ai kai

aj ; aj k

j D 1; : : : ; m:

152

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

Proposition 4.36

e vertex aj solves problem (4.23) if and only if kRj k  1.

Proof. Employing the Fermat rule from Proposition 2.35 and the subdifferential sum rule from Corollary 2.45 shows that the vertex aj solves problem (4.23) if and only if 0 2 @'.aj / D

Rj C IB.0; 1/;

which is clearly equivalent to the condition kRj k  1.



Proposition 4.36 from convex analysis allows us to essentially simplify the proof of the following result established by Kuhn. We use the notation:  f s .x/ WD f f s 1 .x/ for s 2 N with f 0 .x/ WD x; x 2 Rn :

Proposition 4.37 condition 0 < kx

Suppose that the vertex aj does not solve (4.23). en there is ı > 0 such that the aj k  ı implies the existence of s 2 N with kf s .x/

aj k > ı and kf s

1

.x/

(4.31)

aj k  ı:

Proof. For any x … fa1 ; : : : ; am g, we have from (4.29) that Pm

ai aj kx ai k and so 1 kx ai k

iD1;i¤j

f .x/

aj D

Pm

iD1

Pm

ai aj f .x/ aj kx ai k lim D lim D Rj ; Pm x!aj kx x!aj kx aj k aj k 1 C iD1;i ¤j kx ai k iD1;i ¤j

j D 1; : : : ; m:

It follows from Proposition 4.36 that lim

x!aj

kf .x/ aj k D kRj k > 1; kx aj k

j D 1; : : : ; m;

which allows us to select  > 0; ı > 0 such that kf .x/ aj k > 1 C  whenever 0 < kx kx aj k

aj k  ı:

(4.32)

4.4. THE FERMAT-TORRICELLI PROBLEM

us for every s 2 N the validity of 0 < kf kf s .x/

Since .1 C /s kx

aj k  .1 C /kf s

q 1 1

.x/

.x/

153

aj k  ı with any q D 1; : : : ; s yields aj k  : : :  .1 C /s kx

aj k:

aj k ! 1 as s ! 1, it gives us (4.31) and completes the proof.



We finally present the main result on the convergence of the Weiszfeld algorithm. eorem 4.38 Let fxk g be the sequence generated by Weiszfeld algorithm (4.30) and let xk … fa1 ; : : : ; am g for k 2 N . en fxk g converges to the unique solution xN of (4.23).

Proof. If xk D xkC1 for some k D k0 , then fxk g is a constant sequence for k  k0 , and hence it converges to xk0 . Since f .xk0 / D xk0 and xk0 is not a vertex, xk0 solves the problem. us we can assume that xkC1 ¤ xk for every k . It follows from Proposition 4.35 that the sequence f'.xk /g of nonnegative numbers is decreasing, so it converges. is gives us

lim '.xk /

k!1

 '.xkC1 / D 0:

(4.33)

By the construction of fxk g, we have that each xk belongs to the compact set co fa1 ; : : : ; am g. is allows us to select from fxk g a subsequence fxk g converging to some point zN as  ! 1. We have to show that zN D xN , i.e., it is the unique solution to (4.23). Indeed, we get from (4.33) that  lim '.xk / ' f .xk / D 0; !1

which implies by the continuity of the functions ' and f that '.z/ N D '.f .z// N , i.e., f .z/ N D zN and thus zN solves (4.23) if it is not a vertex. It remains to consider the case where zN is a vertex, say a1 . Suppose by contradiction that zN D a1 ¤ xN and select ı > 0 sufficiently small so that (4.31) holds for some s 2 N and that IB.a1 I ı/ does not contain xN and ai for i D 2; : : : ; m. Since xk ! zN D a1 , we can assume without loss of generality that this sequence is entirely contained in IB.a1 I ı/. For x D xk1 select l1 so that xl1 2 IB.a1 I ı/ and f .xl1 / … IB.a1 I ı/. Choosing now k > l1 and applying Proposition 4.37 give us l2 > l1 with xl2 2 IB.a1 I ı/ and f .xl2 / … IB.a1 I ı/. Repeating this procedure, find a subsequence fxl g such that xl 2 IB.a1 I ı/ while f .xl / is not in this ball. Extracting another subsequence if needed, we can assume that fxl g converges to some point qN satisfying f .q/ N D qN . If qN is not a vertex, then it solves (4.23) by the above, which is a contradiction because the solution xN is not in IB.a1 I ı/. us qN is a vertex, which must be a1 since the other vertexes are also not in the ball. en we have kf .xl / a1 k D 1; lim !1 kxl a1 k  which contradicts (4.32) in Proposition 4.37 and completes the proof of the theorem.



154

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

4.5

A GENERALIZED FERMAT-TORRICELLI PROBLEM

ere are several generalized versions of the Fermat-Torricelli problem known in the literature under different names; see, e.g., our papers [15, 16, 17] and the references therein. Note that the constrained version of the problem below is also called a “generalized Heron problem” after Heron from Alexandria who formulated its simplest version in the plane in the 1st century AD. Following [15, 16], we consider here a generalized Fermat-Torricelli problem defined via the minimal time functions (3.9) for finitely many closed, convex target sets in Rn with a constraint set of the same structure. In [6, 11, 29] and their references, the reader can find some particular versions of this problem and related material on location problems. We also refer the reader to [15, 17] to nonconvex problems of this type. Let the target sets ˝i for i D 1; : : : ; m and the constraint set ˝0 be nonempty, closed, convex subsets of Rn . Consider the minimal time functions TF .xI ˝i / D T˝Fi .x/ from (3.9) for ˝i , i D 1; : : : ; m, with the constant dynamics defined by a closed, bounded, convex set F  Rn that contains the origin as an interior point. en the generalized Fermat-Torricelli problem is formulated as follows: minimize H.x/ WD

m X iD1

TF .xI ˝i / subject to x 2 ˝0 :

(4.34)

Our first goal in this section is to establish the existence and uniqueness of solutions to the optimization problem (4.34). We split the proof into several propositions of their own interest. Let us start with the following property of the Minkowski gauge from (3.11). Proposition 4.39 Assume that F is a closed, bounded, convex subset of Rn such that 0 2 int F . en F .x/ D 1 if and only if x 2 bd F .

Proof. Suppose that F .x/ D 1. en there exists a sequence of real numbers ftk g that converges to t D 1 and such that x 2 tk F for every k . Since F is closed, it yields x 2 F . To complete the proof of the implication, we show that x … int F . By contradiction, assume that x 2 int F and choose t > 0 so small that x C tx 2 F . en x 2 .1 C t/ 1 F . is tells us F .x/  .1 C t/ 1 < 1, a contradiction. Now let x 2 bd F . Since F is closed, we have x 2 F , and hence F .x/  1. Suppose on the contrary that WD F .x/ < 1. en there exists a sequence of real numbers ftk g that converges to with x 2 tk F for every k . en x 2 F D F C .1 /0  F . By Proposition 3.32, the Minkowski gauge F is continuous, so the set fu 2 Rn j F .u/ < 1g is an open subset of F containing x . us x 2 int F , which completes the proof. 

Recall further that a set Q  Rn is said to be strictly convex if for any x; y 2 Q with x ¤ y and for any t 2 .0; 1/ we have tx C .1 t/y 2 int Q.

4.5. A GENERALIZED FERMAT-TORRICELLI PROBLEM

Proposition 4.40 0 2 int F . en

155

n

Assume that F is a closed, bounded, strictly convex subset of R and such that F .x C y/ D F .x/ C F .y/ with x; y ¤ 0

(4.35)

if and only if x D y for some  > 0. Proof. Denote ˛ WD F .x/, ˇ WD F .y/ and observe that ˛; ˇ > 0 since F is bounded. If the linearity property (4.35) holds, then x C y  F D 1; ˛Cˇ

which implies in turn that F

is allows us to conclude that

y ˇ  ˛ C D 1: ˛˛Cˇ ˇ˛Cˇ

x

x ˛ y ˇ C 2 bd F: ˛˛Cˇ ˇ˛Cˇ

Since

y ˛ x 2 F , 2 F , and 2 .0; 1/, it follows from the strict convexity of F that ˛ ˇ ˛Cˇ x y F .x/ D ; i.e., x D y with  D : ˛ ˇ F .y/

e opposite implication in the proposition is obvious.



Proposition 4.41 Let F be as in Proposition 4.40 and let ˝ be a nonempty, closed, convex subset of Rn . For any x 2 Rn , the set ˘F .xI ˝/ is a singleton.

Proof. We only need to consider the case where x … ˝ . It is clear that ˘F .xI ˝/ ¤ ;. Suppose by contradiction that there are u1 ; u2 2 ˘F .xI ˝/ with u1 ¤ u2 . en

TF .xI ˝/ D F .u1 which implies that

u1

x r

2 bd F and

u2

x/ D F .u2 x

r

2 bd F . Since F is strictly convex, we have

1 u1 x 1 u2 x 1  u1 C u2 C D 2 r 2 r r 2

and so we get F .u

x/ < r D TF .xI ˝/ with u WD

the proof of the proposition.

x/ D r > 0;

 x 2 int F;

u1 C u2 2 ˝ . is contradiction completes 2 

156

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

In what follows we identify the generalized projection ˘F .xI ˝/ with its unique element when both sets F and ˝ are strictly convex.

Let F be a closed, bounded, convex subset of Rn such that 0 2 int F and let ˝ be a nonempty, closed set ˝  Rn with xN … ˝ . If !N 2 ˘F .xI N ˝/, then !N 2 bd ˝ . Proposition 4.42

Proof. Suppose that !N 2 int ˝ . en there is  > 0 such that IB.!I N /  ˝ . Define z WD !N C 

xN kxN

!N 2 IB.!I N /: !k N

and observe from the construction of the Minkowski gauge (3.11) that F .z

 xN x/ N D F !N C  kxN

!N !k N

  xN D 1

 kxN

!k N

 F .!N

is shows that !N must be a boundary point of ˝ .

x/ N < F .!N

x/ N D TF .xI N ˝/: 

Figure 4.3: ree sets that satisfy assumptions of Proposition 4.43. Proposition 4.43 In the setting of problem (4.34) suppose that the sets ˝i for i D 1; : : : ; m are strictly convex. If for any x; y 2 ˝0 with x ¤ y there is i0 2 f1; : : : ; mg such that

L.x; y/ \ ˝i0 D ; for the line L.x; y/ connecting x and y , then the function

H.x/ D

m X

TF .xI ˝i /

iD1

defined in (4.34) is strictly convex on the constraint set ˝0 .

4.5. A GENERALIZED FERMAT-TORRICELLI PROBLEM

157

Proof. Suppose by contradiction that there are x; y 2 ˝0 , x ¤ y , and t 2 .0; 1/ with  t/y D t H.x/ C .1

H tx C .1

t/H.y/:

Using the convexity of each function TF .xI ˝i / for i D 1; : : : ; m, we have  TF tx C .1 t/yI ˝i D t TF .xI ˝i / C .1 t/TF .yI ˝i /; i D 1; : : : ; m:

(4.36)

Suppose further that L.x; y/ \ ˝i0 D ; for some i0 2 f1; : : : ; mg and define u WD ˘F .xI ˝i0 / and v WD ˘F .yI ˝i0 /:

en it follows from (4.36), eorem 3.33, and the convexity of the Minkowski gauge that tF .u

x/ C .1

t/F .v

y/ D t TF .xI ˝i0 / C .1 t/TF .yI ˝i0 / D TF tx C .1 t/yI ˝i0   F t u C .1 t /v .tx C .1 t/y/  tF .u x/ C .1 t /F .v y/;

which shows that tu C .1 t/v D ˘F .tx C .1 t/yI ˝i0 /. us u D v , since otherwise we have tu C .1 t/v 2 int ˝i0 , a contradiction. is gives us u D v D ˘F .tx C .1 t /yI ˝i0 /. Applying (4.36) again tells us that    F u .tx C .1 t/y D F t.u x/ C F .1 t /.u y/ ; so u x; u y ¤ 0 due to x; y … ˝i0 . By Proposition 4.40, we find  > 0 such that t.u .1 t /.u y/, which yields u

y/ with D

x D .u

.1

t/ t

x/ D

¤ 1:

is justifies the inclusion uD

1 1

x

1

y 2 L.x; y/;

which is a contradiction that completes the proof of the proposition.



Now we are ready to establish the existence and uniqueness theorem for (4.34). eorem 4.44 In the setting of Proposition 4.43 suppose further that at least one of the sets among ˝i for i D 0; : : : ; m is bounded. en the generalized Fermat-Torricelli problem (4.34) has a unique solution.

158

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

Proof. By the strict convexity of H from Proposition 4.43, it suffices to show that problem (4.34) admits an optimal solution. If ˝0 is bounded, then it is compact and the conclusion follows from the (Weierstrass) eorem 4.10 since the function H in (4.34) Lipschitz continuous by Corollary 3.34. Consider now the case where one of the sets ˝i for i D 1; : : : ; m is bounded; say it is ˝1 . In this case we get that for any ˛ 2 R the set ˇ ˇ ˚ ˚ L˛ D x 2 ˝0 ˇ H.x/  ˛  x 2 ˝0 ˇ TF .xI ˝1 /  ˛  ˝0 \ .˝1 ˛F /

is compact. is ensures the existence of a minimizer for (4.34) by eorem 4.10.



Next we completely solve by means of convex analysis and optimization the following simplified version of problem (4.34): minimize H.x/ D

3 X iD1

d.xI ˝i /;

x 2 R2 ;

(4.37)

where the target sets ˝i WD IB.ci I ri / for i D 1; 2; 3 are the closed balls of R2 whose centers ci are supposed to be distinct from each other to avoid triviality. We first study properties of this problem that enable us to derive verifiable conditions in terms of the initial data .ci ; ri / ensuring the uniqueness of solutions to (4.37). Proposition 4.45

e solution set S of (4.37) is a nonempty, compact, convex set in R2 .

Proof. e existence of solutions (4.37) follows from the proof of eorem 4.44. It is also easy to see that S is closed and bounded. e convexity of S can be deduced from the convexity of the distance functions d.xI ˝i /. 

To proceed, for any u 2 R2 define the index subset ˇ ˚ I .u/ WD i 2 f1; 2; 3g ˇ u 2 ˝i ;

(4.38)

which is widely used in what follows. Proposition 4.46 Suppose that xN does not belong to any of the sets ˝i , i D 1; 2; 3, i.e., I .x/ N D ;. en xN is an optimal solution to (4.37) if and only if xN solves the classical Fermat-Torricelli problem (4.23) generated by the centers of the balls ai WD ci for i D 1; 2; 3 in (4.22).

Proof. Assume that xN is a solution to (4.37) with xN … ˝i for i D 1; 2; 3. Choose ı > 0 such that IB.xI N ı/ \ ˝i D ; as i D 1; 2; 3 and get following for every x 2 IB.xI N ı/: 3 X iD1

kxN

ci k

3 X iD1

ri D inf H.x/ D H.x/ N  H.x/ D x2R2

3 X iD1

kx

ci k

3 X iD1

ri :

4.5. A GENERALIZED FERMAT-TORRICELLI PROBLEM

159

is shows that xN is a local minimum of the problem minimize F .x/ WD

3 X iD1

kx

ci k;

x 2 R2 ;

(4.39)

so it is a global absolute minimum of this problem due to the convexity of F . us xN solves the classical Fermat-Torricelli problem (4.23) generated by c1 ; c2 ; c3 . e converse implication is also straightforward.  e next result gives a condition for the uniqueness of solutions to (4.37) in terms of the index subset defined in (4.38).

Let xN be a solution to (4.37) such that I .x/ N D ; in (4.38). en xN is the unique solution to (4.37), i.e., S D fxg N .

Proposition 4.47

Proof. Suppose by contradiction that there is u 2 S such that u ¤ xN . Since I .x/ N D ;, we have xN … ˝i for i D 1; 2; 3. en Proposition 4.46 tells us that xN solves the classical Fermat-Torricelli problem generated by c1 ; c2 ; c3 . It follows from the convexity of S in Proposition 4.45 that Œx; N u  S . us we find v ¤ xN such that v 2 S and v … ˝i for i D 1; 2; 3. It shows that v also solves the classical Fermat-Torricelli problem (4.39). is contradicts the uniqueness result for (4.39) and hence justifies the claim of the proposition. 

e following proposition verifies a visible geometric property of ellipsoids.

Given ˛ > 0 and a; b 2 R2 with a ¤ b , consider the set ˇ ˚ E WD x 2 R2 ˇ kx ak C kx bk D ˛

Proposition 4.48

and suppose that ka

bk < ˛ . For x; y 2 E with x ¤ y , we have

x C y

2

which ensures, in particular, that

x C y

a C 2

b < ˛;

xCy … E. 2

Proof. It follows from the triangle inequality that

x C y

2

x C y

a C 2

1

b D kx 2 1  kx 2

aCy ak C ky

ak C kx ak C kx

bCy

bk

bk C ky

  bk D ˛:

160

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

Suppose by contradiction that

x C y

2

which means that

x C y

a C 2

b D ˛;

(4.40)

xCy 2 E . Since our norm is (always) Euclidean, it gives 2 x

a D ˇ.y

a/ and x

b D .y

b/

for some numbers ˇ; 2 .0; 1/ n f1g since x ¤ y . is implies in turn that aD

1 1

ˇ

ˇ

x

1

ˇ

y 2 L.x; y/ and b D

1 1

x

1

y 2 L.x; y/:

xCy 2 Œa; b, impose bk, we see that x; y 2 L.a; b/ n Œa; b. To check now that 2 xCy without loss of generality the following order of points: x , , a; b , and y . en 2

x C y

x C y



a C b < kx ak C kx bk D ˛;

2 2

Since ˛ > ka

xCy 2 Œa; b, it yields 2

x C y

x C y



a C b D ka

2 2

which contradicts (4.40). Since

bk < ˛;

xCy … E and the proof is complete.  2 e next result ensures the uniqueness of solutions to (4.37) via the index set (4.38) differently from Proposition 4.47.

which is a contradiction. us

Proposition 4.49 Let S be the solution set for problem (4.37) and let xN 2 S satisfy the conditions: I .x/ N D fig and xN … Œcj ; ck , where i; j; k are distinct indices in f1; 2; 3g. en S is a singleton, namely S D fxg N .

Proof. Suppose for definiteness that I .x/ N D f1g. en we have xN 2 ˝1 , xN … ˝2 , xN … ˝3 , and xN … Œc2 ; c3 . Denote ˛ WD kxN

and observe that ˛ > kc2

c2 k C kxN

c3 k D inf H.x/ C r2 C r3 x2R2

c3 k. Consider the ellipse ˇ ˚ E WD x 2 R2 ˇ kx c2 k C kx

c3 k D ˛



4.5. A GENERALIZED FERMAT-TORRICELLI PROBLEM

161

for which we get xN 2 E \ ˝1 . Arguing by contradiction, suppose that S is not a singleton, i.e., there is u 2 S with u ¤ xN . e convexity of S implies that Œx; N u  S . We can choose v 2 Œx; N u sufficiently close to but different from xN so that Œx; N v \ ˝2 D ; and Œx; N v \ ˝3 D ;. Let us first show that v … ˝1 . Assuming the contrary gives us d.vI ˝1 / D 0, and hence

H.v/ D inf H.x/ D d.vI ˝2 / C d.vI ˝3 / D kv x2R2

c2 k

r2 C kv

c3 k

r3 ;

xN C v which implies that v 2 E . It follows from the convexity of S and ˝1 that 2 S \ ˝1 . We 2 xN C v xN C v xN C v … ˝2 and … ˝3 , which tell us that 2 E . But this have furthermore that 2 2 2 cannot happen due to Proposition 4.48. erefore, v 2 S and I .v/ D ;, which contradict the result from Proposition 4.47 and thus justify that S is a singleton. 

Figure 4.4: A Fermat-Torricelli problem with jI.x/j N D 2.

Now we verify the expected boundary property of optimal solutions to (4.37). Proposition 4.50

Let xN 2 S and let I .x/ N D fi; j g  f1; 2; 3g. en xN 2 bd .˝i \ ˝j /.

Proof. Assume for definiteness I .x/ N D f1; 2g, i.e., xN 2 ˝1 \ ˝2 and xN … ˝3 . Arguing by contradiction, suppose that xN 2 int .˝1 \ ˝2 /, and therefore there exists ı > 0 such that IB.xI N ı/  ˝1 \ ˝2 . Denote p WD ˘.xI N ˝3 /, WD kxN pk > 0 and take q 2 Œx; N p \ IB.x; N ı/ satisfying the condition kq pk < kxN pk D d.xI N ˝3 /. en we get

H.q/ D

3 X iD1

d.qI ˝i / D d.qI ˝3 /  kq

pk < kxN

pk D d.xI N ˝3 / D

which contradicts the fact that xN 2 S and thus completes the proof.

3 X iD1

d.xI N ˝i / D H.x/; N 

e following result gives us verifiable necessary and sufficient conditions for the solution uniqueness in (4.37).

162

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

Figure 4.5: A generalized Fermat-Torricelli problem with infinite solutions.

Let \3iD1 ˝i D ;. en problem (4.37) has more than one solutions if and only if there is a set Œci ; cj  \ ˝k for distinct indices i; j; k 2 f1; 2; 3g that contains a point u 2 R2 belonging to the interior of ˝k and such that the index set I .u/ is a singleton. eorem 4.51

Proof. Let u 2 Œc1 ; c2  \ ˝3 , u 2 int ˝3 satisfy the conditions in the “if ” part of the theorem. en u … ˝1 , u … ˝2 and we can choose v ¤ u with v 2 Œc1 ; c2 ], v … ˝1 , v … ˝2 , and v 2 int ˝3 . is shows that the set I .v/ is a singleton. Let us verify that in this case u is a solution to (4.37). To proceed, we observe first that

H.u/ D d.uI ˝1 / C d.uI ˝2 / C d.uI ˝3 / D kc1

c2 k

r1

r2 :

Choose ı > 0 such that IB.uI ı/ \ ˝1 D ;, IB.uI ı/ \ ˝2 D ;, and IB.uI ı/  ˝3 . For every x 2 IB.uI ı/, we get

H.x/ D d.xI ˝2 / C d.xI ˝3 / D kx

c1 k C kx

c2 k

r1

r2  kc1

c2 k

r1

r2 D H.u/;

which implies that u is a local (and hence global) minimizer of H. Similar arguments show that v 2 S , so the solution set S is not a singleton. To justify the “only if ” part, suppose that S is not a singleton and pick two distinct elements x; N yN 2 S , which yields Œx; N y N  S by Proposition 4.45. If there is a solution to (4.37) not belonging to ˝i for every i 2 f1; 2; 3g, then S reduces to a singleton due to Proposition 4.47, a contradiction. us we can assume by the above that ˝1 contains infinitely many solutions. en its interior int ˝1 also contains infinitely many solutions by the obvious strict convexity of the ball ˝1 . If there is such a solution u with I .u/ D f1g, then u 2 int ˝1 and u … ˝2 as well as u … ˝3 . is implies that u 2 Œc2 ; c3 , since the opposite ensures the solution uniqueness by Proposition 4.49. Consider the case where the set I .u/ contains two elements for every solution belonging to int ˝1 . en there are infinitely many solutions belonging to the intersection of these two sets, which is strictly convex in this case. Hence there is a solution to (4.37) that belongs to the interior of this

4.5. A GENERALIZED FERMAT-TORRICELLI PROBLEM

163

intersection, which contradicts Proposition 4.50 and thus completes the proof of the theorem.  Let us now consider yet another particular case of problem (4.34), where the constant dynamics F is given by the closed unit ball. It is formulated as minimize D.x/ WD

m X iD1

d.xI ˝i / subject to x 2 ˝0 :

For x 2 Rn , define the index subsets of f1; : : : ; mg by ˇ ˇ ˚ ˚ I.x/ WD i 2 f1; : : : ; mg ˇ x 2 ˝i and J.x/ WD i 2 f1; : : : ; mg ˇ x … ˝i :

(4.41)

(4.42)

It is obvious that I.x/ [ J.x/ D f1; : : : ; mg and I.x/ \ J.x/ D ; for all x 2 Rn . e next theorem fully characterizes optimal solutions to (4.41) via projections to ˝i . eorem 4.52

e following assertions hold for problem (4.41):

(i) Let xN 2 ˝0 be a solution to (4.41). For ai .x/ N  Ai .x/ N as i 2 J.x/ N , we have X X ai .x/ N 2 Ai .x/ N C N.xI N ˝0 /; i2J.x/ N

(4.43)

i2I.x/ N

where each set Ai .x/ N , i D 1; : : : ; m, is calculated by 8 xN ˘.xI N ˝i / ˆ ˆ for xN … ˝i ; < d.xI N ˝i / Ai .x/ N D ˆ ˆ : N.xI N ˝i / \ IB for xN 2 ˝i :

(4.44)

(ii) Conversely, if xN 2 ˝0 satisfies (4.43), then xN is a solution to (4.41). Proof. To verify (i), fix an optimal solution xN to problem (4.41) and equivalently describe it as an optimal solution to the unconstrained optimization problem:

minimize D.x/ C ı.xI ˝/;

x 2 Rn :

(4.45)

Applying the subdifferential Fermat rule to (4.45), we characterize xN by 02@

n X iD1

 d.I ˝i / C ı.I ˝/ .x/: N

(4.46)

164

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

Since all of the functions d.I ˝i / for i D 1; : : : ; n are convex and continuous, we employ the subdifferential sum rule of eorem 2.15 to (4.46) and arrive at X X 02 @d.xI N ˝i / C d.xI N ˝i / C N.xI N ˝0 / D D

i2J.x/ N

X

i2J.x/ N

X

i2J.x/ N

Ai .x/ N C ai .x/ N C

N Xi2I.x/ Ai .x/ N C N.xI N ˝0 /

i2I.x/ N

X

i2I.x/ N

Ai .x/ N C N.xI N ˝0 /;

which justifies the claimed inclusion X X ai .x/ N 2 Ai .x/ N C N.xI N ˝0 /: i2J.x/ N

i2I.x/ N

e converse assertion (ii) is verified similarly by taking into account the “if and only if ” property of the proof of (i).  Let us specifically examine problem (4.41) in the case where the intersections of the constraint set ˝0 with the target sets ˝i are empty. Corollary 4.53

fine the elements

Consider problem (4.41) with ˝0 \ ˝i D ; for i D 1; : : : ; m. Given xN 2 ˝0 , deai .x/ N WD

xN

˘.xI N ˝i / ¤ 0; d.xI N ˝i /

i D 1; : : : ; m:

(4.47)

en xN 2 ˝0 is an optimal solution to (4.41) if and only if m X iD1

ai .x/ N 2 N.xI N ˝0 /:

(4.48)

Proof. In this case we have that xN … ˝i for all i D 1; : : : ; m, which means that I.x/ N D ; and J.x/ N D f1; : : : ; mg. us our statement is a direct consequence of eorem 4.52. 

For more detailed solution of (4.41), consider its special case of m D 3. eorem 4.54 Let m D 3 in the framework of eorem 4.52, where the target sets ˝1 ; ˝2 , and ˝3 are pairwise disjoint, and where ˝0 D Rn . e following hold for the optimal solution xN to N defined by (4.47): (4.41) with the sets ai .x/

(i) If xN belongs to one of the sets ˝i , say ˝1 , then we have ˝ a2 ; a3 i  1=2 and a2 a3 2 N.xI N ˝1 /:

4.5. A GENERALIZED FERMAT-TORRICELLI PROBLEM

165

(ii) If xN does not belong to any of the three sets ˝1 , ˝2 , and ˝3 , then ˚ hai ; aj i D 1=2 for i ¤ j as i; j 2 1; 2; 3 :

Conversely, if xN satisfies either (i) or (ii), then it is an optimal solution to (4.41). Proof. To verify (i), we have from eorem 4.52 that xN 2 ˝1 solves (4.41) if and only if 0 2 @d.xI N ˝1 / C @d.xI N ˝2 / C @d.xI N ˝3 / D @d.xI N ˝1 / C a2 C a3 D N.xI N ˝1 / \ IB C a2 C a3 :

us a2 a3 2 N.xI N ˝1 / \ IB , which is equivalent to the validity of a2 a3 2 N.xI N ˝1 /. By Lemma 4.31, this is also equivalent to ˝ a2 ; a3 i  1=2 and a2 a3 2 N.xI N ˝1 /;

a3 2 IB and a2

which gives (i). To justify (ii), we similarly conclude that in the case where xN … ˝i for i D 1; 2; 3, this element solves (4.41) if and only if 0 2 @d.xI N ˝1 / C @d.xI N ˝2 / C @d.xI N ˝3 / D a1 C a2 C a3 :

en Lemma 4.32 tells us that the latter is equivalent to hai ; aj i D

˚ 1=2 for i ¤ j as i; j 2 1; 2; 3 :

Conversely, the validity of either (i) or (ii) and the convexity of ˝i ensure that 0 2 @d.xI N ˝1 / C @d.xI N ˝2 / C @d.xI N ˝3 / D @D.x/; N

and thus xN is an optimal solution to the problem under consideration.



Figure 4.6: Generalized Fermat-Torricelli problem for three disks.

Example 4.55 Let the sets ˝1 , ˝2 , and ˝3 in problem (4.37) be closed balls in R2 of radius r D 1 centered at the points .0; 2/, . 2; 0/, and .2; 0/, respectively. We can easily see that the point

166

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

.0; 1/ 2 ˝1 satisfies all the conditions in eorem 4.54(i), and hence it is an optimal solution (in fact the unique one) to this problem. More generally, consider problem (4.37) in R2 generated by three pairwise disjoint disks denoted by ˝i , i D 1; 2; 3. Let c1 , c2 , and c3 be the centers of the disks. Assume first that either the line segment Œc2 ; c3  2 R2 intersects ˝1 , or Œc1 ; c3  intersects ˝2 , or Œc1 ; c2  intersects ˝3 . It is not hard to check that any point of the intersections (say, of the sets ˝1 and Œc2 ; c3  for definiteness) is an optimal solution to the problem under consideration, since it satisfies the necessary and sufficient optimality conditions of eorem 4.54(i). Indeed, if xN is such a point, then a2 and a3 are unit vectors with ha2 ; a3 i D 1 and a2 a3 D 0 2 N.xI N ˝1 /. If the above intersection assumptions are violated, we define the three points q1 ; q2 , and q3 as follows. Let u and v be intersection points of Œc1 ; c2  and Œc1 ; c3  with the boundary of the disk centered at c1 , respectively. en we see that there is a unique point q1 on the minor curve generated by u and v such that the measures of angle c1 q1 c2 and c1 q1 c3 are equal. e points q2 and q3 are defined similarly. eorem 4.54 yields that whenever the angle c2 q1 c3 , or c1 q2 c3 , or c2 q3 c1 equals or exceeds 120ı (say, the angle c2 q1 c3 does), then the point xN WD q1 is an optimal solution to the problem under consideration. Indeed, in this case a2 and a3 from (4.47) are unit vectors with ha2 ; a3 i  1=2 and a2 a3 2 N.xI N ˝1 / because the vector a2 a3 is orthogonal to ˝1 . If none of these angles equals or exceeds 120ı , there is a point q not belonging to ˝i as i D 1; 2; 3 such that the angles c1 qc2 D c2 qc3 D c3 qc1 are of 120ı and that q is an optimal solution to the problem under consideration. Observe that in this case the point q is also a unique optimal solution to the classical Fermat-Torricelli problem determined by the points c1 ; c2 , and c3 ; see Figure 4.6.

Finally in this section, we discuss the application of the subgradient algorithm from Section 4.3 to the numerical solution of the generalized Fermat-Torricelli problem (4.34). eorem 4.56 Consider the generalized Fermat-Torricelli problem (4.34) and assume that S ¤ ; for its solution set. Picking a sequence f˛k g of positive numbers and a starting point x1 2 ˝0 ,

form the iterative sequence



xkC1 WD ˘ xk

˛k

m X iD1

 vi k I ˝0 ;

k D 1; 2; : : : ;

(4.49)

with an arbitrary choice of subgradient elements vik 2

@F .!ik

xk / \ N.!i k I ˝i / for some !ik 2 ˘F .xk I ˝i / if xk … ˝i

(4.50)

and vik WD 0 otherwise, where ˘F .xI ˝/ stands for the generalized projection operator (3.21), and where F is the Minkowski gauge (3.11). Define the value sequence ˇ ˚ Vk WD min H.xj / ˇ j D 1; : : : ; k :

4.5. A GENERALIZED FERMAT-TORRICELLI PROBLEM

167

(i) If the sequence f˛k g satisfies conditions (4.13), then fVk g converges to the optimal value V of problem (4.34). Furthermore, we have the estimate P 2 d.x1 I S/2 C `2 1 kD1 ˛k 0  Vk V  ; Pk 2 iD1 ˛k

where `  0 is a Lipschitz constant of cost function H./ on Rn . (ii) If the sequence f˛k g satisfies conditions (4.15), then fVk g converges to the optimal value V and fxk g in (4.49) converges to an optimal solution xN 2 S of problem (4.34). Proof. is is a direct application of the projected subgradient algorithm described in Proposition 4.28 to the constrained optimization problem (4.34) by taking into account the results for the subgradient method in convex optimization presented in Section 4.3. 

Let us specify the above algorithm for the case of the unit ball F D IB in (4.34). Corollary 4.57 Let F D IB  Rn in (4.34), let f˛k g be a sequence of positive numbers, and let x1 2 ˝0 be a starting point. Consider algorithm (4.49) with @F ./ calculated by

@F .!ki

8 ˘.xk I ˝i / xk ˆ ˆ < d.xk I ˝i / xk / D ˆ ˆ : 0;

if xk … ˝i ; if xk 2 ˝i :

en all the conclusions of eorem 4.56 hold for this specification. Proof. Immediately follows from eorem 4.56 with F .x/ D kxk.



Next we illustrate applications of the subgradient algorithm of eorem 4.56 to solving the generalized Fermat-Torricelli problem (4.34) formulated via the minimal time function with non-ball dynamics. Note that to implement the subgradient algorithm (4.49), it is sufficient to calculate just one subgradient from the set on the right-hand side of (4.50) for each target set. In the following example we consider for definiteness the dynamics F given by the square Œ 1; 1  Œ 1; 1 in the plane. In this case the corresponding Minkowski gauge (3.11) is given by the formula ˚ F .x1 ; x2 / D max jx1 j; jx2 j ;

.x1 ; x2 / 2 R2 :

(4.51)

Consider the unconstrained generalized Fermat-Torricelli problem (4.34) with the dynamics F D Œ 1; 1  Œ 1; 1 and the square targets ˝i of right position (with sides parallel to the axes) centered at .ai ; bi / with radii ri (half of the side) as i D 1; : : : ; m. Given a sequence

Example 4.58

168

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

Figure 4.7: Minimal time function with square dynamic

of positive numbers f˛k g and a starting point x1 , construct the iterations xk D .x1k ; x2k / by algorithm (4.49). By eorem 3.39, subgradients vi k can be chosen by 8 .1; 0/ if jx2k bi j  x1k ai and x1k > ai C ri ; ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ . 1; 0/ if jx2k bi j  ai x1k and x1k < ai ri ; ˆ ˆ ˆ ˆ < vik D .0; 1/ if jx1k ai j  x2k bi and x2k > bi C ri ; ˆ ˆ ˆ ˆ ˆ ˆ ˆ .0; 1/ if jx1k ai j  bi x2k and x2k < bi ri ; ˆ ˆ ˆ ˆ ˆ ˆ : .0; 0/ otherwise.

4.6

A GENERALIZED SYLVESTER PROBLEM

In [31] James Joseph Sylvester proposed the smallest enclosing circle problem: Given a finite number of points in the plane, find the smallest circle that encloses all of the points. After more than a century, location problems of this type remain active as they are mathematically beautiful and have meaningful real-world applications. is section is devoted to the study by means of convex analysis of the following generalized version of the Sylvester problem formulated in [18]; see also [19, 20] for further developments. Let F  Rn be a closed, bounded, convex set containing the origin as an interior point. Define the generalized ball centered at x 2 Rn with radius r > 0 by GF .xI r/ WD x C rF:

4.6. A GENERALIZED SYLVESTER PROBLEM

169

It is obvious that for F D IB the generalized ball GF .xI r/ reduces to the closed ball of radius r centered at x . Given now finitely many nonempty, closed, convex sets ˝i for i D 0; : : : ; m, the generalized Sylvester problem is formulated as follows: Find a point x 2 ˝0 and the smallest number r  0 such that GF .xI r/ \ ˝i ¤ ; for all i D 1; : : : ; m:

Observe that when ˝0 D R2 and all the sets ˝i for i D 1; : : : ; m are singletons in R2 , the generalized Sylvester problem becomes the classical Sylvester enclosing circle problem. e sets ˝i for i D 1; : : : ; m are often called the target sets. To study the generalized Sylvester problem, we introduce the cost function ˇ ˚ T .x/ WD max TF .xI ˝i / ˇ i D 1; : : : ; m via the minimal time function (3.9) and consider the following optimization problem: minimize T .x/ subject to x 2 ˝0 :

(4.52)

We can show that the generalized ball GF .xI r/ with xN 2 ˝0 is an optimal solution of the generalized Sylvester problem if and only if xN is an optimal solution of the optimization problem (4.52) with r D T .x/ N . Our first result gives sufficient conditions for the existence of solutions to (4.52). Proposition  Tm 4.59 e generalized Sylvester problem (4.52) admits an optimal solution if the set ˝0 \ ˛F / is bounded for all ˛  0. is holds, in particular, if there exists an index iD1 .˝i i0 2 f0; : : : ; mg such that the set ˝i0 is compact.

Proof. For any ˛  0, consider the level set ˇ ˚ L˛ WD x 2 ˝0 ˇ T .x/  ˛

and then show that in fact we have

L˛ D ˝0 \ Œ

m \

.˝i

˛F /:

(4.53)

iD1

Indeed, it holds for any nonempty, closed, convex set Q that TF .xI Q/  ˛ if and only if .x C ˛F / \ Q ¤ ;, which is equivalent to x 2 Q ˛F . If x 2 L˛ , then x 2 ˝0 and TF .xI ˝i /  ˛ for i D 1; : : : ; m. us x 2 ˝i ˛F for i D 1; : : : ; m, which verifies the inclusion “” in (4.53). e proof of the opposite inclusion is also straightforward. It is clear furthermore that the bound Tm edness of the set ˝0 \ .˝ ˛F / for all ˛  0 ensures that the level sets for the cost i iD1 function in (4.52) are bounded. us it follows from eorem 4.11 that problem (4.52) has an optimal solution. e last statement is obvious. 

170

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

Now we continue with the study of the uniqueness of optimal solutions to (4.52) splitting the proof into several propositions.

Consider a convex function g.x/  0 on a nonempty, convex set ˝  Rn . en the function defined by h.x/ WD .g.x//2 is strictly convex on ˝ if and only if g is not constant on any line segment Œa; b  ˝ with a ¤ b . Proposition 4.60

Proof. Suppose that h is strictly convex on ˝ and suppose on the contrary that g is constant on a line segment Œa; b with a ¤ b , then h is also constant on this line segment, so it cannot be strictly convex. Conversely, suppose g is not constant on any segment Œa; b  ˝ with a ¤ b . Observe that h is a convex function on ˝ since it is a composition of a quadratic function, which is a nondecreasing convex function on Œ0; 1/, and a convex function whose range is a subset of Œ0; 1/. Let us show that it is strictly convex on ˝ . Suppose by contradiction that there are t 2 .0; 1/ and x; y 2 ˝ , x ¤ y , with h.z/ D th.x/ C .1

t /h.y/ with z WD tx C .1

t/y:

en .g.z//2 D t.g.x//2 C .1 t /.g.y//2 . Since g.z/  tg.x/ C .1 t/g.y/, one has 2 2 2 g.z/  t 2 g.x/ C .1 t/2 g.y/ C 2t .1 t /g.x/g.y/; which readily implies that 2 2 2 t g.x/ C .1 t/ g.y/  t 2 g.x/ C .1

t /2 g.y/

2

C 2t .1

t/g.x/g.y/:

us .g.x/ g.y//2  0, so g.x/ D g.y/. is shows that h.x/ D h.z/ D h.y/ for the point z 2 .x; y/ defined above. We arrive at a contradiction by showing that h is constant on the line segment Œz; y. Indeed, fix any u 2 .z; y/ and observe that h.u/  h.z/ C .1

/h.y/ D h.z/ for some  2 .0; 1/:

On the other hand, since z lies between x and u, we have h.z/  h.x/ C .1

/h.u/  h.z/ C .1

/h.z/ D h.z/ for some  2 .0; 1/:

It gives us h.z/ D h.x/ C .1 /h.u/ p D h.z/ C .1 /h.u/, and hence h.u/ D h.z/. is contradicts the assumption that g.u/ D h.u/ is not constant on any line segment Œa; b with a ¤ b and thus completes the proof of the proposition. 

Let both sets F and ˝ in Rn be nonempty, closed, strictly convex, where F contains the origin as an interior point. en the convex function TF .I ˝/ is not constant on any line segment Œa; b  Rn such that a ¤ b and Œa; b \ ˝ D ;. Proposition 4.61

4.6. A GENERALIZED SYLVESTER PROBLEM

171

Proof. To prove the statement of the proposition, suppose on the contrary that there is Œa; b  Rn with a ¤ b such that Œa; b \ ˝ D ; and TF .xI ˝/ D r for all x 2 Œa; b. Choosing u 2 ˘F .aI ˝/ and v 2 ˘F .bI ˝/, we get F .u a/ D F .v b/ D r . Let us verify that u ¤ v . Indeed, the opposite gives us F .u a/ D F .u b/ D r , which implies by r > 0 that u a u b  F D F D 1; r r

and thus

u

a r

2 bd F and

u

1u 2 r

b r a

2 bd F . Since F is strictly convex, it yields

C

1u b  1 D u 2 r r

Hence we arrive at a contradiction by having a C b   TF I ˝  F u 2

a C b 2 int F: 2 a C b < r: 2

To finish the proof of the proposition, fix t 2 .0; 1/ and get   r D TF ta C .1 t/bI ˝  F t u C .1 t/v .ta C .1 t /b/  tF .u a/ C .1 t/F .v b/ D r; which implies that tu C .1 t/v 2 ˘F .ta C .1 t /bI ˝/. us t u C .1 contradicts the strict convexity of ˝ and completes the proof.

t /v 2 bd ˝ , which 

Figure 4.8: Minimal time function for strictly convex sets.

Let ˝  Rn be a nonempty, closed, convex set and let hi W ˝ ! R be nonnegative, continuous, convex functions for i D 1; : : : ; m. Define ˚ .x/ WD max h1 .x/; : : : ; hm .x/ Proposition 4.62

172

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

and suppose that .x/ D r > 0 on Œa; b for some line segment Œa; b  ˝ with a ¤ b . en there exists a line segment Œ˛; ˇ  Œa; b with ˛ ¤ ˇ and an index i0 2 f1; : : : ; mg such that we have hi0 .x/ D r for all x 2 Œ˛; ˇ. Proof. ere is nothing to prove for m D 1. When m D 2, consider the function ˚ .x/ WD max h1 .x/; h2 .x/ :

en the conclusion is obvious if h1 .x/ D r for all x 2 Œa; b. Otherwise we find x0 2 Œa; b with h1 .x0 / < r and a subinterval Œ˛; ˇ  Œa; b, ˛ ¤ ˇ , such that h1 .x/ < r for all x 2 Œ˛; ˇ:

us h2 .x/ D r on this subinterval. To proceed further by induction, we assume that the conclusion holds for a positive integer m  2 and show that the conclusion holds for m C 1 functions. Denote ˚ .x/ WD max h1 .x/; : : : ; hm .x/; hmC1 .x/ : en we have .x/ D maxfh1 .x/; k1 .x/g with k1 .x/ WD maxfh2 .x/; : : : ; hmC1 .x/g, and the conclusion follows from the case of two functions and the induction hypothesis.  Now we are ready to study the uniqueness of an optimal solution to the generalized Sylvester problem (4.52). eorem 4.63 In the setting of problem (4.52) suppose further that the sets F and ˝i for i D 1; : : : ; m are strictly convex. en problem (4.52) has at most one optimal solution if and only if \m iD0 ˝i contains at most one element.

Proof. Define K.x/ WD .T .x//2 and consider the optimization problem

minimize K.x/ subject to x 2 ˝0 :

(4.54)

It is obvious that xN 2 ˝0 is an optimal solution to (4.52) if and only if it is an optimal solution to problem (4.54). We are going to prove that if \m iD0 ˝i contains at most one element, then K is strictly convex on ˝0 , and hence problem (4.52) has at most one optimal solution. Indeed, suppose that K is not strictly convex on ˝0 . By Proposition 4.60, we find a line segment Œa; b  ˝0 with a ¤ b and a number r  0 so that ˇ ˚ T .x/ D max TF .xI ˝i / ˇ i D 1; : : : ; m D r for all x 2 Œa; b: T It is clear that r > 0, since otherwise Œa; b  m iD0 ˝i , which contradicts the assumption that this set contains at most one element. Proposition 4.62 tells us that there exist a line segment Œ˛; ˇ  Œa; b with ˛ ¤ ˇ and an index i0 2 f1; : : : ; mg such that

TF .xI ˝i0 / D r for all x 2 Œ˛; ˇ:

4.6. A GENERALIZED SYLVESTER PROBLEM

173

Since r > 0, we have x … ˝i0 for all x 2 Œ˛; ˇ, which contradicts Proposition 4.61. Conversely, let problem (4.52) have at most one solution. Suppose now that the set \m iD0 ˝i T contains more than one element. It is clear that m ˝ is the solution set of (4.52) with the opi iD0 timal value equal to zero. us this problem has more than one solution, which is a contradiction that completes the proof of the theorem.  e following corollary gives a sufficient condition for the uniqueness of optimal solutions to the generalized Sylvester problem (4.52).

In the setting of (4.52) suppose that the sets F and ˝i for i D 1; : : : ; m are strictly convex. If iD0 ˝i contains at most one element and if one of the sets among ˝i for i D 0; : : : ; m is bounded, then problem (4.52) has a unique optimal solution. Corollary 4.64 Tm

T Proof. Since m iD0 ˝i contains at most one element, problem (4.52) has at most one optimal solution by eorem 4.63. us it has exactly one solution because the solution set is nonempty  by Proposition 4.59.

Next we consider a special unconstrained case of (4.52) given by ˇ ˚ minimize M.x/ WD max d.xI ˝i / ˇ i D 1; : : : ; m subject to x 2 Rn ;

(4.55)

for nonempty, closed, convex sets ˝i under the assumption that \niD1 ˝i D ;. Problem (4.55) corresponds to (4.52) with F D IB  Rn . Denote the set of active indices for (4.55) by ˇ ˚ I.x/ WD i 2 f1; : : : ; mg ˇ M.x/ D d.xI ˝i / ; x 2 Rn : Further, for each x … ˝i , define the singleton ai .x/ WD

x

˘.xI ˝i / d.xI ˝i /

(4.56)

where the projection ˘.xI ˝i / is unique. We have M.x/ > 0 as i 2 I.x/ due to the assumption Tm iD1 ˝i D ;. is ensures that x … ˝i automatically for any i 2 I.x/.

Let us now present necessary and sufficient optimality conditions for (4.55) based on the general results of convex analysis and optimization developed above. e element xN 2 Rn is an optimal solution to problem (4.55) under the assumptions made if and only if we have the inclusion ˇ ˚ xN 2 co ˘.xI N ˝i / ˇ i 2 I.x/ N (4.57)

eorem 4.65

In particular, for the case where ˝i D fai g, i D 1; : : : ; m, problem (4.55) has a unique solution and xN is the problem generated by ai if and only if ˚ ˇ xN 2 co ai ˇ i 2 I.x/ N : (4.58)

174

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

Proof. It follows from Proposition 2.35 and Proposition 2.54 that the condition [ 0 2 co @d.xI N ˝i / i2I.x/ N

is necessary and sufficient for optimality of xN in (4.55). Employing now the distance function subdifferentiation from eorem 2.39 gives us the minimum characterization ˇ ˚ 0 2 co ai .x/ N ˇ i 2 I.x/ N ; (4.59) via the singletons ai .x/ N calculated in (4.56). It is easy to observe that (4.59) holds if and only if P there are i  0 for i 2 I.x/ N such that i2I.x/ N i D 1 and 0D

X

i2I.x/ N

i

xN !N i with !i WD ˘.xI N ˝i /: M.x/ N

is equation is clearly equivalent to X 0D i .xN i2I.x/ N

!N i /; or xN D

X i2I.x/ N

i !N i ;

which in turn is equivalent to inclusion (4.57). Finally, for ˝i D fai g as i D 1; : : : ; m, the latter inclusion obviously reduces to condition (4.58). In this case the problem has a unique optimal solution by Corollary 4.64.  To formulate the following result, we say that the ball IB.xI N r/ touches the set ˝i if the intersection set ˝i \ IB.xI N r/ is a singleton.

Consider problem (4.55) under the assumptions above. en any smallest intersecting ball touches at least two target sets among ˝i , i D 1; : : : ; m.

Corollary 4.66

Proof. Let us first verify that there are at least two indices in I.x/ N provided that xN is a solution to (4.55). Suppose the contrary, i.e., I.x/ N D fi0 g. en it follows from (4.57) that xN 2 ˘.xI N ˝i0 / and d.xI N ˝i0 / D M.x/ N > 0;

a contradiction. Next we show that IB.xI N r/ with r WD M.x/ N touches ˝i when i 2 I.x/ N . Indeed, if on the contrary there are u; v 2 ˝i such that u; v 2 IB.xI N r/ \ ˝i with u ¤ v;

en, by taking into account that d.xI N ˝i / D r , we get ku

xk N  r D d.xI N ˝i / and kv

xk N  r D d.xI N ˝i /;

so u; v 2 ˘.xI N ˝i /. is contradicts the fact that the projector ˘.xI N ˝i / is a singleton and thus completes the proof of the corollary. 

4.6. A GENERALIZED SYLVESTER PROBLEM

175

Now we illustrate the results obtained by solving the following two examples. Let xN solve the smallest enclosing ball problem generated by ai 2 R2 , i D 1; 2; 3. Corollary 4.66 tells us that there are either two or three active indices. If I.x/ N consists of two indices, say I.x/ N D f2; 3g, then xN 2 co fa2 ; a3 g and kxN a2 k D kxN a3 k > kxN a1 k by a2 C a3 and ha2 a1 ; a3 a1 i < 0. If conversely eorem 4.65. In this case we have xN D 2 ha2 a1 ; a3 a1 i < 0, then the angle of the triangle formed by a1 , a2 , and a3 at the vertex a1 is obtuse; we include here the case where all the three points ai are on a straight line. It is a C a  a2 C a3 2 3 easy to see that I D f2; 3g and that the point xN D satisfies the assumption 2 2 of eorem 4.65 thus solving the problem. Observe that in this case the solution of (4.55) is the midpoint of the side opposite to the obtuse vertex. If none of the angles of the triangle formed by a1 ; a2 ; a3 is obtuse, then the active index set I.x/ N consists of three elements. In this case xN is the unique point satisfying

Example 4.67

˚ xN 2 co a1 ; a2 ; a3 and kxN

a1 k D kxN

a2 k D kxN

a3 k:

In the other words, xN is the center of the circumscribing circle of the triangle.

Figure 4.9: Smallest intersecting ball problem for three disks.

Let ˝i D IB.ci ; ri / with ri > 0 and i D 1; 2; 3 be disjoint disks in R2 , and let xN be the unique solution of the corresponding problem (4.55). Consider first the case where one of the line segments connecting two of the centers intersects the other disks, e.g., the line segment connecting c2 and c3 intersects ˝1 . Denote u2 WD c2 c3 \ bd ˝2 and u3 WD c2 c3 \ bd ˝3 , and let xN be the midpoint of u2 u3 . en I.x/ N D f2; 3g and we can apply eorem 4.65 to see that xN is the solution of the problem. Example 4.68

176

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

It remains to consider the case where any line segment connecting two centers of the disks does not intersect the remaining disk. In this case denote u1 WD c1 c2 \ bd ˝1 ; v1 WD c1 c3 \ bd ˝1 ; u2 WD c2 c3 \ bd ˝2 ; v2 WD c2 c1 \ bd ˝2 ; u3 WD c3 c1 \ bd ˝3 ; v3 WD c3 c2 \ bd ˝3 ; a1 WD c1 m1 \ bd ˝1 ; a2 D c2 m2 \ bd ˝2 ; a3 WD c3 m3 \ bd ˝3 ;

where m1 is the midpoint of u2 v3 , m2 is the midpoint of u3 v1 , and m3 is the midpoint of u1 v2 . Suppose that one of the angles u2 a1 v3 , u3 a2 v1 , u1 a3 v2 is greater than or equal to 90ı ; e.g., u2 a1 v3 is greater than 90ı . en I.m1 / D f2; 3g, and thus xN D m1 is the unique solution of this problem by eorem 4.65. If now we suppose that all of the aforementioned angles are less than or equal to 90ı , then I.x/ N D 3 and the smallest disk we are looking for is the unique disk that touches three other disks. e construction of this disk is the celebrated problem of Apollonius.

2

2 2 2

Finally in this section, we present and illustrate the subgradient algorithm to solve numerically the generalized Sylvester problem (4.52). In the setting of problem (4.52) suppose that the solution set S of the problem is nonempty. Fix x1 2 ˝0 and define the sequences of iterates by  xkC1 WD ˘ xk ˛k vk I ˝0 ; k 2 N; eorem 4.69

where f˛k g is a sequence of positive numbers, and where  f0g  vk 2  @F .!k xk / \ N.!k I ˝i /

if xk 2 ˝i ; if xk … ˝i ;

where !k 2 ˘F .xk I ˝i / for some i 2 I.xk /. Define the value sequence ˇ ˚ Vk WD min T .xj / ˇ j D 1; : : : ; k : (i) If the sequence f˛k g satisfies conditions (4.13), then fVk g converges to the optimal value V of the problem. Furthermore, we have the estimate P 2 d.x1 I S/2 C `2 1 kD1 ˛k 0  Vk V  ; P 2 kiD1 ˛k

where `  0 is a Lipschitz constant of cost function T ./ on Rn . (ii) If the sequence f˛k g satisfies conditions (4.15), then fVk g converges to the optimal value V and fxk g in (4.49) converges to an optimal solution xN 2 S of the problem. Proof. Follows from the results of Section 4.3 applied to problem (4.52).



4.6. A GENERALIZED SYLVESTER PROBLEM

177

e next examples illustrate the application of the subgradient algorithm of eorem 4.69 in particular situations.

Example 4.70

Consider problem (4.55) in R2 with m target sets given by the squares

 ˝i D S.!i I ri / WD !1i

  ri ; !1i C ri  !2i

 ri ; !2i C ri ;

i D 1; : : : ; m:

Denote the vertices of the i th square by v1i WD .!1i C ri ; !2i C ri /; v2i WD .!1i ri ; !2i C ri /; v3i WD .!1i ri ; !2i ri /; v4i WD .!1i C ri ; !2i ri /, and let xk WD .x1k ; x2k /. Fix an index i 2 I.xk / and then choose the elements vk from eorem 4.69 by 8 xk v1i ˆ ˆ ˆ kxk v1i k ˆ ˆ ˆ xk v2i ˆ ˆ ˆ ˆ kxk v2i k ˆ ˆ xk v3i ˆ ˆ ˆ ˆ < kxk v3i k xk v4i vk D ˆ ˆ kxk v4i k ˆ ˆ ˆ ˆ .0; 1/ ˆ ˆ ˆ ˆ .0; 1/ ˆ ˆ ˆ ˆ .1; 0/ ˆ ˆ : . 1; 0/

if x1k

!1i > ri and x2k

if x1k

!1i
ri ;

if x1k

!1i
ri ;

!2i
ri ; jx1k !1i j  ri and x2k !2i < ri ; x1k !1i > ri and jx2k !2i j  ri ; x1k !1i < ri and jx2k !2i j  ri :

Figure 4.10: Euclidean projection to a square.

Now we consider the case of a non-Euclidean norm in the minimal time function.

178

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

ˇ ˚ Consider problem (4.52) with F WD .x1 ; x2 / 2 R2 ˇ jx1 j C jx2 j  1 and a nonempty, closed, convex target set ˝ . e generalized ball GF .xI N t/ centered at xN D .xN 1 ; xN 2 / with radius t > 0 has the following diamond shape: ˇ ˚ GF .xI N t/ D .x1 ; x2 / 2 R2 ˇ jx1 xN 1 j C jx2 xN 2 j  t :

Example 4.71

e distance from xN to ˝ and the corresponding projection are given by ˇ ˚ TF .xI N ˝/ D min t  0 ˇ GF .xI N t/ \ ˝ ¤ ; ;

(4.60)

˘F .xI N ˝/ D GF .xI N t / \ ˝ with t D TF .xI N ˝/:

Let ˝i be the squares S.!i I ri /, i D 1; : : : ; m, given in Example 4.70. en problem (4.55) can be interpreted as follows: Find a set of the diamond shape in R2 that intersects all m given squares. Using the same notation as in Example 4.70, we can see that the elements vk in eorem 4.69 are chosen by 8 .1; 1/ if x1k !1i > ri and x2k !2i > ri ; ˆ ˆ ˆ ˆ ˆ . 1; 1/ if x1k !1i < ri and x2k !2i > ri ; ˆ ˆ ˆ ˆ . 1; 1/ if x1k !1i < ri and x2k !2i < ri ; ˆ ˆ < .1; 1/ if x1k !1i > ri and x2k !2i < ri ; vk D .0; 1/ if jx1k !1i j  ri and x2k !2i > ri ; ˆ ˆ ˆ ˆ ˆ .0; 1/ if jx1k !1i j  ri and x2k !2i < ri ; ˆ ˆ ˆ ˆ .1; 0/ if x1k !1i > ri and jx2k !2i j  ri ; ˆ ˆ : . 1; 0/ if x1k !1i < ri and jx2k !2i j  ri :

4.7

EXERCISES FOR CHAPTER 4

Exercise 4.1 Given a lower semicontinuous function f W Rn ! R, show that its multiplication .˛f / 7! ˛f .x/ by any scalar ˛ > 0 is lower semicontinuous as well.

Exercise 4.2 Given a function f W Rn ! R, show that its lower semicontinuity is equivalent to openess, for every  2 R, of the set

ˇ ˚ U WD x 2 Rn ˇ f .x/ >  :

Exercise 4.3 Give an example of two lower semicontinuous real-valued functions whose product is not lower semicontinuous.

4.7. EXERCISES FOR CHAPTER 4

179

Figure 4.11: Minimal time function with diamond-shape dynamic.

Exercise 4.4 Let f W Rn ! R. Prove the following assertions: (i) f is upper semicontinuous (u.s.c.) at xN in the sense of Definition 2.89 if and only if f is lower semicontinuous at this point. (ii) f is u.s.c. at xN if and only if lim supk!1 f .xk /  f .x/ N for every xk ! xN . (iii) f is u.s.c. at xN if and only if we have lim supx!xN f .x/  f .x/ N . n n (iv) f is u.s.c. on R if and only if the upper level set fx 2 R j f .x/  g is closed for any real number . (v) f is u.s.c. on Rn if and only if its hypergraphical set (or simply ) hypo f WD f.x; / 2 Rn  R j   f .x/g is closed.

Exercise 4.5 Show that a function f W Rn ! R is continuous at xN if and only if it is both lower semicontinuous and upper semicontinuous at this point.

Exercise 4.6 Prove the equivalence (iii)”(iv) in eorem 4.11. Exercise 4.7 Consider the real-valued function g.x/ WD

m X iD1

kx

ai k2 ; x 2 Rn ;

where ai 2 Rn for i D 1; : : : ; m. Show that g is a convex function and has an absolute minimum a1 C a2 C : : : C am at the mean point xN D . m

180

4. APPLICATIONS TO OPTIMIZATION AND LOCATION PROBLEMS

Exercise 4.8 Consider the problem: minimize kxk2 subject to Ax D b; where A is an p  n matrix with rank A D p , and where b 2 Rp . Find the unique optimal solution of this problem.

Exercise 4.9 Give a direct proof of Corollary 4.15 without using eorem 4.14. Exercise 4.10 Given ai 2 Rn for i D 1; : : : ; m, write MATLAB programs to implement the

subgradient algorithm for minimizing the function f given by: P (i) f .x/ WD m kx ai k; x 2 Rn . PiD1 m (ii) f .x/ WD iD1 kx ai k1 ; x 2 Rn . P (iii) f .x/ WD m ai k1 ; x 2 Rn . iD1 kx

Exercise 4.11 Write a MATLAB program to implement the projected subgradient algorithm for solving the problem: minimize kxk1 subject to Ax D b; where A is an p  n matrix with rank A D p , and where b 2 Rp .

Exercise 4.12 Consider the optimization problem (4.9) and the iterative procedure (4.10) under the standing assumptions imposed in Section 4.3. Suppose that the optimal value V is known and that the sequence f˛k g is given by 8 ˆ if 0 2 @f .xk /; 0g and ˝2 WD f.x; y/ 2 R2 j y  1=x; x > 0g . en ˝1 and ˝2 are nonempty, closed, convex sets while ˝1 ˝2 D f.x; y/ 2 R2 j y > 0 is not closed. Exercise 1.4. (i) Use Exercise 1.3(ii). (ii) We only consider the case where m D 2 since the general case is similar. From the definition of cones, both sets ˝1 and ˝2 contain 0, and so ˝1 [ ˝2  ˝1 C ˝2 . en co f˝1 [ ˝2 g  ˝1 C ˝2 . To verify the opposite inclusion, fix any x D w1 C w2 2 ˝1 C ˝2 with w1 2 ˝1 and w2 2 ˝2 and then write x D Œ.2w1 C 2w2 /=2 2 co f˝1 [ ˝2 g. Exercise 1.7. Compute the first or second derivative of each function. en apply either eorem 1.45 or Corollary 1.46. Exercise 1.8. (i) Use f .x; y/ WD xy; .x; y/ 2 R2 . (ii) Take f1 .x/ WD x and f2 .x/ WD x . Exercise 1.9. Let f1 .x/ WD x and f2 .x/ WD x 2 , x 2 R. Exercise 1.12. Use Proposition 1.39. Note that the function f .x/ WD x q is convex on Œ0; 1/. ! Rp by F .x/ WD K , x 2 Rn . en gph F is a convex subset of Exercise 1.16. Define F W Rn ! Rn  Rp . e conclusion follows from Proposition 1.50.

184

SOLUTIONS AND HINTS FOR EXERCISES

Exercise 1.18. (i) gph F D C . (ii) Consider the function f .x/ WD

( p 1 1

x2

if jxj  1; otherwise:

(iii) Apply Proposition 1.50. Exercise 1.20. (i) Use the definition. (ii) Take f .x/ WD Exercise 1.22. Suppose that v C Pm i D 1, and so iD1  vD

Pm

iD1

m X i iD1



p

jxj.

i vi D 0 with  C

Pm

iD1

i D 0. If  ¤ 0, then

˚ vi 2 aff v1 ; : : : ; vm ;

which is a contradiction. us  D 0, and hence i D 0 for i D 1; : : : ; m because the elements v1 ; : : : ; vm are affinely independent. Exercise 1.23. Since m  ˝  aff ˝ , we have aff m  aff ˝ . To prove the opposite inclusion, it remains to show that ˝  aff m . By contradiction, suppose that there are v 2 ˝ such that v … aff m . It follows from Exercise 1.22 that the vectors v; v1 ; : : : ; vm are affinely independent and all of them belong to ˝ . is contradicts the condition dim ˝ D m. e second equality can also be proved similarly. Exercise 1.24. (i) We obviously have aff ˝  aff ˝ . Observe that the set aff ˝ is closed and contains ˝ since aff ˝ D w0 C L, where w0 2 ˝ and L is the linear subspace parallel to aff ˝ , which is a closed set. us ˝  aff ˝ and the conclusion follows. (ii) Choose ı > 0 so that IB.xI N ı/ \ aff ˝  ˝ . Since xN C t.xN x/ D .1 C t/xN C . t/x is an affine combination of some elements in aff ˝ D aff ˝ , we have xN C t .xN x/ 2 aff ˝ . us u WD xN C t.xN x/ 2 ˝ when t is small. (iii) It follows from (i) that ri ˝  ri ˝ . Fix any x 2 ri ˝ and xN 2 ri ˝ . By (ii) we have z WD x C t.x x/ N 2 ˝ when t > 0 is small, and so x D z=.1 C t / C t x=.1 N C t / 2 .z; x/ N  ri ˝ . Exercise 1.26. If ˝ 1 D ˝ 2 , then ri ˝ 1 D ri ˝ 2 , and so ri ˝1 D ri ˝2 by Exercise 1.24(iii). Exercise 1.27. (i) Fix y 2 B.ri ˝/ and find x 2 ri ˝ such that y D B.x/. en pick a vector yN 2 ri B.˝/  B.˝/ and take xN 2 ˝ with yN D B.x/ N . If x D xN , then y D yN 2 ri B.˝/. Consider the case where x ¤ xN . Following the solution of Exercise 1.24(iii), find e x 2 ˝ such that x 2 .e x ; x/ N and let e y WD B.e x / 2 B.˝/. us y D B.x/ 2 .B.e x /; B.x// N D .e y ; y/ N , and so y 2 ri B.˝/. To complete the proof, it remains to show that B.˝/ D B.ri ˝/ and then use Exercise 1.26 to obtain

SOLUTIONS AND HINTS FOR EXERCISES

185

the inclusion ri B.˝/ D ri B.ri ˝/  B.ri ˝/: Since B.ri ˝/  B.˝/, the continuity of B and Proposition 1.73 imply that B.˝/  B.˝/ D B.ri ˝/  B.ri ˝/:

us we have B.˝/  B.ri ˝/ and complete the proof. (ii) Consider B.x; y/ WD x y defined on Rn  Rn . Exercise 1.28. (i) Let us show that

aff .epi f / D aff .dom f /  R :

(4.62)

Fix any .x; / 2 aff .epi f /. en there exist i 2 R and .xi ; i / 2 epi f for i D 1; : : : ; m with Pm iD1 i D 1 such that m X .x; / D i .xi ; i /: iD1

Since xi 2 dom f for i D 1; : : : ; m, we have x D

Pm

iD1

i xi 2 aff .dom f /, which yields

aff .epi f /  aff .dom f /  R : To verify the opposite inclusion, fix any .x; / 2 aff .dom f /  R and find xi 2 dom f and i 2 P Pm R for i D 1; : : : ; m with m iD1 i D 1 and x D iD1 i xi . Define further ˛i WD f .xi / 2 R and Pm ˛ WD iD1 i ˛i for which .xi ; ˛i / 2 epi f and .xi ; ˛i C 1/ 2 epi f . us m X iD1 m X iD1

Letting  WD ˛

i .xi ; ˛i / D .x; ˛/ 2 aff .epi f /; i .xi ; ˛i C 1/ D .x; ˛ C 1/ 2 aff .epi f /:

C 1 gives us .x; / D .x; ˛/ C .1

/.x; ˛ C 1/ 2 aff .epi f /;

which justifies the opposite inclusion in (4.62). (ii) See the proof of [9, Proposition 1.1.9]. ha; xi a. kak2 (ii) ˘.xI ˝/ D x C AT .AAT / 1 .b Ax/. (iii) ˘.xI ˝/ D maxfx; 0g (componentwise maximum).

Exercise 1.30. (i) ˘.xI ˝/ D x C

b

186

SOLUTIONS AND HINTS FOR EXERCISES

EXERCISES FOR CHAPTER 2 Exercise 2.1. (i) Apply Proposition 2.1 and the fact that tx 2 ˝ for all t  0. (ii) Use (i) and observe that x 2 ˝ whenever x 2 ˝ . Exercise 2.2. Observe that the sets ri ˝i for i D 1; 2 are nonempty and convex. By eorem 2.5, there is 0 ¤ v 2 Rn such that hv; xi  hv; yi for all x 2 ri ˝1 ; y 2 ri ˝2 :

Fix x0 2 ri ˝1 and y0 2 ri ˝2 . It follows from eorem 1.72(ii) that for any x 2 ˝1 , y 2 ˝2 , and t 2 .0; 1/ we have that x C t.x0 x/ 2 ri ˝1 and y C t.y0 y/ 2 ri ˝2 . en apply the inequality above for these elements and let t ! 0C . Exercise 2.3. We split the solution into four steps.

Step 1: If L is a subspace of Rn that contains a nonempty, convex set ˝ with xN … ˝ and xN 2 L, then there is 0 ¤ v 2 L such that ˇ ˚ N (4.63) sup hv; xi ˇ x 2 ˝ < hv; xi: Proof. By Proposition 2.1, find w 2 Rn such that ˇ ˚ N sup hw; xi ˇ x 2 ˝ < hw; xi:

(4.64)

Represent Rn as L ˚ L? , where L? WD fu 2 Rn j hu; xi D 0 for all x 2 Lg. en w D u C v with u 2 L? and v 2 L. Substituting w into (4.64) and taking into account that hu; xi D 0 for all x 2 ˝ and that hu; xi N D 0, we get (4.63). Note that (4.63) yields v ¤ 0.

Step 2: Let ˝  Rn be a nonempty, convex set such that 0 … ri ˝ . en the sets ˝ and f0g can be separated properly. Proof. In the case where 0 … ˝ , we can separate ˝ and f0g strictly, so the conclusion is obvious. Suppose now that 0 2 ˝ n ri ˝ . Let L WD aff ˝ D span ˝ , where the latter holds due to 0 2 aff ˝ , which follows from the fact that ˝  L since L is closed. Repeating the proof of Lemma 2.4, find a sequence fxk g  L with xk ! 0 and xk … ˝ for every k . It follows from Step 1 that there is fvk g  L with vk ¤ 0 and ˇ ˚ sup hvk ; xi ˇ x 2 ˝ < 0: Dividing both sides by kvk k, we suppose without loss of generality that kvk k D 1 and that vk ! v 2 L with kvk D 1. en ˇ ˚ sup hv; xi ˇ x 2 ˝  0: To finish this proof, it remains to show that there exists x 2 ˝ with hv; xi < 0. By contradiction, suppose that hv; xi  0 for all x 2 ˝ . en hv; xi D 0 on ˝ . Since v 2 L D aff ˝ , we can repP Pm resent v as v D m iD1 i wi , where iD1 i D 1 and wi 2 ˝ for i D 1; : : : ; m. is implies that

2

kvk D hv; vi D

Pm

iD1

SOLUTIONS AND HINTS FOR EXERCISES

187

i hv; wi i D 0, which is a contradiction.

Step 3: Let ˝  Rn be a nonempty, convex set. If the sets ˝ and f0g can be separated properly, then we have 0 … ri ˝ . Proof. By definition, find 0 ¤ v 2 Rn such that ˇ ˚ sup hv; xi ˇ x 2 ˝  0

and then take xN 2 ˝ with hv; xi N < 0. Arguing by contradiction, suppose that 0 2 ri ˝ . en Exercise 1.24(ii) gives us t > 0 such that 0 C t.0 x/ N D t xN 2 ˝ . us hv; t xi N  0, which yields hv; xi N  0, a contradiction.

Step 4: Justifying the statement of Exercise 2.3. Proof. Define ˝ WD ˝1

˝2 . By Exercise 1.27(ii) we have ri ˝1 \ ri ˝2 D ; if and only if 0 … ri .˝1

˝2 / D ri ˝1

ri ˝2 :

en the conclusion follows from Step 2 and Step 3. Exercise 2.4. Observe that ˝1 \ Œ˝2

.0; / D ; for any  > 0.

Exercise 2.5. Employing eorem 2.5 and arguments similar to those in Exercise 2.2, we can show that there is 0 ¤ v 2 Rn such that hv; xi  hv; yi for all x 2 ˝1 ; y 2 ˝2 :

By Proposition 2.13, choose 0 ¤ v 2 N.xI N ˝1 / \ Œ N.xI N ˝2 / and, using the normal cone definition, arrive at the conclusion with v1 WD v , v2 WD v , ˛1 WD hv1 ; xi N , and ˛2 WD hv2 ; xi: N Exercise 2.6. Fix xN  0 and v 2 N.xI N ˝/. en hv; x

xi N  0 for all x 2 ˝:

Using this inequality for 0 2 ˝ and 2xN 2 ˝ , we get hv; xi N D 0. Since ˝ D . 1; 0  : : :  . 1; 0, it follows from Proposition 2.11 that v  0. e opposite inclusion is a direct consequence of the definition. Exercise 2.8. Suppose that this holds for any k convex sets with k  m and m  2. Consider the sets ˝i for i D 1; : : : ; m C 1 satisfying the qualification condition h mC1 X iD1

i   vi D 0; vi 2 N.xI N ˝i / H) vi D 0 for all i D 1; : : : ; m C 1 :

188

SOLUTIONS AND HINTS FOR EXERCISES

Since 0 2 N.xI N ˝mC1 /, this condition also holds for the first m sets. us m m  \  X N xI N ˝i D N.xI N ˝i /: iD1

en apply the induction hypothesis to 1 WD

iD1

\m iD1 ˝i

and 2 WD ˝mC1 .

Exercise 2.10. Since v 2 @f .x/ N , we have v.x

x/ N  f .x/

f .x/ N for all x 2 R :

en pick any x1 < xN and use the inequality above. Exercise 2.12. Define the function g.x/ WD f .x/ C ıfxg N .x/ D

( f .x/ N 1

if x D x; N

otherwise

via the indicator function of fxg N . en epi g D fxg N  Œf .x/; N 1/, and hence N..x; N g.x//I N epi g/ D n n n N D R and @ıfxg N D R . Employing R . 1; 0. It follows from Proposition 2.31 that @g.x/ N .x/ the subdifferential sum rule from Corollary 2.45 gives us Rn D @g.x/ N D @f .x/ N C Rn ;

which ensures that @f .x/ N ¤ ;. Exercise 2.13. We proceed step by step as in the proof of eorem 2.15. It follows from Exercise 1.28 that ri 1 D ri ˝1  .0; 1/ and ˇ ˚ N : ri 2 D .x; / ˇ x 2 ri ˝1 ;  < hv; x xi

It is easy to check, arguing by contradiction, that ri 1 \ ri 2 D ;. Applying the separation result from Exercise 2.3, we find 0 ¤ .w; / 2 Rn  R such that hw; xi C 1  hw; yi C 2 for all .x; 1 / 2 1 ; .y; 2 / 2 2 :

Moreover, there are .e x; e 1 / 2 1 and .e y; e 2 / 2 2 satisfying hw;e xi C e 1 < hw; e yi C e 2 :

As in the proof of eorem 2.15, it remains to verify that < 0. By contradiction, assume that

D 0 and then get hw; xi  hw; yi for all x 2 ˝1 ; y 2 ˝2 ; hw;e x i < hw; e y i with e x 2 ˝1 ; e y 2 ˝2 :

SOLUTIONS AND HINTS FOR EXERCISES

189

us the sets ˝1 and ˝2 are properly separated, and hence it follows from Exercise 2.3 that ri ˝1 \ ri ˝2 D ;. is is a contradiction, which shows that < 0. To complete the proof, we only need to repeat the proof of Case 2 in eorem 2.15. Exercise 2.14. Use Exercise 2.13 and proceed as in the proof of eorem 2.44. Exercise 2.16. Use the same procedure as in the proof of eorem 2.51 and the construction of the singular subdifferential to obtain ˇ  ˚ @1 .f ı B/.x/ N D A @1 f .y/ N D A v ˇ v 2 @1 f .y/ N : (4.65)

Alternatively, observe the equality dom .f ı B/ D B and Corollary 2.53 to complete the proof.

1

.dom f /. en apply Proposition 2.23

Exercise 2.17. Use the subdifferential formula from eorem 2.61 and then the representation of the composition from Proposition 1.54. Exercise 2.19. For any v 2 Rn , we have ˝1 .v/ D kvk1 and ˝2 .v/ D kvk1 . Exercise 2.21. Use first the representations ˇ ˇ ˚ ˚ .ı˝ / .v/ D sup hv; xi ı.xI ˝/ ˇ x 2 Rn D sup hv; xi ˇ x 2 ˝ D ˝ .v/;

v 2 Rn :

Applying then the Fenchel conjugate one more time gives us ˇ ˚ .˝ / .x/ D sup hx; vi ˝ .v/ ˇ v 2 Rn D ıco ˝ .x/: Exercise 2.23. Since f W Rn ! R is convex and positively homogeneous, we have @f .0/ ¤ ; and f .0/ D 0. is implies in the case where v 2 ˝ D @f .0/ that hv; xi  f .x/ for all x 2 Rn , and so the equality holds when x D 0. us we have ˇ ˚ f  .v/ D sup hv; xi f .x/ ˇ x 2 Rn D 0:

If v … ˝ D f .0/, there is xN 2 Rn satisfying hv; xi N > f .x/ N . It gives us by homogeneity that ˇ ˇ ˚ ˚ f  .v/ D sup hv; xi f .x/ ˇ x 2 Rn  sup hv; t xi N f .t x/ N ˇ t > 0 D 1: Exercise 2.26. f 0 . 1I d / D 1 and f 0 .1I d / D 1. Exercise 2.27. Write xN C t2 d D

t2 .xN C t1 d / C .1 t1

t2 /xN for fixed t1 < t2 < 0: t1

190

SOLUTIONS AND HINTS FOR EXERCISES

Exercise 2.28. (i) For d 2 Rn , consider the function ' defined in Lemma 2.80. en show that the inequalities  < 0 < t imply that './  '.t /. (ii) It follows from (i) that '.t /  '.1/ for all t 2 .0; 1/. Similarly, '. 1/  '.t / for all t 2 . 1; 0/. is easily yields the conclusion. Exercise 2.31. Fix xN 2 Rn , define P .xI N ˝/ WD fw 2 ˝ j kxN wk D ˝ .x/g N , and consider the following function fw .x/ WD kx wk for w 2 ˝ and x 2 Rn :

en we get from eorem 2.93 that ˇ ˚ @˝ .x/ N D co @fw .x/ N ˇ w 2 P .xI N ˝/ :

In the case where ˝ is not a singleton we have @fw .x/ N D

co P .xI N ˝/ : ˝ .x/ N

xN

@˝ .x/ N D

xN w , and so ˝ .x/ N

e case where ˝ is a singleton is trivial. Exercise 2.33. (i) Fix any v 2 Rn with kvk  1 and any w 2 ˝ . en hx; vi

˝ .v/  hx; vi

It follows therefore that

hw; vi D hx

˚ sup hx; vi

w; vi  kx

wk  kvk  kx

wk:

˝ .v/  d.xI ˝/:

kvk1

To verify the opposite inequality, observe first that it holds trivially if x 2 ˝ . In the case where x … ˝ , let w WD ˘.xI ˝/ and u WD x w ; thus kuk D d.xI ˝/. Following the proof of Proposition 2.1, we obtain the estimate kuk2 for all z 2 ˝:

hu; zi  hu; xi

Divide both sides by kuk and let v WD u=kuk. en ˝ .v/  hv; xi ˚ d.xI ˝/  sup hx; vi ˝ .v/ :

d.xI ˝/, and so

kvk1

EXERCISES FOR CHAPTER 3 Exercise 3.1. Suppose that f is Gâteaux differentiable at xN with fG0 .x/ N D v . en hfG0 .x/; N vi D lim

t!0

f .xN C td / t

f .x/ N

D lim

t!0C

f .xN C td / t

f .x/ N

for all d 2 Rn :

SOLUTIONS AND HINTS FOR EXERCISES 0

191 n

is implies by the definition of the directional derivative that f .xI N d / D hv; d i for all d 2 R . n 0 Conversely, suppose that f .xI N d / D hv; d i for all d 2 R . en f 0 .xI N d / D hv; d i D lim

t !0C

f .xN C t d / t

f .x/ N

for all d 2 Rn :

N d / D f 0 .xI N d / D hv; d i D hv; d i for all d 2 Rn . us Moreover, f 0 .xI f 0 .xI N d / D hv; d i D lim t !0

f .xN C t d / t

f .x/ N

for all d 2 Rn :

is implies that f is Gâteaux differentiable at xN with fG0 .x/ N D v. Exercise 3.2. (i) Since the function '.t / WD t 2 is nondecreasing on Œ0; 1/, following the proof N D of Corollary 2.62, we can show that if g W Rn ! Œ0; 1/ is a convex function, then @.g 2 /.x/ 2g.x/@g. N x/ N for every xN 2 Rn , where g 2 .x/ WD Œg.x/2 . It follows from eorem 2.39 that ( f0g if xN 2 ˝; @f .x/ N D 2d.xI N ˝/@d.xI N ˝/ D ˚ 2ŒxN ˘.xI N ˝/ otherwise:

˚ us in any case @f .x/ N D 2ŒxN ˘.xI N ˝/ is a singleton, which implies the differentiability of f at xN by eorem 3.3 and rf .x/ N D 2ŒxN ˘.xI N ˝/. (ii) Use eorem 2.39 in the out-of-set case and eorem 3.3. 1 Exercise 3.3. For every u 2 F , the function fu .x/ WD hAx; ui kuk2 is convex, and hence f 2 is a convex function by Proposition 1.43. It follows from the equality hAx; ui D hx; A ui that rfu .x/ D A u:

1 kuk2 has a unique maximizer on 2 F denoted by u.x/. By eorem 2.93 we have @f .x/ D fA u.x/g, and thus f is differentiable at x by eorem 3.3. We refer the reader to [22] for more general results and their applications to optimization.

Fix x 2 Rn and observe that the function gx .u/ WD hAx; ui

Exercise 3.5. Consider the set Y WD fd 2 Rn j f 0 .xI N d / D f 0 .xI N d /g and verify that Y is a subspace containing the standard orthogonal basis fei j i D 1; : : : ; ng. us V D Rn , which implies the differentiability of f at xN . Indeed, fix any v 2 @f .x/ N and get by eorem 2.84(iii) n 0 that f .xI N d / D hv; d i for all d 2 R . Hence @f .x/ N is a singleton, which ensures that f is differentiable at xN by eorem 3.3. Exercise 3.6. Use Corollary 2.37 and follow the proof of eorem 2.40.

192

SOLUTIONS AND HINTS FOR EXERCISES

Exercise 3.7. An appropriate example is given by the function f .x/ WD x 4 on R. Exercise 3.10. Fix any sequence fxk g  co ˝ . By the Carathéodory theorem, find k;i  0 and wk;i 2 ˝ for i D 0; : : : ; n with xk D

n X

k;i wk;i ;

iD0

n X iD0

k;i D 1 for every k 2 N :

en use the boundedness of fk;i gk for i D 0; : : : ; n and the compactness of ˝ to extract a subsequence of fxk g converging to some element of co ˝ . Exercise 3.16. By Corollary 2.18 we have N.xI N ˝1 \ ˝2 / D N.xI N ˝1 / C N.xI N ˝2 /. Applying eorem 3.16 tells us that  ı T .xI N ˝1 \ ˝2 / D N.xI N ˝1 \ ˝2 /  ı  ı  ı D N.xI N ˝1 / C N.xI N ˝2 / D N.xI N ˝1 / \ N.xI N ˝2 / D T .xI N ˝1 / \ T .xI N ˝2 /: Exercise 3.18. Consider the set ˝ WD f.x; y/ 2 R2 j y  x 2 g.

Exercise 3.19. Fix any a; b 2 Rn with a ¤ b . It follows from eorem 3.20 that there exists c 2 .a; b/ such that f .b/

en jf .b/

f .a/j  kvk  kb

f .a/ D hv; b ak  `kb

ai for some c 2 @f .c/: ak.

Exercise 3.21. (i) F1 D f0g  RC . (ii) F1 D F . (iii) F1 D RC  RC . (iv) F1 D f.x; y/ j y  jxjg. Exercise 3.23. (i) Apply the definition. (ii) A condition is A1 \ . B1 / D f0g. Exercise 3.24. Suppose that T˝F .x/ D 0. en there exists a sequence of tk ! 0, tk  0, with .x C tk F / \ ˝ ¤ ; for every k 2 N . Choose fk 2 F and wk 2 ˝ such that x C tk fk D wk . Since F is bounded, the sequence fwk g converges to x , and so x 2 ˝ since ˝ is closed. e converse implication is trivial. Exercise 3.25. Follow the proof of eorem 3.37 and eorem 3.38.

EXERCISES FOR CHAPTER 4 Exercise 4.2. e set U D Lc is closed by Proposition 4.4.

SOLUTIONS AND HINTS FOR EXERCISES

193

Exercise 4.3. Consider the l.s.c. function f .x/ WD 0 for x ¤ 0 and f .0/ WD 1. en the product of f  f is not l.s.c. Exercise 4.7. Solve the equation rg.x/ D 0, where rg.x/ D

Pm

iD1

2.x

ai /.

Exercise 4.8. Let ˝ WD fx 2 Rn j Ax D bg. en N.xI ˝/ D fAT  j  2 Rp g for x 2 ˝ . Corollary 4.15 tells us that x 2 ˝ is an optimal solution to this problem if and only if there exists  2 Rp such that 2x D AT  and Ax D b:

Solving this system of equations yields x D AT .AAT / 1 b .

Exercise 4.11. e MATLAB function sign x gives us a subgradient of the function f .x/ WD kxk1 for all x 2 R. Considering the set ˝ WD fx 2 Rn j Ax D b , we have ˘.xI ˝/ D x C AT .AAT / 1 .b Ax/, which can be found by minimizing the quadratic function kx uk2 over u 2 ˝ ; see Exercise 4.8. Exercise 4.12. If 0 2 @f .xk0 / for some k0 2 N , then ˛k0 D 0, and hence xk D xk0 2 S and f .xk / D V for all k  k0 . us we assume without loss of generality that 0 … @f .xk / as k 2 N . It follows from the proof of Proposition 4.23 that 2

0  kxkC1

xk N  kx1

xk N

2

2

k X

˛i Œf .xi /

iD1

VC

k X iD1

˛i2 kvi k2

for any xN 2 S . Substituting here the formula for ˛k gives us 0  kxkC1

xk N 2  kx1

xk N 2

D kx1

xk N 2

k X Œf .xi / iD1

2

k X Œf .xi /

iD1 k X iD1

V 2

kvi k2

V 2

kvi k2

C

Œf .xi / V 2 kvi k2  kx1

xk N 2:

 kx1

xk N 2;

k X Œf .xi / iD1

V 2

kvi k2

and

Since kvi k  ` for every i , we get k X Œf .xi / iD1

`2

V 2

which implies in turn that k X iD1

Œf .xi /

V 2  `2 kx1

xk N 2:

(4.66)

194

SOLUTIONS AND HINTS FOR EXERCISES

P en the series 1 V 2 is convergent. Hence f .xk / ! V as k ! 1, which justifies kD1 Œf .xk / (i). From (4.66) and the fact that Vk  f .xj / for i 2 f1; : : : ; kg it follows that k.Vk

us 0  Vk

V 

V /2  `2 kx1

xk N 2:

`kx1 xk N . Since xN was taken arbitrarily in S , we arrive at (ii). p k

Exercise 4.17. Define the MATLAB function Pm ai iD1 kx ai k f .x/ WD ; Pm 1 iD1 kx ai k

x … fa1 ; : : : ; am g

and f .x/ WD x for x 2 fa1 ; : : : ; am g. Use the FOR loop with a stopping criteria to find xkC1 WD f .xk /, where x0 is a starting point. Exercise 4.18. Use the subdifferential Fermat rule. Consider the cases where m is odd and where m is even. In the first case the problem has a unique optimal solution. In the second case the problem has infinitely many solutions. Exercise 4.22. For the interval ˝ WD Œa; b  R, the subdifferential formula for the distance function 2.39 gives us 8 ˆ f0g if xN 2 .a; b/; ˆ ˆ ˆ ˆ ˆ if xN D b; ˆ b; ˆ ˆ ˆ ˆ Œ 1; 0 if xN D a; ˆ ˆ ˆ :f 1g if xN < b: Exercise 4.23. For ˝ WD IB.cI r/  Rn , a subgradient of the function d.xI ˝/ at xN is 8