Regularization Theory for Ill-posed Problems: Selected Topics 9783110286496, 9783110286465

Table of contents :
Preface
1 An introduction using classical examples
1.1 Numerical differentiation. First look at the problem of regularization. The balancing principle
1.1.1 Finite-difference formulae
1.1.2 Finite-difference formulae for nonexact data. A priori choice of the stepsize
1.1.3 A posteriori choice of the stepsize
1.1.4 Numerical illustration
1.1.5 The balancing principle in a general framework
1.2 Stable summation of orthogonal series with noisy coefficients. Deterministic and stochastic noise models. Description of smoothness properties
1.2.1 Summation methods
1.2.2 Deterministic noise model
1.2.3 Stochastic noise model
1.2.4 Smoothness associated with a basis
1.2.5 Approximation and stability properties of -methods
1.2.6 Error bounds
1.3 The elliptic Cauchy problem and regularization by discretization
1.3.1 Natural linearization of the elliptic Cauchy problem
1.3.2 Regularization by discretization
1.3.3 Application in detecting corrosion
2 Basics of single parameter regularization schemes
2.1 Simple example for motivation
2.2 Essentially ill-posed linear operator equations. Least-squares solution. General view on regularization
2.3 Smoothness in the context of the problem. Benchmark accuracy levels for deterministic and stochastic data noise models
2.3.1 The best possible accuracy for the deterministic noise model
2.3.2 The best possible accuracy for the Gaussian white noise model
2.4 Optimal order and the saturation of regularization methods in Hilbert spaces
2.5 Changing the penalty term for variance reduction. Regularization in Hilbert scales
2.6 Estimation of linear functionals from indirect noisy observations
2.7 Regularization by finite-dimensional approximation
2.8 Model selection based on indirect observation in Gaussian white noise
2.8.1 Linear models given by least-squares methods
2.8.2 Operator monotone functions
2.8.3 The problem of model selection (continuation)
2.9 A warning example: an operator equation formulation is not always adequate (numerical differentiation revisited)
2.9.1 Numerical differentiation in variable Hilbert scales associated with designs
2.9.2 Error bounds in L2
2.9.3 Adaptation to the unknown bound of the approximation error
2.9.4 Numerical differentiation in the space of continuous functions
2.9.5 Relation to the Savitzky–Golay method. Numerical examples
3 Multiparameter regularization
3.1 When do we really need multiparameter regularization?
3.2 Multiparameter discrepancy principle
3.2.1 Model function based on the multiparameter discrepancy principle
3.2.2 A use of the model function to approximate one set of parameters satisfying the discrepancy principle
3.2.3 Properties of the model function approximation
3.2.4 Discrepancy curve and the convergence analysis
3.2.5 Heuristic algorithm for the model function approximation of the multiparameter discrepancy principle
3.2.6 Generalization in the case of more than two regularization parameters
3.3 Numerical realization and testing
3.3.1 Numerical examples and comparison
3.3.2 Two-parameter discrepancy curve
3.3.3 A numerical check of Proposition 3.1 and use of a discrepancy curve
3.3.4 Experiments with three-parameter regularization
3.4 Two-parameter regularization with one negative parameter for problems with noisy operators and right-hand side
3.4.1 Computational aspects for regularized total least squares
3.4.2 Computational aspects for dual regularized total least squares
3.4.3 Error bounds in the case B D I
3.4.4 Error bounds for B ¤ I
3.4.5 Numerical illustrations. Model function approximation in dual regularized total least squares
4 Regularization algorithms in learning theory
4.1 Supervised learning problem as an operator equation in a reproducing kernel Hilbert space (RKHS)
4.1.1 Reproducing kernel Hilbert spaces and related operators
4.1.2 A priori assumption on the problem: general source conditions
4.2 Kernel independent learning rates
4.2.1 Regularization for binary classification: risk bounds and Bayes consistency
4.3 Adaptive kernel methods using the balancing principle
4.3.1 Adaptive learning when the error measure is known
4.3.2 Adaptive learning when the error measure is unknown
4.3.3 Proofs of Propositions 4.6 and 4.7
4.3.4 Numerical experiments. Quasibalancing principle
4.4 Kernel adaptive regularization with application to blood glucose reading
4.4.1 Reading the blood glucose level from subcutaneous electric currentmeasurements
4.5 Multiparameter regularization in learning theory
5 Meta-learning approach to regularization – case study: blood glucose prediction
5.1 A brief introduction to meta-learning and blood glucose prediction
5.2 A traditional learning theory approach: issues and concerns
5.3 Meta-learning approach to choosing a kernel and a regularization parameter
5.3.1 Optimization operation
5.3.2 Heuristic operation
5.3.3 Learning at metalevel
5.4 Case-study: blood glucose prediction
Bibliography
Index

Citation preview

Inverse and Ill-Posed Problems Series 58 Managing Editor Sergey I. Kabanikhin, Novosibirsk, Russia; Almaty, Kazakhstan

Shuai Lu, Sergei V. Pereverzev

Regularization Theory for Ill-posed Problems Selected Topics

De Gruyter

Mathematics Subject Classification 2010 47A52, 65J10, 65J20, 65J22, 65N15, 65N20 Authors Dr. Shuai Lu Fudan University School of Mathematical Sciences No. 220, Road Handan 200433 Shanghai PR China [email protected] Prof. Dr. Sergei V. Pereverzev Austrian Academy of Sciences Johann Radon Institute for Computational and Applied Mathematics (RICAM) Altenbergerstraße 69 4040 Linz Austria [email protected]

ISBN 978-3-11-028646-5 e-ISBN 978-3-11-028649-6 Set-ISBN 978-3-11-028650-2 ISSN 1381-4524 Library of Congress Cataloging-in-Publication Data A CIP catalog record for this book has been applied for at the Library of Congress. Bibliographic information published by the Deutsche Nationalbibliothek The Deutsche Nationalbibliothek lists this publication in the Deutsche Nationalbibliografie; detailed bibliographic data are available in the Internet at http://dnb.dnb.de. © 2013 Walter de Gruyter GmbH, Berlin/Boston Typesetting: P T P-Berlin Protago-TEX-Production GmbH, www.ptp-berlin.de Printing and binding: Hubert & Co. GmbH & Co. KG, Göttingen Printed on acid-free paper Printed in Germany www.degruyter.com

This book was written with love for Anna although it was not intended to be her favorite one. The book is just evidence that her efforts were not useless. And to Yanqun for her consistent support.

Preface

The theory of inverse problems has a wide variety of applications because any mathematical model needs to be calibrated before it can be used, and such a calibration is a typical inverse problem. Regularization theory, in turn, is the algorithmic part of the theory of inverse problems. It provides and analyzes the methods for dealing with ill-posedness, which is one of the main issues for inverse problems. In spite of a growing number of monographs on regularization theory (at the time of writing, the latest published one is [84]), there are quite a few topics which have only recently been developed and which are not yet reflected in the literature. The present book is motivated by some of these. The first novelty of this book is that it simultaneously analyzes the ill-posed problems with deterministic and stochastic data noises. Not only does such analysis allow uniform theoretical justification of a general regularization scheme for both of the above-mentioned noise models, it also provides a link to a large class of learning theory algorithms, which are essentially all of the linear regularization schemes. Note that the chapter on regularization algorithms in learning theory is another feature which distinguishes this book from existing monographic literature on inverse problems. A further novelty of the book is Chapter 3, entitled “Multiparameter regularization”. It is interesting to observe that in existing publications the performance of multiparameter regularization schemes have been variously judged by authors. Some of them found that multiparameter regularization only marginally improved the oneparameter version, while others reported on most satisfactory decisions given by multiparameter algorithms in cases where their one-parameter counterparts failed. We hope that Chapter 3 will shed light on this slightly controversial subject. Note that in this book the term “multiparameter regularization” is used as a synonym for “multiple penalty regularization”. At the same time, in modern numerical analysis, the approximation and regularization algorithms are becoming more sophisticated and dependent on various parameters parameterizing even the spaces where penalization, or regularization, is performed. The self-tuning of such parameters means that a regularization space is automatically adjusted to the considered problem. On the other hand, classical regularization theory restricts itself to studying the situation where a regularization space is assumed to be given a priori. Therefore, to the best of our knowledge, Chapter 5 of the present book is the first attempt in the monographic literature to analyze the adaptive choice of the regularization space. This analysis is

viii

Preface

based on the concept of meta-learning, which is also first introduced in the context of regularization theory. The meta-learning concept presupposes that the design parameters of algorithms are selected on the basis of previous experience with similar problems. Therefore, parameter selection rules developed in this way are intrinsically problem-oriented. In Chapter 5 we demonstrate a meta-learning-based approach to regularization on a problem from diabetes technology, but it will also be shown that the main ingredients can be exploited in other applications. At the same time, the material of the chapter describes one of the first applications of regularization theory in diabetes therapy management, which is an extremely important medical care area. We hope that such a context makes the book of interest for a wide audience, within the inverse problems community and beyond. The first part of the book can also be recommended for use in lectures. For example, the sections of the introductory chapter can be used independently in general courses on numerical analysis as examples of ill-posed problems and their treatments. At the same time, Chapter 2 contains a compact and rather general presentation of regularization theory for linear ill-posed problems in the Hilbert space setting under deterministic and stochastic noise models and general source conditions. This chapter can provide material for an advanced master course on regularization theory. Such a course was given at the Technical University of Kaiserslautern, Germany, and at the Stefan Banach International Center in Warsaw, Poland. In Chapter 2 we have really tried to adapt the presentation for this purpose. For example, we avoided the numeration of the formulae in order to make the material more convenient for presentation on the blackboard. The second part of the book can be seen as a presentation of some further developments of the basic theory. This material is new in monographic literature on regularization theory and can be used in students’ seminars. The book was written in the stimulating atmosphere of the Johann Radon Institute for Computational and Applied Mathematics (RICAM). The preliminary plan for the project was discussed with its Founding Director, Professor Heinz Engl. The book would not be possible without our colleagues. We are especially grateful to Peter Mathé (WIAS, Berlin), Alexander Goldenshluger (University of Haifa), Eberhard Schock (Technical University of Kaiserslautern), Ulrich Tautenhahn (12.01.1951 – 10.10.2011), Lorenzo Rosasco (MIT, Cambridge), Bernd Hofmann (Technical University of Chemnitz), Hui Cao (Sun Yat-sen University), Sivananthan Sampath (IIT, Delhi), Valeriya Naumova (RICAM). We are also grateful to Christoph von Friedeburg and Anja Möbius from De Gruyter for the final impulse to start writing this book. Special thanks to Jin Cheng (Fudan University, Shanghai), who recommended the first author to the second one as a Ph.D. student, which was the beginning of the story. Finally we gratefully acknowledge financial support from the Austrian Fonds zur Förderung der wissenschaftlichen Forschung (FWF) (projects P17251-N12 and P20235-N18), Alexander von Humboldt Foundation, National Natural Science

Preface

ix

Foundation of China (Key Projects 91130004 and 11101093), Shanghai Science and Technology Commission (11ZR1402800 and 11PJ1400800) and the Programme of Introducing Talents of Discipline to Universities (B08018), China. Shanghai – Linz December 2012

Shuai Lu and Sergei V. Pereverzev

Contents

Preface 1

An introduction using classical examples

vii 1

1.1 Numerical differentiation. First look at the problem of regularization. The balancing principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.1 Finite-difference formulae . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.2 Finite-difference formulae for nonexact data. A priori choice of the stepsize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1.3 A posteriori choice of the stepsize . . . . . . . . . . . . . . . . . . . . . . 6 1.1.4 Numerical illustration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.1.5 The balancing principle in a general framework . . . . . . . . . . . . 10

2

1.2 Stable summation of orthogonal series with noisy coefficients. Deterministic and stochastic noise models. Description of smoothness properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Summation methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.2 Deterministic noise model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.3 Stochastic noise model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.4 Smoothness associated with a basis . . . . . . . . . . . . . . . . . . . . . 1.2.5 Approximation and stability properties of -methods . . . . . . . 1.2.6 Error bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12 13 14 15 18 19 21

1.3 The elliptic Cauchy problem and regularization by discretization . . . . 1.3.1 Natural linearization of the elliptic Cauchy problem . . . . . . . . 1.3.2 Regularization by discretization . . . . . . . . . . . . . . . . . . . . . . . . 1.3.3 Application in detecting corrosion . . . . . . . . . . . . . . . . . . . . . .

25 27 36 39

Basics of single parameter regularization schemes

47

2.1 Simple example for motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 2.2 Essentially ill-posed linear operator equations. Least-squares solution. General view on regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 2.3 Smoothness in the context of the problem. Benchmark accuracy levels for deterministic and stochastic data noise models . . . . . . . . . . . . . . . . 65 2.3.1 The best possible accuracy for the deterministic noise model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

xii

Contents

2.3.2

The best possible accuracy for the Gaussian white noise model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

2.4 Optimal order and the saturation of regularization methods in Hilbert spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 2.5 Changing the penalty term for variance reduction. Regularization in Hilbert scales . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 2.6 Estimation of linear functionals from indirect noisy observations . . . . 101 2.7 Regularization by finite-dimensional approximation . . . . . . . . . . . . . . 113 2.8 Model selection based on indirect observation in Gaussian white noise 2.8.1 Linear models given by least-squares methods . . . . . . . . . . . . . 2.8.2 Operator monotone functions . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8.3 The problem of model selection (continuation) . . . . . . . . . . . . 2.9 A warning example: an operator equation formulation is not always adequate (numerical differentiation revisited) . . . . . . . . . . . . . . . . . . . . 2.9.1 Numerical differentiation in variable Hilbert scales associated with designs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9.2 Error bounds in L2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9.3 Adaptation to the unknown bound of the approximation error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9.4 Numerical differentiation in the space of continuous functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9.5 Relation to the Savitzky–Golay method. Numerical examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Multiparameter regularization

124 127 131 137 141 143 147 150 151 155 163

3.1 When do we really need multiparameter regularization? . . . . . . . . . . . 163 3.2 Multiparameter discrepancy principle . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 Model function based on the multiparameter discrepancy principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 A use of the model function to approximate one set of parameters satisfying the discrepancy principle . . . . . . . . . . . . 3.2.3 Properties of the model function approximation . . . . . . . . . . . . 3.2.4 Discrepancy curve and the convergence analysis . . . . . . . . . . . 3.2.5 Heuristic algorithm for the model function approximation of the multiparameter discrepancy principle . . . . . . . . . . . . . . . 3.2.6 Generalization in the case of more than two regularization parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

165 168 170 172 173 174 175

3.3 Numerical realization and testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 3.3.1 Numerical examples and comparison . . . . . . . . . . . . . . . . . . . . 177

xiii

Contents

3.3.2 3.3.3 3.3.4

Two-parameter discrepancy curve . . . . . . . . . . . . . . . . . . . . . . 182 A numerical check of Proposition 3.1 and use of a discrepancy curve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 Experiments with three-parameter regularization . . . . . . . . . . . 187

3.4 Two-parameter regularization with one negative parameter for problems with noisy operators and right-hand side . . . . . . . . . . . . . . . . 3.4.1 Computational aspects for regularized total least squares . . . . . 3.4.2 Computational aspects for dual regularized total least squares . 3.4.3 Error bounds in the case B D I . . . . . . . . . . . . . . . . . . . . . . . . 3.4.4 Error bounds for B ¤ I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.5 Numerical illustrations. Model function approximation in dual regularized total least squares . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Regularization algorithms in learning theory

189 191 192 193 195 197 203

4.1 Supervised learning problem as an operator equation in a reproducing kernel Hilbert space (RKHS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 4.1.1 Reproducing kernel Hilbert spaces and related operators . . . . . 205 4.1.2 A priori assumption on the problem: general source conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 4.2 Kernel independent learning rates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 4.2.1 Regularization for binary classification: risk bounds and Bayes consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 4.3 Adaptive kernel methods using the balancing principle . . . . . . . . . . . . 4.3.1 Adaptive learning when the error measure is known . . . . . . . . 4.3.2 Adaptive learning when the error measure is unknown . . . . . . 4.3.3 Proofs of Propositions 4.6 and 4.7 . . . . . . . . . . . . . . . . . . . . . . 4.3.4 Numerical experiments. Quasibalancing principle . . . . . . . . . .

218 220 223 225 231

4.4 Kernel adaptive regularization with application to blood glucose reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 4.4.1 Reading the blood glucose level from subcutaneous electric current measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 4.5 Multiparameter regularization in learning theory . . . . . . . . . . . . . . . . . 249 5

Meta-learning approach to regularization – case study: blood glucose prediction

255

5.1 A brief introduction to meta-learning and blood glucose prediction . . . 255 5.2 A traditional learning theory approach: issues and concerns . . . . . . . . 259 5.3 Meta-learning approach to choosing a kernel and a regularization parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 5.3.1 Optimization operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263

xiv

Contents

5.3.2 5.3.3

Heuristic operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 Learning at metalevel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267

5.4 Case-study: blood glucose prediction . . . . . . . . . . . . . . . . . . . . . . . . . . 269 Bibliography

277

Index

289

Chapter 1

An introduction using classical examples

1.1

Numerical differentiation. First look at the problem of regularization. The balancing principle

Numerical differentiation, which is a determination of the derivative y 0 .t / of the function y.t / from its values at discrete points, is a classic example of ill-posed problems. Ill-posedness here means that every small perturbation in function values may cause huge alterations in the values of the derivative. In the note [65], one can find an example of such extreme behavior of the derivative provided by the function yı .t / D y.t / C ı sin

t , ı2

(1.1)

whose deviation from y.t / is arbitrarily small (for ı > 0 sufficiently small), and whose derivative yı0 .t / deviates from y 0 .t / as much as desired. In the classical books [52, 161], numerical differentiation is used to introduce regularization theory because it encompasses many subtleties that a complex ill-posed problem can exhibit; yet it is very easy to understand and analyze. In this section numerical differentiation serves a slightly different purpose. We use it to demonstrate the balancing principle which will be applied extensively throughout the book.

1.1.1

Finite-difference formulae

Unless otherwise stated, y represents a function having a continuous bounded derivative on the interval Œ0, 1. Given a point t 2 .0, 1/, perhaps the simplest method of approximating y 0 .t / uses the forward difference formula Dh y.t / D

y.t C h/  y.t / . h

If y has second order continuous bounded derivative on Œ0, 1 and h is so small that t C h 2 Œ0, 1, then it follows immediately from Taylor’s theorem that the error in the approximation can be estimated as jy 0 .t /  Dh y.t /j 

h 00 ky kC , 2

where k  kC is a standard norm in the space of continuous functions C Œ0, 1.

2

Chapter 1 An introduction using classical examples

It can easily be checked that for the forward difference formula the order of accuracy O.h/ cannot be improved in general. On the other hand, if y has third order continuous bounded derivative and one requires an accuracy of a higher order of smallness (as h ! 0), one can use the central difference formula Dhc y.t / D

y.t C h/  y.t  h/ . 2h

Then Taylor expansion yields jy 0 .t /  Dhc y.t /j 

h2 000 ky kC . 6

In the literature on numerical analysis one can find many sophisticated finitedifference formulae calculating the approximate value of the derivative y 0 at the point t 2 .0, 1/ as Dhl y.t /

Dh

1

l X

ajl y.t C j h/,

(1.2)

j Dl

where ajl are some fixed real numbers, and a stepsize h is so small that t C j h 2 Œ0, 1 for j D 0, ˙1, ˙2, : : : , ˙l. Moreover, for properly chosen coefficients ajl one usually has the bound jy 0 .t /  Dhl y.t /j  cr hr1 ky .r/ kC ,

(1.3)

for the consistency error provided that y .r/ 2 C Œ0, 1. For example, in [137] it has been shown that the formulae Dh2 with coefficients a02 D 0,

2 2 a12 D a1 D , 3

2 D a22 D a2

1 , 12

(1.4)

and Dh4 with coefficients a04 D 0,

6528 1272 4 D , a24 D a2 , 8760 8760 128 3 4 D D , a44 D a4 , 8760 8760

4 a42 D a1 D

4 a34 D a3

(1.5)

1 4 , c8 D 511 , respectively. meet (1.3) with values r D 6, 8 and c6 D 30 As one can see from equation (1.3), the consistency error of a finite-difference formula Dhl crucially depends on the smoothness of the function to be differentiated. At the same time, all known formulae are usually robust in the sense that their consistency error should converge to zero with h ! 0 even for the function y having very modest smoothness properties. For example, if y is just a continuously differentiable function

3

Section 1.1 Numerical differentiation

then the consistency error of the forward difference formula Dh can be estimated as follows: ˇ ˇZ ˇ ˇ tCh 1 ˇ ˇ ¹y 0 .t /  y 0 . /ºd  ˇ jy 0 .t /  Dh y.t /j D ˇ ˇ hˇ t Z tCh 1 jy 0 .t /  y 0 . /jd   !.y 0 ; h/,  h t where the quantity !.f ; h/ :D

sup

t,;jtjh

jf .t /  f . /j

is known in the Function Theory as a modulus of continuity of a real-valued function f . It is well-known that for f 2 C Œ0, 1, !.f ; h/ ! 0 as h ! 0. It means that the consistency error of the forward difference formula converges to zero for any continuously differentiable function. Thus, dealing with a standard finite-difference formula (1.2), it is natural to assume that for any continuously differentiable function y there exists a nondecreasing function .h/ D .y, Dhl ; h/ , such that 0 D .0/  .h/ and jy 0 .t /  Dhl y.t /j  .h/.

1.1.2

(1.6)

Finite-difference formulae for nonexact data. A priori choice of the stepsize

Suppose now that only an approximate version yı of y is available such that ky  yı kC  ı,

(1.7)

where ı > 0 is a positive number giving a bound for the perturbation in function values. In accordance with the formula (1.2) , the approximation to the derivative of y is then given by l X ajl yı .t C j h/. Dhl yı .t / D h1 j Dl

Since yı is not necessarily differentiable, limh!0 Dhl yı .t / does not necessarily exist. Even when yı is differentiable, it may happen, as we saw in the case of function (1.1), that this limit can be a far cry from y 0 .t /. How bad can things get? We will see this by analyzing the total error of the formula Dhl for nonexact data yı which can obviously be estimated as jy 0 .t /  Dhl yı .t /j  jy 0 .t /  Dhl y.t /j C jDhl y.t /  Dhl yı .t /j,

4

Chapter 1 An introduction using classical examples

where the first term on the right-hand side is the consistency error of Dhl , whereas the second term is a noise propagation error. It can be bounded as jDhl y.t /



Dhl yı .t /j

l ı X l  jaj j, h

(1.8)

j Dl

and under the assumption (1.7) this bound is the best possible one because for a fixed t it is attainable for a perturbed version of y given by i ıh yı . / D yı .t ;  / D y. / sign.ajl C1 /.  t  j h/ C sign.ajl /.t C j h C h   / , h  2 Œt C j h, t C j h C h, j D 0, ˙1, ˙2, : : : , ˙.l  1/. Combining (1.6) with (1.8), one arrives at the total error bound ı jy 0 .t /  Dhl yı .t /j  .h/ C dl , h where l X jajl j. dl D

(1.9)

j Dl

Note that for a fixed perturbation level ı > 0, the error bound (1.9) blows up as h ! 0. Therefore, dealing with nonexact data we cannot let h ! 0 in an arbitrary manner, but must choose an appropriate strategy for relating h to ı . For example, there is no reason to choose h smaller than ı , because for such a choice the error bound (1.9) becomes larger than dl and does not even decrease as ı ! 0. Moreover, assuming t C j h 2 Œ0, 1 for j D ˙1, ˙2, : : : , ˙l, one presupposes that h  1. Thus, in view of (1.9), the best possible choice of the stepsize is h D hmin D hmin ., Dhl / such that ² ³ ı ı D min .h/ C dl , h 2 Œı, 1 . .hmin / C dl hmin h Observe that such a choice of stepsize crucially depends on the consistency error bound (1.6). If, for example, y 00 2 C Œ0, 1 then, as we saw earlier, the inequality (1.6) for the forward difference formula Dh is satisfied with .h/ D .y, Dh ; h/ D h 00 2 ky kC . Moreover, for this formula l D 1, dl D d1 D 2. Then it can be easily found that ² ³ h 00 2ı 1=2 min ky kC C D 2ky 00 kC ı 1=2 , 2 h 1=2

and this value of the total error bound is realized by hmin D 2ky 00 kC ı 1=2 . On the other hand, it has been shown above that the consistency error of the formula Dh can be also bounded by a modulus of continuity !.y 0 ; h/. If y has modest smoothness properties such that y 0 just meets a Hölder condition jy 0 .1 /  y 0 .2 /j  kj1  2 jˇ ,

1 , 2 2 Œ0, 1,

0 < ˇ < 1,

Section 1.1 Numerical differentiation

5

and !.y 0 ; h/ D khˇ , then ³ ² ˇ ˇ 1 2ı min khˇ C D .2=ˇ/ 1Cˇ .1 C ˇ/k ˇC1 ı ˇC1 , h h 1   ˇC1 2ı . and hmin D kˇ These calculations show that the choice of the stepsize h D hmin ., Dhl / minimizing the total error bound (1.9) can only be implemented when the smoothness of the function to be differentiated is given very precisely. However, in applications this smoothness is usually unknown, as one can see in the following example. Example 1.1. In the simplest 1-dimensional case, the cooling of hot glass is modeled by the parabolic system of the form @2 u @u D 2 ; u.0, x/ D uin .x/, @t @t @u @u (1.10) .t , 0/ D ˇ0 .t /, .t , 1/ D ˇ1 .t /, @t @t .t , x/ 2 .0, T   Œ0, 1, and one is interested in determining the heat exchange coefficients ˇ0 , ˇ1 by means of measurements of the boundary temperature. Assume that we have only the noisy measurement uı0 .t /, uı1 .t / of the boundary temperature u.t , 0/, u.t , 1/ on the whole time interval Œ0, T . Such data allow the determination of an approximate distribution u D uı .t , x/ of the temperature u.t , x/ in the whole interval Œ0, 1 as a solution of initial-boundary-value problem with a noisy Dirichlet condition: @2 u @u (1.11) D 2 ; u.0, x/ D uin .x/, u.t , 0/ D uı0 .t /, u.t , 1/ D uı1 .t /, @t @t .t , x/ 2 .0, T   Œ0, 1. From the unique solvability of equations (1.10), (1.11) , it follows that the solution u.t , x/ of equation (1.10) corresponding to the “true” coefficients ˇ0 , ˇ1 is the same as the solution of equation (1.11) with “pure” boundary data u.t , 0/, u.t , 1/ instead of uı0 , uı0 . In view of the well-posedness of (1.11) , the deviation of u.t , x/ from uı .t , x/ is of the same order of magnitude as a data noise. Then without loss of generality one can assume that ju.t , x/  uı .t , x/j  ı. As soon as uı has been determined from equation (1.11), the heat exchange coefficients can be approximated by means of forward and backward difference formulae respectively: uı .t , h/  uı .t , 0/ uı .t , h/  uı0 .t / D , h h uı .t /  uı .t , 1  h/ ˇ1 .t /  1 . h ˇ0 .t / 

(1.12)

6

Chapter 1 An introduction using classical examples

At this point it is important to note that one does not know a priori the smoothness of the function u.t , x/ to be differentiated. This smoothness depends on the so-called compatibility conditions dui n .1/ dui n .0/ (1.13) D ˇ0 .0/, D ˇ1 .1/. dx dx If they are not satisfied then @u .t , / may be discontinuous for t D 0, 1. On the other @x hand, one cannot check equation (1.13), because ˇ0 .t / and ˇ1 .t / are just the functions that should be recovered. Thus, a stepsize h in (1.12) should be chosen without knowledge of the smoothness of the function u.t , x/ to be differentiated.

1.1.3 A posteriori choice of the stepsize A priori choice of the stepsize h D hmin ., Dhl / is tempting since it can be made beforehand, i.e., without dealing with data. Inasmuch as it can seldom be used in practice, one is forced to explore some finite set HN of possible stepsizes ı D h1 < h2 <    < hN D 1, and choose one of them a posteriori considering the approximate values Dhl yı .t / of i

the derivatives produced by the formula Dhl for h D hi , i D 1, 2, : : : , N . The performance of any such a posteriori choice strategy can be measured against the best possible accuracy that can be guaranteed by a formula Dhl under the assumption (1.7). To introduce such a benchmark we observe that for a fixed function y there are many functions  bounding the consistency error of a given formula Dhl . For example, in our calculations above we saw that for a two times continuously differentiable function y the functions .h/ D !.y 0 ; h/ and .h/ D h2 ky 00 kC could bound the consistency error of the forward difference formula. In the following, any nondecreasing function  on the interval Œ0, 1 will be called admissible for t 2 .0, 1/, y and Dhl if it satisfies (1.9) for any h and, moreover, .ı/ < dl . The latter inequality is not a real restriction; if it is not satisfied, then the total error bound (1.9) involving such a function  is too rough (its right-hand side is larger than dl for any h 2 Œı, 1). Let ˆ t .y/ D ˆ t .y, Dhl / be the set of all such admissible functions. In view of (1.9) the quantity ³ ² ı l eı .y, Dh / D inf min .h/ C dl h 2ˆ t .y/ h2Œı,1 is the best possible accuracy that can be guaranteed for approximation of y 0 .t / by the formula Dhl under the assumption (1.7). Now we are going to present a principle for a posteriori choice of the stepsize hC 2 HN that allows us to reach this best possible accuracy up to the multiplier 62 , where  D .HN / D max¹hiC1 = hi ,

i D 1, 2, : : : , N  1º.

7

Section 1.1 Numerical differentiation

We will call it the balancing principle because the chosen hC will almost balance the terms .h/ and dl ı= h in equation (1.9). As we will see, such hC will be automatically adapted to the best possible, but unknown,  2 ˆ t .y/. In this book it will also be shown that the balancing principle can be applied not only to the problem of numerical differentiation. ı .Dhl / be the set of all hi 2 HN such that for any hj  hi , hj 2 HN , Let HN jDhl i yı .t /  Dhl j yı .t /j  4dl

ı , hj

j D 1, 2, : : : , i .

The stepsize hC we are interested in is now defined as ı hC D max¹hi 2 HN .Dhl /º.

We stress that the admissible functions  2 ˆ t .y/, as well as any other information concerning the smoothness of the function to be differentiated, are not involved in the process of the choice of hC . To find hC for a given formula Dhl , one should be able to calculate the values Dhl yı .t / from perturbed data yı for all h D hi 2 HN , and know a bound ı for the perturbation in function values. Now we are ready to formulate the main result of this subsection. Proposition 1.1. For any continuously differentiable function y : Œ0, 1 ! .1, 1/ jy 0 .t /  Dhl C yı .t /j  62 eı .y, Dhl /. Proof. Let  2 ˆ t .y, Dhl / be any admissible function and let us temporarily introduce the following quantities: ² ³ ı hj0 D hj0 ./ D max hj 2 HN : .hj /  dl , hj ² ³ ı hj1 D hj1 ./ D arg min .h/ C dl , h D hj 2 HN . h The latter relation means that min¹.h/ C dl ı= h, h 2 HN º is attained at h D hj1 . Observe that   ı dl ı   .hj1 / C dl , (1.14) hj0 hj1 because either hj1  hj0 , in which case   dl ı dl ı dl ı    .hj1 / C , hj0 hj1 hj1

 > 1,

or hj0 < hj0 C1  hj1 . But then, by the definition of hj0 , it holds true that dl ı= hj0 C1  .hj0 C1 / and   hj C1 dl ı dl ı dl ı D 0 < .hj0 C1 /  .hj1 /   .hj1 / C . hj0 hj0 hj0 C1 hj1

8

Chapter 1 An introduction using classical examples

Now we show that hj0 < hC . Indeed, for any hj  hj0 , hj 2 HN , jDhl j yı .t /  Dhl j yı .t /j  jy 0 .t /  Dhl j yı .t /j C jy 0 .t /  Dhl j yı .t /j 0

0

dl ı dl ı C .hj / C  .hj0 / C hj0 hj dl ı dl ı  2.hj0 / C C hj0 hj dl ı dl ı dl ı 3 C 4 . hj0 hj hj

(1.15)

ı It means that hj0 2 HN .Dhl /, and ı hj0  hC D max¹hi 2 HN .Dhl /º.

Using this and (1.14) one can continue as follows: jy 0 .t /  Dhl C yı .t /j  jy 0 .t /  Dhl j yı .t /j C jDhl j yı .t /  Dhl C yı .t /j 0

0

dl ı dl ı dl ı C4 6  .hj0 / C hj0 hj hj0  0 dl ı  6 .hj1 / C hj1 ³ ² dl ı D 6 min .h/ C . h h2HN

(1.16)

Recall that an ideal choice of the stepsize for the considered formula Dhl and an admissible function  2 ˆ.y, Dhl / would be ² dl ı , hmin D hmin ., Dhl / D arg min .h/ C h

³ h 2 Œı, 1 .

For this hmin an index l 2 ¹1, 2, : : : , N  1º and hl , hlC1 2 HN exist, such that hl  hmin  hlC1 . Then ³ ² dl ı dl ı dl ı min .h/ C  .hl / C D .hmin / C h hmin hlC1 h2Œı,1  ³  ² 1 dl ı 1 dl ı  .hl / C min .h/ C .   hl  h2HN h Combining this with equation (1.16) we obtain

³ ² dl ı . .h/ C h h2Œı,1

jy 0 .t /  Dhl C yı .t /j  62 min

9

Section 1.1 Numerical differentiation

This estimation holds true for an arbitrary admissible function  2 ˆ t .y, Dhl /. Therefore, we can conclude that ³ ² dl ı jy 0 .t /  Dhl C yı .t /j  62 inf min .h/ C . l h2Œı,1 h 2ˆ t .y,Dh / The proof is complete. Remark 1.1. In view of equation (1.15) it is easy to check that Proposition 1.1 is also valid for hC chosen as   3 1 l l C hC D max¹hi 2 HN : jDhi yı .t /  Dhj yı .t /j  dl ı , j D 1, 2, : : : , i º, hi hj or



hC D max¹hi 2 HN :

jDhl i yı .t /  Dhl j yı .t /j

 1 1  2dl ı C , j D 1, 2, : : : , i º. hi hj

On the other hand, in practical computations, these rules sometimes produced more accurate results.

1.1.4

Numerical illustration

To illustrate the balancing principle for the choice of the stepsize we will test it on the function y.t / D jt j7 C jt  0.25j7 C jt  0.5j7 C jt  0.75j7 C jt  0.85j7 ,

t 2 Œ0, 1.

This function was used for numerical experiments in the paper [137], where several new and rather sophisticated finite-difference formulae were proposed. We borrow two of them. Namely, Dh2 with coefficients (1.4), and Dh4 with coefficients (1.5). Moreover, we also consider central difference formula Dh1 D Dhc with coefficients a01 D 0, a11 D a11 D 12 . These formulae meet equation (1.3) with the values r D 6, 8, and 3, respectively. Note that the function y under consideration has only six continuous derivatives. On the other hand, it is a common belief that if r in (1.3) is larger than the actual smoothness of the function to be differentiated then corresponding higher order formulae produce even worse results. For example, it has been suggested in [137, Remark 5.1] that finite-difference formulae should be used according to the regularity of the function, and in accordance with this suggestion the formula Dh2 with coefficients (1.4) and h D 0.1 has been applied in [137] for estimating y 0 .0.5/ D 0.09650714062500. The obtained accuracy is 0.001071  103 . Experiments with perturbed data were not performed in [137]. In our tests we use perturbed values yı .t C j h/ which are the results of a simple program producing the perturbation y.t C j h/  yı .t C j h/ 2 Œı, ı, ı D 0.01  y.0.5/  105 (1% percent noise) by a uniform random number generator.

10

Chapter 1 An introduction using classical examples

Then the above-mentioned formulae Dh2 , Dh4 , Dh1 with stepsizes h D hi D 0.02  i , i D 1, 2, : : : , 15, were applied to noisy data yı .t C j h/, and the first rule from Remark 1.1 was used for determining hC . For the formula Dh2 it gave hC D 0.12 and the value y 0 .0.5/ was estimated with an error 0.001885  103 , i.e., for noisy data the same order of accuracy as in [137] was obtained. It may also be instructive to see that for the stepsize h D 0.2, for example, the error is 0.016934, i.e., almost ten times larger. For the formulae Dh1 and Dh4 the results are even better. Namely, Dh1 , h D hC D 0.06, and Dh4 , h D hC D 0.2, give the value y 0 .0.5/ with errors 5.3209  104 and 2.44106  104 respectively. These tests do not support the suggestion from Remark 5.4 [137] , that in practice only the lower order formulae, such as Dh1 , should be used so that no unexpected errors may occur. In the case considered both formulae give the same order of accuracy, provided that the stepsize is chosen properly. Note that for each specific y and yı the best possible stepsize hideal 2 HN could be defined as hideal D arg min¹jy 0 .t /  Dhl yı .t /j, h 2 HN º. Of course, such hideal is not numerically feasible. Our next test shows how far hC can be from this ideal stepsize. Consider y.t / D sin.t  0.4/=.t  0.4/ and simulate noisy data in such a way that yı .t ˙ j h/ D y.t ˙ j h/ ˙ .1/j ı, ı D 105 . Then we use the central difference formula Dh1 with H15 D ¹h D hi D 0.02  i , i D 1, 2, : : : , 15º. For t D 0.5 it gives hideal D 0.16, hC D 0.28. On the other hand, the error of the formula Dh1 with h D hideal is 1.47654  104 , while for h D hC it approximates y 0 .0.5/ with the error 2.96007  104 . As one can see in the considered case, hC differs from hideal . Nevertheless, the formula Dh1 with h D hC gives the accuracy of the same order as with the ideal stepsize.

1.1.5 The balancing principle in a general framework We have already mentioned that the balancing principle can be applied not only to the problem of numerical differentiation, as it has been done in Proposition 1.1. In modern numerical analysis one often needs to find a balance between approximation and stability. This book contains several examples where such a balance is of interest. In the context of numerical differentiation it can be seen as a balance between consistency and propagation error bounds (1.6), (1.8). Within a general framework some element of interest u (solutions of the problem) can be in principle approximated in some metric space U by an ideal element u depending on a positive parameter  2 Œ0, max  in such a way that a distance d.u , u / goes to zero as  ! 0. It means that a nondecreasing continuous function exists, '. / D '.u ;  / such that '.0/ D 0 and for any

11

Section 1.1 Numerical differentiation

 2 Œ0, max  d.u , u /  '. /.

(1.17)

In practice, however, these ideal approximations u are usually not available. One reason for this is that the data required for constructing u are given with errors. Another reason is that u itself is defined as a solution of some infinite dimensional problem and becomes numerically feasible only after appropriate discretization. Somehow or other we have at our disposal some elements e u 2 U instead of u . For ill-posed problems, such as the numerical differentiation discussed above, it is typical that errors in data propagate in approximations in such a way that a distance u / can increase up to infinity as  ! 0. Such lack of stability can be described d.u ,e in the following form: u /  d.u ,e

1 , . /

 2 Œ0, max ,

(1.18)

where . / is some known increasing continuous function such that .0/ D 0. For example, a propagation error bound (1.8) can be represented as equation (1.18) u D Dhl yı .t /, and . / D  .ıdl /1 . with  D h, u D Dhl y.t /, e The triangle inequality combined with equations (1.17), (1.18) yields the estimation u /  '. / C d.u ,e

1 , . /

(1.19)

which tells us that a balance between approximation and stability bounds is required for good accuracy. In concrete applications one can easily indicate the smallest value min of the parameter  for which the estimation (1.19) is reliable. In the case of numerical differentiation, for example, one can take min D ı, where ı is a bound for perturbations in the values of function to be differentiated. Then in view of equation (1.19) it is reasonable to define the best possible accuracy u as that can be guaranteed for approximation of u by elements e ³ ² 1 inf min , '. / C e.u / D . / '2ˆ.u ,/ 2Œmin ,max  where ˆ.u , / is the set of nondecreasing continuous functions which satisfy equation (1.19) for all  2 Œmin , max , and such that 0 < '.min / < .1min / . Analyzing a numerical differentiation problem we have already explained that the latter inequality is not restrictive at all. The functions from the set ˆ.u , / will be called admissible. We are now ready to formulate a general form of the balancing principle which allows us to reach the best possible accuracy up to some constant multiplier and does not require any knowledge of admissible functions.

12

Chapter 1 An introduction using classical examples

Proposition 1.2 (Balancing principle). Consider an ordered set of parameters †N D ¹i ºN iD1 such that min D 1 < 2 <    < N D max . Let C be selected from †N in accordance with one of the following rules: ² ³ 4 ui ,e uj /  , j D 1, 2, : : : , i  1 , C D max i 2 †N : d.e .j / ² ³ 3 1 ui ,e uj /  C , j D 1, 2, : : : , i  1 , C D max i 2 †N : d.e .i / .j / ² ³ 1 1 ui ,e uj /  2. C /, j D 1, 2, : : : , i  1 . C D max i 2 †N : d.e .i / .j / If the bounds (1.17), (1.18) are valid then for any admissible function ' 2 ˆ.u , / it holds that ² ³ 1 j0 D j0 .'/ :D max j 2 †N : '.j /   C , .j / and uC /  62 e.u /, d.u ,e where

²

³ .iC1 / , i D 1, 2, : : : , N  1 .  D  .†N / D max .i /

To prove this theorem one can use the same argument as in the proof of Proposiui , u , and the distance tion 1.1, where hi , Dhl yı .t /, y 0 .t / should be replaced by i , e i d., / should be used instead of modulus. The general form of the balancing principle presented in Proposition 1.2 will allow us to use it in various situations which will be discussed in the following. The balancing principle has its origin in the paper by Lepskii [101] devoted to nonparametric regression estimation. In the context of inverse problems it has been introduced in [61]. Since then many authors have adopted this approach towards various regularization techniques.

1.2 Stable summation of orthogonal series with noisy coefficients. Deterministic and stochastic noise models. Description of smoothness properties If we are given the Fourier expansion of a continuous function y.t /, and we know that the coefficients in the expansion are slightly perturbed, can we obtain a good approximation to the sum y.t / from the approximate expansion? This problem has been widely discussed in the literature beginning with the classical paper [160] and

Section 1.2 Stable summation of orthogonal series with noisy coefficients

13

book [161]. It is well known that the naive application of a particular summation method to the approximate expansion fails in general, because of the ill-posedness of the problem in the space of continuous functions. In the present section we study this famous example of ill-posed problems and use it to illustrate two models of noise in data.

1.2.1

Summation methods

The recovery of a continuous function y 2 C Œ0, 1 from its Fourier coefficients Z 1 yk D hy, ek i :D y.t /ek .t /dt , k D 1, 2, : : : , 0

with respect to certain given orthonormal systems of functions ¹ek º1 is one of the kD1 classical problems of numerical analysis. It is well known that in general the partial sums n X hy, ek iek .t / Sn y.t / D kD1

of corresponding Fourier series may not converge to y.t / in the space C Œ0, 1 as n!1. This fact gives rise to a number of summation techniques known in approximation theory as -methods; see, e.g., [89, Sect. 2.2.4]. Each of these is determined by a certain triangular array  D .nk /, k D 1, 2, : : : , and transforms a partial sum Sn y.t / into n X nk hy, ek iek .t /. Tn y.t / :D kD1

The idea of -methods originated with Féjer who proved that for the trigonometric system p p e1 .t / 1, e2l .t / D 2 cos 2lt , e2lC1 .t / D 2 sin 2lt , l D 1, 2, : : : , (1.20) the choice 2mC1 1

D 1,

2mC1 2l

D

2mC1 2lC1

  l 1 D 1 , l D 1, 2, : : : , m, m

(1.21)

 y.t / for any y 2 C Œ0, 1, i.e., ensures the convergence T2mC1  ykC !0 as m!1. ky  T2mC1

For the Féjer method the rate of convergence cannot be better than O.m2 / in general. Therefore, more sophisticated -methods were designed to improve this rate for functions y with additional smoothness properties. For a thorough discussion on methods we recommend [89, 163] and the references cited therein.

14

Chapter 1 An introduction using classical examples

1.2.2 Deterministic noise model In practice, Fourier coefficients yk are not usually known exactly, so that we are given determined by only a sequence of perturbed coefficients ¹y,k º1 kD1 y,k D hy, ek i C k , k D 1, 2, : : : ,

(1.22)

where D ¹ k º1 is the noise. kD1 It is easy to realize that applying certain -methods to perturbed Fourier coefficients we cannot guarantee the convergence of ¹y,k º1 kD1 Tn y .t / :D

n X

nk y,k ek .t /

kD1

to the function y.t /, but it can be approximated to a certain degree of accuracy. This accuracy crucially depends on the way the coefficients have been perturbed. If k , k D 1, 2, : : : , are assumed to be chosen by an “antagonistic opponent”, subject to the constraint 1 X k2  ı 2 , (1.23) kD1

the recovery of a continuous function y.t / from its perturbed Fourier coefficients ¹y,k º1 is a classical ill-posed problem discussed in the first monograph on this subkD1 ject [161, Chap. 6]; we refer also to the survey [102]. A positive number ı in (1.23) is used to measure the noise intensity and assumed to be small enough. as a sequence of Fourier In view of equation (1.23) one can consider D ¹ k º1 kD1 coefficients of some function .t / defined as a sum of the series .t / D

1 X

k ek .t /

kD1

that converges in the Hilbert space L2 .0, 1/. In this noise model it is assumed, in fact, that instead of a function y 2 C Œ0, 1 we can only observe a function y D y C 2 L2 .0, 1/, k kL2  ı.

(1.24)

In the worst case the error of any given -method Tn at the function y 2 C Œ0, 1 is measured as  e det .Tn , y, ı/ D sup¹ky  T2mC1 y k1 , k kL2  ıº,

where kgk1 :D sup¹jg.t /j, t 2 Œ0, 1º.

15

Section 1.2 Stable summation of orthogonal series with noisy coefficients

1.2.3

Stochastic noise model

If one assumes that perturbations k , k D 1, 2, : : : , appear randomly in equation (1.22), becomes a problem of statistical estimation. then the recovery of y from ¹y,k º1 kD1 Approaching the problem this way we have in mind that there is an underlying probability space . , F , /, where we consider the Hilbert space L2 ./ of random variables a.!/, ! 2 , with finite second moments Z Ea :D 2



a2 .!/d.!/ < 1.

The inner product h, iL2 . / and norm k  kL2 . / are defined in L2 ./ in the usual way Z ha1 , a2 iL2 . / :D Ea1 a2 D



a1 .!/a2 .!/d.!/, kakL2 . / :D .Ea2 /1=2 .

In nonparametric statistics one adopts an observation model of the form (1.22), where perturbations k D k .!/ are assumed to be random variables from L2 ./. Since each of them is assigned to a particular function ek 2 L2 .0, 1/, this establishes a mapping : ek ! P k from a linear span of e1 , e2 , : : : , em , m D 2, 3, : : : , into L2 ./ so that for m any f D m kD1 fk ek 2 span¹ek ºkD1 we have .f / D

m X

fk .ek / D

kD1

m X

fk k 2 L2 ./.

kD1

Under rather natural assumptions this mapping can be extended to a linear bounded operator : L2 .0, 1/!L2 ./. This is possible, for example, when k , k D 1, 2, : : : , meet the following assumptions: (i) 9c > 0: E k2  c, k D 1, 2, : : :; (ii) k , k D 1, 2, : : : are zero-mean, i.e., E k D 0; (iii) for k ¤ l the random variables k and l are uncorrelated, i.e., E k  l D E k  E l . Keeping in mind that any g 2 L2 .0, 1/ can be represented as a sum g D P1 hg, ek iek C g ? , where g ? is such that hg ? , ek i D 0, k D 1, 2, : : : , the natkD1 ural extension of the mapping appears as .g/ D

1 X

hg, ek i k .

kD1

(1.25)

16

Chapter 1 An introduction using classical examples

In view of (i)–(iii) such an extension is well defined as a linear bounded operator : L2 .0, 1/!L2 ./ since X  X 2  1 1 2 k .g/kL2 . / D E hg, ek i k D E hg, ek ihg, el i k l kD1

D

1 X

k,lD1

2 hg, ek i2 E k2  ckgkL . 2 .0,1/

kD1

In accordance with the usual terminology the operator just defined can be called a Hilbert space valued (generalized) random process. It can also be seen as a generator of the noise in observations (1.22), because by its very definition .ek / D k , k D 1, 2, : : : . On the other hand, as in the deterministic noise model (1.24), one can try to associate random errors k with the values of Fourier coefficients of some (this time random) function. Naturally such a function, also denoted by , is defined so that for all g 2 L2 .0, 1/ we have h , gi :D .g/. By analogy, with the notion of generalized function interpreted as a bounded linear functional on some functional space, this is also named generalized random element. Thus, for any g 2 L2 .0, 1/, the inner product h , gi is a random variable in L2 ./, and its mean value Eh , gi is well-defined. At the same time, Eh , gi can be seen as a bounded linear functional on L2 .0, 1/, since Z 1=2 Z 2 Eh , gi D .g/d  .g/d D k .g/kL2 . /



 k kL2 .0,1/!L2 . / kgkL2 .0,1/ (in the last expression appears as a linear bounded operator from L2 .0, 1/ into L2 ./). Therefore, for the functional Eh , gi the Ritz representer x 2 L2 .0, 1/ exists, such that for any g 2 L2 .0, 1/ hx , gi D Eh , gi. Such a representer is called a mean value of the generalized random element and denoted by E , i.e., Eh , gi D hE , gi. If the assumption (ii) is satisfied, then E D 0. In the following we will always assume a random noise to be zero mean. Keeping in mind that can be seen as a bounded linear operator from L2 .0, 1/ to L2 ./, for any g 2 L2 .0, 1/ the variance of a random variable h , gi admits a representation Eh , gi2 D h .g/, .g/iL2 . / D h  . .g//, gi, where  : L2 ./!L2 .0, 1/ is the adjoint operator of . The operator  : L2 .0, 1/!L2 .0, 1/ is called the covariance operator of and denoted by Cov. /. It is easy to see that for any g1 , g2 2 L2 .0, 1/ E.h , g1 ih , g2 i/ D hCov. /g1 , g2 i.

Section 1.2 Stable summation of orthogonal series with noisy coefficients

17

In terms of the covariance operator Cov. /, one can answer the question whether it is possible to consider as an element of the same Hilbert space in which it is defined as a Hilbert space valued random process. Note that a mean value E always belongs to the space where a generalized random element is defined, whereas itself may not. The best known example is the white noise process with Cov. / equaling to identity. is some orthonormal basis, then for white noise Indeed, if ¹ek º1 kD1 P we have Eh , ek i2 D hek , ek i D 1. It means that the expected value of the sum nkD1 h , ek i2 tends to infinity as n!1, whereas for any element of a Hilbert space such a sum should be bounded. At the same time, if Cov. / is a Hilbert-Schmidt type operator, then 1 X hCov. /ek , ek i < 1. kD1

Moreover, the latter sum does not depend on a particular basis ¹ek º. It is called the trace of Cov. / and denoted by tr.Cov. //. So, if has the covariance operator of the Hilbert-Schmidt type then 1 X Eh , ek i2 D tr.Cov. // < 1, Ek k2 D kD1

and it means that with probability one the generalized random element belongs to the Hilbert space in which it is defined as a Hilbert space valued random process, and the converse is also true. If, for example, the assumptions (i)–(iii) are satisfied, then from (1.25) it follows that 1 1 X X h .ek /, .ek /iL2 . / D E k2 . t r.Cov. // D kD1

kD1

Thus, if in addition to (i)–(iii) we assume that the latter sum is finite, then a generalized random variable given by equation (1.25) belongs to L2 .0, 1/ with probability one. Moreover, it can be represented as 1 X k .!/ek .t /, .!, t / 2  Œ0, 1, (1.26) D .!, t / D kD1

where the latter series converges in the space L2 ./ ˝ L2 .0, 1/ of random valued functions f .!, t / equipped with a norm k  k D .Ek  kL2 .0,1/ /1=2 . Up until now we have characterized random variables h , gi in terms of mean value and variance-covariance, but no assumption of distribution has been made. The distribution most extensively employed in nonparametric statistics is the Gaussian one. The generalized random element is called Gaussian if the random variables h , gi are Gaussian for all g. In view of our discussion above it means that Z t 1=2 exp¹ 2 =2hCov. /g, giºd  . ¹h , gi < t º D ¹2hCov. /g, giº 1

18

Chapter 1 An introduction using classical examples

By their very definitions, generalized Gaussian random elements admit estimations of their distribution in terms of the norm of variance. The main conclusions about the behavior of ¹k k > t º are obtained in the form of the so-called concentration inequalities. In the following we will use the following one [99, p. 59]: if tr.Cov. // < 1 then for any t > 0 ¹k k > t º  4 exp¹t 2 =8Ek k2 º. In view of this inequality for Gaussian generalized random elements the series in equation (1.26) converges now in probability, and in case of continuous functions ek .t / k D 1, 2, : : :, one may ask about the behavior of sup t j .!, t /j. The answer can be found using Dudley’s theorem [99, Theorem 11.17]. This asserts that for Gaussian generalized random elements .!, t / we have Z Dq log N.d ,  /d  , (1.27) E sup j ., t /j  24 t2Œ0,1

0

where D is the diameter of the interval Œ0, 1 in the metric d .s, t / :D .Ej ., t //  ., s/j2 /1=2 , and N.d ,  / denotes the minimal number of  -balls required to cover Œ0, 1 in the space equipped with d -metric. We have presented a stochastic noise model using generalized random elements defined on the Hilbert space L2 .0, 1/. It fits in with the context of the stable summation problem we are discussing in this section. Of course, L2 -space can easily be substituted for any one of Hilbert spaces. Therefore, in the following we will use the concept of generalized (Gaussian) random elements in the general Hilbert space setting. Within the framework of stochastic noise model the error of a given -method Tn at the function y 2 C Œ0, 1 is measured by the so-called risk e ran .Tn , y, / D Eky  Tn y k1 , P where Tn y has the same form as before, i.e., Tn y D nkD1 nk yk, ek .t /, but this time yk, are random variables.

1.2.4 Smoothness associated with a basis In accordance with the paradigm of approximation theory (see, e.g., [158]), the smoothness of a function is determined by the convergence rate of its Fourier coefficients to zero. This rate is usually measured against the rate at which a value .t / of some prescribed continuous nondecreasing function  : Œ0, 1!Œ0, 1/, .0/ D 0, tends to zero as t !0. Then standard assumption on the smoothness of a function y.t / is expressed in terms of spaces W  D W  .¹ek º/, associated with a given basis system ¹ek º, in the following way: ´ μ 1 2 X jhy, e ij k W  D W  .¹ek º/ :D y 2 L2 .0, 1/ : kyk2 :D 0 if there is an .n, k/-independent constant  such that j1  nk j  

  k , n

k D 1, 2, : : : , n;

n D 1, 2, : : : .

(1.29)

Note that the qualification of classical -methods can easily be estimated, and it turns out that it is not so large. For example, from equation (1.21) one can see that the Féjer method has a qualification of the order 1. Moreover, we also indicate the Bernstein–Rogosinsky method  used mainly for the trigonometric system and determined by T2mC1 2mC1 D 1, 2mC1 D 2mC1 D cos 1 2l 2lC1

l , l D 1, 2, : : : , m. 2m

20

Chapter 1 An introduction using classical examples

Since ˇ   ˇ ˇ ˇ ˇ ˇˇ 2 l 2 l ˇˇ ˇ ˇ 2mC1 ˇ 2mC1 ˇ 2 l ˇ ,  D 2 sin ˇ1  2l ˇ D ˇ1  2lC1 ˇ D ˇ1  cos 2m ˇ 4m 8 m one can see that the Bernstein–Rogosinsky method has a qualification of the order 2. At the same time, the partial sum Sn y.t / of the Fourier series can be seen as a method of arbitrary high order of qualification, because it corresponds to nk D 1, k D 1, 2, : : : , n. Another example of a high order qualification method is given by the Tikhonov summation method: Tn˛,s y.t / :D

n X

.1 C ˛k s /1 hy, ek iek .t /,

(1.30)

kD1

where ˛ and s are design parameters. If one chooses ˛ D ˛.n/ D ns then ˇ ˇ  s ˇ ˇ k ˛k s 1 n ˇ ˇ  . j1  k j D ˇ1  D 1 C ˛k s ˇ 1 C ˛k s n Now the qualification of the Tikhonov method is governed by the parameter s and can be made as large as necessary. As we have already mentioned, the recovery of a continuous function from its Fourier coefficients blurred by deterministic or stochastic noise is an ill-posed problem. One of the intrinsic features of such problems is that data noise will unavoidably propagate in an approximate solution. In the considered case the rate of noise propagation depends on the growth of kek k1 when k!1. belongs to the class Kˇ , if kek k1  We say, that an orthonormal system ¹ek º1 kD1 k ˇ , k D 1, 2, : : :, for some ˇ  0. Here and below a relation a.u/  b.u/ means that there are two u-independent constants c1 , c2 > 0, such that for any relevant u, c1 a.u/  b.u/  c2 a.u/. It is apparent that the trigonometric system (1.20) belongs to Kˇ with ˇ D 0, whereas the system of Legendre polynomials (1.28) requires taking ˇ D 1=2. In the stochastic noise model we have to make an additional assumption on the orthonormal system, expressed in terms of Lipschitz properties, namely that there are some positive constants p and cp , for which jek .t /  ek .s/j  cp k p kek k1 js  t j, s, t 2 Œ0, 1.

(1.31)

Note that this assumption is fulfilled for the trigonometric system (1.20) with p D 1. For algebraic polynomials ek of degree k  1, the Markov inequality (see, e.g., [89, Theorem 3.5.8]) asserts kek0 k1  2.k  1/2 kek k1 , such that equation (1.31) is satisfied with p D 2.

Section 1.2 Stable summation of orthogonal series with noisy coefficients

1.2.6

21

Error bounds

We begin error analysis with the following definition. Definition 1.1. Assume that a -method Tn has a qualification of order . We say that the qualification of Tn covers a smoothness of y 2 W  if there is c > 0, such that  t  inf , 0 < t  1. (1.32) c .t / t1 . / Note that if the function  7!   =. / is increasing, then (1.32) is certainly satisfied with c D 1. Proposition 1.3. From equations (1.29) and (1.32) it follows that for k D 1, 2, : : : , n     1  1 n   . j1  k j k c n     Proof. Using equation (1.32) with t D n1 ,  D k1 we have  k1 k    n1 n =c. Then equation (1.29) gives us the required estimate:       1 1  1  D j1  nk jk   k   n  n =c j1  nk j k k n    1 D  . c n Lemma 1.1. Assume that the qualification of Tn covers a smoothness of y 2 W  ¹ek º, and ¹ek º 2 Kˇ . If  is such that for t 2 Œ0, 1 and some p > ˇ C 1=2 the function t !.t /=t p is nondecreasing, then   1 kyk , ky  Tn yk1  cnˇ C1=2  n where c does not depend on n and y.     Proof. Note that under the assumptions of the lemma  k1 k p   n1 np , k D n C 1, n C 2, : : : . Then an application of the Cauchy–Schwarz inequality provides  X    1 X  1  1 jhy, ek ij     kek k1 hy, e ie   k k  k  k1 1 kDnC1 kDnC1  X   1=2 1 2 1 2ˇ  ckyk  k k kDnC1  X   1=2 1 2 1 2p 2.pˇ / D ckyk  k k k kDnC1

22

Chapter 1 An introduction using classical examples

 c

1 n 1

np kyk

 X 1

k 2.pˇ /

kDnC1

1

nˇ C1=2 kyk . n n Among other things, this implies that under the conditions of the lemma, the partial sums Sn y converge to y in k  k1 -norm, and the functions from W  ¹ek º are continuous, if the system ¹ek º consists of continuous functions. Then for any -method Tn one can decompose the error as    n  1   X  X  n    .1  k /hy, ek iek  C  hy, ek iek  ky  Tn yk1    .  c

kD1

p pCˇ C1=2

1=2

n n

kyk D c

1

kDnC1

1

The second summand has just been estimated. Using Proposition 1.3 one can estimate the first summand as ˇ ˇ   n n  1  ˇ hy, e i ˇ X  X ˇ k ˇ  .1  nk /hy, ek iek  j1  nk j ˇ  1  ˇ kek k1    ˇ ˇ k  1 k kD1 kD1   X 1=2 n  1 2  kek k1  kyk c n kD1   X 1=2 n 1  c k 2ˇ kyk n kD1   1 ˇ C1=2  c kyk . n n This proves the lemma. Now we are ready to present the main result of this section. Proposition 1.4. Let the assumptions of Lemma 1.1 be satisfied. Then within the framework of deterministic noise model (1.24) for n D Œ1= 1 .ı/ ( Œa is the integer part of a) we have e det .Tn , y, ı/  ckyk ıŒ 1 .ı/ˇ 1=2 .

(1.33)

If in addition condition (1.31) is satisfied, and isp a Gaussian generalized random element with Cov. / D ı 2 I , then for n D Œ1= 1 .ı log.1=ı// r p 1 ran  (1.34) e .Tn , y, /  ckyk ı log Œ 1 .ı log.1=ı//ˇ 1=2 . ı In bounds (1.33), (1.34) the constant c does not depend on ı and y.

23

Section 1.2 Stable summation of orthogonal series with noisy coefficients

Remark 1.2. Note that the stochastic noise model analyzed in the proposition corresponds to a situation when noise components k in equation (1.22) are assumed to be zero-mean independent identically distributed Gaussian random variables with variance E k2 D ı 2 . Such calibration of noise intensity allows a comparison of the results obtained for stochastic and deterministic noise models. Proof. From Lemma 1.1 and equation (1.22) we obtain ky  Tn y k1  ky  Tn yk1 C kTn y  Tn y k1   n    X 1 ˇ C1=2   n  c ky k C  k k ek  n   n kD1

.

(1.35)

1

Note also, that as a consequence of inequality (1.29), it holds that jnk j  1 C  , k D 1, 2, : : : , n; n D 1, 2, : : : . Then for deterministic noise the second summand in (1.35) can be estimated as   n X 1=2  X 1=2 n n  X n 2 2   k k ek   .1 C  / kek k1 k  cınˇ C1=2 ,  kD1

1

kD1

kD1

and the choice of n D Œ1= 1 .ı/ balances the orders of two summands in (1.35) with respect to ı, such that we arrive at (1.33). The Gaussian white noise case remains to be treated, where we have to bound EkTn y  Tn y k1 , which will be done using Dudley’s inequality (1.27) applied to the Gaussian generalized random element .n/ .!, t / :D

n X

nk k .!/ek .t /, .!, t / 2  Œ0, 1.

kD1

Using assumptions (ii), (iii) we can estimate the distance X 1=2 n n 2 2 .k / jek .s/  ek .t /j , s, t 2 Œ0, 1, d .n/ .s, t / D ı kD1

such that it is easy to bound the diameter D of the interval Œ0, 1 in d .n/ as X X 1=2 1=2 n n 2 2ˇ kek k1  cı k  c1 nˇ C1=2 ı. D  2ı.1 C  / kD1

kD1

We also can bound distances d .n/ .s, t / by a multiple of js  t j, using assumption (1.31). Indeed, X 1=2 n cp2 k 2p .nk /2 kek k21 js  t j  c2 ınpCˇ C1=2 js  t j. d .n/ .s, t /  ı kD1

24

Chapter 1 An introduction using classical examples

Let us consider b1 D c1 nˇ C1=2 ı and b2 D c2 ınpCˇ C1=2 . Then the number of N.d .n/ ,  / in Dudley’s inequality can be bounded by b2 = , and inequality (1.27) yields Z Dp log.b2 = /d  EkTn y  Tn y k1 D Ek .n/ k1  c Z c 0

0

b1

p

p

Z

log.b2 = /d   c b1

!1=2

b1

log.b2 = /d  0

p  cb1 .1 C log.b2 =b1 //1=2  cınˇ C1=2 log n. Combining this estimate with Lemma 1.1 we obtain     p 1  ˇ C1=2 kyk  C ı log n . Eky  Tn y k1  cn n

(1.36)

Note that under the assumptions of the proposition ct  .1/  .t /  t p .1/. c3 t 1=p  So there are positive constants c3 , c4 depending only on  and such that p  1 .t /  c4 t 1= . Then log.1= 1 .t //  log 1t , and for n D Œ1= 1 .ı log.1=ı//, we have log n  c log ı1 . Substituting it in equation (1.36) we arrive at equation (1.34). Let us discuss the results presented in the proposition. To make the discussion more transparent, we restrict ourselves to the classical Sobolev case, when y 2 W  with .t / D t r . In this case for r > 34 , deterministic noise, and a system ¹ek º 2 Kˇ , ˇ D 0, an error bound of order O.ı 1=3 / for the Tikhonov summation method was proven in [77] and [6]. To the best of our knowledge (we refer also to the survey [102]) it was the culmination of all previous work on this particular case. Then the result (1.33) in Proposition 1.4 can be seen as an essential step forward. Moreover, in [114] we argue r1=2 that within the deterministic noise model the order of accuracy O.ı r / cannot be improved for y 2 W  ¹ek º, .t / D t r , and ¹ek º being the trigonometric system (1.20). Recovery of a function y 2 W2r in k  k1 -norm from its trigonometric Fourier coefficients blurred byqGaussian white noise , Cov. / D ı 2 I , was studied in [166], r1=2 where the order O..ı log ı1 / r / of the risk was shown to be optimal. This order is contained in estimation (1.34) as a particular case. Note that in the proposition the result for the Gaussian white noise model is only worse by a log-factor than the corresponding result for deterministic noise. This is a rare phenomenon because, as we will see in the following, the error caused by deterministic noise differs usually by a factor of order O.ı /, > 0, from its stochastic counterpart.

Section 1.3 The elliptic Cauchy problem and regularization by discretization

1.3

25

The elliptic Cauchy problem and regularization by discretization

In this section we discuss one more classically ill-posed problem, namely, the Cauchy problem for elliptic partial differential equations, which was historically the first problem labeled as an “ill-posed” problem. Let be a regular bounded domain in R2 with the unit normal to the boundary @ oriented outward. Also let P be the second order elliptic differential operator defined by 2 X Di .aij Dj u/, P u :D  i,j D1 @ @ , D2 D @y , and real functions aij D aij .x, y/, .x, y/ 2 , are such where D1 D @x that the matrix A.x, y/ D ¹aij .x, y/º2i,j D1 a.e. satisfies T A.x, y/  k k2 for any vector D . 1 , 2 / 2 R2 and a positive constant given independently of .x, y/. Assume a given source function f .x, y/, a flux g.x, y/ and data h.x, y/ the regularity of which will be supplied later on. The Cauchy problem for the operator P consists of finding the solution of the equation .P u/.x, y/ D f .x, y/, .x, y/ 2 , (1.37)

satisfying the following Neumann and Dirichlet conditions: @u .x, y/ D g.x, y/, .x, y/ 2 N , @ u.x, y/ D h.x, y/, .x, y/ 2 D ,

(1.38) (1.39)

where N , D are the accessible parts of the boundary @ , and it is assumed that the linear measures of the sets un :D @ n.N [D / and c :D N \D are positive. The Cauchy problem in such a formulation arises in various applications, where some parts of the boundary of the object are not accessible for measurements. We supply a nonexhaustive list of papers dedicated to such applications [29,30,57,59,88]. At the same time, it is well known that the elliptic Cauchy problem (1.37)–(1.39) is ill-posed in the sense that the existence of solutions for arbitrary Cauchy data .g, h/ cannot be assured, and even if the solution exists, it is not continuously dependent on the Cauchy data. This can be seen from the next example encountered by Hadamard [68], who analyzed the family of problems (1.37)–(1.39) for the Laplace operator P D @2 @2  :D  @x 2  @y 2 , D .0, 1/  .0, 1/, N D D D .0, 1/  ¹0º: u.x, y/ D 0, @u @  D u.x, 0/ D gk .x/, x 2 .0, 1/, @ @y u.x, 0/ D 0, x 2 .0, 1/, where gk .x/ D .k/1 sin.kx/.

(1.40)

26

Chapter 1 An introduction using classical examples

It is easy to see that the solutions of problems (1.40) for k D 1, 2, : : : , are u.x, y/ D uk .x, y/ D .k/2 sin.kx/ sinh.ky/, and in view of the Cauchy–Kowalewskaia theorem they are unique, since the differential operator P D  has analytic coefficients and the data .gk , 0/ are analytic. On the other hand, in equation (1.40) the sequence of Cauchy data uniformly converges to zero as k!1, and in the limit we have a Cauchy problem with data g D h 0, which admits only the trivial solution u 0. However, for every fixed y > 0 we have juk .x, y/j!1 as k!1. Consequently the sequence of the solutions ¹uk º does not converge to u 0 in any reasonable topology showing that in the solution of equations (1.37)–(1.39), in general, there is no continuous dependence of the data .g, h/. Moreover, as has been noted in [100], if the solution of equation (1.40) exists in the classical sense for Cauchy data .g, 0/, then g must be analytic. This means that in general one should be satisfied with a weak solution of equation (1.37), even if C 1 -smooth functions .g, h/ are used as Cauchy data. Recall that an integrable function u is said to be a weak solution of (1.37) if for every smooth function  with compact support in it holds Z Z  u.x, y/.P /.x, y/d D f .x, y/.x, y/d , (1.41)



where the differential operator P   :D 

P

is the formal adjoint of P and given by the formula

2 X

Di Dj .aij /  Dj ..Di a//.

i,j D1

Note that for a weak solution the derivatives appearing in the equation (1.37) may not all exist. In view of the Hadamard example (1.40), it is clear that even in such a weak formulation the Cauchy problem (1.37)–(1.40) is still ill-posed, and an efficient numerical treatment of it can scarcely be achieved without appropriate regularization. As we have already seen in the previous sections, an acute issue in regularization is the choice of the corresponding parameter. There is considerable literature concerned with regularization of the Cauchy problem where this issue is addressed on the basis of a priori knowledge of the smoothness of unknown solution. Sometimes such a priori smoothness even allows a recovery of a continuous dependence of the solution on the Cauchy data (we refer to [69, 78] for details). In practical applications, however, a priori information about solution smoothness is usually very pessimistic or not available at all. On the other hand, we can indicate only a few publications where a regularization parameter choice without knowledge of smoothness has been discussed. In [9] use of a heuristic parameter choice rule known as the L-curve method has been proposed. In [37] only convergence of a regularized approximation with a posteriori

Section 1.3 The elliptic Cauchy problem and regularization by discretization

27

parameter choice has been proven, provided data noise level tends to zero. In [53, 100] the solution of the elliptic Cauchy problem is characterized as a fixed point of some operator equation solved by Picard iterations, and the number of iteration steps is used as a regularization parameter. A posteriori iteration stopping rules proposed there allow error estimations in terms of the data noise level, but a large number of iterative steps are required, as is seen from numerical experiments performed in [53] for a problem, similar to the Hadamard example (1.40). In this section our goal is threefold. Firstly, we use the elliptic Cauchy problem to illustrate the use of the balancing principle presented in Proposition 1.2 for a posteriori parameter choice in regularizing Cauchy problems. In view of the discussion above, the proposed choice rule can be seen as a step forward in the numerical treatment of the Cauchy problem. Secondly, we reformulate the Cauchy problem in the form of a linear operator equation with a compact operator. In the following, such operator equations will be extensively used in our analysis of regularization methods as a general form of linear illposed problems, and we will refer to the Cauchy problem as an illustrative example. Thirdly, in order to regularize the Cauchy problem we use an approach known as regularization by discretization. To the best of our knowledge this approach has not been used for this problem until now. It therefore seems to be instructive to present, analyze and test it in a new situation.

1.3.1

Natural linearization of the elliptic Cauchy problem

There are several ways to reduce the Cauchy problem to a linear operator equation. Presumably, the most ancient of them is the one used in the quasireversibility method by Lattés and Lions [98], where the involved linear operator is unbounded. Here we follow the way leading to an equation with a compact operator for unknown Dirichlet trace s.x, y/ of the solution u.x, y/ at the part of the boundary un where no data was prescribed. Note that in the existing literature on elliptic Cauchy problems most of the efforts concentrate on reconstructing Dirichlet or Neumann traces at the inaccessible part un of @ , because with such data at hand one can recover the whole (weak) solution u.x, y/, .x, y/ 2 in a stable way from the corresponding well-posed mixed boundary value problem. For example, if a Dirichlet trace s.x, y/, .x, y/ 2 un , has already been found, then the solution u.x, y/ can be recovered from the system .P u/.x, y/ @u .x, y/ @ u.x, y/ u.x, y/

D f .x, y/, .x, y/ 2 , D g.x, y/, .x, y/ 2 N , D s.x, y/, .x, y/ 2 un , D h.x, y/, .x, y/ 2 D n N ,

(1.42)

28

Chapter 1 An introduction using classical examples

with a completed set of boundary data. Here, the Dirichlet condition on D n N will vanish automatically when D n N D ;. To derive a linear operator equation for the unknown Dirichlet trace s.x, y/, .x, y/ 2 un , we presuppose that the following mixed boundary value problem .P u/.x, y/ D 0, .x, y/ 2 , u.x, y/ D s.x, y/, .x, y/ 2 un , (1.43) u.x, y/ D 0, .x, y/ 2 D n N , @u .x, y/ D 0, .x, y/ 2 N , @ has a (weak) solution for any s 2 L2 .un /, and its trace on c D D \ N is well defined in L2 .c /. Moreover, it is presupposed that the mixed boundary problem .P u/.x, y/ u.x, y/ @u .x, y/ @ u.x, y/

D f .x, y/, D 0,

.x, y/ 2 , .x, y/ 2 un ,

D g.x, y/,

.x, y/ 2 N ,

D h.x, y/,

.x, y/ 2 D n N ,

(1.44)

also has a (weak) solution with L2 -trace on c D D \ N for the exact Cauchy data .g, h/, as well as for their noisy measurements .g ı , hı /. In fact, the solvability of equations (1.43), (1.44) does not pose a real restriction, as will be seen in the following example. Example 1.2. Consider the problem that has been used as a test example in [95]. For D .0, 1/  .0, 0.5/ R2 define the following subsets of @ : 1 :D ¹.x, 0/; x 2 .0, 1/º, 2 :D ¹.x, 0.5/; x 2 .0, 1/º, 3 :D ¹.0, y/; y 2 .0, 0.5/º, 4 :D ¹.1, y/; y 2 .0, 0.5/º, and consider the Cauchy problem u u @u @ u u

D 0, D h1 ,

in , on 1 ,

D g,

on 1 ,

D h3 , D h4 ,

on 3 , on 4 ,

where h1 , h3 , h4 and g are Cauchy data with noisy measurements hı1 , hı3 , hı4 , g ı . In this example c D 1 , un D 2 and the auxiliary problem (1.43) becomes u u @u @y u

D 0, D s,

in , on 2 ,

D 0,

on 1 ,

D 0,

on 3 [ 4 .

(1.45)

Section 1.3 The elliptic Cauchy problem and regularization by discretization

29

p Keeping in mind that the functions ¹ 2 sin.kx/º1 form an orthonormal basis in kD1 L2 .2 / D L2 .0, 1/, one can represent the weak solution us of equation (1.45) as us .x, y/ D

1 X

1

kD1

cosh. k 2 /

hs./,

p

p 2 sin.k/i 2 sin.kx/ cosh.ky/,

where h, i means the inner product in the corresponding Hilbert space. Then for its trace on c D 1 one has the representation us jc .x/ D

1 X

1

cosh. k 2 / kD1

hs./,

p

p 2 sin.k/i 2 sin.kx/,

and it is easy to see that for any s./ 2 L2 .un / D L2 .2 / this trace is well defined as an element of L2 .c / D L2 .1 / such that kus jc kL2 .c /  kskL2 .un / . Verification of the solvability of the auxiliary problem (1.44) is a bit more involved. In the considered case this problem consists of solving the following system: u @u @y u u u

D 0,

in ,

D g,

on 1 ,

D 0, D h3 , D h4 ,

on 2 , on 3 , on 4 .

(1.46)

Its solution u0 can be represented as the sum u0 D u0,2 C u0,3 C u0,4 , where u0,j , j D 2, 3, 4, solve mixed boundary value problems u u @u @y u

D 0, D 0,

in , on 2 ,

D g,

on 1 ,

D 0,

on 3 [ 4 ,

(1.47)

and u @u @y u u respectively.

D 0,

in ,

D 0,

on 1 ,

D 0, D hj ,

on @ n ¹1 [ j º, on j , j D 3, 4,

(1.48)

30

Chapter 1 An introduction using classical examples

p Using the orthonormal basis ¹ 2 sin.kx/º1 of the space L2 .1 / D L2 .0, 1/ kD1 one can represent the solution u0,2 of equation (1.47) as p    1 X p hg./, 2 sin.k/i 1 2 sin kx sinh k  y . u0,2 .x, y/ D k 2 k cosh 2 kD1 Then, for its trace on on c D 1 we have ku0,2 jc kL2 .c / X  12 1 p 2 2  .k/ hg./, 2 sin.k/i D: kgkW21 ,

(1.49)

kD1

where W21 is the adjoint space to the standard Sobolev space W21 introduced in the previous section. Let us turn to the problem (1.48) and consider the case j D 3 (the case j D 4 is analyzed similarly). Inspecting the proof of Theorem 2 [23], one can deduce that the form an orthonormal basis in L2 .3 / D L2 .0, 0.5/. functions ¹2 cos.2k C 1/yº1 kD0 Using this basis we introduce in L2 .3 / a scale of Hilbert spaces ´ μ 1 X ˇ 2ˇ 2 L2,log :D f : kf kLˇ :D .log.k C 1/ C 1/ jak .f /j < 1 , ˇ 2 R, 2,log

kD0

where ak .f / denotes hf ./, 2 cos..2k C 1/.//iL2 .0,0.5/ . Note that for any ˇ > 0 ˇ the space L2,log is so wide that it contains a Sobolev space W2 with arbitrary small positive . The solution of equation (1.48) for j D 3 can now be represented as u0,3 .x, y/ D

1 X

2ak .h3 /

kD1

1 sinh..2k C 1/.1  x// cos..2k C 1/y/. sinh.2k C 1/

ˇ L2,log ,

If h3 2 ˇ > 1=2, then for the trace u0,3 jc D u0,3 .x, 0/ on c we have the following estimation: 2 ku0,3 jc kL 2 .c / 2 Z 1 X 1  ak .h3 /e .2kC1/ x dx 4 0

kD0

Z

1 1X

 16kh3 k2 ˇ

L2,log



0

e 2.2kC1/ x .log.k C 1/ C 1/2ˇ dx

kD0

Cˇ kh3 k2 ˇ , L2,log

where Cˇ D 16

1 X kD0

1 .log.k C 1/ C 1/2ˇ < 1. .2k C 1/

31

Section 1.3 The elliptic Cauchy problem and regularization by discretization ˇ

Using this estimation with equation (1.49), one can conclude that for h3 , h4 2 L2,log , g 2 W21 , a trace of the weak solution u0 of the problem (1.46) on c is well defined in L2 .c / and ku0 jc kL2 .c /  C.kh3 kLˇ

2,log

C kh4 kLˇ

2,log

C kgkW21 /.

(1.50)

Thus, in the considered case, the existence of weak solutions with L2 -traces on c can be guaranteed for auxiliary problems (1.43) and (1.44) under rather mild assumptions that Dirichlet and Neumann data as well as their noisy counterparts belong to ˇ L2,log , ˇ > 12 and W21 respectively. Under these assumptions one can expect that the exact (weak) solution u belongs only to L2 . /. At the same time, in many papers devoted to elliptic Cauchy problems, it is a priori assumed that a weak solution u belongs to Sobolev space H 1 . / involving all the functions that are in L2 . / as are their first-order derivatives. The reason for such an assumption is related to the use of the finite element method for approximating a weak solution of equation (1.42) when the data on the inaccessible part of the boundary have already been recovered. For example, the variational problem (1.41) for solving the Laplace equation in a weak sense reads: find u 2 L2 . / such that Z  u d D 0 (1.51)

for any  2 H02 . / :D ¹v : v 2 L2 . /, v.x, y/ D @v .x, y/ D 0, .x, y/ 2 @ º. @ To use equation (1.51) within the framework of the finite element method one needs to construct a finite-dimensional trial space V H02 . /, and hence to use finite elements of the class C 1 which have second-order derivatives. To reduce this smoothness requirement, one needs to impose more smoothness assumptions on u. Then an integration by parts allows a transformation of equation (1.51) into a variational problem: find u 2 H 1 . / such that Z ru  r d D 0

for any  2 H01 . / :D ¹v : jrvj 2 L2 . /, v.x, y/ D 0, .x, y/ 2 @ º. Now it is possible to use more simple finite elements of the class C 0 , which form a trial space V H01 . /. On the other hand, to guarantee that a weak solution of equation (1.51) belongs to H 1 . /, one needs to impose additional smoothness assumptions on the Dirichlet and Neumann traces h, g, as well as on their noisy measurements hı , gı . For example, in [53] the proximity of hı to h is, in fact, assumed in the space H 1=2 .D / of traces over D of all functions of H 1 . /, which is a much stronger ˇ assumption than L2,log -smoothness emerging in this example. From this viewpoint it seems more practically relevant to look for a weak solution to the Cauchy problem in L2 . / rather than in H 1 . /. At the same time, for unknown

32

Chapter 1 An introduction using classical examples

Dirichlet trace s, we can construct some smooth approximation in L2 .un /, say snı , which allows the use of simple finite elements from H01 . / for solving the boundary value problem (1.43), where s is substituted for snı . We will present numerical experiments showing that a reliable reconstruction of the whole solution of the Cauchy problem still can be obtained in this way. Assuming existence and unicity of the weak solutions of equations (1.43), (1.44) with well-defined L2 -traces on c , we now derive an equation for unknown Dirichlet trace. First we define an operator A : s 7! us jc 2 L2 .c /,

(1.52)

where us denotes the solution of equation (1.43) for s 2 L2 .un /. It is obvious that A is a linear continuous operator from L2 .un / into L2 .c /. Moreover, if s.x/ is a trace of the solution of the Cauchy problem on the inaccessible part of the boundary un , then this solution can be represented as a sum us C u0 , where u0 is the solution of the auxiliary problem (1.44). Therefore, on c we can write down the following operator equation for unknown Dirichlet data at the inaccessible part of the boundary: As D r :D hjc  u0 jc .

(1.53)

The proof of the next proposition is now obvious. Proposition 1.5. Assume that the auxiliary boundary value problem (1.43) has a unique solution for any s 2 L2 .un /. If the Cauchy problem (1.37)–(1.39) and the auxiliary boundary value problem (1.44) are uniquely solvable, then the problem of the reconstruction of the Dirichlet trace of the solution of equation (1.37) at un is equivalent to solving equation (1.53). For noisy measurements hı and g ı , the right-hand term r in equation (1.53) also appears in the form of a noisy version r ı . If uı0 is the solution of the auxiliary problem (1.44), where h, g are substituted for hı and g ı , then r ı D hı jc uı0 jc . As in [53,95], we assume that we are given noisy Cauchy data hı , g ı , or alternatively r ı , such that kr  r ı k  ı.

(1.54)

In the context of Example 1.2, the inequality (1.50) gives us the estimate (1.54) for h D .h3 , h4 /, hı D .hı3 , hı4 /, g, g ı such that khj  hjı kLˇ

2,log

 cı,

j D 3, 4,

kg  g ı kW21  cı,

where c is some generic constant. In the following reconstruction of the unknown Dirichlet trace we will operate only with the noisy right-hand term r ı , which can be calculated in a stable way from hı , g ı .

Section 1.3 The elliptic Cauchy problem and regularization by discretization

33

If the goal is to reconstruct the whole solutions of equations (1.37)–(1.39), then one also needs noisy Cauchy data hı , g ı . Now it is clear that using the auxiliary problems (1.43), (1.44) one can reduce the reconstruction of unknown Dirichlet trace s 2 L2 .un / to the linear operator equation As D r

(1.55)

In the context of Example 1.2, for instance, the operator A can be written as As.x/ D

1 X

1 

kD1 cosh

k 2

 hs./,

p

p 2 sin.k/i 2 sin.kx/.

(1.56)

p form a basis in L2 .c / D L2 .0, 1/, we can Since the functions ¹ 2 sin.kx/º1 kD1 also write 1 X p p hr./, 2 sin.k/i 2 sin.kx/. r.x/ D kD1

Then the solution s D

sC

of equation (1.55) is given as the sum   1 X p p k C s .x/ D cosh hr./, 2 sin.k/i 2 sin.kx/. 2

(1.57)

kD1

At the same time, one should be aware of the fact that equation (1.55) is ill-posed in L2 .un /, which means that a perturbed problem As D r ı

(1.58)

may have no solution. For example, for the operator (1.56) and p 1 2 3 X sin.kx/ ı r D r C , .x/ D ı , k kL2 .c / D ı,  k kD1

the solution of equation (1.58) can be formally written as   p 1 cosh k X 2 2 3 ı sin.k/, sı D sC C  k

(1.59)

kD1

but for any arbitrary p small ı > 0 the series does not converge in L2 .un /, since its Fourier coefficients ı 6 cosh. k 2 /=k!1 as k!1. Nevertheless, one can consider a partial sum   n X p p k cosh hr ı ./, 2 sin.k/i 2 sin.kx/ snı .x/ D 2 kD1

34

Chapter 1 An introduction using classical examples

of equation (1.59) as an approximate solution of the original problem (1.55), (1.56), and coordination between n and the amount of noise ı in the problem (1.58) can produce a regularization effect. Indeed, in view of equation (1.54), the propagation of the noise in an approximate solution snı can be estimated by ksnı

 sn kL2 .un / D

X n kD1

 cosh

2

 1=2 p n k ı 2  ıe 2 . (1.60) hr  r , 2 sin.k/i 2

On the other hand, assuming convergence of partial sums sn of the series (1.57), we can express it in the form of inequality ks C  sn kL2 .un /  .n1 /,

(1.61)

where . / is some nondecreasing continuous function used for measuring the rate of convergence sn !s C , so .0/ D 0. As we saw in the previous section, a function  in equation (1.61) is determined by the smoothness of the exact solution. In interesting practical cases one cannot expect that . / tends to zero faster than at a polynomial rate, which means that for some r1 > 0 (1.62) . / >  r1 ,  2 .0, 1/. Moreover, it is usually assumed that  obeys the so-called 2 -condition, i.e., there is a constant c > 0 such that (1.63) .2 /  c . /. In view of equations (1.60), (1.61), the error of an approximation snı can be estimated as n (1.64) ks C  snı kL2 .un /  .n1 / C ıe 2 . From this estimate one concludes that for n > 2 ln ı1 the error bound becomes so large that an accuracy smaller than 1 cannot be guaranteed, and the balance between convergence and noise propagation is provided by ° n 2 1± . (1.65) n D nopt D arg min .n1 / C ıe 2 , n D 1, 2, : : : , n  ln  ı Of course , this nopt cannot be found without knowledge of . At the same time, a general form of the balancing principle presented by Proposition 1.2 allows a choice of n that gives an accuracy which is only worse than the ideal one by a constant factor, but does not require any knowledge of . To apply Proposition 1.2 in the considered case one may formally introduce a sequence i D .N  i C 1/1 , i D 1, 2, : : : , N , where N D Œ 2 ln ı1 , and use one of the formulae for C , where ı  =2i =ı, d.uQ i , uQ j / D kuQ i  uQ j kL2 .un / . uQ i D sN iC1 , .i / D e

35

Section 1.3 The elliptic Cauchy problem and regularization by discretization

Of course, any of the rules given by Proposition 1.2 can be reformulated directly in terms of n and snı . In this section we will deal with the rule suggesting the choice ± ° ı kL2 .un /  ı.3e bn C e bm /, m D N , N  1, : : : , m C 1 . n.b/ D min n : ksnı  sm (1.66) This rule is a direct reformulation of the first rule from Proposition 1.2. In view of this proposition and equations (1.64), (1.65), the rule (1.66) for b D gives us an order-optimal bound ° ± n ı ks C  sn. =2/ kL2 .un /  6e =2 min .n1 / C ıe 2 , n D 1, 2, : : : , N .

2

It is instructive to estimate the optimal bound ° ± e., ı/ D min .n1 / C ıe bn , n D 1, 2, : : : , in terms of a noise level ı. As we already noted, the largest admissible n has the value N D Œ b1 ln ı1 . Then  1 . e., ı/  .N 1 /   b ln1 ı Observe now that from the 2 -condition (1.63), it follows that for any positive constant  such that  ,  2 Dom./, we have c1 . /  . /  c2 . /,

(1.67)

where c1 , c2 depends only on  and . Indeed, assuming  > 1 (the case 0 <  < 1 is analyzed similarly), then the left inequality in equation (1.67) holds true with c1 D 1 because of the monotony of . The right inequality is deduced from equation (1.63), since integer j exists for  > 1, such that 2j 1    2j . Then . /  .2j /  .c /j . /  clog2 2 . /, and from equations (1.57), (1.67) we have

 1 e., ı/  cb,  ln1 . (1.68) ı In case  satisfies equation (1.62), the estimate (1.68) tells us that knowing data with accuracy of ı, one can expect to approximate the solution of the problem at best with an accuracy of order lnr1 ı1 . Such a dramatic loss of accuracy is typical for the socalled severely ill-posed problems. In [21] it has been shown that the elliptic Cauchy problems we are discussing really fall into this category. At the same time, one can show that for some ı-independent constant c b,  1 e., ı/  c b,  ln1 . ı

(1.69)

36

Chapter 1 An introduction using classical examples

To see this we assume without loss of generality that .1/ > ı,

.N 1 / < 1

(1.70)

.1/ C ıe b

< cı, which contradicts (1.68); (assuming .1/  ı, we obtain e., ı/  1 .N /  1 would mean that even for noise-free data the approximation error cannot be better than 1). Let ° ± n .b/ D min n : .n1 /  ıe bn , n D 1, 2, : : : . From equation (1.70) it is clear that n .b/  N . Therefore, e., ı/  .1=n .b// C ıe bn .b/  2ıe bn .b/ .

  On the other hand, n .b/ < nı C 1, where nı solves the equation  n1 D ıe bn for n. Then   1 bn .b/ b bnı b e., ı/  2ıe  2e ıe D 2e  . nı Moreover, in view of equation (1.62), and the definition of nı we have   1 1 1 1 1 < .b C r1 / ln1 , bnı D ln C ln  > ln  r1 ln nı ) ı nı ı nı ı and using equation (1.67) we  arrive at (1.69).  Thus, the value  ln1 ı1 can be taken as a benchmark for the order of the approximation error, when the latter is bounded as in equation (1.64). The analysis above shows that this benchmark can be attained without any knowledge of , when the second term in equation (1.64) is known a priori. In the next subsection we discuss a situation where this case does not apply.

1.3.2 Regularization by discretization In the previous subsection we have shown how the problem of reconstructing an unknown Dirichlet trace can be reduced to a linear operator equation (1.55). We have also shown how a noisy version (1.58) of this equation can be regularized in the case of

being a rectangle .0, 1/  .0, 1/, when the operator A can be written as equation (1.56). But such operator representation, called the singular value decomposition/expansion, is not available for domains with more complicated geometry or/and for general elliptic differential operators P . We are now going to discuss how to construct an analog of partial sums snı in this case. Let n-dimensional subspaces Xn , n D 1, 2, : : : , be such that X1 X2 : : : Xn : : : , and

1 [

Xn D X ,

nD1

where X is an appropriate Hilbert space. In the present context X D L2 .un /.

Section 1.3 The elliptic Cauchy problem and regularization by discretization

37

In order to find the approximate solution snı of equation (1.58) in the subspace Xn one needs to fix an independent linear system ¹ˆni ºniD1 forming a basis in Xn . Then snı can be represented in the form of a linear combination snı D

n X

i ˆni

(1.71)

iD1

¹ˆni º,

with a coefficient vector D ¹ i ºniD1 . of the fixed system If the vector is chosen in such a way that kAsnı  r ı k2 D min¹kAs  r ı k2 , s 2 Xn º, then snı is called a least-squares solution of equation (1.58) in the space Xn . It is natural to assume that no ˆni belongs to the nullspace of A (in the opposite case the coefficient i has no impact on the value of kAsnı  r ı k). Then, as will be shown in Chapter 2, snı is the unique solution of the equation Pn A APn s D Pn A r ı ,

(1.72)

where Pn is the orthogonal projector from X onto Xn , and A : L2 .c /!L2 .un / is the adjoint of A. Adding the representation (1.71) to (1.72), and keeping in mind that Pn ˆni D ˆni , i D 1, 2, : : : , n, and hˆni , A Aˆjn i D hAˆni , Aˆjn i, we obtain the following system of linear algebraic equations for the vector M D Yı ,

(1.73)

where the matrix and right-hand side vector are given as M :D .hAˆni , Aˆjn i/i,j D1,:::,n , Yı :D .hAˆni , r ı i/iD1,:::,n .

p

(1.74)

It is easy to see that for the operator (1.56) and ˆni D 2 sin.i /, the matrix M is diagonal, and the least-squares solution snı is just the n-th partial sum of the Fourier series (1.57) with noisy coefficients. Generally, in order to obtain the matrix and the vector (1.74), one needs to know Aˆni , i D 1, 2, : : : , n, at least approximately. In the case of the operator (1.52), this means that a mixed boundary value problem (1.43) should be solved n times for s D ˆni , i D 1, 2, : : : , n. Usually in practice it is enough to take n much less than 100 (see, for example, the numerical test presented below). At the same time, some other methods for solving elliptic Cauchy problems may require hundreds or even thousands of calls of the direct solver for similar mixed boundary value problems (see, for example, numerical experiments reported in [53] for Mann–Maz’ya iterations). From our discussions above it follows that the operator .Pn A APn /1 exists. Thus the solution of equation (1.72) is given as snı D .Pn A APn /1 Pn A r ı .

38

Chapter 1 An introduction using classical examples

Moreover, the operator Pn A APn is a positive compact operator, and as such, it admits a singular value decomposition Pn A APn D

n X

nk ekn hekn , i,

(1.75)

kD1

where ¹ekn º is some X -orthonormal basis of Xn , and nk , k D 1, 2, : : : , n, are eigenvalues of Pn A APn indexed in decreasing order. It is clear that nk are nothing but the eigenvalues of the matrix (1.74), and they can be found using any of existing linear algebra tool boxes. In view of equation (1.75), the operator Pn A has the singular value decomposition of the form P n A D

n q X nk ekn hfkn , i, kD1

where ¹fkn º is some orthonormal system in the finite-dimensional space AXn D ¹u D As, s 2 Xn º. Then n q X snı D ekn hfkn , r ı i= nk . (1.76) kD1

Note that to construct one does not need to know the orthonormal systems ¹ekn º, ¹fkn º. It is enough to solve the system (1.73). At the same time, by analogy with equation (1.60), one can use equation (1.76) for estimating the rate of noise propagation in an approximate solution snı : snı

ksnı

 sn k D

X n

1=2 ı

hr 

r, fkn i2 =nk

kD1

ı p , nn

(1.77)

where nn is the minimal eigenvalue of the matrix M which can be calculated simultaneously to solving the system (1.73). Without loss of generality we assume that the sequence ¹nn º is nonincreasing (if n n n1 n > n1 n1 for some n, then putting n :D n1 one obtains a nonincreasing sequence such that the stability estimate (1.77) remains true). It is natural to expect that for noise-free data r D r 0 least-squares solutions sn D sn0 converge to the exact solution s C as n!1. However, it is not always the case, as was observed in [149]. At the same time, the following result is known. Proposition 1.6 ([52, 66]). sn !s C if and only if lim sup ksn k  ks C k. n!1

Sufficient conditions for the convergence of ¹sn º to s C can be found in [110].

Section 1.3 The elliptic Cauchy problem and regularization by discretization

39

In this subsection our main interest is related to the choice of the discretization parameter n under the convergence assumptions. Therefore, by analogy with equation (1.61), we assume that an increasing continuous function  exists, such that .0/ D 0, and equation (1.61) is satisfied for a least-squares solution sn D sn0 . In view of equations (1.61) and (1.77), the error of a least-squares solution can be estimated as ı (1.78) ks C  snı k  .n1 / C p . nn Given this error bound we are interested only in n such that nn > ı 2 . Let Nı be the largest integer meeting this inequality. Then, as in Section 1.1.5, an increasing continuous function  is called admissible if it meets equation (1.78) for n D 1, 2, : : : , Nı , p and .0/ D 0, .Nı1 / < ı= nn . For fixed s C the set of all such admissible functions is denoted by ˆ.s C /. The best error bound of the form (1.78) is then given as ´ μ ı .n1 / C p , n D 1, 2, : : : , Nı . eı .s C / :D inf 2ˆ.s C / nn 1 Now, q using Proposition 1.2 (balancing principle), with i D .Nı  i C 1/ , .i / D ı iC1 ı N Q i D sN , we conclude that the choice Nı iC1 =ı, u ı iC1

® 3 1 ı n D nC D min n : ksnı  sm k  ı. p C p /, n n m m ¯ m D Nı , Nı  1, : : : , n C 1 automatically achieves the best error bound up to the factor 6, where  D max¹nn =nC1 nC1 , n D 1, 2, : : : , Nı º, i.e., ks C  snı C k  6eı .s C /. We would like to emphasize that a near optimal choice of the discretization parameter n D nC is entirely data driven. It only relies on the quantities nn and approximate solutions snı appearing in a computational process. In the next subsection we present a numerical example of an application of the approach presented above. At the same time, we would like to mention that sometimes in practice the stability bound (1.77) is too pessimistic. Then more flexible, but also more sophisticated, parameter choice rules based on the balancing principle can be used. Here we refer to [34].

1.3.3

Application in detecting corrosion

The aim of corrosion detection is to determine the possible presence of corrosion damage by performing current and voltage measurements on the boundary. This means

40

Chapter 1 An introduction using classical examples

applying a current of density g to an accessible portion of the boundary of the conductor and measuring the corresponding voltage potential u on that same portion. This physical problem is modeled as follows. A regular bounded domain represents the region occupied by the electrostatic conductor which contains no electric sources, and this is modeled by the Laplace equation, so that the voltage potential u satisfies u D 0 in . (1.79) We assume that the boundary @ of the conductor is decomposed into three open nonempty and disjointed portions c , un , D . The portion c is accessible for electrostatic measurements, whereas the portion un , where the corrosion may take place, is inaccessible. The remaining portion D is assumed to be grounded, which means that the voltage potential u vanishes there, i.e., u D 0 on D .

(1.80)

We impose a nontrivial current density g on c and measure the corresponding potential h on the same portion of the boundary. Using the pair of boundary measurements u D h on c @u D g on c @

(1.81)

on the we want to recover a relationship between voltage u and current density @u @ inaccessible boundary portion un suspected to be corroded. In other words, we are interested in the identification of a nonlinearity f such that @u (1.82) on un . @ There is a substantial amount of literature devoted to the problem of corrosion detection. Appropriate references can be found, for example, in [5]. Our aim here is to demonstrate how the approach presented in Sections 1.3.1 and 1.3.2 can be applied to this problem. It is easy to realize that the reconstruction of a nonlinearity f in equation (1.82) can be achieved in two steps. First, an elliptic Cauchy problem (1.79)–(1.81) can be reformulated as a linear operator equation with a compact operator, which can be regularized by discretization. Then, using an approximate solution of the problem (1.79)– (1.81), one is able to find its Dirichlet and Neumann traces at un , and reconstruct f from equation (1.82) pointwise. In Section 1.3.1 we have described how the elliptic Cauchy problem (1.79)–(1.81) can be reduced to a linear operator equation with respect to a Dirichlet trace at un . By solving such an equation one can complete a set of data for the corresponding (mixed) boundary-value problem and find its solution in a stable way. In view of uniqueness f .u/ D

Section 1.3 The elliptic Cauchy problem and regularization by discretization

41

results the latter solution solves also starting Cauchy problem (1.79)–(1.81). So its Neumann trace at un gives us an approximation for the right-hand side in equation (1.82). But to find such a trace one should perform a numerical differentiation procedure. As we saw in Section 1.1, such a procedure requires additional regularization, and it should be provided with a reliable estimate of data noise level. In the present context this data noise level is nothing but an error bound for a Dirichlet trace reconstruction. In spite of the fact that a near optimal regularization of the latter problem has been proposed in the previous subsection, we cannot reliably estimate corresponding errors a posteriori. Such a situation is typical for ill-posed problems. A remedy has been proposed in [33]. To avoid this issue here we reduce the Cauchy problem (1.79)– (1.81) to an operator equation with respect to Neumann trace at un . This allows us to combine two ill-posed problems, such as a numerical differentiation and a Dirichlet trace reconstruction, into one problem. At the same time, the ill-posedness of this problem can only increase compared to a Dirichlet trace reconstruction. Nevertheless, the numerical experiment presented below shows that a regualrization by discretization with a parameter n D nC given by the balancing principle still allows an efficient treatment of this increased ill-posedness. An operator equation for an unknown Neumann trace at un can be derived in the same way as in Section 1.3.1. Namely, we introduce a linear continuous operator A : L2 .un /!L2 .c /, that assigns to s 2 L2 .un / a Dirichlet trace of the (weak) solution us of the mixed boundary value problem 8 u D 0, in

ˆ ˆ ˆ ˆ @u ˆ ˆ < D s, on un @ (1.83) ˆ @u ˆ ˆ ˆ D 0, on c ˆ ˆ : @ u D 0 in D at c , i.e., As D us jc . Then we consider a function u0 solving an auxiliary mixed boundary value problem 8 ˆ u D 0, in

ˆ ˆ ˆ ˆ @u ˆ ˆ < D 0, on un @ (1.84) ˆ @u ˆ ˆ ˆ D g, on c ˆ ˆ ˆ : @ u D 0, in D . If s in equation (1.83) is such that us jc C u0 jc D h, then in view of uniqueness results, the function us C u0 solves the Cauchy problem (1.79)–(1.81). Therefore, an unknown Neumann trace s can be found from the operator equation As D r, r D h  u0 jc .

(1.85)

42

Chapter 1 An introduction using classical examples 1 Гc,1

Гun

0.5 0

Ω

− 0.5

Гc,3 Гc,2

−1 − 1.5 −2 −2

− 1.5

−1

− 0.5

−0

− 0.5 Г 1 D

1.5

2

Figure 1.1. Electrostatic conductor grounded to D ; a boundary portion un is suspected to be corroded; a boundary portion c D c,1 [ c,2 [ c,3 is accessible for electrostatic measurements.

If only noisy measurements g ı , hı are available, then one can construct a noisy version of equation (1.85) (1.86) As D r ı , r ı D hı  uı0 jc , where uı0 is the solution of (1.84) with g substituted for g ı . Equation (1.86) can then be regularized by discretization in the same way as in Subsection 1.3.2. To illustrate this approach we consider a problem (1.79)–(1.82) in the domain displayed in Figure 1.1, where c D c,1 [ c,2 [ c,3 , c,1 D ¹.x, y/ : x 2 C y 2 D 1, x  0, y  0º, c,2 D ¹.x, y/ : 3y C 4x C 4 D 0, x 2 Œ1, 0.5º, c,3 D ¹.x, y/ : x D 1, y 2 Œ2, 0º, D D ¹.x, y/ : x 2 Œ0.5, 1, y D 2º, un D ¹.x, y/ : x 2 C y 2 D 1, x  0, y  0º. In equation (1.81) synthetic data h, g are chosen in such a way that the exact solution is given as yC2 u.x, y/ D , .x, y/ 2 . .y C 2/2 C x 2 Then it is easy to check that at un yC2 5y C 4 @u , uD D , @ .4y C 5/2 4y C 5

Section 1.3 The elliptic Cauchy problem and regularization by discretization

43

and these two functions are related by the equation (1.82), where 11 2 1 2 f .t / D 4t 2  t C , t  . 3 3 3 5 To implement a regularization by discretization in the form described in Section 1.3.2 one needs to choose a system ¹ˆni º. In our test, we take the basis functions ˆni , i D 1, 2, : : : , n; n D 1, 2, : : : , 66, as the traces at un of two-dimensional polynomials 1, x, y, x 2 , xy, y 2 , : : : , x k1 , x k2 y, : : : , y k1 , k D 1, 2, : : : , 11, n D k.k C 1/=2. Then to calculate the matrix M in equation (1.74) one needs to find Aˆni , i D 1, 2, : : : , n. In this example Aˆni is the Dirichlet trace at c of the solution of equation (1.83), where s is substituted for ˆni . To find such solutions we use Matlab PDE toolbox with the FEM method, where the domain is triangulated into 622592 triangles, and an approximate solution is constructed as a corresponding sum of piece-wise linear finite elements. The auxiliary problem (1.84) is solved in the same way. The inner products hAˆni , Aˆjn i, hAˆni , r ı i in L2 .c / are computed approximately by the trapezoidal rule using the apexes of triangles forming -triangulation as knots on c . Moreover, calculating hAˆni , r ı i, we add the corresponding values of h and u0 to the values of independent zero mean random variables uniformly distributed on Œı, ı, ı D 105 . In this way noisy data are simulated. To implement the discretization parameter choice n D nC we need to compute the minimal eigenvalues nn of the matrices .hAˆni , Aˆjn i/ni,j D1 , n D 1, 2, : : : , Nı , Nı D 66. In Figure 1.2 we present the first ten values of nn plotted in log10 -scale. To obtain a sequence ¹snı º66 nD1 of approximate Neumann traces at un we solve linear systems (1.73) for calculated matrices and vectors. Then the balancing principle provided with the computed values of ¹nn º66 nD1 suggests a discretization parameter choice nC D 3. 1 0 −1

log10

−2 −3 −4 −5 −6 −7 −8 1

2

3

4

5

6

7

8

9

10

Figure 1.2. The minimal eigenvalues of the matrices .hAˆni , Aˆjn i/ni,j D1 , n D 1, 2, : : : , 10, plotted in log10 -scale.

44

Chapter 1 An introduction using classical examples − 0.11 − 0.12 − 0.13 − 0.14 − 0.15 − 0.16 − 0.17 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

exact n=3 0.9 1

.5yC4/ Figure 1.3. The exact Neumann trace @u D  .4yC5/ 2 , y 2 Œ0, 1 and its regularized ap@ proximation snı with the parameter n D 3 chosen in accordance with the balancing principle (from [35]).

In Figure 1.3 we plot approximate reconstruction snı C , nC D 3, together with the

D .5y C 4/=.4y C 5/2 , y 2 Œ0, 1. exact Neumann trace @u @ Using the approximate reconstruction snı C , we can find an approximate voltage potential in as the solution uınC of the mixed boundary value problem (1.79), (1.80), completed by the boundary conditions @u @u D g ı in c , D snı C in un . @ @ The Dirichlet trace uınC jun is displayed in Figure 1.4 with the trace u.x, y/ D p .y C 2/=.4y C 5/, x D 1  y 2 , y 2 Œ0, 1, of the exact solution. Now the desired nonlinearity f can be reconstructed pointwise from (1.82), where are substituted for uınC jun and snı C respectively. This reconstruction is u and @u @ displayed in Figure 1.5. Here we essentially assume that the approximate Dirichlet trace uınC jun has been obtained as a monotone function (see Figure 1.4). One cannot guarantee that this will be always the case. If uınC jun is a nonmonotone, then the relation (1.87) f .uınC .x, y// D snı C .x, y/, .x, y/ 2 un , can be used in general only for reconstructing f as a multi-valued function. In [33,51] it has been shown how this difficulty can be overcome by regularization of (1.87). In the example considered a regularized approximation snı C is so accurate that uınC jun preserves a monotony of the exact trace, and there is no need for an additional regularization.

Section 1.3 The elliptic Cauchy problem and regularization by discretization 0.4

45

exact n=3

0.39 0.38 0.37 0.36 0.35 0.34 0.33 0

0.1

0.2

0.3

0.4

0.5

Figure 1.4. The exact Dirichlet trace u.x, y/ D p uınC , x D 1  y 2 , y 2 Œ0, 1.

0.6

yC2 ,y 4yC5

0.7

0.8

1

0.9

2 Œ0, 1, and its approximate solution

− 0.11

exact n=3

− 0.12 − 0.13 − 0.14 − 0.15 − 0.16 − 0.17 0.33

0.34

0.35

0.36

0.37

Figure 1.5. The exact nonlinearity f .t / D 4t 2  reconstruction obtained from (1.87) (from [35]).

11 t 3

0.38

C 23 ,

1 3

0.39

0.4

 t 

2 , 5

and its pointwise

Chapter 2

Basics of single parameter regularization schemes

2.1

Simple example for motivation

Some people consider analysis and statistics as areas of mathematics. At the same time, a lot of statisticians do not agree with this. Nevertheless, the majority of statistical problems can be treated as mathematical ones. Therefore, even statisticians agree that there is mathematical statistics which provides the methodology for statistical inferences. Moreover, statisticians also agree that statistical problems are basically ill-posed, and should be treated with great caution. The same is true, however, regarding almost all inverse problems of mathematical analysis. The present chapter can be considered an attempt to find a unified approach to the joint problems of analysis and statistics. To begin with, we would like to show a standard statistical problem that is also a standard problem in numerical analysis.

The estimation of probability density function Let D .!/ be some random variable taking the value in R D .1, 1/, i.e., is a mapping from some probability space . , F , / equipped with  -algebra F and a probability measure , to the real line R. We assume that this mapping is measurable with respect to natural Borel  -algebra on R, i.e., 8t 2 .1, 1/ ¹! : .!/ < t º 2 F . It allows definition of a function F .t / D ¹! : .!/ < t º, which is called the probability distribution function. This function can be sufficiently smooth such that Z t f . /d  , F .t / D 1

where f . / is called the probability density function, or simply density. Note that it is very important to know such a density because it allows prediction of the behavior of the random variable . Namely, for any t1 , t2 Z t2 ¹! : t1   t2 º D f . /d  . t1

48

Chapter 2 Basics of single parameter regularization schemes

A standard statistical problem is therefore to estimate the probability density function. The information that is usually available for such an estimate takes the form of samples ¹ i 2 R, i D 1, 2, : : : , N º of the values of .!/. Using such samples, one can assume that .!/ takes the values between min1iN i and max1iN i , i.e., at least with large probability min i  .!/  max i . The interval Œmin i , max i  is then divided into subintervals Œtk1 , tk , tk D min i C  kh, k D 0, 1, : : : , n, h D max min . Sometimes such intervals are called classes. n The empirical distribution function Fn .t / is determined as Fn .t / D

j X Nk , N

t 2 Œtj 1 , tj /,

Nk D card¹ i : tk1  i < tk º,

kD1

where NNk is called the frequency and considered as an empirical estimate of the probability that 2 Œtk1 , tk /. This empirical distribution function is a piecewise constant approximation for F .t /. Putting it into the equation for density one obtains Z t f . /d  D Fn .t /. min i

What can we tell about this equation? First of all, it is clear that there is no solution for such an equation because the left-hand side is a differentiable function, while the right-hand one is discontinuous. At the same time, we do not wish to solve this equation, we only want to recover or to estimate f . / from the information presented in the form of an empirical distribution function, and we realize that this information is contaminated by noise. Moreover, the above-mentioned statistical problem is nothing but the problem of numerical differentiation which was discussed in Section 1.1. If we have some methods for the first problem, we can use them for the second one, and vice versa. Last but not least, the above-mentioned equation is a particular case of illposed linear operator equations. If a general approach to such problems exists, it can be applied in this particular case. To conclude this foreword, let us summarize that there are a lot of similarities between statistical problems and inverse problems of numerical analysis. The difference between them is mainly related to the stochastic nature of the noise presenting in statistical information. Thus the situation appears to be that there is a common scope of ill-posed problems and there are two different noise models for each problem. If the nature of the noise is taken into account, then there is hope that the results of numerical analysis will be applied to statistics and vice versa. This is exactly the general topic for discussion within the framework of the current chapter.

Section 2.2 Essentially ill-posed linear operator equations

2.2

49

Essentially ill-posed linear operator equations. Least-squares solution. General view on regularization

Let X and Y be Hilbert spaces. The relation Ax D y between x 2 X , y 2 Y , A 2 L.X , Y / when considered as an equation with respect to x represents an ill-posed problem if the inverse A1 of A does not exist, or it is not continuous in the topology L.Y , X /, i.e., A1 … L.Y , X /. There are three causes leading to ill-posedness of the equation Ax D y. First cause: N.A/ :D ¹v : v 2 X , Av D 0º ¤ ¹0º; Second cause: Range.A/ :D ¹u : 9v 2 X : Av D uº ¤ Range.A/, i.e., the range of A is not closed in Y ; Third cause: Range.A/ D Range.A/ ¤ Y . Of course, the first cause can occur simultaneously with the second or third one, but it is not so serious in itself. It simply means that our equation has a variety of solutions, because if x0 is one of them, then for any z 2 N.A/, x0 C z also meets the equation, i.e., A.x0 C z/ D Ax0 C Az D Ax0 D y. Sometimes we are even glad to have such a variety, since then we have the possibility of picking a solution which fulfills additional criteria. The third cause is also not so severe. Theoretically it means that in choosing Y as an observation space we were not so careful, because as it is a closed subspace of the Hilbert space Y , the space Range.A/ is a Hilbert space itself. The same is true for the space N.A/? :D ¹x : x 2 X , hx, vi D 0, 8v 2 N.A/º, which is an orthogonal complement to the nullspace N.A/ in X . Moreover, Range.A/ D Range.A/ if and only if N.A/? ¤ ¹0º, and  :D

inf

v2N.A/? , kvkX D1

kAvkY > 0.

The operator A then acts as an isomorphism from the Hilbert space N.A/? to the Hilbert space Range.A/. Thus, if instead of the pair of Hilbert spaces .X , Y / one takes the pair of Hilbert spaces X1 D N.A/? and Y1 D Range.A/, then the problem Ax D y will be well-posed with respect to the new pair .X1 , Y1 /, and 1 kA1 kY1 !X1 D , 

50

Chapter 2 Basics of single parameter regularization schemes

i.e., if Range.A/ D Range.A/ ) A1 2 L.Range.A/, N.A/? /. Roughly speaking, if the ill-posedness of the problem is caused by the third and / or the first cause, then by changing the spaces we can keep the Hilbert structure and avoid ill-posedness. This is not true for the ill-posedness induced by the second cause. In this case  D 0 and A1 , even if it exists, cannot be a bounded linear operator from Range.A/ to N.A/? . Therefore, a problem Ax D y with an operator A 2 L.X , Y /, such that Range.A/ ¤ Range.A/ in Y , is called an essentially ill-posed problem. The most important example of an operator with nonclosed range is a compact operator. It is known that any compact operator A acting between Hilbert spaces X and Y can be represented in the form of the so-called singular value expansion Ax D

rank.A/ X

sk uk hx, vk i,

kD1

where rank.A/ :D dim Range.A/, and positive numbers sk are the square roots of the nonzero eigenvalues of operators A A 2 L.X , X / and AA 2 L.Y , Y /, written in decreasing order with multiplicities. Here, A is the adjoint of the operator A, and ¹vk º, ¹uk º are corresponding complete orthonormal systems of eigenvectors of A A, AA , i.e., A Avk D sk2 vk ,

AA uk D sk2 uk ,

kuk kY D kvk kX D 1,

k D 1, 2, : : : , rank.A/. It is clear that A y D

rank.A/ X

sk vk hy, uk i

kD1

and A uk D sk vk .

Avk D sk uk ,

If there are an infinite number of singular values sk , then rank.A/ D 1, and compactness means that lim sk D 0.

k!1

In such a case the range of A cannot be closed in Y because for any k D 1, 2, : : : 0

inf

v2N.A/? kvkX D1

k!0

kAvkY  kAvk kY D sk kuk kY D sk ! 0.

Thus, inf

v2N.A/? kvkX D1

kAvkY D 0.

Section 2.2 Essentially ill-posed linear operator equations

51

This means that in the case of infinite-dimensional compact operators, the necessary and sufficient condition for closedness of Range.A/ is broken. The nonclosedness of Range.A/, in its turn, leads to instability of the problem Ax D y, because even minor perturbation of the right-hand side can make the problem unsolvable. Indeed, by definition, for any yN 2 Range.A/nRange.A/, one can find y 2 Range.A/, such that the distance ky  yk N can be arbitrarily small, and the equation Ax D y is solvable, whereas the equation Ax D yN has no solution because yN … Range.A/. It should be noted that in general, checking whether or not y 2 Range.A/ is equivalent to solving the equation Ax D y. On the other hand, in practice one is usually given an equation Ax D y, and even if y … Range.A/, one should assign to each pair .A, y/ some element x considered as an approximation to the “ideal solution”, that will never be known if y … Range.A/. In this situation, a reasonable compromise between reality and its model would be the so-called least-squares solution xN such that kAxN  yk D inf¹kAx  yk, x 2 X º. If N.A/ ¤ ¹0º, then the least-squares solution is not unique because 8v 2 N.A/, v ¤ 0, A.xN C v/ D Ax. N In this case, one can search for a minimal-norm least-squares solution x  such that x  2 N.A/? and kAx   yk D inf¹kAx  yk, x 2 X º. Let Q be the orthoprojector onto Range.A/. Then kAx   yk2 D kAx   Qy C .I  Q/yk2 D kAx   Qyk2 C k.I  Q/yk2 , because Ax  2 Range.A/ ) Ax  2 Range.A/ ) Ax   Qy 2 Range.A/, and ? .I  Q/y 2 Range.A/ . Keeping in mind that k.I  Q/yk2 does not depend on x  , one can see that if Qy 2 Range.A/ then x  is a solution of the equation Ax D Qy. In practice, it is not always possible to construct Q. In order to do this one should know the elements of singular value expansion of the operator A. However, the equation Ax D Qy can be equivalently transformed in such a way that Q will not be involved. More precisely, the following proposition holds true. Proposition 2.1. Equations Ax D Qy and A Ax D A y have the same set of solutions. Proof. Let x0 be the solution to A Ax D A y. Then for any v 2 X , we have hA Ax0  A y, vi D 0, where h, i is the inner product in X . By definition hA Ax0  A y, vi D hAx0  y, Avi D 0,

52

Chapter 2 Basics of single parameter regularization schemes ?

and this means that Ax0  y 2 Range.A/? Range.A/ . So   Ax0 2 Range.A/ D Ax0  Qy D 0 Q.Ax0  y/ D QAx0  Qy D QAx0 D Ax0 Thus, x0 is the solution of Ax D Qy. Let x0 now be the solution to Ax D Qy, i.e., Ax0 D Qy. Keeping in mind that Q D Q , because it is an orthoprojector, and QA D A, one has A D .QA/ D A Q D A Q, and by then applying A D A Q to both sides of the equality Ax0 D Qy, we obtain A Ax0 D A Qy D A y, i.e., x0 is the solution of the equation A Ax D A y. Remark 2.1. The transformation of the equation Ax D y to the equation A Ax D A y is sometimes called a Gaussian symmetrization. Thus, applying Gaussian symmetrization to the initial equation Ax D y, in the case of Qy 2 Range.A/, we obtain an equation with solutions minimizing the norm of the residual kAx  yk. The main reason to pass from the initial equation Ax D y to a symmetrized equation A Ax D A y is that if y 2 Range.A/, then each solution of the initial equation will be a solution for the symmetrized one. However, for y … Range.A/, such that Qy 2 Range.A/, the initial equation has no solution, while the symmetrized one does have one. Moreover, each solution of the symmetrized equation will minimize the residual kAx  yk. Now we formulate the condition which is necessary and sufficient for the solvability of the problem A Ax D A y in the space X . Proposition 2.2. Let A 2 L.X , Y / be a compact operator with rank.A/ D 1 and with the singular value expansion Ax D

1 X

sk uk hvk , xi.

kD1

Then the equation A Ax D A y is solvable in X or, similarly, Qy 2 Range.A/, where Q is the orthoprojector onto Range.A/, if and only if 1 X

sk2 huk , xi2 < 1.

kD1

Under this condition the minimal norm solution of A Ax D A y has the form x D

1 X kD1

sk1 vk huk , yi.

53

Section 2.2 Essentially ill-posed linear operator equations

Proof. It is easy to see that A Ax D

1 X

A y D

sk2 vk hvk , xi,

kD1

1 X

sk vk huk , yi.

kD1

Here, for simplicity’s sake, we use the same symbol h, i for inner products in different Hilbert spaces. Namely, hvk , xi is the inner product in X , while huk , yi is the inner product in Y . The elements ¹vk º form the orthonormal system, and therefore they are linearly independent. The equation A Ax D A y is then equivalent to the following relations hvk , xi D sk1 huk , yi,

k D 1, 2, : : : .

The Fourier coefficients hvk , xi, k D 1, 2, : : : , define an element of the Hilbert space X if and only if 1 X

hvk , xi D

kD1

2

1 X

sk2 huk , yi2 < 1.

kD1

The theorem is proved. Remark 2.2. The condition for the solution existence outlined in Proposition 2.2 is sometimes called the Picard criterion. The essence of this criterion is that the Fourier coefficients huk , yi of the right-hand side y should decay fast enough, much faster than sk ! 0, when k ! 1. Next we will measure the smoothness of x  against the decay of hvk , x  i D sk1 huk , yi to zero. Note also that Proposition 2.2 contains, in fact, the definition of the operator A acting from the space Dom.A / :D ¹y : y 2 Y ; Qy 2 Range.A/º to the space X . This operator is sometimes called the Moore–Penrose generalized inverse. It assigns to each y 2 Dom.A / the element x  D A y D

1 X

vk sk1 huk , yi,

kD1

which is the minimal norm solution of the equation A Ax D A y. So, for y 2 Dom.A / the Moore–Penrose generalized inverse can be formally represented as A D .A A/1 A . For a compact operator A with an infinite-dimensional rank Range.A/, A is a linear but unbounded operator from Dom.A / to X . This means that there is no stable

54

Chapter 2 Basics of single parameter regularization schemes

procedure for solving the equation A Ax D A y exactly. Even minor perturbation of y, for example, can change the solution dramatically. The most we can expect in this situation is to have a stable procedure for approximating the solution of the above equation. The representation A D .A A/1 A provides a clue for this. Namely, the unboundness of A caused by the multiplier .A A/1 , which can be considered to be a value of the operator-valued function t 1 at the “point” A A, but can be pointwise approximated by some bounded function g.t / in such a way that g.A A/A y will be sufficiently close to .A A/1 A y. To continue, the discussion will now need some facts from functional calculus (spectral calculus). First of all, we observe that A A and AA are compact, self-adjoint nonnegative operators. Indeed .A A/ D A .A / D A A, and 8u 2 X , hA Au, ui D hAu, Aui D kAuk2 . Recall that any compact self-adjoint nonnegative operator B admits the singular value expansion BD

rank.B/ X

k huk , iuk ,

kD1

with kBk D 1  2     k     > 0. The functional calculus for such operators isP based on the observation that for any rank.B/ bounded function f : Œ0, kBk ! R, the sum kD1 f .k /huk , iuk determines a linear bounded operator from X to X , and using the Parseval inequality we get   rank.B/   rank.B/   X   X     f .k /huk , iuk  D sup  f .k /uk huk , vi      kvk1 kD1 X!X kD1 X !1=2 rank.B/ X D sup Œf .k /2 huk , vi2 kvk1

 

kD1

sup

jf ./j sup

sup

jf ./j.

2Œ0,kBk

2Œ0,kBk

kvk1

rank.B/ X

!1=2 jhuk , vij

2

kD1

Keeping in mind that this is true for any self-adjoint, nonnegative and compact operator B, one can consider the operator f .B/ :D

rank.B/ X

f .k /huk , iuk

kD1

as a value of operator-valued function f at the operator B. Any bounded function f : Œ0, a ! R generates an operator-valued function f .B/ for all compact, self-adjoint,

55

Section 2.2 Essentially ill-posed linear operator equations

nonnegative operators B, such that kBk  a. This notion can be extended to the case of noncompact and unbounded, but still self-adjoint and nonnegative operators, but such an extension will not be used in the present monograph. We will use only the following general properties of operator-valued functions (1)

kf .B/kX!X 

(2)

f .B/g.B/ D .fg/.B/ D g.B/f .B/;

(3)

f .A A/A D A f .AA /;

(4)

kf .A A/A kY !X 

sup

2Œ0,kBk

jf ./j;

sup

p

2Œ0,kAk2 

jf ./j.

Note that .1/ has been proved and .2/ is obvious. Therefore we will prove only .3/ and .4/. Prank.A/ Let A D sk uk hvk , i be a singular value expansion of A, then A D kD1 Prank.A/ sk vk huk , i and kD1 

f .A A/ D

rank.A/ X



f .sk2 /hvk , ivk ,

f .AA / D

rank.A/ X

kD1

8 rank.A/ P ˆ ˆ   ˆ f .sk2 /sk huk , ivk < f .A A/A D kD1

rank.A/ ˆ P ˆ ˆ f .sk2 /sk huk , ivk : A f .AA / D

f .sk2 /huk , iuk ,

kD1

) f .A A/A D A f .AA /.

kD1

By the way, in this equality A can be replaced by A, and as a byproduct we have f .AA /A D Af .A A/. The proof of 4/ is now straightforward 



kf .A A/A kY !X

 rank.A/   X    2 D sup  f .sk /sk huk , vivk   kvk1  kD1 Y !1=2 rank.A/ X D sup jf .sk2 /sk j2 huk , vi2 kvk1

 .Note sk2 2 Œ0, kA Ak Œ0, kAk2 / 

kD1

sup sk jf .sk2 /j sup kvk1 k sup

2Œ0,kAk2 

p

rank.A/ X

!1=2 huk , vi

2

kD1

jf ./j.

We are now ready to continue our discussion about the approximation of the Moore– Penrose operator A D .A A/1 A by bounded operators of the form g.A A/A .

56

Chapter 2 Basics of single parameter regularization schemes

Recall that the idea is to choose a function g./ in such a way that for any righthand side y satisfying the Picard criterion the element A y D .A A/1 A y will be sufficiently close to g.A A/A y. At first glance, it would be enough to take g./ providing a good approximation of 1 for any  2 .0, kAk2 . However, such an approximation cannot be provided by a single function g./, because for any bounded g : j1  g./j ! 1 with  ! 0. Therefore we should use at least one parametric family ¹g˛ ./º of bounded functions, where ˛ is some parameter, which will go by the name of regularization parameter in the following, such that lim j1  g˛ ./j D 0,

˛!0

8 2 .0, kAk2 /.

Note that in principle one can take another parametrization of the family of bounded functions, say ¹gr ./º, and demand that lim j1  gr ./j D 0,

r!1

8 2 .0, kAk2 /.

However such a parametrization can be reduced to the previous one by changing the variable ˛ D r1 . Proposition 2.3. Let for all ˛ > 0, the functions g˛ ./ : Œ0, kAk2  ! R meet the following assumptions: (1) 9c : jg˛ ./j  c, (2) 8 2 .0, kAk2 , lim g˛ ./ D 1 . ˛!0

Then for any right-hand side y which meets the Picard criterion we have lim kA y  g˛ .A A/A ykX D 0.

˛!0

P Proof. Let A D 1 kD1 sk uk hvk , i be a singular value expansion of the operator A. Without loss of generality, we assume that rank.A/ D 1. The Picard criterion then has the form 1 X

sk2 huk , yi2 < 1,

kD1

and for any  > 0, there is a k0 D k0 ./, such that 1 X kDk0 C1

sk2 huk , yi2
0, 9˛k D ˛k ./, such that for ˛  ˛k and  D sk2 , ˇ2 ˇ ˇ ˇ1 2 ˇ ˇ . ˇ 2  g˛ .sk2 /ˇ  ˇ ˇ sk 2kA yk2 0 , then the above inequality is valid for all k D 1, 2, : : : , k0 and Let ˛  min¹˛k ºkkD1

X 1

D

k0 X kD1

"

#2 k0 X 2 1 2 2 2  g .s / s hu , yi  sk2 huk , yi2 ˛ k k k sk2 2kA yk2 kD1

2 2  2  kA yk  . 2kA yk2 2 Thus, for any  > 0, one can find ˛ D ˛., y/, such that kA y  g˛ .A A/A ykX  .

58

Chapter 2 Basics of single parameter regularization schemes

This means that lim kA y  g˛ .A A/A ykX D 0

˛!0

for any y which meets the Picard criterion. Comment on the notion of qualification of a regularization method. Under the conditions of Proposition 2.3, A y is the solution of the symmetrized equation A Ax D A y. Then kA y  g˛ .A A/A yk D k.I  g˛ .A A/A A/A yk  k.I  g˛ .A A/A A/kX!X kA ykX  ck.I  g˛ .A A/A A/kX!X c

sup

2Œ0,kAk2 

j1  g˛ ./j.

It is clear now that for the convergence of g˛ .A A/A y to A y with ˛ ! 0, it is enough to have lim

sup

˛!0 2Œ0,kAk2 

j1  g˛ ./j D 0.

This is nothing but the uniform convergence of g˛ ./ to 1 in the so-called weighted supnorm, with a weight   ˇ1 ˇ 1  ˇ ˇ    g˛ ./ D sup  ˇ  g˛ ./ˇ D sup j1  g˛ ./j .   2Œ0,a   It is easy to see that the pointwise convergence of g˛ ./ to 1 follows from the convergence in the weighted supnorm. However, considering convergence in such a norm, we can speak about the rate of convergence uniformly with respect to . It should be noted that the use of weighted norms is common in approximation theory when the approximation of functions with singularities is studied. In such a case, one usually uses the weights vanishing at the points of singularities (see, e.g., www.encyclopediaofmath. org/index.php/Weighted_space). The larger the vanishing order, the better the approximation obtained in the corresponding weighted norm. In the theory of ill-posed problems one is interested in the approximation of the function 1 which has singularity at zero. Therefore, using the approximation theory approach, it is natural to consider the weighted supnorms with weights of the form p . The rate of approximation of 1 by g˛ ./ in the weighted supnorm with weight p is measured against the rate ˛ p when ˛ ! 0, p  1. Definition 2.1. Qualification of the regularization generated by the family ¹g˛ ./º is the maximal p, such that sup

2Œ0,kAk2 

p j1  g˛ ./j  p ˛ p ,

where p depends only on p and g.

59

Section 2.2 Essentially ill-posed linear operator equations

Remark 2.3. If the family ¹g˛ º generates a regularization with the qualification p0 , and all g˛ are uniformly bounded in weighted supnorm with the weight , then the regularization generated by ¹g˛ º has a qualification p for any 1 < p  p0 . Indeed, uniform boundness in weighted supnorm with the weight  means that sup j1  g˛ ./j  1 C sup jg˛ ./j  c1 , 



where c1 does not depend on  and ˛. Then for any 1 < p  p0 p

p

p

sup j1  g˛ ./jp D sup j1  g˛ ./j p0  p0 p0 j1  g˛ ./j1 p0 





p p  sup j1  g˛ ./j1 p0 sup j1  g˛ ./jp0 p0 



1 p c1 p0 1 pp0

 c1



p0 ˛ p0

p p0 p0



pp



0

˛ p D p ˛ p .

The previous remark does not mean that for p0 > p, the norm corresponding to the weight p is weaker than the norm corresponding to the weight p0 , because sup jp0 g./j  sup p0 p sup jp g./j  ap0 p sup jp g./j.

2Œ0,a

2Œ0,a

2Œ0,a

2Œ0,a

p 2 Œ0, p0 , the strongest norm corTherefore, among the norms with weights responds to p D 0, the weakest is the norm with the weight p0 . Note also that the uniform boundness of the family ¹g˛ ./º in weighted supnorm with the weight  is one of the conditions of Proposition 2.3. However, as we will see in the following, the behavior of the weighted supnorm of g˛ with the weight 1=2 is very important. Usually, regularization methods are generated by families ¹g˛ º meeting the so-called “one-half” condition: ˇ ˇ 1=2 ˇ ˇ 9 1=2 : sup ˇ1=2 g˛ ./ˇ  p . 2 ˛ 2.0,kAk  Now we will consider some examples of regularization methods and indicate corresponding families generating these methods. p ,

Example 2.1 (Spectral cut-off method). This regularization method is generated by the family ¹g˛ ./º such that 8 1 < , ˛   < 1;  g˛ ./ D : 0, 0   < ˛. It is easy to see that sup jg˛ ./j D 1,

2Œ0,1/

60

Chapter 2 Basics of single parameter regularization schemes gα

0

α

λ

Figure 2.1. A generator of the spectral cut-off regularization method.

and for any 0 2 Œ0, 1/ and  > 0, there is ˛0 D 0 such that for any ˛ < ˛0 ˇ ˇ ˇ1 ˇ ˇ  g˛ .0 /ˇ D 0  . ˇ ˇ 0 This means that for any  2 Œ0, 1/,

ˇ ˇ ˇ1 ˇ ˇ lim ˇ  g˛ ./ˇˇ D 0. ˛!0 

Therefore, all conditions of Proposition 2.3 are fulfilled. Thus, for any right-hand side meeting the Picard criterion we have lim kA y  g˛ .A A/A ykY D 0.

˛!0

Moreover, for the considered family ¹g˛ º the “one-half” condition is satisfied with the constant 1=2 D 1: sup 1=2 g˛ ./ D

2.0,1/

sup

1

2Œ˛,1/ 1=2

1 p . ˛

To estimate qualification of the spectral cut-off regularization we observe that for any p0 sup p j1  g˛ ./j D sup p D ˛ p ,

2Œ0,1/

2Œ0,˛/

and there is no maximal p for which this equality is true. Therefore, the spectral cutoff regularization has infinite qualification. However, one should realize that this regularization method works only in cases where the singular system ¹sk , uk , vk º of the

61

Section 2.2 Essentially ill-posed linear operator equations

underlying operator A is known, because in the considered case g˛ .A A/ is nothing but a partial sum of the formal singular value expansion of .A A/1 and X g˛ .A A/A y D sk1 vk huk , yi. k:sk2 ˛

Example 2.2 (Tikhonov–Phillips regularization). This method is generated by the 1 family ¹g˛ ./ D ˛C º. First of all, it should be noted that for an implementation of this method it is not necessary to know the singular system of the underlying operator A because x˛ D g˛ .A A/A y D .˛I C A A/1 A y is nothing but the solution of the operator equation of the second kind ˛x C A Ax D A y which can be solved, at least numerically, without knowledge of the singular system. Moreover, it should be noted that the above-mentioned operator equation is the Euler equation of the variational problem 2 ! min, kAx  yk2Y C ˛kxkX

x2X

which provides one more approach to construct x˛ . It is also notable that because of the simplicity of its realization, Tikhonov–Phillips regularization is one of the most popular regularization techniques. It is easy to check the conditions of Proposition 2.3  sup jg˛ ./j D sup D 1; 2Œ0,1/ ˛ C  ˇ1 ˇ ˛ ˇ ˇ D 0. 8 2 .0, 1/, lim ˇ  g˛ ./ˇ D lim ˛!0  ˛!0 .˛ C / 2Œ0,1/

Moreover, for p > 0  p 0 pp1 .˛ C /  p p1  D D .p˛  .1  p//. ˛C .˛ C /2 .˛ C /2 This means that for p 2 .0, 1/ p D 2Œ0,kAk2  ˛ C  sup



p 1p

p

˛p ˛ 1p

D ˛ p1 p p .1  p/1p ,

while for p  1 and sufficiently small ˛ < kAk2  p 0 1 p  > 0 ) kAk2.p1/  sup  kAk2.p1/ . C˛ 2 2Œ0,kAk2  ˛ C 

62

Chapter 2 Basics of single parameter regularization schemes

Using these simple observations, one can easily check that g˛ ./ D “one-half” condition with the constant 1=2 D 12 sup

2Œ0,kAk2 

j1=2 g˛ ./j D

1 ˛C

meets the

1 1=2 D p 2  C ˛ 2Œ0,kAk  2 ˛ sup

and the qualification of the Tikhonov–Phillips method is equal to 1, because for p  1 p 1 ˛kAk2.p1/  sup p j1  g˛ ./j D sup ˛  ˛kAk2.p1/ , 2 2Œ0,kAk2  2Œ0,kAk2   C ˛ while for p 2 .0, 1/ sup

2Œ0,kAk2 

p j1  g˛ ./j D

p ˛ p D ˛ sup D ˛ p p p .1  p/1p . 2Œ0,kAk2   C ˛  C˛ sup

It is worth mentioning here that such a small qualification p D 1 is the main drawback of the Tikhonov–Phillips regularization, as will become clear in the following. Example 2.3 (Iterated Tikhonov regularization). Assume for the moment that we have some a priori information concerning the unknown element A y. Namely, we suppose that A y is close to some element x0 2 X . Roughly speaking, x0 is our initial guess for A y. In this situation it seems to be reasonable to alter the minimization problem corresponding to the Tikhonov–Phillips regularization as follows kAx  yk2 C ˛kx  x0 k2 ! min,

x 2 X.

Changing the variables x1 D x x0 , we arrive at the following minimization problem: kAx1  .y  Ax0 /k2 C ˛kx1 k2 ! min,

x1 2 X .

This is just the original Tikhonov functional, where y is replaced by y  Ax0 . From the previous example we know that the minimizer of such a functional is the solution of the equation ˛x1 C A Ax1 D A .y  Ax0 /. Keeping in mind that x1 D x  x0 , the solution of the new minimization problem with the initial guess x0 can be found from the equation ˛x C A Ax D ˛x0 C A y. In principle, the role of the initial guess x0 can play the Tikhonov–Phillips approximation x˛ D .˛I C A A/1 A y, which, in turn, corresponds to the initial guess x0 D 0. We can then repeat this procedure in such a way that an approximation obtained from

63

Section 2.2 Essentially ill-posed linear operator equations

the previous step will play the role of the initial guess for the next approximation. In this way, we construct the following iteration procedure: x0,˛ D 0; x1,˛ D x˛ ! ˛x C A Ax D A y; x2,˛ D : : : ! ˛x C A Ax D ˛x1,˛ C A y; . . . xm,˛ D : : : ! ˛x C A Ax D ˛xm1,˛ C A y. Here one should solve an operator equation of the second kind in each step. Using the representation of the Tikhonov–Phillips method in terms of operator-valued functions g˛ .A A/ we have x1,˛ D g˛ .A A/A y x2,˛ D g˛ .A A/.˛x1,˛ C A y/ D g˛ .A A/A y C ˛g˛2 .A A/A y . . . xm,˛ D

m X

˛ k1 g˛k .A A/A y D gm,˛ .A A/A y,

kD1

where gm,˛ ./ D

m X

˛ k1 g˛k ./ D g˛ ./

kD1

m1 X

˛ k g˛k ./

kD0

1  ˛ m g˛m ./ . D g˛ ./ 1  ˛g˛ ./  1  ˛g˛ ./ D C˛ , we obtain   1 ˛m 1 gm,˛ ./ D .  . C ˛/m

Keeping in mind that g˛ ./ D

1 , C˛

The main advantage of the iterated Tikhonov regularization is the possibility of obtaining a regularization method with arbitrary high qualification. Namely, for any p  m, one can use the same argument as in Example 2.2 to prove that ˛m sup p D sup p .1  . C ˛/1 /m . C ˛/m 2Œ0,1/  h p im D sup  m .1  g˛ .//

sup p j1  gm,˛ ./j D

2Œ0,1/

2.0,1/

64

Chapter 2 Basics of single parameter regularization schemes

  p  m p p 1 m p p m 1  ˛m m m  p p  p mp p 1 D ˛ . m m This means that after m-iteration of the Tikhonov–Phillips regularization method, we obtain a regularization method with qualification m. To check the “one-half” condition we note that k m1 m1  X 1 X m ˛ k k ˛ jg˛ ./j D  . jgm,˛ ./j  jg˛ ./j C˛ ˛C ˛ kD0

Then 1=2

sup 

2Œ0,1/

kD0

 ˛m jg˛,m ./j D sup 1=2 1  . C ˛/m      1=2  1=2 ˛m 1 ˛m D sup 1 1  .˛ C /m . C ˛/m     1=2 ˛m 1  sup 1  .˛ C /m  1



m1=2 D supŒg˛,m ./1=2  p . ˛  Example 2.4 (Landweber iteration). This is another example of an iteration procedure that is used as regularization. In the previous example the number of iteration steps was fixed and the regularization was governed by the parameter ˛. The distinguishing feature of the Landweber iterative regularization is that the number steps of iteration 1 , or m D ˛1 , where ˛1 is an m is used as a regularization parameter. So ˛ D m integer part of ˛1 . The iteration formula for the Landweber iteration is xn D xn1  A .Axn1  y/, n D 1, 2, : : : , where  is some fixed number from the interval .0, kAk2 /. It is easy to check that 1 , and xm D g˛ .A A/A y, where ˛ D m g˛ ./ D

m1 X

.1  /k  D

kD0

The “one-half” condition is checked as before

1 Œ1  .1  /m . 

p  1 1=2 jg˛ ./j  .m/1=2 D p , ˛ D . 2 m ˛ 2Œ0,kAk  sup

Section 2.3 Smoothness in the context of the problem

65

Moreover, straightforward calculations show that for any 0 < p < 1, sup

2Œ0,kAk2 

p j1  g˛ ./j  p mp D p ˛ p ,

where p D .p=e/p . Thus, the Landweber iteration can be considered a regularization method with arbitrary high qualification, but it should be noted that p ! 1 with p ! 1.

2.3

Smoothness in the context of the problem. Benchmark accuracy levels for deterministic and stochastic data noise models

In previous section we presented a general form of regularization methods which allow construction of approximate solutions converging to the Moore–Penrose generalized solution of an operator equation Ax D y, where A is an operator with nonclosed range. However, the above-mentioned convergence can only be provided in cases where a regularization method is applied to the exact right-hand side y. As was illustrated in the first chapter, in reality such exact information is very rarely available. At the same time, even minor perturbation of the right-hand side of the ill-posed operator equation can cause major perturbation of the solution. This means that in applying regularization methods to a perturbed righthand side, we cannot guarantee convergence to the solution. Roughly speaking, using a perturbed right-hand side, we generally cannot approach the solution closer than some fixed separation distance, which depends on the way a right-hand side has been perturbed, and of course on the amount of perturbation. In order to discuss this dependence, we should specify both above-mentioned items, which can be done in the form of the so-called noise-model. We will therefore discuss approximate solutions of operator equations Ax D y in the presence of noise, which means we are given yı D Ax C ı , where denotes the noise element which is normalized in some sense, and ı is a small positive number used for measuring the noise level (or the amount of perturbation as above). Then a regularization method generated by a family ¹g˛ ./º will give us approximate solutions x˛ı D g˛ .A A/A yı . Modern theory of ill-posed problems is dominated by two noise models. In one of them, the noise element is assumed to be chosen by some antagonistic opponent

66

Chapter 2 Basics of single parameter regularization schemes

subject to a constraint k kY  1. In other words, in order to recover the solution from noisy data yı , we should take into account that there is a troublemaker choosing in such a way that it will make as much trouble as possible. Within this model, the error between a regularized approximation x˛ı D g˛ .A A/A yı and the Moore–Penrose generalized solution x  D A y D .A A/1 A y will be measured as e det .A, x  , g˛ , ı/ D

sup

yı :kyyı kY ı

kx   x˛ı kX .

Sometimes this quantity is called the worst-case error. Another noise model, which is also widely accepted, reflects the stochastic nature of the perturbations. It is known as the Gaussian white noise model and supposes to be the so-called generalized random element such that (G.1) 8f 2 Y , hf , i is a Gaussian random variable on some probability space ¹ , F , º, i.e., hf , i D f .!/ : ! R. R (G.2) Ehf , i :D f .!/d.!/ D 0, 8f 2 Y . R (G.3) Ehf , i2 :D f2 .!/d.!/ D kf k2Y , 8f 2 Y . Conditions (G.1)–(G.3) mean that 8f 2 Y , 8t 2 R,

1  ¹hf , i  t º D p 2kf kY

Z

t

1

2

e

 2kfu k2

Y

du.

Moreover, from (G.3) it follows that 8f1 , f2 2 Y ,

Ehf1 , ihf2 , i D hf1 , f2 i.

Indeed,   kf1 C f2 k2 D Ehf1 C f2 , i2 D E hf1 , i2 C 2hf1 , ihf2 , i C hf2 , i2 D Ehf1 , i2 C 2Ehf1 , ihf2 , i C Ehf2 , i2 D kf1 k2 C kf2 k2 C 2Ehf1 , ihf2 , i kf1  f2 k2 D kf1 k2 C kf2 k2  2Ehf1 , ihf2 , i. Thus, 4Ehf1 , ihf2 , i D kf1 C f2 k2  kf1  f2 k2 D 4hf1 , f2 i.

67

Section 2.3 Smoothness in the context of the problem

At first glance, the description of Gaussian white noise model looks too formal. What is the idea behind it? Let ¹fk º be some orthonormal basis of a Hilbert space Y . Then the exact right-hand side of the equation Ax D y can be represented as yD

1 X

hfk , yifk .

kD1

Accepting the Gaussian white noise model we assume, in fact, that instead of the sequence of Fourier coefficients ¹hfk , yiº, we are dealing with a noisy sequence ¹hfk , yı iº, such that the deviations k D hfk , y  yı i D ıhfk , i,

k D 1, 2, : : : ,

are independent identically distributed (i.i.d.) Gaussian random variables such that E k D ıEhfk , i D 0,

E k2 D ı 2 Ehfk , i2 D ı 2 kfk k2 D ı 2 ,

and, moreover, the stochastic behavior of noisy Fourier coefficients does not depend on the chosen basis. By the way, within the Gaussian white noise model it is not assumed that is an element of the Hilbert space Y , because for any orthonormal basis ¹fk º of Y , Ehfk , i2 D kfk k2 D 1, k D 1, 2, : : :, while for 2 Y it would be limk!1 hfk , i2 D 0. Within the framework of the Gaussian white noise model, the error of the regularization method associated with the family ¹g˛ º is measured by the risk 1=2  2 e ran .A, x  , g˛ , ı/ D Ekx   x˛ı kX . As has been mentioned at the beginning of the section, using a perturbed right-hand side we can approach the solution of the original equation only up to a certain distance. This distance is nothing but the best possible accuracy of the approximation of the solution under the presence of noise with fixed intensity. However, the accuracy of the approximation, in turn, depends not only on the noise level, but also on the smoothness of the solution. In accordance with the paradigm of the Hilbert space theory (see, e.g., [113]), the smoothness of an element is measured by the rate at which its Fourier coefficients converge to zero with respect to some orthonormal basis. Which basis should be chosen? The Picard criterion gives us a hint for the answer. Recall that if the operator A has a singular value expansion Ax D

1 X

sk uk hvk , xi,

kD1

then the Picard criterion provides necessary and sufficient conditions for the Moore– Penrose generalized solution to be an element from X . Namely, x  D A y 2 X if

68

Chapter 2 Basics of single parameter regularization schemes

and only if 1 X

sk2 huk , yi2 < 1.

kD1

Roughly speaking, the Picard criterion guarantees that x  possesses the minimal possible smoothness, namely, it is in X . For more smoothness more restrictions must be imposed on the coefficients huk , yi. They should remain a square-summable sequence multiplied not only by sk2 , but also by sk2 ' 2 .sk2 /, where ' is some increasing function, such that '.0/ D 0, or, more precisely 1 X huk , yi2 < 1. sk2 ' 2 .sk2 /

kD1

If we put vD

1 X kD1

vk

huk , yi 2 X, sk '.sk2 /

then x  D A y D

1 X

vk sk1 huk , yi D

kD1

1 X

'.sk2 /hvk , vivk D '.A A/v.

kD1

Thus, additional smoothness of x  can be expressed in the form of inclusion ® ¯ x  2 A' .R/ :D x 2 X : x D '.A A/v, kvkX  R . In the literature, conditions of such a type are usually known as “source conditions”, or “source conditions generated by the index function '”.

2.3.1 The best possible accuracy for the deterministic noise model When dealing with noisy data yı D AxCı , one cannot distinguish the “true” solution x  from other elements meeting a source condition A' .R/. Therefore, the best possible accuracy which one can guarantee within the framework of deterministic noise models is measured as Eıdet .A' .R// D

sup

inf

x2A' .R/ g˛

sup

yı :kyyı kı y2N.A /C¹Axº

kx  g˛ .A A/A yı k,

where N.A / C ¹Axº D ¹u C Ax, u 2 N.A /º. Note that for any x 2 A' .R/ and y 2 N.A / C ¹Axº we have A y D x, because x 2 N.A/? and y D u C Ax, u 2 N.A /. Therefore, A y D .A A/1 A y D .A A/1 .A u C A Ax/ D .A A/1 .A A/x D x.

69

Section 2.3 Smoothness in the context of the problem

The quantity Eıdet .A' .R// just introduced can be estimated from below in terms of the so-called modulus of continuity of Moore–Penrose generalized inverse associated with A' .R/ !.ı, A' .R// :D sup¹kxkX : x 2 A' .R/, kAxkY  ıº. Proposition 2.4. Eıdet .A' .R//  !.ı, A' .R//. Proof. Let Q be the orthoprojector onto Range.A/. It is clear that N.Q/ N.A / because A Q D A . For any z 2 A' .R/ such that kAzkY  ı the set N.Q/ C ¹Azº contains the elements yN such that A yN D z, because, as above, QyN D Az, and z 2 N Then N.A /? . For any such yN we consider yNı D .I  Q/y. kyN  yNı kY D kQyk N Y D kAzkY  ı. Note also that A yNı D A .I  Q/yN D A yN  A yN D 0. Thus, Eıdet .A' .R//  D

sup

kz  g˛ .A A/A yNı k

sup

inf

sup

kzk D !.ı, A' .R//.

z2A' .R/ g˛ yN ı D.I Q/yN y2N.Q/C¹Azº N kAzkY ı z2A' .R/ kAzkY ı

Proposition 2.5 ([81]). If the index function ' is such that the function .t / D is strictly increasing, and ' 2 .. 2 /1 .t // is concave, then  2 ı 2 2 , ! .ı; A' .R// D R Sp R2

p

t '.t /

where Sp .t / is a piecewise linear function interpolating the values ' 2 .sk2 / at points  2 .sk2 /, i.e., Sp . 2 .sk2 // D ' 2 .sk2 /, and sk , k D 1, 2, : : : are the singular values of the operator A. Proof. If AD

1 X kD1

sk uk hvk , i,

70

Chapter 2 Basics of single parameter regularization schemes

then for x 2 A' .R/ we have xD

1 X

'.sk2 /vk hvk , vi, Ax D

kD1

1 X

sk '.sk2 /uk hvk , vi,

kD1

where v is such that kvkX  R, i.e., 1 X

hvk , vi2  R2 .

kD1

Moreover, kxk2 D

1 X

' 2 .sk2 /hvk , vi2 ; kAxk2 D

kD1

1 X

sk2 ' 2 .sk2 /hvk , vi2 .

kD1

Consider the sequences k D sk2 ' 2 .sk2 /, ˇk D ' 2 .sk2 /, k D 1, 2, : : : , 0 D ˇ0 D 0. It is easy to check that ˇk D ' 2 .. 2 /1 . k //, k D 1, 2, : : : because  2 .t / D t ' 2 .t / is strictly increasing and 0 D . 2 /1 .0/, so that   sk2 D . 2 /1 . k / ) ˇk D ' 2 . 2 /1 . k / ; k D 1, 2, : : : . Keeping in mind that the function ' 2 .. 2 /1 .t // is concave, one can consider the points with coordinates . k , ˇk / as vertex points of the convex polygon shown in Figure 2.2. β (γ2,β2) (γk,βk) (γ1,β1) 2

(γk+1,βk+1)

(γ0,β0)

ω (δ) R2

δ2 R2

Figure 2.2. Illustration of the proof of Proposition 2.5.

γ

71

Section 2.3 Smoothness in the context of the problem

Let us now consider one more sequence hvk , vi2 , k D 1, 2, : : : k D k .v/ D R2 1 X 0 D 1  k  0. kD1

Thus by construction, k  0 and ˇ.x/ D

P1

kD0 k

D 1. The values

1 1 X X kxk2 kAxk2 D  ˇ and .x/ D D k k k k R2 R2 kD0

kD0

can then be considered convex combinations of ¹ˇk º, ¹ k º respectively. This means that for x traveling within the set A' .R/, the point . .x/, ˇ.x// is traveling within the above mentioned convex polygon. One can now easily realize that the value ! 2 .ı, A' .R//=R2 is the maximal abscissa ˇ.x/ of the points . .x/, ˇ.x// belonging ı2 to this convex polygon, such that .x/  R 2. ı2  , i.e., 9: Let k be such that kC1  R 2 k 2

ı ı2 R2  k D  C .1  / ;  D . kC1 k R2 kC1  k Using Figure 2.2, it is easy to realize that the point with the maximal abscissa ! 2 .ı,A' .R// R2

belongs to the segment between points . kC1 , ˇkC1 /, . k , ˇk / and ! ! ı2 ı2   ! 2 .ı, A' .R// 2 2 k kC1 R R D ˇkC1 C .1  /ˇk D ˇkC1 C ˇk R2 kC1  k kC1  k  2 ı , D Sp R2

where Sp .t / is piecewise linear functions meeting interpolation conditions Sp . k / D ˇk , k D 1, 2, : : : or, similarly, Sp . 2 .sk2 // D ' 2 .sk2 /. p p Moreover, it is easy to check that for v D R vkC1 C R 1  vk , the following equalities hold true: kvk2 D R2  C R2 .1  / D R2 ; 2 kxk2 D k'.A A/vk2 D R2 ' 2 .skC1 / C R2 .1  /' 2 .sk2 / D ! 2 .ı, A' .R//; 2 2 ' 2 .skC1 / C R2 .1  /sk2 ' 2 .sk2 / D ı 2 . kAxk2 D R2 skC1

The proposition is proved.

72

Chapter 2 Basics of single parameter regularization schemes

Corollary 2.1. Assume that the index function ' meets the assumptions of Proposition 2.5 and obeys 2 -condition, i.e., there is  > 1 for which '.2t /  '.t /, 0  t  s kAk. Assume furthermore, that the singular numbers of an operator A obey kC1 sk  , 0 <  1. There is then a constant c,,R , such that !.ı, A' .R//  c,,R '. 1 .ı//. In particular c,,R can be taken as 2

c,,R D R log2 4R2 . Proof. For any q 2 .0, 1/ iterating 2 -condition one can find i , such that 1 , and 2i 1     t t i '.t /  '     '   i '.qt /. 2 2i

1 2i

q

Keeping in mind that i  log2 q2 , and  > 1, we have q

'.qt /   log2 2 '.t /, q 2 .0, 1/. As in the proof of Proposition 2.5 we consider k, such that 2  2 .skC1 /

Then



! .ı, A' .R//  R Sp 2

2

ı2 R2



ı2   2 .sk2 /. R2

  2 /  R2 Sp  2 .skC1

.interpolation property/ 2

2 /  R2 ' 2 . 2 sk2 /  R2  2 log2 2 ' 2 .sk2 / D R2 ' 2 .skC1   2     2 2 ı . D R2  2 log2 2 ' 2 . 2 /1  2 .sk2 /  R2  2 log2 2 ' 2 . 2 /1 R2 ı 1 . ı /, because Note that . 2 /1 . R 2/ D  R  2   ı2 ı ı ı 2 1 a D . 2 /1 .a/ D ) .a/ D ) a D  . )  2 2 R R R R 2

Therefore, !.ı, A' .R//  R

log2

2 2

   ı 1 '  . R

ı / D a, then Moreover, if b D  1 .ı/ >  1 . R r p ı .b/ b b'.b/ b RD ı D Dp ) a  2.  .a/ a R a'.a/ R

73

Section 2.3 Smoothness in the context of the problem

Thus, !.ı, A' .R//  R

log2

 R log2

2 2 2 2

     2 1 1 ı 1 log2 2 '  '  .ı/  R R R2 

 log2 2R2 '. 1 .ı// D R 2 log2 2R '. 1 .ı//. 1

The corollary is proved. As you can see, in the proof of Proposition 2.5, we essentially used the concavity of the function ' 2 .. 2 /1 .t //. Keeping in mind that ..'/2 .t //0 .log ' 2 .t //00 ' 4 .t / d 2 2  2 1  . ' / .t / D  C . 0 dt 2 .. /2 .t // .. /2 .t //0 One can conclude that a sufficient condition for the function ' 2 .. 2 /1 .t // to be concave is that the function log '.t / is concave. Example 2.5. Let us consider the index function '.t / D t  ,  > 0. It is clear that this function meets the 2 -condition with  D 2 , i.e., '.2t / D 2 t  D 2 '.t /. 2 Moreover, .t / D t C1=2 is strictly increasing,  1 .t / D t 2C1 . It is thus easy to see 2 1 that  2 .t / D t 2C1 , . 2 /1 .t / D t 2C1 , ' 2 .. 2 /1 .t // D t 2C1 , and the last function is concave. Therefore, one can apply Corollary 2.1 to any operator A with singular s .A/  > 0, and it yields the following lower estimate: numbers obeying kC1 s .A/ k

2

!.ı, A' .R//  R2 log2 4R2 '. 1 .ı// D

2.3.2

2 2 2 ı 2C1  ı 2C1 .  21 4 R

The best possible accuracy for the Gaussian white noise model

Recall that within the framework of the Gaussian white noise model we are given yı D y C ı , where is assumed to be the so-called Gaussian generalized random element, such that for any f1 , f2 2 Y Ehf1 , i D Ehf2 , i D 0, and Ehf1 , ihf2 , i D hf1 , f2 i. On the other hand, the Moore–Penrose generalized solution x  D A y which we are interested in can be represented as x D

1 X kD1

sk1 vk huk , yi,

74

Chapter 2 Basics of single parameter regularization schemes

where ¹sk , uk , vk º form the singular value expansion of A. Thus, our problem is to estimate the coefficients xk D hvk , x  i D sk1 huk , yi, k D 1, 2, : : : , from noisy observations yı D y C ı . These observations provide the following discrete information hk D xk C k , k D 1, 2, : : : , where hk D huk , yı isk1 , k D ıhuk , isk1 , and k , k D 1, 2, : : :, are Gaussian random variables such that E k D ısk1 Ehuk , i D 0; E k2 D k2 D ı 2 sk2 Ehuk , i2 D ı 2 sk2 . On the other hand, in view of the representation x˛ı D g˛ .A A/A yı D

1 X

g˛ .sk2 /sk vk huk , yı i D

kD1

1 X

g˛ .sk2 /sk2 hk vk

kD1

a regularization estimates the coefficients xk as xO k D gk hk , k D 1, 2, : : : , where gk D g˛ .sk2 /sk2 , and vice versa. Any estimation method estimating the coefficients xk from their noisy values hk , such that xO k D gk hk , k D 1, 2, : : :, can be viewed as a regularization x˛ı D g˛ .A A/A yı , where g˛ is any function such that g˛ .sk2 / D gk sk2 . Thus,   Eıran A' .R/ D

sup

inf e ran .A, x  , g˛ , ı/ D

x  2A' .R/ g˛

D

sup

P

sup

 1=2 inf Ekx   x˛ı k2

x  2A' .R/ g˛

 X 1=2 1 2 inf E .xk  gk .xk C k // ,

gk ¹xk º 2 k xk ak 1

kD1

where ak D ' 2 .sk2 /=R2 ; here we use one-to-one correspondence between the eleP1 2   ment x  2 AP ' .R/, x D P kD1 '.sk /vk hvk , vi and the values xk D hvk , x i D 2 2 2 2 '.sk /hvk , vi, k xk ak D k R hvk , vi  1. ConsideringP a lower bound for Eıran .A' .R//, one can assume without loss of gen2 erality that E. 1 kD1 .xk  gk .xk C k // / < 1. For any n we then have X  n n X 2 2 E .xk  gk .xk C k // D E .xk  gk .xk C k // kD1

kD1

X  1 2 E .xk  gk .xk C k // < 1. kD1

75

Section 2.3 Smoothness in the context of the problem

This means that

P1

kD1 E .xk

Eıran .A' .R//



sup

P



 gk .xk C k //2 < 1, and 1 X

inf

gk ¹xk º: 2 k xk ak 1

sup

E.xk  gk .xk C k //

kD1

1 X

¹x º: P 2k k xk ak 1

!1=2 2

kD1

!1=2 inf E.xk  gk .xk C k //2 gk

.

Using the properties of random variables k we have E.xk  gk .xk C k //2 D .1  gk /2 xk2 C 2.1  gk /gk xk E k Cgk2 E k2 „ ƒ‚ … D .1  gk /2 xk2 C gk2 k2 ;

D0

d E.xk  gk .xk C k //2 D 2.gk k2  .1  gk /xk2 /. dgk The minimum of the above expected value is then reached for gk D inf E.xk  gk .xk C k //2 D gk

sup

P

¹xk º: 2 k xk ak 1

1 X xk2 k2 kD1

, and

xk2 k2 . xk2 C k2

Finally, we arrive at the estimate Eıran .A' .R// 

xk2 2 xk Ck2

xk2 C k2

!1=2 ,

where ak D ' 2 .sk2 /=R2 , k2 D ı 2 sk2 , k D 1, 2, : : : depending only on A, R, ' and ı. In this way the problem of the lower bound is reduced to an optimization problem with one inequality constraint. Keeping in mind, however, that all partial derivatives of the function 1 X xk2 k2 f .x1 , x2 , : : : , xk , : : :/ D xk2 C k2 kD1

have the form 2xk k2 .xk2 C k2 /  2xk xk2 k2 2xk k4 @f D D , @xk .xk2 C k2 /2 .xk2 C k2 /2 and may vanish only for xk D 0, k D 1, 2, : : :, which is definitely not a point of the maximum, one can conclude that f may reach its maximum value only on the boundary of the set μ ´ 1 X 2 x k ak  1 , Q D ¹xk º : kD1

76

Chapter 2 Basics of single parameter regularization schemes

P 2 i.e., for ¹xk º, such that 1 1 kD1 ak xk D 0. This means that our problem is a problem of conditional maximum, and we can apply the method of Lagrange multipliers. Let us therefore consider ! 1 X 2 ak x k . L.x1 , x2 , : : : , xk , : : : , / D f .x1 , x2 , : : : , xk , : : :/ C  1  kD1

Our problem is then reduced to the system @L @L D 0, k D 1, 2, : : : , D 0, @xk @ which after the transformations 2xk k4 @f @L D  2xk ak  D 2  2xk ak  D 0, @xk @xk .xk C k2 /2 ! ! k2 k2 p p  ak  C ak  D 0, xk xk2 C k2 xk2 C k2 can be rewritten as follows ² 2 8 k ..ak /1=2  1/, ak < 1 ˆ 2 ˆ D x ˆ ˆ 0, ak  1 < k ˆ ˆ ˆ ˆ :

X

k2 ak ..ak /1=2  1/ D 1.

k:ak 0 x2A'.R/ ı2 C 2 f 2A .R1 / x2A' .R/

kAxk

From Lemma 2.4 it now follows that '. 1 .// . 1 .// p >0 ı2 C 2 1 Dı '. .ı// . 1 .ı//  cıR1 p  c'. 1 .ı// . 1 .ı//, 2ı

Eıran .A , A' /  cıR1 sup

and the required lower bound is proved. The corresponding upper bound follows from Proposition 2.14 or from Proposition 2.15. Note that within the framework of the deterministic noise model one can obtain the same estimation for the optimal error. To be more precise, we consider the quantities eıdet .A' , f , z/ D Eıdet .A , A' / D

sup

sup jhf , xi  hz, Ax C ı ij,

sup

inf eıdet .A' , f , z/.

x2A' .R/ kk1 f 2A .R1 / z

From Remark 2.6 it follows that for ˛ D  1 .ı/ Eıdet .A , A' / 

sup

f 2A .R1 /

eıdet .A' , f , z˛ /  c'. 1 .ı// . 1 .ı//.

To establish the corresponding lower bound one needs a deterministic analog of Lemma 2.5. Lemma 2.6. inf

sup

sup

c u2Œ, 2Œ,

jqu C c.u C /j D q min. ,  /.

112

Chapter 2 Basics of single parameter regularization schemes

We omit a proof of this lemma, since it is elementary. Now in the same spirit as in the proof of Proposition 2.16, we have ˇ ˇ  ˇ hz, iı ˇˇ det ˇ sup sup ˇhf , xit  hz, Axi t C eı .A', f , z/  sup hz, Axi ˇ x2A' .R/ t2Œ1,1 kk1   z h1 D kzk h1 2 Œ1, 1 ˇ ˇ  ˇ ˇ kzkı h 1 ˇ  sup sup sup ˇˇhf , xit  hz, Axi t C hz, Axi ˇ A' .R/ t2Œ1,1 h1 2Œ1,1 ! h1 ı 1 kzkı  kAxk h D hhz,Axi ı h 2 Œ ,  ,   kAxk D

sup

sup

sup

x2A' .R/ t2Œ1,1 h2Œ,

jhf , xit  hz, Axi.t C h/j .

The application of Lemma 2.6 allows us to continue as follows: eıdet .A' , f , z/  

sup

jhf , xit C c.t C h/j

sup

inf sup

sup

jhf , xij min¹1,  º 

x2A' .R/ c t2Œ1,1 h2Œ,

x2A' .R/

sup

x2A' .R/

² jhf , xij min 1,

³ ı . kAxk

Keeping in mind that this estimation is valid for any z and using Lemma 2.4 we have ³ ² ı Eıdet .A , A' /  sup sup jhf , xij min 1, kAxk x2A' .R/ f 2A .R1 / ² ³ ı  D sup k .A A/xk min 1, kAxk x2A' .R/ ² ³ ı  sup k .A A/xk min 1, kAxk x2A' .R/ kAxkı .Dı/

 c'. 1 .ı// . 1 .ı//.

Thus, the following proposition has been proved. Proposition 2.17. Under the condition of Proposition 2.16 Eıdet .A , A' /  Eıran .A , A' /  '. 1 .ı// . 1 .ı//.

Section 2.7 Regularization by finite-dimensional approximation

2.7

113

Regularization by finite-dimensional approximation

In this section we continue the discussion on regularization by discretization begun in Subsection 1.3.2 of the introduction using examples. Let X and Y be two Hilbert spaces, and A 2 L.X , Y / be a compact operator with singular value expansion AD

1 X

sj uj hvj , i.

j D1

Proposition 2.2 tells us that the Moore–Penrose generalized solution x  for the equation Ax D y is determined from the equation A Ax D A y and the latter is solvable in X if the so-called Picard criterion is satisfied, i.e., 1 X

sj2 huj , yi2 < 1.

j D1

As long as we cannot meet this criterion with any y 2 Y , we are in need of regularization. At the same time, if the operator A is replaced by some finite-dimensional operator An , rank.An / D n, then the equation An An x D An y will be solvable for any y, even when one substitutes noisy data yı instead of y. In the latter case, the minimal norm solution xnı of the equation An An x D An yı can be considered as a regularized approximation for the Moore–Penrose generalized solution x  of the initial equation Ax D y. In this way, the regularization effect can be achieved by a finite-dimensional approximation alone, if the dimension n plays the role of a regularization parameter. Such a method of regularization is not covered by the general schemes discussed in previous sections. We therefore propose to analyze its advantages and disadvantages. We are going to do this within the framework of the Gaussian white noise model, because analysis is simpler than with the deterministic noise model. We will use the following fact (see [99] p. 59). Fact 2.1 (Concentration inequality). Let be a generalized Gaussian random element in a Hilbert space Y (the definition can be found in Section 2.3). Consider a linear

114

Chapter 2 Basics of single parameter regularization schemes

compact operator B 2 L.X , Y / with singular value expansion BD

1 X

sj .B/fj hgj , i,

j D1

such that 1 X

sj2 .B/ < 1.

j D1

Note that the operators possessing such a property are called kernel-type operators. Then, by the very definition of we have X  X 1 1 1 X sj2 .B/hgj , i2 D sj2 .B/Ehgj , i2 D sj2 .B/. EkB k2 D E j D1

Keeping in mind that B  B D

j D1

P1

2 j D1 sj .B/gj hgj , i,

j D1

it can be written as

EkB k2 D tr.B  B/. The fact which will be used in the following is that for any compact operator B 2 L.X , Y / and for any t > 0  ¹kB kY > t º  4 exp.t 2 =8tr.B  B//. We now begin our analysis of the regularization by finite-dimensional approximation. By definition xnı D .An An /1 An yı D .An An /1 An y C ı.An An /1 An . The term xn0 D .An An /1 An y is noise-free. This is nothing but the approximate solution which would be obtained within the framework of a finite-dimensional scheme applied to “pure” data y. The norm kx   xn0 kX is therefore just an approximation error which depends only on the approximation power of the space Range.An / containing the approximate solution, as well as on the smoothness properties of the exact solution x  itself. In order to illustrate this point we consider the following example. Example 2.10. Consider a family of orthonormal systems ¹ein ºniD1 , n D 1, 2, : : : , ein 2 Y , hein , ejn i D ıij , and let Qn D

n X iD1

ein hein , i

Section 2.7 Regularization by finite-dimensional approximation

115

be an orthoprojector from Y onto Yn D span¹ein ºniD1 . For An D Qn A the equation An An x D An yı can be written as A Qn Ax D A Qn yı , and its minimal norm solution xnı D arg min¹kQn Ax  Qn yı k2 , x 2 X º belongs to the space Xn D span¹A ein ºniD1 . Let us denote by Pn the orthoprojector from X onto Xn . Then An x D Qn Ax D

n X

n X

ein hein , AxiY D

iD1

D

n X

ein hA ein , xiX

iD1

ein hPn A ein , xi D

iD1

n X

ein hein , APn xi D An Pn x.

iD1

Thus, in the considered case for y D Ax  we have xn0 D .An An /1 An y D .An An /1 An Ax  D .An An /1 An An x   .An An /1 An .An  A/x  D .An An /1 An An Pn x   .An An /1 An Qn .An  A/x  D Pn x   .An An /1 An .Qn An  Qn A/x  D Pn x  , and kx   xn0 k D kx   Pn x  k D inf kx   uk, u2Xn

this is the best approximation of x  by elements from Xn D Range.An /. This best approximation depends really on the smoothness of x  and on the approximation power of the space span¹A ein ºniD1 . Therefore in general we will assume that there is a continuous, positive and monotonically decreasing function : Œ0, 1/ ! Œ0, d , such that kx   xn0 kX 

.n/.

Let us now discuss the second term of the representation xnı D xn0 C ın ,

n D .An An /1 An .

This term depends only on the operator An and on the noise. It does not depend on the unknown solution, but is random and if An has a singular value expansion An D

n X j D1

sn,j ujn hvjn , i

116

Chapter 2 Basics of single parameter regularization schemes

then Ekn k2 D Ek

n X

1 n n sn,j vj huj , ik2

j D1 n X 2 D sn,j Ehujn , i2 j D1

D

n X

2 sn,j .

j D1

It is easy to see that the latter quantity is nothing but the trace tr..An An /1 /. It is well known that the trace is invariant of a matrix representation of the operator. It is therefore not necessary to know the singular value expansion of An in order to estimate the trace. To make it more clear, consider this example. P P Example 2.11. Let Pn D nkD1 hfk , ifk and Qn D niD1 ei hei , i be two orthoprojectors in the spaces X and Y respectively. Consider the case that An D Qn APn D

n X n X

ei hei , Afk ihfk , i.

iD1 kD1

Then An An D

n n X X

akl fk hfl , i,

akl D

n X

hei , Afk ihei , Afl i.

iD1

kD1 lD1

In any way, within the framework of regularization by finite-dimensional approximation one should solve the equation An An x D An yı which corresponds to a linear system with a matrix A D ¹akl ºnk,lD1 . If it is done by constructing the inverse matrix A1 D ¹akl ºnk,lD1 , then .An An /1

D

n n X X

akl fk hfl , i

kD1 lD1

and tr..An An /1 / D

n X

akk D

kD1

n X

2 sn,j

j D1

can be calculated during the numerical procedure. If the system corresponding to the equation An An x D An yı is solved without inestimated by the Monte–Carlo method. verting the matrix A then tr..An An /1 / can beP The idea consists in using the elements b D nkD1 k fk , where k , k D 1, 2, : : : , n are independent identically distributed random variables with E k D 0, E k2 D 1,

117

Section 2.7 Regularization by finite-dimensional approximation

k D 1, 2, : : :. Modern software allows effective simulation of such random variables. The solution of the equation An An x D b is then also a random element that can be formally represented as xD

.An An /1 b

D

n n X X

akl fk hfl , bi D

kD1 lD1

n n X X

akl l fk .

kD1 lD1

Now one can observe that

X  n n X Ehb, xi D E k akl l kD1

D D

n n X X

lD1

akl E k l

kD1 lD1 n X

akk D tr..An An /1 /.

kD1

Thus, simulating a random vector m-times, D . 1 , 2 , : : : , n / we obtain a sequence of random right-hand sides b .j / D

n X

.j /

k fk , j D 1, 2, : : : , m,

kD1 .j /

where k is a realization of the random variable k obtained in j -th simulation step. Pn .j / This sequence ¹b .j / ºjmD1 gives the sequence of solutions x .j / D kD1 xk fk of equations An An x D b .j / . One can then easily calculate the inner products hx

.j /

,b

.j /

iD

n X

.j / .j /

xk k ,

kD1

Pm

1 .j / , b .j / i can be considered an empirical estimation and the quantity tm D m j D1 hx  1 for tr.An An / . Note that all calculations mentioned above can be done in advance, without using data yı . Moreover, in practice one usually simulates a random right-hand side only once. The trace can therefore be estimated by running the solution procedure for the system corresponding to the equation An An x D u twice. Once with u D An yı in order to construct xnı , and the second time with u D b .1/ in order to estimate the trace tr..An An /1 / for any fixed n. Thus, it is natural to assume that there is a continuous, positive and monotonically decreasing function  : Œ1, 1/ ! Œ0, d , such that

Ekn k2 D Ek.An An /1 An k2 

1 2 .n/

.

118

Chapter 2 Basics of single parameter regularization schemes

The previous example tells us that .n/ can be known a priori, or it can be estimated within the numerical procedure framework. Due to monotonicity, the inverse function 1 : .0, d  ! Œ1, 1/ exists, and we will assume that for sufficiently small u 1 ln 1 .u/  ln . u A situation in which this does not hold has been considered in Section 1.3, in particular for the deterministic noise model (see estimation (1.64)). In the stochastic noise model case such a situation has been analyzed in [15]. At the same time, in many interesting practical situations .u/ decreases with a power rate, as is the case for the numerical differentiation problem considered at the end of this chapter. The above-mentioned assumption then holds true. Proposition 2.18. If there is a constant c1 , such that 8u 2 Œ1, 1/, .uC1/  c1 .u/, then under the above-mentioned assumptions about x  , xn0 , , ,  for ² ³ ı nopt D min n : .n/  , n D 1, 2, : : : , .n/ it holds that 



Ekx 

1=2 xnı opt k2

p 

2

c1

  . /1 .ı/ .

Proof. First of all we obtain an expression for Ekx   xnı k2 . Note that kx   xnı k2 D hx   xn0  ın , x   xn0  ın i D kx   xn0 k2  2ıhx   xn0 , n i C ı 2 kn k2 . From the property (G.2) of Gaussian white noise in Section 2.3 it follows that Ehx   xn0 , n i D Ehx   xn0 , .An An /1 An i D EhAn .An An /1 .x   xn0 /, i D 0. Then for any n Ekx   xnı k2 D kx   xn0 k2 C ı 2 Ekn k2 

2

.n/ C

ı2 . 2 .n/ 2

As usual, the optimal choice of n balances the values of 2 .n/ (bias) and 2ı.n/ (variance), but keeping in mind that n takes only integer values, one cannot take nopt D u0 D . /1 .ı/,

119

Section 2.7 Regularization by finite-dimensional approximation

because in general the latter value is not integer. So, in general the ideal choice for n is ² ³ ı . nopt D min n : .n/  .n/ Thus, .nopt /.nopt /  ı D

.u0 /.u0 /

.nopt  1/.nopt  1/  ı D

.u0 /.u0 /.

Keeping in mind that .u/, .u/ are monotonically decreasing functions, from these inequalities one can conclude that u0  nopt  u0 C 1. Then Ekx   xnı opt k2 

2

2

ı2 2 .nopt /

2 .u /2 .u / ı2 0 0  2 2 .nopt / 2 .u0 C 1/

2 D

.nopt / C

2 c12

2 .u

2 0 / .u0 / c12 2 .u0 / 2

D

2 c12

2

.u0 /

.. /1 .ı//.

The proposition is proved. Remark 2.8. The optimal choice of n indicated in Proposition 2.18 can be realized only when ,  and ı are known a priori. The estimation of  was discussed in Example 2.11. Therefore  can be assumed to be known. As to , it is rarely known in practice. Therefore Proposition 2.18 provides only a benchmark for the accuracy of regularization by finite-dimensional approximation. We are now going to discuss the criterion for the choice of n that will provide accuracy almost coinciding with the above-mentioned benchmark in the sense of order. First of all we restrict our attention to the final set ® ¯ ƒı D n : n D 1, 2, : : : , N , N  1 .ı/ , because for n > 1 .ı/, we have .n/ < ı ( is a decreasing function), and the estimate for the random component of the error ı 2 Ekn k2  becomes trivial,

ı2 2 .n/

ı2 2 .n/

 1, and gives the value which is far from optimal.

120

Chapter 2 Basics of single parameter regularization schemes

As has already been observed in the proof of Proposition 2.18, nopt belongs to the area, where the approximation error kx   xn0 k estimated by .n/ is dominated by the component caused by data noise. If one can describe this area without using information concerning , then the minimal n belonging to such an area would be a good substitute for nopt . The problem is, however, that the behavior of the component caused by data noise cannot be specified completely even for known  and ı because of ran domness . We mean that the inequality Ekn k2  21.n/ does not allow us to say that 

1 kn k  .n/ for all n. Nevertheless, as we will see in the following, for sufficiently large , which does not depend on n, the estimate  kn k  .n/

holds with large probability. Namely, the operator B D .An An /1 An is compact because it is finite-dimensional. Moreover, 1 tr.B  B/ D Ek.An An /1 An k2  2 .  .n/ 

Then from the concentration inequality (Fact 2.1 above) it follows that for n D B and any n it holds that   ± ° t2   kn k > t  4 exp  8tr.B  B/  2 2  t  .n/ .  4 exp  8 For t D

 .n/

it can be written as ± ° 2  kn k.n/ >   4e  8 ,

and now it is easy to see that by choosing a large enough  we can make the indicated probability as small as we wish. We will use it later in a rigorous proof. Now we are only going to give an idea of the criterion for selecting n that is close to nopt . Let us assume that we have chosen  in such a way that for n 2 ƒı the estimation  kn k  .n/ holds true with large probability. If n < m and both of them belong to the area where .n/ is dominated by the stochastic error caused by data noise then ı ı kxnı  xm k  kx   xnı k C kx   xm k 0 k C ıkm k  kx   xn0 k C ıkn k C kx   xm ı ı  .n/ C C .m/ C .n/ .m/ ı ı ı 2 C2 4 . .n/ .m/ .m/

Section 2.7 Regularization by finite-dimensional approximation

121

Using this observation we can formulate our criterion as follows: ² ³ 4ı ı ı 1 , 8m D n, n C 1, : : : , N D Œ .ı/ . n D min n : kxn  xm k  .m/ Note that n is well-defined; the set in the brackets is not empty, at least the maximal N D Œ1 .ı/ belongs to it. Note also, that the unknown .n/ is not involved in the choice of n . This choice depends only on , ı which are assumed to be known, and on the design parameter . The next statement relates this design parameter to ı and , so that actually n depends only on , ı and, of course, on random noisy data yı . p Proposition 2.19. If ı is small enough, one can always choose  D 4 p ln 1 .ı/  ln1=2 ı1 , and the constant p in such a way that 2 2 Œ1 .ı/pC1 D ı 2 11

and

r 1=2  1  ı 2 1 Ekx  xn k  c .. / .ı// ln , ı where c does not depend on ı. Proof. First of all we would like to recall that xnı  D xnı  . .!// D xnı  .!/ is a random element defined on a probability space . , F , /, so that Z kx   xnı  .!/k2 d.!/. Ekx   xnı  k2 D

If we introduce one more random variable „ D „ . .!// D max kn k.n/ n2ƒı

we can divide into two subspaces

 D ¹! 2 : „  º,

 D ¹! 2 : „ > º.

For ! 2  we can rely on the estimate ı ı .!/k C kxnı   xN .!/k kx   xnı  .!/k  kx   xN ı   .N / C ıkN k C 4 .N / 

D

ıkN k.N / ı .N / C C4 .N / .N / 

2

ıkN k.N / ı C4 .N / .N /   N  1 .ı/, .N /  ı 1  ı1 .N / 

 2kN k.N / C 4  6„ .

122

Chapter 2 Basics of single parameter regularization schemes

Let now ! 2  and nopt be the same as in Proposition 2.18, i.e., ² ³ ı . nopt D min n : .n/  .n/ Then for any n  nopt kxnı  xnı opt k  kx   xnı opt k C kx   xnı k .nopt / C ıknopt k C .n/ C ıkn k ı ı C .n/ C  .nopt / C .nopt / .n/ ı ı ı ı C C C  .nopt / .nopt / .n/ .n/ ı 4 . .n/ 

This means that for ! 2  , nopt  n , because n is the minimal n for which the above inequality holds. Thus, for ! 2  we have kx   xnı  k  kx   xnı opt k C kxnı   xnı opt k ı ı  .nopt / C C4 .nopt / .nopt / ı 6 .nopt / 6   .. /1 .ı//. c1 Using these estimations we can continue as follows: Z Z Ekx   xnı  .!/k2 D kx   xnı  k2 d.!/ C kx   xnı  k2 d.!/



36 2  c12

2

.. /1 .ı// C 36

Z





„2 .!/d.!/.

Now we look closer at the second integral, ²Z ³1=2 Z Z 2 4 „ .!/d.!/  „ .!/d.!/



®



¯1=2  ¹! E.„4 /

By definition,

Z E.„4 / D

0

1



1=2 2  º .

 4 dF . /,

1=2 d.!/

123

Section 2.7 Regularization by finite-dimensional approximation

where F . / is a distribution function of the random variable „ , i.e., F . / D ¹„   º. At the same time, using our previous observation, we can estimate the function G. / D 1  F . / D ¹„ >  º ² ³  D  ! 2 : max kn k.n/ >  n2ƒı ° ±  card.ƒı / ! 2 : kn k.n/ >  2

 4Ne  8 . For  D  it gives us, in particular 2

¹! 2  º  4Ne  8 . Moreover, integrating by part we obtain Z 1 Z 1 4 4 E.„ / D  dF . / D   4 d.1  F . // 0 0 Z 1 4 1 G. / 3 d  D  G. /j0 C 4 0 Z 1  2 G. /d  2 D2 0 Z 1 2  8N  2e 8 d  2 Z0 1 D 83 N ue u du 0 Z 1 3 u 1 3 e u du D 83 N . D 8 N ue j0 C 8 N 0

Thus,

Z

2

„2 d.!/  2 2 Ne  16  2 2 1 .ı/e  11

11

16p ln 1 .ı/ 16

pC1 11   2 2 1 .ı/ .

Keeping in mind that ln 1 .ı/  ln ı1 , one can find a constant c0 such that ln 1 .ı/  c0 ln ı1 )

pC1 1 .ı/  .ı/c0 ) 1 .ı/  ı .p1/c0 . This means that for sufficiently small ı, one always can find p such that

pC1 11 11 2 2 1 .ı/  2 2 ı .p1/c0  ı 2 .

124

Chapter 2 Basics of single parameter regularization schemes

Summing up, we arrive at the estimate 

Ekx 

xnı  .!/k2



2  36 2 c1 

72 2 c12

2

2

1

.. / .ı// C ı

2

.. /1 .ı//,

or, similarly, 1=2 24p2  p  ı 2  .. /1 .ı// p ln 1 .ı/ Ekx  xn .!/k c1 r 1  c .. /1 .ı// ln . ı The proposition is proved. Remark 2.9. Comparison q of estimates from Propositions 2.18 and 2.19 allows an interpretation of the factor ln ı1 , enlarging the order of the risk, as a payment for adaptation, payment for the lack of knowledge. At the same time, a close look at the proof of Proposition 2.19 allows the conclusion that there will be no additional logfactor for the deterministic noise model (this is because of the set  , which is simply empty for the deterministic noise). In any case, the proposed criterion for adaptive selection of n allows us to keep the order of accuracy at least in the power scale, because for .n/  nr , .n/  nˇ r r r r 1 1 1 1 rCˇ rCˇ .. / .ı//  ı ; .. / .ı// ln  ı ln ı ı and 8 > 0

r lim ı 

ı!0

ln

1 D 0. ı

2.8 Model selection based on indirect observation in Gaussian white noise The problem of model selection is one of the basic problems in statistical inference. Here, by “model” we have in mind any possible finite-dimensional linear space. In the spirit of the paper by Barron & Cover [10], the dimension of such a space is interpreted as the description length of the corresponding model. The goal is to obtain an accurate estimate for the unknown element (function) x on the basis of a finite family of models ¹Fn º, n D 1, 2, : : : , N . We do not mean that x belongs to any of the models, although this might be the case. Therefore, we shall always think of a model Fn as an approximate space for the true x with controlled complexity, i.e., the description length. The

Section 2.8 Model selection based on indirect observation in Gaussian white noise

125

case study involves estimation of an unknown x from indirectly observed data near y D Ax, where A is some compact linear operator acting from some Hilbert space X , x 2 X , to a Hilbert space Y , and describing how the observations are indirect. In practice indirect observations are usually observed in the presence of some noise, so that we observe yı given by yı D Ax C ı , where, as before, denotes a normalized noise, and ı is a small positive number used for measuring the noise level. The problem of estimating x from yı becomes a statistical one when is assumed to be zero-mean Gaussian noise, for example, meeting the conditions (G.1)–(G.2) from Section 2.3. In this case, for each model Fn , n D 1, 2, : : : , N one builds an estimator xnı D xnı .!/ based on noisy data yı , and the risk of this at the true x is given as 1=2  e ran .A, x, xnı / D Ekx  xnı .!/k2 . An ideal model from the list ¹Fn º should minimize the risk when n varies. Since x is unknown one cannot determine such an ideal model exactly. Therefore, we should content ourselves with considering, instead of the minimal risk, some accuracy index of the form eıran .A' .R/, Fn / D

sup

inf

ı x2A' .R/ xn ./2Fn

e ran .A, x, xnı /,

and in finding a suitable list ¹Fn º of the models which can guarantee that for given ' inf

¹Fn ºN nD1

eıran .A' .R/, Fn /  Eıran .A' .R//,

i.e., the best possible order of accuracy can be reached within the framework of the models from the list ¹Fn º. If we are given such a list of models then the aim is to find the simplest one that provides the best possible order of accuracy by choosing its description length to be as short as possible. Assume, just for simplicity, that the models are ordered in the list ¹Fn º in accordance with their description length, i.e., d i mFn D n, n D 1, 2, : : : , N . Then the above mentioned minimal description length is determined as ® ¯ nı .A' .R// D min n : eıran .A' .R/, Fn /  cEıran .A' .R// . Of course, this quantity depends on the constant c, but we will not specify this in the following, because we are interested only in the order of nı .A' .R// expressed in the terms of ı and '. Let PFn be an orthoprojector of X onto the finite-dimensional subspace Fn X . It is well known that for any x 2 X inf kx  ukX D kx  PFn xk.

u2Fn

126

Chapter 2 Basics of single parameter regularization schemes

Then

³1=2

²Z eıran .A' .R/, Fn /

D

sup

inf

ı x2A' .R/ xn ./2Fn

²Z

 D 

sup

x2A' .R/

sup

x2A' .R/

inf





kx 

xnı . .!//k2 d.!/

³1=2 kx  PFn xk2 d.!/

kx  PFn xk sup

inf kx  uk

Fn X x2A' .R/ u2Fn dimFn Dn

D dn .A' .R/, X /. The latter quantity is well-known as Kolmogorov’s n-width of A' .R/. The following fact is also widely known. Fact 2.2 (Propositions 11.6.2, 11.3.3. [135]). Let B be a compact operator acting between Hilbert spaces X and Y , and having singular value expansion BD

1 X

sj .B/uj hvj , iX .

j D1

Then inf

sup

inf kx  uk D RsnC1 .B/.

Fn Y xDBv u2Fn dimFn Dn kvkX R

From this fact it is immediately clear that for B D '.A A/ we have 2 dn .A' .R/, X / D R'.snC1 /,

where snC1 is .n C 1/-th singular value of the operator A. Proposition 2.20. Assume that an operator A and an index function ' meet the conditions of Proposition 2.6, and Proposition 2.9. In particular sn  nr . Then

1=2r nı .A' .R//  c',R,r r1 .ı/ , where r .t / D '.t /t

2rC1 4r

, and the constant c',R,r depends only on ', R and r.

Proof. From the assumption that ' meets the 2 -condition one can derive an estimation 2 eıran .A' .R/, Fn /  R'.snC1 /  '.cN',R n2r /

Section 2.8 Model selection based on indirect observation in Gaussian white noise

127

with the constant c',R depending only on ' and R. Moreover, from Proposition 2.9 it follows that Eıran .A' .R//  c'.r1 .ı//  '.cN',R r1 .ı//. Then ® ¯ nı .A' .R// D min n : eıran .A' .R/, Fn /  cEıran .A' .R// ¯ ®  min n : '.cN',R n2r /  '.cNN',R r1 .ı// ¯ ® D min n : cN',R n2r  cN',R r1 .ı/  1 cN',R 2r 1  2r1 r .ı/  cN',R

 1 D c',R,r r1 .ı/ 2r . The proposition is proved.

2.8.1

Linear models given by least-squares methods

In practice even indirect observations yı D Ax C ı cannot be observed exactly, they can only be observed in discretized or binned form. n n ºj D1 2 Rn , n D 1, 2, : : : , N , To be more precise, we have only a vector vn .yı / D ¹yı,j defined by n D hyı , ujn iY D hAx, ujn iY C ıh , ujn iY , j D 1, 2, : : : , n yı,j

where ujn , j D 1, 2, : : : , n is some orthonormal system, usually called design. When fixing a design ¹ujn ºjnD1 we may rewrite discretized noisy indirect observations as Qn yı D Qn .Ax C ı /, where Qn denotes the orthogonal projection onto Un D span¹un1 , un2 , : : : , unn º. Note that the latter equation is a particular case of a finite-dimensional approximation, as was discussed in Example 2.11 from the previous section. Thus, using discretized noisy observations, one can take as estimator an element xnı D arg min¹kQn Av  Qn yı k2 , v 2 X º

128

Chapter 2 Basics of single parameter regularization schemes

that solves the equation A Qn Axnı D A Qn yı and has the form of a linear combination xnı D

n X

cj A ujn ,

j D1

where the vector ¹cj ºjnD1 is the solution of the following system of linear algebraic equations n X

cj hA ujn , A ujn i D huni , yı i, i D 1, 2, : : : , n.

j D1

In other words, a finite-dimensional least-square approximation based on the family of design spaces Un D span¹ujn ºjnD1 , n D 1, 2, : : : , N gives us the list of linear models Fn D Fn .A/ D span¹A uni ºniD1 , n D 1, 2, : : : , N , i.e., the linear finite-dimensional spaces containing the estimators xnı , n D 1, 2, : : : , N . In principle, the criterion for the adaptive choice of dimension n proposed in Proposition 2.19, can be used as a procedure for model selection. But to use it one should control the sum n X

2 sn,j

j D1

for each n D 1, 2, : : : , N , where sn,j is j th singular value of An D Qn A. At this point it should be noted that the necessity of controlling the above-mentioned sum is basically the main drawback of regularization by finite-dimensional approximation, because as can be seen from Example 2.10, this increases computational costs. Even if we know, for example, that the singular values sj of the underlying operator A are such that c1 j r  sj  c2 j r , but the design elements ¹uni º are fixed and do not coincide with the elements from singular value expansion of A, then the abovementioned sum can be controlled in general only from below, because sn,j  sj ) cn

2rC1



n X j D1

sj2



n X

2 sn,j .

j D1

Indeed, the operators Qn A and A Qn D .Qn A/ have the same singular values sn,j . 2 is the eigenvalue of the operator Qn AA Qn , and according to the Moreover, sn,j

Section 2.8 Model selection based on indirect observation in Gaussian white noise

129

maximum principle for eigenvalues of self-adjoint, nonnegative operators it holds that 2 D sn,j



min

vi 2Y iD1,2,:::,j 1

min

vi 2Y iD1,2,:::,j 1

max

hQn AA Qn u, ui

max

hAA u, ui,

u2Y ,kukD1 hu,vi iD0, iD1,2,:::,j 1

u2Y ,kukD1 hu,vi iD0, iD1,2,:::,j 1

where the latter quantity is just sj2 by definition. Knowledge of the asymptotics of sj therefore does not guarantee a good a priori 2 . estimate from above for the sum of sn,j In this situation, one may sometimes switch from a regularization by finite-dimensional approximation to a finite-dimensional approximation of a regularization. The simplest and most widely used way to do this is by applying the Tikhonov–Phillips regularization to discretized noisy observation represented in the form Qn Ax D Qn yı . This leads to the estimator ı D arg min¹kQn Av  Qn yı k2 C ˛kvk2 , v 2 X º, xnı D xn,˛

which solves the equation ı ı C A Qn Axn,˛ D A Qn yı ˛xn,˛

and has the form of a linear combination ı D xn,˛

n X

cj A ujn ,

j D1

where ¹ujn ºjnD1 is n-th member of our design family, and ¹cj ºjnD1 is determined from the following system of linear algebraic equations: ˛ci C

n X

cj hA uni , A ujn i D huni , yı i, i D 1, 2, : : : , n.

j D1

This system almost coincides with the system coming from a least-square approximation. Only the diagonal elements of corresponding matrices differ by ˛. This means ı is basically the same as that the complexity of the construction of the estimator xn,˛ for xnı . Moreover, it is easy to see that we are still dealing with the same list of linear models Fn D Fn .A/ D span¹A uni ºniD1 , n D 1, 2, : : : , N . The main advantage of the

130

Chapter 2 Basics of single parameter regularization schemes

method discussed is that one can use the parameter ˛ to control the value of the noisedependent term, variance, from the error decomposition. Namely, if x is the solution of “pure” equation Ax D y, then 

ı 0 D xn,˛ C ın,˛ , xn,˛

0 xn,˛ D .˛I C A Qn A/1 A Qn Ax

n,˛ D .˛I C A Qn A/1 A Qn , 

and, as in the previous section, 

ı 0 Ekx  xn,˛ k2 D kx  xn,˛ k2 C ı 2 Ekn,˛ k2 .

Now we are interested in Ekn,˛ k2 D Ek.˛I C A Qn A/1 A Qn Qn k2 

 k.˛I C A Qn A/1 A Qn k2 EkQn k2 . Using functional calculus for B D Qn A one has k.˛I C A Qn A/1 A Qn k D k.˛I C B  B/1 B  k ˇ p ˇ ˇ  ˇˇ 1 ˇ  sup ˇ ˇ p . 2 ˛ 2Œ0,1/ ˇ ˛ C  ˇ Moreover, using the property (G.3) of the white noise model (see Section 2.3), we have X  n 2 n 2 EkQn k D E huj , i j D1

D D

n X j D1 n X

Ehujn , i2 kujn k2 D n.

j D1

Thus, 

Ekn,˛ k2 

n . 4˛

If it is known a priori that sj .A/  j r then putting ˛ D ˛.n/ D n2r we obtain 

Ekn,˛.n/ k2 

n2rC1 . 4

Section 2.8 Model selection based on indirect observation in Gaussian white noise

131

Note that with respect to the rate in n this is the best possible variance bound for a finite-dimensional least-square approximation applied to noisy observation of the form Qn Ax D Qn yı , because, as mentioned above, for any fixed n inf Ekn k2 D inf Ek.A Qn A/1 A Qn k

Qn

Qn

D inf Qn



n X

2 sn,j .Qn A/

j D1

n X

sj2 .A/ 

j D1

n X

j 2r  n2rC1 .

j D1

Thus, if it is known a priori that sj .A/  j r , then for each model Fn .A/ D ı 2 Fn .A/, span¹A uni ºniD1 from our list one can construct an estimator xN nı D xn,˛.n/ 



˛.n/ D n2r , with the variance N n D n,˛.n/ subordinated to the inequality EkN n k2 

1 2 .n/

, .n/ D

2 nrC1=2

.

Putting this .n/ in the criterion from Proposition 2.19, one can select a model Fn .A/ and an estimator xN nı  with a guaranteed risk bound r 1 ran ı 1 e .A, x, xN n /  c .. / .ı// ln , ı where

.n/ is such that for any n D 1, 2, : : : , N , kx  xN n0 k 

.n/.

Our goal now is to compare this risk with the minimal one that can be obtained under the assumption that x 2 A' .R/. To do this, we should establish the relationship between and the index function '. Such a relationship is discussed in the next subsection.

2.8.2

Operator monotone functions

In our previous analysis we have used only simple geometric characteristics of the shape of the index function ', such as monotonicity, concavity/convexity. This was sufficient because we used ' for measuring smoothness properties with respect to one fixed self-adjoint nonnegative operator A A. Assume for the moment, however, that

132

Chapter 2 Basics of single parameter regularization schemes

we are given two such operators B1 , B2 , and the smoothness of some element x 2 X has been measured with respect to one of them, say B1 . What can we say about the smoothness of x with respect to B2 ? If B1 , B2 are unbounded operators, then B1 smoothness of x means that it belongs to the so-called energy space W .B1 / of B1 , equipped with the inner-product hu, viW .B1 / :D hB1 u, vi and corresponding norm kuk2W .B1 / D hB1 u, ui. If all B1 smooth elements are also B2 smooth ones then W .B1 / ,! W .B2 / and for all u kuk2W .B2 / D hB2 u, ui  kuk2W .B1 / D hB1 u, ui, or h.B1  B2 /u, ui  0. If this is the case then one usually writes that B1  B2 . The index function ' is now involved to refine the notion of smoothness, and it is reasonable to expect that for B1  B2 all elements that are '.B1 /-smooth would also be '.B2 /smooth. In other words, it is reasonable to require that index functions should retain the order for self-adjoint, nonnegative operators, i.e., B1  B2 ) '.B1 /  '.B2 /. Such functions are called operator monotone functions. The formal definition follows. Definition 2.5. A function ' : .0, D/ ! R is operator monotone, if for any pair of self-adjoint operators B1 , B2 with spectra in .0, D/ such that B1  B2 , we have '.B1 /  '.B2 /. Fact 2.3 (Corollary of Löwner’s Theorem, [22] V. 4.14). Each operator monotone function ' : .0, D/ ! R admits an integral representation as a Pick function  Z   1  2 .d / '. / D a C b C   C1 for some a > 0, a real number b Rand a finite positive measure  on R that does not have mass on .0, D/ and satisfies .2 C 1/1 .d / < 1. Example 2.12. (1) The function '. / D is operator monotone on .0, 1/ for any 2 .0, 1. (2) The function '. / D  1 is operator monotone on .0, 1/.   (3) The function '. / D logp 1 is operator monotone on (0,1) for any p 2 .0, 1.   (4) The function '. / D logp log 1 is operator monotone on .0, 1/ for any p 2 .0, 1.

Section 2.8 Model selection based on indirect observation in Gaussian white noise

133

In the theory of ill-posed inverse problems one usually deals with compact operators, and the smoothness is expressed in terms of source conditions of the form x D '.A A/v. This means that x belongs to the energy space W .B/ of the unbounded operator B D Œ'.A A/1 because

1 kxk2W .B/ D hBx, xi D h '.A A/ '.A A/v, '.A A/vi D hv, '.A A/vi < 1. Thus, for compact operators A1 , A2 the relation A1 A1  A2 A2 means that A1 is smoother than A2 because W ..A1 A1 /1 / ,! W ..A2 A2 /1 /. Indeed, A1 A1  A2 A2

! 1 operator monontone

)

.A1 A1 /1  .A2 A2 /1

i.e.,

h .A2 A2 /1 C .A1 A1 /1 u, ui  0

) kukW ..A1 A1 /1 / D h.A1 A1 /1 u, ui  h.A2 A2 /1 u, ui D kukW ..A2 A2 /1 / .

condition In this context it is natural to expect that for A1 A1  A2 A2 from the source

1 x D '.A1 A1 /v1 it would follow that x belongs to the energy space of '.A2 A2 / . It is really true if ' is operator monotone on some interval containing the spectra of A1 A1 , A2 A2 . One can see it from the following chain: A1 A1  A2 A2 ) '.A1 A1 /  '.A2 A2 /

) Œ'.A1 A1 /1  Œ'.A2 A2 /1 ) Œ'.A2 A2 /1  Œ'.A1 A1 /1

) W .Œ'.A1 A1 /1 / ,! W .Œ'.A2 A2 /1 /;

x 2 W .Œ'.A1 A1 /1 / ) x 2 W .Œ'.A2 A2 /1 /.

Thus, if we are going to compare the smoothness given by the source conditions with different operators we should use index functions which are operator monotone ones. Operator monotone functions allow the comparison not only of smoothness, but also the approximation power, as one can see it from the following propositions. Proposition 2.21. Let ' be an operator monotone function on .0, 1/ with '.0/ D 0. For any pair B1 , B2 of nonnegative self-adjoint operators on the Hilbert space X we have k'.B1 /  '.B2 /k  4'.kB1  B2 k/. Proof. From the corollary of Löwner’s Theorem (Fact 2.3) it follows that  Z 0   1 '. / D a C b C  2 .d /.  C1 1  

134

Chapter 2 Basics of single parameter regularization schemes

Assuming '.0/ D 0 we can deduce Z 0  bD 1

Then it holds

 1   .d /. 2 C 1 

 1 1  .d / '. / D a C   Z1 0 .d / D a C 1 .  / Z 1 ! D a C .d Q /. . C / 0 Z

0



Using the latter representation we have k'.B1 /  '.B2 /k   Z 1 

.d Q /  1 1   .I C B1 / B1  .I C B2 / B2 D a.B1  B2 / C   0 Z 1   Q / .I C B1 /1 B1  .I C B2 /1 B2  .d .  akB1  B2 k C  0 Now we observe that .I C B1 /1 B1  .I C B2 /1 B2 D .I C B1 /1 .I C B1  I /  .I C B2 /1 .I C B2  I / D I  .I C B1 /1  I C .I C B2 /1

D  .I C B2 /1  .I C B1 /1 D .I C B1 /1 Œ.I C B1 /  .I C B2 / .I C B2 /1 D .I C B1 /1 .B1  B2 /.I C B2 /1 . Keeping in mind that B1 , B2 are nonnegative self-adjoint operators, from functional calculus we obtain k.I C B1 /1 B1  .I C B2 /1 B2 k  k.I C B1 /1 kkB2  B1 kk.I C B2 /1 k  2 kB2  B1 k 1  kB2  B1 k sup  . t0  C t  Moreover, we will also use a rough estimation k.I C B1 /1 B1  .I C B2 /1 B2 k  k.I C B1 /1 B1 k C k.I C B2 /1 B2 k t  2 sup  2. t0  C t

Section 2.8 Model selection based on indirect observation in Gaussian white noise

135

Summing up all these estimations and letting :D kB2  B1 k, we conclude that Z  Z 1 Q / .d Q / .d k'.B1 /  '.B2 /k  a C 2 C ,    0  where we have used a rough estimation for  2 .0, /, and a more accurate one for  2 . , 1/. Keeping in mind that 1

2 C

for  2 .0, /,

and

2  ,  C

 2 . , 1/,

we can continue as follows:

 Z k'.B1 /  '.B2 /k  4 a C

1

Q / .d C  0 D 4'. / D 4'.kB1  B2 k/.



The proposition is proved. Remark 2.10. The result of the proposition actually holds with the constant 1, i.e., under the conditions of Proposition 2.21 k'.B1 /  '.B2 /k  '.kB1  B2 k/, A proof leading to the better values of the constant would be longer, however, and involve additional technicalities (see [22], Theorem X.1.1). Another observation is that for a particular case when X D R1 and B1 , B2 are simply positive numbers (i.e., linear self-adjoint nonnegative operators on R1 ), one has j'.B1 /  '.B2 /j  '.jB1  B2 j/, or, similarly, '.u C v/  '.u/ C '.v/ ) '.2u/  2'.u/. This means that if a function ' is operator monotone on .0, 1/ and '.0/ D 0, then it cannot tend to zero faster than u ! 0. Proposition 2.22. Fix D > 0. Let ' : Œ0, D/ ! RC be operator monotone with '.0/ D 0. For each 0 < d < D there is a constant c D c.d , '/, such that for any pair of nonnegative self-adjoint operators B1 , B2 with kB1 k, kB2 k  d , we have k'.B1 /  '.B2 /k  c'.kB1  B2 k/. Proof. As in the proof of Proposition 2.21 from the equality '.0/ D 0 and from the Löwner’s Theorem corollary (Fact 2.3), we deduce Z .d /. '. / D a C .  /

136

Chapter 2 Basics of single parameter regularization schemes

The corollary from Löwner’s Theorem also tells us that  does not have a mass on Œ0, D/. Then Z 1 Z 0 .d / C .d / '. / D a C 1 .  / D .  / Z 1 .d /. D '1 . / C D .  / As observed in the proof of Proposition 2.21, '1 is operator monotone on .0, 1/. Then   k'.B1 /  '.B2 /k D '1 .B1 /  '1 .B2 / Z 1

.d /   .I  B1 /1 B1  .I  B2 /1 B2 C   D  '1 .kB1  B2 k/ Z 1   .I  B1 /1 B1  .I  B2 /1 B2  .d / . C  D In order to estimate the second integral, one can use the same argument as in the proof of Proposition 2.21. This leads to Z 1 .d / k.I  B1 /1 B1  .I  B2 /1 B2 k  D Z 1 k.I  B1 /1 .B2  B1 /.I  B2 /1 k.d /. D D

Using functional calculus for nonnegative self-adjoint B1 , B2 and   D > d  kB1 k, kB2 k, we have k.I  Bi /1 k  sup

0 kAk , it holds that ˇ ˇ ˇ   ˇ t  1   ŒI  .˛I C A Qn A/ A Qn A'.A Qn A/v   R sup ˇ.1  /'.t /ˇˇ ˇ ˛Ct t2Œ0,D  R'.˛/. On the other hand, Proposition 2.22 allows the following estimation of the second term kŒI  .˛I C A Qn A/1 A Qn A.'.A A/  '.A Qn A//vk t  R sup j1  jk'.A A/  '.A Qn A/k t ˛Ct  cR,' '.kA A  A Qn Ak/. Thus, 0 kx  xn,˛ k  R'.˛/ C cR,' '.kA A  A Qn Ak/,

where cR,' depends on ' and R only. Now everything depends on the approximation power of the design space. We will assume that k.I  Qn /AkX!Y  cr nr , n D 1, 2, : : : , N ,

138

Chapter 2 Basics of single parameter regularization schemes

where cr depends only on r. Note that by assumption of Proposition 2.20 snC1 .A/ :D inf k.I  Pn /AkX!Y  nr , Pn

where the infinum has been taken over all admissible orthogonal projections of rank.Pn /  n. We thus assume that the best possible order of approximation is achieved by our chosen design. In fact, this assumption is not so restrictive. There are a lot of examples with which it is possible to indicate the design with such order-optimal approximation properties, even without knowledge of the singular value expansion of the operator A. We will now consider one such example. Example 2.13 (Abel’s integral operator). Z 1 x. / d, Ax.t / D p  t t

t ,  2 Œ0, 1,

X D Y D L2 .0, 1/. It is known that for Abel’s integral operator sn .A/  n1=2 . Moreover, it is well-known that Abel’s integral operator A acts continuously from 1=2 L2 .0, 1/ to the Banach space H2 L2 .0, 1/ of functions for which !2 .f , h/ < 1, p 0 2. The latter  assumption is not restrictive, since as mentioned in Remark 2.11, the inclusion W2

1 0 H2 , which is necessary for considering y as an element of L2 , can be guaranteed for  > 2. Now we present an adaptive choice of the truncation level n D nC , which is based on the balancing principle and is read as follows: nC D min¹n 2 N : kDn yı  Dm yı kL2  4ƒ.m/ı, m D n, : : : , N º.

(2.21)

The main result of the subsection given below shows that the adaptively chosen truncation level nC provides a risk bound that is only worse by a log factor than the one obtained with the knowledge of '.k/. Proposition 2.25. Let the truncation level nC be chosen in accordancepwith equation p (2.21), and a design parameter  chosen as  D 4 p ln ƒ1 .1=ı/  ln.1=ı/ with a constant p be chosen in such a way that the following equality holds 211=2 .ƒ1 .1=ı//pC1 D ı 2 , then

p 2  c ln.1=ı/ Eky 0  DnC yı kL 2

where  D max

°

ƒ.nC1/ , ƒ.n/

inf

min ¹.' 2 .n/ C ƒ2 .n/ı 2 /1=2 º,

'2ˆ.,y,ı/ n2N

± n2N .

Remark 2.13. The constant p, used to determine the design parameter , can be numerically calculated for a given noise level, e.g., for ı D 106 , p D 5.34, for ı D 104 , p D 5.50, and for ı D 102 , p D 6.11. Proposition 2.25 can be proven by the same argument used for Proposition 2.19.

2.9.4

Numerical differentiation in the space of continuous functions

In this subsection we consider the problem of numerical differentiation when a derivative needs to be reconstructed in the space C D C Œ1, 1 of continuous functions,

152

Chapter 2 Basics of single parameter regularization schemes

while a function to be differentiated is observed in L2 space. As we already mentioned, this problem can be seen as an extension of the classical ill-posed problem of the reconstruction of a continuous function from its observations in L2 . Here we show in particular that for a wide variety of analytic functions y, their data yı blurred by deterministic noise allow the recovery of y 0 with accuracy of order O.ı log3 ı1 /. This is in contrast to the recovery of y 0 from the ill-posed equation (2.1) by means of the standard regularization methods, where even such a simple analytic function as y.t / D at C b cannot generally be reconstructed from yı with better accuracy than 1 O.ı 3 / [58]. We first note the following property of the derivatives of Legendre polynomials [118], which will be useful below: .k C r/Š p .r/ .r/ k C 1=2. (2.22) jPk .t /j  jPk .1/j D r 2 .k  r/ŠrŠ Lemma 2.9. For y 2 W2 the approximation error has the following bound, provided the integral below exists: Z 1 5 1=2 t 0 kyk . (2.23) dt ky  Dn ykC  C 2 .t / n In cases .t / D t  and .t / D e th , h > 0, from equation (2.23) we can derive the following bounds respectively: ky 0  Dn ykC  C n3 .2  6/1=2 kyk , and ky 0  Dn ykC  C



n5 n5 C 6 h h

1=2

 > 3,

e nh kyk .

(2.24)

(2.25)

Proof. Using equation (2.22) we can show that ky 0  Dn ykC  sup t

1 X kDnC1

1 X

jPk0 .t /j

kDnC1

j .k/j

 sup t

jhy, Pk ijj .k/jjPk0 .t /j

1 X

1 j .k/j

!2 !1=2 kyk

k 2 .k C 1/2 .k C 1=2/  4 2 .k/ kDnC1 Z 1 5 1=2 t C kyk . dt 2 .t / n

!1=2 kyk

The bounds (2.24), (2.25) are obtained by direct calculation from equation (2.23).

153

Section 2.9 A warning example

Next we provide and prove an explicit bound for the noise propagation error. Lemma 2.10. In the Gaussian white noise model the following bound holds true p (2.26) EkDn y  Dn yı kC  cın3 log n. Proof. Keeping in mind that hPk , yı i D hPk , yi C ıhPk , i, k D 1, 2, : : : , n, where hPk , i D k is a centered Gaussian random variable on a probability space . , F , /, the noise propagation error can be estimated as follows: EkDn y  Dn yı kC  ıEk

n X

k Pk0 kC .

(2.27)

kD1

In order to bound the right-hand side of equation (2.27), we will use Dudley’s inequality (1.27). Note that n X k Pk0 .t /, T D Œ1, 1 (2.28) X D .X t / t2T :D kD1

is a zero mean Gaussian random element because it is a finite linear combination of real-valued functions with Gaussian random coefficients k D hPk , i, E k D 0 and E k2 D 1 for k D 1, 2, : : : , n. To employ Dudley’s inequality we define on T the metric dX induced by X as follows: !1=2 n X 2 1=2 0 0 2 jPk .s/  Pk .t /j , s, t 2 T . dX .s, t / :D .EjXs  X t j / D kD1

In view of (2.22) with r D 1 the diameter D D D.T / of T D Œ1, 1 in this metric admits the estimation !1=2 !1=2 n n X X 0 2 5 jPk .1/j c k  cn3 . (2.29) D D D.T /  2 kD1

kD1

Moreover, using mean value theorem and equation (2.22) with r D 2, we can bound the distance dX .s, t / by a multiple of js  t j : dX .s, t / 

n X

!1=2 kPk00 k2C

js  t j

kD1

D

!1=2 n X 1 2 2 2 2 js  t j .k  1/ k .k C 1/ .k C 2/ .k C 1=2/ 64

kD1 5

 cn js  t j.

(2.30)

154

Chapter 2 Basics of single parameter regularization schemes

Recall that Dudley’s inequality (1.27) is written in terms of the minimal number N.dX ,  / of  -balls in metric dX required to cover T . From equation (2.30) one can conclude that N.dX ,  /  cn5  1 . Then Dudley’s estimate yields   n  Z D  X n5 1=2  0 k P k   c d E log    0 kD1 C Z D 1=2 p n5 log d  c D  0  1=2 5 p n D c D D log . CD D By combining this with equations (2.27)–(2.29) we get the lemma statement . We now summarize the above observations from Lemmas 2.9 and 2.10 into the following convergence result. Proposition 2.26. Assume the Gaussian white noise model and y 2 W2 with .k/ D k  . Then for  > 3 and n D cı 1= we have   3 Eky 0  Dn yı kC D O ı log1=2 .1=ı/  . If y 2 W2 with

.k/ D e kh , h > 0, then for n D

c h

(2.31)

log.1=ı/ we obtain

  Eky 0  Dn yı kC D O ı log7=2 .1=ı/ .

(2.32)

Remark 2.14. Examining the proof of Lemma 2.10, one can easily see that in the deterministic noise model instead of equation (2.26) it holds that kDn y  Dn y ı kC  cın3 .   The bounds (2.31), (2.32) are then only worse by a factor log1=2 ı1 than those which can be obtained for the deterministic noise model. In particular, for this model and y 2 W2 , .k/ D e kh , we have   (2.33) ky 0  Dn yı kC D O ı log3 .1=ı/ . Note also that the risk bounds indicated in Propositions 2.24 and 2.26 are achieved for the same order of the truncation level n D O.ı 1= /, or n D O.h1 log.1=ı//. Therefore, one may expect that the truncation level n D nC chosen in accordance with equation (2.21) is effective not only in L2 space, but also in the space of continuous functions.

155

Section 2.9 A warning example

2.9.5

Relation to the Savitzky–Golay method. Numerical examples

In 1964, Savitzky and Golay [146] provided a method for smoothing and differentiating data by the least-squares technique. Since then the Savitzky–Golay approach has been widely used; in fact, the proposed algorithm is very attractive for its exceptional simplicity and its ability to significantly improve computational speed. Moreover, the paper [146] is one of the most cited papers in the journal “Analytical Chemistry” and is considered by that journal to be one of its “10 seminal papers” saying “it can be argued that the dawn of the computer-controlled analytical instrument can be traced to this article” [143]. In this section we would like to discuss the relationship between the considered approach (2.7) and the well-known filtering technique [146]. As will be shown, the approach (2.7) is very similar to the Savitzky–Golay method. At the same time, it is worthwhile mentioning that the Savitzky–Golay filter produces excellent results provided the degree of the polynomial n is correctly chosen [133]. However, this issue has not been well studied in the literature and there is no general rule to advise the choice of the polynomial degree n. In this situation, and in view of the similarity between the approach (2.7) and the Savitzky–Golay method, the adaptive parameter choice rule (2.21) can be used to address the above-mentioned issue. Moreover, we are going to demonstrate the numerical experiments which confirm the robustness of the adaptive choice n D nC and our error bounds for infinitely smooth functions. We are interested in this function class here because as has been mentioned, standard regularization methods operating with the equation (2.1) cannot in general guarantee the accuracy of such high order as equation (2.19), (2.32), or (2.33) which has been proven for the method (2.7). Consider the function with known Fourier–Legendre coefficients 1 X p 2 1=2 D k k C 1=2Pk .t /, y1 .t / D .1  2t  C  / kD0

and take  D 1=3, just to be specific. In our numerical experiments we simulate the noisy Fourier coefficients ¹ykı ºnkD1 , n D 50, as follows: p ı ykı D ykı .j / D k k C 1=2 C k,j , "

ı D ı k" k,j and random vectors "j D ."1,j , "2,j , : : : , "n,j / 2 Rn , j D where k,j j kRn 1, 2, : : : , 10, are drawn from the uniform distribution with zero mean and variance 1. In order to verify the convergence rate mentioned in Remark 2.14, we estimate the average value of the constants from the error bound (2.33)

Cn D

10 ı 0 1 X ky1  Dn y1,j kC , 10 jı log3 ıj j D1

156

Chapter 2 Basics of single parameter regularization schemes

Table 2.1. Numerical tests with y1 . Average values of the constants from the error bounds over 10 noise realizations. ı Cn

102.5 0.5680

102.9 0.4534

103.3 0.3927

103.7 0.3215

104.1 0.4124

104.5 0.4570

a priori n

11

13

14

16

18

19

P ı where y1,j .t / D nkD0 ykı .j /Pk .t /. For different noise levels ı, the values of Cn are presented in Table 2.1. Recall that the truncation level n in equation (2.33) is chosen a priori as n D Ch log ı1 . In order to calculate the truncation level for different noise levels, at first we estimate the multiplier Ch as follows: we simulate the data with some noise level, say ı D 103 , and find a value n D n0 that leads to good performance of the method (2.7) on simulated data. The multiplier Ch D n0 = log ı1 is then used in other numerical experiments. As a result of such a procedure, we have found Ch  1.88. As can be seen from Table 2.1, for different noise levels  of Cn exhibit a  the values rather stable behavior supporting the convergence rate O ı log3 ı1 indicated in equation (2.33). In the rest of the subsection we discuss the case where instead of Fourier–Legendre coefficients hy, Pk i, k D 0, 1, : : : , n, we are only able to use noisy function values y ı .xi /, i D 1, 2, : : : , m, such that m X .y.ti /  y ı .ti //2  ı 2 , (2.34) iD1

where ı depends on m. One can then approximate the partial sums Sn y.t / of the Fourier–Legendre series of y.t / by using the numerical integration for calculating Fourier–Legendre coefficients from given noisy data. A fast algorithm for such calculation was recently proposed in [79] for the case of noise-free data, i.e., y ı .ti / D y.ti /, i D 1, 2, : : : , m. Here we consider another popular method of numerical differentiation from noisy data ¹y ı .ti /º, known as the Savitzky–Golay technique [146], and demonstrate how the adaptive procedure based on the balancing principle can be used in this technique. We consider a version of the Savitzky–Golay technique consisting of the approximation of y 0 .t / by n d X ı ck Pk .t /, (2.35) S Gn yı .t / D dt kD0

is found by the method of leastwhere the vector of coefficients D squares from data ¹yı .ti /ºm as the minimizer of iD1 2 m  n X X dk Pk .ti / , LSm,n .¹yı .ti /º, ¹dk º/ D yı .ti /  cı

.c0ı , c1ı , : : : , cnı /

iD1

over d D .d0 , d1 , : : : , dn / 2 RnC1 .

kD0

Section 2.9 A warning example

157

Using standard linear algebra we can represent the coefficients vector cı in matrix terms as follows: cı D .ATn An /1 ATn yı , (2.36) where yı D .yı .t1 /, yı .t2 /, : : : , yı .tm // 2 Rm , and An is a .n C 1/  mmatrix with n D Pk .ti /, i D 1, 2, : : : , m, k D 0, 1, : : : , n. elements ai,k D ai,k In view of the previous discussion it is clear that the accuracy of the Savitzky–Golay method (2.35) depends on the coordination of the number of the used data values m (also known as filter length), the truncation level n, (sometimes referred to as a filter size), and the noise level ı. The importance of such coordination has been demonstrated numerically in [133]. On the other hand, the assumption (2.34) already relates noise level ı to the filter length m. Therefore, under this assumption we need only specify the choice of the filter size n as a function of m and ı. This can be done by means of the general framework described in Section 1.1.5, where we use a stability bound of the form kS Gn y  S Gn yı k  ƒ.n/ı

(2.37)

to control the noise propagation error in the Savitzky–Golay scheme. Here S Gn y means the approximation (2.35) with ı D 0 and the coefficients vector c0 D .c00 , c10 , : : : , cn0 / calculated for noise-free data vector y D .y.t1 /, y.t2 /, : : : , y.tm //. In view of equations (2.34), (2.36) we have ı kc0  cı kRnC1  k.ATn An /1 ATn .y  yı /k  p , sn

(2.38)

where sn D sn .m/ is the minimal nonzero eigenvalue of the matrix ATn An . Note that sn can be effectively calculated simultaneously to solve the linear system .ATn An /cı D ATn yı for cı , and it turns p out that one may choose data points ti 2 Œ1, 1, i D 1, 2, : : : , m, such that 1= sn .m/  1 for all n D 1, 2, : : : , nmax  m. For example, i1 , i D 1, 2, : : : , m, the values of such for equidistributed points ti D 1 C 2 m1 nmax are presented in Table 2.2 for m D 11, 101, 401. In view of (2.38) this means that under the assumption (2.34) for all n D 1, 2, : : : , nmax , nmax D nmax .m/, noise does not propagate in coefficients ckı of the Savitzky–Golay method (2.35) based on equidistributed points. Moreover, using equations (2.13), (2.22) and the same arguments as in the proofs of Lemmas 2.7 and 2.10, we have kS Gn y  S Gn yı k  ƒ.n/kc0  cı kRnC1 , where for kk D kkL2 the function ƒ.n/ p is the same as in (2.21), while for kk D kkC we take ƒ.n/ D n.n C 1/.n C 2/=.2 6/. Then from Table 2.2 we can conclude that for equidistributed points, for example, the stability bound (2.37) holds true for all n D 1, 2, : : : , nmax .m/. Using this bound

158

Chapter 2 Basics of single parameter regularization schemes

Table 2.2. The maximum values of the filter size n D nmax .m/ for which a noise constrained by equation (2.34) does not propagate in coefficients of the Savitzky–Golay method (2.35) based on m equidistributed points. m nmax D nmax .m/

11

101

401

8

35

77

in the same way as in Proposition 1.2 we can choose a filter size n D nC as ® nC D min n : kS Gn yı  S Gp yı k  3ƒ.n/ı C ƒ.p/ı,

¯ p D nmax .m/, nmax .m/  1, : : : , n C 1

(2.39)

and prove that ky 0  S GnC yı k  6

inf

min

¹'.n/ C ƒ.n/ıº,

'2ˆ.ƒ,y,ı/ nD1,2,:::,nmax .n/

where ˆ.ƒ, y, ı/ is the set of nonincreasing admissible functions ' such that for all n D 1, 2, : : : , nmax .m/ ky 0  S Gn yk  '.n/, and '.nmax .m//  ıƒ.nmax .m//. To the best of our knowledge the balancing principle formulated in the form (2.39) is the first data-driven strategy for choosing a filter size in the Savitzky–Golay method. In what follows we present some numerical illustrations of this strategy. We consider the approximations (2.35) with adaptively chosen filter sizes n. We assume that we are provided with noisy function values yı .ti / D y.ti / C iı , i D 1, 2, : : : , 101, at 101 points ti equally distributed in Œ1, 1, where iı D ı k"k"i , l D 101, and Rl

Rl

is drawn from the uniform distribution with random vector " D ."1 , "2 , : : : , "l / 2 zero mean and variance 1. Then, the given data .ti , yı .ti //, i D 1, 2, : : : , 101, are used to find the coefficients (2.36). We start with numerical experiments confirming the robustness of the adaptive choice n D nC for an infinitely smooth function y1 .t / appearing above. In the upper panel of Figure 2.3 we depict the C norm error of the derivative approximation for different filter sizes n and ı D 103 . As can be seen from the figure, the proposed adaptive choice rule n D nC D 9 allows us to pick one of the filter sizes giving the best accuracy (see the middle panel of the figure). For comparison, in the lower panel we show the derivative approximation by the method (2.35) with a priori parameter choice n D 30. As one can see, near x D 1 the approximation based on the adaptive method (2.39) performs much better compared to the one in the lower panel.

159

Section 2.9 A warning example 1.4 1.2 1 0.8 0.6 0.4 0.2 0 0

5

10

15

20

25

30

35

40

50

45

0.3 Exact derivative Approximated derivative with the size n+ = 9

0.28 0.26 0.24 0.22 0.2 0.18 0.16 0.14 0.12 −1

− 0.9

− 0.8

− 0.7

− 0.6

− 0.5

0.3 Exact derivative Approximated derivative with the filter size n = 30

0.28 0.26 0.24 0.22 0.2 0.18 0.16 0.14 0.12 −1

− 0.9

− 0.8

− 0.7

− 0.6

− 0.5

Figure 2.3. Numerical example for y1 . Upper: C -norm error for filter sizes from 1 to 50. Middle: approximated derivative with the size nC D 9 chosen in accordance with equation (2.39). Lower: approximated derivative with the filter size n D 30 chosen a priori.

These observations are also true for the second example with another infinitely smooth function y2 .t / D t 2 .

160

Chapter 2 Basics of single parameter regularization schemes

2.5 2 1.5 1 0.5 0 0 − 1.8 − 1.82 − 1.84 − 1.86 − 1.88 − 1.9 − 1.92 − 1.94 − 1.96 − 1.98 −2 −1 − 1.8 − 1.82 − 1.84 − 1.86 − 1.88 − 1.9 − 1.92 − 1.94 − 1.96 − 1.98 −2 −1

5

10

15

20

25

30

35

40

45

50

Exact derivative Approximated derivative with the size n+ = 13

− 0.98

− 0.96

− 0.94

− 0.92

− 0.9

− 0.92

− 0.9

Exact derivative Approximated derivative with the filter size n = 23

− 0.98

− 0.96

− 0.94

Figure 2.4. Numerical example for y2 . Upper: C -norm error for filter sizes from 1 to 50. Middle: approximated derivative with the size nC D 13 chosen in accordance with equation (2.39). Lower: approximated derivative with the filter size n D 23 chosen for comparison.

From Figure 2.4 it can be seen that the proposed adaptive choice rule n D nC D 13 automatically picks up one of the filter sizes giving the best accuracy whereas, by

Section 2.9 A warning example

161

comparison, the derivative approximation with n D 23 exhibits poor performance near t D 1. Note that in one of the recent papers [76] on numerical differentiation, where the authors consider a method based on the discrete cosine transform, the Gibbs phenomenon was observed in the approximated derivative near the endpoint of the interval of the definition. From Figure 2.4 one may conclude that the method (2.35) with n D nC D 13 is superior to [76] in weakening the Gibbs phenomenon.

Chapter 3

Multiparameter regularization

3.1

When do we really need multiparameter regularization?

As mentioned in Example 2.2, the Tikhonov–Phillips method is one of the most popular regularization techniques because of the simplicity of the implementation. In the version suggested by Phillips in 1962, a regularized solution x˛ı of the equation Ax D yı is defined as x˛ı D .˛I C A A/1 A yı D arg min¹kAx  yı k2 C ˛kxk2 º.

(3.1)

Keeping in mind that this method has the qualification p D 1, it follows from Proposition 2.8 and Example 2.6 that within the framework of the deterministic noise model the best accuracy that may potentially be achieved by such regularization is of order O.ı 2=3 /. At the same time, this order can be increased if, following Tikhonov’s idea (1963), one changes the identity operator I for some densely defined unbounded self-adjoint strictly positive operator B 2 and defines a regularized solution as x˛ı D .˛B 2 C A A/1 A yı D arg min¹kAx  yı k2 C ˛kBxk2 , x 2 Dom.B/º. (3.2) Then, in view of the transformation .˛B 2 C A A/1 A D B 1 .˛I C B 1 A AB 1 /1 B 1 A it follows from Proposition 2.12 that the best accuracy potentially achieved by (3.2) aC2 has the order O.ı 2.aC1//, provided that the operators B and A are linked in such a way that for any x 2 X it holds d kB a xk  kAxk  DkB a xk.

(3.3)

So, if the link condition (3.3) is satisfied with a 2 .0, 2/ then the Tikhonov regularized solution (3.2) is superior to the Phillips form (3.1). As we have already noted in Section 2.5, the condition (3.3) is a serious restriction, but it is crucial for the theoretical justification of the superiority of equation (3.2) over equation (3.1). Moreover, the link condition (3.3) is sometimes hardly verifiable, and it may happen that the regularization (3.2) performs poorly when this condition is violated. This can be illustrated by Figures 3.1, 3.2, 3.8, and 3.9 below.

164

Chapter 3 Multiparameter regularization

So, what to do in the situation when it is not clear which of the regularization schemes (3.1) or (3.2) is more useful for the problem at hand? An obvious answer is to allow multiple regularization parameters, so that both of the approaches can be used to balance goodness of the data fit kAx  yı k2 against multiple penalties kxk2 , kBxk2 . Some of the first examples can be found in statistical literature [11, 67], as well as in the literature on regularization theory [7]. Examples also include the paper [29], where multiparameter regularization was applied to the inverse problem of electrocardiography, [19], where multiparameter regularization was used for restoration of blurred images, [49], where multiparameter regularization was implemented on an inverse problem of option pricing, [104, 174] on a multiparameter regularization in the determination of geopotential, and the paper [20], where multiparameter regularization was proposed to solve the problem of learning from labeled and unlabeled data. This problem has attracted significant attention in recent years and will be also discussed later in Chapter 4. At this point it is worth noting that in the monograph [74] the term “multi-parameter regularization” was used in a case where several equations with the same solution are regularized with the use of the Tikhonov single penalty scheme. In the present chapter “multiparameter regularization” is a synonym for “multiple penalty regularization”. It is interesting to observe that in the existing literature the performance of multiparameter regularization has been variously judged by authors. For example, the authors of [174] found that multiparameter regularization only improved one parameter version marginally, while the authors of [20] reported on most satisfying decisions given by multiparameter learning algorithms in cases where their one parameter counterparts failed (see also our results in Section 4.5). At the same time, all authors agree that the choice of multiple regularization parameters is a crucial and challenging issue for multiparameter regularization. On the other hand, in spite of the growing interest in multiparameter regularization, we can only indicate a few papers where the choice of multiple regularization parameters has been discussed systematically. Among these are [18], where a multiparameter generalization of the heuristic L-curve rule has been proposed, [13, 16, 39], where knowledge of a noise (covariance) structure is required for a choice of parameters, and [28], where some reduction to a single parameter choice is suggested. At the same time, the discrepancy principle, which is widely used and known as the first parameter choice strategy proposed in the regularization theory [134], has never been discussed in a multiparameter context. The application of this principle for choosing multiple regularization parameters is the primary goal of the next section. Of course, it was clear from the outset that there may be many combinations of regularization parameters satisfying the discrepancy principle. In the next section we show that under standard assumptions regarding smoothness of reconstructed solutions, all such combinations correspond to a reconstruction accuracy of optimal order. In Section 3.3 we consider replacing the discrepancy with a surrogate model function which is far easier to control. Here we develop a multiparameter generalization of

Section 3.2 Multiparameter discrepancy principle

165

the approach proposed in [96], where a model function for the single parameter case has been considered. We show that our model function approximation of the multiparameter discrepancy principle leads to efficient iteration algorithms for choosing regularization parameters. For the sake of transparency in the sections just mentioned all constructions are given for the case of two regularization parameters. The extension to an arbitrarily finite number of parameters is straightforward and presented in Section 3.2.6. In Section 3.3 the performance of our approach is evaluated through numerical experiments involving standard inverse problems drawn from Hansen’s regularization toolbox [71]. We then discuss two-parameter regularization for problems with noisy operators with relation to regularized total least-squares and its dual counterpart. Moreover, in Chapter 4 we demonstrate how model function approximation of the multiparameter discrepancy principle can be used in learning from labeled and unlabeled examples.

3.2

Multiparameter discrepancy principle

Let A : X ! Y be a bounded linear operator between real Hilbert spaces X and Y with norms k  k and inner products h, i. As in Chapter 2, we assume that the range Range.A/ is not closed, then the equation Ax D y

(3.4)

is ill-posed. Moreover, for the sake of simplicity, we assume that the operator A is injective and y belongs to Range.A/, such that a unique minimum norm solution x  2 X of the equation (3.4) exists. As usual, we consider the situation when instead of y 2 Range.A/, noisy data yı 2 Y are given with ky  yı k  ı.

(3.5)

Assume B to be a densely defined unbounded self-adjoint strictly positive operator in the Hilbert space X , satisfying Dom.B/ D Dom.B  / is the dense subspace in X , hBx, yi D hx, Byi

for all x, y 2 Dom.B/,

and > 0 exists, such that kBxk  kxk

for all x 2 Dom.B/.

In the multiparameter Tikhonov regularization a regularized solution x ı .˛, ˇ/ is defined as the minimizer of the functional ˆ .˛, ˇ; x/ :D kAx  yı k2 C ˛kBxk2 C ˇkxk2 ,

(3.6)

166

Chapter 3 Multiparameter regularization

where ˛ > 0 and ˇ > 0 play the role of regularization parameters. In this section, for the sake of transparency, we will concentrate on two-parameter regularization. The results for multiparameter regularization consisting of minimizing the functional ˆ .˛1 , ˛2 , : : : , ˛l , ˇ; x/ :D kAx  yı k2 C

l X

˛i kBi xk2 C ˇkxk2

(3.7)

iD1

can be found in Section 3.2.6. To complete the picture of multiparameter regularization we shall now discuss the topic of the choice of parameters. The goal is to find an a posteriori strategy for choosing regularization parameters .˛, ˇ/. Here we consider an extension of the classical discrepancy principle [117, 134] and look for a parameter set .˛, ˇ/ satisfying the socalled multiparameter discrepancy principle, i.e., kAx ı .˛, ˇ/  yı k D cı,

c  1.

(3.8)

The idea behind the discrepancy principle is to find in Dom.B/ such a solution of an equation involving A that its right-hand side approximates the unknown y with the same order of accuracy as a given yı , assuming that the equation Ax D yı has no solutions in Dom.B/. To analyse the accuracy for such a choice of parameters, we use the standard smoothness assumptions formulated in terms of the operator B. As in Section 2.5, we consider a Hilbert scale ¹Xr ºr2R induced by the operator B, where Xr is the completion of Dom.B r / with respect to the Hilbert space norm kxkr D kB r xk, r 2 R, and assume that positive constants E and p exist, such that x  2 Mp,E :D ¹x 2 X j kxkp  Eº,

(3.9)

which means that x  admits a representation as x  D B p v with v 2 X and kvk  E. The next proposition shows, that in the situation when the problem at hand meets the link condition (3.3), the regularization (3.6) equipped with the multiparameter discrepancy principle (3.8) achieves the order of accuracy that generally cannot be improved upon under the assumption (3.9). From Proposition 2.12 it follows that if equation (3.3) is satisfied, then the same order can be also achieved by a single parameter regularization. In spite of this multiparameter regularization still has an advantage in that the above-mentioned order can be achieved for a variety of regularization parameters. In the following we present numerical examples of how such a variety can be used in multi-task approximation. Proposition 3.1. Let x ı :D x ı .˛, ˇ/ be a Tikhonov two-parameter regularized solution. Then under the assumptions (3.3), (3.9) an order optimal error bound p=.aCp/  p a=.aCp/ .c C 1/ı ı  D O.ı pCa /. (3.10) kx  x k  .2E/ d

167

Section 3.2 Multiparameter discrepancy principle

is valid for p 2 .0, a C 2, and for any positive regularization parameters ˛, ˇ satisfying the multiparameter discrepancy principle (3.8). Proof. Note that the inclusion B p x  2 X is equivalent to the representation B p1 x  D B 1 v, kvk  E, and under the conditions of Proposition 2.13 we may 1 write B 1 v D '.Ls A ALs /vN with L D B, s D p  1, './ D  2.pCa1/ . Then from Proposition 2.13 it immediately follows that for these L, s and ', the order of p error O.ı pCa / cannot generally be improved upon, which means equation (3.10) is optimal. We prove the error bound (3.10) for p 2 Œ1, a. For p 2 .0, 1/ [ .a, a C 2/ the proof is more technical and can be found in [106]. Assume that .˛, ˇ/ is a set of positive parameters satisfying the multiparameter discrepancy principle and x ı is the solution corresponding to .˛, ˇ/. Taking into account that x ı minimizes the functional (3.6), we have kAx ı  yı k2 C ˛kBx ı k2 C ˇkx ı k2  kAx   yı k2 C ˛kBx  k2 C ˇkx  k2  ı 2 C ˛kBx  k2 C ˇkx  k2 . Since the multiparameter discrepancy principle is satisfied, the previous inequality yields c 2 ı 2 C ˛kBx ı k2 C ˇkx ı k2  ı 2 C ˛kBx  k2 C ˇkx  k2 . Keeping in mind that the parameters ˛, ˇ are here positive, and c  1, we have kBx ı k  kBx  k

or

kx ı k  kx  k.

We will now analyse these two cases separately. (i) Using the inequality kBx ı k  kBx  k, we can conclude that kB.x ı  x  /k2 D hBx ı , Bx ı i  2hBx ı , Bx  i C hBx  , Bx  i  2hBx  , Bx  i  2hBx  , Bx ı i D 2hB 2p .x   x ı /, B p x  i  2EkB 2p .x ı  x  /k, or, similarly, kx ı  x  k21  2Ekx ı  x  k2p . The rest of the proof for (i) is now based on the interpolation inequality kxkr  kxk.sr/=.sCa/ kxk.aCr/=.sCa/ , a s which holds for all r 2 Œa, s, a C s ¤ 0 (see Section 2.5).

(3.11)

168

Chapter 3 Multiparameter regularization

By taking r D 2  p and s D 1, we can continue as follows: .aC2p/=.aC1/

kx ı  x  k21  2Ekx ı  x  k.p1/=.aC1/ kx ı  x  k1 a

.

Then from assumptions (3.3), (3.5), (3.8) we obtain kx ı  x  k1  .2E/.aC1/=.aCp/ kx ı  x  k.p1/=.aCp/ a .p1/=.aCp/  cC1  .2E/.aC1/=.aCp/ ı .p1/=.aCp/ . d This estimate, together with the interpolation inequality (3.11), where r D 0, s D 1, gives us the error bound (3.10), which is valid for 1  p  a C 2. (ii) Assume now that the inequality kx ı k  kx  k is valid. Then using equation (3.11) with r D p, s D 0 we obtain kx ı  x  k2 D hx ı , x ı i  2hx ı , x  i C hx  , x  i  2hx   x ı , x  i  2Ekx ı  x  kp ı  .ap/=a  2Ekx ı  x  kp=a , a kx  x k

or, similarly, . kx ı  x  k  .2E/a=.aCp/ kx ı  x  kp=.aCp/ a Again using equations (3.3), (3.5), (3.8), we arrive at the error bound (3.10), which is valid for 0  p  a in the case considered. Thus, Proposition 3.1 tells us that the multiparameter discrepancy principle (3.8) defines a manifold of regularization parameters .˛, ˇ/, such that the same order of accuracy is granted for all its points. Staying on such a manifold one still leaves enough freedom to perform other approximation tasks. This freedom in choosing regularization parameters can be seen as a possibility to incorporate various features of one-parameter regularized solutions into a multiparameter one, or as a possibility for balancing different one-parameter regularizations, when it is not clear how to decide between them. In the next subsection, we discuss a way to realize the above-mentioned freedom with the help of the multiparameter model function.

3.2.1 Model function based on the multiparameter discrepancy principle In this subsection, we discuss a numerical realization of the multiparameter discrepancy principle based on the model function approximation [96, 107, 109, 173]. Note that the minimizer x ı D x ı .˛, ˇ/ of the functional (3.6) should solve the corresponding Euler equation, which in this case takes the form    A A C ˛B 2 C ˇI x ı D A yı . (3.12)

169

Section 3.2 Multiparameter discrepancy principle

This equation can be rewritten in a variational form as follows: hAx ı , Agi C ˛hBx ı , Bgi C ˇhx ı , gi D hyı , Agi,

8g 2 Dom.B/. (3.13)

For our analysis we will need the following statements which can be proven similar to [96]. Lemma 3.1. The function x ı D x ı .˛, ˇ/ is infinitely differentiable at every ˛, ˇ > 0. @n ı The partial derivatives may be obtained recursively. The partial derivative x D @˛ nx solves the following variational equation: hAx, Agi C ˛hBx, Bgi C ˇhx, gi D nhB

@n1 ı x , Bgi, @˛ n1

8g 2 Dom.B/ (3.14)

while the partial derivative z D

@n @ˇ n

x ı solves the equation

hAz, Agi C ˛hBz, Bgi C ˇhz, gi D nh

@n1 ı x , gi, @ˇ n1

8g 2 Dom.B/.

(3.15)

Lemma 3.2. The first partial derivatives of F .˛, ˇ/ :D ˆ.˛, ˇ; x ı .˛, ˇ// are given by @˛ F .˛, ˇ/ :D

@F .˛, ˇ/ D kBx ı .˛, ˇ/k2 , @˛

@F .˛, ˇ/ D kx ı .˛, ˇ/k2 . @ˇ In view of equation (3.6) and Lemma 3.2, the multiparameter discrepancy principle (3.8) can be rewritten as @ˇ F .˛, ˇ/ :D

F .˛, ˇ/  ˛@˛ F .˛, ˇ/  ˇ@ˇ F .˛, ˇ/ D c 2 ı 2 . Now the idea is to approximate F .˛, ˇ/ using a simple surrogate function, namely the model function m.˛, ˇ/, such that one could easily solve the corresponding approximate equation for ˛ or ˇ, i.e., m.˛, ˇ/  ˛@˛ m.˛, ˇ/  ˇ@ˇ m.˛, ˇ/ D c 2 ı 2 . To derive an equation for such a model function, we note that for g D x ı the variational form (3.13) gives us kAx ı k2 C ˛kBx ı k2 C ˇkx ı k2 D hyı , Ax ı i. Then, F .˛, ˇ/ D hAx ı  yı , Ax ı  yı i C ˛kBx ı k2 C ˇkx ı k2 D kAx ı k2 C kyı k2  2hyı , Ax ı i C ˛kBx ı k C ˇkx ı k2 D kyı k2  kAx ı k2  ˛kBx ı k2  ˇkx ı k2 .

(3.16)

170

Chapter 3 Multiparameter regularization

Now, as in [96, 109, 173], we approximate the term kAx ı k2 by T kx ı k2 , where T is a positive constant to be determined. This approximation, together with Lemma 3.2, gives us the approximate formula F .˛, ˇ/  kyı k2  ˛@˛ F .˛, ˇ/  .ˇ C T /@ˇ F .˛, ˇ/. By a model function we mean a parameterized function m.˛, ˇ/, for which this formula is exact, that is, m.˛, ˇ/ should solve the differential equation m.˛, ˇ/ C ˛@˛ m.˛, ˇ/ C .ˇ C T /@ˇ m.˛, ˇ/ D kyı k2 . It is easy to check that a simple parametric family of the solutions of this equation is given by m.˛, ˇ/ D kyı k2 C

C D C , ˛ T Cˇ

(3.17)

where C , D, T are constants to be determined. Now we are ready to present an algorithm for the approximate solution of the equation (3.8), where the discrepancy is approximated by means of a model function.

3.2.2 A use of the model function to approximate one set of parameters satisfying the discrepancy principle We use model functions of the form (3.17) to construct an iterative procedure that produces a sequence ¹.˛k , ˇk /º, k D 1, 2, : : :, approximating a pair .˛  , ˇ  /, which satisfies the discrepancy principle. Assume that a pair .˛k , ˇk / has already been found, and x D x ı .˛k , ˇk / solves the equation (3.12), where ˛ D ˛k , ˇ D ˇk . We then determine the coefficients C , D, T in such a way that the corresponding function (3.17) interpolates the function F .˛, ˇ/, and its first partial derivatives at the point .˛k , ˇk /, i.e., 8 ˆ ˆ D F .˛k , ˇk /, m.˛k , ˇk / D kyı k2 C ˛Ck C ˇ D ˆ k CT < @˛ m.˛k , ˇk / D  ˛C2 D @˛ F .˛k , ˇk / D kBx ı .˛k , ˇk /k2 , ˆ k ˆ ˆ : @ m.˛ , ˇ / D  D 2 D @ F .˛ , ˇ / D kx ı .˛ , ˇ /k2 . ˇ k k ˇ k k k k .ˇ CT / k

It is easy to check that these interpolation conditions uniquely determine the values of the coefficients C D C.˛k , ˇk /, D D D.˛k , ˇk /, T D T .˛k , ˇk / as follows: 8 2 ı 2 ˆ ˆ < C.˛k , ˇk / D ˛k kBx .˛k , ˇk /k , D.˛k , ˇk / D .kAx ı .˛k , ˇk /k2 C ˇk kx ı .˛k , ˇk /k2 /2 =kx ı .˛k , ˇk /k2 , ˆ ˆ : T .˛k , ˇk / D kAx ı .˛k , ˇk /k2 =kx ı .˛k , ˇk /k2 . (3.18)

171

Section 3.2 Multiparameter discrepancy principle

Using a model function (3.17) with these coefficients we can find an updated value of the regularization parameter ˛ D ˛kC1 by solving the equation m .˛, ˇk /  ˛@˛ m .˛, ˇk /  ˇk @ˇ m .˛, ˇk / D c 2 ı 2 , which corresponds to the model function approximation of the discrepancy principle mentioned above. It is easy to see that this equation is equivalent to a linear equation, and its solution ˛ D ˛kC1 can be found explicitly as ˛kC1 D

2C.˛k , ˇk / c2ı2

 kyı

k2



D.˛k ,ˇk / ˇk CT .˛k ,ˇk /



ˇk D.˛k ,ˇk / .ˇk CT .˛k ,ˇk //2

.

(3.19)

Performing an intermediate iteration step we use the partially updated parameter set .˛kC1 , ˇk /, to find the solution x D x ı .˛kC1 , ˇk / of the equation (3.12), where ˛ D ˛kC1 , ˇ D ˇk , and then to calculate the coefficients C D C.˛kC1 , ˇk /, D D D.˛kC1 , ˇk /, T D T .˛kC1 , ˇk / given by the formula (3.18), where ˛k should be substituted by ˛kC1 . These coefficients determine another model function (3.17) which interpolates the function F .˛, ˇ/ and its derivatives at the point .˛kC1 , ˇk /. This new model function m.˛, ˇ/ is used to find an updated value of the parameter ˇ D ˇkC1 by solving the equation of the approximate discrepancy principle m .˛kC1 , ˇ/  ˛kC1 @˛ m .˛kC1 , ˇ/  ˇ@ˇ m .˛kC1 , ˇ/ D c 2 ı 2 , which is equivalent to a quadratic equation   2C.˛kC1 , ˇk / .ˇ C T .˛kC1 , ˇk //2 c 2 ı 2  kyı k2  ˛kC1  2D.˛kC1 , ˇk /.ˇ C T .˛kC1 , ˇk // C T .˛kC1 , ˇk /D.˛kC1 , ˇk / D 0. (3.20) Thus, an iterative algorithm based on the model function approximation can be formulated in the form of an alternating procedure as follows: (1) Given ı, c, yı , A, ˛0 > ˛ , ˇ0 > ˇ , set k D 0. (2) Solve the equation (3.12), where ˛ D ˛k , ˇ D ˇk , to obtain x ı .˛k , ˇk /; calculate the coefficients C D C.˛k , ˇk /, D D D.˛k , ˇk /, T D T .˛k , ˇk / in accordance with equation (3.18); update ˛ D ˛kC1 in accordance with equation (3.19). (3) Solve the equation (3.12), where ˛ D ˛kC1 , ˇ D ˇk , to obtain x ı .˛kC1 , ˇk /; calculate C D C.˛kC1 , ˇk /, D D D.˛kC1 , ˇk /, T D T .˛kC1 , ˇk /; find ˇ D ˇkC1 as the minimal positive solution of equation (3.20). (4) STOP if a stopping criteria is satisfied; otherwise set k :D k C 1, GOTO (1).

172

Chapter 3 Multiparameter regularization

3.2.3 Properties of the model function approximation In the algorithm of model function approximation described above one goes from .˛k , ˇk / to .˛kC1 , ˇk /, and then to .˛kC1 , ˇkC1 /. In each updating step the discrepancy function G.˛, ˇ/ D kAx ı .˛, ˇ/  y ı k2 is approximated by the function G m .˛, ˇ/ D m.˛, ˇ/  ˛@˛ m.˛, ˇ/  ˇ@ˇ m.˛, ˇ/. By definition G.˛, ˇ/ D F .˛, ˇ/  ˛@˛ F .˛, ˇ/  ˇ@ˇ F .˛, ˇ/, and for any k D 0, 1, 2, : : : we have G.˛k , ˇk / D G m .˛k , ˇk /,

G.˛kC1 , ˇk / D G m .˛kC1 , ˇk /.

It is easy to derive from equations (3.16) and (3.18) that for any current value of the regularization parameter ˇ D ˇk we have ˇCT D

kAx ı k2 C ˇkx ı k2 > 0. kx ı k2

Moreover, from equations (3.18) we can conclude that 8 C 2C ˆ ˆ @2˛ m .˛, ˇ/ D 3 < 0; < @˛ m .˛, ˇ/ D  2 > 0, ˛ ˛ D 2D ˆ ˆ : @ˇ m .˛, ˇ/ D  > 0, @2ˇ m .˛, ˇ/ D < 0. 2 .ˇ C T / .ˇ C T /3

(3.21)

Proposition 3.2. Assume that for .˛k , ˇk / we have kAx ı .˛k , ˇk /  y ı k > cı. If ˛ D ˛kC1 is given by the formula (3.19) as a positive solution of the equation G m .˛, ˇk / D c 2 ı 2 corresponding to the model function approximation of the discrepancy principle, then ˛kC1 < ˛k . Proof. Observe that g.˛/ :D G m .˛, ˇk / is an increasing function of ˛ because of dg.˛/ D @˛ m .˛, ˇk /  @˛ m .˛, ˇk /  ˛@2˛ m .˛, ˇk / D ˛@2˛ m .˛, ˇk / > 0. d˛ Since ˛kC1 satisfies g.˛kC1 / D G m .˛kC1 , ˇk / D c 2 ı 2 , from g.˛k / G m .˛k , ˇk / D G.˛k , ˇk / > c 2 ı 2 and the monotonicity of g.˛/, we have ˛kC1 < ˛k .

D

Section 3.2 Multiparameter discrepancy principle

173

A similar statement is also valid for ˇ. Proposition 3.3. Assume that for .˛kC1 , ˇk / we have kAx ı .˛kC1 , ˇk /  yı k > cı. If ˇ D ˇkC1 is the minimal positive solution of the equation G m .˛kC1 , ˇ/ D c 2 ı 2 , then ˇkC1 < ˇk . Thus, the propositions just proven tell us that the algorithm of the multiparameter model function approximation produces decreasing sequences of regularization parameters ˛k , ˇk provided that in each updating step the discrepancy is larger than a threshold value cı.

3.2.4

Discrepancy curve and the convergence analysis

In this subsection we discuss the use of model functions for an approximate reconstruction of a discrepancy curve DC.A, yı , c/ 2 R2 , which is defined as follows: DC.A, yı , c/ D ¹.˛, ˇ/ : ˛, ˇ  0, kAx ı .˛, ˇ/  yı k D cıº, c  1. In view of Proposition 3.1, the points .˛, ˇ/ on this curve are of interest, since all of them correspond to regularized solutions x ı .˛, ˇ/, giving an accuracy of the same (optimal) order. It follows from [106] that the existence of the curve DC.A, yı , c/ is guaranteed under the condition that kyı k > cı. Assume that ˇ D ˇ   0 is such that for some ˛  > 0 we have ¹.˛, ˇ/ 2 R2 , ˇ D ˇ  º \ DC.A, yı , c/ D ¹.˛  , ˇ  /º,

(3.22)

which means that kAx ı .˛  , ˇ  /  yı k D cı. Consider the sequence ¹˛k .ˇ  /º given by the formula (3.19), where ˇk D ˇ  , k D 0, 1, : : :. Note that the sequence ¹˛k .ˇ  /º can be produced by the algorithm of model function approximation described in Section 3.2.2, if one skips the updating step (3) and uses ˇk D ˇ  for all k D 0, 1, 2, : : : . Proposition 3.4. Assume that for ˛ D ˛1 .ˇ  /, ˇ D ˇ  kAx ı .˛1 .ˇ  /, ˇ  /  yı k > cı, and the condition (3.22) is satisfied. Then either there is an index k D k  , such that for k D 1, 2, : : : , k   1, kAx ı .˛k .ˇ  /, ˇ  /  yı k > cı, while kAx ı .˛k  .ˇ  /, ˇ  /  yı k < cı, or ˛k .ˇ  / ! ˛  as k ! 1. Proof. It is clear that we need to prove only the convergence ˛k .ˇ  / ! ˛  under the assumption that kAx ı .˛k .ˇ  /, ˇ  /  yı k > cı for all k D 1, 2, : : :. Using the same argument as in [52], one can prove that for any ˇ the discrepancy kAx ı .˛, ˇ/  yı k is a monotonically increasing function of ˛, which means that ˛k .ˇ  / > ˛  for all k D 1, 2, : : : . From Proposition 3.2 it follows that ¹˛k .ˇ  /º is a decreasing sequence. Then

174

Chapter 3 Multiparameter regularization

there exists ˛N  ˛  > 0 such that limn!1 ˛k .ˇ  / D ˛. N Moreover, from Lemma 3.1 ı  it follows that x .˛, ˇ / is a continuous function of ˛. This allows the conclusion that kAx ı .˛, ˇ  /k, kx ı .˛, ˇ  /k, hyı , Ax ı .˛, ˇ  /i are continuous functions of ˛. Then from equation (3.13) it follows that ˛kBx ı .˛, ˇ  /k is also a continuous function of ˛, and taking limits in both sides of the formula (3.19) we obtain ˛N D  D

N ˇ  /k2 ˛N 2 2kBx ı .˛, ı .˛, c 2 ı 2  F .˛, N ˇ  /  ˛kBx N N ˇ  /k2 C ˇ  kx ı .˛, N ˇ  /k2 2kBx ı .˛, N ˇ  /k2 ˛N 2 , c 2 ı 2  kAx ı .˛, N ˇ  /  yı k2  2kBx ı .˛, N ˇ  /k2 ˛N

or, similarly, N ˇ  /  yı k2 . c 2 ı 2 D kAx ı .˛, Using the assumption (3.22) we can conclude that limk!1 ˛k .ˇ  / D ˛  . Note that if the discrepancy curve admits a parameterization by means of some continuous function h such that DC.A, yı , c/ D ¹.˛, ˇ/ : ˛ D h.ˇ/, ˇ 2 Œ0, ˇ0 º, then the assumption (3.22) is satisfied. From Proposition 3.4 it follows that in this case the discrepancy curve can be approximately reconstructed by taking a grid ¹ˇ.q/ºM qD1 Œ0, ˇ0 , and constructing a sequence ¹˛k .ˇ  /º for each ˇ  D ˇ.q/, q D 1, 2, : : : , M . The points .˛k .ˇ  /, ˇ  / will either converge to corresponding points on the discrepancy curve or the final point of the sequence will be below the curve. From Proposition 3.1 it follows that in the latter case the corresponding regularized solution will also provide accuracy of the optimal order. Note that, as demonstrated in [106], Newton iterations can be profitably employed for a pointwise reconstruction of discrepancy curves. In this section we are more interested in the concept of the discrepancy curve and consider multiparameter model function approximation as one possible tool allowing use of this concept.

3.2.5 Heuristic algorithm for the model function approximation of the multiparameter discrepancy principle In the algorithm of model function approximation presented in Section 3.2.2, each iteration consists of two updating steps: first one updates ˛ going from .˛k , ˇk / to .˛kC1 , ˇk / by solving a linear equation, then ˇ is updated by solving a quadratic equation. Both these equations can easily be solved explicitly, however to specify their coefficients one needs to find x ı .˛k , ˇk / and x ı .˛kC1 , ˇk /. This means that in each iteration step a regularized problem (3.12) must be solved twice, and this may be

175

Section 3.2 Multiparameter discrepancy principle

undesirable from a computational standpoint. In this subsection we present a heuristic algorithm that allows us to go from .˛k , ˇk / to .˛kC1 , ˇkC1 / in one step, and to solve a regularized problem (3.12) only once in each iteration. Note that the situation in the Tikhonov two-parameter regularization is in some sense similar to that in the iteratively regularized Gauss–Newton method for nonlinear ill-posed problems (see, e.g., [85], p. 77). Recall that in that method each iteration step k consists of minimizing a Tikhonov-type functional with a single penalty parameter ˇk , and both a number of iteration k and a penalty value ˇk are considered regularization parameters. In such a multiparameter regularization scheme a penalty parameter changes from iteration to iteration, and its values are a priori prescribed to be the terms of a fixed decreasing geometric sequence, say ˇkC1 D ˇk ,

0 < < 1,

(3.23)

while the iteration number k is chosen in accordance with the discrepancy principle. We use the same idea in our heuristic algorithm for model function approximation of the discrepancy principle. From Proposition 3.3 we know that the algorithm described in Section 3.2.2 produces a decreasing sequence of parameters ¹ˇk º. Our heuristic consists of setting this sequence to a geometric one (3.23). An iterative procedure for the model function approximation of the discrepancy principle can then be realized as follows: (1) Given ı, c, yı , A, ˛0 , ˇ0 , , set k D 0. (2) Solve the equation (3.12), where ˛ D ˛k , ˇ D ˇk to obtain x ı .˛k , ˇk /; calculate the coefficients C D C.˛k , ˇk /, D D D.˛k , ˇk /, T D T .˛k , ˇk / in accordance with equation (3.18); update ˛ D ˛kC1 in accordance with equation (3.19) and take ˇ D ˇkC1 D ˇk . (3) STOP if kAx ı .˛kC1 , ˇkC1 /  yı k < cı or ˛kC1 is smaller than the smallest positive normalized floating-point number, otherwise set k D k C 1, GOTO (1). We test this heuristic algorithm in the numerical experiments presented in Section 3.3.

3.2.6

Generalization in the case of more than two regularization parameters

Recall that a multiparameter regularized solution x ı .˛1 , ˛2 , : : : , ˛l , ˇ/ is defined as the minimizer of the Tikhonov functional ˆ .˛1 , ˛2 , : : : , ˛l , ˇ; x/ :D kAx  yı k2 C

l X iD1

˛i kBi xk2 C ˇkxk2 ,

176

Chapter 3 Multiparameter regularization

where .˛1 , ˛2 , : : : , ˛l , ˇ/ is a parameter set to be determined. The multiparameter discrepancy principle suggests choosing such a set .˛1 , ˛2 , : : : , ˛l , ˇ/ that    ı  c  1. Ax .˛1 , ˛2 , : : : , ˛l , ˇ/  yı  D cı, The same reasons as in the two-parameter case lead to the following form of model function: m .˛1 , ˛2 , : : : , ˛l , ˇ/ :D kyı k C 2

l X Ci iD1

˛i

C

D . ˇCT

(3.24)

As in Section 3.2.2, we can construct an iterative process to approximate one of the solutions of the discrepancy equation. Let @m.˛1 , ˛2 , : : : , ˛l , ˇ/ G m .˛1 , ˛2 , : : : , ˛l , ˇ/ D m.˛1 , ˛2 , : : : , ˛l , ˇ/  ˇ @ˇ l X @m.˛1 , ˛2 , : : : , ˛l , ˇ/ ˛i ,  @˛i iD1

and .˛1,k , ˛2,k , : : : , ˛l,k , ˇk / be an approximation constructed in k-th iteration step. Then in .k C 1/-th iteration step we go from .˛1,kC1 , ˛2,kC1 , : : : , ˛j 1,kC1 , ˛j ,k , : : : , ˛l,k , ˇk / to .˛1,kC1 , ˛2,kC1 , : : : , ˛j 1,kC1 , ˛j ,kC1 , ˛j C1,k , : : : , ˛l,k , ˇk / j D 1, 2, : : : , l  1, by solving for ˛ D ˛j ,kC1 the equation G m .˛1,kC1 , ˛2,kC1 , : : : , ˛j 1,kC1 , ˛, ˛j C1,k , : : : , ˛l,k , ˇk / D c 2 ı 2 ,

(3.25)

which is equivalent to a linear equation. Once all parameters ˛1 , ˛2 , : : : , ˛l have been updated, the updated value of the parameter ˇ D ˇkC1 can be found from the equation G m .˛1,kC1 ,    , ˛l,kC1 , ˇ/ D c 2 ı 2 ,

(3.26)

which is equivalent to a quadratic equation. Similar to Propositions 3.2 and 3.3, one can prove the following statement. Proposition 3.5. Assume that kAx ı .˛1,kC1 , ˛2,kC1 , : : : , ˛j 1,kC1 , ˛j ,k , : : : , ˛l,k , ˇk /  yı k > cı. If ˛ D ˛j ,kC1 is given as a positive solution of equation (3.25), then ˛j ,kC1 < ˛j ,k . Moreover, if kAx ı .˛1,kC1 , ˛2,kC1 , : : : , ˛l,kC1 , ˇk /  yı k > cı, and ˇ D ˇkC1 is given as the minimal positive solution of equation (3.26), then ˇkC1 < ˇk .

177

Section 3.3 Numerical realization and testing

An extension of our heuristic algorithm in the case of more than two regularization parameters is straightforward. In Section 3.3 we test it in the case of 3-parameter regularization. Remark 3.1. In order to receive a version of the error bound (3.10) for a multiparameter regularized solution x ı .˛1 , ˛2 , : : : , ˛l , ˇ/, one may assume that positive constants di , ai , Di exist, such that for any x 2 X di kBiai xk  kAxk  Di kBiai xk

(3.27)

and, moreover, for some Ei , pi > 0 p

kBi i x  k  Ei ,

i D 1, 2, : : : , l.

(3.28)

At the same time, it is rather restrictive to assume that the inequalities (3.27), (3.28) are satisfied simultaneously for all penalization operators Bi , i D 1, 2, : : : , l. On the other hand, numerical experiments presented in Section 3.3 show that even if the conditions (3.27), (3.28) are violated, the multiparameter discrepancy principle with, say, l D 2 produces a regularized solution x ı .˛1 , ˛2 , ˇ/ performing similarly to the best of the corresponding single-parameter regularizations.

3.3

Numerical realization and testing

In this section we first test the two-parameter Tikhonov regularization against the single-parameter ones. The regularization parameters for both schemes are chosen in accordance with the discrepancy principle. In the two-parameter case, this principle is implemented in combination with a model function approximation, as discussed in Subsection 3.2.2. To demonstrate the reliability of such an approach, we also compare two-parameter discrepancy curves constructed pointwise with their approximations based on Proposition 3.4. We then numerically check the statement of Proposition 3.1 and illustrate how discrepancy curves can be used in multi-task approximation. The three-parameter regularization will also be discussed at the end of the section.

3.3.1

Numerical examples and comparison

To perform numerical experiments we generate test problems of the form (3.4) by using the functions ilaplace.n, 1/, shaw.n/ and deriv2.n/ in the Matlab regularization toolbox [71]. These functions occur as the results of a discretization of a first kind Fredholm integral equation of the form Z b k.s, t /f .t /dt D g.s/, s 2 Œa, b, (3.29) a

178

Chapter 3 Multiparameter regularization

with a known solution f .t /. In such a context the operators A and the solutions x  are represented in the form of n  n-matrices and n-dimensional vectors obtained by discretizing corresponding integral operators and solutions. The exact right-hand sides y are then produced as y D Ax  . Noisy data yı are generated in the form yı D y C ıe, where e is n-dimensional normally distributed random vector with zero mean and unit standard deviation, which is generated 100 times, so that for each ı, A, x  we have 100 problems with noisy data yı . For our numerical experiments we simulate two noise levels ı D 0.01kAx  k and ı D 0.05kAx  k, which corresponds to data noise of 1% and 5% respectively; here k  k means the standard norm in the ndimensional Euclidean space Rn . First we consider a regularized approximation x ı D x ı .˛, ˇ/ defined by the formula x ı .˛, ˇ/ D .ˇI C ˛D T D C AT A/1 AT yı , where I is the identity matrix, and D is a discrete approximation of the first derivative on a regular grid with n points given as follows: 0 1 1 1 B C 1 1 B C DDB C. .. .. @ A . . 1 1 Note that such x ı .˛, ˇ/ can be seen formally as the minimizer of equation (3.6), where B is n  n-matrix defined as B D jDj D .D T D/1=2 . In our experiments we compare the performance of the two-parameter regularization x ı .˛, ˇ/ and the standard single-parameter ones xˇı D x ı .0, ˇ/ D .ˇI C AT A/1 AT yı , x˛ı D x ı .˛, 0/ D .˛D T DCAT A/1 AT yı . The relative error kxx kxk

k

with x D x ı .˛, ˇ/, x D x˛ı , x D xˇı is used as a performance measure. In all cases the discrepancy principle is used as the criterion for choosing regularization parameters. In single-parameter regularizations it is implemented routinely [71]. In the case of two-parameter regularization, the implementation of this principle is based on a model function approximation, as discussed in Section 3.2.1. In our experiments we choose the regularization parameters ˛, ˇ by using the algorithm described in Section 3.2.4, where we take the starting value ˛ D 0.2, ˇ D 0.1. Moreover, we take in equation (3.23) D 0.5 and in the stopping rule c D 1. The first experiment is performed with the function ilaplace.n, 1/ [71], which occurs in the discretization of the inverse Laplace transformation by means of the Gauss– Laguerre quadrature with n knots. This case corresponds to equation (3.29) with a D 0, b D 1, k.s, t / D exp.st /, f .t / D exp.t =2/, g.s/ D .s C 1=2/1 . We choose n D 100. The results are displayed in Figure 3.1, where each circle exhibits a relative error in solving the problem with one of 100 simulated noisy data. The circles

179

Section 3.3 Numerical realization and testing

100 tests for ilaplace(100,1) with 1% noise Min = 0.1192, Max = 0.1674

DP (β)

Min = 0.0099, Max = 0.1240

DP (α)

Min = 0.0106, Max = 0.0378

DP (α, β)

0

0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 Relative error 100 tests for ilaplace(100,1) with 5% noise Min = 0.1370, Max = 0.2217

DP (β)

Min = 0.0182, Max = 0.2199

DP (α)

Min = 0.0269, Max = 0.0962

DP (α, β)

0

0.05

0.1 0.15 Relative error

0.2

0.25

Figure 3.1. Experiment with ilaplace.100, 1/. The upper figure presents the results for 1% noise, while the lower figure displays the results for 5% noise.

on the horizontal lines labeled by DP .ˇ/, DP .˛/, DP .˛, ˇ/ correspond to errors of single-parameter regularizations xˇı , x˛ı , and two-parameter regularization x ı .˛, ˇ/ respectively. The two-parameter regularization outperforms the competitors, as can be seen from the figure and from a comparison of Tables 3.1–3.3 below. The best result for the single-parameter regularization is obtained in the case of x˛ı . The second experiment is performed with the function shaw.n/ [71]. It is a discretization of the equation (3.29) with a D =2, b D =2, where the kernel and the solution are given by   sin.u/ 2 , u D .sin.s/ C sin.t //, k.s, t / D .cos.s/ C cos.t //2 u f .t / D a1 e c1 .tt1 / C a2 e c2 .tt2 / , 2

a1 D 2,

a2 D 1,

c1 D 6,

c2 D 2,

2

t1 D 0.8,

t2 D 0.5.

This equation is discretized using a simple collocation with n equidistant points. In the experiment we take n D 100. The results are displayed in Figure 3.2, where the

180

Chapter 3 Multiparameter regularization

100 tests for shaw(100) with 1% noise Min = 0.0596, Max = 0.1620

DP (β)

Min = 0.0688, Max = 0.4300

DP (α)

Min = 0.0917, Max = 0.1618

DP (α, β)

0.05

0.1

0.15

0.2 0.25 0.3 Relative error

0.35

0.4

0.45

0.6

0.7

100 tests for shaw(100) with 5% noise Min = 0.0855, Max = 0.2037

DP (β)

Min = 0.1697, Max = 0.5667

DP (α)

Min = 0.0639, Max = 0.2118

DP (α, β)

0

0.1

0.2

0.3 0.4 Relative error

0.5

Figure 3.2. Experiment with shaw.100/. The upper figure presents the results for 1% noise, while the lower figure displays the results for 5% noise.

same notation as in Figure 3.1 is used. This time the best result for a single-parameter regularization is delivered by xˇı . Two-parameter regularization still performs well compared to each of its competitors, however. This can be seen from Tables 3.1–3.3, where mean values, median values and standard deviations of the relative error are given for each of the problems and methods tested. In all cases multiparameter regularization outperforms the competitors in terms of at least two of three performance measures. For example, in the shaw.100/ problem with 1% noise, multiparameter regularization outperforms the competitors in more than 50% of tests, as can be seen by comparing median values of relative errors. Moreover, multiparameter regularization seems to be more robust to different noise realization, as can be concluded from a comparison of standard deviations. Two experiments presented above clearly demonstrate the compensatory property of multiparameter regularization: it performs similar to the best single-parameter regularization which can be constructed using penalizing operators involved in the multiparameter scheme. Note also that in both experiments the algorithm realizing the multiparameter regularization was rather fast: usually only 2 or 3 iterations were performed to satisfy the stopping criteria.

181

Section 3.3 Numerical realization and testing

Table 3.1. Statistical performance measures for two-parameter regularization (from [105]). Problem

Mean relative error

Median relative error

Standard deviation of relative error

Mean reguparameters ˛, ˇ

ilaplace.100, 1/ with 1% noise

0.0216

0.0217

0.0061

0.0535, 0.0023

ilaplace.100, 1/ with 5% noise

0.0611

0.0594

0.0166

0.0757, 0.0162

shaw(100) with 1% noise

0.1288

0.1297

0.0151

6.14 104 , 0.0034

shaw(100) with 5% noise

0.1665

0.1684

0.0217

0.0068, 0.0209

Table 3.2. Statistical performance measures for the single-parameter regularization xˇı (from [105]). Problem

Mean relative error

Median relative error

Standard deviation of relative error

Mean reguparameter ˇ

ilaplace(100,1) with 1% noise

0.1455

0.1453

0.0109

0.0032

ilaplace(100,1) with 5% noise

0.1810

0.1818

0.0166

0.023

shaw(100) with 1% noise

0.1271

0.1317

0.0240

0.0039

shaw(100) with 5% noise

0.1696

0.1707

0.0175

0.0294

Table 3.3. Statistical performance measures for single-parameter regularization x˛ı (from [105]). Problem

Mean relative error

Median relative error

Standard deviation of relative error

Mean reguparameter ˛

ilaplace(100,1) with 1% noise

0.0509

0.0489

0.0287

0.1418

ilaplace(100,1) with 5% noise

0.0911

0.0763

0.0486

0.7002

shaw(100) with 1% noise

0.2066

0.1969

0.0746

0.0529

shaw(100) with 5% noise

0.4935

0.5384

0.0895

12.8933

182

Chapter 3 Multiparameter regularization

Note that the operator A and B used in our experiments do not satisfy the condition (3.3). Such a situation frequently appears in application (see, e.g., [174]). The presented experiments show that multiparameter regularization can also work reliably beyond the assumptions used in the convergence analysis.

3.3.2 Two-parameter discrepancy curve In this subsection we demonstrate the adequacy of the multiparameter model function approximation using it as a tool to reconstruct discrepancy curves. Recall that in view of Proposition 3.1, these curves are of interest because under the condition of the proposition any point .˛, ˇ/ of such a curve corresponds to a regularized solution x ı .˛, ˇ/ realizing an accuracy of optimal order. Moreover, along the discrepancy curve one can implement another parameter choice rule with the aim of finding a regularized solution meeting one or more performance criterion in addition to the order optimal approximation in a Hilbert space norm. Examples will be given in the next subsection. In Section 3.2.4 we described a procedure for reconstructing a discrepancy curve using Proposition 3.4. In accordance with this procedure we take a grid of parameters n.q/ m ˇ 2 ¹ˇ.q/ D 0.01  q1 m ºqD1 and generate a sequence ¹˛k .ˇ.q//ºkD1 using the formula (3.19), where for each fixed q D 1, 2, : : : , m, all ˇk D ˇ.q/, k D 0, 1, : : :. We terminate with ˛ D ˛n.q/ D ˛n.q/ .ˇ.q//, where n.q/ is the minimal integer number such that either kAx ı .˛n.q/ , ˇ.q//  yı k  cı, or j˛n.q/ .ˇ.q//  ˛n.q/1 .ˇ.q//j < 104 . In view of Proposition 3.4, a line running through the points .˛n.q/ , ˇn.q/ / 2 R2 can then be seen as an approximate reconstruction of the discrepancy curve. At the same time, a straightforward approach to approximate this curve consists in the direct calculation of the discrepancy kAx ı .˛, ˇ/  yı k for all grid points ° p1 .˛, ˇ/ 2 Mm D .˛.p/, ˇ.q// : ˛.p/ D 0.2  , m q1 , p, q D 1, 2, : : : , mº. ˇ.q/ D 0.01  m A line passing through the points .˛.pq /, ˇ.q// 2 Mm such that kAx ı .˛.pq /, ˇ.q//  yı k  cı, but kAx ı .˛.pq C 1/, ˇ.q//  yı k > cı, q D 1, 2, : : : , m, then provides us with an accurate reconstruction of the discrepancy curve. Figures 3.3 and 3.4 display reconstructions of the discrepancy curves for problems ilaplace.100, 1/ and shaw.100/ with 1% data noise. The upper pictures in both figures present reconstructions obtained by means of model function approximation, as described above, while on lower pictures reconstructions made by the full search over grid points M200 are shown. For the problems considered both reconstructions are very

183

Section 3.3 Numerical realization and testing 0.16 Discrepancy curve by model function approximation

α value

0.12 0.08 0.04 0 0

||Axδ − γδ|| < cδ Parameter sets region which satisfies 0.002

0.004 0.006 β value

0.008

0.01

0.14 Discrepancy curve by straight forward approach

α value

0.12 0.08 0.04 0 0

||Axδ − γδ|| ≤ cδ Parameter sets region which satisfies 0.002

0.004 0.006 β value

0.008

0.01

Figure 3.3. Reconstruction of the discrepancy curve for the problem ilaplace.100, 1/ with 1% data noise by means of model function approximation (upper picture) and by the full search over grid points Mm , m D 200 (lower picture).

similar, which can be seen as evidence that the algorithm based on multiparameter model function approximation is rather accurate. Remark 3.2. One could think about using multiparameter regularization x ı .˛, ˇ/, where one parameter, say ˇ, is kept fixed, and not set to zero, while the other is improved by some means. Using a discrepancy curve we can illustrate that in such an approach the choice of a fixed parameter is crucial. For example, from Figure 3.4 it can be seen that the line ˇ D 0.01 does not intersect the discrepancy curve for the problem shaw.100/ with 1% data noise (ı  0.2331). Calculations show that if this ˇ value is kept fixed, then for ˛ D 10k , k D 1, 2 : : : , the smallest relative error kx   x ı .˛, 0.01/k=kx  k is 14%. At the same time, for ˇ D 0.001 and the corresponding point ˛ D 0.0086 at the discrepancy curve, the relative error is 7%, i.e., two times smaller.

184

Chapter 3 Multiparameter regularization 0.16 Discrepancy curve by model function approximation

α value

0.12 0.08 0.04 0 0

||Axδ − γδ|| ≤ cδ Parameter sets region which satisfies 0.002

0.004 0.006 β value

0.008

0.01

0.16 Discrepancy curve by straight forward approach

α value

0.12 0.08 0.04 0 0

||Axδ − γδ|| ≤ cδ Parameter sets region which satisfies 0.002

0.004 0.006 β value

0.008

0.01

Figure 3.4. Reconstruction of the discrepancy curve for the problem shaw.100/ with 1% data noise by means of model function approximation (upper picture) and by the full search over grid points Mm , m D 200 (lower picture) (from [105]).

3.3.3 A numerical check of Proposition 3.1 and use of a discrepancy curve In this subsection we first numerically test the statement of Proposition 3.1 that for regularization parameters .˛, ˇ/ from a discrepancy curve the errors kx ı .˛, ˇ/  x  k do have the same order of magnitude. For this test we consider the problem deriv2 from [71] corresponding to equation (3.29) with a kernel function ´ s.1  t / for s  t K.s, t / D t .1  s/ for s  t and Œa, b D Œ0, 1. For this kernel function, the condition (3.3) Pis satisfied with d D D D  2 , a D 2, and a first order differential operator B D j1D1 j hej , iej , where p ej .t / D 2 sin jt , and h, i is the standard inner product in L2 .0, 1/. We consider three subexamples in which the right-hand sides y.s/, the corresponding solutions x  .t / and the maximal smoothness parameters p0 , for which condition (3.9) is guar-

185

Section 3.3 Numerical realization and testing

anteed for all p 2 .0, p0 /, are given by (i)

y.s/ D

(ii)

y.s/ D

(iii) y.s/ D

s 6 s 3

.s 2  1/ ,

x  .t / D t ,

p0 D 12 ,

.s 3  2s 2 C 1/ , x  .t / D 4t .1  t /, p0 D 52 ,

1 4 2

sin 2s,

x  .t / D sin 2 t ,

p0 D 1.

The discretization of (3.29) has been done using Galerkin approximation based on piecewise constant functions with n equidistant jump points. As a discrete approximation of the first order differential operator B we use the square root of n  n–matrix 1 0 2 1 C B . . C .n C 1/2 B C B 1 . . . . 2 B D C. B C B 2 . . . . @ . 1 A . 1 2 The left diagrams in Figures 3.5 – 3.7 show the numerical results of computing discrepancy curves using the first algorithm described in Section 3.3.2. In addition, the middle diagrams of these figures show the values err.q/ D kxqı x  k, q D 1, 2, : : : , m, where xqı stands briefly for x ı .˛n.q/ , ˇ.q//, and .˛n.q/ , ˇ.q// is a point on the discrepancy curve found as described in Section 3.3.2. From Figures 3.5–3.7 and our computations we observe that the values of err.q/ along the discrepancy curves do not essentially vary and have basically the same order of magnitude, as predicted by Proposition 3.1. We now illustrate how one can use the computed discrepancy curves. Assuming that one is simultaneously interested in an order optimal approximation of the solution x  with respect to L2 -norm, and in an estimation of the point value x  .0.5/. Note that an order optimal L2 -approximation does not guarantee a good estimation of the point value, since the latter functional is not well-defined in L2 . Keeping in mind that all regularized approximations x ı corresponding to the points .˛, ˇ/ on the discrepancy curve provide an order optimal accuracy in L2 -norm, it is natural to look among them for the estimation of x  .0.5/ by x ı .0.5/. The freedom in selecting a point .˛, ˇ/ on the discrepancy curve can now be realized, for example, by means of the well-known quasioptimality criterion [162]. In the present context this criterion consists of choosing .˛n.q/ , ˇn .q//, q D q0 , such that ˇ ˇ± ˇ °ˇ ˇ ˇ ˇ ı ˇ ı .0.5/ˇ . ˇxq0 .0.5/  xqı0 1 .0.5/ˇ D min ˇxqı .0.5/  xq1 2qm

The results are displayed in the right diagrams of Figures 3.5 – 3.7. As can be seen from Figures 3.5 and 3.6, for subexamples (i) and (ii) the error of the estimation jx  .0.5/  xqı .0.5/j varies almost by a factor of 100. Nevertheless, the proposed approach allows a choice of .˛, ˇ/ 2 DC corresponding to the estimation with the smallest error.

186

Chapter 3 Multiparameter regularization × 10− 8

0.252

α value

4.5 3.5

0.248

2.5

0.244

1.5

0.24

0.5 0

0.5

1

1.5 β value

0.012 0.008 0.004

0.236 2 2.5 2 −6 × 10

0 6

10

14

18

4

8

12

16

Figure 3.5. Left: discrepancy curve for subexample (i) n D 100, m D 20. Middle: corresponding values of err.q/. Right: the values of jx  .0.5/x ı .0.5/j for .˛, ˇ/ lying on the discrepancy curve; quasioptimality criterion suggests q0 D 15 (from [106]). × 10− 5

0.03

α value

0.9

0.01 0.026

0.7 0.5

0.006

0.022

0.3 0.002

0.018

0.1 0

0.5

1

1.5

β value

2 2.5 × 10− 5

10

30

50

70

10

30

50

70

Figure 3.6. Left: discrepancy curve for subexample (ii) n D 100, m D 80. Middle: corresponding values of err.q/. Right: the values of jx  .0.5/  x ı .0.5/j for .˛, ˇ/ lying on the discrepancy curve; quasioptimality criterion suggests q0 D 4 (from [106]).

α value

−7

6 × 10 5 4 3 2 1 0

0.5

0.026

0.0025

0.022 0.018 0.014 1

1.5

β value

2

2.5 3 × 10− 6

0.002 4

8

12

16

20

4

8

12

16

Figure 3.7. Left: discrepancy curve for subexample (iii) n D 100, m D 20. Middle: corresponding values of err.q/. Right: the values of jx  .0.5/  x ı .0.5/j for .˛, ˇ/ lying on the discrepancy curve; quasioptimality criterion suggests q0 D 4 (from [106]).

In the subexample (iii), the choice suggested by the quasioptimality criterion does not correspond to the smallest error, but in this case the variation of the error is not essential: from 0.002 to 0.0026. The illustrations presented show how the proposed approach to multiparameter regularization can be effectively used in multi-task approximation, when several quantities of interest are to be estimated simultaneously.

187

Section 3.3 Numerical realization and testing

3.3.4

Experiments with three-parameter regularization

In this subsection we consider regularized approximation x ı D x ı .˛1 , ˛2 , ˇ/ defined by the formula x ı .˛1 , ˛2 , ˇ/ D .ˇI C ˛1 D T D C ˛2 DN T DN C AT A/1 AT yı , where I , D, A are the same as in Section 3.3.1, and .n2/n-matrix DN is the discrete approximation to the second derivative operator on a regular grid with n points given as follows: 1 0 1 2 1 C B 1 2 1 C B DN D B C. .. .. .. A @ . . . 1 2 1 x ı .˛1 , ˛2 , ˇ/ can be formally seen as the minimizer of equation (3.7), where l D 3, N D .DN T D/ N 1=2 . As in Section 3.3.1, we use B1 D jDj D .D T D/1=2 , B2 D jDj the problems ilaplace.100, 1/ and shaw.100/ with 1% and 5% data noise to compare the performances of the three-parameter regularization x ı .˛1 , ˛2 , ˇ/ and the standard single-parameter ones xˇı D x ı .0, 0, ˇ/, x˛ı 1 D x ı .˛1 , 0, 0/ and x˛ı 2 D x ı .0, ˛2 , 0/. In single-parameter regularizations we routinely use the discrepancy principle for choosing a regularization parameter, while in three-parameter regularization this principle is implemented by means of the model function approximation, which is realized as the algorithm described in Sections 3.2.5 and 3.2.6, and we keep the same values of the design and starting parameters D 1=2, c D 1, ˛1 D ˛2 D 0.2, ˇ D 0.1. The results are displayed in Figures 3.8 and 3.9, where the notations are similar to those in Figures 3.1 and 3.2. We also present Tables 3.4 and 3.5 for comparison, where mean values, median values and standard deviations of the relative error are Table 3.4. Statistical performance measures for three-parameter regularization x ı .˛1 , ˛2 , ˇ/ (from [105]). Problem

Mean relative error

Median relative error

Standard deviation of relative error

Mean parameters ˛1 , ˛ 2 , ˇ

ilaplace.100, 1/ with 1% noise

0.023

0.0216

0.0081

3.5 104 , 0.0823, 0.0012

ilaplace.100, 1/ with 5% noise

0.0527

0.0489

0.0186

0.0119 0.0949, 0.0110

shaw.100/ with 1% noise

0.1232

0.1202

0.0284

1.8 108 , 0.0051, 0.0026

shaw.100/ with 5% noise

0.1620

0.1652

0.0254

4.3 104 , 0.0263, 0.0183

188

Chapter 3 Multiparameter regularization

Table 3.5. Statistical performance measures for the single-parameter regularization x˛ı 2 (from [105]). Problem

Mean relative error

Median relative error

Standard deviation of relative error

Mean reguparameter ˛2

ilaplace.100, 1/ with 1% noise

0.7062

0.6265

0.4884

0.7172

ilaplace.100, 1/ with 5% noise

1.8298

1.9014

0.8478

38.1258

shaw.100/ with 1% noise

0.2117

0.1858

0.0776

5.7343

shaw.100/ with 5% noise

0.5410

0.6072

0.1393

6.7 103

100 tests for ilaplace(100,1) with 1% noise Min = 0.1152, Max = 0.1656

DP (β)

Min = 0.0094, Max = 0.1374

DP (α1)

Min = 0.0592, Max = 2.4300, 74 relative error > 0.3

DP (α2)

Min = 0.0083, Max = 0.0505

DP (α1, α2, β) 0

0.05

0.1

0.15 0.2 Relative error

0.25

0.3

100 tests for ilaplace(100,1) with 5% noise Min = 0.1379, Max = 0.2144

DP (β)

Min = 0.0161, Max = 0.2738

DP (α1)

Min = 0.083, Max = 3.5570, 96 relative error > 0.3

DP (α2)

Min = 0.0167, Max = 0.0996

DP (α1, α2, β) 0

0.05

0.1

0.15 0.2 Relative error

0.25

0.3

Figure 3.8. The experiment with 3-parameter regularization of ilaplace.100, 1/ with 1% (upper picture) and 5% (lower picture) data noise (from [105]).

given for three-parameter regularization x ı .˛1 , ˛2 , ˇ/ and the single-parameter one x˛ı 2 (the results for xˇı and x˛ı 1 are similar to what is presented in Tables 3.2 and 3.3). Again multiparameter regularization exhibits a compensatory property, this can be seen as an automatic adjustment of the regularization scheme to a problem setup.

189

Section 3.4 Two-parameter regularization with one negative parameter 100 tests for shaw(100) with 1% noise Min = 0.0527, Max = 0.1686

DP (β)

Min = 0.0527, Max = 0.5392

DP (α1)

Min = 0.1241, Max = 0.5429

DP (α2)

Min = 0.0700, Max = 0.1700

DP (α1, α2, β)

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.55 Relative error 100 tests for shaw(100) with 5% noise Min = 0.1020, Max = 0.2066

DP (β)

Min = 0.1115, Max = 0.5690

DP (α1)

Min = 0.1375, Max = 0.6899

DP (α2)

Min = 0.0835, Max = 0.2157

DP (α1, α2, β) 0

0.1

0.2

0.3 0.4 Relative error

0.5

0.6

0.7

Figure 3.9. The experiment with 3-parameter regularization of shaw.100/ with 1% (upper picture) and 5% (lower picture) of noise in data (from [105]).

3.4

Two-parameter regularization with one negative parameter for problems with noisy operators and right-hand side

In this section we are interested in ill-posed problems A0 x D y0

(3.30)

where A0 : X !Y is a bounded linear operator between infinite dimensional real Hilbert spaces X and Y with the nonclosed range Range.A0 /. We assume throughout the section that the operator A0 is injective, and that y0 belongs to Range.A0 / so that equation (3.30) has a unique solution x0 2 X , but (i) instead of the exact right-hand side y0 2 Range.A0 /, we have noisy data yı 2 Y with ky0  yı k  ı,

(3.31)

190

Chapter 3 Multiparameter regularization

(ii) instead of the exact operator A0 2 L.X , Y /, we have some noisy operator Ah 2 L.X , Y / with (3.32) kA0  Ah k  h. Since Range.A0 / is assumed to be nonclosed, the solution x0 of the problem (3.30) does not depend continuously on the data. Hence, the numerical treatment of problems (3.30)–(3.32) requires the application of some appropriate regularization methods. One of them is the classical total least-squares method (TLS method), where some O for .x0 , y0 , A0 / from the given data .yı , Ah / is determined by solvestimate .x, O y, O A/ ing the constrained minimization problem kA  Ah k2 C ky  yı k2 ! min

subject to Ax D y,

(3.33)

which we call the TLS problem, see [168]. Due to the ill-posedness of the operator equation (3.30), it may happen that a solution xO of the TLS problem does not exist in the space X . Furthermore, if a solution xO 2 X of the TLS problem exists, this solution may be far removed from the desired solution x0 . Therefore, it is quite natural to restrict the set of admissible solutions by searching for approximations xO that belong to some prescribed set K, which is the philosophy of the regularized total ˇ least squares. ® ¯ The simplest case occurs when the set K is a ball K D x 2 X ˇ kBxk  R with a prescribed radius R. This leads us to the regularized total least squares method (RTLS O for .x0 , y0 , A0 / is determined by solving the method) in which some estimate .x, O y, O A/ constrained minimization problem kA  Ah k2 C ky  yı k2 ! min

subject to

Ax D y, kBxk  R,

(3.34)

which we call the RTLS problem, see [62, 141, 150]. In the special case of exactly given operators Ah D A0 , this philosophy leads us to the method of quasisolution of Ivanov, see [80], in which xO is determined by solving the constrained minimization problem kA0 xyı k2 ! min subject to x 2 K. This approximation xO is also sometimes called the K-constrained least squares solution. It is worth noting that in the special case B D I , xO can be obtained by solving the quadratically constrained least squares problem, and appropriate references are [50, 60, 63]. One disadvantage of the RTLS method (3.34) is that this method requires a reliable bound R for the norm kBx0 k. In many practical applications, however, such a bound is unknown. On the other hand, in different applications reliable bounds for the noise levels ı and h in equations (3.31) and (3.32) are known. In this case, it makes O which satisfy the side conditions Ax D y, sense to look for approximations .x, O y, O A/ ky  yı k  ı and kA  Ah k  h. The solution set characterized by these three side conditions is nonempty. Selecting the element which minimizes kBxk from the O for .x0 , y0 , A0 / is solution set leads us to a problem in which an estimate .x, O y, O A/ determined by solving the constrained minimization problem kBxk! min

subject to Ax D y, ky  yı k  ı, kA  Ah k  h.

(3.35)

Section 3.4 Two-parameter regularization with one negative parameter

191

This method can be seen as a dual of the method of regularized total least squares (3.34), because the same functional kBxk plays the role of a constraint in (3.34), while in (3.35) it is a target. Therefore, we propose calling this method the dual regularized total least squares method (dual RTLS or DRTLS), in spite of the fact that the minimization problem (3.35) is not the dual of equation (3.34) in the well-established sense. The constrained minimization problem (3.35) we call the dual RTLS problem. In this section, we characterize regularized solutions given by the RTLS or the DRTLS methods under the assumptions that the corresponding inequality constraints in equations (3.34) and (3.35) are active. These assumptions are important. For example, in [62] it was observed that when the solution xO to (3.34) satisfies kB xk O < R, then equation (3.34) is just the standard TLS method, which may exhibit poor performance. The assumption that the inequality constraints in equation (3.35) are active can be interpreted as perfect knowledge of the data noise level. In the next subsections we discuss some computational aspects of the RTLS method (3.34) and of the dual RTLS method (3.35) in finite dimensional spaces. As a result, both methods lead, in the general case where B 6D I , to special multiparameter regularization methods with two regularization parameters where one of the regularization parameters is negative. We then provide error bounds for the regularized solutions obtained by the methods (3.34) and (3.35). First we treat the special case B D I and derive error bounds under the classical source condition x0 D A0 v with p v 2 Y , which show that the accuracy of the regularized solutions is of the order O. ı C h /. For the general case B 6D I the link condition (3.3) between A0 and B and a smoothness condition (3.9) for x  D x0 in terms of B, are exploited to derive error bounds. Finally, some numerical experiments are given which shed a light on the relationship between RTLS, dual RTLS and the standard Tikhonov regularization.

3.4.1

Computational aspects for regularized total least squares

Computational aspects are studied in the literature for discrete versions of (3.30) in finite-dimensional spaces. Therefore, we restrict our attention to the case when X D Rn and Y D Rm , equipped with the Euclidian norm k  k2 and use the Frobenius norm k  kF as the matrix norm. Moreover, when discussing a finite-dimensional case we use the symbol AT instead of A , where the latter means the adjoint of A. Note that the TLS method, which is the problem (3.34) without the constraint kBxk2  R, is a successful method for noise reduction in linear least-squares methods in various applications. For an overview of the computational aspects and analysis of the TLS method see the monograph [168]. The TLS method is suited for finite dimensional problems where both the coefficient matrix and the right-hand side are not precisely known and where the coefficient matrix is not very ill-conditioned. For discrete ill-posed problems where the coefficient matrix is very ill-conditioned and also for infinite dimensional ill-posed problems, some additional stabilization is necessary as in the RTLS formulation (3.34).

192

Chapter 3 Multiparameter regularization

Previous results regarding properties and computational aspects of the RTLS method may be found in [17, 62, 141, 150]. From [62] we have Proposition 3.6. If the constraint kBxk2  R of the RTLS method (3.34) is active, then the RTLS solution x D xO satisfies the equations .ATh Ah C ˛B T B C ˇI /x D ATh yı

and

kBxk2 D R.

(3.36)

The parameters ˛ and ˇ are related by ˛ D R2 .ˇ C yıT .yı  Ah x//. Moreover, ˇD

(3.37)

kAh x  yı k22 D kA  Ah k2F  ky  yı k22 . 1 C kxk22

(3.38)

Thus, Proposition 3.6 allows a characterization of the RTLS method as a twoparameter regularization where one of the regularization parameters is negative.

3.4.2 Computational aspects for dual regularized total least squares In the special case h D 0, the method (3.35) reduces to the Tikhonov regularization with ˛ chosen by the discrepancy principle, see [52,117]. The general case h ¤ 0 was studied in [108], where the following proposition was proven. Proposition 3.7. If in equation (3.35) the two constraints ky  yı k2  ı and kA  ı,h solves the equation Ah kF  h are active, then the dual RTLS solution x D xO D x˛,ˇ .ATh Ah C ˛B T B C ˇI /x D ATh yı .

(3.39)

The parameters ˛ and ˇ satisfy the system of equations ı,h kAh x˛,ˇ

 yı k2 D ı C

ı,h hkx˛,ˇ k2

and

ˇD

ı,h k / h.ı C hkx˛,ˇ 2 ı,h kx˛,ˇ k2

.

(3.40)

From Proposition 3.7 it follows that, similar to RTLS, the DRTLS method can be seen as a two-parameter regularization where one of the regularization parameters is negative. Remark 3.3. It is interesting to note that even in case of noisy operators and noise-free right-hand sides, the DRTLS method leads to a two-parameter regularization. In this case, however, the negative regularization parameter can be chosen a priori. Namely,it follows directly from Proposition 3.7 that if the constraint kA  Ah kF  h is active, h solves the equation then the DRTLS solution x D xO D x˛,ˇ .ATh Ah C ˛B T B C ˇI /x D ATh y0 ,

Section 3.4 Two-parameter regularization with one negative parameter

193

where the parameters ˛ and ˇ satisfy the system of equations h h  y0 k2 D hkx˛,ˇ k2 kAh x˛,ˇ

3.4.3

and

ˇ D h2 .

Error bounds in the case B D I

Our aim in this subsection is to prove error bounds for the special case when B D I and the classical source condition N x0 D A0 v D .A0 A0 /1=2 v,

with

kvk D kvk, N

v 2 Y,

(3.41)

holds. Then intermediate bounds obtained will be used in the analysis of the general case when B ¤ I . Proposition 3.8 (Error bounds for RTLS). Assume that the exact solution x0 of the problem (3.30) satisfies the source condition (3.41) and the side condition kx0 k D R. In addition let xO be the RTLS solution of the problem (3.34), then p p (3.42) kxO  x0 k  .2 C 2 2/1=2 kvk1=2 max¹1, R1=2 º ı C h . O satisfy the two side conditions Ax D y O y, O A/ Proof. Since both .x0 , y0 , A0 / and .x, and kxk  R of the RTLS method (3.34), we obtain from equations (3.34) and (3.31), (3.32) that kAO  Ah k2 C kyO  yı k2  kA0  Ah k2 C ky0  yı k2  h2 C ı 2 .

(3.43)

O 2  kx0 k2 , or equivalently, Next, since kxk O  R and kx0 k D R, we have kxk O kxO  x0 k2  2hx0 , x0  xi. Due to equation (3.41) and the Cauchy–Schwarz inequality, we obtain O  2kvk kA0 x0  A0 xk. O kxO  x0 k2  2hA0 v, x0  xi

(3.44)

Using the triangle inequality, we have O  kA0 x0  yk O C kyO  A0 xk O kA0 x0  A0 xk  kA0 x0  yı k C kyO  yı k C kyO  Ah xk O C kA0 xO  Ah xk. O (3.45) In order to estimate the sum of the first and fourth summand on the right-hand side of equation (3.45), we use the identity A0 x0 D y0 , apply equations (3.31) and (3.32) and obtain O  ı C hkxk O  max¹1, kxkº.ı O C h/. kA0 x0  yı k C kA0 xO  Ah xk

(3.46)

194

Chapter 3 Multiparameter regularization

In order to estimate the sum of the second and third summand on the right-hand side (3.45), we use the identity yO D AOx, Opapply the inequality a C b  p pof 2equation 2 2 a C b , the estimate (3.43) and the inequality a2 C b 2  a C b to obtain kyO  yı k C kyO  Ah xk O  kyO  yı k C kAO  Ah k kxk O    max¹1, kxkº O kyO  yı k C kAO  Ah k q p 2 kyO  yı k2 C kAO  Ah k2  max¹1, kxkº O p p  max¹1, kxkº O 2 ı 2 C h2 p  max¹1, kxkº O 2 .ı C h/ . (3.47) Combining equations (3.45), (3.46), and (3.47), we have p kA0 xO  A0 x0 k  max¹1, kxkº.1 O C 2/.ı C h/.

(3.48)

Since kxk O  kx0 k D R, this estimate and equation (3.44) provide equation (3.42). Proposition 3.9 (Error bounds for DRTLS). Assume that the exact solution x0 of the problem (3.30) satisfies the source condition (3.41) and let xO be the dual RTLS solution of the problem (3.35). Then, p (3.49) kxO  x0 k  2kvk1=2 ı C hkx0 k. O satisfy the three side conditions Ax D y, Proof. Since both .x0 , y0 , A0 / and .x, O y, O A/ ky  yı k  ı and kA  Ah k  h of the dual RTLS method (3.35), and since xO is the solution of equation (3.35) we have kxk O 2  kx0 k2 ,

(3.50)

or equivalently, kxO  x0 k2  2.x0 , x0  x/. O Using equation (3.41) and the Cauchy– Schwarz inequality we obtain O  2kvk kA0 x0  A0 xk. O kxO  x0 k2  2.A0 v, x0  x/

(3.51)

From equation (3.31) we have ky0  yı k  ı, and from equation (3.35) we have kyO  yı k  ı. Consequently, using the triangle inequality and the identity A0 x0 D y0 we have kA0 x0  yk O  ky0  yı k C kyO  yı k  2ı. (3.52) From equation (3.32) we have kA0  Ah k  h and from equation (3.35) we have kAO  Ah k  h. Hence, using the triangle inequality, the identity yO D AOxO and the estimate (3.50) we have   kyO  A0 xk O  kAO  Ah k C kA0  Ah k kxk O  2hkxk O  2hkx0 k. (3.53)

195

Section 3.4 Two-parameter regularization with one negative parameter

We again apply the triangle inequality together with equations (3.52) and (3.53) and obtain O  kA0 x0  yk O C kyO  A0 xk O  2ı C 2hkx0 k. kA0 x0  A0 xk

(3.54)

From this estimate and equation (3.51) we obtain equation (3.49). Remark 3.4. It is interesting to note that, as it follows from [167, p. 15], under the assumptions (3.31), (3.32), and (3.41), the best guaranteed accuracy of the reconstrucp tion of x0 from noisy data .Ah , yı / has the order O. ı C h/. On the other hand, from Propositions 3.8 and 3.9 we know that accuracy of the same order can be obtained by means of RTLS and dual RTLS. This allows the conclusion that under the conditions of Propositions 3.8 and 3.9 these methods are order optimal in the sense of accuracy.

3.4.4

Error bounds for B ¤ I

In this subsection our aim is to provide order optimal error bounds for the general case B ¤ I under the assumption that the link condition (3.3) and the inclusion (3.9) hold true for A D A0 and x  D x0 . These error bounds are not restricted to finite dimensional spaces X and Y but are also valid for infinite dimensional Hilbert spaces. Proposition 3.10. Assume that the exact operator A0 meets the link condition (3.3), and the exact solution x0 satisfies the smoothness condition (3.9) with p 2 Œ1, 2 C a, as well as the side condition kBx0 k D R. In addition, let xO be the RTLS solution of the problem (3.34), then 

max¹1, kxkº.1 O C kxO  x0 k  .2E/ d   p D O .ı C h/ pCa . a pCa

p

2/

p  pCa .ı C h/

(3.55)

Proof. Since kB xk O  R and kBx0 k D R we have kB xk O 2  kBx0 k2 . Consequently, due to equation (3.9), O B x/ O  2.B x, O Bx0 / C .Bx0 , Bx0 / kxO  x0 k21 D .B x,  2.Bx0 , Bx0 /  2.B x, O Bx0 / D 2.B 2p .x0  x/, O B p x0 /  2EkxO  x0 k2p .

(3.56)

In order to estimate kxO  x0 k2p , we use the interpolation inequality (3.11) with r D 2  p, s D 1, and obtain from equation (3.56) the estimate .aC2p/=.aC1/

kxO  x0 k1 kxO  x0 k21  2EkxO  x0 k.p1/=.aC1/ a

.

(3.57)

196

Chapter 3 Multiparameter regularization

Rearranging the terms in equation (3.57) gives . kxO  x0 k1  .2E/.aC1/=.aCp/ kxO  x0 k.p1/=.aCp/ a

(3.58)

From the link condition (3.3) and estimate (3.48) of Proposition 3.8, which is also valid here, we obtain p kA0 xO  A0 x0 k max¹1, kxkº.1 O C 2/.ı C h/  . (3.59) kxO  x0 ka  d d This estimate and (3.58) yield  .aC1/=.aCp/

kxO  x0 k1  .2E/

max¹1, kxkº.1 O C d

p

2/.ı C h/

.p1/=.aCp/ .

(3.60) Now the desired estimate (3.55) follows from equations (3.59) and (3.60) and the interpolation inequality (3.11) with r D 0 and s D 1. Proposition 3.11. Assume that the exact operator A0 meets the link condition (3.3), and the exact solution x0 satisfies the smoothness condition (3.9) with p 2 Œ1, 2 C a, and let xO be the dual RTLS solution to problem (3.35). Then, kxO  x0 k  2E

a pCa



ı C hkBx0 k d

p  pCa

  p D O .ı C h/ pCa .

(3.61)

O satisfy the three side conditions Ax D y, Proof. Since both .x0 , y0 , A0 / and .x, O y, O A/ ky  yı k  ı and kA  Ah k  h and xO is the solution to equation (3.35), it follows that kB xk O 2  kBx0 k2 . This yields kxO  x0 k21  2EkxO  x0 k2p .

(3.62)

From the link condition (3.3) and the estimate (3.54) of Proposition 3.9, we obtain kxO  x0 ka 

kA0 xO  A0 x0 k 2ı C 2hkBx0 k  . d d

(3.63)

The required estimate (3.61) can be proven as in Proposition 3.10, where instead of equation (3.59) the estimate (3.63) must be used. Remark 3.5. From Theorem 3.6 of [167] and the results of [120, Theorem 2.2 and Example 2.3], it follows that under the assumptions (3.3), (3.9), (3.31), and (3.32) the best guaranteed accuracy of the reconstruction of x0 from noisy data .Ah , yı / has the p order O..ıCh/ pCa /. Comparing this with the error bounds given by Propositions 3.10 and 3.11, one can conclude that under the conditions of these propositions RTLS and dual RTLS are order optimal in the sense of accuracy.

Section 3.4 Two-parameter regularization with one negative parameter

3.4.5

197

Numerical illustrations. Model function approximation in dual regularized total least squares

Note that, as can be seen from Proposition 3.6, the RTLS method requires an accurate bound R for the norm kx0 k of the unknown solution. It has been demonstrated in [108] that the RTLS method is sometimes too sensitive to a misspecification of the value of R. Therefore, it is recommended that RTLS be applied in the situation where a reliable bound for kx0 k is known a priori. One such situation was discussed recently in [104]. On the other hand, as has been reported in [108], the DRTLS method demonstrates stable behavior even with a rough estimation of input parameters, namely, noise levels ı and h. Therefore, when reasonable estimations of the noise levels are known, the DRTLS method can be suggested as a method of choice. As can be seen from Proposition 3.7, however, realization of DRTLS is related to solving a system of highly nonlinear equations (3.40), and computational difficulties may arise. At the same time, one might observe that the first equation of (3.40) is similar to one appearing in the multiparameter discrepancy principle (3.8), where the model function approximation has been used. In this subsection we discuss how the same idea can be used for solving the nonlinear system (3.40) appearing in DRTLS. Then using some standard numerical tests from the Matlab regularization toolbox [71], we demonstrate the efficiency of the proposed model function approach. For the discrete ill-posed problem (3.30) we know from Proposition 3.7 that in the ı,h of problem (3.35) can be case of active constraints the dual RTLS solution xO D x˛,ˇ obtained by solving the minimization problem min ˆh .˛, ˇ; x/,

x2X

ˆh .˛, ˇ; x/ D kAh x  yı k2 C ˛kBxk2 C ˇkxk2

with regularization parameters .˛, ˇ/ chosen by an a posteriori rule (3.40). ı,h /, we can rewrite Using Lemma 3.2 with similar notations F .˛, ˇ/ D ˆh .˛, ˇ; x˛,ˇ the first equation in (3.40) as follows: q   (3.64) F .˛, ˇ/  ˛@˛ F  ˇ@ˇ F D ı C h @ˇ F . The idea now is to approximate F .˛, ˇ/ by a simple model function m.˛, ˇ/, such that one could easily solve the corresponding approximate equation q   m.˛, ˇ/  ˛@˛ m  ˇ@ˇ m D ı C h @ˇ m for ˛ and ˇ. The equation for such a model function can be derived similar to that given in Section 3.2.1 as C D , m.˛, ˇ/ D kyı k2 C C ˛ T Cˇ where C , D, T are constants to be determined. Now we are ready to present an algorithm for computation of the regularization parameters .˛, ˇ/ according to our DRTLS

198

Chapter 3 Multiparameter regularization

rule. That is, we present an algorithm for the approximate solution of the equations in system (3.40) by a special three-parameter model function approach consisting of the following steps. Given ˛0 , ˇ0 , yı , Ah , ı and h. Set k :D 0. (1) Solve equation (3.39) with .˛k , ˇk / to get x˛ı,h,ˇ . Compute F1 D F .˛k , ˇk /, k

k

F2 D @˛ F D kBx˛ı,h,ˇ k2 and F3 D @ˇ F D kx˛ı,h,ˇ k2 . In the formula (3.17) k k k k for m.˛, ˇ/ set C D Ck , D D Dk , T D Tk such that 8 C D ˆ ˆ C D F1 , m.˛k , ˇk / D kyı k2 C ˆ ˆ ˛k ˇk C T ˆ ˆ ˆ < C @˛ m.˛k , ˇk / D  2 D F2 , ˆ ˛ ˆ k ˆ ˆ ˆ D ˆ ˆ D F3 . : @ˇ m.˛k , ˇk / D  .ˇk C T /2 Then,

8 Ck D F2 ˛k2 , ˆ ˆ ˆ ˆ ˆ .kyı k2  F1  F2 ˛k /2 < , Dk D  F3 ˆ ˆ ˆ ky k2  F1  F2 ˛k ˆ ˆ : Tk D ı  ˇk . F3

(3.65)

Update ˇ D ˇkC1 using the second equation (3.40) as ˇkC1 D 

h.ı C hkx˛ı,h,ˇ k/ k

k

kx˛ı,h,ˇ k k k

and update ˛ D ˛kC1 as the solution of the linear algebraic equation m.˛, ˇkC1 /  ˛@˛ m.˛, ˇkC1 /  ˇkC1 @ˇ m.˛, ˇkC1 / D q  2 ı C h @ˇ m.˛, ˇkC1 / . (2) STOP if the stopping criterion j˛kC1  ˛k j C jˇkC1  ˇk j   is satisfied; otherwise set k :D k C 1, GOTO (1). The proposed algorithm is a special fixed point iteration for realizing the DRTLS rule in the method of Tikhonov regularization with two regularization parameters .˛, ˇ/. Although this algorithm works well in experiments, we don’t have any convergence results. The only result which we have is following: If the iteration converges, then the limit .˛  , ˇ  / D limk!1 .˛k , ˇk / is a solution of the nonlinear system (3.40).

Section 3.4 Two-parameter regularization with one negative parameter

199

Before starting numerical illustrations of the algorithm just described we note that Proposition 3.11 allows a theoretical comparison of dual RTLS with the one-parameter Tikhonov regularization scheme, where a regularized solution x D x˛ı,h is obtained from the following equation .ATh Ah C ˛B T B/x D ATh yı .

(3.66)

The regularization theory [52, 167] tells us that under the assumptions of Proposip tion 3.11 the order of accuracy O..ı C h/ pCa / is optimal, and it can be realized within the framework of one-parameter Tikhonov scheme. If the smoothness index p in equation (3.9) is unknown, then this scheme should be equipped with some a posteriori parameter choice rule. Note that a knowledge of the smoothness index p is not necessary for performing dual RTLS. From Proposition 3.11 it follows that this two-parameter regularization scheme automatically adapts to the unknown smoothness index and, like the oneparameter Tikhonov scheme, gives the optimal order of accuracy, at least for some range of p. In our numerical tests we compare the performance of dual RTLS with one-parameter Tikhonov regularization equipped with the quasioptimality criterion for choosing the regularization parameter. This heuristic a posteriori rule has recently been advocated in [14]. Our dual RTLS rule will be realized by the model function approach discussed above while the parameter ˛ in the Tikhonov scheme (3.66) will be chosen in accordance with the quasioptimality criterion as follows: p Consider a set of regularization parameters N D ¹˛i : ˛i D ˛0 p i , i D p 1, 2, : : : , N º, p > 1. The quasioptimality criterion selects such ˛  D ˛m 2 N ,  x˛ı,h k has the minimum value v.˛m / in the for which the quantity v.˛i / D kx˛ı,h i i 1 p chosen set N . Similar to Section 3.3 in our numerical experiments we consider Fredholm integral equations (3.29) with known solutions f .t /. The operators A0 and solutions x0 are then obtained in the form of n  n-matrices and n-dimensional vectors by discretizing the corresponding integral operators and solutions f . Noisy data Ah , yı are simulated by Ah D A0 C hkEk1 F E,

yı D Ah x  C  kek1 e,

where E, e are n  n-random matrices and n-dimensional random vectors from a normal distribution with zero mean and unit standard deviation. These were generated 50 times, so that each integral equation gives rise to 50 noisy matrix equations. Similar to Section 3.3.1, in our experiments with dual RTLS, for B we choose the matrix D, which is a discrete approximation of the first derivative on a regular grid with n points. The single-parameter Tikhonov method (3.66) has been implemented with B D I and B D D respectively. Moreover, in the quasioptimality criterion the set of regularization parameters is chosen as 50 D ¹˛i D 103  p i , i D 1, 2, : : : , 50º, p

p D 1.1.

200

Chapter 3 Multiparameter regularization

The first series of experiments is performed with the i laplace.n, 1/ function from [71] which has been already used in Subsection 3.3.1. Noisy data are simulated with h D  D 0.2. In our algorithm for the DRTLS rule we choose ˛0 D 10, ˇ0 D 0.2 as starting values, and  D 0.01 for stopping the iteration. The results are displayed in Figure 3.10.

Min = 0.1279, Max = 0.2929

Quasi B = I

Min = 0.0132, Max = 0.2799

Quasi B = D

Min = 0.0824, Max = 0.0987

Multiparameter choice

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

Figure 3.10. Comparison of relative errors in the approximate solutions for ilaplace.100, 1/ computed using the Tikhonov method (3.66), B D I , B D D, with quasioptimality criterion, and by dual RTLS, which has been realized by our model function approach (from [109]).

As can be seen from this figure, the Tikhonov method with B D I exhibits poor performance; the relative error is between 0.1 and 0.2. For B D D the results are better, but the performance is not stable; the relative error varies between 0.01 and 0.16. At the same time, dual RTLS (which has been realized by our model function approach) exhibits stable performance; the relative error is between 0.08 and 0.1. The second example is performed with the ilaplace.n, 2/ function in [71]. Similar to ilaplace.n, 1/, we have the same kernel k in equation (3.29), but f .t / D 1  exp.t =2/. This time we take a discretization level n D 64, since we would like to compare our results with [62], which used the same discretization level. In noise simulations, h D  D 0.1 has been used. The results are presented in Figure 3.11. It is known [71] that the Tikhonov method with B D I fails to handle the example ilaplace.n, 2/. For B D D the results are reasonable, but again, the performance is not stable. As to dual RTLS, it exhibits stable performance, and it is interesting to note that similar relative errors were reported in [62] for ilaplace.64, 2/ (a slight change in the solution) and RTLS, but the parameters ˛, ˇ were chosen in [62] “by hand” using the knowledge of the exact solution. In our experiments similar accuracy has been obtained automatically. In our final example we use the function gravity.n, example, 0, 1, 0.25/ [71]. It corresponds to the Fredholm equation (3.29) with k.s, t / D 0.25..0.25/2 C .s  t /2 /3=2 ,

f .t / D sin. t / C 0.5 sin.2 t /

201

Section 3.4 Two-parameter regularization with one negative parameter

Min = 0.8306, Max = 0.8486

Quasi B = I

Min = 0.0048, Max = 0.3232

Quasi B = D

Min = 0.1455, Max = 0.1571

Multiparameter choice

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Figure 3.11. Comparison of relative errors in the approximate solutions for ilaplace.64, 2/ computed using the Tikhonov method (3.66), B D I , B D D, with quasioptimality criterion, and by dual RTLS, which has been realized by our model function approach (from [109]).

Quasi B = I Quasi B = D

Min = 0.0290, Max = 0.1083 Min = 0.0382, Max = 0.1678

Multiparameter choice

0.02 0.04

Min = 0.0768, Max = 0.0867

0.06 0.08

0.1

0.12 0.14

0.16

0.18

Figure 3.12. Comparison of relative errors in the approximate solutions for gravity .100, 1, 0, 1, 0.25/ computed by Tikhonov method (3.66), B D I , B D D, with quasioptimality criterion and by dual RTLS which has been realized by our model function approach (from [109]).

discretized on the interval Œ0, 1 by means of a collocation scheme with n D 100 knots. Noisy data are simulated with h D  D 0.2. The results are displayed in Figure 3.12. In this example, the standard Tikhonov method with B D I produces the best results, while the method with B D D is extremely unstable. As to dual RTLS (which has been realized by our model function approach), its maximal relative error is almost the same as for the standard Tikhonov method and it is still the most stable among the tested methods. Thus, from our tests we conclude that the performance of the Tikhonov method with B D I and B D D depends on the problem, while dual RTLS (computed by our model function approach) exhibits a stable reliability independent of the problems under consideration.

Chapter 4

Regularization algorithms in learning theory

4.1

Supervised learning problem as an operator equation in a reproducing kernel Hilbert space (RKHS)

In this section we discuss the relationship between learning theory and the regularization of linear ill-posed problems. It is well-known that the Tikhonov–Phillips regularization scheme can be profitably used in the context of supervised learning, where it usually goes under the names “proximal vector machines” or “least squares support vector machines” (see, e.g., [136]). The remarkable fact is that our analysis shows that the same properties are shared by a large class of learning algorithms which are essentially all the linear regularization schemes discussed in Chapter 2. The concept of operator monotone functions used in Section 2.8.2 turns out to be an important tool for the analysis of regularized learning. The techniques becoming known as “supervised learning” or “learning from examples” refer to algorithms that are used to predict an output, say y, of a system under study on the basis of a set of training examples, that is a set of input-output pairs zi D .xi , yi /, i D 1, 2, : : : , n, observed in the same system. Algorithms which could teach from examples zi to perform a prediction of y for a new input x would have many applications. For instance, algorithms have been developed which can teach prediction of future blood-sugar levels of diabetic patients from a set of examples, that is from examples of patterns of blood-sugar monitoring of a particular patient which took part in clinical trials. More details will be forthcoming in the next chapter. Throughout this section the inputs x are assumed to be taken from a compact domain or a manifold X in the Euclidean space Rd , and the input space is Y Rk . For convenience we will mainly consider k D 1, where a common assumption is Y D ŒB, B for some B > 0. We also let Z D X  Y be the sample set containing a training set ¹zi D .xi , yi /º. In supervised learning the input x and the output y are assumed to be related by a probabilistic relation because generally an element x 2 X does not uniquely determine an element y 2 Y , but rather a conditional probability .yjx/ of y given x, which is assumed to be unknown. Moreover, the input x is also assumed to be random and governed by an unknown marginal probability X on X , so that there is an unknown probability measure .x, y/ D X .x/.yjx/ on the product space Z D X  Y , from which a training data set z D .x, y/ D ¹.x1 , y1 /, : : : , .xn , yn /º is independently drawn. As mentioned in [119], it is important to clarify the role of probability in this approach: we always have incomplete data about a system, and in order to understand

204

Chapter 4 Regularization algorithms in learning theory

its input-output relationship we do not need to know all the things which affect the system being studied. The uncertainty in a given situation is therefore rather a matter of efficiency, when one seeks to replace all complicated affecting factors by stochastic perturbations. What is assumed in learning theory is that a machine can be trained to perform a prediction task, given data of the form .xi , yi /, i D 1, 2, : : : , n. Training here means synthesizing a function y D f .x/ which will represent the relationship between the inputs x and the corresponding outputs y. In this sense, learning is similar to fitting a function f D fz to a certain number of given data z D ¹.xi , yi /, i D 1, 2, : : : , nº. The key point is that the fitting fz should in principle uncover the underlying input-output relationship, which is then used in a predictive way. The central question in learning theory is how well this function fz estimates the outputs y for previously unseen inputs x. For a chosen loss function l.y, f .x//, say l.y, f .x// D .y  f .x//2 , the error incurred by a function fz can be measured by the expected risk Z .y  fz .x//2 d.x, y/. E.fz / D XY

Note that in the space L2 .X , X / of square integrable functions with respect to the marginal probability measure X the risk E.fz / is minimized by the so-called target or regression function Z yd.yjx/. f .x/ D Y

To see this observe that due to a version of Fubini’s theorem  Z Z Z Z .y  f .x//d.x, y/ D y yd.yjx/ d.yjx/dX .x/ D 0, XY

X

and then

Y

Y

Z

E.fz / D

.fz .x/  f .x/ C f .x/  y/2 d.x, y/ Z ZXY .fz .x/  f .x//.f  y/d.x, y/ D .fz .x/  f .x//2 dX .x/ C 2 X XY Z .f .x/  y/2 d.x, y/ C XY

D kfz  f k2 C E.f /,

(4.1)

where k  k is the norm in L2 .X , X /. Thus, equation (4.1) implies that f has the smallest possible risk E.f / among all functions fz : X ! Y . Unfortunately, this ideal estimator cannot be found in practice, because the conditional probability .yjx/ that defines f is unknown, and only a sample of it, the training set z, is available. Therefore, in view of equation (4.1)

205

Section 4.1 Supervised learning problem

we can restate the problem as that of approximating the regression function f in the norm k  k . The goal of learning theory might be said to be finding f minimizing the approximation error kf  f k . In order to search for such an f , it is important to have a hypothesis space H in which to work. It is clear that such H should be a proper subset of L2 .X , X /, since for H D L2 .X , X /, the problem becomes tautological, i.e., f D f . Once we choose a hypothesis space H , the best achievable error is clearly inf E.f /.

(4.2)

f 2H

In general, such an achievable error can be bigger than E.f / and the existence of an extremal function is not even ensured. Now let IH : H ! L2 .X , X / be the inclusion operator of H into L2 .X , X /, and P : L2 .X , X / ! Range.IH / the orthoprojector of L2 .X , X / onto the closure of the range of IH in L2 .X , X /. In view of equation (4.1) the problem (4.2) can then be written as follows: inf E.f / D inf kIH f  f k2 C E.f /,

f 2H

f 2H

(4.3)

and from Proposition 2.1 it follows that the inclusion of Pf 2 Range.IH / is a sufficient condition for the existence and uniqueness of the minimizer of equation (4.2),  which is nothing but the Moore–Penrose generalized solution fH of the embedding equation I H f D f .

(4.4)

Thus, under the assumption that Pf 2 Range.IH /, we search for an estimator fz of  the solution fH of equation (4.4) from given training data z. In view of equation (4.1) we have 



kfz  fH k2 D E.fz /  E.fH /,

(4.5)

and, as we discuss in the following, under some more assumptions this also ensures a good approximation for f . For example, if f 2 H (that is f 2 Range.IH /), clearly   f D fH (that is, f D IH fH ).

4.1.1

Reproducing kernel Hilbert spaces and related operators

A Reproduction Kernel Hilbert Space (RKHS) is a Hilbert space H D H .X / of pointwise defined functions f : X ! R, with the property that for each x 2 X and f 2 H the evaluation functional lx .f / :D f .x/ is continuous (i.e., bounded) in the topology of H .

206

Chapter 4 Regularization algorithms in learning theory

It is known [171] that every RKHS can be generated from a unique symmetric and positive definite function K : X  X ! R of two variables in X , called the reproducing kernel of H D H .X ; K/. Recall that a function K : X  X ! R is called positive definite on X if for any n and distinct x1 , x2 , : : : , xn 2 X , the quadratic form PnanyPpairwise n b b K.x hKb, biRn D i , xj / is positive for all n-dimensional vectors iD1 j D1 i j n b D .b1 , b2 , : : : , bn / 2 R n ¹0º, where K D ¹K.xi , xj /ºni,j D1 is sometimes called the Gram matrix of K : X  X ! R. e D To illustrate the way a kernel K generates an RKHS we consider a linear space H e .K/ of all possible finite linear combinations of kernel sections Kx D K.xi , /, H i xi 2 X . For any two such combinations f1 D

n X

f2 D

ci,1 Kx1,i ,

m X

c2,j Kx2,j ,

j D1

iD1

one can introduce the inner product hf1 , f2 iH :D

m n X X

c1,i c2,j K.x1,i , x2,j /,

iD1 j D1

which is well-defined since the kernel K is symmetric and positive definite. e .K/ with Then RKHS H D H .X ; K/ D HK can be built as the completion of H respect to the inner product h, iH , and moreover, for any f 2 H the following reproducing property easily follows: f .x/ D hf , Kx iH , x 2 X . Simple examples of kernels are the Gaussian kernel K.x, t / D exp. kx  t k2Rd /, and the polynomial kernel of degree p K.x, t / D .1 C hx, t iRd /p . Assume that we have a finite or infinite sequence of positive numbers k and linearly independent functions fk : X ! R, such that they define a function K : X  X ! R in the following way: X K.x, t / :D k fk .x/fk .t /, (4.6) k

where, in the case of infinite sequences, the series is well-defined, say, it converges uniformly. It can easily be shown that the function defined by equation (4.6) is positive definite. Vice versa, for any RKHS there is a unique kernel K and corresponding ¹ k º,

207

Section 4.1 Supervised learning problem

¹fk º that satisfy equation (4.6). Moreover, the ¹fk º is a basis for the RKHS. The mathematical details can be found in [171]. In the following we always assume that K : X  X ! R is a continuous and bounded kernel, that is, p (4.7) sup K.x, x/   < 1. x2X

Then using the Cauchy–Schwartz inequality for f 2 H .K/ it holds that sup jf .x/j  kf kH .

x2X

Moreover, in view of equation (4.7), the inclusion operator IK D IH .K/ : H ! L2 .X , X / which has been introduced above is continuous.  : L .X ,  /!H , the Following [152], we also consider the adjoint operator IK 2 X  covariance operator T : H !H , such that T D IK IK and the operator LK :  . It can easily be proved that L2 .X , X /!L2 .X , X /, such that LK D IK IK Z Z  Kx f .x/dX .x/ T D IK f ./ D h, Kx iH Kx dX .x/. X

X

The operators T and LK can be proved to be positive trace class operators (and hence compact). For a function f 2 H we can relate the norm in H and L2 .X , X / using T . In fact, if we regard f 2 H as a function in L2 .X , X / we can write   p  f  D  T f  . (4.8)

H This fact can easily be proved by recalling thatpthe inclusion operator is continuous and partial isometry [145]. hence admits a polar decomposition IK D U T , where U is aP Finally, replacing X by the empirical measure x D n1 niD1 ıxi on a sample x˝ D ¹xi˛ºniD1 , we can define the sampling operator Sx : H !Rn by .Sx f /i D f .xi / D f , Kxi H ; i D 1, : : : , n, where the norm kkn in Rn is 1=n times the Euclidean norm. Moreover, we can define Sx : Rn !H , the empirical covariance operator Tx : H !H , such that Tx D Sx Sx and the operator Sx Sx : Rn !Rn . It follows that for D . 1 , : : : , n / 1X Kxi i n n

Sx D

iD1

˛ 1 X˝ , Kxi H Kxi . n n

Tx D

iD1

Moreover, Sx Sx D n1 K, where K is the kernel matrix, such that .K/ij D K.xi , xj /. Throughout we indicate with kk the norm in the Banach space L.H / of bounded linear operators from H to H .

4.1.2

A priori assumption on the problem: general source conditions

Thus, the supervised learning problem can been reduced to the embedding operator equation, and it is well known that to obtain bounds on equation (4.5) we have to

208

Chapter 4 Regularization algorithms in learning theory

restrict the class of possible probability measures generating f . In learning theory this is related to the so called “no free lunch” theorem [47], but a similar kind of phenomenon occurs in the regularization of ill-posed inverse problems as discussed in Section 2.3. Essentially what happens is that we can always find a solution with convergence guarantees to some prescribed target function, but the convergence rates can be arbitrarily slow. In our setting this turns into the impossibility of stating finite sample bounds holding uniformly with respect to any probability measure . A standard way to impose restrictions on the class of possible problems is to consider a set of probability measures M.W /, such that the associated regression function satisfies f 2 W . Such a condition is called the prior. The set W is usually a compact set determined by smoothness conditions. In the context of RKHS it is natural to describe the prior in term of the compact operator LK , considering f 2 Wr,R with Wr,R D ¹f 2 L2 .X , X / : f D LrK u, kuk  Rº. (4.9)  r  The above condition is often written as LK f   R (see [152]). Note that when r D 1=2, such a condition is equivalent to assuming f 2 H and is independent of the measure , but for arbitrary r it is distribution dependent. Note that the condition f 2 Wr,R corresponds to what is called a source condition in Section 2.3. Indeed, if we consider Pf 2 Wr,R , r > 1=2, then Pf 2 Range.IK /,  and we can equivalently consider the prior fH 2 WQ ,R with WQ ,R D ¹f 2 H : f D T v, kvkH  Rº,

(4.10)

 where D r 1=2 (see, for example, [45] Proposition 3.2). Recalling that T D IK IK , we see that the above condition is the standard source condition for the linear problem  IK f D f , namely Hölder source condition x  D fH 2 A .R/, A D IK , .s/ D s . Following on from Chapter 2, we wish to extend the class of possible probability measures M.W / considering general source condition. We assume throughout that  Pf 2 Range.IK /, which means that fH exists and solves the normalized embedding  f . equation Tf D IK  Thus, additional smoothness of fH can be expressed as an inclusion

fH 2 WQ ,R :D ¹f 2 H : f D .T /v, kvkH  Rº, 

(4.11)

that in accordance with the terminology of Section 2.3 is called the source condition generated by an index function . There is good reason to further restrict the class of possible index functions. In general, the smoothness expressed through source conditions is not stable with respect to perturbations in the involved operator T . In learning theory only the empirical covariance operator Tx is available and it is desirable to control .T /  .Tx /. As we know from Section 2.8.2, this can be achieved by requiring  to be operator monotone. Recall that the function  is operator monotone on Œ0, b if for any pair of self-adjoint operators U , V , with spectra in Œ0, b such that U  V , we have .U /  .V /. The

209

Section 4.2 Kernel independent learning rates

partial ordering B1  B2 for self-adjoint operators B1 , B2 on some Hilbert space H means that for any h 2 H , hB1 h, hi  hB2 h, hi. It follows from Löwner’s theorem (see, for example, Section 2.8.2 [Fact 2.3] or [70]), that each operator monotone function on .0, b/ admits an analytic continuation in the corresponding strip of the upper half-plane with a positive imaginary part. Proposition 2.22 tells us that operator monotone index functions allow a desired norm estimate for .T /  .Tx /. Therefore in the following we consider index functions from the class FC D ¹

: Œ0, b!RC , operator monotone, .0/ D 0, .b/  C , b >  2 º.

Note that from Remark 2.9 it follows that an index function 2 FC cannot converge faster than linearly to 0. To overcome this limitation of the class FC we also introduce the class F of index functions  : Œ0,  2 !RC , which can be split into a part 2 FC and a monotone Lipschitz part # : Œ0,  2 !RC , #.0/ D 0, i.e., . / D #. / . /. This splitting is not unique such that we implicitly assume that the Lipschitz constant for # is equal to 1 which means k#.T /  #.Tx /k  kT  Tx k. The fact that an operator valued function # is Lipschitz continuous if a real function # is Lipschitz continuous follows from Theorem 8.1 in [24]. Remark 4.1. Observe that for 2 Œ0, 1, a Hölder-type source condition (4.10) can be seen as equation (4.11) with . / D  2 FC , C D b , b >  2 while for > 1 we can write . / D #. / . / where #. / D  p =C1 and . / D C1  p 2 FC , C D C1 b p , b >  2 , C1 D p 2.p1/ and p D Œ  is an integer part of or p D  1 if is an integer number. It is clear that the Lipschitz constant for such a #. / is equal to 1. At the same time, source conditions (4.11) with  2 F cover all types of smoothness studied so far in regularization theory. For example, . / D  p log 1= with p D 0, 1, : : : , 2 Œ0, 1 can be split into a Lipschitz part #. / D  p and an operator monotone part . / D log 1= .

4.2

Kernel independent learning rates

As already mentioned, Tikhonov regularization can be used profitably in the context of supervised learning and many theoretical properties have been demonstrated. The question whether other regularization techniques from the theory of ill-posed inverse problems can be valuable in the context of learning theory has been considered in [144], motivated by some connections between learning and inverse problems [136]. In this section we follow the same approach and provide a refined analysis for algorithms defined by fz D g .Tx /Sx y,

(4.12)

210

Chapter 4 Regularization algorithms in learning theory

where ¹g º is a one-parameter regularization family covered by Definition 2.2, and the final estimator is defined providing the above scheme with a parameter choice n D .n, z/, so that fz D fzn . Note that in [144] an extra condition was required on g compared to Definition 2.2, namely a Lipschitz condition. Here we show that at least in the prior range considered such a condition can be dropped and the conditions listed in Definition 2.2 are sufficient for learning. Note that in contrast to [36], who analyzed the Tikhonov regularization in the context of learning, the results of this section are more general and valid simultaneously for all kernels meeting the assumption (4.7). In this sense the results are kernel independent, which is reflected in the title of the section. The advantage of this is that such results can provide justification for a learning algorithm (4.12) in the situation, where a kernel is not given a priori, but needs to be chosen from some admissible set. More details will be given in the next chapter. The following result has been proved in [46] and is useful in the analysis of error estimates for a fixed value of the regularization parameter . Lemma 4.1. Assume that there are positive constants †, M , such that for almost all inputs x 2 X it holds  Z  jyf  .x/j  jy  fH .x/j †2 H  .  1 d.yjx/  e M M 2M 2 Y If the inequality (4.7) is satisfied, then for any  2 .0, 1/ with probability at least 1   we have 

kTx fH  Sx ykH  ı1 ,

kT  Tx k  ı2 ,

where 

 M † 4 Cp ı1 :D ı1 .n, / D 2 log , n  n 1 p 2 4 ı2 :D ı2 .n, / D p 2 2 log .  n The above result provides us with the probabilistic perturbation measures which quantify the effect of random sampling. Note that in [36] it was mentioned that the assumption of Lemma 4.1 is weaker than a common assumption that Y 2 ŒB, B for some B > 0. We are now ready to state the following proposition. Proposition 4.1. Let  2 .0, 1. Assume the conditions of Lemma 4.1. Moreover, as sume that Pf 2 Range.IK / and fH 2 WQ ,R , where  D # 2 F , 2 FC . We let fz as in equation (4.12), satisfying Definition 2.2 and assume that the regularization

211

Section 4.2 Kernel independent learning rates

p ¹g º has a qualification which covers . /  . If 1 p 4   p 2 2 2 log  n

(4.13)

for 0 <  < 1, then with probability at least 1     p 1 4   kfz  fH k  C1 ./  C C2 p log ,  n

(4.14)

where C1 D 2.1 C c / N R, N and c D c are the constants from the statements of Propositions 2.7 and 2.22, and    p p M . 0 C 1/ 1 C 1 † C p . C2 D .1 C N / 0 CR2 2 2 C 2 Moreover, with probability at least 1     1 4   kfz  fH kH  C3 ./ C C4 p log ,   n    p 2 where C3 D .1 C c / N R and C4 D 0 CR2 2 C 1 † C pM2 .

(4.15)

Proof. We let r . / D 1  g . /, and consider the following decomposition into two terms fH  fz D fH  g .Tx /Sx y 



D .fH  g .Tx /Tx fH / C .g .Tx /Tx fH  g .Tx /Sx y/. 





(4.16)

The idea is then to separately bound each term both in the norm in H and in L2 .X , X /. We start dealing with the first term. Using equation (4.11) we can write 



fH  g .Tx /Tx fH D .I  g .Tx /Tx /.T /v D r .Tx /.Tx /v C r .Tx /..T /  .Tx //v D r .Tx /.Tx /v C r .Tx /#.Tx /. .T / 

.Tx //v

C r .Tx /.#.T /  #.Tx // .T /v.

(4.17)

When considering the norm in H , we know that Proposition 2.7 applies since  (as well as #) is covered by the qualification of g . The fact that # is covered by the qualification 0 of g can be seen from the following chain of inequalities: inf

 2

 0  0 . / D inf  #. / #. / . /  2  0  0 D ,  ./ ./ #./

./

inf

 2

 0 . /

212

Chapter 4 Regularization algorithms in learning theory

where we rely on the fact that ./ D ./#./ is covered by the qualification of g , and an operator monotone index function ./ is nondecreasing. We can then use equation (4.11), Definition 2.2, Proposition 2.7, and Proposition 2.22 to get the bound 



N C N c R#./ .kT  Tx k/ C 0 CRkT  Tx k, kfH  g .Tx /Tx fH kH  R./ and from Lemma 4.1 with probability at least 1   we have 



kfH  g .Tx /Tx fH kH  .1 C c / N R./ C 0 CRı2 ,

(4.18)

where we used equation (4.13) to have #./ .kT  Tx k/  #./ .ı2 /  #./ ./ D ./. Some more reasoning is needed to get the bound in L2 .X , X /. To this aim in place of equation (4.17) we consider p p p    T .fH  g .Tx /Tx fH / D . T  Tx /.I  g .Tx /Tx /fH p  C Tx .I  g .Tx /Tx /fH . (4.19) The first addend is easy p to bound since from the condition (4.13) and operator monotonicity of . / D  with probability at least 1   we get p p p p p (4.20) k T  Tx k  kT  Tx k  ı2  . From the above inequality and from equation (4.18) we then get p  p p p   T  T /.I  g .T /T /f  .1 C c / N R./  C 0 CR ı2 . .  x  x x H H

(4.21) On the other hand, the second addend in equation (4.19) can be further decomposed using equation (4.11) p p Tx .I  g .Tx /Tx /.T /v D Tx r .Tx /.Tx /v p C Tx r .Tx /#.Tx /. .T /  .Tx //v p C Tx r .Tx /.#.T /  #.Tx // .T /v. Using equation (4.11), Definition 2.2, and Propositions 2.7 and 2.22 we get with probability at least 1   p  p p    Tx .I  g .Tx /Tx /fH   .1 C c / N R./  C N 0 CR ı2 , H

where again we used equation (4.13) to have .kT  Tx k/  .ı2 /  ./. Now we can put the above inequality, equation (4.8), and equation (4.21) together to obtain the following bound in the -norm p  p p     T .f  g .T /T f /  2.1 C c / N R./  C .1 C N / 0 CR ı2 .   x x H  H H

(4.22)

213

Section 4.2 Kernel independent learning rates

We are now ready to consider the second term in equation (4.16). If we consider the norm in H we can write g .Tx /Tx fH  g .Tx /Sx y D g .Tx /.Tx fH  Sx y/, 



and in view of Definition 2.2 from Lemma 4.1 it follows that with probability at least 1   we have   1    (4.23) ı1 . g .Tx /Tx fH  g .Tx /Sx y  H  Moreover, when considering the norm in L2 .X , X / we simply have p p   T .g .Tx /Tx fH  g .Tx /Sx y/ D Tx g .Tx /.Tx fH  Sx y/ p p  C . T  Tx /g .Tx /.Tx fH  Sx y/. (4.24) It is easy to show that

 p.1 C / p 0 1   , p  Tx g .Tx /  

because for any h 2 H from the Cauchy–Schwartz inequality we have ˇDp E ˇ p ˇ ˇ ˇ Tx g .Tx /h, Tx g .Tx /h ˇ D jhg .Tx /h, Tx g .Tx /hij H

 kg .Tx /hkH kTx g .Tx /hkH .1 C 0 / 1  khk2H ,  where we used Definition 2.2. We can use the definition of ı1 with the above inequality to bound the first addend in the second equation (4.24) and the definition of ı1 with the inequality (4.20) topboundp addend in equation (4.24). Then, using equation (4.13), we have ı2   so that with probability at least 1   p p  p 1 .1 C 0 / 1     ı1 C ı2 p ı1  T .g .Tx /Tx fH  g .Tx /Sx y/  H   p . .1 C 0 / 1 C 1 / ı1 .  p (4.25)  We are now in a position to derive the desired bounds. Recalling (4.16), we can put equations (4.18) and (4.23) together to get with probability at least 1  ,   1    ı1 . fz  fH   .1 C c / N R./ C 0 CRı2 C H 

214

Chapter 4 Regularization algorithms in learning theory

We can then simplify the above bound. In fact ı2  ı2 = since   1 so that p 4 1 0 CRı2  log 0 CR2 2 2 p .   n Moreover, from the explicit expression of ı1 , using equation (4.13) and   1, it is easy to prove that   4 1 M 1 p . ı1  log 1 † C p   2  n Putting everything together we have equation (4.15). Similarly we can use equation (4.8) to write p            f D T .f  f / f   z z H H 

H

and from equations (4.22) and (4.25) we get with probability at least 1   p  p p     T .fz  fH /  2.1 C c / N R./  C .1 C N / 0 CR ı2 H p . .1 C 0 / 1 C 1 / ı1 , C p  which can be further simplified as above to get equation (4.14). Remark 4.2 (Assumption on the regularization parameter). A condition similar to equation (4.13) has been considered in [152]. It simply indicates the range of regularization parameters for which the error estimates (4.14) and (4.15) are nontrivial. For example, if  does not satisfy equationp(4.13), then the right-hand side of (4.15) becomes larger than a fixed constant C4 =.2 2 2 /, which is not reasonable. Thus, the condition (4.13) is not restrictive at all. In fact, it is automatically satisfied for the best a priori choice of the regularization parameter (see Proposition 4.2 below) balancing the values of the terms in the estimates (4.14) and (4.15). Finally, the condition  < 1 is considered only to simplify the results and can be replaced by  < a for some positive constant a (and in particular for a D ) which would eventually appear in the bound. Remark 4.3 (Assumption on the best in the model). If H is dense in L2 .X , X / or   f 2 H we can clearly replace fH with f , since E.fH / D inff 2H E.f / D E.f /. 

A drawback in our approach is that we have to assume the existence of fH . Though this assumption is necessary to study results in the H -norm, it can be relaxed when  looking for bounds in L2 .X , X /. In fact, as discussed in [45, 175], if fH does not exist we can still consider 2  E.fz /  inf E.f / D fz  Pf 

H

215

Section 4.2 Kernel independent learning rates

in place of equation (4.5). For this kind of prior (but Hölder source condition), the results of Proposition 4.1 were obtained in [152] for Tikhonov regularization. The result on Landweber iteration in [175] also cover this case, though the dependence on the number of examples is worse than for the Tikhonov one. Results for general regularization schemes were obtained in [144] requiring the regularization g to be Lipschitz, but the dependence on the number of examples was again spoiled. Remark 4.4 (Bounds uniform w.r.t. ). Inspecting the proof of the above proposition we see that the family of good training sets where the bounds hold with high probability do not depend on the value of the regularization parameter. This turns out to be useful to define a data-driven strategy for the choice of  (an issue discussed in the section below). From the above results we can immediately derive a data independent (a priori) parameter choice n D .n/. The next proposition shows the error bounds obtained providing the one-parameter family of algorithms in equation (4.12) with such a regularization parameter choice. Proposition 4.2. We let ‚./ D ./. Under the same assumptions of Proposition 4.1 we choose n D ‚1 .n 2 /, 1

and let fz D fzn . Then for 0 <  < 1 and n 2 N, such that p 1 1 4 ‚1 .n 2 /n 2  2 2 2 log , 

(4.26)

(4.27)

the following bound holds with probability at least 1   q   1 4 1   fz  fH   .C1 C C2 /.‚1 .n 2 // ‚1 .n 2 / log ,

 with C1 and C2 as in Proposition 4.1. Moreover with probability at least 1     1 4   fz  fH   .C3 C C4 /.‚1 .n 2 // log , H  with C3 and C4 as in Proposition 4.1. Proof. If we choose n as in equation (4.26), then for n such that equation (4.27) it holds that condition (4.13) is verified and we can apply the bounds of Proposition 4.1 to n . The results easily follow, noting that the proposed parameter choice is the one balancing the two terms in equation (4.14); in fact the following equation is verified for  D n : p 1 ./  D p n 1 1=2 (./ D  n for the H -norm).

216

Chapter 4 Regularization algorithms in learning theory

Remark 4.5 (Kernel-independent lower bounds). As far as we know, no minimax lower bounds exist for the class of priors considered here. In fact, in [36] lower bounds are presented for  2 M.Wr,R /, that is, Hölder source condition, and considering the case where the eigenvalues of T have a polynomial decay sk .T / D O.k b /, b > 1. rb  In this case the lower rate kfz  fH k D O.n 2rbC1 /, 1=2 < r  1 is shown to be optimal. Here we do not make any assumptions about the kernel and, in this sense, our results are kernel-independent. This situation can be thought of as the limit case when b D 1. As can be seen from Proposition 4.3, we share the same dependence on the smoothness index r. The following result considers the case of Hölder source conditions, that is, the case when condition (4.11) reduces to equation (4.10). Recalling the equivalence between equations (4.9) and (4.10), we state the following result, considering D r  1=2, in order to have an easier comparison with previous results. Proposition 4.3. Under the same assumption of Proposition 4.2, let . / D  ,

D r  1=2. Now choose n as in equation (4.26), and let fz D fzn . Then for 0 <  < 1 and  4rC2  p 2 4 2rC3 n > 2 2 log 

(4.28)

the following bounds hold with probability at least 1     r 4   fz  fH   .C1 C C2 /n 2rC1 log ,

 with C1 and C2 as in Proposition 4.1 and   r1=2 4   fz  fH   .C3 C C4 /n 2rC1 log , H  with C3 and C4 as in Proposition 4.1. Proof. By a simple computation we have n D ‚1 .n1=2 / D n 2rC1 . Moreover, the condition (4.27) can now be written explicitly as in equation (4.28). The proof follows plugging the explicit form of  and n into the bounds of Proposition 4.2. 1

Remark 4.6. Clearly, if in place of Pf 2 Wr,R we take f 2 Wr,R with r > 1=2,  then f 2 H and we can replace fH with f , since inff 2H E.f / D E.f /. In particular we discuss the bounds corresponding to the examples of regularization algorithms discussed in Section 2.2, and for the sake of clarity we restrict ourselves to polynomial source conditions and H dense in L2 .X , X /. Tikhonov regularization. In the considered range of prior (r > 1=2), the above results match those obtained in [152] for Tikhonov regularization. We observe that this

217

Section 4.2 Kernel independent learning rates

kind of regularization suffers from a saturation effect, and the results no longer improve after a certain regularity level, r D 1 (or r D 3=2 for the H -norm) is reached. This is a well-known fact in the theory of inverse problems (see discussion in Section 2.4). Landweber iteration. In the considered range of prior (r > 1=2), the above results improve on those obtained in [175] for gradient descent learning. Moreover, as pointed out in [175], such an algorithm does not suffer from saturation and the rate can be extremely good if the regression function is regular enough (that is, if r is big enough), though the constant gets worse. Spectral cut-off regularization. The spectral cut-off regularization does not suffer from the saturation phenomenon, and moreover the constant does not change with the regularity of the solution, allowing extremely good theoretical properties. Note that such an algorithm is computationally feasible if one can compute the SVD of the kernel matrix K. 

Remark 4.7. Note that, though assuming that fH exists, we improve the result in [144] and show that in the considered range of prior we can drop the Lipschitz assumption on g and obtain the same dependence on the number of examples n and on the confidence level  for all regularization g satisfying Definition 2.2. This class of algorithms includes all the methods considered in [144], and in general all the linear regularization algorithms to solve ill-posed inverse problems. The key to avoiding the Lipschitz assumption on g is exploiting the stability of the source condition w.r.t. to operator perturbation.

4.2.1

Regularization for binary classification: risk bounds and Bayes consistency

We briefly discuss the performance of the proposed class of algorithms in the context of binary classification [26, 165], that is, when Y D ¹1, 1º. The problem is that of discriminating the elements of two classes, and as usual we can take the signfz as our decision rule. In this case some natural error measures can be considered. The risk or misclassification error is defined as R.f / D  .¹.x, y/ 2 X  Y j signf .x/ ¤ yº/ , whose minimizer is the Bayes rule signf . The quantity we aim to control is the excess risk R.fz /  R.f /. Moreover, as proposed in [152] it is interesting to consider   signfz  signf  .

218

Chapter 4 Regularization algorithms in learning theory

  To obtain bounds on the above quantities the idea is to relate them to fz  f  . A straightforward result can be obtained recalling that   R.fz /  R.f /  fz  f  , see [12, 175]. In any case it is interesting to consider the case when some extra information is available on the noise affecting the problem. This can be done considering the Tsybakov noise condition [165] X .¹x 2 X : jf .x/j  Lº/  Bq Lq , 8L 2 Œ0, 1, where q 2 Œ0, 1. As shown in Proposition 6.2 in [175], the following inequalities q : hold for ˛ D qC1   2 R.fz /  R.f /  4c˛ fz  f  2˛ ,     ˛ signfz  signf   4c˛ fz  f  2˛ .



with c˛ D Bq C 1. A direct application of Proposition 4.2 immediately leads to the following result. Proposition 4.4. Assume that H is dense in L2 .X , X /, and that the same assumptions of Proposition 4.2 hold. Choose n according to equation (4.26) and let fz D fzn . Then for 0 <  < 1 and n satisfying equation (4.27) the following bounds hold with probability at least 1    2  q 4 2˛ 1  12  12 1 , R.fz /  R.f /  4c˛ .C1 C C2 /.‚ .n / ‚ .n / log  ˛  2˛  q   signfz  signf   4c˛ .C1 C C2 /.‚1 .n 12 / ‚1 .n 12 / log 4 ,

 with C1 ,C2 ,C3 and C4 given in Proposition 4.1. Proposition 4.3 shows that for polynomial source conditions this means2r all the proposed algorithms achieve risk bounds on R.fz /  R.f / of order n .2rC1/.2˛/ if n is big enough (satisfying equation (4.28)). In other words, the algorithms we propose are Bayes-consistent with fast rates of convergence.

4.3 Adaptive kernel methods using the balancing principle In the previous section we considered a learning algorithm as a map z!fz , but in practice most algorithms can be seen as a two-step procedure. The first step defines a family of solutions depending on a real regularization parameter z!fz ,  > 0,

219

Section 4.3 Adaptive kernel methods using the balancing principle

whereas the second step determines how to choose the regularization parameter . The final estimator is obtained only once both steps have been defined. One fundamental approach to model selection in learning theory [26, 42, 157, 169] is based on deriving excess risk bounds for any , and choosing a priori the value optimizing the bound. As can be seen, for example, from equation (4.5) and Proposition 4.1, excess risk bounds are usually given by the sum of two competing terms, i.e., 



E.fz /  E.fH / D kfz  fH k2

 .S.n, , / C A., //2 .

(4.29)

The term S.n, , / is the so-called sample error and quantifies the error due to random sampling. The term A., / is called approximation error, it does not depend on the data, but requires prior knowledge of the unknown probability distribution. The typical behavior of the two terms (for a fixed n) is depicted in Figure 4.1.

A (η, λ)

log10

S (n, η, λ)

λ0

λ

0

Figure 4.1. The figure represents the behavior of sample and approximation errors; respectively S.n, , / and A., /, as functions of , for fixed n,  (from [44]).

The best possible regularization parameter choice is found by solving a sampleapproximation (or bias-variance) trade-off, that is, from the balancing of these two terms. In view of Proposition 4.2, rather than the value optimizing the bound, we consider the value of o D o .n/  n making the contribution of the two terms equal (the crossing point in Figure 4.1). One can see that the corresponding error estimate is, with probability at least 1   

E.fzo .n/ /  E.fH /  4S 2 .n, , o / D 4A2 ., o /.

(4.30)

220

Chapter 4 Regularization algorithms in learning theory

Note that the a priori choice  D o as in equation (4.30) leads to a learning rate which is only worse than the optimal rate in equation (4.28) by a factor of 4, given as inf .S.n, , / C A., //2 . 

Indeed, assume for simplicity, that 1 D 1 .n/ exists, for which the above infimum is achieved. Then S 2 .n, , o / D A2 ., o /  inf .S.n, , / C A., //2 

because either o  1 , in which case A., o /  A., 1 /  inf .S.n, , / C A., // , 

(see Figure 4.1), or 1 < o and S.n, , o /  S.n, , 1 /  inf .S.n, , / C A., // . 

From the above discussion one can see that the parameter choice  D o .n/ depends  on the regularity properties of fH which are coded in A., / and usually unknown. This stimulates the interest in the adaptive parameter choice which we will study in this section. At first glance, in view of the relation between learning theory and theory of regularization, as discussed in the previous section, a posteriori parameter choice strategies developed for regularization, such as the balancing principle, for example, can also be used in the context of learning. In fact a direct use of such strategies is problematic, however, since they are based on estimates of the stability of regularization methods having, for example, the form (1.18) and are measured in the space where the element of interest should be reconstructed, such as, for instance, in equation (1.17). The problem is that in the context of learning theory, this reconstruction space is given as L2 .X , X /, which depends on the unknown measure X . In this section we discuss how the general framework of the balancing principle presented in Section 1.1.5 can be adapted to overcome this difficulty.

4.3.1 Adaptive learning when the error measure is known 

We assume both the estimator fz and the best in the model fH to be elements of some normed space whose norm we denote with kk. Such a norm is assumed to be known (note that in contrast the risk is not). Again we assume that an error bound of the form 

kfz  fH k  S.n, , / C A., /

Section 4.3 Adaptive kernel methods using the balancing principle

is available and further assume that ˛./ , S.n, , / D !./ .n/

221

A., / D ˛./A./,

where ˛./ > 1 and !, , A are positive functions. This latter assumption is typically satisfied and is made only to simplify the exposition. For example, in the case of k  k D k  kH , under the assumptions of Proposition 4.1, one may take ˛./ D log 4 , p .n/ D n, !./ D C4 , A./ D C3 ./. Then the parameter choice  D o .n/ corresponding to the crossing point in Figure 4.1 gives, with probability 1  , the rate    o .n/   f  2˛./A.o .n//. fz H To define a parameter strategy we first consider a suitable discretization for the possible values of the regularization parameter, which is an ordered sequence ¹i ºi2N , such that the value o .n/ falls between two successive values, say l and lC1 : l  o .n/  lC1 . Then, as in Proposition 1.2, the balancing principle estimate for o .n/ is defined via   4˛./    , j D 0, 1, : : : , i º. C D max¹i : fzi  fz j   !.j / .n/ Such estimates no longer depend on A, and the reason we can still expect it to be sufficiently close to o .n/ is better illustrated by Figure 4.1 and by the following reasoning. Observe that if we take two values , ˇ, such that   ˇ  o .n/, then with probability at least 1                fz  fzˇ   fz  fH  C fzˇ  fH      1 1 C ˛./ A.ˇ/ C  ˛./ A./ C .n/!./ .n/!.ˇ/ ˛./ 4 . (4.31) .n/!./ The intuition is that when such a condition is violated we are close to the intersection point of the two curves, that is to o .n/. The above discussion is made precise in the following. 

Assumption 4.1. For  > 0 both fz and fH belong to some normed space and moreover, with probability at least 1       1    C A./ fz  fH   ˛./ !./ .n/ where 

!./ is a continuous, increasing function;



A./ is a continuous, increasing function with A.0/ D 0;



!./A./  c,

222

Chapter 4 Regularization algorithms in learning theory

and ˛./ > 1, .n/ > 0. Moreover, assume that the bound holds uniformly with respect to , meaning that the collection of training sets for which it holds with confidence 1   does not depend on . It is easy to check that the last item in the above assumption ensures that for some c > 0 it holds o .n/  1=.c .n//, so that if we choose the first value start in the sequence ¹i ºi2N so that start  1=.c .n//, the best possible parameter choice will fall within the parameter range we consider. The last condition in the assumption requires the bound to be uniform with respect to , and is needed since the parameter choice we consider is data dependent. As mentioned in Remark 4.4, Section 4.2, this assumption is satisfied in all the examples of algorithms which we consider in this section. The following statement shows that the choice C provides the same error estimate of o .n/ up to a constant factor. Proposition 4.5. If Assumption 4.1 holds and moreover, we consider a sequence of regularization parameter values such that start  1=.c .n// !.iC1 /  q!.i /,

q > 1,

then with probability at least 1   we have    C  fz  fH   6q˛./A.o .n//. The above proposition is just a reformulation of Proposition 1.2, Section 1.1.5 and it shows that the balancing principle can adaptively achieve the best possible learning rate. In its basic formulation the balancing principle requires an extensive comparison of solutions at different values i . In fact, the procedure can be simplified, at the cost of slightly spoiling the constant in the bound. In fact, we can take a geometric sequence i D start i

with

 > 1,

start  1=.c .n//,

(4.32)

and introduce the choice       N D max¹i : fz j  fz j 1  

4˛./ , j D 0, 1, : : : , i  1º, (4.33) .n/!.j 1 / requiring only comparison of solutions for adjacent parameter values. The next proposition studies the error estimate obtained with this choice. Proposition 4.6. If Assumption 4.1 holds and moreover, there are b > a > 1 such that for any  > 0, !.2/=b  !./  !.2/=a,

(4.34)

then, taking a sequence of regularization parameter values as in equation (4.32), we have with probability at least 1      N  fz  fH   C ˛./A.o .n//, where C might depend on a, b, .

Section 4.3 Adaptive kernel methods using the balancing principle

223

The proof of Proposition 4.6 will be provided below in Section 4.3.3.

4.3.2

Adaptive learning when the error measure is unknown 

A further goal is adaptation with respect to the error as measured by kfz  fH k . Then, as we mentioned before, the application of the balancing principle is not straightˇ forward since we should evaluate kfz  fz k , which depends on the unknown measure . Since both the empirical norm 1X D f .xi /2 , n n

kf

k2 x

iD1

and the RKHS norm are known, we can consider p   4CO ˛./ j j   i , j D 0, 1, : : : , i º,  x D max¹i : fz  fz  

x .n/!.j / and

     H D max¹i : fzi  fz j 

H



4˛./ , j D 0, 1, : : : , i º, .n/!.j /

where the parameter CO is specified in Proposition 4.8 below. Our main result shows that the choice O D min¹ x , H º

(4.35)

allows achievement of the best error rate for the expected risk in an adaptive way. To show this we need the following assumption. Assumption 4.2. Assume that, for   n1=2 , the following bounds hold with probability at least 1      p  1    C A./ fz  fH   ˛./  p

n!./ and

     fz  fH 

H

  1 C A./ ,  ˛./ p n!./

where p  !./ is a continuous, increasing function; p  A./ is a continuous, increasing function with A.0/ D 0, 

!./A./  c,

and ˛./ > max¹log.4=/1=2 , 1º. Moreover, assume the bound to hold uniformly with respect to , meaning that the collection of training sets for which it holds with confidence 1   does not depend on .

224

Chapter 4 Regularization algorithms in learning theory

The way we wrote the estimates is no coincidence since it corresponds to how the two error estimates are typically related (see, for example, Proposition 4.1 above). The fact that pthe reproducing kernel Hilbert space can be viewed as the image of L2 .X , X / under T (see equation (4.8)) lies essentially at the root of this relationship. Because of this fact, an estimator will have different norms (that is lie in spheres with different radius) in L2 .X , X / and HK . Given Assumption 4.2, the parameter choice  D o .n/ corresponding to the crossing point in Figure 4.1 is the same in both cases but the rates are different, in fact we have   p  o .n/   fH   2˛.n/ o .n/A.o .n// (4.36) fz

for the expected risk and    o .n/   fH  fz

H

 2˛.n/A.o .n//

(4.37)

for the RKHS norm. Note that the fact that the above-mentioned parameter choice is the same for both error measures is a promising indication and a possible idea would be to recall that in view of equation (4.7) it holds for the RKHS norm that jf .x/j   kf kH ,

8x 2 X , f 2 H ,

so that we can think of using the bound in the RKHS norm to bound the expected risk and use the balancing principle as presented above. Unfortunately, in this way we are not going to match the best error rate for the expected risk, as can be seen by comparing equations (4.36) and (4.37). The following proposition is our main result and examines the property of the choice (4.35). Proposition 4.7. Assume that Assumption 4.2 holds. p Consider a sequence of regularization parameter values such that start  1=.c n/ and !.iC1 /  q!.i /. If we choose O as in equation (4.35), then the following bound holds with probability at least 1      O   f  qC ˛./o .n/A.o .n//, fz H

where the value of C can be explicitly given. As an illustration of the above result we show how it allows us to adaptively achieve the same rate which was proved to be optimal for learning algorithms based on the Tikhonov regularization in RKHS and obtained in [152] under the assumption that A./ D cr1=2 , and r is known (see also Proposition 4.3). To the best of our knowledge the balancing principle is the first strategy that allows achievement of this result without requiring any data splitting.

225

Section 4.3 Adaptive kernel methods using the balancing principle

Example 4.1. From Proposition 4.1 it follows that for learning based on Tikhonov regularization we have with probability at least 1     1  C 2r , 1=2 < r  1, E.fz /  E.fH /  C log.4=/ n but also     1    p C r1=2 , 1=2 < r  3=2. fz  fH   C log.4=/ H  n Applying the above result we find that the parameter choice (4.35) satisfies with probability at least 1   O

E.fz /  E.fH /  6qC log.4=/n 2rC1 , 2r



1=2 < r  1.

This is exactly the rate obtained in [152] for a priori chosen  D n (see also Proposition 4.3). We end this subsection with the following remark which shows how to practically compute equation (4.35). Remark 4.8 (Computing balancing principle). The proposed parameter choices can P be computed exploiting the properties of RKHS. In fact, for f D niD1 ˛i K.xi , / we have

X  n n X 2 ˛i K.xi , /, ˛i K.xi , / kf kH D D

iD1 n X

iD1

H

˛i ˛j K.xi , xj / D ˛K˛,

i,j D1

where we used the reproducing property hK.x, /, K.s, /iH D K.x, s/. Then we can P P ˇ ˇ check that for fz D niD1 ˛i K.xi , /, fz D niD1 ˛i K.xi , / we have  2  ˇ  fz  fz  D ˛ ˇ K˛ ˇ  2˛ ˇ K˛  C ˛  K˛  H

D .˛ ˇ  ˛  /K.˛ ˇ  ˛  /. Similarly, one can see that  2  ˇ  ˇ  2 ˇ   f fz z  D .˛  ˛ /K .˛  ˛ /.

x

4.3.3

Proofs of Propositions 4.6 and 4.7

Recall that if Assumption 4.1 holds the best parameter choice achieving the order optimal learning rate (4.30) and it can be shown that the last condition in Assumption 4.1

226

Chapter 4 Regularization algorithms in learning theory

ensures o .n/  1=.c .n//. Note that if we now restrict our attention to some discrete sequence .i /i with start  1=.c .n//, then it is easy to see that the best estimate for o .n/ is ² ³ 1  D max i jA.i /  , !.i / .n/ which still depends on A. Given these observations, we can provide the proof of Proposition 4.6. Proof of Proposition 4.6. Note that all the inequalities in the proof are to be intended as holding with probability at least 1  . Recall that by equation (4.31) for , ˇ such that   ˇ  o .n/ we have      fz  fzˇ  

4˛./ . !./ .n/

N Indeed, by definition   o .n/, and we know that, It is easy to prove that   . for any l1    o .n/,    l1   fz   f z

4˛./ , !.l1 / .n/

N The key observation is that we can easily control the so that, in particular,   . N In fact, if we let  D ` distance between the solutions corresponding to  and . and N D m clearly m  ` and we can use the definition of N to write m     X   N   j  fz  fz   fz  fz j 1  j D`C1

 4˛./

m 1 1 X .n/ !.j 1 / j D`C1

 4˛./

m`1 X 1 1 . .n/ !. j / j D0

Now for any  > 1, ˛ > 1 let p, s 2 N be such that 2p    2pC1 and 2s  ˛  2sC1 . Using equation (4.34) we then get 1 1 1 1   s  log ˛ s 2 !.˛ / !.2  / a !. / a !. / !.i / D !.i1 /  b pC1 !.i1 /  b log2 2 !.i1 /.

Section 4.3 Adaptive kernel methods using the balancing principle

227

In view of the fact that by definition  D l  o .n/  lC1 , the last inequality shows that 1=!. /  b log2 2 =!.o .n//. Moreover, m`1 X j D0

1 alog2 2 1  . !. / alog2   1 !. j /

Finally we can use the above inequalities and the definition of  to get         N   N   fz  fH   fz  fH  C fz  fz  alog2 2 1 1 C 4˛./ log  2 .n/!. / a  1 .n/!. /   alog2 2 b log2 2  2˛./ 1 C 2 log  . a 2  1 .n/!.o .n//

 2˛./

Now we make some preparations to prove the main result of the section allowing adaptive regularization for kernel-based algorithms. The following concentration result will be crucial. Lemma 4.2. Assume that H is a RKHS with bounded kernel (4.7). For f 2 H we have with probability at least 1    j kf k  kf k x j  C

log.4=/ p n

1=2 kf kH ,

p and C2 D 2 2 2 . Proof. Let Kx D K.x, /, if f 2 H , we have f .x/ D hf , Kx iH using the reproducing property. Then we can write Z hf , Kx iH hf , Kx iH dX .x/ kf k2 D X 

Z D: hf , Tf iH . D f, hf , Kx iH Kx dX .x/ H

X

Reasoning in the same way we get n ˛ ˝ ˛ 1 X˝ f , Kxi H f , Kxi H n iD1

 n ˛ 1 X˝ f , Kxi H Kxi D f, D: hf , Tx f iH . n H

kf k2 x D

iD1

228

Chapter 4 Regularization algorithms in learning theory

From the above reasoning, it follows that 8f 2 H it holds p j kf k  kf k x j  kT  Tx k kf kH .

(4.38)

The quantity kT  Tx k has been estimated in Lemma 4.1 as follows:   log 4 C2 . p kT  Tx k  n The lemma is proved plugging the above estimate into (4.38). We add the following remark. Remark 4.9. Using the Hoeffding inequality, one can show that for jf .x/j < C , the following estimate holds true: P .j kf k  kf k x j > /  2e 

n 2 kf k2 2C 4

,

that is we have with probability at least 1   j kf k  kf k x

1  p 2 log.2=/ 2 j  2C kf k1

. n

(4.39)

Comparing the above result to Lemma 4.2, we see that one has the order n 2 versus 1 n 4 . It is hence tempting to use this estimate instead of that in Lemma 4.2 to avoid dealing with RKHS. The point is that we need to estimate j kf k  kf k x j for f D  fH  fz , and it is expected that the norm kf k is rather small, say 1

r 1  kf k D kfH  fz k  cn 2rC1 , r > , 2

as in Proposition 4.3. Note that for such f the bound (4.39) is too rough. Namely, 1  p 2 log.2=/ 2 1 1  2.2rC1/ 2C kf k1 / >> n 4 .

D O.n n The simple application of the Hoeffding inequality is then not enough to prove optimal learning rates and in the following we will use the bound given in Lemma 4.2. Lemma 4.2 and Assumption 4.2 immediately yield the following result. Proposition 4.8. If Assumption 4.2 holds, then with probability at least 1      p  1    p C A./ , fz  fH   ˛./CO 

x !./ n with CO D 1 C ˛./C .

Section 4.3 Adaptive kernel methods using the balancing principle

229

Proof. From Lemma 4.2 it holds that      ˛./C           f , f  fz  fH   fz  fH  C z H H

x

n1=4 so that the proof follows substituting p the inequalities of Assumption 4.2 in the above 1=4 inequality and noting that n   since   n1=2 . Given the above results we can finally prove Proposition 4.7. Proof of Proposition 4.7. We note a few useful facts. Let ‚./ D !./A./. First, from Assumption 4.2 - item 3, if we take  D o .n/ we have p 1 1 (4.40) ‚.o .n//  co .n/ ) p  co .n/ ) 1=4  c o .n/. n n p Second, noting that conditions of Proposition 4.2 imply !.iC1 /= iC1  q!.i /= p i , and recalling the reasoning from the proof of Proposition 4.6, we have p p q o .n/   . !. / !.o .n// This immediately yields q 1  , !. x / !.o .n//

(4.41)

since  x   , and p

p q o .n/ H  , !.H / !.o .n//

(4.42)

p since H   and =!./ is assumed to be a decreasing function. We now consider the two cases:  x < H and  x > H . Case 1. First, consider the case O D  x < H . From Lemma 4.2 we have      ˛./C   x  O       f f   fz  fH   fz x  fH  C z H

x H n1=4     ˛./C   x    H   f  fz x  fH  C f   z z

x H n1=4  ˛./C   H  C fz  fH  . H n1=4

(4.43)

We consider the various terms separately. Applying Propositions 4.5 and 4.8 we get   p  x  (4.44) fz  fH   6q˛./CO o .n/A.o .n//.

x

230

Chapter 4 Regularization algorithms in learning theory

Applying again Proposition 4.5, and with the aid of equation (4.40), we obtain  p p ˛./C   H   f  6q˛./2 cC o .n/A.o .n//. (4.45) f  z H H 1=4 n Recalling the definition of H , we also have   4˛./  x  . (4.46) fz  fzH   p H n!. x / We can now use equations (4.40), (4.41) and the definition of o .n/ to get  p p ˛./C   x H   f (4.47) f    4q˛./2 cC o .n/A.o .n//. z z 1=4 H n If we now plug equations (4.44), (4.45), and (4.47) into equation (4.43) we get   p  O  fz  fH   q˛./C o .n/A.o .n//, p with C D 6CO C 10˛./ cC .



Case 2. Consider the case O D H <  x . From Lemma 4.2 we have      ˛./C    H  O     f f  fz  fH   fzH  fH  C z H H 1=4

x  n          fzH  fz x  C fz x  fH 

x

x   ˛./C  H  C fz  fH  . H n1=4 Applying Proposition 4.5 and using equation (4.40), we immediately get  p p ˛./C   H   f  6q c˛./2 C o .n/A.o .n//. f  z H H 1=4 n Another straightforward application of Propositions 4.5 and 4.8 gives   p  x  fz  fH   6q˛./CO o .n/A.o .n//.

x

Finally we have, from the definition of  x p   4˛./CO H x   H , fz  fz   p

x n!.H /

(4.48)

(4.49)

(4.50)

(4.51)

so that using equations (4.40), (4.42), and the definition of o .n/, we can write   p    H (4.52) fz  fz x   4˛./q CO o .n/A.o .n//.

x

The proof is finished by plugging equation (4.49), (4.50), and (4.52) into equation (4.48) to get   p  O  fz  fH   ˛./qC o .n/A.o .n//,

p where C D 6˛./ cC C 10CO .

Section 4.3 Adaptive kernel methods using the balancing principle

4.3.4

231

Numerical experiments. Quasibalancing principle

In this subsection we consider some numerical experiments discussing how the balancing principle can be approximatively implemented in the presence of a very small sample. When the number of samples is very small, as is often the case in practice, we observed that one cannot completely rely on the theoretical constructions since the bounds are conservative and tend to select a large parameter which will oversmooth the estimator. For our numerical experiments, besides the standard kernel-based learning, we also consider the more complex situation when the kernel is not fixed in advance, but is found within the regularization procedure. We first give a brief summary of this latter approach. Indeed, once a regularized kernel-based learning method is applied, two questions should be answered. One of them is how to choose a regularization parameter. The balancing principle discussed in previous subsections provides an answer to this question. Another question is how to choose the kernel, since, as we will see below, in several practically important applications, a kernel is not given a priori. This question is much less studied. It was discussed recently in [115], where selecting a kernel has been suggested K D K./ from some set K, such that K./ D arg min¹Qz .K, /, K 2 Kº, where Qz .K, / D min

f 2HK

(4.53)

! n 2 1 X 2 yi  f .xi / C kf kHK , n iD1

and HK is the RKHS generated by K. By definition, the selected kernel K D K./ is -dependent, so that this kernel choice rule is only applicable for a priori given regularization parameter .  At the same time, as shown in Section 4.2, the best candidate in the model fH 2 HK can be approximated by minimizers fz 2 HK of Qz .K, / in such a way that Assumption 4.2 is satisfied. In accordance with Proposition 4.7 the parameter choice O rule  D O D .K/ then allows an accuracy which is only worse by a constant factor than the optimal one for the fixed K 2 K. Let ƒ: RC ! RC be the function, such that its value at point  is the number O O D .K.// for estimators based on the kernel K./ 2 K given by equation (4.53). V O V then K./ V can be seen as the kernel of If  is a fixed point of ƒ, i.e., V D .K. //, optimal choice in the sense of [115], since it satisfies the criterion Qz .K, / ! min V which is order-optimal for this kernel. for the regularization parameter  D , The existence of this fixed point  D V depends on the set K, and deserves consideration in the future. This issue is partially discussed in the next section. In the computational experiment below we find such a fixed point numerically for an academic example from [115]. At this point it is worth noting that the balancing principle

232

Chapter 4 Regularization algorithms in learning theory

can be capacity-independent in the sense that it does not require knowledge of the spectral properties of the underlying kernel K. This feature of the balancing principle makes its combination with the rule (4.53) numerically feasible. To simplify a numerical realization of the balancing principle, and especially in the presence of very small samples, one can approximate the values  x , H using the well-known quasioptimality criterion [162]. As was observed in [128], this criterion N theoretically can be seen as a heuristic counterpart of the parameter choice rule  D , j  justified by Proposition 4.6. It also operates with norms  .j / D kfz fz j 1 k, j D start  j , and selects q0 D l , such that for any j D 1, 2, : : : , N .  .j /   .l/, i.e., l D arg min¹ .j /, j D 1, 2, : : : , N º. In our experiments we approximate  x and H by j

q0

x D l ,

l D arg min¹ x .j / D kfz

q0

m D arg min¹H .j / D kfz

j 1

 fz

k x ,

j D 1, 2, : : : , N º,

and H

D m ,

j

j 1

 fz

kH ,

j D 1, 2, : : : , N º,

respectively. In accordance with equation (4.35) we then choose a regularization parameter q0 O D min¹q0

x , H º.

(4.54)

In view of the similarity of the rule (4.54) to the quasioptimality criterion [162], and to the balancing principles (4.33) and (4.35), we find it appropriate to refer to equation (4.54) as the quasibalancing principle. Below we provide first illustrations of the applicability of the rule. As in [115], we consider a target function 4  3 1 2 2 2 f .x/ D .x C 2.e 8. 3 x/  e 8. 2 x/  e 8. 2 x/ //. x 2 Œ0, 2, (4.55) 10 and a training set z D zn D ¹.xi , yi /ºniD1 , where xi D 2 .i1/ n1 , yi D f .xi / C i , and i are random variables uniformly sampled in the interval [-0.02, 0.02]. In our first experiment we test the approximate version (4.54) of the balancing principle using the priori information that the target function (4.55) belongs to an 2 RKHS H D HK generated by the kernel K.x, t / D K .x, t / D xt C e 8.tx/ , t , x 2 Œ0, 2. Figures 4.2 and 4.3 display the values  x .j /, H .j / calculated for regularized least 

squares estimators fz j , which are constructed using the kernel K for training sets z D z21 and z D z51 respectively. Here and in the next experiment j 2 ¹start  j , j D 1, 2, : : : , 20º,

start D 106 ,  D 1.5.

233

Section 4.3 Adaptive kernel methods using the balancing principle 1.4 1.2 1 1.8 0.6 0.4 0.2 0 0

2

4

6

8

10

12

14

16

18

20

Figure 4.2. The values of  x .j / (dots) and H .j / (crosses) for z D z21 (from [44]). 3 2.5 2 1.5 1 0.5 0 0

2

4

6

8

10

12

14

16

18

20

Figure 4.3. The values of  x .j / (dots) and H .j / (crosses) for z D z51 (from [44]).

It is instructive to see that the sequences  x .j /, H .j /, j D 1, 2, : : : , 20, exhibit different behavior for training sets z21 and z51 . At the same time, they attain the minimal values at the same j . Therefore, in accordance with the rule (4.54) we take q0 q0 O D  x D H D 1.5  106 in case of z D z21 , while for z D z51 O D q0

 x

q0

D H

D 0.0033.

O

Figures 4.4 and 4.5 show that for the chosen values of parameters, the estimator fz provides an accurate reconstruction of the target function. In our second experiment we do not use a priori knowledge of the space HK , K D K , containing the target function (4.55). Instead, we choose a kernel K adaptively from the set. K D ¹K.x, t / D .xt /ˇ C e .xt/ , ˇ 2 ¹0.5, 1, : : : , 4º, 2 ¹1, 2, : : : , 10ºº, 2

234

Chapter 4 Regularization algorithms in learning theory 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 − 0.1 0

1

2

3

4

5

6

7

O Figure 4.4. The estimator fz (thin line) and the target function f (solid line) for O D 6 1.5  10 and training set z D z21 (dots) (from [44]).

0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 − 0.1 0

1

2

3

4

5

6

7

O Figure 4.5. The estimator fz (thin line) and the target function f (solid line) for O D 0.0033 and training set z D z51 (dots) (from [44]).

O O trying to find a fixed point of the function ƒ:  ! .K.//, where .K.// is the number (4.54) calculated for the kernel K./, which minimizes Qz .K, /, z D z21 , over the set K. In the experiment we take .s/ 2 ¹j ºj20D1 and find the minimizer K..s/ / 2 K by the simple full-search over the finite set K. The next value .sC1/ 2 ¹j ºj20D1 is 

then defined as the number (4.54) calculated for estimators fz j based on the kernel K..s/ /. This iteration procedure terminates when j.sC1/  .s/ j  104 . It gives us a required approximate fixed point V D 18  0.0014 and the corresponding V D K.; V x, t / D xt C e 10.xt/2 , which is a good approximation for the kernel K./

Section 4.4 Kernel adaptive regularization with application to blood glucose reading

235

0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 − 0.1 0

1

2

3

4

5

6

7

V

Figure 4.6. The target function f (solid line) and its estimator fz (this line) based on the V x, t / D xt C e 10.xt/2 , V D 0.0014, and training set z D z21 adaptively chosen kernel K.; (dots) (from [44]). V V provides a good ideal kernel K .x, t /. The estimator fz based on the kernel K./ reconstruction of the target function (4.55), as can be seen in Figure 4.6. The numerical experiments presented demonstrate the reliability of the quasibalancing principle (4.54), as well as its underlying concept justified by Proposition 4.7, and show that it can be used also in learning the kernel function via regularization. In the next section we continue the discussion of the latter issue.

4.4

Kernel adaptive regularization with application to blood glucose reading

As may be observed from our discussion above, from the point of view of learning theory, regularization is an approach to approximate reconstruction of the unknown functional dependency, say fun , which is based on a compromise between the attempt to fit given data and the desire to reduce the complexity of a data fitter. For example, a well-known Tikhonov regularization in its rather general form defines the approximant for fun as the minimizer f D arg min T .f / of the sum T .f / :D DataFitting.f / C   Complexity.f /,

(4.56)

where the above-mentioned compromise is governed by the value of the regularization parameter . Note that in equation (4.56), the form of the first summand is usually dictated by the given data, while a measure of complexity can, in principle, be chosen at will. Beginning with the papers [86, 159], classical regularization theory restricts itself to

236

Chapter 4 Regularization algorithms in learning theory

studying the situation, when in equation (4.56) the complexity term is assumed to be given a priori. For example, in the case of data given by noisy function values yi D fun .xi / C i , xi 2 U Rd , i D 1, 2, : : : , n,

(4.57)

the classical regularization theory suggests taking equation (4.56) in the form X 1 .f .xi /  yi /2 C kf k2HK , (4.58) T .f / D T .f , K, W / :D jW j .xi ,yi /2W

where the complexity is measured by the norm kkHK in a Reproducing Kernel Hilbert Space (RKHS) HK , such as a Sobolev space H2r , for example, and for generality we use the notations W D ¹.xi , yi /, i D 1, 2, : : : , nº, jW j D n. Note that the choice C omplexi ty.f / D kf k2HK allows a framework, which includes most of the classical regularization schemes discussed in Section 4.2. A recent trend in regularization theory is the growing interest in Tikhonov regularization in Banach spaces which can still be seen as the minimization of equation (4.56), where the complexity term is represented by some proper stabilizing functional R.f / such as, for instance, total variation (see, e.g., the paper by Hintermüller and Kunisch [73]). This type of Tikhonov regularization is not covered by an RKHSsetting. Important contributions to the study of this area have been made recently by Burger, Osher [31], Louis, Schöpfer et al. [148], Resmerita, Scherzer [142], Hofmann, Kaltenbacher et al. [75], Bonesky, Kazimierski et al. [25], Jin, Lorenz [82], and others. However, it is important to note that in all above-mentioned studies, the form of the complexity term was assumed to be given a priori. At the same time, as has been mentioned, for example, by Micchelli, Pontil [115], even for the classical RKHS-setting, when Complexity.f / D kf k2HK , a challenging and central problem is the choice of the kernel K itself. Indeed, since in view of Wahba’s classic representer theorem [86], the minimizer f of equation (4.58) can be written in the form f D f .; K, W / D

jW j X

ci K., xi /, ci 2 R1 , i D 1, 2, : : : , jW j,

(4.59)

iD1

c of coefficients c , i D 1, 2, : : : , jW j, is defined as follows: where a real vector ! i  ! c D .jW jI C K/1!  y,  where I is the unit matrix of the size jW j  jW j, K is Gram matrix with entries  y D .y1 , y2 , : : : , yjW j /. The choice of the kerK.xi , xj /, i , j D 1, 2, : : : , jW j, and ! nel K is then tied to the problem of choosing the basis for the approximation of the unknown function fun . Clearly, this choice is problem-dependent and can make a significant difference in practice.

Section 4.4 Kernel adaptive regularization with application to blood glucose reading

237

Thus, despite the great success of the regularization theory, the choice of the suitable regularization space still remains an issue. The present section aims to shed light on this important but as yet under-researched problem. At first glance it may seem that the kernel choice issue can be resolved with the concept of universal kernels advocated by Micchelli, Xu and Zhang [116]. Recall that a universal kernel K has the following property: given any prescribed compact subset UQ U , any positive number  and any function g continuous on UQ , there is a function f of the form (4.59) with ¹xi º UQ , such that in the maximum norm we have kg  f k  . Therefore, one may think that the kernel choice issue will be resolved by incorporating a universal kernel K into Tikhonov regularization schemes (4.58) and (4.59), because then the minimizer f .; K, W / may potentially have a good approximating property. This is partially true when the regularized approximation (4.59) is used for a reconstruction / prediction of the unknown functional dependency fun within the scope UQ of input points ¹xi º which is sometimes called prediction by interpolation. At the same time, the concept of the kernel’s universality does not guarantee that the space of linear combinations of kernel sections (4.59) will approximate well at points outside the scope of inputs ¹xi º. This can be seen with the example of two well-known 2 universal kernels [116] K.u, x/ :D e !kuxk , K.u, x/ :D .ˇ Ckuxk2 /! , u, x 2 d R , !, ˇ > 0, for which the kernel sections K.u, xi / are vanishing with increasing distance ku  xi k of the point of interest u to the set of inputs ¹xi º. Therefore, in order to predict the unknown functional dependency outside the scope of inputs ¹xi º, also known as prediction by extrapolation, the question about the proper choice of a regularization kernel is, in general, open until now. At this point it is worth noting that prediction by extrapolation is important in many applications such as, for example, diabetes technology, as will be discussed below. For such applications, one therefore needs to choose a kernel from the set K.U / of all continuous, symmetric positive-definite functions defined on U , dependent on the input data, say (4.57). Lanckriet et al. were among the first to emphasize the need to consider the multiple kernels or parameterizations of kernels, and not a single a priori fixed kernel, since practical problems often involve multiple, heterogeneous data sources. In their work [97] the authors consider the set K.¹Ki º/ D ¹K D

m X

ˇi Ki º

iD1

of linear combinations of some prescribed kernels ¹Ki ºm iD1 and propose different criteria to select the kernel from it. It is worthy of notice that for some practical applications such a set of admissible kernels is not rich enough. Therefore, more general parameterizations are also of interest.

238

Chapter 4 Regularization algorithms in learning theory

Let be a compact metric space and G : ! K.U / be an injection, such that for any x1 , x2 2 U , the function w ! G.w/.x1 , x2 / is a continuous map from to R, where G.w/.x1 , x2 / is the value of the kernel G.w/ 2 K.U / at .x1 , x2 / 2 U  U . Each such mapping of G determines a set of kernels K. , G/ D ¹K : K D G.w/, K 2 K.U /, w 2 º parameterized by elements of . In contrast to K.¹Ki º/, K. , G/ may be a nonlinear manifold. Example 4.2. Consider D Œa, b3 , 0 < a < b, .˛, ˇ, / 2 Œa, b3 , and define a mapping G : .˛, ˇ, / ! .x1 x2 /˛ Cˇ expŒ .x1 x2 /2 , where x1 , x2 2 U .0, 1/. It is easy to see that G.˛, ˇ, / is positive-definite as the sum of two positive-definite functions of .x1 , x2 /. Moreover, for any fixed x1 , x2 2 U the value G.˛, ˇ, /.x1 , x2 / depends continuously on .˛, ˇ, /. Thus, kernels from the set K. , G/ D ¹K : K.x1 , x2 / D .x1 x2 /˛ C ˇ expŒ .x1  x2 /2 ,

.˛, ˇ, / 2 Œa, b3 º

are parameterized by points of D Œa, b3 in the sense described above. Note that for the extrapolation problem there is good reason to concentrate attention on this set of kernels, since for any the summands K .x1 , x2 / D expŒ .x1  x2 /2  are the universal kernels in the sense mentioned above, and all continuous functions defined on a compact set U .0, 1/ are uniform limits of functions of the form (4.59), where K D K and ¹xi º U . At the same time, for a fixed set ¹xi º of data points and x … co¹xi º, the approximation performance of functions (4.59) with K D K may be poor due to the fact that the values of all kernel selections K .x, xi / D expŒ .x  xi /2  are decreasing with an increase of distance from the point x to the set co¹xi º. The summands K˛ .x, xi / D .xxi /˛ then serve to compensate such decrease. Once the set of kernels is fixed, one may follow the approach [115] described in Section 4.3.4 and select a kernel in accordance with (4.53) by minimizing the Tikhonov regularization functional (4.58), such that Kopt D arg min¹T .f .; K, W /, K, W /, K 2 K. , G/º. To illustrate that such an approach may fail in a prediction of the extrapolation type we use the same target function (4.55) as in Section 4.3.4 and the given set W D

i ¹.xi , yi /º14 iD1 consists of points xi D 10 , i D 1, 2, : : : , 14 and yi D f .xi / C i , where i are random values sampled uniformly in the interval Œ0.02, 0.02. Recall that the target function (4.55) has been chosen in such a way that it belongs to the RKHS generated by the “ideal” kernel Kopt .x1 , x2 / D x1 x2 C expŒ8.x1  x2 /2 . The performance of the approximant f .x; Kopt , W / with the best  is shown in the

14 , 10  the value of the target Figure 4.7. This figure illustrates that for x 2 co¹xi º D Œ 10

Section 4.4 Kernel adaptive regularization with application to blood glucose reading

239

0.6 0.5 0.4 0.3 0.2 0.1 0 0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

5.5

Figure 4.7. The performance of the approximant f .x; K, W / (thin dotted line) based on the kernel K.x1 , x2 / D x1 x2 C expŒ8.x1  x2 /2  generating the target function (4.55) (from [122]).

function is estimated well by f .x; Kopt , W /, while for x … co¹xi º the performance of the approximant based on the Kopt is rather poor. Observe that the figure displays the performance of the approximation (4.59) with the best . This means that the choice of the regularization parameter  cannot improve the performance of the approximation given by the ’ideal’ kernel, that is, the kernel Kopt .x1 , x2 / D x1 x2 CexpŒ8.x1 x2 /2  used to generate the target function. Figure 4.8 displays the performance of the approximation (4.59) constructed for the same data set W , but with the use of the kernel K.x1 , x2 / D .x1 x2 /1.9 CexpŒ2.7.x1  x2 /2 . As one can see, the approximation based on this kernel performs much better compared to Figure 4.7. Note that the regularization parameter  for this approximation has been chosen by means of the quasibalancing criterion (4.54). The kernel improving the approximation performance has been chosen from the set K D ¹K.x1 , x2 / D .x1 x2 /˛ C ˇ expŒ .x1  x2 /2 , ˛, ˇ, 2 Œ104 , 3º

(4.60)

as follows. jW j First, let us split the data W D ¹.xi , yi /ºiD1 , such that W D WT [ WP and co¹xi : .xi , yi / 2 WT º \ ¹xi : .xi , yi / 2 WP º D ¿. The approximant displayed in Figure 4.8 corresponds to the splitting WT D ¹.xi , yi /º7iD1 , WP D ¹.xi , yi /º14 iD8 . For fixed WT and the corresponding Tikhonov-type regularization functional X 1 .yi  f .xi //2 C kf k2HK , (4.61) T .f , K, WT / D jWT j i:.xi ,yi /2WT

240

Chapter 4 Regularization algorithms in learning theory

0.5 0.4 0.3 0.2 0.1 0 0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Figure 4.8. The performance of the approximant f .x; K, W / (thin dot line) based on the kernel K.x1 , x2 / D .x1 x2 /1.9 C expŒ2.7.x1  x2 /2 , that has been chosen as a minimizer of equation (4.62) (from [122]).

we consider a rule  D .K/ which selects for any K 2 K.U / a regularization parameter from some fixed interval Œmin , max , min > 0. It is worth noting again that we are interested in constructing regularized approximations of the form (4.59), which will reconstruct the values of the function at points inside / outside the scope of x. Therefore, the performance of each regularization estimator f .x; K, WT / is checked on the rest of a given data WP and measured, for example, by the value of the functional X 1 .f .xi /, yi /, (4.62) P .f , WP / D jWP j i:.xi ,yi /2WP

where ., / is a continuous function of two variables. To construct the approximant displayed in Figure 4.8 we take .f .xi /, yi / D .yi  f .xi //2 . However, the function ., / can be adjusted to the intended use of the approximant f .x; K, WT /. In the next subsection we present an example of such an adjustment. Finally, the kernel of our choice is K D K.K, , , W ; x1 , x2 / which minimizes the functional Q .K, , W / D T .f .; K, WT /, K, WT /C.1/P .f .; K, WT /, WP / (4.63) over the set of admissible kernels K. , G/. Note that the parameter  here can be seen as a performance regulator on the sets WT and WP . Taking  closer to zero we put more emphasis on the ability to extrapolate, while for  > 12 we are more interested in interpolation. The kernel choice rule based on the minimization of the functional (4.63) is rather general. We call this rule the kernel adaptive regularization algorithm (KAR-

Section 4.4 Kernel adaptive regularization with application to blood glucose reading

241

algorithm). The next proposition justifies the existence of the kernel and the regularization parameter which minimize the functional (4.63). Proposition 4.9. There are K 0 2 K. , G/ and 0 2 Œmin , max  such that for any parameter choice rule  D .K/ Q .K 0 , 0 , W / D inf¹Q .K, .K/, W /, K 2 K. , G/º. Proof. Let ¹Kl º 2 K. , G/ be a minimizing sequence of kernels such that lim Q .Kl , .Kl /, W / D inf¹Q .K, .K/, W /, K 2 K. , G/º.

l!1

Since, by construction, all .Kl / 2 Œmin , max , one can find a subsequence n D .Kln /, n D 1, 2, : : : such that n ! 0 2 Œmin , max . Consider the subsequence of kernels K n D Kln , n D 1, 2, : : : . Let also fn .x; K n , WT / D

X

cin K n .x, xi /,

n D 1, 2, : : :

i:.xi ,yi /2WT

be the minimizers of the functional (4.61) for K D K n . From equation (4.59) we know that the vector cn D .cin / 2 RjWT j admits a representation cn D .n jWT jI C KnT /1 yT , where I is the unit matrix of the size jWT j  jWT j and the matrix KnT and the vector yT are respectively formed by the values K n .xi , xj / and yi with i , j , such that .xi , yi /, .xj , yj / 2 WT . Using the definition of K. , G/, the sequence ¹K n º 2 K. , G/ is associated with a sequence ¹wn º 2 , such that K n D G.wn /. Since is assumed to be a compact metric space, there is a subsequence ¹wnk º

¹wn º which converges in to some w0 2 . Consider the kernel K 0 D G.w0 / 2 K. , G/. Keeping in mind that for any fixed x1 , x2 2 U , the function w ! G.w/.x1 , x2 / is continuous on , one can conclude that the entries K nk .xi , xj / D G.wnk /.xi , xj / of the matrices KnTk converge to the corresponding entries K 0 .xi , xj / D G.w0 /.xi , xj / of the matrix K0T . Therefore, for any  > 0 a natural number k D k./ exists, depending only on , such that for any .xi , yi / 2 WT and k > k./, we have jK 0 .xi , xj /  K nk .xi , xj /j < . This means that the matrices KnTk converge to K0T in a standard matrix norm k  k. Consider the vector c0 D .0 jWT jI C K0T /1 yT , of coefficients .ci0 / from the representation f0 .x; K 0 , WT / D

X

i:.xi ,yi /2WT

ci0 K 0 .x, xi /

242

Chapter 4 Regularization algorithms in learning theory

of the minimizer of the functional (4.61) for K D K 0 . Since for K nk , K 0 2 K. , G/ corresponding matrices KnTk , K0T are positive-definite, for any vector y 2 RjWT j we have k.nk jWT jI C KnTk /1 yk  .nk jWT j/1 kyk, k.0 jWT jI C K0T /1 yk  .0 jWT j/1 kyk. Therefore, kc0  cnk k D k.nk jWT jI C KnTk /1 ..nk jWT jI C KnTk /

 .0 jWT jI C K0T //.0 jWT jI C K0T /1 yT k

D k.nk jWT jI C KnTk /1 .KnTk  K0T /.0 jWT jI C K0T /1 yT

C .nk jWT jI C KnTk /1 .nk  0 /jWT j.0 jWT jI C K0T /1 yT k

 .min jWT j/2 kyT kkKnTk  K0T k

C .min jWT j/2 kyT kjnk  0 jjWT j,

and in view of our observation that nk ! 0 and KnTk ! K0T , we can conclude that cnk ! c0 in RjWT j . Now we note that for any K 2 K. , G/ and X cj K.x, xj /, f .x; K, WT / D i:.xi ,yi /2WT

the functional (4.63) can be seen as a continuous function Q .K, , W / D Q .¹K.xi , xj /º, , W , ¹cj º/ of , cj and K.xi , xj /, i D 1, 2, : : : , jW j, j : .xj , yj / 2 WT . Therefore, summarizing our reasons we have Q .K 0 , 0 , W / D lim Q .K nk , nk , W / k!1

D inf¹Q .K, .K/, W /, K 2 K. , G/º.

4.4.1 Reading the blood glucose level from subcutaneous electric current measurements In this subsection, we discuss the possibility of using the approach described in the previous section in diabetes therapy management, in particular, for reading the blood glucose level from subcutaneous electric current measurements. Continuous glucose monitoring (CGM) systems provide indirect estimates of current blood glucose almost in real-time, which is highly valuable for insulin therapy in diabetes. For example, needle-based electrochemical sensors, such as the Abbott Freestyle Navigator [3], measure electrical signals in the interstitial fluid (ISF) and

Section 4.4 Kernel adaptive regularization with application to blood glucose reading

243

return ISF glucose concentration (mg/dL) by exploiting some internal calibration procedure. This ISF glucose reading is taken as an estimate of current blood glucose concentration. At the same time, a recalibration of Abbott CGM-sensors should sometimes be made several times per day. On the other hand, it is known (see [56] and references therein, [91]) that the equilibration between blood and ISF glucose is not instantaneous. As a result, CGM devices sometimes provide a distorted estimate of blood glucose level, and as has been pointed out in [56], further improvements of blood glucose reconstruction require more sophisticated procedures than the standard calibration, by which ISF glucose is determined in CGM systems, such as the Abbott Freestyle Navigator. In this section we consider how the approach based on the minimization of the functional (4.63) can be adapted for reading blood glucose level from subcutaneous electric current measurements. To illustrate this approach we use data sets of nine type 1 diabetic subjects studied within the framework of the EU-project ‘DIAdvisor’ [2] in the Montpellier University Hospital Center (CHU) and the Padova University Hospital (UNIPD). The chosen number of data sets is consistent with earlier research [56,140], where correspondingly 9 and 6 subjects were studied. Blood glucose concentration and subcutaneous electric current were measured parallel in each subject for 3 days under hospital conditions. The blood glucose concentration was measured 30 times per day by the HemoCue glucose meter [4]. Blood samples were collected every hour during the day, every 2 hours during the night and every 15 minutes after meals for 2 hours. A specific sampling schedule was adopted after breakfast: 30 minutes before, mealtime, 10, 20, 30, 60, 90, 120, 150, 180, 240, 300 minutes after. The subcutaneous electric current was measured by the Abbott Freestyle Navigator every minute. A data set W D ¹.xi , yi /º30 iD1 was formed for each subject by data collected during the first day. Here xi 2 Œ1, 1024 are the values of subcutaneous electric current (ADC counts), and yi 2 Œ0, 450 are the corresponding values of blood glucose (BG) concentrations (mg/dL). For each subject the corresponding data set W was then used to choose a kernel from the set (4.60) in the method described above. For this purpose, the data set W has been split into two parts, namely WP D ¹.xi , yi /º, jWP j D 4, formed by two minimum and two maximum values of xi ; WT D W nWP . The idea behind such a splitting is that we try to incorporate more data in the construction of the estimator f .; K, WT /, which is why jWT j > jWP j. At the same time, however, we attempt to test the ability of the estimator to extrapolate to extreme cases from observed data. The possibility of using Proposition 4.9 in the context of the blood glucose reading has already been discussed in [122]. In that paper, the form of the functional (4.62) was inspired by the notion of the risk function which was introduced similar to [94]. In experiments with clinical data it turned out that a kernel which was found as the minimizer of (4.63) with such form for data W of one patient, does not allow a clini-

244

Chapter 4 Regularization algorithms in learning theory

cally acceptable blood glucose reading for another’s. This means that for each patient the kernel choice procedure based on the minimization of the functional (4.63) should be repeated. This may be seen as a disadvantage. Here we propose another form of the functional (4.62), which not only leads to more accurate blood glucose readings, but also, as a by-product, allows a patient independent kernel choice. Define X 1 jyi  f .xi /jA, , (4.64) P .f , WP / D jWP j .xi ,yi /2WP

where the quantity jyi  f .xi /jA, is aimed at penalizing the overestimation of low glucose levels, as well as the underestimation of high ones. It is set out as follows: Case 1. If yi < 70 (mg/dL) and for sufficiently small , say  D 5 (mg/dL), we have yi C  < f .xi /, then we put jyi  f .xi /jA, D A, where A is large enough. Case 2. If f .xi /  yi < 70 (mg/dL) then jyi  f .xi /jA, D yi  f .xi /. Case 3. In the range of 70 C   yi  180   (mg/dL), we put jyi  f .xi /jA, D jyi  f .xi /j. Case 4. If yi > 180 (mg/dL) and yi   > f .xi /, we put jyi  f .xi /jA, D A. Case 5. If 180 < yi  f .xi / (mg/dL) then jyi  f .xi /jA, D f .xi /  yi . Otherwise, we define the penalizing quantity jyi  f .xi /jA, by linear interpolation to make it continuously dependant on .yi  f .xi //, since such continuity is necessary for the application of Proposition 4.9. Let us summarize the proposed approach. To select a kernel K 0 from a given class K D K. , G/, one needs a training data set W D ¹.xi , yi /ºniD1 , consisting of values of the subcutaneous electric current xi and values of the blood glucose concentration yi measured at the same moments in time. This training set W is split into two parts WT and WP . The first is used to construct regularized blood glucose readers f .; K, WT / with K 2 K. , G/ and  D .K/. The performance of these readers is measured on the second data set WP by the values of the functional (4.64). In accordance with Proposition 4.9, a kernel K 0 2 K. , G/ and  D 0 exist which optimizes the weighted performance in the sense of equations (4.63) and (4.64). K 0 is then the kernel of our choice, and for any current value x the regularized blood glucose reading is given as f.K 0 / .x; K 0 , W /. In our experiments the role of K. , G/ is played by the set (4.60), and for each K 2 K. , G/, the value  D .K/ is chosen in accordance with the quasibalancing principle (4.54). To illustrate this approach, the set W corresponding to the subject CHU102 was used for choosing the kernel K 0 from the set (4.60). Then the kernel K 0 .x, u/ D .xu/0.89 C 0.5e 0.0003.xu/

2

(4.65)

was found as an approximate minimizer of the functional (4.63), (4.64) with  D 0.5, and  D .K/ given by the quasibalancing principle (4.54) with j D 104 .1.01/j .

Section 4.4 Kernel adaptive regularization with application to blood glucose reading

245

The minimization has been performed by full search over the grid of parameters ˛i D 104 i , ˇj D 104 j , l D 104 l, i , j , l D 1, 2, : : : , 3  104 . Of course, the application of the full search method in finding the minimum of (4.63), (4.64) is computationally intensive, but in the present context it can be performed off-line. This kernel K 0 was used for all considered nine subjects to construct regularized estimators which, starting from a raw electric signal x 2 Œ1, 1024, return a blood glucose concentration y D f.K 0 / .x; K 0 , W /, where W D ¹.xi , yi /º30 iD1 are the subject’s data collected during the first day. To quantify the clinical accuracy of the constructed regularized blood glucose estimators we used the original Clarke EGA, which is accepted as one of the “gold standards” for determining the accuracy of blood glucose meters [40]. In accordance with EGA methodology, for each of the nine subjects the available blood glucose values obtained by the HemoCue meter have been compared with the estimates of the blood glucose y D f.K 0 / .x; K 0 , W /. Here x is a subcutaneous electric current value at the moment when the corresponding HemoCue measurement was executed. Since HemoCue measurements made during the first day were used to construct regularized estimators, only the data from another 2 days (at least 60 HemoCue measurements) have been used as references in the Clarke’s analysis. In this analysis each pair (reference value, estimated value) identifies a point in the Cartesian plane, where the positive quadrant is subdivided into five zones, A to E, of varying degrees of accuracy and inaccuracy of glucose estimations (see Figure 4.9, for example). Points in zones A and B represent accurate or acceptable glucose estimations. Points in zone C may prompt unnecessary corrections which could lead to a poor outcome. Points in zones D and E represent a dangerous failure to detect and treat. In short, the more points that appear in zones A and B, the more accurate the estimator is in terms of clinical utility. Abbott navigator

Estimated glucose (mg/dl)

400 350 300

E

A

C

250 200 B

150 100

D

B

D

50 0 0

E C 50 100 150 200 250 300 350 400 HemoCue glucose (mg/dl)

Figure 4.9. Clarke EGA for Abbott Freestyle Navigator (from [122]).

246

Chapter 4 Regularization algorithms in learning theory Regularized estimator

Estimated glucose (mg/dl)

400 350 300

A

C

E

250 200 B

150 100

D

D

B

50 0 0

E C 50 100 150 200 250 300 350 400 HemoCue glucose (mg/dl)

Figure 4.10. Clarke EGA for the regularized estimator (from [122]).

Glucose concentration (mg/dL)

A representative Clarke error grid (subject UNIPD203) for the proposed regularized blood glucose estimator is shown in Figure 4.10. For comparison, in Figure 4.9 the results of the EGA for blood glucose estimations determined from the internal readings of the Abbott Freestyle Navigator, calibrated according to the manufacturer’s instructions, are presented for the same subject and reference values. Comparison shows that the regularized estimator is more accurate, especially in case of low blood glucose. This can be also seen from Figure 4.11.

350 300 250 Hyper threshold

200 150 100 50

Hypo threshold 1500

2000

2500 3000 Time (min)

3500

4000

Figure 4.11. A graph plot of the blood glucose reading from subcutaneous electric current for Subject ID – UNIPD203: The solid line is formed by blood glucose estimations produced by the regularized blood glucose reader. The estimations given with 10 minutes frequency by the Abbott Freestyle Navigator are plotted on the thin line. The star points correspond to blood glucose measurements made by HemoCue meter (references).

Section 4.4 Kernel adaptive regularization with application to blood glucose reading

247

Table 4.1. Percentage of points in EGA-zones for estimators based on the kernel (4.65) (from [122]). Subject

A

B

C

D

E

CHU102 CHU105 CHU111 CHU115 CHU116 CHU119 CHU128 UNIPD202 UNIPD203

86.42 87.5 88.46 92.21 92.11 89.87 87.76 81.08 93.33

13.58 11.25 10.26 7.79 7.89 10.13 12.24 17.57 6.67

– – – – – – – – –

– 1.25 1.28 – – – – 1.35 –

– – – – – – – – –

Average

88.75

10.82

–

0.43

–

Table 4.2. Percentage of points in EGA-zones for the Abbott Freestyle Navigator (from [122]). Subject

A

B

C

D

E

CHU102 CHU105 CHU111 CHU115 CHU116 CHU119 CHU128 UNIPD202 UNIPD203

93.83 92.5 85.9 94.81 86.84 83.54 48.98 89.19 76

6.17 5 12.82 5.19 10.53 16.46 44.9 8.11 21.33

– – – – – – – – –

– 2.5 1.28 – 2.63 – 6.12 2.7 2.67

– – – – – – – – –

Average

83.51

14.5

–

1.99

–

The results of the EGA for all subjects are summarized in Table 4.1 (regularized estimator) and Table 4.2 (Abbott Freestyle Navigator). From the D-columns in these tables one may conclude that the regularized estimator is, on average, at least 4 times more accurate than the considered commercial system. To make the presented results even more transparent, let us look closely at the blood glucose reading in the case of low blood glucose, for which errors may have important clinical implications for patient health and safety. From Table 4.3 one can see that the Abbott Freestyle Navigator failed to detect 66.7% of hypoglycemic events, whereas the regularized estimator accurately detected 75% of these events. From the comparison of these tables it is clear that in the considered clinical trial the proposed blood glucose estimators outperform the commercially available CGM systems which use the same input information. Moreover, in Table 4.4 we also present the results of the EGA for the blood glucose estimator constructed with the use of the kernel (4.65) and the training set of only one subject, CHU102. The exhibited per-

248

Chapter 4 Regularization algorithms in learning theory

formance was demonstrated in a 3-day test without any calibration to other subjects. Nevertheless, the clinical accuracy of the estimator is still acceptable, compared to the required 94% in A+B-zone and 60% in A-zone [1]. This result demonstrates the potential transferability of a regularized estimator from patient to patient with no calibration. In Table 4.2 one may observe a poor performance of the commercial CGM system in the case of subject CHU128, which is the only type 2 diabetic patient in the considered group. At the same time, as can be seen from Table 4.1, the regularized estimator performs well for this patient, which may be seen as robustness to a diabetes type. These results allow the conclusion that on average the proposed approach for reading blood glucose levels from subcutaneous electric current is more accurate than estimations given by the Abbott Freestyle Navigator on the basis of the standard calibration procedure. The proposed approach can be seen as an answer to the request [56] for “more sophisticated calibration procedure.” We would like to stress that no recalibrations of regularized glucose estimators were made during the assessment period. At the same time, recalibrations of the Abbott Freestyle Navigator should sometimes be made several times per day. Moreover, the proposed algorithm can provide the estimated glucose at any requested time, unlike the existing CGM sensors which have fixed sampling frequency.

Table 4.3. Quantification of blood glucose readings in hypo zone for all subjects in total (12 hypoglycemic events). Accurate

Erroneous

4 9

8 3

Abbott Freestyle Navigator Regularized estimator

Table 4.4. Percentage of points in EGA-zones for the estimator constructed with the use of the kernel (4.65) and the training set of only one subject CHU102. The exhibited performance was demonstrated in a 3-day test without any calibration to other subjects. Subject

A

B

C

D

E

CHU102 CHU105 CHU111 CHU115 CHU116 CHU119 CHU128 UNIPD202 UNIPD203

84.55 52.72 77.78 39.25 60.29 85.32 45.57 86.54 37.38

15.45 45.45 22.22 60.75 38.24 14.68 54.43 13.46 57.94

– – – – – – – – 1.87

– 1.82 – – 1.47 – – – 2.81

– – – – – – – – –

Average

63.27

35.85

0.21

0.68

–

Section 4.5 Multiparameter regularization in learning theory

4.5

249

Multiparameter regularization in learning theory

In this section we use the results of Chapter 3 to discuss how the ideas of the multiparameter discrepancy principle and the model function approximation can be applied to the problem of learning from labeled and unlabeled data. This problem has attracted considerable attention in recent years, as can be seen from the references in [20]. In this section we mainly follow the notation of Chapter 3, and therefore the previous notations will be correspondingly changed. For example, the input-output functions will be denoted by x, while inputs/outputs will be marked by letters t and u accordingly. Recall the standard framework of learning from examples. There are two sets of variables t 2 T Rd and u 2 U R, such that U contains all possible responses/outputs of a system under study on inputs from T . We are provided with a training data set ¹.ti , ui / 2 T  U ºniD1 , which is just a collection of inputs ti labeled by corresponding outputs ui given by the system. The problem of learning consists in, given the training data set, providing a predictor, which is a function x : T ! U , which can be used, given any input t 2 T , to predict a system output as u D x.t /. Learning from examples (labeled data) can be regarded as a reconstruction of a function from sparse data. This problem is ill-posed and requires regularization. For example, Tikhonov regularization gives a predictor xn D xn .ˇ/ as the minimizer of a one-penalty functional 1X .ui  x.ti //2 C ˇkxk2HK . n n

ˆ.ˇ; x/ :D

iD1

We do not touch here the issue of the kernel choice, since it has been discussed in the previous sections. In our experiments below we use the same kernel as in [20], since all required data are borrowed from that paper. Consider now the situation where one is informed that tnC1 , tnC2 , : : : , tnC 2 T may also appear as inputs to the system under study, but responses on these inputs are unknown, i.e., these additional data are not labeled. The reason for this may be that the labeling is very expensive, or difficult to perform. For example, the labeling might require the performance of expensive tests or experiments, such as in medical diagnostics. Therefore, there is considerable practical interest in methods incorporating labeled and unlabeled data in a learning algorithm. One such method was recently proposed in [20]. It consists of adding one more penalty term to the one-penalty functional ˆ.ˇ; x/, such that a predictor xn, D xn, .˛, ˇ/ is constructed as the minimizer of the functional ˆ.˛, ˇ; x/ D

nC n X nC X 1X 2 .ui  x.ti //2 C ˇkxkK C˛ .x.ti /  x.tj //2 Wij , n iD1

iD1 j D1

(4.66)

250

Chapter 4 Regularization algorithms in learning theory

where Wij are some design parameters/weights. For example, in [20] it is suggested that exponential weights Wij D exp.kti tj k2 =4b/ are chosen, where kk D kkRd , and b is some positive number. Then using the graph Laplacian L associated to the graph with .n C / nodes t1 , t2 , : : : , tnC 2 T Rd , adjacent according to a partial ordering induced by Wij , one can rewrite ˆ.˛, ˇ; x/ as ˆ.˛, ˇ; x/ D ˆ.ˇ; x/ C ˛hx, LxiRnC ,

(4.67)

where x D .x.t1 /, x.t2 /, : : : , x.tnC //, and the matrix L of the graph Laplacian admits nC a representation L D D  W with W D .Wij /i,j D1 and the diagonal matrix D given PnC by Di i D j D1 Wij . Observe now that ˆ.˛, ˇ; x/ is just a particular case of the two-parameter Tikhonov functional (3.6). To see this, define the sampling operator SnC : HK ! RnC by .SnC x/i D x.ti /, i D 1, 2, : : : , n C . Keeping in mind that the matrix L is symmetric and positive-semidefinite we have nC X nC X

.x.ti /  x.tj //2 Wij D hx, LxiRnC D hSnC x, LSnC xiRnC

iD1 j D1

 D hSnC LSnC x, xiHK D kBxk2HK ,

(4.68)

 : RnC ! HK is the adjoint of SnC , and B D B  : HK ! HK is where SnC  LSnC . Then the functional (4.66) is nothing but equation (3.6), such that B 2 D SnC  LSnC /1=2 , where X D HK , Y D Rn , yı D .u1 , u2 , : : : , un / 2 Rn , B D .SnC A D JSnC , and J is a .n C /  .n C / diagonal matrix given by J D d i ag.1, 1, : : : , 1, 0, 0, : : : , 0/, with the first n diagonal entries as 1 and the rest 0. Thus, in order to choose regularization parameters ˛, ˇ in the construction of a predictor xn, .˛, ˇ/, one can employ two-parameter discrepancy principle equipped with the model function approximation in the same way as proposed in Section 3.2. The question is only one of a value ı in equation (3.8), where it is interpreted as a noise level. From the analysis presented in Section 4.2 (see Lemma 4.1), it follows that in the context of learning ı D O.n1=2 /, where n is the amount of labeled data. However, this relation has an asymptotic character and gives a pessimistic estimation of the noise level when only a few labeled data are given. Keeping in mind that in the considered case, the discrepancy principle (3.8) has the form n 1X .ui  x.ti //2 D ı 2 , (4.69) n iD1

we can interpret ı here as a level of tolerance to data misfit. For a comparatively small n, such interpretation allows us to use the same value of ı for problems with different amounts of labeled data which is demonstrated in numerical experiments below. Keeping in mind that in our experiments ui takes only two values 1 or 1, any ı 2 .0, 1/ seems to be a reasonable choice.

251

Section 4.5 Multiparameter regularization in learning theory

The experiments are performed with “two moons data set” which was also used in [20]. All software and data has been borrowed from http://www.cs.uchicago.edu/ vikass/manifoldregularization.html. The two moons data set is shown in Figure 4.12. The set contains 200 points divided into two moon-shaped subsets: M1 (the upper moon) and M1 (the lower moon). A successful predictor x D x.t /, t 2 R2 , should distinguish points t 2 M1 from points t 2 M1 . For example, x.t / > 0 for t 2 M1 and x.t / < 0 for t 2 M1 . Laplacian least-squares method 2.5 2 1.5 1 0.5 0 − 0.5 −1 − 1.5 − 1.5

−1

− 0.5

0 1 1.5 0.5 α = 0.37814, β = 0.0078125

2

2.5

Figure 4.12. A successful run of the model function approximation in the Laplacian regularized least squares method with only two labeled points. The ˘ indicates the labeled example ui D 1; the ı indicates the labeled example ui D 1; the other  points are unlabeled examples (from [105]).

We are provided with a training data set ¹.ti , ui /ºniD1 , where ui D 1 for ti 2 M1 and ui D 1 for ti 2 M1 . For n D 2k only k labeled points from each subset M1 , M1 are given. Moreover, the coordinates of all other  D 200  n points ti from M1 and M1 are also given, but they are not labeled. In our experiments a predictor x D xn, .˛, ˇ/ is constructed as the minimizer of the functional (4.66). It allows a representation xn, .˛, ˇ/ D

nC X

ci K.t , ti /,

iD1

where the vector c D .c1 , c2 , : : : , cnC / is given as c D .JK C nˇI C n˛LK/1 u, nC

and u D .u1 , u2 , : : : , un , 0, 0, : : : , 0/ 2 RnC , K D .K.ti , tj //i,j D1 . As in [20], the kernel K is chosen as K.t ,  / D exp.kt   k2 /,  D 0.35.

252

Chapter 4 Regularization algorithms in learning theory

We perform experiments, whereby the number of labeled points in M1 and M1 take the values of n D 2, 6, 10, 20, 40, 60. 500 tests with labeled points randomly chosen in M1 and M1 were performed for each of these values, and we have calculated the percentage (SP) of successful predictors xn,200n .˛, ˇ/ constructed during the tests. In all our experiments the parameters ˛, ˇ are chosen by the algorithm described in Section 3.2.5 with the starting values ˛ D 2, ˇ D 1, and ı 2 D 0.01, D 0.5. The number of iterations does not exceed 10. In the case considered all involved norms have been specified above and can be explicitly calculated. For example, kBxk2 can be found using equation (4.68). The results are presented in Table 4.5, where we also indicate the maximal number of points which were wrongly classified (WC) within one test, as well as the average number (AW) of wrongly classified points over all 500 tests. For comparison we also present Table 4.6, where the performance of single-parameter predictor xn .ˇ/ constructed as the minimizer of ˆ.ˇ; x/ is shown. The predictor xn .ˇ/ with ˇ chosen by the discrepancy principle has been used in parallel to xn,200n .˛, ˇ/. It is clear that in all considered cases two-parameter regularization outperforms single-parameter regularization. Typical classification produced by a successful preTable 4.5. Performance of the model function approximation in Laplacian regularized least squares algorithm over 500 tests. Symbols: labeled points (n); successful predictors (SP); maximum of wrongly classified points (WC); average of wrongly classified points (AW). (from [105])

n D 60 n D 40 n D 20 n D 10 nD6 nD2

SP

WC

AW

100% 100% 100% 100% 99.8% 49.4%

0 0 0 0 2 42

0 0 0 0 0.04 3.371

Table 4.6. Performance of the single-parameter discrepancy principle in regularized least squares algorithm over 500 tests. Symbols are the same as in Table 4.5 (from [105]).

n D 60 n D 40 n D 20 n D 10 nD6 nD2

SP

WC

AW

81% 70.8% 43.6% 14% 3.2% 0%

11 16 34 85 91 91

0.412 0.916 3.276 14.294 27.71 27.71

Section 4.5 Multiparameter regularization in learning theory

253

dictor xn,200n .˛, ˇ/ constructed with the use of only n D 2 labeled points is shown in Figure 4.12. A data-driven choice of the regularization parameters ˛ D 0.37814, ˇ D 0.0078125 was automatically made by our algorithm. Similar pictures can also be found in [20], where the regularization parameters have been hand-chosen. Moreover, the authors of [20] mention that they “do not as yet have a good understanding of how to choose these parameters”. We hope that the proposed model function approximation for the multiparameter discrepancy principle sheds some light on this issue.

Chapter 5

Meta-learning approach to regularization – case study: blood glucose prediction 5.1

A brief introduction to meta-learning and blood glucose prediction

Previously discussed approaches to data-driven regularization can be characterized as online strategies, since the same input information, say equation (4.57), is used for choosing a regularization scheme and performing regularization. At the same time, several authors [32, 38, 154, 155] discuss the possibility of choosing the instances of regularization algorithms, e.g., kernels, on the basis of experience with similar or simulated approximation tasks. Such choice rules may be called offline strategies, because they are essentially specified in advance. To make the discussion more concrete we consider here only the case of input data given in the form (4.57). Moreover, we assume that the unknown functional dependency fun is supposed to belong to some family H , and several representers .j / fknown 2 H , j D 1, 2, : : : , J , of this family are available. In medical applications, such as diabetes technology, for example, these representers can be given as the shapes of blood glucose profiles of a particular patient, which were observed in previous days, while the aim is to predict a future shape fun .x/ from the current day data (4.57). In this context the points x1 < x2 < : : : < xn from (4.57) remain for moments during the day, and fun .x/ should be predicted by extrapolation from (4.57) for x 2 Œxn , xn C PH , where PH > 0 is a prediction horizon. .j / In general, with the use of given representers fknown , one can gather a training set .j / of simulated tasks consisting of reconstructing the values fknown .xi / at some points xi , i D nC1, nC2, : : : , nCm, being outside the scope of the inputs ¹xi ºniD1 (e.g., xi 2 .j / .j / Œxn , xn C PH / from synthetic data W .j / D ¹.xi , yi /, i D 1, 2, : : : , nº, yi D .j / .j / .j / fknown .xi /C"i , i D 1, 2, : : : , n, of the same form as (4.57), where "i are artificially .j / .j / created errors. Note that since the values yi D fknown .xi /, i D nC1, nC2, : : : , nC m, are known but not supposed to be taken for the reconstruction, they can be used for the accuracy assessment. In particular, if the reconstruction f ./ D f .; K, W .j / / is expected to be of the form (4.59), then its accuracy can be assessed in terms of the quantity .j / .j / .j / P .f .; K, W .j / /, Wass / defined by (4.62), where WP D Wass D ¹.xi , yi /, i D n C 1, n C 2, : : : , n C mº, and  : R  R ! RC is a prescribed loss function, e.g., .u, v/ D ju  vj2 .

256

Chapter 5 Meta-learning approach and blood glucose prediction .j /

Note that for a fixed j the quantity P .f .; K, W .j / /, Wass / may be seen as a function of K and , but as we know, the regularization parameter  in equations (4.58) and (4.59) depends on the choice of the kernel K. Therefore, in general, it is reasonable to considert  a functional of K, i.e.,  D .K/. Assume now that one is interested in choosing a kernel K from some admissible set K. / of parameter-dependent kernels K D K.!; , /, where, for the moment, all parametric dependencies are collected in the parameter vector ! 2 RN . One may then follow the strategy outlined in [38] and construct a kernel selection criterion as follows: K D K.!0 ; , / 9 8 J = t1 > t2 >    > tmC1 within the sampling horizon SH D t0  tmC1 . The goal is to construct a predictor which uses these past measurements to predict BG-concentration as a function of time g D g.t / for n subsequent future points in time ¹tj ºjnD1 within the prediction horizon PH D tn  t0 , such that t0 < t1 < t2 <    < tn . At this point, it is noteworthy to recall that CGM systems provide estimations ¹gi º of BG-values every 5 or 10 minutes, such that ti D t0 C i t , i D 1, 2, : : : , where t D 5 (min) or t D 10 (min). For mathematical details see Section 4.4.1. Thus, the promising concept in diabetes therapy management is the prediction of future BG-evolution using CGM data [151]. The importance of such predictions has been shown by several applications [125]. From the above discussion, one can see that CGM technology allows us to form a training set z D ¹.x , y /,  D 1, 2, : : : , M º, jzj D M , where 







x D ..tmC1 , gmC1 /, : : : , .t0 , g0 // 2 .R2C /m ,

  y D ..t1 , g1 /, : : : , .tn , gn // 2 .R2C /n ,     and tmC1 < tmC2 <    < t0 < t1
12 , we put more emphasis on the ability to mimic the input data from W1 , while for  closer to zero, we are more interested in making a generalization from those data. The minimization of the functional (5.9) is performed in the first and last meta-learning phases. In the first case, we minimize equation (5.9) with  D 0, while in the second we put  D 12 . The existence of the kernel K 0 and the regularization parameter 0 minimizing the functional (5.9) was proven in Proposition 4.9. To illustrate the assumptions of this proposition, we consider two cases which are needed to set up our meta-learning predictor. In both cases the quasibalancing principle (4.54) is used as a parameter choice rule  D .K/ 2 Œ104 , 1. In the first case, we use the data (5.3), and for any j D 1, 2, : : : , M define the sets j

j

j

j

W1 D W .j / D xj D ..tmC1 , gmC1 /, : : : , .t0 , g0 //, .j /

j

j

j

j

W2 D Wass D yj D ..t1 , g1 /, : : : , .tn , gn //,

j

j

t0  tmC1 D SH , j

tnj  t0 D PH .

In this case, the input environment U is assumed to be a time interval, i.e., U

.0, 1/, while the output environment V D Œ0, 450 is the range of possible BG values (in mg/dL). For this case, we choose D ¹! D .!1 , !2 , !3 / 2 R3 , !i 2 Œ104 , 3, i D 1, 2, 3º, and the set of admissible kernels is chosen similar to that of equation (4.60) as K. , G/ D ¹K : K! .t ,  / D .t  /!1 C !2 e !3 .t/ , 2

.!1 , !2 , !3 / 2 º. (5.10)

265

Section 5.3 Meta-learning approach

To apply Proposition 4.9 in this case, we modify the functional (5.8) involved in the representation of equation (5.9) as in equation (4.64) with the idea of heavily penalizing the failure of detection of dangerous glycemic levels. As a result of the application of Proposition 4.9, we relate input-output BG observa0 0 0 , !2,j , !3,j / of our favorite kernels tions .xj , yj / to the parameters ! 0 D !j0 D .!1,j 0 0 0,j K D K D K!j0 and j D j . As already mentioned, the corresponding optimization is executed only for the data set of one particular patient. Thus, the operation in this case does not require considerable computational effort and time. The second case of the use of Proposition 4.9 corresponds to the optimization which should be performed in the final meta-learning phase. We consider the transformed 0 /, j D 1, 2, : : : , M º, i D 1, 2, 3, obtained after performing data sets zi D ¹.uj , !i,j the first two meta-learning operations. In this case the input environment U is formed by two-dimensional metafeatures vectors uj 2 R2 computed for the inputs xj , i.e., U R2 , whereas the output environment V D Œ104 , 3 is the range of parameters !i of the kernels from equation (5.10). Recall that in the final meta-learning phase the goal is to assign the parameters ! 0 D .!10 , !20 , !30 /, 0 of a favorite algorithm to each particular input x, and this assignment should be made by comparing the metafeature u calculated for x with the metafeatures uj of inputs xj , for which the favorite parameters have already been found in the first meta-learning phase. In the meta-learning literature one usually makes the above-mentioned comparison by using some distance between metafeature vectors u and uj . For two-dimensional .1/ .2/ metafeatures u D .u.1/ , u.2/ /, uj D .uj , uj /, one of the natural distances is the weighted Euclidean distance .1/

.2/

1

ju  uj j :D . 1 .u.1/  uj /2 C 2 .u.2/  uj /2 / 2 which may potentially be used in meta-learning ranking methods in the same way as the distance suggested in [154] (see also Section 5.3.3 below). Here we refine this approach by learning the dependence of parameters 0 , !i0 , i D 1, 2, 3, on the metafeature u in the form of functions F .u/ D

M X

cj '! .ju  uj j /,

j D1

where ! D .!1 , !2 , !3 , !4 / 2 D Œ0, 2  Œ0, 15  Œ0, 2  Œ0, 15, '! . / D  !1 C !4 !2 e !3  , and the corresponding coefficients cj for 0 , !i0 , i D 1, 2, 3, are defined in accordance with the formula (5.13) below. This means that the final meta-learning phase can be implemented as the optimization procedure described in Proposition 4.9, where the set of admissible kernels is

266

Chapter 5 Meta-learning approach and blood glucose prediction

chosen as follows: K D K . , G/ ´ O D M 1 D K : K!, .u, u/

M X

μ '! .ju  uj j /'! .juO  uj j /, ! 2 . (5.11)

j D1

It is well known (see, e.g., [55]) that for anyP continuous and linearly independent M 1 O is positive-definite. functions gi , i D 1, 2, : : : , M , the sum M iD1 gi .u/gi .u/ O from equation (5.11) are really scalar-valued This means that all functions K!, .u, u/ kernels. Moreover, it is clear that for any fixed  the value '! . / depends continuously on !. Therefore, in the case of the set (5.11), the conditions of Proposition 4.9 are satisfied. To apply the optimization procedure above, we rearrange the sets zi , so that zi D 0 0 0 /º, where !i,j < !i,j , k D 1, 2 : : : , M  1, and define the sets W1 , W2 ¹.ujk , !i,j k k kC1 as follows: 0 /, k D 3, : : : , M  2º, W1 D W1,i D ¹.ujk , !i,j k

W2 D W2,i D zi n W1,i ,

so that the performance estimation sets W2 D W2,i contain the two smallest and two largest values of the corresponding parameters. Moreover, for the case considered we use the functional (5.8) with .f , v/ D jf  2 vj . For i D 1, 2, 3, using the optimization procedure described in Proposition 4.9 one can then find the kernels K 0 D Ki0 2 K . , G/ determined by the values of parameters that are presented in Table 5.1. In addition, using the set ¹.uj , j0 /º in the same way, one can obtain the kernel K40 2 K . , G/ whose parameters are also given in Table 5.1. Summing up, as a result of the optimization operations we first find for each inputoutput pair .xj , yj /, j D 1, 2, : : : , M , the parameters of the favorite kernel K 0 D Table 5.1. The parameters of the kernels from (5.11) which are selected for learning at metalevel (from [130]). 1

2

!1

!2

!3

!4

K10

1

0

1.6

5

0.001

0.016

K20

1

0

1.2

0.001

3

0.01

K30

1

0

0

1

0.001

0.003

K40

1

1

0.2

0.02

0.1

0.2

267

Section 5.3 Meta-learning approach

K 0,j from equation (5.10) and 0 D j0 2 Œ104 , 1. Using these parameters we then construct the kernels K 0 D Ki0 , i D 1, 2, 3, 4, from equation (5.11) which will relate .K 0,j , j0 / to the corresponding metafeatures uj . In both cases the minimization of the corresponding functionals (5.9) was performed by a full search over grids of parameters ! determining the kernels from equations (5.10) and (5.11). Of course, the application of the full search method is computationally intensive, but, as already mentioned, in our application this minimization procedure should be performed only once and only for one particular patient.

5.3.2

Heuristic operation

The goal of this operation is to extract special characteristics ¹uj º called metafeatures of inputs ¹xj º which can be used to explain the relationship between ¹xj º and the parameters of optimal algorithms predicting training outputs ¹yj º from ¹xj º. Note that it is common belief [27] that such metafeatures should reflect the nature of the problem to be solved. Keeping in mind that practically all predictions of the future blood glucose concentration are currently based on a linear extrapolation of glucose values [92], it seems .1/ .2/ natural to consider the vector uj D .uj , uj / of coefficients of a linear extrapola.1/

.2/

tor gjli n .t / D uj t C uj , producing the best linear fit for given input data xj D j

j

j

j

..tmC1 , gmC1 /, : : : , .t0 , g0 //, as a good candidate for a metafeature of xj . For any given input x D ..tmC1 , gmC1 /, : : : , .t0 , g0 // the components of the corresponding metafeature u D .u.1/ , u.2/ / are then determined by the linear least squares fit as follows: u.1/ D

0 X iDmC1

.ti  tN/.gi  g/ N , 0 P 2 .ti  tN/

u.2/ D gN  u.1/ tN,

(5.12)

iDmC1

here N is an average. Note that in principle the linear extrapolator g li n .t / D u.1/ t C u.2/ can be used to predict future BG concentration from x. As can been seen from [151], for prediction horizons of clinical interest (PH > 10 min) such a predictor is outperformed by more sophisticated algorithms. Therefore, we are going to use the coefficient vector u D .u.1/ , u.2/ / only as a metafeature (label) of the corresponding prediction input.

5.3.3

Learning at metalevel

The goal of the final phase of the meta-learning approach, which is also called learning at metalevel, is the construction of the so-called metachoice rule for selecting the vector ! D .!1 , !2 , !3 / of the parameters of favorite algorithm which will be applied

268

Chapter 5 Meta-learning approach and blood glucose prediction

to input x in question labeled by a metafeature u. Recall that at this stage the abovementioned metachoice rule is constructed on the basis of the transformed training sets 0 /º, i D 1, 2, 3. zi D ¹.uj , !i,j In this section, we describe two metachoice rules. The first, the k-Nearest Neighbors (k-NN) ranking, is one of the most popular methods found in meta-learning literature. This method has been suggested in [154] and the idea behind it is to identify a set of k metafeatures ¹uj º containing those which are most similar to the considered metafeature u, and then combine the corresponding ¹!j0 º to select the vector ! for the new input x. In their numerical experiments the authors [154] observed a clear tendency for the accuracy of k-NN ranking to decrease with an increasing number of k neighbors. We therefore only consider the 1-NN ranking method as one which produces more accurate results than other k-NN rankings. Using [154] we describe how 1-NN ranking can be adjusted to the task of blood glucose prediction, in particular in order to deal with transformed training sets zi . The use of 1-NN ranking meta-learning involves the three following steps: (1) Calculate the distances between the metafeature u D .u.1/ , u.2/ / of the input x .1/ .2/ in question and all other uj D .uj , uj /, j D 1, 2, : : : , M as follows: dist.u, uj / D

.i/

2 X

ju.i/  uj j

iD1

max.uj /  min.uj /

.i/

.i/

.

(2) Find j 2 ¹1, 2, : : : , M º, such that dist.u, uj / D min¹dist.u, uj /, j D 1, 2, : : : , M º. (3) For the input x in question take the vector ! D !j0 which results in choice of the kernel K 0 D K!j0 from the set (5.10) and  D j0 . 

The second metachoice rule we propose is based on the kernel choice procedure deO : : : , K40 .u, u/ O scribed in Proposition 4.9 or, more specifically, on the kernels K10 .u, u/, obtained in the second case of its application. This rule can be executed as follows: 0 /º, i D 1, 2, 3 and ¹.u , 0 /º, (1) Using the transformed training sets zi D ¹.uj , !i,j j j 0 0 we define the following functions: !i D !i .u/, i D 1, 2, 3, 0 D 0 .u/ of a metafeature vector u 2 R2 : ´ μ M 0 1 P 0 2 2 .!.uj /  !i,j / C ˛i jj!jjH 0 , i D 1, 2, 3, !i D arg min M

0 D arg min

K i

j D1

´ 1 M

M P j D1

μ

..uj /  j0 /2 C ˛4 jjjj2H

K40

(5.13) ,

where the regularization parameters ˛i D ˛i .Ki0 / 2 Œ0 , 1, 0 D 104 are chosen in accordance with the quasibalancing principle (4.54).

269

Section 5.4 Case-study: blood glucose prediction

(2) Calculate the metafeature u D u.x/ 2 R2 for a prediction input x in question and choose kernel and regularization parameter as follows: K.t ,  / D K! 0 .u/ .t ,  / D .t  /!1 .u/ C !20 .u/e !3 .u/.t/ , 0 D 0 .u/. 0

0

2

(5.14)

Once any of the above-mentioned metachoice rules are employed, the prediction g.t / of future BG concentration for the point in time t 2 Œt0 , t0 C PH  can be constructed from past BG estimates x D ..tmC1 , gmC1 /, : : : , .t0 , g0 //, t0  tmC1 D SH as follows. First we calculate a metafeature vector u D u.x/ D .u.1/ , u.2/ / as a result of the heuristic operation (5.12). Using the metachoice rule employed, we then specify a kernel K D K! 0 .u/ from the set (5.10) and  D 0 .u/. Finally, the prediction g D g.t / is defined by means of the regularization performed in the space H D HK . Here one may use, for example, two iterations of the Tikhonov regularization, defined as follows: g .0/ D 0, g

. /

´

1 D arg min m

g.t / D g

.2/

0 X

μ .g.ti /  gi / C jjg  2

g . 1/ jj2HK

, D 1, 2,

iDmC1

.t /,

(5.15)

where  is chosen from Œ0 .u/, 1 by means of the quasibalancing principle (4.54).

5.4

Case-study: blood glucose prediction

In this section, we compare the performance of state-of-the-art BG predictors [127, 140] with that of meta-learning based predictors described in Section 5.3. It is remarkable, in retrospect, that in all cases the meta-learning based predictors outperform their counterparts in terms of clinical accuracy. Even more remarkable, for some prediction horizons BG predictors based on the FARL approach perform at the level of the clinical accuracy achieved by CGM systems providing the prediction input. Clearly such accuracy cannot generally be beaten by CGM-based predictors. The performance assessment has been made with the use of the classic Clarke Error Grid Analysis (EGA) described in Section 4.4.1 (see Figures 4.9 and 4.10). Since the developed BG predictors are more dedicated to patients with high glucose variability, including a significant risk of hypoglycemia, the performance tests were performed mainly on type 1 diabetic patients. The lack of residual insulin secretion in these patients is considered to be a determining factor for higher glucose variability and poorer blood glucose predictability.

270

Chapter 5 Meta-learning approach and blood glucose prediction

Input means: Current and past BG-estimates g m ; g mC1 ; : : : ; g0 made at time moments < t0 t m < t mC1
t0

.gj ; tj / Metafeature extraction: Labeling input data .gj ; tj /, m; m C 1; : : : ; 0, j by a coefficient vector u of the linear fit

u

Metalearning stage: Optimal .K / and corresponding meta-features (labels) u are used for learning how to learn a kernel K D and a parameter for any input data .gj ; tj /, j m; mC 1; : : : ; 0, labeled by a meta-feature (label) u .K

Prediction setting stage

;u /

Metafeature extraction: Labeling historical data .gj ; tj /, j m; m C 1; : : : ; 0, by coefficients vectors u of the best linear fits .K

/

Learning stage: Representative segments of historical BG-measurements g m ; g mC1 ; : : : ; g0 g1 ; : : : ; gn taken from one particular patient (ld. CHU 102) at time moments t m < t mC1 < < t0 < t1 < < tn are used for learning optimal kernels K D K and parameters D , such that P0 gi m cj gj K .ti ; tj /; i D 1; 2; : : : ; n; j D 1; 2; : : : ; M .

Figure 5.2. Meta-learning approach to BG prediction: Fully Adaptive Regularized Learning algorithm (from [130]).

The performance tests were made using of clinical data from trials executed within the EU-project “DIAdvisor” [2] at the Montpellier University Hospital Center (CHU), France, and at the Institute of Clinical and Experimental Medicine (IKEM), Prague, Czech Republic. In general, patients which met the following inclusion criteria were enrolled into the study: male or female between 18 and 70 years of age, diagnosed with type 1 diabetes according to the World Health Organization criteria at least one year prior to study entry; HbA1C between 7.5% and 10.5% and body mass index lower than 35 (Kg/m2 /. In the considered trials (DAQ-trials), each clinical record of a diabetic patient contains nearly 10 days of CGM data collected with the use of the CGM system Abbott’s Freestyle Navigator [3], with a sampling frequency t D 10 (min).

Section 5.4 Case-study: blood glucose prediction

271

For comparison with the state-of-the-art, we considered two BG predictors described in the literature, such as the data-driven autoregressive model-based predictor (ARpredictor) proposed in [140] and the neural network model-based predictor (NNMpredictor) presented in [127]. It is instructive to see that these predictors required more information to produce a BG prediction than necessary for our approach. More precisely, AR-predictors use past CGM-measurements sampled every minute for input. As to NNM-predictors, their input consists of CGM-measurements sampled every 5 minutes, as well as meal intake, insulin dosage, patient symptoms and emotional factors. On the other hand, the FARL-based predictor uses only CGM-measurements from the past 30 minutes for input and, more importantly, these measurements do not need to be equi-sampled. Recall that in Section 5.3 we mentioned that an important feature of our algorithm is transferability from individual to individual. To be more specific, for metalevel learning we use CGM-measurements performed with only one patient (patient ID: CHU102). These measurements were collected over one day of the DAQ-trial with the use of the Abbott sensor. The training data set z D ¹.xj , yj /, j D 1, 2, : : : , M º, M D 24, was formed from the data of the patient CHU102 with the sampling horizon SH D 30 minutes and the training prediction horizon PH D 30 minutes. Application of the procedure described in Proposition 4.9 in the first case transforms the training set z into the values !j0 D 0 0 0 , !2,j , !3,j /, j0 , j D 1, 2, : : : , M , defining the favorite kernel and regularization .!1,j parameters. The transformed training sets ¹.xj , yj /º ! ¹.uj , !j0 /º, ¹.uj , j0 /º, j D 1, 2, : : : , 24, were then used for metalevel learning with the FARL method, as well as with the 1-NN ranking method. First the obtained fully trained BG predictors were tested without any readjustment on the data that were collected during 3 days in hospital and 5 days outside the hospital under real-life conditions from the other 10 patients taking part in the DAQ-trial. Since the goal of the trial was to gain a true picture of blood glucose fluctuation and insulin-glucose interaction in different environmental conditions, no specific intervention regarding the normal diabetic treatment of the patients was made. The number of patients is comparable to those used to test the AR- and NNMpredictors, but testing periods for those predictors were shorter than ours. Moreover, transferability from patient to patient was demonstrated only for the AR-predictor, and only for 2 patients [140]. As to NNM-predictors [127], they were trained using data from 17 patients and tested on data from 10 other patients. To assess the clinical accuracy of the predictors compared we employed EGA since this performance measure was used in [127, 140] to quantify the accuracy of AR- and NNM-predictors. For the prediction horizons PH D 30 (min) and PH D 60 (min), the clinical accuracy of the FARL-predictors is demonstrated in Tables 5.2 and 5.6. For the same

272

Chapter 5 Meta-learning approach and blood glucose prediction

prediction horizons the comparison of the FARL-predictors to AR-predictors [140], as well as to the predictors based on 1-NN ranking, can be made using Tables 5.4, 5.5 and 5.3, 5.7 respectively. Tables 5.8–5.10 can be used for the comparison of the FARL-predictors against the predictors based on neural networks modeling and on 1-NN ranking. These tables display the prediction accuracy for PH D 75 (min), since only this horizon was discussed in [127]. From the comparison of Tables 5.2–5.10 one can expect that the proposed FARLpredictors have higher clinical accuracy than their counterparts based on data-driven autoregressive models or on neural networks models. Table 5.2. Performance of FARL-predictors for PH D 30 (min) (from [130]). Patient ID

A (%)

B (%)

C (%)

D (%)

E (%)

CHU101 CHU102 CHU105 CHU107 CHU108 CHU115 CHU116 IKEM305 IKEM306 IKEM309

85.07 94.38 93.26 91.69 87.31 96.18 93.26 89.88 89.81 92.12

14.93 5.62 6.74 8.03 12.69 3.05 6.74 9.29 10.19 7.88

– – – – – – – – – –

– – – 0.28 – 0.76 – 0.83 – –

– – – – – – – – –

Average

91.3

8.51

–

0.19

–

Table 5.3. Performance of 1-NN ranking predictors for PH D 30 (min) (from [130]). Patient ID

A (%)

B (%)

C (%)

D (%)

E (%)

CHU101 CHU102 CHU105 CHU107 CHU108 CHU115 CHU116 IKEM305 IKEM306 IKEM309

82.84 92.13 90.64 86.9 88.43 92.75 90.64 89.55 90.78 89.16

17.16 7.87 9.36 12.25 11.57 6.49 9.36 9.95 9.22 10.84

– – – – – – – 0.17 – –

– – – 0.85 – 0.76 – 0.33 – –

– – – – – – – – – –

Average

89.38

10.41

0.02

0.19

–

273

Section 5.4 Case-study: blood glucose prediction Table 5.4. Performance of AR-predictors for PH D 30 (min) (from [130]). Patient ID

A (%)

B (%)

C (%)

D (%)

E (%)

6-6 6-8 8-6 8-8

85.3 84.4 82.2 90

13.3 14.2 15 9.8

– – – –

1.4 1.4 2.8 0.2

– – – –

Average

85.48

13.07

—

1.45

—

Table 5.5. Performance of AR-predictors for PH D 60 (min) (from [130]). Patient ID

A (%)

B (%)

C (%)

D (%)

E (%)

6-6 6-8 8-6 8-8

66.2 64.2 60.7 72.9

31.1 32.5 32.9 25.1

0.6 0.2 0.8 –

2.1 3.1 5.4 2.0

– – – –

Average

66

30.4

0.4

3.15

—

Table 5.6. Performance of FARL-predictors for PH D 60 (min) (from [130]). Patient ID

A (%)

B (%)

C (%)

D (%)

E (%)

CHU101 CHU102 CHU105 CHU107 CHU108 CHU115 CHU116 IKEM305 IKEM306 IKEM309

70.15 76.03 78.28 73.24 69.4 77.48 76.4 79.27 75.73 75.37

29.85 23.97 21.72 26.48 30.6 20.61 22.1 18.57 22.82 24.63

– – – – – – 0.75 0.33 0.49 –

– – – 0.14 1.91 0.75 1.66 0.97 –

– – – 1.14 – – – 0.17 – –

Average

75.14

24.13

0.16

0.54

0.13

One further interesting observation can be made from the comparison of Tables 5.8 and 5.10, where the clinical accuracy of FARL- and NNM-predictors is reported. As already mentioned, the input for NNM-predictors is much more informative than that for FARL systems. In addition to CGM-measurements it also contains meal intakes and insulin dosages, which, of course, directly influence BG levels. At the same time, FARL-systems need only past CGM-values to produce predictions. Neverthe-

274

Chapter 5 Meta-learning approach and blood glucose prediction

Table 5.7. Performance of 1-NN ranking predictors for PH D 60 (min) (from [130]). Patient ID

A (%)

B (%)

C (%)

D (%)

E (%)

CHU101 CHU102 CHU105 CHU107 CHU108 CHU115 CHU116 IKEM305 IKEM306 IKEM309

63.06 56.93 50.19 41.13 73.13 51.15 34.46 66.83 48.06 41.38

36.57 43.07 49.81 54.79 26.87 43.89 62.55 31.01 47.57 52.22

– – – – – – – 0.33 – –

– – – 3.66 4.96 3 1.66 4.37 6.4

0.37 – – 0.42 – – – 0.17 – –

Average

52.63

44.84

0.03

2.4

0.1

Table 5.8. Performance of FARL-predictors for PH D 75 (min) (from [130]). Patient ID

A (%)

B (%)

C (%)

D (%)

E (%)

CHU101 CHU102 CHU105 CHU107 CHU108 CHU115 CHU116 IKEM305 IKEM306 IKEM309

68.28 68.91 70.41 72.83 64.55 67.18 71.91 71.64 67.96 64.04

31.72 30.71 29.59 27.17 35.45 31.3 25.09 25.04 28.16 35.47

– – – – – – 1.5 – 2.43 –

– 0.37 – – – 1.53 1.5 2.82 1.46 0.49

– – – – – – – 0.5 – –

Average

68.77

29.97

0.39

0.82

0.05

less, comparing Tables 5.8 and 5.10, one may conclude that even without the use of above-mentioned additional information, FARL-predictors have higher clinical accuracy than NNM-models. A possible explanation for this is that in the case considered FARL-predictors use BG estimations from the past 30 minutes, and since the effects of insulin and meal responses on BG levels are independent [90] and occur within a shorter time frame [153], the influence of these factors, if they take place when collecting a prediction input, may already be seen in the data. In this way information about them is indirectly taken into account by FARL-predictors. Note that the accuracy reported in Tables 5.2–5.10 was measured against the estimates of blood glucose given by a commercial CGM system, which, in fact, reads the glucose in the interstitial fluid and is not always accurate in reporting blood glu-

275

Section 5.4 Case-study: blood glucose prediction Table 5.9. Performance of 1-NN ranking predictors for PH D 75 (min) (from [130]). Patient ID

A (%)

B (%)

C (%)

D (%)

E (%)

CHU101 CHU102 CHU105 CHU107 CHU108 CHU115 CHU116 IKEM305 IKEM306 IKEM309

61.19 46.82 36.7 30.7 66.04 41.98 26.22 58.87 36.41 35.96

38.43 52.81 49.81 62.96 33.96 51.53 68.91 37.98 58.25 52.71

– – – – – – – 0.33 – –

– 0.37 – 5.49 – 6.49 4.87 2.32 5.34 11.33

0.37 – – 0.85 – – – 0.5 – –

Average

44.09

50.73

0.03

3.62

0.17

Table 5.10. Performance of NNM-predictors for PH D 75 (min) (from [130]). Patient ID

A (%)

B (%)

C (%)

D (%)

E (%)

1 2 3 4 5 6 7 8 9 10

57.2 38.7 58.2 58.8 68.2 64.9 42.4 71.8 71.9 78.6

38 40.3 37.3 28.4 24.4 30.4 37.7 28.2 23.7 18.6

1.5 1.2 0.5 0.2 1.2 0.3 – – – –

3.3 19 3.9 12.2 6.2 4.5 19.4 – 4.4 2.8

– 7 – 0.4 – – 0.5 – – –

Average

62.3

30

0.4

7.1

0.1

cose concentration (see, for example, [122]). Although such CGM systems provide the inputs for predictors, the goal is to predict the real blood glucose. Therefore, it is interesting to estimate the prediction accuracy against the blood glucose measurements. We can do this using the same clinical data as in Section 4.4.1. Since we presented a sophisticated approach to the blood glucose reading problem in that subsection, however, it seems reasonable to try to use the input from the blood glucose estimator based on the kernel (4.65) as the prediction input. The results of an EGA for nine subjects who participated in the DAQ-trial are summarized in Table 5.11. Comparison of Tables 4.2 and 5.11 allows the conclusion that on average the FARLpredictors based on the input from the developed BG estimators is even more accurate than the commercial CGM systems for PH D 20 (min.). This is clearly seen from

276

Chapter 5 Meta-learning approach and blood glucose prediction

Table 5.11. Performance of FARL-predictors for PH D 20 (min) based on the input from the estimators with the kernel (4.65). Patient ID

A (%)

B (%)

C (%)

D (%)

E (%)

CHU102 CHU105 CHU111 CHU115 CHU116 CHU119 CHU128 UNIPD202 UNIPD203

81.82 73.86 77.01 82.35 80.43 80.23 86.89 75.9 80.23

18.18 22.73 21.84 17.65 19.57 19.77 11.47 22.89 19.77

– – – – – – – – –

– 3.41 1.15 – – – 1.64 1.2 –

– – – – – – – – –

Average

79.86

19.32

–

0.82

–

the percentage of points in D-zones: 20 min ahead blood glucose predictions are half as erroneous as blood glucose readings provided by the commercially available CGM systems. It should be noted that the results reported in Table 5.11 correspond to the case when only 2 blood glucose measurements/ estimations g0 , g1 with a time interval t D 5 (min) are used as a predictor input, i.e., m D 2. To the best of our knowledge, such a “poor input” has never been used for predicting blood glucose concentration before. Nevertheless, reported results show that for clinically important 20 min ahead blood glucose predictions, the reliability of predictors based on the FARL-algorithm is level with blood glucose meters reliability. This can be seen as a proof-of-concept for the use of the Fully Adaptive Regularized Learning algorithm in estimating and predicting blood glucose concentrations.

Bibliography

[1] http://www.childrenwithdiabetes.com, last accessed on 2013 April 13. [2] http://www.diadvisor.eu, last accessed on 2013 April 13. [3] http://www.freestylenavigator.com, last accessed on 2013 April 13. [4] http://www.hemocue.com, last accessed on 2013 April 13. [5] G. Alessandrini and E. Sincich, Solving elliptic Cauchy problems and the identification of nonlinear corrosion, J. Comput. Appl. Math. 198 (2007), 307–320. [6] B. Aliev, Estimation of the regularization method for the problem of the summability of the Fourier series, (in Russian) Dokl. Akad. Nauk Tadzhik. SSR 21 (1978), 3–6. [7] B. Aliev, Two-parameter regularization method for finding L-pseudosolution, Vest. Mosk. Univ. 15 (1986), 45–50. [8] R. S. Anderssen, The linear functional stragegy for improperly posed problems, in: Inverse Problems, pp. 11–30, Birkhäuser, Basel, 1986. [9] M. Azaïez, F. Ben Belgacem, and H. El Fekih, On Cauchy’s problem: II. Completion, regularization and approximation, Inverse Problems 22 (2006), 1307–36. [10] A. Barron and T. Cover, Minimum complexity density estimation, IEEE Trans. Inform. Theory 37 (1991), 1034–1054. [11] D. Barry, Nonparametric Bayesian regression, Ann. Statist. 14 (1986), 934–953. [12] P. L. Bartlett, M. J. Jordan, and J. D. McAuliffe, Convexity, classification, and risk bounds, J. Amer. Statist. Assoc. 101 (2006), 138–156. [13] F. Bauer and O. Ivanyshyn, Optimal regularization with two interdependent regularization parameters, Inverse Problems 23 (2007), 331–342. [14] F. Bauer and S. Kindermann, The quasi-optimality criterion for classical inverse problems, Inverse Problems 24 (2008), 035002–0350022. [15] F. Bauer and S. V. Pereverzev, Regularization without preliminary knowledge of smoothness and error behaviour, European J. Appl. Math. 16 (2005), 303–317. [16] F. Bauer and S. V. Pereverzev, An utilization of a rough approximation of a noise covariance within the framework of multi-parameter regularization, Int. J. Tomogr. Stat. 4 (2006), 1–12. [17] A. Beck, A. Ben-Tal, and M. Teboulle, Finding a global optimal solution for a quadratically constrained fractional quadratic problem with applications to the regularized total least squares, SIAM J. Matrix Anal. Appl. 28 (2006), 425–445.

278

Bibliography

[18] M. Belge, M.E. Kilmer, and E. L. Miller, Efficient determination of multiple regularization parameters in a generalized L-curve framework, Inverse Problems 18 (2002), 1161–1183. [19] M. Belge and E. L. Miller, Wavelet domain Bayesian image restoration using edgepreserving prior models, in: Prc. ICIP’98 (1998). [20] M. Belkin, P. Niyogi, and V. Sindhwani, Manifold regularization: a geometric framework for learning from labeled and unlabeled examples, J. Mach. Learn. Res. 7 (2006), 2399–2434. [21] F. Ben Belgacem, Why is the Cauchy problem severely ill-posed?, Inverse Problems 23 (2007), 823–836. [22] R. Bhatia, Matrix Analysis, Springer-Verlag, New York, 1997. [23] B. Bilalov, Bases of Exponentials, Cosines, and Sines Formed by Eigenfunctions of Differential Operators, Differential Equations 39 (2003), 652–57. [24] M. S. Birman and M. Solomyak, Double Operators Integrals in a Hilbert Space, Integral Equations and Oper. Theory 47 (2003), 131–168. [25] T. Bonesky, K. Kazimierski, P. Maass, F. Schöpfer, and T. Schuster, Minimization of Tikhonov functionals in Banach spaces, Abstr. Appl. Anal. (2008), Art. ID. 192679. [26] S. Boucheron, O. Bousquet and G. Lugosi, Theory of classification: a survey of some recent advances, ESAIM Probab. Stat. 9 (2005), 323–375. [27] P. Brazdil, C. Giraud-Carrier, C. Soares, and R. Vilalta, Metalearning: Applications to Data Mining, Springer-Verlag, Berlin Heidelberg, 2009. [28] C. Brezinski, M. Redivo-Zaglia, G. Rodriguez, and S. Seatzu, Multi-parameter regularization techniques for ill-conditioned linear systems, Numer. Math. 94 (2003), 203–228. [29] D. H. Brooks, G. F. Ahmad, R. S. MacLeod, and M. G. Maratos, Inverse electrocardiography by simultaneous imposition of multiple constraints, IEEE Trans. Biomed. Eng. 46 (1999), 3–18. [30] M. Brühl, M. Hanke and M. Pidcock, Crack detection using electrostatic measurements, Math. Model. Numer. Anal. 35 (2001), 595–605. [31] M. Burger and S. Osher, Convergence rates of convex variational regularization, Inverse Problems 20 (2004), 1411–1421. [32] C. Calvetti and E. Somersalo, Statistical elimination of boundary artefacts in image deblurring, Inverse Problems 21 (2005), 1697–1714. [33] H. Cao and S. V. Pereverzev, Natural linearization for the identification of a diffusion coefficient in a quasi-linear parabolic system from short-time observations, Inverse Problems 22 (2006), 2311–2330. [34] H. Cao and S. V. Pereverzev, The balancing principle for the regularization of elliptic Cauchy problems, Inverse Problems 23 (2007), 1943–1961.

Bibliography

279

[35] H. Cao, S. V. Pereverzev, and E. Sincich, Natural linearization for corrosion identification, Journal of Physics: Conference Series 135 (2008), 012027. [36] A. Caponnetto and E. De Vito, Optimal rates for the regularized least-squares algorithm, Found. Comput. Math. 7 (2007), 331–368. [37] A. Chakib and A. Nachaoui, Convergence analysis for finite element approximation to an inverse Cauchy problem, Inverse Problems 22 (2006), 1191–1206. [38] O. Chapelle, V. Vapnik, O. Bousquet, and S. Mukherjee, Choosing multiple parameters for support vector machines, Machine Learning 46 (2002), 131–159. [39] Z. Chen, Y. Lu, Y. Xu, and Y. H., Multi-parameter Tikhonov regularization for linear ill-posed operator equations, J. Comp. Math. 26 (2008), 37–55. [40] W. L. Clarke, D. J. Cox, L. A. Gonder-Frederick, W. Carter, and S. L. Pohl, Evaluating clinical accuracy of systems for self-monitoring of blood glucose, Diabetes Care 5 (1987), 622–628. [41] T. M. Cover and J. A. Thomas, Elements of Information Theory, John Wiley & Sons Inc., New York, 1991. [42] F. Cucker and D. X. Zhou, Learning Theory: An Approximation Theory Viewpoint, Cambridge Monographs on Applied and Computational Mathematics, Cambridge University Press, Cambridge, 2007. [43] E. De Vito and A. Caponnetto, Optimal Rates for the Regularized Least-Squares Algorithm, Found. Comput. Math. 7 (2007), 331–368. [44] E. De Vito, S. Pereverzyev, and L. Rosasco, Adaptive kernel methods using the balancing principle, Found. Comput. Math. 10 (2010), 455–479. [45] E. De Vito, L. Rosasco, and A. Caponnetto, Discretization error analysis for Tikhonov regularization, Anal. Appl. (Singap.) 4 (2006), 81–99. [46] E. De Vito, L. Rosasco, A. Caponnetto, U. De Giovannini, and F. Odone, Learning from examples as an inverse problem, J. Mach. Learning Res. 6 (2005), 883–904. [47] L. Devroye, L. Györf, and G. Lugosi, A Probabilistic Theory of Pattern Recognition, Applications of Mathematics 31, Springer, New York, 1996. [48] T. F. Dolgopolova and V. K. Ivanov, On Numerical Differentiation, U.S.S.R. Comp. Math. and Math. Physics 6 (1966), 570–576. [49] D. Düvelmeyer and B. Hofmann, A multi-parameter regularization approach for estimating parameters in jump diffusion process, J. Inverse Ill-posed Probl. 14 (2006), 861–880. [50] L. Eldén, Solving quadratically constrained least squares problem using a differentialgeometric approach, BIT Numerical Mathematics 42 (2002), 323–335. [51] H. W. Engl, P. Fusek, and S. V. Pereverzev, Natural linearization for the identification of nonlinear heat transfer laws, J. Inverse Ill-Posed Probl. 13 (2005), 567–582.

280

Bibliography

[52] H. W. Engl, M. Hanke, and A. Neubauer, Regularization of Inverse Problems, Mathematics and Its Applications, Kluwer Academic Publishers, Dordrecht, 1996. [53] H. W. Engl and A. Leitão, A Mann iterative regularization method for elliptic Cauchy problems, Numer. Funct. Anal. Optim. 22 (2001), 861–884. [54] M. Eren-Oruklu, A. Cinar, L. Quinn, and D. Smith, Estimation of future glucose concentrations with subject-specific recursive linear models, Diabetes Technol. Ther. 11 (2009), 243–253. [55] T. Evgeniou, M. Pontil, and T. Poggio, Regularization networks and support vector machines, Adv. Comp. Math. 13 (2000), 1–50. [56] A. Facchinetti, G. Sparacino, and C. Cobelli, Reconstruction of glucose in plasma from interstitial fluid continuous glucose monitoring data: role of sensor calibration, J. Diabetes Sci. and Tech. 1 (2007), 617–623. [57] D. Fasino and G. Inglese, An inverse Robin problem for Laplace’s equation: theoretical results and numerical methods, Inverse Problems 15 (1999), 41–48. [58] J. Flemming, B. Hofmann, and P. Mathé, Sharp converse results for the regularization error using distance functions, Inverse Problems 27 (2011), 025006. [59] C. P. Franzone and E. Magenes, On the inverse potential problem of electrocardiology, Calcolo 16 (1979), 459–538. [60] W. Gander, Least squares with a quadratic constraint, Numer. Math. 36 (1981), 291–307. [61] A. Goldenshluger and S. V. Pereverzev, Adaptive estimation of linear functionals in Hilbert scales from indirect white noise observations, Probab. Theory Related Fields 118 (2000), 169–186. [62] G. H. Golub, P. C. Hansen, and D. P. O’Leary, Tikhonov regularization and total least squares, SIAM J. Matrix Anal. Appl. 21 (1999), 185–194. [63] G. H. Golub and U. von Matt, Quadratically constrained least squares and quadratic problems, Numer. Math. 59 (1991), 561–580. [64] T. Gomes, R. Prudencio, C. Soares, A. Rossi, and A. Carvalho, Combining metalearning and search techniques to select parameters for support vector machines, Neurocomputing 75 (2012), 3–13. [65] C. W. Groetsch, Differentiation of approximately specified functions, Amer. Math. Monthly 98 (1991), 847–850. [66] C. W. Groetsch and A. Neubauer, Convergence of a general projection method for an operator equation of the first kind, Houston J. Math. 14 (1988), 201–208. [67] C. Gu and G. Wahba, Minimizing GCV/GML scores with multiple smoothing parameter via the Newton method, SIAM J. Sci. Stat. Comput. 12 (1991), 383–398. [68] J. Hadamard, Le probléme de Cauchy et les équations aux dérivées partielles linéaires hyperboliques, p. 542, Hermann, Paris, 1932.

Bibliography

281

[69] H. Han and H. J. Reinhardt, Some stability estimates for Cauchy problems for elliptic equations, J. Inverse Ill-Posed Probl. 5 (1997), 437–454. [70] F. Hansen, Operator inequalities associated with Jensen’s inequality, in: T. M. Rassias (ed.), Survey on Classical Inequalities, pp. 67–98, Springer Science+Business Media Dordrecht, 2000. [71] P. C. Hansen, Regularization tools: a Matlab package for analysis and solution of discrete ill-posed problems, Numer. Algo. 6 (1994), 1–35. [72] M. Hegland, Variable Hilbert scales and their interpolation inequalities with application to Tikhonov regularization, Appl. Anal. 59 (1995), 207–223. [73] M. Hintermüller and K. Kunisch, Total Bounded Variation Regularization as a Bilaterally Constrained Optimization Problem, SIAM J. Appl. Math. 64 (2004), 1311–1333. [74] B. Hofmann, Regularization for Applied Inverse and Ill-posed Problems, Teubner, Leipzig, 1986. [75] B. Hofmann, B. Kaltenbacher, C. Pöschl and O. Scherzer, A convergence rates result for Tikhonov regularization in Banach spaces with non-smooth operators, Inverse Problems 23 (2007), 987–1010. [76] B. Hu and S. Lu, Numerical differentiation by a Tikhonov regularization method based on the discrete cosine transform, Appl. Anal. 91 (2012), 719–736. [77] V. A. Il’in and E. G. Poznyak, Foundations of Mathematical Analysis, Nauka, Moscow, 1973. [78] V. Isakov, Inverse Problems for Partial Differential Equations, 2nd edn., Springer, New York, 2006. [79] A. Iserles, A fast and simple algorithm for the computation of Legendre coefficients, Numer. Math. 117 (2011), 529–53. [80] V. K. Ivanov, The approximate solution of operator equations of the first kind, U.S.S.R. Comp. Math. Math. Phys. 6 (1966), 1089–1094. [81] V. K. Ivanov and T. I. Koroljuk, The estimation of errors in the solution of linear illposed problems, U.S.S.R. Comput. Math. and Math. Phys. 9 (1969), 30–41. [82] B. Jin and D. Lorenz, Heuristic Parameter-Choice Rules for Convex Variational Regularization Based on Error Estimates, SIAM J. Numer. Anal. 48 (2010), 1208–1229. [83] I. M. Johnstone and B. W. Silverman, Discretization effects in statistical inverse problems, J. Complexity 7 (1991), 1–34. [84] S. I. Kabanikhin, Inverse and Ill-posed Problems: Theory and Applications, De Gruyter, Berlin Boston, 2012. [85] B. Kaltenbacher, A. Neubauer, and O. Scherzer, Iterative Regularization Methods for Nonlinear Ill-Posed Problems, De Gruyter, Berlin, 2008. [86] G. Kimeldorf and G. Wahba, Some results on Tchebycheffian spline function, J. Math. Anal. Appl. 33 (1971), 82–95.

282

Bibliography

[87] S. Kindermann and A. Neubauer, On the convergence of the quasioptimality criterion for (iterated) Tikhonov regularization, Inverse Problems and Imaging 2 (2008), 291–299. [88] R. V. Kohn and M. Vogelius, Determining conductivity by boundary measurements. II. Interior results, Comm. Pure Appl. Math. 38 (1985), 643–667. [89] N. Korneichuk, Exact constants in approximation theory, translated from Russian by K. Ivanov, Encyclopedia of Mathematics and its Applications 38, Cambridge University Press, Cambridge, 1991. [90] T. V. Korsgaard, New ways to test beta cell functionality in health and diabetes, Ph.D. thesis, Technical University of Denmark, Copenhagen, 2011. [91] B. Kovatchev, S. Anderson, L. Heinemann, and W. Clarke, Comparison of the numerical and clinical accuracy of four continuous glucose monitors, Diabetes Care 31 (2008), 1160–1164. [92] B. Kovatchev and B. Clarke, Peculiarities of the Continuous Glucose Monitoring Data Stream and Their Impact on Developing Closed-Loop Control Technology, J. Diabetes Sci. Technol. 2 (2008), 158–163. [93] B. Kovatchev, D. Shields, and M. Breton, Graphical and numerical evaluation of continuous glucose sensing time lag, Diabetes Technol. Ther. 11 (2009), 139–143. [94] B. P. Kovatchev, M. Straume, D. J. Cox, and L. S. Farhy, Risk analysis of blood glucose data: a quantitative approach to optimizing the control of insulin dependent diabetes, J. Theor. Med. 3 (2000), 1–10. [95] P. Kügler and A. Leitão, Mean value iterations for nonlinear elliptic Cauchy problems, Numer. Math. 96 (2003), 269–293. [96] K. Kunisch and J. Zou, Iterative choices of regularization parameters in linear inverse problems, Inverse Problems 14 (1998), 1247–1264. [97] G. Lanckriet, N. Cristianini, P. Bartlett, L. El Ghaoui, and M. I. Jordan, Learning the kernel matrix with semidefinite programming, J. Mach. Learn. Res. 5 (2004), 27–72. [98] R. Lattès and J. L. Lions, Méthode de quasi-réversibilité et applications, Travaux et Recherches Mathématiques, Paris, 1967. [99] M. Ledoux and M. Talagrand, Probability in Banach spaces: isoperimetry and processes, Ergebnisse der Mathematik und ihrer Grenzgebiete 3, Springer-Verlag, Berlin, 1991. [100] A. Leitão, An iterative method for solving elliptic Cauchy problems, Numer. Funct. Anal. Optim. 21 (2000), 715–742. [101] O. Lepskii, A problem of adaptive estimation in Gaussian white noise, Theory Probab. Appl. 35 (1990), 454–466. [102] O. A. Liskovets, Theory and methods of solving ill-posed problems (in Russian), Mathematical Analysis 20 (1982), 116–178. [103] A. K. Louis, Inverse und schlecht gestellte Probleme, Teubner Studienbücher Mathematik, Stuttgart, 1989.

Bibliography

283

[104] S. Lu and S. V. Pereverzev, Multiparameter regularization in downward continuation of satellite data, in: W. Freeden, M. Z Nashed, and T. Sonar (eds.), Handbook of Geomathematics, Chap. 27, Springer-Verlag, Berlin, Heidelberg, 2010. [105] S. Lu and S. V. Pereverzev, Multi-parameter regularization and its numerical realization, Numer. Math. 118 (2011), 1–31. [106] S. Lu, S. V. Pereverzev, Y. Shao and U. Tautenhahn, Discrepancy curves for multiparameter regularization, J. Inv. Ill-Posed Problems 18 (2010), 655–676. [107] S. Lu, S. V. Pereverzev and U. Tautenhahn, Dual regularized total least squares and multi-parameter regularization, Comput. Methods Appl. Math. 8 (2008), 253–262. [108] S. Lu, S. V. Pereverzev and U. Tautenhahn, Regularized total least squares: computational aspects and error bounds, SIAM J. Matrix Anal. Appl. 31 (2009), 918–941. [109] S. Lu, S. V. Pereverzev and U. Tautenhahn, A model function method in regularized total least squares, Appl. Anal. 89 (2010), 1693–1703. [110] G. R. Luecke and K. R. Hickey, Convergence of approximate solutions of an operator equation, Houston J. Math. 11 (1985), 345–354. [111] B. A. Mair and F. H. Ruymgaart, Statistical inverse estimation in Hilbert scales, SIAM J. Appl. Math. 56 (1996), 1424–1444. [112] P. Mathé, Saturation of regularization methods for linear ill-posed problems in Hilbert spaces, SIAM J. Numer. Anal. 42 (2004), 968–973. [113] P. Mathé and B. Hofmann, How general are general source conditions?, Inverse Problems 24 (2008), 015009. [114] P. Mathé and S. V. Pereverzev, Stable summation of orthogonal series with noisy coefficients, J. Approx. Theory 118 (2002), 66–80. [115] C. Micchelli and M. Pontil, Learning the kernel function via regularization, J. Mach. Learn. Res. 6 (2005), 1099–1125. [116] C. A. Micchelli, Y. Xu, and H. Zhang, Universal Kernels, J. Mach. Learn. Res. 7 (2006), 2651–2667. [117] V. A. Morozov, On the solution of functional equations by the method of regularization, Soviet Math. Dokl. 7 (1966), 414–417. [118] C. Müller, Foundations of the Mathematical Theory of Electromagnetic Waves, Springer-Verlag, Berlin Heidelberg New York, 1969. [119] D. Mumford, Pattern Theory: The Mathematics of Perception, in: ICM, Vol. 1, pp. 401–422, High Ed. Press, Beijing, 2002. [120] M. T. Nair, S. V. Pereverzev, and U. Tautenhahn, Regularization in Hilbert scales under general smoothing conditions, Inverse Problems 21 (2005), 1851–1869. [121] F. Natterer, Error bounds for Tikhonov regularization in Hilbert scales, Appl. Anal. 18 (1984), 29–37.

284

Bibliography

[122] V. Naumova, S. V. Pereverzyev, and S. Sivananthan, Extrapolation in variable RKHSs with application to the blood glucose reading, Inverse Problems 27 (2011), 075010, 13 pp. [123] A. Neubauer, When do Sobolev spaces form a Hilbert scale?, Proc. Amer. Math. Soc. 103 (1988), 557–562. [124] M. Nussbaum and S. V. Pereverzev, The degree of ill-posedness in stochastic and deterministic noise models, preprint no. 509 (1999), WIAS-Berlin. [125] C. Palerm and B. Wayne Bequette, Hypoglycemia detection and prediction using continuous glucose monitoring – a study on hypoglycemic clamp data, J. Diabetes Sci. Technol. 1 (2007), 624–629. [126] S. Pappada, B. Cameron, and P. Rosman, Development of a neural network for prediction of glucose concentration in type 1 diabetes patients, J. Diabetes Sci. Technol. 2 (2008), 792–801. [127] S. Pappada, B. Cameron, P. Rosman, R. Bourey, T. Papadimos, W. Olorunto, and M. Borst, Neural networks-based real-time prediction of glucose in patients with insulin-dependent diabetes, Diabetes Technol. Ther. 13 (2011), 135–141. [128] S. V. Pereverzev and E. Schock, On the adaptive selection of the parameter in regularization of ill-posed problems, SIAM J. Numer. Anal. 43 (2005), 2060–2076. [129] S. V. Pereverzev and S. Sivananthan, Regularized learning algorithm for prediction of blood glucose concentration in “no action period”, in: 1st International Conference on Mathematical and Computational Biomedical Engineering – CMBE 2009, pp. 395– 398, Swansea, UK, 2009. [130] V. Naumova, S. V. Pereverzyev, and S. Sivananthan, A meta-learning approach to the regularized learning, Neural Networks 33 (2012), 181–193. [131] S. V. Pereverzyev, S. Sivananthan, J. Randlöv, and S. McKennoch, Glucose predictor based on regularization networks with adaptively chosen kernels and regularization parameters, patent publication WO 2012/143505 A2 filing date April 20, 2011. [132] C. Perez-Gandia, A. Facchinetti, G. Sparacino, C. Cobelli, E. J. Gomez, M. Rigla, A. deLeiva, and M. E. Hernando, Artificial neural network algorithm for online glucose prediction from continuous glucose monitoring, Diabetes Technol. Ther. 12 (2010), 81– 88. [133] P. O. Persson and G. Strang, Smoothing by Savitzky-Golay and Legendre filters, in: Mathematical systems theory in biology, communications, computation, and finance (Notre Dame, IN, 2002), IMA Vol. Math. Appl. 134 (2003), 301–315. [134] D. Phillips, A technique for the numerical solution of certain integral equation of the first kind, J. Assoc. Comput. Mach. 9 (1962), 84–97. [135] A. Pietsch, Operator Ideals, North-Holland Publishing, Amsterdam-New York, 1980. [136] T. Poggio and S. Smale, The mathematics of learning: dealing with data, Notices Amer. Math. Soc. 50 (2003), 537–544.

Bibliography

285

[137] R. Qu, A new approach to numerical differentiation and integration, Math. Comput. Modelling 24 (1996), 55–68. [138] S. Z. Rafal’son, The approximation of functions by Fourier-Jacobi sums (in Russian), Izv. Vysš. Uˇcebn. Zaved. Matematika 71 (1968), 54–62. [139] A. G. Ramm and A. B. Smirnova, On stable numerical differentiation, Math. Comp. 70 (2001), 1131–1153. [140] J. Reifman, S. Rajaraman, A. Gribok, and W. K. Ward, Predictive monitoring for improved management of glucose levels, J. Diabetes Sci. Technol. 1 (2007), 478–86. [141] R. A. Renaut and H. Guo, Efficient algorithms for solution of regularized total least squares, SIAM J. Matrix Anal. Appl. 26 (2004), 457–476. [142] E. Resmerita and O. Scherzer, Error estimates for non-quadratic regularization and the relation to enhancement, Inverse Problems 22 (2006), 801–814. [143] J. Riordon, E. Zubritsky, and A. Newman, Top 10 Articles. Analytical chemistry looks at 10 seminal papers, Analytical Chemistry 72 (2000), 324A–329A. [144] L. Rosasco, De Vito, E., and A. Verri, Spectral methods for regularization in learning theory, Technical Report DISI-TR-05-18 (2005), Retrievable at http://www.dima. unige.it/devito/pub_con/spectral.pdf, last accessed on 2013 April 13. [145] W. Rudin, Functional Analysis, International Series in Pure and Applied Mathematics, McGraw-Hill, Princeton, 1991. [146] A. Savitzky and M. J. E. Golay, Smoothing and Differentiation of Data by Simplified Least Squares Procedures, Analytical Chemistry 36 (1964), 1627–1639. [147] T. Schaul and J. Schmidhuber, Metalearning, Scholarpedia 5 (2010), 4650. [148] F. Schöpfer, A. K. Louis, and T. Schuster, Nonlinear iterative methods for linear illposed problems in Banach spaces, Inverse Problems 22 (2006), 311–329. [149] T. I. Seidman, Nonconvergence results for the application of least-squares estimation to ill-posed problems, J. Optim. Theory Appl. 30 (1980), 535–547. [150] D. Sima, S. Van Huffel, and G. H. Golub, Regularized total least squares based on quadratic eigenvalue problem solvers, BIT Numerical Mathematics 44 (2004), 793– 812. [151] S. Sivananthan, V. Naumova, C. Dalla Man, A. Facchinetti, E. Renard, C. Cobelli, and S. V. Pereverzyev, Assessment of Blood Glucose Predictors: The Prediction-Error Grid Analysis, Diabetes Technol. Ther. 13 (2011), 787–796. [152] S. Smale and D. Zhou, Learning theory estimates via integral operators and their approximations, Constr. Approx. 26 (2007), 153–172. [153] L. Snetselaar, Nutrition counseling skills for the nutrition care process, Jones and Bartlett Publishers, Sudbury, MA, 2009. [154] C. Soares, P. B. Brazdil, and P. Kuba, A Meta-learning approach to select the kernel width in support vector regression, Machine Learning 54 (2004), 195–209.

286

Bibliography

[155] V. Solo, Deterministic adaptive control with slowly varying parameters: an averaging analysis, Internat. J. Control 64 (1996), 99–125. [156] G. Sparacino, F. Zanderigo, S. Corazza, and A. Maran, Glucose concentration can be predicted ahead in time from continuous glucose monitoring sensor time-series, IEEE Trans. Biomed. Eng. 54 (2007), 931–937. [157] I. Steinwart and A. Christmann, Support Vector Machines, Information Sciences and Statistics, Springer, Berlin, 2008. [158] A. I. Stepanets, Classification and approximation of periodic functions, translated from the 1987 Russian original by P. V. Malyshev and D. V. Malyshev and revised by the author, Mathematics and its Applications 333, Kluwer Academic Publishers Group, Dordrecht, 1995. [159] A. N. Tikhonov, On the regularization of ill-posed problems, Dokl. Akad. Nauk SSSR (in Russian), 153 (1963), 49–52. [160] A. N. Tikhonov, On stable summation methods for Fourier series, Dokl. Akad. Nauk SSSR 156 (1964), 268–271. [161] A. N. Tikhonov and V. Y. Arsenin, Solutions of ill-posed problems. Translated from Russian. Preface by translation editor Fritz John, Scripta Series in Mathematics, V. H. Winston & Sons, Washington, D.C.: John Wiley & Sons, New York Toronto Ontario London, 1977. [162] A. N. Tikhonov and V. B. Glasko, Use of the regularization methods in non-linear problems, USSR Comput. Math. Math. Phys. 5 (1965), 93–107. [163] A. F. Timan, Theory of approximation of functions of a real variable, translated from Russian by J. Berry, English translation edited and editorial preface by J. Cossar, International Series of Monographs in Pure and Applied Mathematics 34, Pergamon Press Book, The Macmillan Co., New York, 1963. [164] N. G. Tomin, An application of the interpolation of linear operators to questions of the convergence of series of Fourier coefficients with respect to classical orthogonal polynomials (in Russian), Dokl. Akad. Nauk SSSR 212 (1973), 1074–1077. [165] A. B. Tsybakov, Optimal aggregation of classifiers in statistical learning, Ann. Statist. 32 (2004), 135–166. [166] A. Tsybakov, Pointwise and sup-norm sharp adaptive estimation of functions on the Sobolev classes, Ann. Statist. 26 (1998), 2420–2469. [167] G. M. Vainikko and A. Y. Veretennikov, Iteration Procedures in Ill-Posed Problems, (in Russian) Nauka, Moscow, 1986. [168] S. Van. Huffel and J. Vanderwalle, The Total Least Squares Problem. Computational Aspects and Analysis, SIAM, Philadelphia, 1991. [169] V. N. Vapnik, Statistical Learning Theory (Adaptive and Learning Systems for Signal Processing, Communications, and Control), Wiley, New York, 1998.

Bibliography

287

[170] G. Wahba, Practical approximate solutions to linear operator equations when the data are noisy, SIAM J. Numer. Anal. 14 (1977), 651–667. [171] G. Wahba, Splines Methods for Observational Data, Series in Applied Mathematics 59, SIAM, Philadelphia, 1990. [172] S. D. Wicksell, The corpuscle problem I. Case of spherical corpuscle, Biometrika 17 (1925), 84–89. [173] J. Xie and J. Zou, An improved model function method for choosing regularization parameters in linear inverse problems, Inverse Problems 18 (2002), 631–643. [174] P. L. Xu, Y. Fukuda, and Y. M. Liu, Multiple parameter regularization: numerical solutions and applications to the determination of geopotential from precise satellite orbits, J. Geod. 80 (2006), 17–27. [175] Y. Yao, L. Rosasco and A. Caponnetto, On early stopping in gradient descent learning, Constr. Approx. 26 (2007), 289–315. [176] Z. Zhao, A truncated Legendre spectral method for solving numerical differentiation, Int. J. Comput. Math. 87 (2010), 3209–3217.

Index

balancing principle, 11 best possible accuracy, 6, 7, 11 binary classification, 217 Cauchy problem, 25 compactness, 50 concentration inequalities, 18 covariance operator, 16, 207 2 -condition, 72 design, 127 discrepancy curve, 173 dual regularized total least squares, 191 finite-difference formulae, 2 Hilbert scales, 92 inclusion operator, 207 index function, 18 Iterated Tikhonov regularization, 62 kernel-type operators, 114 Kolmogorov’s n-width, 126 -methods, 13 least-squares solution, 51 Legendre polynomials, 19 model function, 169 modulus of continuity, 3, 69 multiparameter discrepancy principle, 166

numerical differentiation, 1 one-half condition, 59 operator monotone functions, 132 optimal bound, 35 Peierls–Bogoliubov Inequality, 86 Picard criterion, 53 prior, 208 qualification, 19, 58, 80 random process, 16 regularization, 80 regularization parameter, 56 regularized total least squares method, 190 Reproduction Kernel Hilbert Space (RKHS), 205 risk, 18, 204 saturation, 85 Savitzky–Golay technique, 156 singular value expansion, 36, 50 source condition, 68 Spectral cut-off method, 59 Tikhonov–Phillips regularization, 61 total least squares method, 190 trigonometric system, 13 worst-case error, 66