Mathematical Analysis for Machine Learning and Data Mining 9813229683, 9789813229686

Table of contents :
Cover
Title Page
Copyright
Epigraph
Preface
Contents
Part I: Set-Theoretical and Algebraic
Preliminaries
1 Preliminaries
2 Linear Spaces
3 Algebra of Convex Sets
Part II: Topology
4 Topology
5 Metric Space Topologies
6 Topological Linear Spaces
Part III: Measure and Integration
7 Measurable Spaces and Measures
8 Integration
Part IV: Functional Analysis and Convexity
9 Banach Spaces
10 Differentiability of Functions Defined
on Normed Spaces
11 Hilbert Spaces
12 Convex Functions
Part V: Applications
13 Optimization
14 Iterative Algorithms
15 Neural Networks
16 Regression
17 Support Vector Machines
Bibliography
Index

Citation preview

10702_9789813229686_tp.indd 1

2/5/18 10:45 AM

b2530   International Strategic Relations and China’s National Security: World at the Crossroads

This page intentionally left blank

b2530_FM.indd 6

01-Sep-16 11:03:06 AM

10702_9789813229686_tp.indd 2

2/5/18 10:45 AM

Published by World Scientific Publishing Co. Pte. Ltd. 5 Toh Tuck Link, Singapore 596224 USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE

Library of Congress Cataloging-in-Publication Data Names: Simovici, Dan A., author. Title: Mathematical analysis for machine learning and data mining / by Dan Simovici (University of Massachusetts, Boston, USA). Description: [Hackensack?] New Jersey : World Scientific, [2018] | Includes bibliographical references and index. Identifiers: LCCN 2018008584 | ISBN 9789813229686 (hc : alk. paper) Subjects: LCSH: Machine learning--Mathematics. | Data mining--Mathematics. Classification: LCC Q325.5 .S57 2018 | DDC 006.3/101515--dc23 LC record available at https://lccn.loc.gov/2018008584

British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library.

Copyright © 2018 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the publisher.

For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher. For any available supplementary material, please visit http://www.worldscientific.com/worldscibooks/10.1142/10702#t=suppl Desk Editors: V. Vishnu Mohan/Steven Patt Typeset by Stallion Press Email: [email protected] Printed in Singapore

Vishnu Mohan - 10702 - Mathematical Analysis for Machine Learning.indd 1

23-04-18 2:48:01 PM

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page v

v

Making mathematics accessible to the educated layman, while keeping high scientific standards, has always been considered a treacherous navigation between the Scylla of professional contempt and the Charybdis of public misunderstanding.

Gian-Carlo Rota

b2530   International Strategic Relations and China’s National Security: World at the Crossroads

This page intentionally left blank

b2530_FM.indd 6

01-Sep-16 11:03:06 AM

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page vii

Preface

Mathematical Analysis can be loosely described as is the area of mathematics whose main object is the study of function and of their behaviour with respect to limits. The term “function” refers to a broad collection of generalizations of real functions of real arguments, to functionals, operators, measures, etc. There are several well-developed areas in mathematical analysis that present a special interest for machine learning: topology (with various flavors: point-set topology, combinatorial and algebraic topology), functional analysis on normed and inner product spaces (including Banach and Hilbert spaces), convex analysis, optimization, etc. Moreover, disciplines like measure and integration theory which play a vital role in statistics, the other pillar of machine learning are absent from the education of a computer scientists. We aim to contribute to closing this gap, which is a serious handicap for people interested in research. The machine learning and data mining literature is vast and embraces a diversity of approaches, from informal to sophisticated mathematical presentations. However, the necessary mathematical background needed for approaching research topics is usually presented in a terse and unmotivated manner, or is simply absent. This volume contains knowledge that complements the usual presentations in machine learning and provides motivations (through its application chapters that discuss optimization, iterative algorithms, neural networks, regression, and support vector machines) for the study of mathematical aspects. Each chapter ends with suggestions for further reading. Over 600 exercises and supplements are included; they form an integral part of the material. Some of the exercises are in reality supplemental material. For these, we include solutions. The mathematical background required for

vii

May 2, 2018 11:28

viii

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page viii

Mathematical Analysis for Machine Learning and Data Mining

making the best use of this volume consists in the typical sequence calculus — linear algebra — discrete mathematics, as it is taught to Computer Science students in US universities. Special thanks are due to the librarians of the Joseph Healy Library at the University of Massachusetts Boston whose diligence was essential in completing this project. I also wish to acknowledge the helpfulness and competent assistance of Steve Patt and D. Rajesh Babu of World Scientific. Lastly, I wish to thank my wife, Doina, a steady source of strength and loving support.

Dan A. Simovici Boston and Brookline January 2018

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page ix

Contents

Preface

vii

Part I. Set-Theoretical and Algebraic Preliminaries 1. Preliminaries 1.1 Introduction . . . . . . . . . . . . 1.2 Sets and Collections . . . . . . . 1.3 Relations and Functions . . . . . 1.4 Sequences and Collections of Sets 1.5 Partially Ordered Sets . . . . . . 1.6 Closure and Interior Systems . . 1.7 Algebras and σ-Algebras of Sets 1.8 Dissimilarity and Metrics . . . . 1.9 Elementary Combinatorics . . . . Exercises and Supplements . . . . . . . Bibliographical Comments . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

2. Linear Spaces 2.1 Introduction . . . . . . . . . . . . . . . . . . 2.2 Linear Spaces and Linear Independence . . 2.3 Linear Operators and Functionals . . . . . . 2.4 Linear Spaces with Inner Products . . . . . 2.5 Seminorms and Norms . . . . . . . . . . . . 2.6 Linear Functionals in Inner Product Spaces 2.7 Hyperplanes . . . . . . . . . . . . . . . . . . Exercises and Supplements . . . . . . . . . . . . . Bibliographical Comments . . . . . . . . . . . . . ix

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

1

. . . . . . . . . . .

3 3 4 8 16 18 28 34 43 47 54 64

. . . . . . . . .

65 65 65 74 85 88 107 110 113 116

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page x

Mathematical Analysis for Machine Learning and Data Mining

x

3. Algebra of Convex Sets 3.1 Introduction . . . . . . . . . . . . . 3.2 Convex Sets and Affine Subspaces 3.3 Operations on Convex Sets . . . . 3.4 Cones . . . . . . . . . . . . . . . . 3.5 Extreme Points . . . . . . . . . . . 3.6 Balanced and Absorbing Sets . . . 3.7 Polytopes and Polyhedra . . . . . . Exercises and Supplements . . . . . . . . Bibliographical Comments . . . . . . . .

117 . . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

Part II. Topology

159

4. Topology 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . 4.2 Topologies . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Closure and Interior Operators in Topological Spaces 4.4 Neighborhoods . . . . . . . . . . . . . . . . . . . . . 4.5 Bases . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Compactness . . . . . . . . . . . . . . . . . . . . . . 4.7 Separation Hierarchy . . . . . . . . . . . . . . . . . . 4.8 Locally Compact Spaces . . . . . . . . . . . . . . . . 4.9 Limits of Functions . . . . . . . . . . . . . . . . . . . 4.10 Nets . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.11 Continuous Functions . . . . . . . . . . . . . . . . . 4.12 Homeomorphisms . . . . . . . . . . . . . . . . . . . . 4.13 Connected Topological Spaces . . . . . . . . . . . . . 4.14 Products of Topological Spaces . . . . . . . . . . . . 4.15 Semicontinuous Functions . . . . . . . . . . . . . . . 4.16 The Epigraph and the Hypograph of a Function . . Exercises and Supplements . . . . . . . . . . . . . . . . . . Bibliographical Comments . . . . . . . . . . . . . . . . . . 5. Metric Space Topologies 5.1 5.2 5.3

117 117 129 130 132 138 142 150 158

161 . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

161 162 166 174 180 189 193 197 201 204 210 218 222 225 230 237 239 253 255

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 255 Sequences in Metric Spaces . . . . . . . . . . . . . . . . . 260 Limits of Functions on Metric Spaces . . . . . . . . . . . . 261

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Contents

xi

5.4 5.5 5.6 5.7 5.8 5.9 5.10 5.11

Continuity of Functions between Metric Spaces Separation Properties of Metric Spaces . . . . . Completeness of Metric Spaces . . . . . . . . . Pointwise and Uniform Convergence . . . . . . The Stone-Weierstrass Theorem . . . . . . . . . Totally Bounded Metric Spaces . . . . . . . . . Contractions and Fixed Points . . . . . . . . . The Hausdorff Metric Hyperspace of Compact Subsets . . . . . . . . . . . . . . . . . . . . . . 5.12 The Topological Space (R, O) . . . . . . . . . . 5.13 Series and Schauder Bases . . . . . . . . . . . . 5.14 Equicontinuity . . . . . . . . . . . . . . . . . . Exercises and Supplements . . . . . . . . . . . . . . . Bibliographical Comments . . . . . . . . . . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

264 270 275 283 286 291 295

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

300 303 307 315 318 327

6. Topological Linear Spaces

329

6.1 Introduction . . . . . . . . . . . . . . . . . . 6.2 Topologies of Linear Spaces . . . . . . . . . 6.3 Topologies on Inner Product Spaces . . . . 6.4 Locally Convex Linear Spaces . . . . . . . . 6.5 Continuous Linear Operators . . . . . . . . 6.6 Linear Operators on Normed Linear Spaces 6.7 Topological Aspects of Convex Sets . . . . . 6.8 The Relative Interior . . . . . . . . . . . . . 6.9 Separation of Convex Sets . . . . . . . . . . 6.10 Theorems of Alternatives . . . . . . . . . . 6.11 The Contingent Cone . . . . . . . . . . . . 6.12 Extreme Points and Krein-Milman Theorem Exercises and Supplements . . . . . . . . . . . . . Bibliographical Comments . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

Part III. Measure and Integration

Introduction . . . . . . Measurable Spaces . . Borel Sets . . . . . . . Measurable Functions

. . . .

. . . .

329 329 337 338 340 341 348 351 356 366 370 373 375 381

383

7. Measurable Spaces and Measures 7.1 7.2 7.3 7.4

page xi

385 . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

385 385 388 392

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page xii

Mathematical Analysis for Machine Learning and Data Mining

xii

7.5 Measures and Measure Spaces . 7.6 Outer Measures . . . . . . . . . 7.7 The Lebesgue Measure on Rn . 7.8 Measures on Topological Spaces 7.9 Measures in Metric Spaces . . . 7.10 Signed and Complex Measures 7.11 Probability Spaces . . . . . . . Exercises and Supplements . . . . . . Bibliographical Comments . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

8. Integration 8.1 8.2

Introduction . . . . . . . . . . . . . . . . . . . . . . . The Lebesgue Integral . . . . . . . . . . . . . . . . . 8.2.1 The Integral of Simple Measurable Functions 8.2.2 The Integral of Non-negative Measurable Functions . . . . . . . . . . . . . . . . . . . . 8.2.3 The Integral of Real-Valued Measurable Functions . . . . . . . . . . . . . . . . . . . . 8.2.4 The Integral of Complex-Valued Measurable Functions . . . . . . . . . . . . . . . . . . . . 8.3 The Dominated Convergence Theorem . . . . . . . . 8.4 Functions of Bounded Variation . . . . . . . . . . . . 8.5 Riemann Integral vs. Lebesgue Integral . . . . . . . 8.6 The Radon-Nikodym Theorem . . . . . . . . . . . . 8.7 Integration on Products of Measure Spaces . . . . . 8.8 The Riesz-Markov-Kakutani Theorem . . . . . . . . 8.9 Integration Relative to Signed Measures and Complex Measures . . . . . . . . . . . . . . . . . . . 8.10 Indefinite Integral of a Function . . . . . . . . . . . . 8.11 Convergence in Measure . . . . . . . . . . . . . . . . 8.12 Lp and Lp Spaces . . . . . . . . . . . . . . . . . . . . 8.13 Fourier Transforms of Measures . . . . . . . . . . . . 8.14 Lebesgue-Stieltjes Measures and Integrals . . . . . . 8.15 Distributions of Random Variables . . . . . . . . . . 8.16 Random Vectors . . . . . . . . . . . . . . . . . . . . Exercises and Supplements . . . . . . . . . . . . . . . . . . Bibliographical Comments . . . . . . . . . . . . . . . . . .

398 417 427 450 453 456 464 470 484 485

. . . 485 . . . 485 . . . 486 . . . 491 . . . 500 . . . . . . .

. . . . . . .

. . . . . . .

505 508 512 517 525 533 540

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

547 549 551 556 565 569 572 577 582 593

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Contents

xiii

Part IV. Functional Analysis and Convexity

595

9. Banach Spaces

597

9.1 Introduction . . . . . . . . . . . . . . . . . . . . 9.2 Banach Spaces — Examples . . . . . . . . . . . 9.3 Linear Operators on Banach Spaces . . . . . . 9.4 Compact Operators . . . . . . . . . . . . . . . 9.5 Duals of Normed Linear Spaces . . . . . . . . . 9.6 Spectra of Linear Operators on Banach Spaces Exercises and Supplements . . . . . . . . . . . . . . . Bibliographical Comments . . . . . . . . . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

10. Differentiability of Functions Defined on Normed Spaces 10.1 Introduction . . . . . . . . . . . . . . . . . . 10.2 The Fr´echet and Gˆateaux Differentiation . . 10.3 Taylor’s Formula . . . . . . . . . . . . . . . 10.4 The Inverse Function Theorem in Rn . . . . 10.5 Normal and Tangent Subspaces for Surfaces Exercises and Supplements . . . . . . . . . . . . . Bibliographical Comments . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . in Rn . . . . . . . .

Introduction . . . . . . . . . . . . . . . . . . . . Hilbert Spaces — Examples . . . . . . . . . . . Classes of Linear Operators in Hilbert Spaces . 11.3.1 Self-Adjoint Operators . . . . . . . . . 11.3.2 Normal and Unitary Operators . . . . . 11.3.3 Projection Operators . . . . . . . . . . 11.4 Orthonormal Sets in Hilbert Spaces . . . . . . 11.5 The Dual Space of a Hilbert Space . . . . . . . 11.6 Weak Convergence . . . . . . . . . . . . . . . . 11.7 Spectra of Linear Operators on Hilbert Spaces 11.8 Functions of Positive and Negative Type . . . . 11.9 Reproducing Kernel Hilbert Spaces . . . . . . . 11.10 Positive Operators in Hilbert Spaces . . . . . . Exercises and Supplements . . . . . . . . . . . . . . . Bibliographical Comments . . . . . . . . . . . . . . .

597 597 603 610 612 616 619 623 625

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

11. Hilbert Spaces 11.1 11.2 11.3

page xiii

625 625 649 658 663 666 675 677

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

. . . . . . . . . . . . . . .

677 677 679 681 683 684 686 703 704 707 712 722 733 736 745

May 2, 2018 11:28

xiv

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page xiv

Mathematical Analysis for Machine Learning and Data Mining

12. Convex Functions

747

12.1 Introduction . . . . . . . . . . . . . . . . . . . 12.2 Convex Functions — Basics . . . . . . . . . . 12.3 Constructing Convex Functions . . . . . . . . 12.4 Extrema of Convex Functions . . . . . . . . . 12.5 Differentiability and Convexity . . . . . . . . 12.6 Quasi-Convex and Pseudo-Convex Functions 12.7 Convexity and Inequalities . . . . . . . . . . . 12.8 Subgradients . . . . . . . . . . . . . . . . . . Exercises and Supplements . . . . . . . . . . . . . . Bibliographical Comments . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

Part V. Applications

817

13. Optimization

819

13.1 Introduction . . . . . . . . . . . . . . . . . . . . 13.2 Local Extrema, Ascent and Descent Directions 13.3 General Optimization Problems . . . . . . . . . 13.4 Optimization without Differentiability . . . . . 13.5 Optimization with Differentiability . . . . . . . 13.6 Duality . . . . . . . . . . . . . . . . . . . . . . 13.7 Strong Duality . . . . . . . . . . . . . . . . . . Exercises and Supplements . . . . . . . . . . . . . . . Bibliographical Comments . . . . . . . . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

14. Iterative Algorithms 14.1 Introduction . . . . . . . . . . . . . . 14.2 Newton’s Method . . . . . . . . . . . 14.3 The Secant Method . . . . . . . . . 14.4 Newton’s Method in Banach Spaces 14.5 Conjugate Gradient Method . . . . . 14.6 Gradient Descent Algorithm . . . . . 14.7 Stochastic Gradient Descent . . . . . Exercises and Supplements . . . . . . . . . Bibliographical Comments . . . . . . . . .

747 748 756 759 760 770 775 780 793 815

819 819 826 827 831 843 849 854 863 865

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

865 865 869 871 874 879 882 884 892

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Contents

xv

15. Neural Networks

893

15.1 Introduction . . . . . . . . . . . . . . . . . . . 15.2 Neurons . . . . . . . . . . . . . . . . . . . . . 15.3 Neural Networks . . . . . . . . . . . . . . . . 15.4 Neural Networks as Universal Approximators 15.5 Weight Adjustment by Back Propagation . . Exercises and Supplements . . . . . . . . . . . . . . Bibliographical Comments . . . . . . . . . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

16. Regression

893 893 895 896 899 902 907 909

16.1 Introduction . . . . . . . . . . . . . . . . 16.2 Linear Regression . . . . . . . . . . . . . 16.3 A Statistical Model of Linear Regression 16.4 Logistic Regression . . . . . . . . . . . . 16.5 Ridge Regression . . . . . . . . . . . . . 16.6 Lasso Regression and Regularization . . Exercises and Supplements . . . . . . . . . . . Bibliographical Comments . . . . . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

17. Support Vector Machines 17.1 Introduction . . . . . . . . . . . . . . 17.2 Linearly Separable Data Sets . . . . 17.3 Soft Support Vector Machines . . . . 17.4 Non-linear Support Vector Machines 17.5 Perceptrons . . . . . . . . . . . . . . Exercises and Supplements . . . . . . . . . Bibliographical Comments . . . . . . . . .

page xv

909 909 912 914 916 917 920 924 925

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

925 925 930 933 939 941 947

Bibliography

949

Index

957

PART I

Set-Theoretical and Algebraic Preliminaries

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 3

Chapter 1

Preliminaries

1.1

Introduction

This introductory chapter contains a mix of preliminary results and notations that we use in further chapters, ranging from set theory, and combinatorics to metric spaces. The membership of x in a set S is denoted by x ∈ S; if x is not a member of the set S, we write x ∈ S. Throughout this book, we use standardized notations for certain important sets of numbers: C R0 R0 ˆ C ˆ R ˆ 0 R Q Z

R R>0

the set of complex numbers the set of non-negative real numbers the set of non-positive real numbers the set C ∪ {∞} the set R ∪ {−∞, +∞} the set R0 ∪ {−∞} the set of rational numbers the set of integers

R0 R I N

the set of real numbers the set of positive real numbers the set of negative real numbers the the the the

set set set set

R0 ∪ {+∞} R0 ∪ {+∞} of irrational numbers of natural numbers

ˆ by −∞ < x < The usual order of real numbers is extended to the set R +∞ for every x ∈ R. Addition and multiplication are extended by x + ∞ = ∞ + x = +∞, and , x − ∞ = −∞ + x = −∞, for every x ∈ R. Also, if x = 0 we assume that  +∞ if x > 0, x·∞ =∞·x = −∞ if x < 0, 3

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 4

Mathematical Analysis for Machine Learning and Data Mining

4

and

 x · (−∞) = (−∞) · x =

−∞ if x > 0, ∞

if x < 0.

Additionally, we assume that 0 ·∞ = ∞·0 = 0 and 0 ·(−∞) = (−∞)·0 = 0. Note that ∞ − ∞, −∞ + ∞ are undefined. Division is extended by x/∞ = x/ − ∞ = 0 for every x ∈ R. The set of complex numbers C is extended by adding a single “infinity” element ∞. The sum ∞ + ∞ is not defined in the complex case. If S is a finite set, we denote by |S| the number of elements of S. 1.2

Sets and Collections

We assume that the reader is familiar with elementary set operations: union, intersection, difference, etc., and with their properties. The empty set is denoted by ∅. We give, without proof, several properties of union and intersection of sets: (1) S ∪ (T ∪ U ) = (S ∪ T ) ∪ U (associativity of union), (2) S ∪ T = T ∪ S (commutativity of union), (3) S ∪ S = S (idempotency of union), (4) S ∪ ∅ = S, (5) S ∩ (T ∩ U ) = (S ∩ T ) ∩ U (associativity of intersection), (6) S ∩ T = T ∩ S (commutativity of intersection), (7) S ∩ S = S (idempotency of intersection), (8) S ∩ ∅ = ∅, for all sets S, T, U . The associativity of union and intersection allows us to denote unambiguously the union of three sets S, T, U by S ∪ T ∪ U and the intersection of three sets S, T, U by S ∩ T ∩ U . Definition 1.1. The sets S and T are disjoint if S ∩ T = ∅. Sets may contain other sets as elements. For example, the set C = {∅, {0}, {0, 1}, {0, 2}, {1, 2, 3}} contains the empty set ∅ and {0}, {0, 1},{0, 2},{1, 2, 3} as its elements. We refer to such sets as collections of sets or simply collections. In general, we use calligraphic letters C, D, . . . to denote collections of sets.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Preliminaries

page 5

5

If C and D are two collections, we say that C is included in D, or that C is a subcollection of D, if every member of C is a member of D. This is denoted by C ⊆ D. Two collections C and D are equal if we have both C ⊆ D and D ⊆ C. This is denoted by C = D. Definition 1.2. Let C be a collection of sets. The union of C, denoted by  C, is the set defined by  C = {x | x ∈ S for some S ∈ C}.  If C is a non-empty collection, its intersection is the set C given by  C = {x | x ∈ S for every S ∈ C}.  If C = {S, T }, we have x ∈ C if and only if x ∈ S or x ∈ T and  x ∈ C if and only if x ∈ S and y ∈ T . The union and the intersection of this two-set collection are denoted by S ∪ T and S ∩ T and are referred to as the union and the intersection of S and T , respectively. The difference of two sets S, T is denoted by S − T . When T is a subset of S we write T for S − T , and we refer to the set T as the complement of T with respect to S or simply the complement of T . The relationship between set difference and set union and intersection is well-known: for every set S and non-empty collection C of sets, we have     S − C = {S − C | C ∈ C} and S − C = {S − C | C ∈ C}. For any sets S, T, U , we have S − (T ∪ U ) = (S − T ) ∩ (S − U ) and S − (T ∩ U ) = (S − T ) ∪ (S − U ). With the notation previously introduced for the complement of a set, the above equalities become: T ∪ U = T ∩ U and T ∩ U = T ∪ U . For any sets T , U , V , we have (U ∪ V ) ∩ T = (U ∩ T ) ∪ (V ∩ T ) and (U ∩ V ) ∪ T = (U ∪ T ) ∩ (V ∪ T ). Note that if C and D are two collections such that C ⊆ D, then     C⊆ D and D⊆ C. We initially excluded the empty collection from the definition of the intersection of a collection. However, within the framework of collections of subsets of a given set S, we extend the previous definition by taking

May 2, 2018 11:28

6

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 6

Mathematical Analysis for Machine Learning and Data Mining



∅ = S for the empty collection of subsets of S. This is consistent with  the fact that ∅ ⊆ C implies C ⊆ S. The symmetric difference of sets denoted by ⊕ is defined by U ⊕ V = (U − V ) ∪ (V − U ) for all sets U, V . We leave to the reader to verify that for all sets U, V, T we have (i) U ⊕ U = ∅; (ii) U ⊕ V = V ⊕ T ; (iii) (U ⊕ V ) ⊕ T = U ⊕ (V ⊕ T ). The next theorem allows us to introduce a type of set collection of fundamental importance. Theorem 1.1. Let {{x, y}, {x}} and {{u, v}, {u}} be two collections such that {{x, y}, {x}} = {{u, v}, {u}}. Then, we have x = u and y = v. Proof. Suppose that {{x, y}, {x}} = {{u, v}, {u}}. If x = y, the collection {{x, y}, {x}} consists of a single set, {x}, so the collection {{u, v}, {u}} also consists of a single set. This means that {u, v} = {u}, which implies u = v. Therefore, x = u, which gives the desired conclusion because we also have y = v. If x = y, then neither (x, y) nor (u, v) are singletons. However, they both contain exactly one singleton, namely {x} and {u}, respectively, so x = u. They also contain the equal sets {x, y} and {u, v}, which must be equal. Since v ∈ {x, y} and v = u = x, we conclude that v = y.  Definition 1.3. An ordered pair is a collection of sets {{x, y}, {x}}. Theorem 1.1 implies that for an ordered pair {{x, y}, {x}}, x and y are uniquely determined. This justifies the following definition. Definition 1.4. Let {{x, y}, {x}} be an ordered pair. Then x is the first component of p and y is the second component of p. From now on, an ordered pair {{x, y}, {x}} is denoted by (x, y). If both x, y ∈ S, we refer to (x, y) as an ordered pair on the set S. Definition 1.5. Let X, Y be two sets. Their product is the set X × Y that consists of all pairs of the form (x, y), where x ∈ X and y ∈ Y . The set product is often referred to as the Cartesian product of sets. Example 1.1. Let X = {a, b, c} and let Y = {1, 2}. The Cartesian product X × Y is given by X × Y = {(a, 1), (b, 1), (c, 1), (a, 2), (b, 2), (c, 2)}.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Preliminaries

page 7

7

 Definition 1.6. Let C and D be two collections of sets such that C =  D. D is a refinement of C if, for every D ∈ D, there exists C ∈ C such that D ⊆ C. This is denoted by C D. Example 1.2. Consider the collection C = {(a, ∞) | a ∈ R} and D =   {(a, b) | a, b ∈ R, a < b}. It is clear that C = D = R. Since we have (a, b) ⊆ (a, ∞) for every a, b ∈ R such that a < b, it follows that D is a refinement of C. Definition 1.7. A collection of sets C is hereditary if U ∈ C and W ⊆ U implies W ∈ C. Example 1.3. Let S be a set. The collection of subsets of S, denoted by P(S), is a hereditary collection of sets since a subset of a subset T of S is itself a subset of S. The set of subsets of S that contain k elements is denoted by Pk (S). Clearly, for every set S, we have P0 (S) = {∅} because there is only one subset of S that contains 0 elements, namely the empty set. The set of all finite subsets of a set S is denoted by Pfin (S). It is clear that Pfin (S) =  k∈N Pk (S). Example 1.4. If S = {a, b, c}, then P(S) consists of the following eight sets: ∅, {a}, {b}, {c}, {a, b}, {a, c}, {b, c}, {a, b, c}. For the empty set, we have P(∅) = {∅}. Definition 1.8. Let C be a collection of sets and let U be a set. The trace of the collection C on the set U is the collection CU = {U ∩ C | C ∈ C}. We conclude this presentation of collections of sets with two more operations on collections of sets. Definition 1.9. Let C and D be two collections of sets. The collections C ∨ D, C ∧ D, and C − D are given by C ∨ D = {C ∪ D | C ∈ C and D ∈ D}, C ∧ D = {C ∩ D | C ∈ C and D ∈ D}, C − D = {C − D | C ∈ C and D ∈ D}. Example 1.5. Let C and D be the collections of sets defined by C = {{x}, {y, z}, {x, y}, {x, y, z}}, D = {{y}, {x, y}, {u, y, z}}.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 8

Mathematical Analysis for Machine Learning and Data Mining

8

We have C ∨ D = {{x, y}, {y, z}, {x, y, z}, {u, y, z}, {u, x, y, z}}, C ∧ D = {∅, {x}, {y}, {x, y}, {y, z}}, C − D = {∅, {x}, {z}, {x, z}}, D − C = {∅, {u}, {x}, {y}, {u, z}, {u, y, z}}. Unlike “∪” and “∩”, the operations “∨” and “∧” between collections of sets are not idempotent. Indeed, we have, for example, D ∨ D = {{y}, {x, y}, {u, y, z}, {u, x, y, z}} = D. The trace CK of a collection C on K can be written as CK = C ∧ {K}. We conclude this section by introducing a special type of collection of subsets of a set. Definition 1.10. A partition of a non-empty set S is a collection π of non-empty subsets of S that are pairwise disjoint and whose union equals S. The members of π are referred to as the blocks of the partition π. The collection of partitions of a set S is denoted by PART(S). A partition is finite if it has a finite number of blocks. The set of finite partitions of S is denoted by PARTfin (S). If π ∈ PART(S) then a subset T of S is π-saturated if it is a union of blocks of π. Example 1.6. Let π = {{1, 3}, {4}, {2, 5, 6}} be a partition of S = {1, 2, 3, 4, 5, 6}. The set {1, 3, 4} is π-saturated because it is the union of blocks {1, 3} and 4. 1.3

Relations and Functions

Definition 1.11. Let X, Y be two sets. A relation on X, Y is a subset ρ of the set product X × Y . If X = Y = S we refer to ρ as a relation on S. The • • •

relation ρ on S is: reflexive if (x, x) ∈ ρ for every x ∈ S; irreflexive if (x, x) ∈ ρ for every x ∈ S; symmetric if (x, y) ∈ ρ implies (y, x) ∈ ρ for all x, y ∈ S;

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Preliminaries

9in x 6in

b3234-main

page 9

9

• antisymmetric if (x, y) ∈ ρ and (y, x) ∈ ρ imply x = y for all x, y ∈ S; • transitive if (x, y) ∈ ρ and (y, z) ∈ ρ imply (x, z) ∈ ρ for all x, y, z ∈ S. Denote by REFL(S), SYMM(S), ANTISYMM(S) and TRAN(S) the sets of reflexive relations, the set of symmetric relations, the set of antisymmetric, and the set of transitive relations on S, respectively. A partial order on S is a relation ρ that belongs to REFL(S) ∩ ANTISYMM(S) ∩ TRAN(S), that is, a relation that is reflexive, symmetric and transitive. Example 1.7. Let δ be the relation that consists of those pairs (p, q) of natural numbers such that q = pk for some natural number k. We have (p, q) ∈ δ if p evenly divides q. Since (p, p) ∈ δ for every p it is clear that δ is symmetric. Suppose that we have both (p, q) ∈ δ and (q, p) ∈ δ. Then q = pk and p = qh. If either p or q is 0, then the other number is clearly 0. Assume that neither p nor q is 0. Then 1 = hk, which implies h = k = 1, so p = q, which proves that δ is antisymmetric. Finally, if (p, q), (q, r) ∈ δ, we have q = pk and r = qh for some k, h ∈ N, which implies r = p(hk), so (p, r) ∈ δ, which shows that δ is transitive. Example 1.8. Define the relation λ on R as the set of all ordered pairs (x, y) such that y = x + t, where t is a non-negative number. We have (x, x) ∈ λ because x = x + 0 for every x ∈ R. If (x, y) ∈ λ and (y, x) ∈ λ we have y = x+t and x = y+s for two non-negative numbers t, s, which implies 0 = t + s, so t = s = 0. This means that x = y, so λ is antisymmetric. Finally, if (x, y), (y, z) ∈ λ, we have y = x + u and z = y + v for two non-negative numbers u, v, which implies z = x + u + v, so (x, z) ∈ λ. In current mathematical practice, we often write xρy instead on (x, y) ∈ ρ, where ρ is a relation of S and x, y ∈ S. Thus, we write pδq and xλy instead on (p, q) ∈ δ and (x, y) ∈ λ. Furthermore, we shall use the standard notations “ | ” and “” for δ and λ, that is, we shall write p | q and x  y if p divides q and x is less or equal to y. This alternative way to denote the fact that (x, y) belongs to ρ is known as the infix notation. Example 1.9. Let P(S) be the set of subsets of S. It is easy to verify that the inclusion between subsets “⊆” is a partial order relation on P(S). If U, V ∈ P(S), we denote the inclusion of U in V by U ⊆ V using the infix notation.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 10

Mathematical Analysis for Machine Learning and Data Mining

10

Functions are special relation that enjoy the property described in the next definition. Definition 1.12. Let X, Y be two sets. A function (or a mapping) from X to Y is a relation f on X, Y such that (x, y), (x, y  ) ∈ f implies y = y  . In other words, the first component of a pair (x, y) ∈ f determines uniquely the second component of the pair. We denote the second component of a pair (x, y) ∈ f by f (x) and say, occasionally, that f maps x to y. If f is a function from X to Y we write f : X −→ Y . Definition 1.13. Let X, Y be two sets and let f : X −→ Y . The domain of f is the set Dom(f ) = {x ∈ X | y = f (x) for some y ∈ Y }. The range of f is the set Ran(f ) = {y ∈ Y | y = f (x) for some x ∈ X}. Definition 1.14. Let X be a set, Y = {0, 1} and let L be a subset of S. The characteristic function is the function 1L : S −→ {0, 1} defined by:  1 if x ∈ L, 1L (x) = 0 otherwise for x ∈ S. The indicator function of L is the function IL : S −→ rr ˆ defined by  0 if x ∈ L, IL (x) = ∞ otherwise for x ∈ S. It is easy to see that: 1P ∩Q (x) = 1P (x) · 1Q (x), 1P ∪Q (x) = 1P (x) + 1Q (x) − 1P (x) · 1Q (x), 1P¯ (x) = 1 − 1P (x), for every P, Q ⊆ S and x ∈ S. Theorem 1.2. Let X, Y, Z be three sets and let f : X −→ Y and g : Y −→ Z be two functions. The relation gf : X −→ Z that consists of all pairs (x, z) such that y = f (x) and g(y) = z for some y ∈ Y is a function.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Preliminaries

Proof.

page 11

11

Let (x, z1 ), (x, z2 ) ∈ gf . There exist y1 , y2 ∈ Y such that y1 = f (x), y2 = f (x), g(y1 ) = z1 , and g(y2 ) = z2 .

The first two equalities imply y1 = y2 ; the last two yield z1 = z2 , so gf is indeed a function.  Note that the composition of the function f and g has been denoted in Theorem 1.2 by gf rather than the relation product f g. This manner of denoting the function composition is applied throughout this book. Definition 1.15. Let X, Y be two sets and let f : X −→ Y . If U is a subset of X, the restriction of f to U is the function g : U −→ Y defined by g(u) = f (u) for u ∈ U . The restriction of f to U is denoted by f U . Example 1.10. Let f be the function defined by f (x) = |x| for x ∈ R. Its restriction to R 1 there exists y ∈ Y such that (x, y) ∈ ρ. Let X  = X − {x}, Y  = Y − {y}, and let ρ = ρ ∩ (X  × Y  ). Note that for every B ⊆ X  we have |ρ [B]|  |B| + 1 because for every subset A of X we have |ρ[A|  |A| + 1 and deleting a single element y from ρ[A] still leaves at least |A| elements in this set. By the inductive hypothesis, there exists a perfect matching f  for ρ . This matching extends to a matching f for ρ by defining f (x) = y. In the second case, let A be a proper subset of X such that |ρ[A]| = |A|. Define the sets X  , Y  , X  , Y  as X  = A, X  = X − A, Y  = ρ[A], Y  = Y − ρ[A] and consider the relations ρ = ρ ∩ (X  × Y  ), and ρ = ρ ∩ (X  × Y  ). We shall prove that there are perfect matchings f  and f  for the relations ρ and ρ . A perfect matching for ρ will be given by f  ∪ f  . Since A is a proper subset of X we have both |A|  n and |X − A|  n. For any subset B of A we have ρ [B] = ρ[B], so ρ satisfies the condition of the theorem and a perfect matching f  for ρ exists. Suppose that there exists C ⊆ X  such that |ρ [C]| < |C|. This would imply |ρ[C ∪ A]| < |C ∪ A| because ρ[C ∪ A] = ρ [C] ∪ ρ[A], which is impossible. Thus, ρ also satisfies the condition of the theorem and a  perfect matching exists for ρ . 1.4

Sequences and Collections of Sets

Definition 1.24. A sequence of sets S = (S0 , S1 , . . . , Sn , . . .) is expanding if i < j implies Si ⊆ Sj for every i, j ∈ N. If i < j implies Sj ⊆ Si for every i, j ∈ N, then we say that S is a contracting sequence of sets. A sequence of sets is monotone if it is expanding or contracting. Definition 1.25. Let S be an infinite sequence of subsets of a set S, where S(i) = Si for i ∈ N.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Preliminaries

b3234-main

page 17

17

∞ ∞ The set i=0 j=i Sj is referred to as the lower limit of S; the set ∞ ∞ i=0 j=i Sj is the upper limit of S. These two sets are denoted by lim inf S and lim sup S, respectively.  If x ∈ lim inf S, then there exists i such that x ∈ ∞ j=i Sj ; in other words, x belongs to all but finitely many sets Si . If x ∈ lim sup S, then, for every i there exists j  i such that such that x ∈ Sj ; in this case x belongs to infinitely many sets of the sequence. Clearly, we have lim inf S ⊆ lim sup S. Definition 1.26. A sequence of sets S is convergent if lim inf S = lim sup S. In this case the set L = lim inf S = lim sup S is said to be the limit of the sequence S and is denoted by lim S. Example 1.11. Every expanding sequence of sets is convergent. Indeed, ∞ ∞ since S is expanding we have j=i Sj = Si . Therefore, lim inf S = i=0 Si . ∞ ∞ On the other hand, j=i Sj ⊆ i=0 Si and, therefore, lim sup S ⊆ lim inf S. This shows that lim inf S = lim sup S, that is, S is convergent. A similar argument can be used to show that S is convergent when S is contracting. In this chapter we will use the notion of set countability discussed, for example, in [56]. Definition 1.27. Let C be a collection of subsets of a set S. The collection Cσ consists of all countable unions of members of C. The collection Cδ consists of all countable intersections of members of C, ⎧ ⎫ ⎧ ⎫ ⎨ ⎬ ⎨ ⎬ Cn | Cn ∈ C and Cδ = Cn | Cn ∈ C . Cσ = ⎩ ⎭ ⎩ ⎭ n0

n0

Observe that by taking Cn = C ∈ C for n  0 it follows that C ⊆ Cσ and C ⊆ Cδ . Furthermore, if C, C are two collections of sets such that C ⊆ C , then Cσ ⊆ Cσ and Cδ ⊆ Cδ . Theorem 1.12. For any collection of subsets C of a set S we have (Cσ )σ = Cσ and (Cδ )δ = Cδ . Proof.

The argument is left to the reader.



May 2, 2018 11:28

18

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 18

Mathematical Analysis for Machine Learning and Data Mining

The operations σ and δ can be applied iteratively. We denote sequences of applications of these operations by subscripts adorning the affected collection. The order of application coincides with the order of these symbols in the subscript. For example (C)σδσ means ((Cσ )δ )σ . Thus, Theorem 1.12 can be restated as the equalities Cσσ = Cσ and Cδδ = Cδ . Observe that if C = (C0 , C1 , . . .) is a sequence of sets, then lim sup C = ∞ ∞ ∞ ∞ i=0 j=i Cj ∈ Cσδ and lim inf C = i=0 j=i Cj belongs to Cδσ , where C = {Cn | n ∈ N}.

1.5

Partially Ordered Sets

If ρ is a partial order on S, we refer to the pair (S, ρ) as a partially ordered set or as a poset. A strict partial order, or a strict order on S, is a relation ρ ⊆ S × S that is irreflexive and transitive. Note that if ρ is a partial order on S, the relation ρ1 = ρ − {(x, x) | x ∈ S} is a strict partial order on S. From now on we shall denote by “” a generic partial order on a set S; thus, a generic partially ordered set is denoted by (S, ). Example 1.12. Let δ = {(m, n) | m, n ∈ N, n = km for some k ∈ N}. Since n = 1 n it follows that (n, n) ∈ δ for every n ∈ N, so δ is a reflexive relation. Suppose that (m, n) ∈ δ and (n, m) ∈ δ, so n = mk and m = nh for some k, h ∈ N. This implies n(1 − kh) = 0. If n = 0, it follows that m = 0. If n = 0 we have kh = 1, which means that k = h = 1 because k, h ∈ N, so again, m = n. Thus, δ is antisymmetric. Finally, if (m, n), (n, p) ∈ δ we have n = rm and p = sn for some r, s ∈ N, so p = srm, which implies (m, p) ∈ δ. This shows that δ is also transitive and, therefore, it is a partial order relation on N. Example 1.13. Let π, σ be two partitions in PART(S). We define π  σ if each block C of σ is a π-saturated set. It is clear that “” is a reflexive relation. Suppose that π  σ and σ  τ , where π, σ, τ ∈ PARTfin (S). Then each block D of τ is a union of blocks of σ, and each block of σ is a union of blocks of π. Thus, D is a union of blocks of π and, therefore, π  τ . Suppose now that π  σ and σ  π. Then, each block C of σ is a  union of π-blocks, C = i∈I Bi , and every π-block is a union of σ-blocks.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Preliminaries

b3234-main

page 19

19

Since no block of a partition can be a subset of another block of the same partition, it follows that each block of σ coincides with a block of π, that is σ = π. Definition 1.28. Let (S, ) be a poset and let K ⊆ S. The set of upper bounds of the set K is the set K s = {y ∈ S | x  y for every x ∈ K}. The set of lower bounds of the set K is the set K i = {y ∈ S | y  x for every x ∈ K}. If K s = ∅, we say that the set K is bounded above. Similarly, if K i = ∅, we say that K is bounded below. If K is both bounded above and bounded below we will refer to K as a bounded set. If K s = ∅ (K i = ∅), then K is said to be unbounded above (below). Theorem 1.13. Let (S, ) be a poset and let U and V be two subsets of S. If U ⊆ V , then we have V i ⊆ U i and V s ⊆ U s . Also, for every subset T of S, we have T ⊆ (T s )i and T ⊆ (T i )s . Proof. The argument for both statements of the theorem amounts to a direct application of Definition 1.28.  Note that for every subset T of a poset S, we have both T i = ((T i )s )i

(1.1)

T s = ((T s )i )s .

(1.2)

and Indeed, since T ⊆ (T i )s , by the first part of Theorem 1.13, we have ((T s )i )s ⊆ T s . By the second part of the same theorem applied to T s , we have the reverse inclusion T s ⊆ ((T s )i )s , which yields T s = ((T s )i )s . Theorem 1.14. For any subset K of a poset (S, ρ), the sets K ∩ K s and K ∩ K i contain at most one element. Proof. Suppose that y1 , y2 ∈ K ∩ K s . Since y1 ∈ K and y2 ∈ K s , we have (y1 , y2 ) ∈ ρ. Reversing the roles of y1 and y2 (that is, considering now that y2 ∈ K and y1 ∈ K s ), we obtain (y2 , y1 ) ∈ ρ. Therefore, we may conclude that y1 = y2 because of the antisymmetry of the relation ρ, which shows that K ∩ K s contains at most one element. A similar argument can be used for the second part of the proposition; we leave it to the reader.  Definition 1.29. Let (S, ) be a poset. The least (greatest) element of the subset K of S is the unique element of the set K ∩K i (K ∩K s , respectively) if such an element exists.

May 2, 2018 11:28

20

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 20

Mathematical Analysis for Machine Learning and Data Mining

If K is unbounded above, then it is clear that K has no greatest element. Similarly, if K is unbounded below, then K has no least element. Applying Definition 1.29 to the set S, the least (greatest) element of the poset (S, ) is an element a of S such that a  x (x  a, respectively) for all x ∈ S. It is clear that if a poset has a least element u, then u is the unique minimal element of that poset. A similar statement holds for the greatest and the maximal elements. Definition 1.30. The subset K of the poset (S, ) has a least upper bound u if K s ∩ (K s )i = {u}. K has the greatest lower bound v if K i ∩ (K i )s = {v}. We note that a set can have at most one least upper bound and at most one greatest lower bound. Indeed, we have seen above that for any set U the set U ∩ U i may contain an element or be empty. Applying this remark to the set K s , it follows that the set K s ∩ (K s )i may contain at most one element, which shows that K may have at most one least upper bound. A similar argument can be made for the greatest lower bound. If the set K has a least upper bound, we denote it by sup K. The greatest lower bound of a set will be denoted by inf K. These notations come from the Latin terms supremum and infimum used alternatively for the least upper bound and the greatest lower bound, respectively. Lemma 1.1. Let U, V be two subsets of a poset (S, ). If U ⊆ V then V s ⊆ U s and V i ⊆ U i . Proof. This statement follows immediately from the definitions of the sets of upper bounds and lower bounds, respectively.  Theorem 1.15. Let (S, ) be a poset and let K, L be two subsets of S such that K ⊆ L. If sup K and sup L exist, then sup K  sup L; if inf K and inf L exist, then inf L  inf K. Proof. By Lemma 1.1 we have Ls ⊆ K s and Li ⊆ Ki . By the same Lemma, we have: (K s )i ⊆ (Ls )i and (K i )s ⊆ (Li )s . Let a = sup K and b = sup L. Since {a} = K s ∩ (K s )i and (K s )i ⊆ s i (L ) , it follows that a ∈ (Ls )i . Since b ∈ Ls , this implies a  b. If c = inf K and d = inf L, taking into account that {c} = K i ∩ (K i )s , we have c ∈ (Li )s because K i ∩ (K i )s ⊆ (Li )s . Since d ∈ Li we have d  c. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Preliminaries

9in x 6in

b3234-main

page 21

21

Example 1.14. A two-element subset {m, n} of the poset (N, δ) introduced in Example 1.12 has both an infimum and a supremum. Indeed, let p be the least common multiple of m and n. Since (n, p), (m, p) ∈ δ, it is clear that p is an upper bound of the set {m, n}. On the other hand, if k is an upper bound of {m, n}, then k is a multiple of both m and n. In this case, k must also be a multiple of p because otherwise we could write k = pq + r with 0 < r < p by dividing k by p. This would imply r = k − pq; hence, r would be a multiple of both m and n because both k and p have this property. However, this would contradict the fact that p is the least multiple that m and n share! This shows that the least common multiple of m and n coincides with the supremum of the set {m, n}. Similarly, inf{m, n} equals the greatest common divisor m and n. Example 1.15. Let π, σ be two partitions in PART(S). It is easy to see that the collection θ = {B ∩ C | B ∈ π, C ∈ σ, B ∩ C = ∅} is a partition of S; furthermore, if τ is a partition of S such that π  τ and σ  τ , then each block E of τ is both a π-saturated set and a σ-saturated set, and, therefore a θ-saturated set. This shows that τ = inf{π, σ}. The partition will be denoted by π ∧ σ. Definition 1.31. A minimal element of a poset (S, ) is an element x ∈ S such that {x}i = {x}. A maximal element of (S, ) is an element y ∈ S such that {y}s = {y}. In other words, x is a minimal element of the poset (S, ) if there is no element less than or equal to x other than itself; similarly, x is maximal if there is no element greater than or equal to x other than itself. For the poset (R, ), it is possible to give more specific descriptions of the supremum and infimum of a subset when they exist. Theorem 1.16. If T ⊆ R, then u = sup T if and only if u is an upper bound of T and, for every > 0, there is t ∈ T such that u − < t  u. The number v is inf T if and only if v is a lower bound of T and, for every > 0, there is t ∈ T such that v  t < v + . Proof. We prove only the first part of the theorem; the argument for the second part is similar and is left to the reader. Suppose that u = sup T ; that is, {u} = T s ∪ (T s )i . Since u ∈ T s , it is clear that u is an upper bound for T . Suppose that there is > 0 such that no t ∈ T exists such that u − < t  u. This means that u − is also an

May 2, 2018 11:28

22

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 22

Mathematical Analysis for Machine Learning and Data Mining

upper bound for T , and in this case u cannot be a lower bound for the set of upper bounds of T . Therefore, no such may exist. Conversely, suppose that u is an upper bound of T and for every > 0, there is t ∈ T such that u − < t  u. Suppose that u does not belong to (K s )i . This means that there is another upper bound u of T such that u < u. Choosing = u − u , we would have no t ∈ T such that u − = u < t  u because this would prevent u from being an upper  bound of T . This implies u ∈ (K s )i , so u = sup T . ˆ ) every subset Theorem 1.17. In the extended poset of real numbers (R, has a supremum and an infimum. Proof. If a set is bounded then the existence of the supremum and infimum is established by the Completeness Axiom. Suppose that a subset ˆ has no upper bound in R. Then x  ∞, so ∞ is an upper bound S of R ˆ Moreover ∞ is the unique upper bound of S, so sup S = ∞. of S in R. ˆ Similarly, if S has no lower bound in R, then inf S = −∞ in R.  ˆ ) The definitions of infimum and supremum of the empty set in (R, are sup ∅ = −∞ and inf ∅ = ∞, in order to remain consistent with Theorem 1.15. A very important axiom for the set R is given next. The Completeness Axiom for R: If T is a non-empty subset of R that is bounded above, then T has a supremum.

A statement equivalent to the Completeness Axiom for R follows. Theorem 1.18. If T is a non-empty subset of R that is bounded below, then T has an infimum. Proof. Note that the set T i is not empty. If s ∈ T i and t ∈ T , we have s  t, so the set T i is bounded above. By the Completeness Axiom v = sup T i exists and {v} = (T i )s ∩ ((T i )s )i = (T i )s ∩ T i by equality (1.1). Thus, v = inf T .  We leave to the reader to prove that Theorem 1.18 implies the Completeness Axiom for R. Another statement equivalent to the Completeness Axiom is the following.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Preliminaries

9in x 6in

b3234-main

page 23

23

Theorem 1.19. (Dedekind’s Theorem) Let U and V be non-empty subsets of R such that U ∪ V = R and x ∈ U, y ∈ V imply x < y. Then, there exists a ∈ R such that if x > a, then x ∈ V , and if x < a, then x ∈ U . Proof. Observe that U = ∅ and V ⊆ U s . Since V = ∅, it means that U is bounded above, so by the Completeness Axiom sup U exists. Let a = sup U . Clearly, u ≤ a for every u ∈ U . Since V ⊆ U s , it also follows that a  v for every v ∈ V . If x > a, then x ∈ V because otherwise we would have x ∈ U since U ∪ V = R and this would imply x  a. Similarly, if x < a, then x ∈ U .  Using the previously introduced notations, Dedekind’s theorem can be stated as follows: if U and V are non-empty subsets of R such that U ∪ V = R, U s ⊆ V , V i ⊆ U , then there exists a such that {a}s ⊆ V and {a}i ⊆ U . One can prove that Dedekind’s theorem implies the Completeness Axiom. Indeed, let T be a non-empty subset of R that is bounded above. Therefore V = T s = ∅. Note that U = (T s )i = ∅ and U ∪V = R. Moreover, U s = ((T s )i )s = T s = V and V i = (T s )i = U . Therefore, by Dedekind’s theorem, there is a ∈ R such that {a}s ⊆ V = T s and {a}i ⊆ U = (T s )i . Note that a ∈ {a}s ∩ {a}i ⊆ T s ∩ (T s )i , which proves that a = sup T . By adding the symbols +∞ and −∞ to the set R, one obtains the set ˆ The partial order  defined on R can now be extended to R ˆ by −∞  x R. and x  +∞ for every x ∈ R. ˆ ), the sets T i and T s are non-empty for Note that, in the poset (R, ˆ ˆ every T ∈ P(R) because −∞ ∈ T i and +∞ ∈ T s for any subset T of R. ˆ both sup T and inf T exist in the Theorem 1.20. For every set T ⊆ R, ˆ poset (R, ). Proof. We present the argument for sup T . If sup T exists in (R, ), then ˆ ). it is clear that the same number is sup T in (R, Assume now that sup T does not exist in (R, ). By the Completeness Axiom for R, this means that the set T does not have an upper bound in (R, ). Therefore, the set of upper bounds of T in (Tˆ , ) is T sˆ = {+∞}. ˆ ). It follows immediately that in this case sup T = +∞ in (R,  Theorem 1.21. Let I be a partially ordered set and let {xi | i ∈ I} be a ˆ indexed by I. For i ∈ I let Si = {xj | j ∈ I and i  j}. We subset of R have sup inf Si  inf sup Si .

(1.3)

May 2, 2018 11:28

24

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 24

Mathematical Analysis for Machine Learning and Data Mining

Proof. Note that if i  h, then Sh ⊆ Si for i, h ∈ I. As we saw earlier, ˆ ). each set Si has both an infimum yi and a supremum zi in the poset (R, It is clear that if i  h, then yi  yh  zh  zi . We claim that sup{yi | i ∈ I}  inf{zh | h ∈ I}. Indeed, since then yi  zh for all i, h such that i  h, we have yi  inf{zh | h ∈ I} for all i ∈ I. Therefore, sup{yi | i ∈ I}  inf{zh | h ∈ I}, which can be written as sup inf Si  inf sup Si .



ˆ The image of S under f is the set Let S be a set and let f : S −→ R. ˆ sup f (S) exists. Furthermore, if f (S) = {f (x) | x ∈ S}. Since f (S) ⊆ R, there exists u ∈ S such that f (u) = sup f (S), then we say that f attains its supremum at u. This is not always the case as the next example shows. ˆ be defined by f (x) = 1 . It is clear Example 1.16. Let f : (0, 1) −→ R 1−x that sup f ((0, 1)) = ∞. However, there is no u ∈ (0, 1) such that f (u) = ∞, so f does not attain its supremum on (0, 1). ˆ be a function. We have Let X, Y be two sets and let f : X × Y −→ R sup inf f (x, y)  inf sup f (x, y).

x∈X y∈Y

y∈Y x∈X

(1.4)

Indeed, note that inf y∈Y f (x, y)  f (x, y) for every x ∈ X and y ∈ Y by the definition of the infimum. Note that the left member of the inequality depends only on x. The last inequality implies supx∈X inf y∈Y f (x, y)  supx∈X f (x, y), by the monotonicity of sup and now the current left member is a lower bound of the set {z = supx∈X f (x, y) | y ∈ Y }. This implies immediately the inequality (1.4). If instead of inequality (1.4) the function f satisfies the equality: sup inf f (x, y) = inf sup f (x, y),

x∈X y∈Y

y∈Y x∈X

(1.5)

then the common value of both sides is a saddle value for f . Since inf y∈Y f (x, y) is a function h(x) of x and supx∈X f (x, y) is a function g(y) of y, the existence of a saddle value for f implies that supx∈X h(x) = inf y∈Y g(y) = v.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Preliminaries

page 25

25

If both supx∈X h(x) and inf y∈Y g(y) are attained, that is, there are x0 ∈ X and y0 ∈ Y such that h(x0 ) = sup h(x) = sup inf f (x, y), x∈X

x∈X y∈Y

g(y0 ) = inf g(y) = inf sup f (x, y), y∈Y

y∈Y x∈X

we have h(x0 ) = sup inf f (x, y) x∈X y∈Y

= inf sup f (x, y) y∈Y x∈X

(because of the existence of the saddle value) = inf g(y) = g(y0 ). y∈Y

Therefore, g(y0 ) = sup f (x, y0 ) = f (x0 , y0 ) = inf f (x0 , y) = h(x0 ), y∈Y

x∈X

and f (x, y0 )  f (x0 , y0 )  f (x0 , y). The pair (x0 , y0 ) that satisfies the inequalities f (x, y0 )  f (x0 , y0 )  f (x0 , y) is referred to as a saddle point for f . Conversely, if there exists a saddle point (x0 , y0 ) such that f (x, y0 )  f (x0 , y0 )  f (x0 , y), then f : X × Y −→ R has a saddle value. Indeed, in this case we sup f (x, y0 )  f (x0 , y0 )  inf f (x0 , y),

x∈X

y∈Y

hence inf sup f (x, y)  sup f (x, y0 )  f (x0 , y0 )  inf f (x0 , y)  sup inf f (x, y).

y∈Y x∈X

x∈X

y∈Y

x∈X y∈Y

Since a saddle value exists, these inequalities become equalities and we have a saddle value. Definition 1.32. Let (S, ) be a poset. A chain of (S, ) is a subset T of S such that for every x, y ∈ T such that x = y we have either x < y or y < x. If the set S is a chain, we say that (S, ) is a totally ordered set and the relation  is a total order.

May 2, 2018 11:28

26

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 26

Mathematical Analysis for Machine Learning and Data Mining

Example 1.17. The set of real numbers equipped with the usual partial order (R, ) is a chain since, for every x, y ∈ R, we have either x  y or y  x. Theorem 1.22. If {Ui | i ∈ I} is a chain of the poset (CHAINS(S), ⊆)  (that is, a chain of chains of (S, )), then {Ui | i ∈ I} is itself a chain of (S, ) (that is, a member of (CHAINS(S), ⊆)).  Proof. Let x, y ∈ {Ui | i ∈ I}. There are i, j ∈ I such that x ∈ Ui and y ∈ Uj and we have either Ui ⊆ Uj or Uj ⊆ Ui . In the first case, we have either xi  xj or xj  xi because both x and y belong to the chain Uj . The same conclusion can be reached in the second case when both x and y belong to the chain Ui . So, in any case, x and y are comparable, which   proves that {Ui | i ∈ I} is a chain of (S, ). A statement equivalent to a fundamental principle of set theory known as the Axiom of Choice is Zorn’s lemma stated below. Zorn’s Lemma: If every chain of a poset (S, ) has an upper bound, then S has a maximal element.

Theorem 1.23. The following three statements are equivalent for a poset (S, ): (i) If every chain of (S, ) has an upper bound, then S has a maximal element (Zorn’s Lemma). (ii) If every chain of (S, ) has a least upper bound, then S has a maximal element. (iii) S contains a chain that is maximal with respect to set inclusion (Hausdorff2 maximality principle). Proof. (i) implies (ii) is immediate. (ii) implies (iii): Let (CHAINS(S), ⊆) be the poset of chains of S ordered by set inclusion. Every chain {Ui | i ∈ I} of the poset (CHAINS(S), ⊆)  has a least upper bound {Ui | i ∈ I} in the poset (CHAINS(S), ⊆). 2 Felix Hausdorff was born on November 8th , 1868 in Breslau in Prussia, (now Germany) and died on January 26th , 1942 in Bonn, Germany. Hausdorff is one of the founders of modern topology and set theory, and has major contributions in measure theory, and functional analysis. Hausdorff studied in Leipzig, where he obtained his doctorate in 1891. He taught at the Universities of Bonn, Greifswald, and Leipzig. Life became very difficult for German Jews under the National Socialist regime and on 26 January 1942, Felix Hausdorff, along with his wife and his sister-in-law, committed suicide to avoid being deported.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Preliminaries

b3234-main

page 27

27

Therefore, by (ii), (CHAINS(S), ⊆) has a maximal element that is a chain of (S, ) that is maximal with respect to set inclusion. (iii) implies (i): Suppose that S contains a chain W that is maximal with respect to set inclusion and that every chain of (S, ) has an upper bound. Let w be an upper bound of W . If w ∈ W , then w is a maximal element of S. Indeed, if this were not the case, then S would contain an element t such that w < t and W ∪ {t} would be a chain that would strictly include W . If w ∈ W , then W ∪ {w} would be a chain strictly including W , which, again, would contradict the maximality of W . Thus, w is a maximal element of (S, ).  Let (S, ) and (T, ) be two posets. Definition 1.33. A morphism between (S, ) and (T, ) or a monotonic mapping (or an increasing mapping) between (S, ) and (T, ) is a mapping f : S −→ T such that u, v ∈ S and u > v imply f (u)  f (v). A mapping g : S −→ T is antimonotonic or a decreasing mapping if u, v ∈ S and u > v imply g(u)  g(v). The mapping f is strictly monotonic (or strictly increasing) if u < v implies f (u) < f (v); f is strictly antimonotonic (or strictly decreasing) if u < v implies f (u) < f (v). Note that g : S −→ T is antimonotonic if and only if g is a monotonic mapping between the poset (S, ) and the dual (T, ) of the poset (T, ). Example 1.18. Consider a set M , the poset (P(M ), ⊆), and the functions f, g : (P(M ))2 −→ P, defined by f (K, H) = K ∪ H and g(K, H) = K ∩ H, for K, H ∈ P(M ). If the Cartesian product is equipped with the product partial order, then both f and g are monotonic. Indeed, if (K1 , H1 ) ⊆ (K2 , H2 ), we have K1 ⊆ K2 and H1 ⊆ H2 , which implies that f (K1 , H1 ) = K1 ∪ H1 ⊆ K2 ∪ H2 = f (K2 , H2 ). The argument for g is similar, and it is left to the reader. Theorem 1.24. Let (P, ), (R, ), (S, ) be three posets and let f : P −→ R, g : R −→ S be two monotonic mappings. The mapping gf : P −→ S is also monotonic. Proof. Let x, y ∈ P be such that x  y. In view of the monotonicity of f , we have f (x)  f (y), and this implies (g(f (x)) ≤ g(f (y)) because of the monotonicity of g. Therefore, gf is monotonic. 

May 2, 2018 11:28

28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 28

Mathematical Analysis for Machine Learning and Data Mining

Monotonic functions map chains to chains, as we show next. Theorem 1.25. Let (P, ) and (R, ) be two posets and f : P −→ R be a monotonic function. If L ⊆ P is a chain in (P, ), then f (L) is a chain in (R, ). Proof. Let u, v ∈ f (L) be two elements of f (L). There exist x, y ∈ L such that f (x) = u and f (y) = v. Since L is a chain, we have either x  y or y  x. In the former case, the monotonicity of f implies u  v; in the latter situation, we have v  u.  Let x ∈ Seq(X). A sequence y ∈ Seq(S) is a subsequence of x if there exists a strictly increasing function h : N −→ N such that yn = xh(n) for n ∈ N. Example 1.19. The sequence y = (x0 , x1 , x4 , x9 , . . .) is a subsequence of the sequence x because we can write yk = xk2 for k ∈ N. If x ∈ Seq(X) we will denote the subsequence (xp , xp+1 , . . . , xq ) of x by xp:q .

1.6

Closure and Interior Systems

Closure and interior systems introduced in this section are significant in algebra, measure theory, and topology. Definition 1.34. Let S be a set. A closure system on S is a collection C of subsets of S that satisfies the following conditions: (i) S ∈ C, and  (ii) for every collection D ⊆ C, we have D ∈ C. Example 1.20. Let C be the collection of all intervals [a, b] = {x ∈ R | a  x  b} with a, b ∈ R and a  b together with the empty set and the  set R. Note that C = R ∈ C, so the first condition of Definition 1.34 is satisfied.  Let D be a non-empty subcollection of C. If ∅ ∈ D, then D = ∅ ∈ C.  If D = {R}, then D = R ∈ C. Therefore, we need to consider only the  case when D = {[ai , bi ] | i ∈ I}. Then, D = ∅ unless a = sup{ai | i ∈ I}  and b = inf{bi | i ∈ I} both exist and a  b, in which case D = [a, b]. Thus, C is a closure system.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Preliminaries

b3234-main

page 29

29

Many classes of relations define useful closure systems. Theorem 1.26. The sets REFL(S), SYMM(S) and TRAN(S) are closure systems on S. Proof. Note that S × S is a reflexive, symmetric, and transitive relation   on S. Therefore, REFL(S) = S × S ∈ REFL(S), SYMM(S) = S × S ∈  SYMM(S), and TRAN(S) = S × S ∈ TRAN(S). Now let C = {ρi | i ∈ I} be a collection of transitive relations and let  ρ = {ρi | i ∈ I}. Suppose that (x, y), (y, z) ∈ ρ. Then (x, y), (y, z) ∈ ρi for every i ∈ I, so (x, z) ∈ ρi for i ∈ I because each of the relations ρi  is transitive. Thus, (x, z) ∈ ρ, which shows that C ∈ TRAN(S). This allows us to conclude that TRAN(S) is indeed a closure system. We leave it to the reader to prove that REFL(S) and SYMM(S) are also closure systems.  Theorem 1.27. The set of equivalences on S, EQUIV(S), is a closure system. Proof. The relation θS = S × S, is clearly an equivalence relation. Thus,  EQUIV(S) = θS ∈ EQUIV(S). Now let C = {ρi | i ∈ I} be a collection of transitive relations and  let ρ = {ρi | i ∈ I}. It is immediate that ρ is an equivalence on S, so EQUIV(S) is a closure system.  Definition 1.35. A mapping K : P(S) −→ P(S) is a closure operator on a set S if it satisfies the conditions (i) U ⊆ K(U ) (expansiveness), (ii) U ⊆ V implies K(U ) ⊆ K(V ) (monotonicity), and (iii) K(K(U )) = K(U ) (idempotency) for U, V ∈ P(S). Example 1.21. Let X, Y be two sets and let f : X −→ Y be a function. Define the mapping Kf : P(X) −→ P(X) as Kf (S) = f −1 (f (S)) for S ∈ P(X). We claim that Kf is a closure operator. Indeed, if x ∈ S, then f (x) ∈ f (S), which implies that x ∈ f −1 (f (S)). Thus, S ⊆ Kf (S). The monotonicity of Kf is immediate. Taking into account the expansiveness and the monotonicity of Kf we have Kf (S) ⊆ Kf (Kf (S)) for S ∈ P(X). To prove the converse inclusion, let x ∈ Kf (Kf (S)) = f −1 f (Kf (S)). We have f (x) ∈ f (Kf (S)), so there exists z ∈ Kf (S)

May 2, 2018 11:28

30

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 30

Mathematical Analysis for Machine Learning and Data Mining

such that f (x) = f (z). Since z ∈ Kf (S), there exists s ∈ S such that f (z) = f (s), so f (x) = f (s). Therefore, x ∈ f −1 (f (s)) ⊆ Kf (S) and we obtain the equality Kf (S) = Kf (Kf (S)). Example 1.22. Let K : P(R) −→ P(R) be defined by ⎧ ⎪ if U = ∅, ⎪∅ ⎨ K(U ) =

[a, b] if both a = inf U and b = sup U exist, ⎪ ⎪ ⎩R otherwise,

for U ∈ P(R). We leave to the reader the verification that K is a closure operator. Definition 1.36. Let S be a set. A collection M of subsets of S is a monotone if the following conditions are satisfied: (i) if C = (Cn ) is an increasing sequence of sets in M, then  n∈N Cn ∈ M; (ii) if D = (Dn ) is a decreasing sequence of sets in M, then  n∈N Dn ∈ M. Note that P(S) is a monotone collection. If {Mi | i ∈ I} is a set of  monotone collections of subsets of S, then i∈I Mi is a monotone collection. This, the family of monotone collections is a closure system. The corresponding closure operator is denoted by Kmon . Closure operators induce closure systems, as shown by the next lemma. Lemma 1.2. Let K : P(S) −→ P(S) be a closure operator. Define the family of sets CK = {H ∈ P(S) | H = K(H)}. Then, CK is a closure system on S.  Proof. Since S ⊆ K(S) ⊆ S, we have S ∈ CK , so CK = S ∈ CK . Let D = {Di | i ∈ I} be a collection of subsets of S such that Di =   K(Di ) for i ∈ I. Since D ⊆ Di , we have K( D) ⊆ K(Di ) = Di for     every i ∈ I. Therefore, K( D) ⊆ D, which implies K( D) = D. This proves our claim.  Note that CK , as defined in Lemma 1.2, equals the range of K. Indeed, if L ∈ Ran(K), then L = K(H) for some H ∈ P(S), so K(L) = K(K(H)) = K(H) = L, which shows that L ∈ CK . The reverse inclusion is obvious. We refer to the sets in CK as the K-closed subsets of S. In the reverse direction from Lemma 1.2, we show that every closure system generates a closure operator.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Preliminaries

b3234-main

page 31

31

Lemma 1.3. Let C be a closure system on the set S. Define the mapping  KC : P(S) −→ P(S) by KC (H) = {L ∈ C | H ⊆ L}. Then, KC is a closure operator on the set S. Proof. Note that the collection {L ∈ C | H ⊆ L} is not empty since it contains at least S, so KC (H) is defined and is clearly the smallest element of C that contains H. Also, by the definition of KC (H), it follows immediately that H ⊆ KC (H) for every H ∈ P(S). Suppose that H1 , H2 ∈ P(S) are such that H1 ⊆ H2 . Since {L ∈ C | H2 ⊆ L} ⊆ {L ∈ C | H1 ⊆ L}, we have

  {L ∈ C | H1 ⊆ L} ⊆ {L ∈ C | H2 ⊆ L},

so KC (H1 ) ⊆ KC (H2 ). We have KC (H) ∈ C for every H ∈ P(S) because C is a closure system. Therefore, KC (H) ∈ {L ∈ C | KC (H) ⊆ L}, so KC (KC (H)) ⊆ KC (H). Since the reverse inclusion clearly holds, we obtain KC (KC (H)) = KC (H).  Definition 1.37. Let C be a closure system on a set S and let T be a subset of S. The C-set generated by T is the set KC (T ). Note that KC (T ) is the least set in C that includes T . Theorem 1.28. Let S be a set. For every closure system C on S, we have C = CKC . For every closure operator K on S, we have K = KCK . Proof. Let C be a closure system on S and let H ⊆ M . Then, we have the following equivalent statements: (1) H ∈ CKC . (2) KC (H) = H. (3) H ∈ C. The equivalence between (2) and (3) follows from the fact that KC (H) is the smallest element of C that contains H. Conversely, let K be a closure operator on S. To prove the equality of K and KCK , consider the following list of equal sets, where H ⊆ S: (1) KCK (H).  (2) {L ∈ CK | H ⊆ L}.  (3) {L ∈ P(S) | H ⊆ L = K(L)}. (4) K(H).

May 2, 2018 11:28

32

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 32

Mathematical Analysis for Machine Learning and Data Mining

We need to justify only the equality of the last two members of the list. Since H ⊆ K(H) = K(K(H)), we have K(H) ∈ {L ∈ P(S) | H ⊆ L =  K(L)}. Thus, {L ∈ P(S) | H ⊆ L = K(L)} ⊆ K(H). To prove the reverse inclusion, note that for every L ∈ {L ∈ P(S) | H ⊆ L = K(L)},  we have H ⊆ L, so K(H) ⊆ K(L) = L. Therefore, K(H) ⊆ {L ∈ P(S) | H ⊆ L = K(L)}.  Theorem 1.28 shows the existence of a natural bijection between the set of closure operators on a set S and the set of closure systems on S. Definition 1.38. Let C be a closure system on a set S and let T be a subset of S. The C-closure of the set T is the set KC (T ). As we observed before, KC (T ) is the smallest element of C that contains T . Example 1.23. Let K be the closure operator given in Example 1.22. Since the closure system CK equals the range of K, it follows that the members of CK , the K-closed sets, are ∅, R, and all closed intervals [a, b] with a  b. Thus, CK is the closure system C introduced in Example 1.20. Therefore, K and C correspond to each other under the bijection of Theorem 1.28. For a relation ρ, on S define ρ+ as KTRAN(S) (ρ). The relation ρ+ is called the transitive closure of ρ and is the least transitive relation containing ρ. Theorem 1.29. Let ρ be a relation on a set S. We have  ρ+ = {ρn | n ∈ N and n  1}.  Proof. Let τ be the relation {ρn | n ∈ N and n  1}. We claim that τ is transitive. Indeed, let (x, z), (z, y) ∈ τ . There exist p, q ∈ N, p, q  1 such that (x, z) ∈ ρp and (z, y) ∈ ρq . Therefore, (x, y) ∈ ρp ρq = ρp+q ⊆ ρ+ , which shows that ρ+ is transitive. The definition of ρ+ implies that if σ is a transitive relation such that ρ ⊆ σ, then ρ+ ⊆ σ. Therefore, ρ+ ⊆ τ . Conversely, since ρ ⊆ ρ+ we have ρn ⊆ (ρ+ )n for every n ∈ N. The transitivity of ρ+ implies that (ρ+ )n ⊆ ρ+ , which implies ρn ⊆ ρ+ for every  n  1. Consequently, τ = {ρn | n ∈ N and n  1} ⊆ ρ+ . This proves the equality of the theorem.  It is easy to see that the set of all reflexive and transitive relations on a set S, REFTRAN(S), is also a closure system on the set of relations on S.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Preliminaries

b3234-main

page 33

33

For a relation ρ on S, define ρ∗ as KREFTRAN(S) (ρ). The relation ρ∗ is called the transitive-reflexive closure of ρ and is the least transitive and reflexive relation containing ρ. We have the following analog of Theorem 1.29. Theorem 1.30. Let ρ be a relation on a set S. We have  ρ∗ = {ρn | n ∈ N}. Proof. The argument is very similar to the proof of Theorem 1.29; we leave it to the reader.  Definition 1.39. Let S be a set and let F be a set of operations on S. A subset P of S is closed under F , or F -closed, if P is closed under f for every f ∈ F ; that is, for every operation f ∈ F , if f is n-ary and p0 , . . . , pn−1 ∈ P , then f (p0 , . . . , pn−1 ) ∈ P . Note that S itself is closed under F . Further, if C is a non-empty  collection of F -closed subsets of S, then C is also F -closed. Example 1.24. Let F be a set of operations on a set S. The collection of all F -closed subsets of a set S is a closure system. Definition 1.40. An interior operator on a set S is a mapping I : P(S) −→ P(S) that satisfies the following conditions: (i) U ⊇ I(U ) (contraction), (ii) U ⊇ V implies I(U ) ⊇ I(V ) (monotonicity), and (iii) I(I(U )) = I(U ) (idempotency), for U, V ∈ P(S). Such a mapping is known as an interior operator on the set S. Interior operators define certain collections of sets. Definition 1.41. An interior system on a set S is a collection I of subsets of S such that (i) ∅ ∈ I and,  (ii) for every subcollection D of I we have D ∈ I. Theorem 1.31. Let I : P(S) −→ P(S) be an interior operator. Define the family of sets II = {U ∈ P(S) | U = I(U )}. Then, II is an interior system on S. Conversely, if I is an interior system on the set S, define the mapping  II : P(S) −→ P(S) by II (U ) = {V ∈ I | V ⊆ U }. Then, II is an interior operator on the set S.

May 2, 2018 11:28

34

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 34

Mathematical Analysis for Machine Learning and Data Mining

Moreover, for every interior system I on S, we have I = III . For every interior operator I on S, we have I = III . Proof. This statement follows by duality from Lemmas 1.2 and 1.3 and from Theorem 1.28.  We refer to the sets in II as the I-open subsets of S. Theorem 1.32. Let K : P(S) −→ P(S) be a closure operator on the set S. Then, the mapping I : P(S) −→ P(S) given by I(U ) = S − K(S − U ) for U ∈ P(S) is an interior operator on S. Proof. Since S − U ⊆ K(S − U ), it follows that I(U ) ⊆ S − (S − U ) = U , which proves property (i) of Definition 1.41. Suppose that U ⊆ V , where U, V ∈ P(S). Then, we have S − V ⊆ S − U , so K(S − V ) ⊆ K(S − U ) by the monotonicity of closure operators. Therefore, I(U ) = S − K(S − U ) ⊆ S − K(S − V ) = I(V ), which proves the monotonicity of I. Finally, observe that we have I(I(U )) ⊆ I(U ) because of the contraction property already proven for I. Thus, we need only show that I(U ) ⊆ I(I(U )) to prove the idempotency of I. This inclusion follows immediately from I(I(U )) = I(S − K(S − U )) ⊇ I(S − (S − U )) = I(U ).



We can prove that if I is an interior operator on a set S, then K : P(S) −→ P(S) defined as K(U ) = S − I(S − U ) for U ∈ P(S) is a closure operator on the same set. 1.7

Algebras and σ-Algebras of Sets

Definition 1.42. Let S be a non-empty set. An algebra of sets on S is a non-empty collection E of subsets of S such that: (i) if U ∈ E, then its complement U = S − U belongs to E; n (ii) if U1 , . . . , Un belong to E, then i=1 Ui ∈ E. In other words, a non-empty family of subsets of S, E, is an algebra of sets on S, it is closed with respect to complement and to finite unions. Every algebra of sets E on a set S contains both S and ∅. Indeed, since E is non-empty, there exists T ∈ E. Therefore, T ∈ E and this implies S = T ∪ T ∈ E. Therefore, ∅ = S ∈ E.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Preliminaries

b3234-main

page 35

35

Example 1.25. The collection E0 = {∅, S} is an algebra on S; moreover, as we saw, every algebra E on S contains E0 . The set P(S) of all subsets of a set S is an algebra on S. If T is a subset of S, then the collection E(T ) = {∅, T, S − T, S} is a algebra on S referred to as the algebra generated by T . An algebra of sets E on S is closed with respect to finite intersections because  n  n   Ai = S − (S − Ai ) . i=1

i=1

The difference A − B of two sets in E belongs to E because A − B = A ∩ B. A very important type of algebras of sets that play a central role in measure theory is introduced next. Definition 1.43. A σ-algebra of sets on S is a non-empty family of subsets E of S such that (i) if A ∈ E, then its complement A = S − A belongs to E; (ii) if {An | n ∈ N} is a countable collection of subsets of S that belong  to E, then n∈N An ∈ E. It is clear that every σ-algebra is also an algebra on S. Example 1.26. Let S be an arbitrary set and let C be the family of sets that consist of sets that either countable or are complements of countable sets. We claim that (S, C) is a measurable space. Note that S ∈ C because S is the complement of ∅, which is countable. Next, if A ∈ C is countable, A¯ is a complement of a countable set, so A¯ ∈ C; otherwise, if A is not countable, then it is the complement of a countable set, which means that A¯ is countable, so A¯ ∈ C. Let A, B be two sets of C. If both are countable, then A ∪ B ∈ C. If ¯ ¯ are countable, then A ∪ ¯ B = A¯ ∩ B, ¯ so A ∪ B ∈ C, because it A and B ¯ is countable, then has a countable complement. If A is countable and B ¯ is countable because is a subset of B. ¯ Therefore, A ∪ B ∈ C as a A¯ ∩ B complement of a countable sets. The case when A¯ and B are countable is treated similarly. Thus, in any case, the union of two sets of C belongs to C. Finally, we have to prove that if {Ai | i ∈ N} is a family of sets included  in C, then the set A = i∈N Ai belongs to C. Indeed, let us split the set I into I  and I  , where i ∈ I  if the set Ai is countable and i ∈ I  if the  complement Ai = S − Ai is countable. Note that both A = i∈I  Ai and

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 36

Mathematical Analysis for Machine Learning and Data Mining

36

 A = i∈I  Ai are countable sets, and that A = A ∪ A . Since both A and A belong to C, it follows that A ∈ C. We give now a technical result that concerns σ-algebras. Theorem 1.33. Let (S, E) be a measurable space and let {Ui ∈ E | i ∈ N} a family of sets from E. There exists a family of sets {Vi ∈ E | i ∈ N} ⊆ E that satisfies the following conditions: (i) the sets Vi are pairwise disjoint, that is, if i, j ∈ N and i = j, then Vi ∩ Vj = ∅; (ii) Vi ⊆ Ui for i ∈ N;   (iii) {Vi | i ∈ N} = {Ui | i ∈ N}. Proof.

The sets Vn are defined inductively by: V0 = U0 , Vi = Ui −

 {Uj | 0  j  i − 1}.

It is immediate that Vi ∈ E for i ∈ N and that the first two conditions of the theorem are satisfied; we prove the last part of the theorem.  For x ∈ {Ui | i ∈ N} let ix be the least i such that x ∈ Ui ; clearly,   x ∈ Uj for j < i, so x ∈ Vi . Thus, {Ui | i ∈ N} ⊆ {Vi | i ∈ N}. The  reverse inclusion follows from the fact that Vi ⊆ Ui for every i ∈ N. Next, we describe the σ-algebra generated by a countable partition of a set. Theorem 1.34. Let π = {Bi | i ∈ I} be a countable partition of a set S. In other words we assume that the set of indices I of the blocks of π is countable. The σ-algebra generated by π is the collection of sets:    Bi | J ⊆ I . Eπ = i∈J

Proof.

We have: π⊆{

 i∈J

Bi | J ⊆ I} ⊆ Eπ .

 We claim that the collection E = { i∈J Bi | J ⊆ I} is a σ-algebra.   Indeed, we have S = {B | B ∈ π}, so S ∈ { i∈J Bi | J ⊆ I}. If  A = {Bi | i ∈ J}, then A¯ = {Bi | i ∈ I − J}, which shows that  A¯ ∈ { i∈J Bi | J ⊆ I}.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Preliminaries

page 37

37

 Finally, suppose that A0 , . . . , An , . . . belong to E, so Ak = {Bi | i ∈    Jk }, where Jk ⊆ I for k ∈ N. Then, k0 Ak = {Bi | i ∈ k0 Jk },   which implies that k0 Ak ∈ { i∈J Bi | J ⊆ I}. Thus, E is a σ-algebra, so Eπ ⊆ E. The converse inclusion is immediate, so E = Eπ , which completes the argument.  Theorem 1.35. Every σ-algebra E is a monotone collection. Proof. Let (Cn ) be an increasing sequence of sets in E. We have  n∈N Cn ∈ E by the definition of σ-algebras. If (Dn ) is a decreasing sequence of sets in E, then (S − Dn ) is an in creasing sequence in E, so by the first part, n∈N (S − Dn ) ∈ E. This implies   Dn = (S − Dn ) ∈ E, n∈N

n∈N

hence E is indeed a monotone collection.



Theorem 1.36. An algebra of sets on a set S that is a monotone collection is a σ-algebra on S. Proof. Let E be an algebra of sets on a set S and let {Uj | j ∈ N} be a countable family of sets included in E. Define the sequence W = n (W0 , W1 , . . .) by Wn = j=0 Uj . It is immediate that W is a monotone   sequence and that j∈N Uj = j∈N Wj . Since E is a monotone collection   it follows that j∈N Wj ∈ E, so j∈N Uj belongs to E, so E is a σ-algebra.  Theorem 1.37. Let E be a collection of algebras (σ-algebras) of sets on a  set S. Then, the collection E is a an algebra (a σ-algebra) on the set S. Proof. We give the argument for a collection of algebras of sets E = {Ei |  i ∈ I} on S. Since S ∈ Ei for every i ∈ I it follows that S ∈ {Ei | i ∈ I}.  Suppose that A ∈ E. Since A ∈ Ei for every i ∈ I it follows that  A¯ ∈ Ei for every i ∈ I, which implies that A¯ ∈ {Ei | i ∈ I}.  Finally, if {Ai | 1  i  n} ∈ {Ei | i ∈ I}, it is easy to see that  n {Ei | i ∈ I}. i=1 Ai ∈ A similar argument can be applied to σ-algebras.  Thus, the class of algebras and the class of σ-algebras define closure systems on P(P(S)). We denote by Kalg and Kσ-alg the closure operators generated by these two classes, respectively. By the properties of closure operators, if C is a collection of subsets of S, Kalg (C) and Kσ-alg (C) are the algebra and the σ-algebra generated

May 2, 2018 11:28

38

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 38

Mathematical Analysis for Machine Learning and Data Mining

by C, respectively. Consequently, if C ⊆ Kσ-alg (C ), we have Kσ-alg (C) ⊆ Kσ-alg (C ). Example 1.27. Let A be a subset of the set S. The σ-algebra generated ¯ S}. by the collection {A} is {∅, A, A, Example 1.28. Let π = {Bi | i ∈ I} be a countable partition of a set S. The σ-algebra generated by π is:       Eπ = Bi J ⊆ I . i∈J

In other words, the σ-algebra generated by π consists of sets that are countable unions of blocks of π. Such sets are referred to as π-saturated sets. Every block Bi belongs to Eπ , so π ⊆ Eπ . To verify that Eπ is a σ-algebra note first that we have S ∈ Eπ since   S = i∈I Bi . If A ∈ Epi , then A = i∈J Bi for some subset J of I, so  A¯ = i∈I−J Bi , which shows that A¯ ∈ Eπ . Let {A | ∈ L} be a family of sets included in Eπ . For each set A there exists a set J such that  A = {Bi | i ∈ J }. Therefore,        A = Bi i ∈ J , ∈L ∈L  which shows that ∈L A ∈ Eπ . This proves that Eπ is a σ-algebra. Moreover, any σ-algebra on S that includes π also includes Eπ , which concludes the argument. If π is a finite partition of S, then the algebra generated by π consists of unions of blocks of π. For a subset U of a set S and a ∈ {0, 1} denote by U a the set:  U if a = 1, a U = S − U if a = 0. This notation allows us to generalize Example 1.28. Theorem 1.38. Let S be a set and let U = {U1 , . . . , Un } be a finite collection of subsets of S. For (a1 , . . . , an ) ∈ {0, 1}n denote by U a1 ···an the set U a1 ···an = U1a1 ∩ · · · ∩ Unan . If E is the algebra of sets generated by U, then the collection A = {U a1 ···an | Ua1 ···an = ∅}, is the set of minimal non-empty elements of E and every element of E is a union of a subcollection of A.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Preliminaries

page 39

39

Proof. Observe that any two distinct sets in A are disjoint and that  A = S. In other words, A is a partition of S. Therefore, the set of all unions of subcollections of A is a algebra of sets E that contains each of the sets Bi . Therefore, E ⊆ E . On the other hand, we have E ⊆ E, so  E = E . Theorem 1.39. (Monotone Collection Theorem) If A is an algebra of subsets of a set S, then Kmon (A) = Kσ-alg (A). Proof. Since every σ-algebra is a monotone class we need to prove that C = Kmon (A) is a σ-algebra. For C ∈ C define M(C) = {B ∈ C | C ∩ B, C − B, B − C ∈ C}.  If (Bn ) is an ascending sequence in M(C) and B = n∈N Bn we have  (C ∩ Bn ) = C ∩ B, n∈N



(Bn − C) = B − C,

n∈N



(C − Bn ) = C − B,

n∈N

which show that M(C) is a monotone collection. Since C is a monotone collection, it follows that C ∩ B, C − B, B − C ∈ C, so B ∈ M(C), which means M(C) is a monotone class for all C ∈ C. If A ∈ A ⊆ C, we have A ∩ B, A − B, B − A ∈ A ⊆ C for all B ∈ A, hence A ⊆ M(A) ⊆ C. Since C = Kmon (A), we have M(A) = C for every A ∈ A. Let B ∈ C. Note that A ∈ M(B) if and only of B ∈ M(A). Therefore, since M(A) = C for all A ∈ A implies A ⊆ M(B) ⊆ C for all B ∈ C. Since C = Kmon (A) and M(B) is a monotone class, we conclude that M(B) = C for all B ∈ C. Therefore, C is closed under complements (since S ∈ A ⊆ C), finite intersections and countable unions, which implies that C is a σalgebra.  Definition 1.44. Let C be a collection of sets and let T be a set. The restriction of C to T is the collection C T defined by: C T = {C ∩ T | C ∈ C}.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 40

Mathematical Analysis for Machine Learning and Data Mining

40

Theorem 1.40. Let E be a σ-algebra of subsets of a set S and let T be a non-empty subset of S. The collection E T = {U ∩ T | U ∈ E} is a σ-algebra of subsets of T . Proof.

Let A ∈ E T . There exists E ∈ E such that A = E ∩ T , hence T − A = T ∩ A = T ∩ (E ∪ T ) = T ∩ E ∈ E T

because E ∈ E. Suppose now that {An | n ∈ N} is a countable collection of sets in E T . There exists a countable collection of sets {En | n ∈ N} ⊆ E such that An = En ∩ T . Therefore, ⎞ ⎛    An = (En ∩ T ) = ⎝ En ⎠ ∩ T ∈ E T because



n∈N n∈N En

n∈N

n∈N

∈ E.



Corollary 1.3. Let C be a collection of subsets of a set S and let T be a non-empty subset of S. We have Kσ-alg (C T ) = Kσ-alg (C) T . Proof.

It is immediate that C T ⊆ Kσ-alg (C) T . This implies Kσ-alg (C T ) ⊆ Kσ-alg (C) T .

Let S = {U ⊆ S | U ∩T ∈ Kσ-alg (C T )}. We claim that S is a σ-algebra of subsets of S. Suppose that U ∈ S, so U ∩ T ∈ Kσ-alg (C T ). Since (S − U ) ∩ T = T − (U ∩ T ) ∈ Kσ-alg (C T ), it follows that S − U ∈ Kσ-alg (C T ). Suppose now that U1 , U2 , . . . is a sequence of sets in S, hence U1 ∩T, U2 ∩ T, . . . is a sequence of sets in Kσ-alg (C T ). Therefore, ⎛ ⎞   ⎝ Un ⎠ ∩ T = (Un ∩ T ) ∈ Kσ-alg (C T ), 

n1

n1

hence n1 Un ∈ S, so S is indeed a σ-algebra. If U ∈ C, then U ∩ T ∈ C T ⊆ Kσ-alg (C T ), hence U ∈ S. Therefore, C ⊆ S, hence Kσ-alg (C) ⊆ S. If B ∈ Kσ-alg (C) T we have B = A ∩ T for some A ∈ Kσ-alg (C) ⊆ S and, therefore, B ∈ Kσ-alg (C T ). This yields Kσ-alg (C) T ⊆ Kσ-alg (C T ).



May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Preliminaries

b3234-main

page 41

41

Definition 1.45. Let S be a set. A π-system on S is a collection of sets C such that U, V ∈ C implies U ∩ V ∈ C. A Dynkin system on a set S is a collection D of subsets of S that satisfies the following conditions: (i) S ∈ D, (ii) U, V ∈ D and U ⊆ V implies V − U ∈ D; (iii) if T = (T0 , T1 , . . .) is an increasing sequence of subsets of S that  belong to D, then i∈N Ti belongs to D. ˆ Example 1.29. The collections of subsets of R: ˆ −∞  a  b  ∞}, I = {(a, b) | a, b ∈ R, ˆ −∞  a  b < ∞}, G = {(a, b] | a, b ∈ R, ˆ −∞ < a  b  ∞}, H = {[a, b) | a, b ∈ R, ˆ −∞ < a  b < ∞}, K = {[a, b] | a, b ∈ R, are π-systems. Indeed, if (a, b), (c, d) ∈ I, then (a, b) ∩ (c, d) = (min{a, c}, max{b, d}} ∈ I. Similar observations can be made about the other collections. Theorem 1.41. A collection D of subsets of a set S is a Dynkin system if and only if the following conditions are satisfied: (i) S ∈ D; (ii) if U ∈ D, then U = S − U belongs to D; (iii) if U = (U0 , U1 , . . .) is a sequence of pairwise disjoint subsets of S  that belong to D, then n∈N Un belongs to D. Proof. Let D be a Dynkin system on S. Since S ∈ D and U ⊆ S for each U ∈ D, it follows that S − U = U ∈ D. Let U = (U0 , U1 , . . .) is a sequence  of pairwise disjoint subsets of S that belong to D, and let Vn = kn for   n ∈ N. It is clear that n∈N Un = n∈N Vn and, since (Vn ) is an increasing  sequence of sets in D, it follows that n∈N Un ∈ D. Conversely, suppose now that D satisfies the conditions of the theorem. If U, V ∈ D and U ⊆ V , then U and V are two disjoint sets, so U ∪V ∈ D by the third condition of the theorem. Therefore, U ∪ V = V −U belongs to S. Finally, suppose that (T0 , T1 , . . .) is an increasing sequence of subsets of S that belong to D. Then, the sequence (T0 , T1 −T2 , T2 −T3 , . . .) is a sequence  of disjoint sets that belong to D, hence the set T0 ∪ n0 (Tn+1 − Tn ) =   n0 Tn belongs to D. Thus, D is indeed a Dynkin system.

May 2, 2018 11:28

42

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 42

Mathematical Analysis for Machine Learning and Data Mining

It is easy to verify that the collection of π-systems and the collection of Dynkin systems on a set S are both closure systems. Their corresponding closure operators are denoted by Kπ and KDyn , respectively. Theorem 1.42. If C is a π-system on a set S, then Kσ-alg (C) = KDyn (C). Proof. Let D = KDyn (C) and let E = Kσ-alg (C). Since a σ-algebra is a Dynkin system, it follows that D ⊆ E. To prove the reverse inclusion, we begin by showing that D = KDyn (C) is closed with respect to finite intersections. Consider the collections of sets defined by D1 = {A ∈ D | A ∩ C ∈ D for each C ∈ C}, and D2 = {B ∈ D | B ∩ A ∈ D for each A ∈ D}. We have D ⊂ D1 and D ⊂ D2 and both D1 and D2 are Dynkin systems. Since C ⊆ D, S ∈ D1 . Taking into account that (A − B) ∩ C = (A ∩ C) − (B ∩ C) ⎞   ⎝ An ⎠ ∩ C = (An ∩ C), ⎛

n∈N

n∈N

it follows that D1 is closed with respect to set difference and to unions of increasing sequences of sets, so it is a Dynkin system. Since C ⊆ D1 , it follows that D = KDyn (C) ⊆ D1 , which implies D = D1 because D1 consists of sets that belong to D. Thus, D1 is a Dynkin system. The equality D1 = D implies that C ⊆ D2 . By an argument similar to the one used for D1 , it follows that D2 is a Dynkin system, and then, D2 = D. Thus, D is closed under finite intersections and, therefore, it is an algebra of sets. Since D is also a monotone collection it is also a σ-algebra that includes C. Therefore, Kσ-alg (C) ⊆ D,  hence Kσ-alg (C) = D. Theorem 1.43. A collection C of subsets of a set S is a σ-algebra of sets if and only if it is both a π-system and a Dynkin system. Proof. If C is a σ-algebra, then it is clearly both a Dynkin system and a π-system. Conversely, suppose that C is both a Dynkin system and a π-system, so KDyn (C) = C. By Theorem 1.42, we have C = Kσ-alg (C), which means that C is a σ-algebra. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Preliminaries

1.8

9in x 6in

b3234-main

page 43

43

Dissimilarity and Metrics

The notion of a metric was introduced in mathematics by the French mathematician Maurice Ren´e Fr´echet3 in [58] as an abstraction of the notion of distance between two points. In this chapter, we explore the notion of metric and the related notion of metric space, as well as a number of generalizations and specializations of these notions. Dissimilarities are functions that allow us to evaluate the extent to which data objects are different. Definition 1.46. A dissimilarity on a set S is a function d : S 2 −→ R0 satisfying the following conditions: (i) d(x, x) = 0 for all x ∈ S; (ii) d(x, y) = d(y, x) for all x, y ∈ S. The pair (S, d) is a dissimilarity space. The set of dissimilarities defined on a set S is denoted by DS . Let (S, d) be a dissimilarity space and let S(x, y) be the set of all nonnull sequences s = (s1 , . . . , sn ) ∈ Seq(S) such that s1 = x and sn = y. The d-amplitude of s is the number ampd (s) = max{d(si , si+1 ) | 1  i  n−1}. Next we introduce the notion of extended dissimilarity by allowing ∞ as a value of a dissimilarity. Definition 1.47. Let S be a set. An extended dissimilarity on S is a ˆ 0 that satisfies the conditions of Definition 1.46. function d : S 2 −→ R The pair (S, d) is an extended dissimilarity space. Additional properties may be satisfied by dissimilarities. A nonexhaustive list is given next. (1) d(x, y) = 0 implies d(x, z) = d(y, z) for every x, y, z ∈ S (evenness); (2) d(x, y) = 0 implies x = y for every x, y (definiteness); (3) d(x, y)  d(x, z) + d(z, y) for every x, y, z (triangular inequality); (4) d(x, y)  max{d(x, z), d(z, y)} for every x, y, z (the ultrametric inequality); (5) d(x, y) + d(u, v)  max{d(x, u) + d(y, v), d(x, v) + d(y, u)} for every x, y, u, v (Buneman’s inequality, also known as the four-point condition). 3 Fr´ echet was born on September 2nd 1878 in Maligny, France and died on June 4th 1973 in Paris. He made major contributions in topology, introduced metric spaces, and is considered as one of the founders of modern analysis. Fr´ echet studied at the ´ Ecole Normale Sup´erieure and taught at the Universities of Poitieres, Strassbourg, and at Sorbonne.

May 2, 2018 11:28

44

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 44

Mathematical Analysis for Machine Learning and Data Mining

If d : S 2 −→ R is a function that satisfies the properties of dissimilarities and the triangular inequality, then the values of d are non-negative numbers. Indeed, by taking x = y in the triangular inequality, we have 0 = d(x, x)  d(x, z) + d(z, x) = 2d(x, z), for every z ∈ S. Various connections exist among these properties. As an example, we can show the following statement. Theorem 1.44. Both the triangular inequality and definiteness imply evenness. Proof. Suppose that d is a dissimilarity that satisfies the triangular inequality, and let x, y ∈ S be such that d(x, y) = 0. By the triangular inequality, we have both d(x, z)  d(x, y) + d(y, z) = d(y, z) and d(y, z)  d(y, x) + d(x, z) = d(x, z) because d(y, x) = d(x, y) = 0. Thus, d(x, z) = d(y, z) for every z ∈ S. We leave it to the reader to prove the second part of the statement.  We denote the set of definite dissimilarities on a set S by DS . Further notations are introduced shortly for other types of dissimilarities. Definition 1.48. A dissimilarity d ∈ DS is (i) a pseudo-metric if it satisfies the triangular inequality; (ii) a metric if it satisfies the definiteness property and the triangular inequality; (iii) a tree metric if it satisfies the definiteness property and Buneman’s inequality; (iv) an ultrametric if it satisfies the definiteness property and the ultrametric inequality. The set of metrics on a set S is denoted by MS . The sets of tree metrics and ultrametrics on a set S are denoted by TS and US , respectively. If d is a metric or an ultrametric on a set S, then (S, d) is a metric space or an ultrametric space, respectively. If d is a metric defined on a set S and x, y ∈ S, we refer to the number d(x, y) as the d-distance between x and y or simply the distance between x and y whenever d is clearly understood from context. Thus, a function d : S 2 −→ R0 is a metric if it has the following properties: (i) d(x, y) = 0 if and only if x = y for x, y ∈ S;

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Preliminaries

b3234-main

page 45

45

(ii) d(x, y) = d(y, x) for x, y ∈ S; (iii) d(x, y)  d(x, z) + d(z, y) for x, y, z ∈ S. If the first property is replaced by the weaker requirement that d(x, x) = 0 for x ∈ S, then we refer to d as a semimetric on S. Thus, if d is a semimetric d(x, y) = 0 does not necessarily imply x = y and we can have for two distinct elements x, y of S, d(x, y) = 0. The notions of extended metric and extended ultrametric are defined starting from the notion of extended dissimilarity using the same process as in the definitions of metrics and ultrametrics. A collection of semimetrics on a set S is said to be a gauge on S. Example 1.30. Let S be a non-empty set. Define the mapping d : S 2 −→ R0 by  1 if u = v, d(u, v) = 0 otherwise, for x, y ∈ S. It is clear that d satisfies the definiteness property. The triangular inequality, d(x, y)  d(x, z) + d(z, y) is satisfied if x = y. Therefore, suppose that x = y, so d(x, y) = 1. Then, for every z ∈ S, we have at least one of the inequalities x = z or z = y, so at least one of the numbers d(x, z) or d(z, y) equals 1. Thus d satisfies the triangular inequality. The metric d introduced here is the discrete metric on S. Example 1.31. Consider the mapping dh : (Seqn (S))2 −→ R0 defined by dh (p, q) = |{i | 0  i  n − 1 and p(i) = q(i)}| for all sequences p, q of length n on the set S. Clearly, dh is a dissimilarity that is both even and definite. Moreover, it satisfies the triangular inequality. Indeed, let p, q, r be three sequences of length n on the set S. If p(i) = q(i), then r(i) must be distinct from at least one of p(i) and q(i). Therefore, {i | 0  i  n − 1 and p(i) = q(i)} ⊆ {i | 0  i  n − 1 and p(i) = r(i)} ∪ {i | 0  i  n − 1 and r(i) = q(i)}, which implies the triangular inequality. This is a rather rudimentary distance known as the Hamming distance on Seqn (S). If we need to compare sequences of unequal length, we can use an extended metric dh defined by  |{i | 0  i  |x| − 1, xi = yi } if |x| = |y|,  dh (x, y) = ∞ if |x| = |y|.

May 2, 2018 11:28

46

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 46

Mathematical Analysis for Machine Learning and Data Mining

Example 1.32. Define the mapping d : R × R −→ R0 as d(x, y) = |x − y| for x, y ∈ R. It is clear that d(x, y) = 0 if and only if x = y and that d(x, y) = d(y, x) for x, y ∈ S; To prove the triangular inequality suppose that x  y  z. Then, d(x, z) + d(z, y) = z − x + z − y = 2z − x − y and we have 2z − x − y > y−x = d(x, y) because z > y. The triangular inequality is similarly satisfied no matter what the relative order of x, y, z is. We use frequently use the notions of closed sphere and open sphere. Definition 1.49. Let (S, d) be a metric space. The closed sphere centered in x ∈ S of radius r is the set Bd [x, r] = {y ∈ S|d(x, y)  r}. The open sphere centered in x ∈ S of radius r is the set Bd (x, r) = {y ∈ S|d(x, y) < r}. The spherical surface centered in x ∈ S of radius r is the set Sd (x, r) = {y ∈ S | d(x, y) = r}. If the metric d is clear from context we drop the subscript d and replace Bd [x, r], Bd (x, r), and Sd (x, r) by B[x, r], B(x, r), and Sd (x, r), respectively. Definition 1.50. Let (S, d) be a metric space. The diameter of a subset U of S is the number diamS,d(U ) = sup{d(x, y) | x, y ∈ U }. The set U is bounded if diamS,d (U ) is finite. The diameter of the metric space (S, d) is the number diamS,d = sup{d(x, y) | x, y ∈ S}. If the metric space is clear from the context, then we denote the diameter of a subset U just by diam(U ). If (S, d) is a finite metric space, then diamS,d = max{d(x, y) | x, y ∈ S}. A notion close to the notion of dissimilarity is given next. Definition 1.51. A similarity on a set S is a function s : S 2 −→ [0, 1] satisfying the following conditions: (i) s(x, x) = 1 for all x ∈ S; (ii) s(x, y) = s(y, x) for all x, y ∈ S. If s(x, y) = 1 implies x = y, then s is a definite similarity. The pair (S, s) is referred to a similarity space.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Preliminaries

page 47

47

In other words, the similarity between an object x and itself is the largest; also, the similarity is symmetric. Example 1.33. Let d : S 2 −→ R0 be a dissimilarity on S. The function d2 (x,y)

s : S 2 −→ [0, 1] defined by s(x, y) = e− 2σ2 for x, y ∈ S and σ ∈ R is easily seen to be a similarity. Note that s is definite if and only if d is definite. Definition 1.52. Let (S, d) and (T, d ) be two metric spaces. An isometry between these spaces is a function f : S −→ T such that d (f (x), f (y)) = d(x, y) for every x, y ∈ S. If an isometry exists between (S, d) and (T, d ) we say that these metric spaces are isometric. Note that if f : S −→ T is an isometry, then f (x) = f (y) implies d(f (x), f (y)) = d(x, y) = 0, which yields x = y for x, y ∈ S. Therefore, every isometry is injective and a surjective isometry is a bijection. 1.9

Elementary Combinatorics

Definition 1.53. A permutation of a set S is a bijection f : S −→ S. A permutation f of a finite set S = {s0 , . . . , sn−1 } is completely described by the sequence (f (s0 ), . . . , f (sn−1 )). No two distinct components of such a sequence may be equal because of the injectivity of f , and all elements of the set S appear in this sequence because f is surjective. Therefore, the number of permutations equals the number of such sequences, which allows us to conclude that there are n(n − 1) · · · 2 · 1 permutations of a finite set S with |S| = n. The number n(n − 1) · · · 2 · 1 is denoted by n!. This notation is extended by defining 0! = 1 to capture the fact that there exists exactly one bijection of ∅, namely the empty mapping. The set of permutations of the set S = {1, . . . , n} is denoted by PERMn . If f ∈ PERMn is such a permutation, we write   1 ··· i ··· n f: , a1 · · · ai · · · an where ai = f (i) for 1  i  n. To simplify the notation, we specify f just by the sequence (a1 , . . . , ai , . . . , an ).

May 2, 2018 11:28

48

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 48

Mathematical Analysis for Machine Learning and Data Mining

Another way to describe a permutation f of a finite set S = {a1 , . . . , an } is by using a {0, 1}-matrix Pf ∈ {0, 1}n×n defined by  1 if f (ai ) = aj , (Pf )ij = 0 otherwise, for 1  i, j  n. Example 1.34. The matrix Pf of permutation f ∈ PERM5 defined as   12345 f: , 24531 is

⎛

01 ⎜0 0 ⎜ Pf = ⎜ ⎜0 0 ⎝0 0 10

⎞ 000 0 1 0⎟ ⎟ 0 0 1⎟ ⎟. 1 0 0⎠ 000

Note that for every row and every column of Pf contains exactly one 1. Conversely, if P is a {0, 1}n×n-matrix such that every row and every column of Pf contains exactly one 1, there exists a permutation f ∈ PERMn such that P = Pf . Indeed, in this case, f can be defined as f (i) = j if Pij = 1. We refer to matrices P ∈ {0, 1}n×n having exactly one 1 entry in each row and each column as permutation matrices. Theorem 1.45. Let f, g ∈ PERMn . We have Pf g = Pg Pf . n Proof. We have (Pg Pf )ij = h=1 (Pg )ih (Pf )hj . Since (Pg )ih = 1 if and only if g(i) = h, it follows that (Pg Pf )ij = (Pf )g(i)j . Therefore, (Pg Pf )ij = 1 if and only if f (g(i)) = j, that is, if and only if (Pf g )ij = 1, which shows  that Pf g = Pg Pf . Definition 1.54. A stochastic matrix is a matrix A ∈ Rn×n that satisfies the conditions: (i) aij ∈ [0, 1] for 1  i  n and 1  j  n; n (ii) j=1 aij = 1 for each i, 1  i  n. A matrix A ∈ Rn×n is doubly stochastic if both A and its transpose A are stochastic matrices.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Preliminaries

page 49

49

It is clear that every permutation matrix Pf is a doubly stochastic matrix. Let S = {1, . . . , n} be a finite set. For every x ∈ {1, . . . , n} and f ∈ PERMn there exists k ∈ N such that x = f k (x). If k is the least number with this property, the set {x, f (x), . . . , f k−1 (x)} is the cycle of x and is denoted by Cf,x . The number |Cf,x | is the length of the cycle. Cycles of length 1 are said to be trivial. Note that each pair of elements f i (x) and f j (x) of Cf,x are distinct for 0  i, j  |Cf,x | − 1. If z ∈ Cf,x and |Cf,x | = k, then z = f j (x) for some j, 0  j  k − 1. Since x = f k (x), it follows that x = f k−j (z), which shows that x ∈ Cf,z . Consequently, Cf,x = Cf,z . Thus, the cycles of a permutation f ∈ PERMn form a partition πf of {1, . . . , n}. Definition 1.55. A k-cyclic permutation of {1, . . . , n} is a permutation such that πf consists of a cycle of length k, (j1 , . . . , jk ) and a number of n − k cycles of length 1. A transposition of {1, . . . , n} is a 2-cyclic permutation. Note that if f is a transposition of {1, . . . , n}, then f 2 = 1S . Theorem 1.46. Let f be a permutation in PERMn , and πf = {Cf,x1 , . . . , Cf,xm } be the cycle partition associated to f . Define the cyclic permutations g1 , . . . , gm of {1, . . . , n} as  f (t) if t ∈ Cf,xp , gp (t) = t otherwise. Then, gp gq = gq gp for every p, q such that 1  p, q  m. Proof. Observe first that u ∈ Cf,x if and only if f (u) ∈ Cf,x for any cycle Cf,x . We can assume that p = q. Then, the cycles Cf,xp and Cf,xq are disjoint. If u ∈ Cf,xp ∪ Cf,xq , then we can write gp (gq (u)) = gp (u) = u and gq (gp (u)) = gq (u) = u. Suppose now that u ∈ Cf,xp −Cf,xq . We have gp (gq (u)) = gp (u) = f (u). On the other hand, gq (gp (u)) = gq (f (u)) = f (u) because f (u) ∈ Cf,xq . Thus, gp (gq (u)) = gq (gp (u)). The case where u ∈ Cf,xq − Cf,xp is treated similarly. Also, note that Cf,xp ∩ Cf,xq = ∅, so, in all cases, we have  gp (gq (x)) = gq (gp (u)).

May 2, 2018 11:28

50

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 50

Mathematical Analysis for Machine Learning and Data Mining

The set of cycles {g1 , . . . , gm } is the cyclic decomposition of the permutation f . Definition 1.56. A standard transposition is a transposition that changes the places of two adjacent elements. Example 1.35. The permutation f ∈ PERM5 given by   12345 f: 13245 is a standard transposition of the set {1, 2, 3, 4, 5}. On the other hand, the permutation   12345 g: 15342 is a transposition but not a standard transposition of the same set because the pair of elements involved is not consecutive. If f ∈ PERMn is specified by the sequence (a1 , . . . , an ), we refer to each pair (ai , aj ) such that i < j and ai > aj as an inversion of the permutation f . The set of all such inversions is denoted by INV(f ). The number of elements of INV(f ) is denoted by inv(f ). A descent of a permutation f ∈ PERMn is a number j such that 1  j  n − 1 and aj > aj+1 . The set of descents of f is denoted by D(f ). Example 1.36. Let f ∈ PERM6 be:   123456 f: . 425163 We have INV(f ) = {(4, 2), (4, 1), (4, 3), (2, 1), (5, 1), (5, 3), (6, 3)} and inv(f ) = 7. Furthermore, D(f ) = {1, 3, 5}. It is easy to see that the following conditions are equivalent for a permutation f ∈ PERMn : (i) f = 1S ; (ii) inv(f ) = 0; (iii) D(f ) = ∅. Theorem 1.47. Every permutation f ∈ PERMn can be written as a composition of transpositions.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Preliminaries

page 51

51

Proof. If D(f ) = ∅, then f = 1S and the statement is vacuous. Suppose therefore that D(f ) = ∅, and let j ∈ D(f ), which means that (aj , aj+1 ) is an inversion f . Let g be the standard transposition that exchanges aj and aj+1 . It is clear that inv(gf ) = inv(f ) − 1. Thus, if gi are the transpositions that correspond to all standard inversions of f for 1  i  p = inv(f ), it follows that gp · · · g1 f has 0 inversions and, as observed above, gp · · · g1 f = 1S . Since g 2 = 1S for every transposition g, we have f = gp · · · g1 , which gives the desired conclusion.  Theorem 1.48. If f ∈ PERMn , then inv(f ) equals the least number of standard transpositions, and the number of standard transpositions involved in any other factorization of f as a product of standard transposition differs from inv(f ) by an even number. Proof. Let f = hq · · · h1 be a factorization of f as a product of standard transpositions. Then, h1 · · · hq f = 1S and we can define the sequence of permutations fl = hl · · · h1 f for 1  l  q. Since each hi is a standard transposition, we have inv(fl+1 ) − inv(fl ) = 1 or inv(fl+1 ) − inv(fl ) = −1. If |{l | 1  l  q − 1 and inv(fl+1 ) − inv(fl ) = 1}| = r, then |{l | 1  l  q − 1 and inv(fl+1 ) − inv(fl ) = −1}| = q − r, so inv(f ) + r − (q − r) = 0, which means that q = inv(f ) + 2r. This implies the desired conclusion.  An important characteristic of permutations is their parity . Namely, the permutation parity is defined as the parity of the number of their inversions: a permutation f ∈ PERMn is even (odd) if inv(f ) is an even (odd) number. Theorem 1.48 implies that any factorization of a permutation as a product m standard transpositions determines whether the permutation is odd or even. Note that any transposition is an odd permutation. Indeed, if f ∈ PERMn is a transposition of i and j, where i < j we have f = (1, 2, . . . , i − 1, j, i + 1, . . . , j − 1, i, j + 1, . . . , n). The number j generates j − i inversions, and each of the numbers i + 1, . . . , j − 1 generates one inversion because they are followed by i. Thus, the total number of inversions is j − i + (j − i − 1) = 2(j − i) − 1, which is obviously an odd number. Theorem 1.49. A cyclic permutation f of length k is the composition of k − 1 transpositions.

May 2, 2018 11:28

52

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 52

Mathematical Analysis for Machine Learning and Data Mining

Proof. Let (j1 , . . . , jk ) be the cycle of length k of f . It is immediate that f is the product of the k − 1 transpositions (j1 , j2 ), (j2 , j3 ), . . . ,  (jk−1 , j1 ). Thus, the parity of a cyclic permutation of even length is odd. Corollary 1.4. Let f ∈ PERMn be a permutation that has c cycles of length for  1. The parity of f is the parity of the number c2 + c4 + · · · ; in other words, the parity of a permutation is given by the parity of the number of its even cycles. Proof. By Theorem 1.49 a cyclic transposition of length is the composition of − 1 transpositions. Thus, if f has c cycles of length , then f is  a product of 1 c ( − 1) transpositions. It is clear that the parity of this sum is determined by those terms where − 1 is impair. Thus the parity of f is given by the parity of c2 + 3c4 + 5c6 + · · · and this equals the parity  of c2 + c4 + c6 + · · · . Let S be a non-empty finite set, S = {s1 , . . . , sn }. We need to evaluate the number of injective functions of the form f : {1, . . . , m} −→ S. To this end we will define such an injection by examining the number of choices we have when f is specified in increasing order of its values f (1), . . . , f (m). Note that f (1) can be chosen as any of the elements of S, so we have n choices; for f (2) we have n− 1 choices since f (2) must be distinct from f (1) and so on. Assuming that we defined f (1), . . . , f (k) there are n − k choices left for f (k + 1) among the elements of S − {f (1), . . . , f (k)}. Thus, there exist n(n − 1) · · · (n − m + 1) injections of the form f : {1, . . . , m} −→ S n! . and this number can be written as (n−m)! Let T be a subset of S such that |T | = m, so m  n Note that there are exactly m! injections of the form f : {1, . . . , m} −→ S such that n! subsets T of S havf ({1, . . . , m}) = T . Therefore, there exist (n−m)!m! n n! and it is known ing m elements. The number (n−m)!m! is denoted by m as a binomial coefficient for reasons that will become apparent later.  n is extended to m = 0 by taking n0 = 1 for every The value of m n ∈ N. This corresponds to the fact that there is exactly one empty subset of a set S. Multinomial coefficients generalize binomial coefficients. If n, p1 , . . . , pk are natural numbers such that n = p1 + · · · + pk we have the following definition of a multinomial coefficient:   n! n . = p1 p2 · · · pk p1 ! p2 ! · · · pk !

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Preliminaries

page 53

53

It is immediate that each multinomial coefficient can be written as a product of binomial coefficients:   p1 + p2 + · · · + pk p1 p2 · · · pk       p1 + · · · + pk pk−2 + pk−1 + pk pk−1 + pk pk = ··· . (1.6) p1 pk−2 pk−1 pk Let p(x) = (x + b1 )(x + b2 ) · · · (x + bn ) be a polynomial of degree n. The coefficient of xn−m in p is the sum of all products of the form bi1 bi2 · · · bim , where {i1 , i2 , . . ., im } is an m-element subset of the set {1, . . . , n}. Recall n such subsets. that there exist m For 1  m  n the function fm : Rn −→ R defined by  fm (b1 , . . . , bn ) = {bi1 bi2 · · · bim | {i1 , . . . , im } ∈ Pm ({1, . . . , n)} is the mth symmetric function in b1 , . . . , bn . If the roots of a polynomial p of degree n, p(x) = xn + a1 xn−1 + · · · + ak xn−k + · · · + an are the complex numbers λ1 , . . . , λn , then p(x) = (x − λ1 ) · · · (x − λn ), and by the previous argument, fm (λ1 , . . . , λn ) = (−1)m am . If we have b1 = b2 = · · · = bn = y in the polynomial p(x) = (x + b1 )(x + b2 ) · · · (x + bn ), we have p(x) = (x + y)n and the coefficient of xn−m is n m and we can write: m y n    n n−m m y . (x + y) = x m m=0 n

This is the well-known Newton’s binomial formula. Newton’s binomial formula can be extended to the multinomial formula: (x1 + · · · + xk )n     k     n p1 p2 pk  = pi = n . x x · · · xk p1 , p2 , . . . , pk ∈ N,  p1 p2 · · · pk 1 2 i=1 The proof of the multinomial formula can be made by induction on k, where k  1.

May 2, 2018 11:28

54

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 54

Mathematical Analysis for Machine Learning and Data Mining

For the base step, k = 1, the equality is immediate. For the induction step suppose that the equality holds for k. This allows us to write (x1 + · ·· + xk + xk+1 )n = (x1 + · · · + (xk + xk+1 ))n    n pk−1 xp1 xp2 · · · xk−1 = (xk + xk+1 )p ) p1 p2 · · · pk 1 2   k−1    pi + p = n p1 , p2 , . . . , pk−1 , p ∈ N,  i=1   

    p n pk−1 pk+1  p pk + pk+1 = p xp11 xp22 · · · xk−1 xkk xk+1 =  pk pk+1 p1 p2 . . . pk   k−1    pi + p = n . p1 , p2 , . . . , pk−1 , p ∈ N,  i=1

Note that for pk , pk+1 , p ∈ N and pk + pk+1 = p we have      p n n = , p1 p2 · · · pk−1 p p1 p2 · · · pk−1 pk pk+1 pk pk+1 due to the definition of multinomial coefficients. This allows us to write    n pk+1 n (x1 + · · · + xk + xk+1 ) = xp11 xp22 · · · xpkk xk+1 p p · · · p p 1 2 k k+1   k+1   pi = n , p1 , p2 , . . . , pk , pk+1 ∈ N,  i=1 which concludes the argument. If we take x1 = · · · = xk in the multinomial formula we obtain the sum of multinomial coefficients:    k    n  pi = n = k n . p1 , p2 , . . . , pk ∈ N, p1 p2 · · · pk  i=1

For binomial coefficients this amounts to taking k = 2, p1 = p and p2 = n−p for 0  p  n and we have n    n = 2n . p p=0 Exercises and Supplements (1) Let C, D be two collections of sets. Prove that: (a) 

 (C ∪ D),    C∩ D = {C ∩ D | C ∈ C and D ∈ D}; C∪



D=

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Preliminaries

55

(b) if D is non-empty, then for each D ∈ D, define FD = C ∈ C}. Prove that 

C−



D=

page 55





{C − D |

{FD | D ∈ D};

if C and D are both non-empty, then 

C∪



D=

 {C ∪ D | C ∈ C and D ∈ D}.

(2) Let A, B be two sets. Prove that A × B ⊆ P(P(A ∪ B)). (3) Let U be a set and let A and B be subsets of U . Prove that (a) The equation A ∩ X = B has a solution X ∈ P(U ) if and only if B ⊆ A. Show that, in this case, X is a solution if and only if there is a P ⊆ U − A with X = B ∪ P . (b) Prove that the equation A ∪ X = B has a solution in X if and only if A ⊆ B. In this case, show that X is a solution if and only if B − A ⊆ X ⊆ B. (4) For each inequality, give examples of sets A, B, and C that satisfy the inequality (and thereby show that various possible distributive laws do not hold). A ∪ (B − C) = (A ∪ B) − (A ∪ C), A ∪ (B × C) = (A ∪ B) × (A ∪ C), A ∩ (B × C) = (A ∩ B) × (A ∩ C), A − (B ∪ C) = (A − B) ∪ (A − C), A − (B ∩ C) = (A − B) ∩ (A − C), A − (B − C) = (A − B) − (A − C), A − (B × C) = (A − B) × (A − C), (B × C) − A = (B − A) × (C − A), A × (B × C) = (A × B) × (A × C), (B × C) × A = (B × A) × (C × A). (5) Prove the following equalities for all sets A, B, C: (a) A ⊕ B = (A ∪ B) − (A ∩ B). (b) If B ⊆ A, then A ⊕ B = A − B. (c) If A and B are disjoint, then A ⊕ B = A ∪ B. (d) A ⊕ ∅ = ∅ ⊕ A = A. (e) A ⊕ B = ∅ if and only if A = B. (f) A ⊕ B = B ⊕ A. (g) A ⊕ (B ⊕ C) = (A ⊕ B) ⊕ C. (h) A ∩ (B ⊕ C) = (A ∩ B) ⊕ (A ∩ C). (i) (B ⊕ C) − A = (B − A) ⊕ (C − A).

May 2, 2018 11:28

56

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 56

Mathematical Analysis for Machine Learning and Data Mining

(j) (k) (l) (m) (n)

A × (B ⊕ C) = (A × B) ⊕ (A × C). (B ⊕ C) × A = (B × A) ⊕ (C × A). A ∪ B = A ⊕ (B ⊕ (A ∩ B)). If A − C = B − C, then A ⊕ B ⊆ C. If A ⊕ B = A ⊕ C, then B = C.

(6) For a subset U of a set S denote by U a the set  a

U =

U S −U

if a = 1, if a = 0.

Let C = {C1  , . . . , Cn } be a finite collection of subsets of a non-empty set S such that n j=1 Cj = S. Define the collection π as containing the nonempty subsets of the form C1a1 ∩ C2a2 ∩ · · · ∩ Cnan , where (a1 , . . . , an ) ∈ {0, 1}n . Prove that: (a) π is a partition of S; (b) every set Ci ∈ C is π-saturated. Solution: For x ∈ S let Cx = {Ci1 , . . . , Cip } be the subcollection of C that consists of those sets in C that contain x. Since C is a cover for S it is clear that Cx = ∅ for every x ∈ S.   Let Kx = pk=1 Cik . Since x ∈ Kx it follows that x∈S Kx = S. We n aj have Kx = j=1 Cj , where  aj =

1 0

if x ∈ Cj , if x ∈  Cj ,

for 1  j  n. This shows that if Kx = Ky we have Kx ∩ Ky = ∅, which proves the first part of this supplement.  The fact that each set Ci ∈ C is π follows from the fact that Ci = {Kx | x ∈ Ci }. (7) Let S be a set and let U1 , . . . , Un be n subsets of S. Prove that S n − (U1 × U2 × · · · × Un ) n  S j−1 × (S − Uj ) × S n−j = j=1

= ((S − U1 ) × S n−1 ) ∪ (U1 × (S − U2 ) × S n−2 ) ∪ (U1 × · · · × Uk−1 × (S − Uk ) × S n−k ) ∪ (U1 × U2 × · · · × Un−1 × (S − Un ). Solution: Note that (x1 , . . . , xn ) ∈ S n − (U1 × U2 × · · · × Un ) amounts to the existence of j, 1  j  n such that xj ∈ Uj . This implies the first equality.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Preliminaries

b3234-main

page 57

57

To prove the second equality, for (x1 , . . . , xn ) ∈ S n −(U1 ×U2 ×· · ·×Un ) let k be the least number such that xk ∈ Uk (hence xk ∈ S −Uk )). Then, (x1 , . . . , xn ) ∈ U1 × · · · × Uk−1 × (S − Uk ) × S n−k and the second equality follows. (8) Let U, V be two subsets of a set S. Prove that: (a) 1U (x)  1V (x) for x ∈ S if and only if U ⊆ V ; (b) if f : S −→ [0, 1], then 1U (x)  f (x)  1V (x) for x ∈ S, if and only if f (x) = 1 for x ∈ U and f (x) = 0 if x ∈ S − V ; (c) if U, W are disjoint subsets of S and g : S −→ [0, 1] is a function, then 1U (x)  g(x)  1S−W (x) if and only if g(x) = 1 for x ∈ U and g(x) = 0 for x ∈ W . (9) Let ρ ⊆ X × Y be a relation on the finite sets X, Y . Prove that ρ is a function defined on X if and only if |X| = |ρ|. (10) Let U, V be two subsets of a set S. Prove that 1U ⊕V (x) = 1U (x) + 1V (x) − 2 · 1U (x)1V (x) for x ∈ S. An interval of R is a subset J of R that is either an open interval (a, b), a closed interval [a, b], or one of the semi-open intervals (a, b] or [a, b). Let (J) = b − a for any of these cases. (11) Let [a, b] be a closed interval in R and set {Jk | 1  k  n} be n intervals  of R such that [a, b] ⊆ n k=1 Jk . Prove that: 1[a,b] (x) 

n 

1Jk (x)

k=1

for x ∈ R. (12) Prove that if J = {J k | 1  k  n} is a collection of n open intervals of n n R such that [a, b] ⊆ k=1 Jk , then b − a  k=1 (Jk ). Solution: Without loss of generality we may assume that every open intervals in J has a non-empty intersection with [a, b]. Let Jk = (ak , bk ) for 1  k  n. If n = 1 we have [a, b] ⊆ (a1 , b1 ), which implies immediately the inequality. Suppose that the inequality holds for collection of fewer than n open intervals and let J be a collection of n open intervals that covers [a, b].  Since [a, b] ⊆ n k=1 Jk , there exists an open interval in J that is not included in [a, b]. Without loss of generality we may assume that this interval is (an , bn ). Since each of the intervals of J has a non-empty intersection with [a, b], we have either [a, b] ⊆ [a, an ] ∪ (an , bn ) or [a, b] ⊆ [a, bn ] ∪ (an , bn ) depending whether an ∈ [a, b] or bn ∈ [a, b], respectively.

May 2, 2018 11:28

58

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 58

Mathematical Analysis for Machine Learning and Data Mining

 In the first case, b−a  an −a+((an , bn )) and [a, an ] ⊆ n−1 k=1 (ak , bk ). n−1 By the inductivehypothesis we have an − a  k=1 ((ak , bk )), which implies b − a  n k=1 ((ak , bk )). The second case can be dealt with in a similar manner. (13) Let {sn | n ∈ N, n  1} be a family of sequences, where sn = (xn1 , . . . , xnm , . . .). Arrange the elements of the set {xnm | n  1, m  1} in a rectangular infinite array x11 x21 x31 .. . xn1 .. .

x12 x22 x32 .. . xn2 .. .

x13 x23 x33 .. . xn3 .. .

··· ··· ··· .. . ··· .. .

x1m x2m x3m .. . xnm .. .

··· ··· ··· .. . ··· .. .

The kth diagonal Dk of this array contains the elements xmn such that m+n = k +1. If the elements of the array are listed diagonally and from top to bottom, prove that there exists a bijection  : P × P −→ P such that xmn is placed at position (n, m) = 12 (n + m − 2)(n + m − 1) + n for m, n ∈ P. We will refer to the sequence obtained in this manner as the amalgam of the sequences sn . Solution: Note that xnm belongs to the diagonal Dn+m−1 . This diagonal is preceded by D1 , . . . , Dn+m−2 which contain a total number of 1 + 2 + · · · + (n + m − 2) = 12 (n + m − 2)(n + m − 1) elements. Since xnm occupies the nth place on the diagonal Dn+m−1 , it follows that (n, m) = 12 (n + m − 2)(n + m − 1) + n. Since the place of xnm is uniquely determined by n, m,  is indeed a bijection. (14) Let K, L be two subsets of R. Prove that if K s = Ls , then sup K = sup L and that K i = Li implies inf K = inf L. Solution: This follows immediately from {sup K} = K s ∩ (K s )i = Ls ∩ (Ls )i = {sup L}. A similar argument can be used for the second part. (15) Let K, L be two subsets of R. Prove that if K s = Li , then sup K  inf L. (16) Let U be a subset of R, and let b ∈ R be such that sup U  b. Prove that if a < b implies a  sup U , then b = sup U . Solution: Suppose that sup U < b. Then, there exists c ∈ R such that sup U < c < b, which implies c  sup U . This contradiction means that so such c exists, so sup U = b. (17) Let T be a subset of R. For a ∈ R define aT = {at | t ∈ T }. Prove that (a) if a  0, sup aT = a sup T and inf aT = a inf T ;

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Preliminaries

page 59

59

(b) if a  0, sup aT = a inf T and inf aT = a sup T . (18) Let U, V be two subsets of R such that U ⊆ V . Prove that: (a) if sup U , sup V exist, then sup U  sup V ; (b) if inf U , inf V exist, then inf U  inf V . (19) For U, V ⊆ R define U + V = {u + v | u ∈ U, v ∈ V }. Prove that if U and V are non-empty then sup(U + V ) = sup U + sup V, inf(U + V ) = inf U + inf V. Solution: Note that (U + V )s = ∅ if and only if U s = ∅ and V s = ∅. Therefore, sup(U + V ) exists if and only if both sup U and sup V exist. If u ∈ U and v ∈ V , then u + v  sup U + sup V , hence sup U + sup V is an upper bound of U + V . Therefore, sup(U + V )  sup U + sup V . Let  > 0. There exist u ∈ U and v ∈ V such that sup U − 2 < u and sup V − 2 < v, hence sup U + sup V −   u + v for every  > 0, which implies sup(U + V )  sup U + sup V . Thus, sup(U + V ) = sup U + sup V . The second equality has a similar argument. (20) Let C be a subset of R and let f, g : C −→ R be two real-valued functions. Prove that inf f (x) + inf g(x)  inf (f + g)(x),

x∈C

x∈C

x∈C

sup f (x) + sup g(x)  sup (f + g)(x). x∈C

x∈C

x∈C

Solution: Let p = inf x∈C f (x) and q = inf x∈C g(x). We have p  f (x) and q  g(x) for every x ∈ C. Therefore, p + q  f (x) + g(x) = (f + g)(x) for every x ∈ C, hence p + q  inf x∈C f + g. The second inequality has a similar argument based on the definition of sup. Let C be a set and let f : C −→ R be a function. Define sup f = sup{f (x) | x ∈ C} and inf f = inf{f (x) | x ∈ C}. C

C

Also, if f, g : C −→ R we write f  g if f (x)  g(x) for every x ∈ C. (21) Let C be a set and let f : C −→ R and g : C −→ R be two functions. Prove that if f  g and supC is finite, then supC f  supC g. Also, prove that if inf C f is finite, then inf C f  inf C g. (22) Give an example of two functions f, g : C −→ R such that f  g but supC f  inf C g.

May 2, 2018 11:28

60

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 60

Mathematical Analysis for Machine Learning and Data Mining

(23) Let f : C −→ R be a function such that supC f is finite and let a ∈ R. Prove that supC (af ) = a supC f and inf C (af ) = a inf C f . (24) Let C be a set and let f, g : C −→ R be two real-valued functions. Prove that inf f (x) + inf g(x)  inf (f + g)(x),

x∈C

x∈C

x∈C

sup f (x) + sup g(x)  sup (f + g)(x). x∈C

x∈C

x∈C

Solution: Let p = inf x∈C f (x) and q = inf x∈C g(x). We have p  f (x) and q  g(x) for every x ∈ C. Therefore, p + q  f (x) + g(x) = (f + g)(x) for every x ∈ C, hence p + q  inf x∈C f + g. The second inequality has a similar argument based on the definition of sup. (25) Let C be a set and let f, g : C −→ R be two real-valued bounded functions. Prove that | supC f − supC g|  supC |f − g|. (26) Let K1 , K2 be two closure operators on a set S and let C be a collection of subsets of S such that K1 (K2 (C)) = K2 (C) and K2 (K1 (C)) = K1 (C). Prove that K1 (C) = K2 (C). Solution: Since K1 (K2 (C)) = K2 (C). (27) Let A be an algebra of subsets of a set S. Prove that if A is closed under formation of unions of ascending sequences of sets or under the formation of intersections of descending sequences of sets, then A is a σ-algebra. (28) Prove that if (Un ) is a sequence of sets in a σ-algebra of subsets of a set S, then lim sup Un and lim inf Un belong to S. (29) Prove that if A is an algebra of subsets of a set S, then the collection Aσ is closed with respect to intersection. (30) Let E be a σ-algebra of subsets of a set S and let T = {t1 , . . . , tn } be a finite set. Prove that the collection E  T = {E ∪ U | E ∈ E and U ⊆ T } is a σ-algebra of sets on the S ∪ T . Solution: If E ∪ U , E ∪ U = (S − E) ∪ (T − U ), so E ∪ U ∈ E  T . collection of Suppose now that {E  n ∪ Un | n ∈ N}is a countable (E ∪ U ) = E ∪ subsets of E  T . Then n n n n∈ N n∈N n∈N Un ∈ E  T ,  because n∈N En ∈ E and n∈N Un ∈ T since E is a σ-algebra and T is a finite set. (31) Let E be an algebra of sets. Prove that if E is closed under countable increasing unions, then E is a σ-algebra. (32) Let C be a collection of subsets of a set S. Kσ -alg (C).

Prove that Kalg (C) ⊆

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Preliminaries

b3234-main

page 61

61

(33) Let S be a set and let C be a collection of subsets of S. Define the collections of sets: C = C ∪ {S − T | T ∈ C}, 



D

D ⊆ C , C = 



 

D D ⊆ C . C = Prove that C equals the σ-algebra generated by C. (34) Let S, T be two sets and let f : S −→ T be a function. Prove that if E is a σ-algebra on T , then {f −1 (V ) | V ∈ E } is a σ-algebra on A. A semi-ring of sets is a collection S of sets that satisfies the following conditions: (i) ∅ ∈ S; (ii) if U, V ∈ S, then U ∩ V ∈ S; (iii) if U, V ∈ S and V ⊆ U , then there exists  a finite collection U of pairwise disjoint sets in S such that U − V = U. (35) An open-closed interval in Rn is a set of the form G = (a1 , b1 ] × · · · × (an , bn ]. Prove that the collection {∅} ∪ {G | G is an open-closed interval in Rn } is a semi-ring on R. Solution: Let G = (a1 , b1 ] × · · · × (an , bn ] be an open-closed interval such that G ⊆ G. We have (aj , bj ] ⊆ (aj , bj ], hence (aj , bj ] = (aj , aj ] ∪ (aj , bj ] ∪ (bj , bj ], where (aj , aj ], (aj , bj ], (bj , bj ] are disjoint. Consider the open-closed intervals R1 × · · · × Rn , where Rj is one of (aj , aj ], (aj , bj ], (bj , bj ]. One of these intervals equals G ; the union of the rest of them equals G − G , so the class of open-closed intervals (together with ∅) is indeed a semi-ring. (36) Prove that if S and T are semi-rings on the sets S and T , respectively, then the collection Y = {U × V | U ∈ S, V ∈ T} is a semi-ring on S × T . Let S be a set. A ring of sets on S is a non-empty family of subsets E of S that satisfies the following conditions: (i) if U, V ∈ E, then U ∩ V ∈ E; (ii) if U, V ∈ E, then U − V ∈ E. If S ∈ E, then E is an algebra of sets on S.

May 2, 2018 11:28

62

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 62

Mathematical Analysis for Machine Learning and Data Mining

(37) Let R be the collection of subsets of R2 that can be written as a finite union of disjoint intervals. Prove that R is a ring. (38) Let B the collection of all bounded subsets of R. Prove that B is a ring but not an algebra. Also, show that the collection I = {[x, y] | x, y ∈ [a, b]} is a semi-ring but not a ring. (39) Prove that the collection of rings of subsets of a set S is a closure system. (40) Prove that if S is a semi-ring of sets, then the collection R of all finite unions of sets in S is a ring of sets. (41) Let S be a semi-ring of sets. Prove that if U, U1 , . . . , Un ∈ S, then there are pairwise disjoint sets V1 , . . . , Vm in S such that U ∩ U1 ∩ · · · ∩ Un = V1 ∪ · · · ∪ Vm . Solution: The argument is by induction on n  1. The base case, n = 1 is immediate from the definition of semi-rings. Suppose that the statement holds to n. Then, U ∩U1 ∩· · ·∩Un ∩Un+1 = (V1 ∩Un+1 )∪· · ·∪ (Vm ∩ Um+1 ). By applying the case n = 1 to each of the sets Vj ∩ Um+1 , the desired conclusion follows. (42) Prove that if E is an infinite σ-algebra on a set S, then E is not countable. Solution: Suppose that E is an infinite σ-algebra of subsets of S. Clearly, this implies that S is infinite. Suppose  that E is countable. Let f : S −→ E be a function such that f (x) = {U ∈ E | x ∈ U }. Note that x ∈ f (x). Since σ-algebras are closed with respect to countable intersections the function f is welldefined. Suppose that f (x) ∩ f (y) = ∅. If x ∈ f (y), then x ∈ f (x) − f (y) ⊂ f (x), which is a contradiction because f (x) − f (y) ∈ E. Therefore, x ∈ f (y) and, similarly, y ∈ f (x). Since f maps a point to the least element of E containing that element, it follows that x ∈ f (x) ⊆ f (y) and y ∈ f (y) ⊆ f (x), so f (x) = f (y). Thus, the collection {f (x) | x ∈ S} is a partition πf of S.  Each set U ∈ E can be written as U = {f (u) | u ∈ U }. Therefore, πf cannot be finite because otherwise S would be finite. Since πf is infinite, the set P({f (x) | x ∈ S}) is not countable. The function φ : E −→ P({f (x) | x ∈ S}) given by φ(E) = {f (x) | x ∈ E} for every E ∈ E is surjective, which implies that E is not countable. (43) Let (S, d) be a metric space and let f : S −→ R be a function. Define the relation on S as x y if f (x) + d(x, y)  f (y). Define M (x) = {t ∈ S | t  x}. Prove that: (a) is a partial order on S; (b) y x if and only if M (y) ⊆ M (x); (c) y is a minimal element of the poset (S, ) if and only if M (y) = {y}.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Preliminaries

b3234-main

page 63

63

(44) Let (S, d) be a metric space and let k be a positive number. Prove that the mapping dk : S × S −→ R defined by dk (x, y) = min{d(x, y), k} is a metric on S. Solution: It is immediate to verify that dk (x, y) = 0 if and only if x = y and that dk (x, y) = dk (y, x) for x, y ∈ S. We need to prove only the triangular inequality. Note that dk (x, y)  d(x, y) for x, y ∈ S. If at least one of the numbers dk (x, z), dk (z, y) equals k, then the triangular inequality dk (x, y)  dk (x, z) + d(z, y) is satisfied. Otherwise, we have dk (x, z) = d(x, z) and dk (z, y) = d(z, y) and, since dk (x, y)  d(x, y)  d(x, z) + d(z, y), the triangular inequality is satisfied. (45) The arctan function is usually defined on R and ranges in the interval ˆ by defining arctan +∞ = π/2 and (−π/2, π/2). Extend arctan to R ˆ by ˆ×R arctan −∞ = −π/2. Prove that the function d defined on R ˆ. ˆ is a metric on R d(x, y) = | arctan x − arctan y| for x, y ∈ R (46) Let p be a prime number. For n ∈ N, n  1 define p (n) as the exponent of the prime p in the factorization on n as a product of prime numbers. Prove that (a) p (mn) = p (m) + p (n) for m, n ∈ N and m, n  1. (b) If q = s m is a rational number, where s ∈ {−1, 1} and m, n ∈ N, n m, n  1, define p (q) = p (m) − p (n). Prove that p (qr) = p (q) + p (r) for any rational numbers q, r. (c) Define the mapping dp : Q2 −→ R as  dp (q, r) = for q, r ∈ Q. q, r, s ∈ Q.

p−p (q,r) 0

if q = r, if q = r

Prove that dp (q, r)  max{dp (q, s), dp (s, r)} for

(47) Let (S, d) be a metric space and let {B(xi , r) | 1  i  n} be a collection of spheres in S. Prove that the set C = n i=1 B(xi , r) is a bounded set and diamS,d (C)  max{d(xi , xj ) | 1  i, j  n} + 2r. (48) Let (S, d) be a metric space. Prove that if h : R −→ R is a differentiable function such that h(0) = 0, h (x) > 0 for x > 0, and h (x) is decreasing on [0, ∞), then hd is a metric on S. (49) Prove that for h ∈ R and h > −1 we have (1 + h)n  1 + nh for n ∈ N. This inequality is known as the Bernoulli inequality. (50) Let φ ∈ PERMn be

1 ··· i ··· n , a1 · · · ai · · · an

φ:

May 2, 2018 11:28

64

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 64

Mathematical Analysis for Machine Learning and Data Mining

and let vp (φ) = |{(ik , il ) | il = p, k < l, ik > il } be the number of inversions of φ that have p as their second component, for 1  p  n. Prove that (a) vp  n − p for 1  p  n; (b) for every sequence of numbers (v1 , . . . , vn ) ∈ Nn such that vp  n − p for 1  p  n there exists a unique permutation φ that has (v1 , . . . , vn ) ∈ Nn as its sequence of inversions.  (51) Let p be the polynomial p(x1 , . . . , xn ) = i y if xi > yi for 1  i  n. If x  0n , we say that x is non-negative; if x ≥ 0n , x is said to be semipositive, and if x > 0n , then x is positive. The relation “” is a partial order on Rn ; the other two relations, “≥” and “>” are strict partial orders on the same set because they lack reflexivity. In some cases, when x, y ∈ Rn and ξ is one of the relations {, ≥, >} we write xξn y in order to stress that the relation involves vectors in Rn . Further, if x, y ∈ Rn , u, v ∈ Rp , and ξ, ζ are symbols in the set {, ≥, >} we write   !  x ξ u y ζ v if and only if x ξ u and y ζ v. Definition 2.2. Let L be an F-linear space and let K be a subset of L. A linear combination of K is a finite sum of the form c1 x1 + · · · + cm xm , where c1 , . . . , cm ∈ F. The empty linear combination corresponds to m = 0 and is defined to be the zero vector 0L of L. The notion of linear combination is used for defining linearly independent sets in linear spaces. Definition 2.3. A subset K of L is linearly independent if for every linear combination c1 x1 + · · · + cm xm of elements of K, the equality c1 x1 + · · · + cm xm = 0L implies c1 = · · · = cm = 0. If K is not linearly independent, we refer to K as a linearly dependent set. If x = 0L , then the set {x} is linearly independent. Of course, the set {0L } is not linearly independent because 1 0L = 0L . If K is a linearly independent subset of a linear space, then any subset of K is linearly independent.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Linear Spaces

Example 2.4. Let

page 69

69

⎛ ⎞ 0 ⎜ .. ⎟ ⎜.⎟ ⎜ ⎟ ⎟ ei = ⎜ ⎜1⎟ ⎜.⎟ ⎝ .. ⎠ 0

be a vector in R that has a unique non-zero component equal to 1 in place i, where 1  i  n. The set E = {e1 , . . . , en } is linearly independent. Indeed, suppose that c1 e1 + · · · + cn en = 0n . This is equivalent to ⎛ ⎞ ⎛ ⎞ c1 0 ⎜ .. ⎟ ⎜ .. ⎟ ⎝ . ⎠ = ⎝.⎠ n

cn

0

that is, with c1 = · · · = cn = 0. Thus, E is linearly independent. Theorem 2.1. Let L be an F-linear space. A subset K of L is linearly inm dependent if and only if for every linear combination x = i=1 ci xi (where xi ∈ K for 1  i  m) the coefficients ci are uniquely determined. Proof. Let K be linearly independent subset of L, and suppose that x = c1 x1 + · · · + cm xm = c1 x1 + · · · + cm xm , where x1 , . . . , xm ∈ K. If there m exists i such that ci = ci , then i=1 (ci − ci )xi = 0L and ci − ci = 0, which contradicts the linear independence of K. Conversely, suppose that K is not linearly independent, that is, there exists a linear combination d1 x1 + · · · + dm xm = 0L such that at least one m m di is not 0. Then, if x = i=1 ci xi we also have x = i=1 (ci + di )xi and at there exists i such that ci = ci + di , which contradicts the uniqueness of the coefficients of x.  Definition 2.4. A subset S of an F-linear space L spans the space L (or S generates the linear space) if every x ∈ L is a linear combination of S. A Hamel basis (or, simply a basis) of the linear space L is a linearly independent subset that spans the linear space. In view of Theorem 2.1, a set B is a Hamel basis if every x ∈ L can be written uniquely as a linear combination of elements of B. Definition 2.5. A subspace of an F-linear space L is a non-empty subset U of L such that x, y ∈ U implies x + y ∈ U and a · x ∈ U for every a ∈ F.

May 2, 2018 11:28

70

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 70

Mathematical Analysis for Machine Learning and Data Mining

If U is a subspace of an F-linear space, then U itself can be regarded as a real linear space and various notions introduced for real linear spaces are applicable to U . The set {0L } is a subspace of any F-linear space L referred to as the trivial subspace of L. Note that {0L } is included in every subspace of L. If {Ui | i ∈ I} is a non-empty collection of subspaces of an F-linear  space L, then {Ui | i ∈ I} is also a subspace of L. Also, L itself is a subspace, so the collection of subspaces of an F-linear space is a closure system. If U is a subset of L and Ksubs is the corresponding closure operator for the closure system of linear subspaces, then we say that U is spanning the subspace Ksubs (U ) of L. The subspace Ksubs (U ) is said to be spanned or generated by U . Frequently, we denote this subspace by  U. Theorem 2.2. If U is a subset of a linear space L, then  U consists of all linear combinations of elements of U . Proof. The set T of linear combinations of elements of U is clearly a subspace of L such that U ⊆ T ⊆  U. Since  U is the minimal subspace that contains U , it follows that  U = T .  Theorem 2.3. Let P, Q be two subspaces of an F-linear space L. Then, the set P + Q = {u + v | u ∈ P, v ∈ Q} is a subspace of L and  P ∪ Q = P + Q. Proof. Suppose that x, y ∈ P + Q, where x = u1 + v1 , y = u2 + v2 , where u1 , u2 ∈ P and v1 , v2 ∈ Q. Since P and Q are subspaces of L, we have u1 + u2 ∈ P and v1 + v2 ∈ Q. This, in turn, implies x + y = u1 + v1 + u2 + v2 = (u1 + u2 ) + (v1 + v2 ) ∈ P + Q. If a ∈ F is a scalar, then au1 ∈ P and av1 ∈ Q, so ax = au1 + av1 ∈ Q + P, which shows that P + Q is indeed a subspace of L. Note that P ∪ Q ⊆ P + Q, so  P ∪ Q ⊆ P + Q. The converse inclusion follows from the fact that every element of P + Q is a linear combination of the elements of P ∪ Q. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Linear Spaces

b3234-main

page 71

71

Theorem 2.4. The following statements that concern an F-linear space L are equivalent: (i) the finite set B = {x1 , . . . , xn } is spanning L and B is minimal with this property; (ii) B is a finite basis for L; (iii) the finite set B is linearly independent, and B is maximal with this property. Proof. (i) implies (ii): We need to prove that B is linearly independent. Suppose that this is not the case. Then, there exist c1 , . . . , cn ∈ F such that c1 x1 + · · · + cn xn = 0 and at least one of c1 , . . . , cn , say ci , is non-zero. Then, xi = − cc1i x1 − · · · − ccni xn , and this implies that B − {xi } also spans the linear space, thus contradicting the minimality of B. (ii) implies (i): Let B be a finite basis. Suppose that B  is a proper subset of S that spans L. Then, if z ∈ B − B  , z is a linear combination of elements of B  , which contradicts the fact that B is a basis. We leave to the reader the proof of the equivalence between (ii) and (iii).  Corollary 2.1. Every F-linear space L that is spanned by a finite subset has a finite basis. Further, if B is a finite basis for an F-linear space L, then each finite subset U of L such that |U | = |B| + 1 is linearly dependent. Proof.

This statement follows directly from Theorem 2.4.



Corollary 2.2. If B and B  are two finite bases for an F-linear space L, then |B| = |B  |. Proof. If B is a finite basis, then |B| is the maximum number of linearly independent elements in L. Thus, |B  |  |B|. Reversing the roles of B and  B  , we obtain |B|  |B  |, so |B| = |B  |. Thus, the number of elements of a finite basis of L is a characteristic of L and does not depend on any particular basis. Definition 2.6. An F-linear space L is n-dimensional if there exists a basis of L such that |B| = n. The number n is the dimension of L and is denoted by dim(L). An F-linear space L is finite-dimensional if there exists n ∈ N such that dim(L) = n. If a linear space V is not finite-dimensional than we say that dim(V ) is infinite.

May 2, 2018 11:28

72

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 72

Mathematical Analysis for Machine Learning and Data Mining

Example 2.5. Let S be a non-empty, finite set. The linear space of functions CS has dimension |S|. Indeed, for each t ∈ S consider the function ft : S −→ C defined by  1 if s = t, ft (s) = 0 otherwise for t ∈ S. If S = {t1 , . . . , tn }, then the set of functions {ft1 , . . . , ftn } is linearly independent, for if c1 ft1 (s) + · · · + cn ftn (s) = 0, then by taking s = tk we obtain ck = 0 for any k, 1  k  n. Furthermore, if f : S −→ C n is a function and f (ti ) = ci , then f = i=1 ci fti , so {ft1 , . . . , ftn } is a basis for CS . Theorem 2.5. Let L be a finite-dimensional F-linear space and let U = {u1 , . . . , uk } be a linearly independent subset of L. There exists an extension of U that is a basis of L. Proof. If  U = L, then U is a basis of L. If this is not the case, let w1 ∈ L −  U. The set U ∪ {w1 } is linearly independent and we have the strict inclusion  U ⊂  U ∪ {w1 }. The subspace  U ∪ {w} is (k + 1)dimensional. This argument can be repeated no more than n − k times, where n = dim(L). Thus, U ∪ {w1 , . . . , wn−k } is a basis for L that extends U.  Definition 2.7. Let L be an F -linear space and let U, V be subspaces of L. The sum of the subspaces U and V is the set U + V defined by: U + V = {u + v | u ∈ U and v ∈ V }. It is easy to verify that U + V is also a subspace of L. Theorem 2.6. Let U, V be two subspaces of a finite-dimensional F-linear space L. We have dim(U + V ) + dim(U ∩ V ) = dim(U ) + dim(V ). Proof. Let {w1 , . . . , wk } be a basis for U ∩ V , where k = dim(U ∩ V ) which can be extended to a basis {w1 , . . . , wk , uk+1 , . . . , up } for U and to a basis {w1 , . . . , wk , vk+1 , . . . , vq } for V . Define B = {w1 , . . . , wk , uk+1 , . . . , up , vk+1 , . . . , vq }. It is clear that  B = U + V . Suppose that there exist c1 , . . . , cp+q−k such that c1 w1 + · · · + ck wk + ck+1 uk+1 + · · · + cp up + cp+1 vk+1 + · · · + cp+q−k vq = 0.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Linear Spaces

page 73

73

The last equality implies c1 w1 + · · · + ck wk + ck+1 uk+1 + · · · + cp up = −cp+1 vk+1 − · · · − cp+q−k vq . Therefore, c1 w1 + · · · + ck wk + ck+1 uk+1 + · · · + cp up belongs to U ∩ V , which implies ck+1 = · · · = cp = 0. Since c1 w1 + · · · + ck wk + cp+1 vk+1 + · · · + cp+q−k vq = 0, and {w1 , . . . , wk , vk+1 , . . . , vq } is a basis for V , it follows that c1 = · · · = ck = cp+1 = · · · = cp+q−k = 0. This allows to conclude that dim(U + V ) = p + q − k and this implies the equality of the theorem.  Example 2.6. Let Pol(R) be the set of one-argument polynomials with real coefficients. This set is a real linear space under the addition of polynomials and multiplication by a real number. Every polynomial in Pol(R), p(x) = an xn + an−1 xn−1 + · · · + a1 x + a0 is a finite linear combination of the set of polynomials B = {1, x, . . . , xm , . . .}. Also if z(x) is the zero polynomial (that is, p is a polynomial that equals 0 for every x ∈ R), then all coefficients of z are 0, which implies that the set B is linearly independent and therefore it is a basis for Pol(R). This shows that dim(Pol(R)) is infinite. Using Zorn’ Lemma it is possible to show that every linear space has a basis. However, this is an existence statement and, in many cases, such a basis cannot be effectively specified. Theorem 2.7. Every non-trivial linear space has a basis. Proof. Let L be a non-trivial linear space. Consider the collection L of linearly independent sets ordered by inclusion. Let C be a chain of linearly  independent sets and let Cˆ = C. We claim that Cˆ is itself linearly independent. Suppose that  a x = 0, where at least one of a is not 0, x ∈ C i i i for i ∈ I and i∈I i i I is a finite set. Since C is a chain one of the sets Ci , say Ci∗ contains xi for each i ∈ I. This contradicts the linear independence of Ci∗ , which implies that Cˆ is linearly independent. Thus shows that every chain of linearly independent sets has an upper bound. By Zorn’s Lemma, the set of linearly independent sets has a maximal element M . Let x ∈ L. The set M ∪ {x} is either linearly independent or x ∈ M . In either case, x is a linear combination of M . Moreover, x can be represented uniquely as a linear combination of M for, otherwise, we would have 0 as a non-trivial linear combination of elements of M , contradicting the linear independence of M . 

May 2, 2018 11:28

74

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 74

Mathematical Analysis for Machine Learning and Data Mining

Corollary 2.3. Let L be a linear space. Every linearly independent subset of L can be extended to a base of L. Proof. This Theorem 2.7. 2.3

statement

follows

immediately

from

the

proof

of 

Linear Operators and Functionals

Linear operators between linear spaces are functions that are compatible with the algebraic operations of linear spaces. Definition 2.8. Let L and K be two F-linear spaces. A linear operator is a function h : L −→ K such that h(ax + by) = ah(x) + bh(y) for every a, b ∈ F and x, y ∈ L. An isomorphism between the F-linear spaces L and K is a linear operator between these spaces which is a bijection. In the special case, when K is the field F, linear operators are referred to as linear functionals. The set of linear operators between two linear spaces L and K is denoted by L(L, K). We denote by 1L the identity linear operator, 1L : L −→ L defined by 1L (x) = x for x ∈ L. The operator 0L : L −→ L maps every x ∈ L into 0L . Theorem 2.8. Let L, K be two F-linear spaces and let h : L −→ K be a linear operator. The sets Null(h) = {x ∈ L | h(x) = 0K }, Img(h) = {y ∈ K | y = h(x) for some x ∈ L} are subspaces of L and K, respectively. Proof. Let u, v ∈ Null(h). Since h(u) = h(v) = 0K it follows that h(au + bv) = ah(u) + bh(v) = 0K , for a, b ∈ F, so au + bv ∈ Null(h). This shows that Null(h) is indeed a subspace of L. Let now s, t ∈ Img(h). There exist x, y ∈ L such that s = h(x) and t = h(y). Therefore, as + bt = ah(x) + bh(y) = h(ax + by), hence as + bt ∈ Img(h). This implies that Img(h) is a subspace of K.  Null(h) and Img(h) are referred to as the null subspace and the image subspace of h, respectively. Theorem 2.9. Let L, K be two F-linear spaces and let h : L −→ K be a linear operator. The operator h is injective if and only if Null(h) = {0L }.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Linear Spaces

b3234-main

page 75

75

Proof. Let h be an injective linear operator. If x = 0L and x ∈ Null(h), then h(x) = h(0L ) = 0K , so h is not injective. Thus, Null(h) = {0L }. Conversely, suppose that Null(h) = {0L }. If h(u) = h(v) for u, v ∈ L, then h(u − v) = 0K , so u − v ∈ Null(h), which implies u − v = 0L , that is, u = v.  In other words, h is injective if and only if h(x) = 0K implies x = 0L . Corollary 2.4. Let L and K be two linear spaces having 0L and 0K as their zero elements, respectively. A linear operator h ∈ L(L, K) is an isomorphism if and only if h(x) = 0K implies x = 0L and h(L) = K. Proof.

This is an immediate consequence of Theorem 2.9.



An endomorphism of an F-linear space L is a linear operator h : L −→ L. The set of endomorphisms of L is denoted by L(L). Often, we refer to endomorphisms of L as linear operators on L. Theorem 2.10. Let L and K be two finite-dimensional linear spaces. The linear operator h ∈ L(L, K) is injective if and only if its surjective. Proof. Suppose that h is an injective linear operator and that {u1 , . . . , un } is a basis in L. The injectivity of h means that we have n distinct vectors h(u1 ), . . . , h(un ) in K and the set {h(u1 ), . . . , h(un )} is linearly independent. Indeed, suppose that c1 h(u1 ) + · · · + cn h(un ) = 0K . Then h(c1 u1 + · · · + cn un ) = 0K , hence c1 u1 + · · · + cn un = 0L . This, in turn, implies c1 = · · · = cn = 0. Thus, {h(u1 ), . . . , h(un )} is a basis in K and every element y ∈ K can be written as y = b1 h(u1 ) + · · · + bn h(un ) = h(b1 u1 + · · · + hn un ) ∈ h(L), which allows us to conclude that h is surjective. Conversely, suppose that h is surjective. If v1 , . . . , vn is a basis in K, there exist u1 , . . . , un in L such that h(ui ) = vi for 1  i  n. Note that the set {u1 , . . . , un } is a basis for L. n Let x ∈ L and assume that h(x) = 0K . If x = i=1 ai ui , then h(x) = n n i=1 ai h(ui ) = i=1 ai vi = 0K , which implies a1 = · · · = ai = 0. This, in turn, means that x = 0L , so h is an injective function by Theorem 2.9.  Corollary 2.5. Let L and K be two finite-dimensional linear spaces and let h ∈ L(L, K). The following statements are equivalent: (i) h is an isomorphism; (ii) h is injective; (iii) h is surjective.

May 2, 2018 11:28

76

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 76

Mathematical Analysis for Machine Learning and Data Mining

Proof. The equivalence of the three statements follows immediately from Theorem 2.10.  Definition 2.9. A linear operator h : L −→ K between the linear spaces L and K is invertible if there exists a linear operator g : K −→ L such that gh = 1L and hg = 1K . In this case, we say that g is the inverse of h and we denote this operator by h−1 . Theorem 2.11. Let L be a finite-dimensional space and let h, g be two linear operators such that gh = 1L . We have hg = 1L , so g is the inverse of h. Proof. Suppose that gh = 1L . Then, for every x ∈ L we have x = g(h(x)), so g is surjective operator and, therefore, by Corollary 2.5, g is isomorphism. If h(u) = h(v) then u = g(h(u)) = g(h(v)) = v, so h is injective and, therefore, an isomorphism. This shows that h and g are inverse isomorphisms.  For an F-linear space, the function 0 : L −→ F given by 0 (x) = 0 is a linear functional referred to as the trivial functional or the zero functional. Theorem 2.12. Let L be an F-linear space. Any non-trivial linear functional on L is a surjective function. Proof. Let : L −→ F be a non-trivial functional on L. There exists x0 ) = 1. Therefore, for each x0 ∈ L such that (x0 ) = 0. Note that ( (x 0) a ∈ F we have   x0

a = a,

(x0 ) which shows that is surjective.



Theorem 2.13. Let L be an F-linear space. If x ∈ L is such that (x) = 0 for every linear functional , then x = 0L . Proof. Suppose that (x) = 0 for every linear functional and x = 0L . Since x = 0L , there exists a basis B of L that includes x. Every y ∈ L can be expressed as a linear combination of elements of B, so there exists c ∈ F such that y = · · · + cx + · · · . It is easy to see that the function g : L −→ F defined by g(y) = c is a linear functional on L. Since  g(x) = 1, this contradicts the hypothesis. Thus, x = 0L .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Linear Spaces

page 77

77

Example 2.7. Let L be an F-linear space and let B be a basis of L. For x ∈ B define ex : L −→ F as ex (y) = a if y = · · · + ax + · · · . The function ex is a linear functional on L. Example 2.8. Let L be an F-linear space. The translation generated by z ∈ L is the mapping tz : L −→ L defined by tz (x) = x + z. Any such mapping is a bijection on L but not a linear operator unless z = 0L . Its inverse is t−z . Example 2.9. Let L be an F-linear space. The homothety generated by a ∈ F is the mapping ha : L −→ L defined by ha (x) = ax. Any homothety is a linear operator; if a = 0, the inverse of ha is h a1 . Example 2.10. Let L be an F-linear space. The reflection is the mapping r : L −→ L defined by r(x) = −x. Any reflection is a linear operator. The inverse of r is r itself. If L, K are two F-linear spaces, then the set L(L, K) is never empty because the zero linear operator 0L : L −→ K given by 0L (x) = 0K for x ∈ L is always an element of L(L, K). Definition 2.10. Let L and K be two F -linear spaces. If f, g ∈ L(L, K), the sum f + g is defined by (f + g)(x) = f (x) + g(x) for x ∈ L. The sum of two linear operators is also a linear operator because (f + g)(ax + by) = f (ax + by) + g(ax + by) = af (x) + bf (y) + ag(x) + bg(y) = f (ax + by) + g(ax + by), for all a, b ∈ F and x, y ∈ L. Theorem 2.14. Let M, P, Q be three real linear spaces. The following properties of compositions of linear operators hold: (i) If f ∈ L(M, P ) and g ∈ L(P, Q), then gf ∈ L(M, Q). (ii) If f ∈ L(M, P ) and g0 , g1 ∈ L(P, Q), then f (g0 + g1 ) = f g0 + f g1 . (iii) If f0 , f1 ∈ L(M, P ) and g ∈ L(P, Q), then (f0 + f1 )g = f0 g + f1 g.

May 2, 2018 11:28

78

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 78

Mathematical Analysis for Machine Learning and Data Mining

Proof. We prove only the second part of the theorem and leave the proofs of the remaining parts to the reader. Let x ∈ M . Then, f (g0 + g1 )(x) = f ((g0 + g1 )(x)) = f (g0 (x) + g1 (x)) =  f (g0 (x)) + f (g1 (x)) for x ∈ M , which yields the desired equality. We leave to the reader to verify that for any linear spaces M and P the algebra (L(M, P ), {h0 , +, −}) is an Abelian group that has the zero linear operator h0 as its zero-ary operations and the addition of linear operators as its binary operation; the opposite of a linear operator h is the operator −h. Moreover, (L(M ), {h0 , iM , +, −, ·}) is a unitary ring, where the multiplication is defined as the composition of linear operators. If M and P are linear spaces, L(M, P ) is itself an linear space, where the multiplication of a linear operator h by a scalar c is the linear operator ch defined by (ch)(x) = c · h(x). Indeed, the operator ch is linear because (ch)(ax + by) = c(ah(x) + bh(y)) = cah(x) + cbh(y)) = ach(x) + bch(y)) = a(ch)(x) + b(ch)(y)), for every a, b, c ∈ F and x, y ∈ M . Definition 2.11. Let h be an endomorphism of a linear space M . The mth iteration of h (for m ∈ N) is defined as (i) h0 = iM ; (ii) hm+1 (x) = h(hm (x)) for m ∈ N. For every m  1, hm is an endomorphism of M ; this can be shown by a straightforward proof by induction on m. Definition 2.12. A linear operator h : L −→ L on a linear space L is idempotent if h(h(x)) = h(x) for every x ∈ L. Theorem 2.15. Let h : L −→ L be an idempotent linear operator on a linear space L. The linear operator g = 1L − h is also idempotent. Furthermore, we have Img(h) = Null(1L − h) and Img(1L − h) = Null(h). Proof.

Note that g(x) = (1L − h)(x) = x − h(x). Therefore, g(g(x)) = g(x) − h(g(x)) = g(x) − h(x) + h(h(x)) = g(x),

because h(h(x)) = h(x). If y ∈ Img(h), then y = h(x). This is equivalent to h(y) = h(h(x)) = h(x) = y, which means that (1L − h)(y) = 0L . Therefore, Img(h) = Null(1L − h). Replacing h by 1K − h we obtain Img(1L − h) = Null(h). 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Linear Spaces

b3234-main

page 79

79

In general, we shall refer to an idempotent linear operator on a linear space as a projection. Theorem 2.16. Let L be an F-linear space and let p : L −→ L be a projection. If Img(p) is a finite-dimensional subspace of L having the basis {v1 , . . . , vn }, then there exist n linear functionals 1 , . . . , n such that  1 if i = j, (2.1)

i (vj ) = 0 otherwise n for 1  i, j  n and p(x) = j=1 j (x)vj . n Proof. Let v ∈ Img(h). If v = j=1 aj vj , define fj (v) = aj for 1  j  n. It is immediate that the mappings fj : Img(h) −→ R are linear functionals. Since p(x) ∈ Img(p) for every x ∈ L, it follows that p(x) = n j=1 fj (p(x))vj . Let j : L −→ F be defined as j (x) = fj (p(x)) for 1  j  n.

j L F p

fj Img(p)

Clearly, j is a linear functional. equalities (2.1) follow.

Since p(vj ) = vj for 1  j  n, 

Definition 2.13. Let L be an F-linear space and let U be a subspace of L. A pair (x, y) ∈ L2 is U -congruent if x − y ∈ U . This is denoted by x ∼U y. For a subspace U of an F-linear space the relation ∼U that consists of all pairs of U -congruent elements is an equivalence relation on L. Indeed, x ∼U x because x − x = 0 ∈ U . Since x − y ∈ U is equivalent to y − x ∈ U , it follows that ∼U is symmetric. Finally, suppose that x ∼U y and y ∼U z. Since x − z = (x − y) + (y − z) ∈ U, it follows that ∼U is symmetric, so ∼U is an equivalence. The equivalence class of x ∈ L relative to ∼U is denoted by [x]U and the set of such equivalence classes is denoted by L/ ∼U . This set can be organized, in turn, as an F-linear space by defining a[x]U = [ax]U for a ∈ F and [x]U + [y]U = [x + y]U for x, y ∈ L. Indeed, it is easy to verify that the definition of these operations is independent of the choice of class representative.

May 2, 2018 11:28

80

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 80

Mathematical Analysis for Machine Learning and Data Mining

Let h ∈ L(L, K) be a linear operator between the two finite-dimensional linear spaces L and K having the Hamel bases R = {r1 , . . . , rm }, and S = {s1 , . . . , sn }, respectively. Since h(rj ) ∈ K we can write: h(rj ) = a1j s1 + a2j s2 + · · · + anj sn . In a more compact form, the last equality can be written using matrices as ⎛ ⎞ a1j ⎜ .. ⎟ h(rj ) = (s1 , . . . , sn ) ⎝ . ⎠ . anj The matrix Ah ∈ Cn×m associated to the linear operator h : L −→ K is the matrix that has ⎛ ⎞ a1j ⎜ a2j ⎟ ⎜ ⎟ h(rj ) = ⎜ . ⎟ ⎝ .. ⎠ anj as its j th column for 1  j  m, where h(rj ) = a1j s1 + a2j s2 + · · · + anj sn for 1  j  m. In other words, the image of rj under h is represented by the j th column of the matrix Ah . Let v ∈ L be a vector such that v = v1 r1 + · · · + vm rm . The image of v under h is ⎞ ⎛ m m   vj rj ⎠ = vj h(rj ) h(v) = h ⎝ j=1

j=1

⎛ m ⎞ a1j vj a1j j=1 m ⎟ m ⎜ a2j ⎟ ⎜  j=1 a2j vj ⎟ ⎜ ⎟ ⎜ ⎜ ⎟, = vj ⎜ . ⎟ = ⎜ .. ⎟ .. ⎠ ⎝ ⎝ . ⎠ j=1 m anj a v nj j j=1 ⎛

⎞

which is easily seen to equal Ah v. The matrix Ah attached to h : L −→ K depends on the bases chosen for the linear spaces L and K. The previous discussion also shows that a linear operator between two finite-dimensional spaces is completely determined by the values of the operator on the elements of the basis in the definition space. Matrix multiplication corresponds to the composition of linear operators, as we show next.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Linear Spaces

9in x 6in

b3234-main

page 81

81

Theorem 2.17. Let L, K, H be three finite-dimensional linear spaces. Let h ∈ L(L, K) and g ∈ L(K, H). Then, Agh = Ag Ah . Proof. If p1 , . . . , pm is a basis for L, then Agh (pi ) = gh(pi ) = g(h(pi )) = g(Ah pi ) = Ag (Ah (pi )) for every i, where 1  i ≤ n. This proves that  Agh = Ag Ah . Starting from a matrix A ∈ Cm×n we can define a linear operator associated to A, hA : Cn −→ Cm as hA (x) = Ax for x ∈ Cn . If e1 , . . . , en is a basis for Cn , then hA (ei ) is the ith column of the matrix A. It is immediate that AhA = A and hAh = h. Definition 2.14. A matrix A ∈ Cn×n is: (i) upper triangular if akj = 0 when j > k; (ii) lower triangular if akj = 0 when j < k; (iii) diagonal if akj = 0 when j = k. If A ∈ Cn×n is a diagonal matrix whose diagonal elements are d1 , . . . , dn we shall use the notation A = diag(d1 , . . . , dn ). Definition 2.15. Let L be a finite-dimensional space with dim(L) = n and let h : L −→ L be a linear operator. The determinant of h, det(h) is the determinant det(A) of the matrix Ah ∈ Cn×n . Example 2.11. The matrix of the operator 1L (where dim(L) = n) is A1L = diag(1, 1, . . . , 1). Also, det(1L ) = 1. We have det(Agh ) = det(Ag ) det(Ah ). If h : L −→ L is invertible, then hh 1L , so det(h) det(h−1 ) = det(1L ) = 1, hence −1

det(h−1 ) =

1 . det(h)

Let L be an n-dimensional C-linear space. The sum f + g of two linear functionals f, g : L −→ C is defined by (f +g)(x) = f (x)+g(x); the product of a linear functional f with a scalar a is defined by (af )(x) = af (x) for x ∈ L. Definition 2.16. The dual of an F-linear space L is the set L∗ of F-valued linear functionals equipped with the sum and the product with scalars defined above.

May 2, 2018 11:28

82

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 82

Mathematical Analysis for Machine Learning and Data Mining

Theorem 2.18. The dual L∗ of an n-dimensional linear space L is an n-dimensional linear space. Proof. Let b1 , . . . , bn be a basis in L. Define n linear functionals gi : L −→ C by gi (x) = xi , where x = x1 b1 + · · · + xn bn . Note that  1 if i = j, gi (bj ) = 0 otherwise, for 1  i, j  n. Let f ∈ L∗ . The linearity of f allows us to write f (x) = f (x1 b1 + · · · + xn bn ) = x1 f (b1 ) + · · · + xn f (bn ) = f (b1 )g1 (x) + · · · f (bn )gn (x), for every x ∈ P which shows that f = f (b1 )g1 + · · · + f (bn )gn . Thus, f is a linear combination of g1 , . . . , gn . Next, we need to show that the set {g1 , . . . , gn } is linearly independent. Suppose that a1 g1 + an gn = 0. This implies a1 g1 (x) + an gn (x) = 0L for every x ∈ L. Choosing x = bj we have aj = 0 for 1  j  n, which proves that the set {g1 , . . . , gn } is linearly independent and, therefore, is a basis  of L∗ . Thus, dim(L∗ ) = dim(L) = n. Theorem 2.19. (Real Hahn-Banach Theorem) Let L be a real linear space and let f : K −→ R be a linear functional defined on a subspace K of L. If p : L −→ R is a function such that p(x + y)  p(x) + p(y), p(ax) = ap(x) if a  0 for all x, y ∈ L, and f (x)  p(x) for x ∈ K, then f has an extension to F to L such that F (x)  p(x) for x ∈ L and F (x) = p(x) for x ∈ K. Proof. Suppose that K ⊂ L and let z ∈ L − K. Consider the set Hz = {y + az | y ∈ K and a ∈ R}. To extend f from K to a linear functional defined on Hz we seek to have f (y + az) = f (y) + af (z)  p(y + az)

(2.2)

for a ∈ R. To this end we need to specify a choice for f (z). If inequality (2.2) holds for a ∈ R, then we have f (y1 + z) = f (y1 ) + f (z)  p(y1 + z) f (y2 − z) = f (y2 ) − f (z)  p(y2 − z) for all y1 , y2 ∈ K (by taking a = 1 and a = −1, respectively). Inequalities (2.3) and (2.3) imply f (y2 ) − p(y2 − z)  f (z)  p(y1 + z) − f (y1 ).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Linear Spaces

b3234-main

page 83

83

Thus, to be able to define f (z) it suffices to have f (y2 ) − p(y2 − z)  p(y1 + z) − f (y1 ), or f (y1 ) + f (y2 )  p(y1 + z) + p(y2 − z) for all y1 , y2 ∈ K. This is indeed the case because f (y1 ) + f (y2 ) = f (y1 + y2 ) (because of the linearity of f )  p(y1 + y2 ) (because y1 + y2 ∈ K) = p(y1 + z + y2 − z)  p(y1 + z) + p(y2 − z) (because p is subadditive). Denote an extension of f to a subspace G that contains K by (G, fG ). We introduce a partial order on these extensions by writing (G, fG )  (G , fG ) if G ⊆ G and fG is the extension of fG to G . Let {(Gi , fi ) | i ∈ I} be a chain of extensions of f . Define the function  f∗ on G∗ = i∈I Gi to coincide with fi on Gi . It is clear that f∗ (x)  p(x) and (G, fG )  (G∗ , f∗ ). By Zorn’s Lemma there exists a maximal extension F of f to the entire space L.  Let now L be a complex linear space and let f : L −→ C be a linear functional defined on L. If g, h are the real and imaginary parts of f , that is, g(x) = (f (x)) and h(x) = (f (x)), then f (x) = g(x) + ih(x) for 1 (f (x) − f (x)), it follows x ∈ L. Since g(x) = 12 (f (x) + f (x)) and h(x) = 2i immediately that both g and h are linear functionals on L. Therefore, 1 g(ix) = (if (x) − if (x)) 2 i = (f (x) − f (x)) = −h(x), 2 which shows that f (x) = g(x) − ig(ix)

(2.3)

for x ∈ L. Theorem 2.20. (Complex Hahn-Banach Theorem) Let L be a complex linear space and let f : K −→ R be a linear functional defined on a subspace K of L. If p : L −→ R0 is a real valued function such that p(x+y)  p(x)+p(y) for all x, y ∈ L and p(ax) = |a|p(x), and |f (x)|  p(x) for x ∈ K, then f has an extension F to L such that |F (x)|  p(x) for x ∈ L and F (x) = p(x) for all x ∈ K.

May 2, 2018 11:28

84

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 84

Mathematical Analysis for Machine Learning and Data Mining

Proof. Let g = (f ). Since p(x + y)  p(x) + p(y) for x, y ∈ L, p(ax) = |a|p(x), and |f (x)|  p(x) for x ∈ K, g has a real linear extension G to the entire space L such that G(x)  p(x) by the real Hahn-Banach theorem. Define F : L −→ C as F (x) = G(x) − iG(ix). F clearly extends f , F (x1 + x2 ) = F (x1 ) + F (x2 ) for x1 , x2 ∈ L, and F (ax) = aF (x) for a ∈ R. Moreover, F (ix) = iG(ix) − iG(x) = iF (x), so F is also linear in the complex field. Note that p(ax) = p(x) if |a| = 1. If F (x) = |F (x)|eiθ , then |F (x)| = e−iθ F (x) = F (e−iθ x). Since F (e−iθ x) = |F (x)| is a real number, so |F (x)| = G(e−θ x)  p(e−iθ x) = p(x).



Definition 2.17. Let L be a linear space and let h : L −→ L be a linear operator. A subspace U is an invariant subspace of h if x ∈ U implies h(x) ∈ U . Example 2.12. Let Pn (R) be the linear space of polynomials of degree not larger than n. Note that if m  n, then Pm (R) is a subspace of Pn (R). The differentiation operator D : Pn (R) −→ Pn (R) defined as D(p) = p has Pm (R) as an invariant subspace because the derivative of a polynomial of degree at most m is a polynomial of degree at most m. Definition 2.18. Let L be a C-linear space. An eigenvalue of a linear operator h : L −→ L is a number λ ∈ C such that there exists x ∈ L − {0L} for which h(x) = λx. The vector x is referred to as an eigenvector of λ. Note that if h(x) = λx, then (h − λ1L )x = 0L , hence h − λ1L is not injective, and therefore, it is not invertible. Definition 2.19. The λ-resolvent of the linear operator h : L −→ L is the linear operator Rh,λ = (h − λ1L )−1 . Note that the set of eigenvectors that correspond to λ constitute the null space of the operator h − λ1L = R−1 h,λ . Example 2.13. Let h : C −→ C be the linear operator defined by h(x + iy) = −y + ix for x, y ∈ R. It is immediate that h is a linear operator. Then, λ is an eigenvalue if h(x + iy) = λ(x + iy) for some x + iy = 0C . This amounts to −y + ix = λ(x + iy), which implies −y = λx and x = λy. Note that y = 0 because this would entail x = 0, and x + iy = 0. Consequently,

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Linear Spaces

9in x 6in

b3234-main

page 85

85

we have λ2 y = −y, so λ2 = −1, which imply λ = i, or λ = −i. Eigenvectors that correspond to λ = i have the form a − ia and those that correspond to λ = −i have the form a + ia. Theorem 2.21. Let L be a linear space and let h : L −→ L be a linear operator. If λ1 , . . . , λm are distinct eigenvalues of h and x1 , . . . , xm are corresponding eigenvectors, then {x1 , . . . , xm } is a linearly independent set. Proof. Suppose that {x1 , . . . , xm } is a linearly dependent set. Let k be the least number such that xk ∈  x1 , . . . , xk−1 . There exist k − 1 scalars a1 , . . . , ak−1 such that xk = a1 x1 + · · · + ak−1 xk−1 . By applying h to both sides we obtain λk xk = a1 λ1 x1 + · · · + ak−1 λk−1 xk−1 . Therefore, 0 = a1 (λk − λ1 )x1 + · · · + ak−1 (λk − λk−1 xk−1 . The definition of k implies that the set {x1 , . . . , xk−1 } is linearly independent. Therefore, a1 = · · · = ak−1 = 0, which implies xk = 0. This contradiction yields the desired conclusion.  Theorem 2.22. Let L be a linear space and let h : L −→ L be a linear operator. Then h has at most dim(L) distinct eigenvalues. Proof. Suppose that λ1 , . . . , λm are m distinct eigenvalues of h. Let x1 , . . . , xm be the corresponding eigenvectors. Theorem 2.21 implies that  the set {x1 , . . . , xm } is linearly independent. Thus, m  dim(L).

2.4

Linear Spaces with Inner Products

Definition 2.20. Let L be a complex linear space. An inner product on L is a two-argument function (·, ·) : L×L −→ C with the following properties: (i) (ax + by, z) = a(x, z) + b(y, z), (ii) (x, y) = (y, x) (the skew-symmetry property), (iii) (x, x) is a non-negative real number, and (iv) (x, x) = 0 implies x = 0L for all a, b ∈ C and x, y, z ∈ L. The pair (L, (·, ·)) is called an inner product space. Observe that for a ∈ C we have (x, ay) = a(x, y) for a ∈ C and x, y ∈ L. Therefore, for a complex inner product we have the skew-linearityskew-linearity property, namely (x, ay + bz) = a(x, y) + b(x, z) for x, y, z ∈ L and a, b ∈ C.

May 2, 2018 11:28

86

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 86

Mathematical Analysis for Machine Learning and Data Mining

In general, a function f : L×L −→ C that is linear in the first argument and skew-linear in the second is said to be sesquilinear. Thus, an inner product on a complex linear space is sesquilinear. If L is a real linear space we assume that (x, y) ∈ R for x, y ∈ L. Thus, for an inner product on a real linear space we have the symmetry property (x, y) = (y, x) for x, y ∈ L. Example 2.14. If a, b ∈ Cn the function (·, ·) : Cn × Cn −→ R defined by (a, b) = bH a = a1 b1 + · · · + an bn is an inner product on Cn . Example 2.15. Let Cn×n be the set of square complex matrices. We remind the reader that the trace of a matrix A ∈ Cn×n is the complex n number trace(A) = i=j ajj . n  H j = 1n The function (A, B) = trace(AB H ) = j=1 (AB )jj = n n×n . k=1 ajk bjk is an inner product on C The linearity in the first argument is immediate. We have (B, A) =

=

=

n  n  j=1 k=1 n  n  j=1 k=1 n  n 

bkj akj bkj akj akj bkj = (A, B),

j=1 k=1

which shows that this function satisfies the second property of Definition 2.20. n n n n Also, (A, A) = j=1 k=1 ajk ajk = j=1 k=1 |ajk |2 , which is a real non-negative number. Finally, it is clear that (A, A) = 0 implies A = On,n , where On,n is the n × n square matrix whose entries are 0. Note that for a fixed y0 the function f : L −→ C given by f (x) = (x, y0 ) is linear. A fundamental property of the inner product defined on Rn is the equality (Ax, y) = (x, A y),

(2.4)

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 87

Linear Spaces

87

which holds for every matrix A ∈ Rn×n and x, y ∈ Rn . Indeed, we have n n  n n n     (Ax)i yi = aij xj yi = xj aij yi (Ax, y) = =

i=1 n 

i=1 j=1

xj

j=1

n 

j=1

i=1

aij yi = (x, A y).

i=1

For the complex linear space Cn and a matrix A ∈ Cn×n we have (Ax, y) = (x, AH y),

(2.5)

H

where A is obtained is the transposed conjugate matrix of A. Indeed, we have n n  n n n     (Ax)i yi = aij xj yi = xj aij yi (Ax, y) = i=1

=

n  j=1

i=1 j=1

xj

n 

j=1

i=1

aij yi = (x, AH y)

i=1

for x, y ∈ C . A very important inequality is presented next. n

Theorem 2.23. (Cauchy1 -Schwarz2 Inequality) Let (L, (·, ·)) be an inner product F-linear space. For x, y ∈ L we have |(x, y)|2  (x, x)(y, y). Proof.

We discuss the complex case. If a, b ∈ C we have

(ax + by, ax + by) = aa(x, x) + ab(x, y) + ba(y, x) + bb(y, y) = |a|2 (x, x) + 2(ab(x, y)) + |b|2 (y, y)  0. Let a = (y, y) and b = −(x, y). We have (y, y)2 (x, x) − 2(y, y)|(x, y)|2 + |(x, y)|2 (y, y)  0, 1 Augustin-Louis Cauchy was born on 21 August 21st 1789 and died on May 23rd 1857 in Sceaux (Seine), France. He was a French mathematician who made pioneering contributions to analysis. He was one of the first to state and prove theorems of calculus rigorously. He founded complex analysis and the study of permutation groups in abstract ´ algebra. Cauchy taught at Ecole Polytechnique starting in 1815 where he became a full professor in 1816, at the University of Turin, at the Facult´ e des sciences of Paris and at Coll` ege de France. 2 Karl Hermann Amandus Schwarz was born on January 23rd 1843 in Hermsdorf, Prussia (now in Poland) and died on November 30th 1921, in Berlin. Schwarz has major contribution in complex analysis and calculus of variations. He received his doctorate from the University of Berlin, and taught at the Universities of Halle, at ETH in Z¨ urch and at the University of Berlin.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 88

Mathematical Analysis for Machine Learning and Data Mining

88

hence (y, y)|(x, y)|2  (y, y)2 (x, x). If y = 0 the inequality obviously holds. If y = 0, the inequality follows. 

2.5

Seminorms and Norms

Definition 2.21. Let L be an F-linear space. A seminorm on L is a mapping ν : L −→ R0 that satisfies the following conditions: (i) ν(x + y)  ν(x) + ν(y) (subadditivity), and (ii) ν(ax) = |a|ν(x) (positive homogeneity), for x, y ∈ L and every scalar a. By taking a = 0 in the second condition of the definition we have ν(0L ) = 0 for every seminorm on a real or complex space. Theorem 2.24. If L is a real linear space and ν : L −→ R is a seminorm on L, then ν(x − y)  |ν(x) − ν(y)| for x, y ∈ L. Proof. We have ν(x)  ν(x − y) + ν(y), so ν(x) − ν(y)  ν(x − y). Since ν(y − x) + ν(x)  ν(y) and ν(y − x) = ν(x − y), we have the inequalities ν(x) − ν(y)  ν(x − y)  ν(y) − ν(x), which imply the inequality of the theorem.



Definition 2.22. Let L be a real or complex linear space. A norm on L is a seminorm ν : L −→ R such that ν(x) = 0 implies x = 0 for x ∈ L. The pair (L, ν) is referred to as a normed linear space. Inner products on linear spaces induce norms. Theorem 2.25. Let (L, (·, ·)) be an inner "product F-linear space. The function  ·  : L −→ R0 defined by x = (x, x) is a norm on L. Proof. We present the argument for inner product complex linear spaces. It is immediate from the properties of the inner product listed in Definition 2.20 that x is a real non-negative number and that ax = |a|x for a ∈ C.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Linear Spaces

page 89

89

Note that x + y2 = (x + y, x + y) = (x, x) + (x, y) + (y, x) + (y, y) = (x, x) + 2(x, y) + (y, y) (because (y, x) = (x, y))  (x, x) + 2|(x, y)| + (y, y) (because (y, x)  |(x, y)|)  x2 + 2xy + y2 (be the Cauchy-Schwarz Inequality) = (x + y)2 , which produces the desired inequality.



The Cauchy-Schwarz Inequality shown in Theorem 2.23 can now be formulated using norms as |(x, y)|  x · y. for any vectors of an inner product linear space. Theorem 2.26. (Complex Polarization Identity) Let  ·  be a norm on a complex linear space L that is generated by the inner product (·, ·). We have the complex polarization identity: 1 x + y2 − x − y2 − ix − iy2 + ix + iy2 (x, y) = 4 for x, y ∈ L. Proof.

We have x + y2 = (x + y, x + y) = x2 + y2 + (x, y) + (y, x) = (x + y, x + y) = x2 + y2 + (x, y) + (x, y) = x2 + y2 + 2(x, y).

Replacing y by −y we have x − y2 = x2 + y2 − 2(x, y), hence x + y2 − x − y2 = 4(x, y). Replacing now y by iy we obtain x + iy2 − x − iy2 = 4(x, iy).

(2.6)

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 90

Mathematical Analysis for Machine Learning and Data Mining

90

Note that (x, iy) = −i(x, y) = −i((x, y) + i(x, y)) = (x, y) − i(x, y), so (x, iy) = (x, y), which implies x + iy2 − x − iy2 = 4(x, iy). Taking into account the previous equalities we can write x+ y2 − x− y2 + ix+ iy2 − ix− iy2 = 4(x, y)+ 4i(x, y) = 4(x, y), which is the desired identity.



Corollary 2.6. (Real Polarization Identity) Let  ·  be a norm on a real linear space L that is generated by the inner product (·, ·). We have the polarization identity 1 x + y2 − x − y2 (x, y) = 4 for x, y ∈ L. Proof.

The proof of this identity follows directly from equality (2.6). 

Theorem 2.27. (Parallelogram Equality) Let  ·  be a norm on a linear space L that is generated by the inner product (·, ·). We have the parallelogram equality: 1 x + y2 + x − y2 x2 + y2 = 2 for x, y ∈ L. Proof. By applying the definition of  ·  and the properties of the inner product we can write: 1 x + y2 + x − y2 2 1 = ((x + y, x + y) + (x − y, x − y)) 2 1 = ((x, x) + 2(x, y) + (y, y) + (x, x) − 2(x, y) + (y, y)) 2 = x2 + y2 , which concludes the proof. Lemma 2.1. For a, b ∈ R0 and t ∈ (0, 1) we have at b1−t  ta + (1 − t)b. Equality takes place when a = b.



May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Linear Spaces

Proof.

page 91

91

Let f : R0 −→ R be the function defined as f (x) = xt − xt + x − 1.

We have f  (x) = txt−1 − t = t(xt−1 − 1). Since t ∈ (0, 1), we have f  (x) > 0 when x ∈ (0, 1) and f  (x) < 0 for x > 1. It follows that f (x)  f (1) = 0 with equality for x = 1. Hence, xt  tx + 1 − t. If b = 0 the inequality holds trivially. Suppose that b = 0. Substituting x = ab we have: ta at + 1 − t.  bt b A final multiplication by b delivers the result. The equality holds when x = 1, that is, when a = b.  Theorem 2.28. (Discrete H¨ older’s Inequality) Let a1 , . . . , an and b1 , . . . , bn be 2n positive numbers, and let p and q be two numbers such that p1 + 1q = 1 and p > 1. We have:  n 1  n 1 n   p p  q q ai b i  ai · bi . i=1

i=1

i=1

Equality holds when ap n i

bq = n i

p k=1 ak

q k=1 bk

for 1  i  n. Proof.

Define the numbers ap xi = n i

p k=1 ak

bq and yi = n i

for 1  i  n. Lemma 2.1 applied to xi , yi with 1 p

1 q

xi yi 

q k=1 bk t = p1 and

1−t =

1 q

implies

1 1 xi + yi , p q

or api bqi 1 1 ai b i    + n 1 n 1 p nk=1 apk q nk=1 bqk ( k=1 apk ) p ( k=1 bqk ) q Adding these inequalities, we obtain  n 1  n  1q n   p p  q ai b i  ai bi . i=1

i=1

i=1

Note that to obtain an equality we must have xi = yi for 1  i  n. This justifies the last claim of the theorem. 

May 2, 2018 11:28

92

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 92

Mathematical Analysis for Machine Learning and Data Mining

Theorem 2.29. Let a1 , . . . , an and b1 , . . . , bn be 2n real numbers and let p and q be two numbers such that p1 + 1q = 1 and p > 1. We have   n  n  p1  n  q1       p q ai b i   |ai | · |bi | .    i=1

Proof.

i=1

i=1

By Theorem 2.28, we have n 

|ai ||bi | 

 n 

i=1

 p1  n  1q  q |ai | · |bi | . p

i=1

i=1

The inequality follows from the fact that   n n       a b |ai ||bi |.  i i   i=1



i=1

Theorem 2.30. (Minkowski’s Inequality) Let a1 , . . . , an and b1 , . . . , bn be 2n non-negative numbers. If p  1, we have 

n 



 p1 

p

(ai + bi )

i=1

n 

 p1 api

+

 n 

i=1

 p1 bpi

.

i=1

 a1 

 b1 

. . . an

For 1 < p < ∞ the equality holds if and only if c

. . . bn

=d

for some

positive numbers c, d. Proof. For p = 1, the inequality is immediate. Therefore, we can assume that p > 1. Note that n 

(ai + bi )p =

i=1

n 

ai (ai + bi )p−1 +

i=1

n 

bi (ai + bi )p−1 .

i=1

By H¨ older’s inequality for p, q such that p > 1 and n 

 p−1

ai (ai + bi )



i=1

 =

n 

 p1  api

i=1 n  i=1

 p1  api

n 

1 p

1 q

+

 1q (p−1)q

(ai + bi )

i=1 n  i=1

= 1, we have

 q1 (ai + bi )p

.

(2.7)

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Linear Spaces

page 93

93

Similarly, we can write 

n 

bi (ai + bi )p−1 

i=1

n 

 p1  bpi

n 

i=1

 1q (ai + bi )p

.

(2.8)

i=1

Adding the last two inequalities yields ⎛  p1  n  p1 ⎞  n  1q n n     ⎠ (ai + bi )p  ⎝ api + bpi (ai + bi )p , i=1

i=1

i=1

i=1

which is equivalent to the desired inequality 

n 



 p1 (ai + bi )p



i=1

n 

 p1 api

+

 n 

i=1

 p1 bpi

.

i=1

To have the equality we must have equalities in each of inequalities (2.7) and (2.8). An elementary computation yields the condition mentioned above.  Note that if p < 1, the inequality sign in Minkowski’s inequality is reversed. Theorem 2.31. For p  1, the function νp : Rn −→ R0 defined by νp (x) =

 n 

 p1 |xi |

p

,

i=1

is a norm on the linear space (Rn , +, ·). Proof. Let x, y ∈ Rn . Minkowski’s inequality (Theorem 2.30) applied to the non-negative numbers ai = |xi | and bi = |yi | amounts to 

n 



 p1 (|xi | + |yi |)p



i=1

n 



 p1 |xi |p

+

i=1

n 

 p1 |yi |p

.

i=1

Since |xi + yi |  |xi | + |yi | for every i, 1  i  n, we have  n  i=1



 p1 (|xi + yi |)

p



n  i=1



 p1 |xi |

p

+

n 

 p1 |yi |

p

,

i=1

that is, νp (x + y)  νp (x) + νp (y). Thus, νp is a norm on Rn .



May 2, 2018 11:28

94

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 94

Mathematical Analysis for Machine Learning and Data Mining

We refer to νp as a Minkowski norm on Rn . Example 2.16. Consider the mappings ν1 , ν∞ : Rn −→ R given by ν1 (x) = |x1 | + |x2 | + · · · + |xn | and ν∞ (x) = max{|x1 |, |x2 |, . . . , |xn |}, for every x ∈ Rn . Both ν1 and ν∞ are norms on Rn . To verify that ν∞ is a norm we start from the inequality |xi + yi |  |xi | + |yi |  ν∞ (x) + ν∞ (y) for 1  i  n. This in turn implies ν∞ (x + y) = max{|xi + yi | | 1  i  n}  ν∞ (x) + ν∞ (y), which gives the desired inequality. This norm can be regarded as a limit case of the norms νp . Indeed, let x ∈ Rn and let M = max{|xi | | 1  i  n} = |xl1 | = · · · = |xlk | for some l1 , . . . , lk , where 1  l1 , . . . , lk  n. Here xl1 , . . . , xlk are the components of x that have the maximal absolute value and k  1. We can write:  n  1  |xi | p p 1 = lim M (k) p = M, lim νp (x) = lim M p→∞ p→∞ p→∞ M i=1 which justifies the notation ν∞ . We frequently use the alternative notation xp for νp (x). We refer to the norm ν2 as the Euclidean norm. # $ Example 2.17. Let x = xx12 ∈ R2 be a unit vector in the sense of the Euclidean norm. We have |x1 |2 + |x2 |2 = 1. Since x1 and x2 are real numbers we can write x1 = cos α and x2 = sin α. This allows us to write   cos α x= . sin α Theorem 2.32. Each norm ν : L −→ R0 on a linear space L generates a metric on the set L defined by dν (x, y) = ν(x − y) for x, y ∈ L. Proof. Note that if dν (x, y) = ν(x − y) = 0, it follows that x − y = 0L , so x = y. The symmetry of dν is obvious and so we need to verify only the triangular axiom. Let x, y, z ∈ L. We have: ν(x − z) = ν(x − y + y − z)  ν(x − y) + ν(y − z) or, equivalently, dν (x, z)  dν (x, y) + dν (y, z), for every x, y, z ∈ L, which concludes the argument. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Linear Spaces

b3234-main

page 95

95

We refer to dν as the metric induced by the norm ν on the linear space L. Note that the metric dν on L induced by a norm is translation invariant, that is, dν (x + z, y + z) = dν (x, y) for every z ∈ L. Also, for every a ∈ R and x, y ∈ L we have the homogeneity property dν (ax, ay) = |a|dν (x, y) for x, y ∈ L. Theorem 2.33. Let L be a real linear space and let d : L × L −→ R0 be a metric on L. If d is translation invariant and homogeneous, then there exists a norm ν of L such that d = dν . Proof. Let d be a metric on L that is translation invariant and homogeneous. Define ν(x) = d(x, 0L ). It follows immediately that ν is a norm on L.  For p  1, then dp denotes the metric dνp induced by the norm νp on the linear space (Rn , +, ·) known as the Minkowski metric on Rn . The metrics d1 , d2 and d∞ defined on Rn are given by d1 (x, y) =

n 

|xi − yi |,

(2.9)

i=1

% & n & |xi − yi |2 , d2 (x, y) = '

(2.10)

i=1

d∞ (x, y) = max{|xi − yi | | 1  i  n},

(2.11)

for x, y ∈ Rn . These metrics are visualized in Figure 2.1 for the special case of R2 . If     x0 y0 x= and y = , x1 y1 then d1 (x, y) is the sum of the lengths of the two legs of the triangle, d2 (x, y) is the length of the hypotenuse of the right triangle and d∞ (x, y) is the largest of the lengths of the legs. Theorem 2.34 to follow allows us to compare the norms νp (and the metrics of the form dp ) that were introduced on Rn . We begin with a preliminary result. Lemma 2.2. Let a1 , . . . , an be n positive numbers. If p and q are two 1 1 positive numbers such that p  q, then (ap1 + · · · + apn ) p  (aq1 + · · · + aqn ) q .

May 2, 2018 11:28

96

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 96

Mathematical Analysis for Machine Learning and Data Mining

6

y = (y0 , y1 )

x = (x0 , x1 )

Fig. 2.1

(y0 , x1 ) -

The distances d1 (x, y), d2 (x, y), and d∞ (x, y).

Proof. Let f : R>0 −→ R be the function defined by f (r) = 1 (ar1 + · · · + arn ) r . Since ln f (r) =

ln (ar1 + · · · + arn ) , r

it follows that f  (r) 1 1 ar ln a1 + · · · + arn ln ar = − 2 (ar1 + · · · + arn ) + · 1 . f (r) r r ar1 + · · · + arn To prove that f  (r) < 0, it suffices to show that ln (ar1 + · · · + arn ) ar1 ln a1 + · · · + arn ln ar .  ar1 + · · · + arn r This last inequality is easily seen to be equivalent to n 

ar i=1 1

ari ari ln r  0, r + · · · + an a1 + · · · + arn

which holds because ar1

ari 1 + · · · + arn

for 1  i  n.



Theorem 2.34. Let p and q be two positive numbers such that p  q. We have up  uq for u ∈ Rn . Proof.

This statement follows immediately from Lemma 2.2.



Corollary 2.7. Let p, q be two positive numbers such that p  q. For every x, y ∈ Rn , we have dp (x, y)  dq (x, y).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Linear Spaces

Proof.

This statement follows immediately from Theorem 2.34.

page 97

97



Theorem 2.35. Let p  1. We have x∞  xp  nx∞ for x ∈ R . n

Proof. The first inequality is an immediate consequence of Theorem 2.34. The second inequality follows by observing that  n  p1  p |xi |  n max |xi | = nx∞ . xp = 1in i=1  Corollary 2.8. Let p and q be two numbers such that p, q  1. For x ∈ Rn we have: 1 xq  xp  nxq . n Proof. Since x∞  xp and xq  nx∞ , it follows that xq  nxp . Exchanging the roles of p and q, we have xp  nxq , so 1 xq  xp  nxq n for every x ∈ Rn .



For p = 1 and q = 2 and x ∈ Rn we have the inequalities % % & n & n n   & & 1' x2i  |xi |  n' x2i . n i=1 i=1 i=1

(2.12)

Corollary 2.9. For every x, y ∈ Rn and p  1, we have d∞ (x, y)  dp (x, y)  nd∞ (x, y). Further, for p, q > 1, there exist c, c ∈ R>0 such that c dq (x, y)  dp (x, y)  c dq (x, y) for x, y ∈ Rn . Proof.

This follows from Theorem 2.35 and from Corollary 2.9.



Corollary 2.7 implies that if p  q, then the closed sphere Bdp [x, r] is included in the closed sphere Bdq [x, r]. For example, we have Bd1 [0, 1] ⊆ Bd2 [0, 1] ⊆ Bd∞ [0, 1].

May 2, 2018 11:28

98

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 98

Mathematical Analysis for Machine Learning and Data Mining

6 @ @-

6 '$

6

-

@ @ (a)

-

&% (b) (c)

Fig. 2.2

Spheres Bdp [0, 1] for p = 1, 2, ∞.

In Figures 2.2(a)–(c) we represent the closed spheres Bd1 [0, 1], Bd2 [0, 1], and Bd∞ [0, 1]. An useful consequence of Theorem 2.28 is the following statement. Theorem 2.36. Let x1 , . . . , xm and y1 , . . . , ym be 2m non-negative numm m bers such that i=1 xi = i=1 yi = 1 and let p and q be two positive numbers such that 1p + 1q = 1. We have m 

1

1

xjp yjq  1.

j=1

1

1

1

1

p q Proof. The H¨ older inequality applied to x1p , . . . , xm and y1q , . . . , ym yields the inequality of the theorem:

m  j=1

1

1

xjp yjq 

m  j=1

xj

m 

yj = 1.

j=1

 The set of real number sequences Seq(R) is a real linear space where the sum of the sequences x = (xn ) and y = (yn ) is defined as x + y = (xn + yn ) and the product of a real with x is ax = (axn ). The subspace 1 (R) of Seq(R) consists of all sequences x = (xn ) such  that n∈N |xn | is convergent. Note that a norm exists on 1 defined by  x = n∈N |xn |. The set of sequences x ∈ Seq∞ (R) such that xp is finite is a real normed linear space. In Example 2.2 we saw that Seq∞ (R) can be organized as a linear space. Let x, y ∈ Seq∞ (R) be two sequences such that xp and yp are finite.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Linear Spaces

page 99

99

By Minkowski’s inequality, if p  1 we have  n  n  n  p1  p1  p1  n  p1     p p p p |xi + yi |  (|xi | + |yi |)  |xi | + |yi | . i=1

i=1

i=1

i=1

When n tends to ∞ we have x + yp  xp + yp , so the function  · p is indeed a norm. If Sp (R) is the set of all sequences x in Seq∞ (R) such that xp < ∞, then (Sp (R),  · p ) is a normed space denoted by p (R). The space ∞ (R) consists of bounded sequences in Seq∞ (R). The Cauchy-Schwarz Inequality implies that |(x, y)|  x2 y2 . Equivalently, this means that −1 

(x, y)  1. x2 y2

This double inequality allows us to introduce the notion of angle between two vectors x, y of a real linear space L. Definition 2.23. The angle between the vectors x and y is the number α ∈ [0, π] defined by cos α =

(x, y) . x2 y2

This angle is denoted by ∠(x, y). # $ Example 2.18. Let u = uu12 ∈ R2 be a unit vector. Since u21 + u22 = 1, there exists α ∈ [0, 2π] such that u1 = cos α and u2 = sin α. Thus, for any two unit vectors in R2 ,     cos α cos β u= and v = sin α sin β we have (u, v) = cos α cos β + sin α sin β = cos(α − β), where α, β ∈ [0, 2π]. Consequently, ∠(u, v) is the angle in the interval [0, π] that has the same cosine as α − β. Theorem 2.37. (The Cosine Theorem) Let x and y be two vectors in Rn equipped with the Euclidean inner product. We have: x − y2 = x2 + y2 − 2xy cos α, where α = ∠(x, y).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 100

Mathematical Analysis for Machine Learning and Data Mining

100

Proof.

Since the norm is induced by the inner product we have x − y2 = (x − y, x − y) = (x, x) − 2(x, y) + (y, y) = x2 − 2xy cos α + y2 , 

which is the desired equality.

Definition 2.24. Let L be an inner product space. Two vectors x and y of L are orthogonal if (x, y) = 0. A pair of orthogonal vectors (x, y) is denoted by x ⊥ y. Definition 2.25. An orthogonal set of vectors in an inner product space L is a subset W of L such that for every distinct u, v ∈ W we have u ⊥ v. If, in addition, u = 1 for every u ∈ W , then we say that W is orthonormal. Theorem 2.38. If W is a set of non-zero orthogonal vectors in an inner product space (V, (·, ·)), then W is linearly independent. Proof. Let a1 w1 + · · · + an wn = 0 for a linear combination of elements of W . This implies ai wi 2 = 0, so ai = 0 because wi 2 = 0, and this holds for every i, where 1  i  n. Thus, W is linearly independent.  Corollary 2.10. Let L be an n-dimensional linear space. If W is an orthonormal set and |W | = n, then W is an orthonormal basis of L. Proof.

This statement is an immediate consequence of Theorem 2.38. 

Corollary 2.11. Let L be an n-dimensional linear space. {v1 , . . . , vn } is an orthonormal basis in V , then

If V

=

x = (x, v1 )v1 + · · · + (x, vn )vn for all x ∈ L. Proof. Since V is a basis, there exist a1 , . . . , an such that x = a1 v1 +  . . . + an vn . This implies (x, vj ) = aj (vj , vj ) = aj for 1  j  n. The Gram-Schmidt algorithm constructs an orthonormal basis for a finitely dimensional linear space L starting from an arbitrary basis of L. Suppose that {u1 , . . . , um } is a basis of L. The orthonormal basis {w1 , . . . , wm } is constructed sequentially such that  w1 , . . . , wk  =  u1 , . . . , uk for 1  k  m.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Linear Spaces

page 101

101

Let Wk =  w1 , . . . , wk . Note that W1 =  w1 =  u1 , which allows us to define w1 as the unit vector w1 = u11 u1 . Suppose that we have constructed an orthonormal basis for Wk =  u1 , . . . , uk and we seek to construct wk+1 such that {w1 , . . . , wk , wk+1 } is an orthonormal basis for Wk+1 =  u1 , . . . , uk , uk+1 . The expansion of uk+1 relative to the orthonormal basis {w1 , . . . , wk , wk+1 } is uk+1 =

k+1 

(uk+1 , wi )wi ,

i=1

which implies that

k uk+1 − i=1 (uk+1 , wi )wi . = (uk+1 , wk+1 )

wk+1

Note that by Fourier expansion of uk+1 with respect to the orthonormal set {w1 , . . . , wk , wk+1 } we have uk+1 2 =

k 

(uk+1 , wi )2 + (uk+1 , wk+1 )2 .

i=1

Therefore,  2 k      (uk+1 , wi )wi  uk+1 −   i=1   k k   = uk+1 − (uk+1 , wi )wi , uk+1 − (uk+1 , wi )wi i=1

= uk+1 2 − 2

i=1

k 

(uk+1 , wi )2 +

i=1

= uk+1 2 −

k 

k 

(uk+1 , wi )2

i=1

(uk+1 , wi )2 = (uk+1 , wk+1 )2 .

i=1

The last equalities imply   k      (uk+1 , wi )wi  = |(uk+1 , wk+1 )|. uk+1 −   i=1

It follows that we can define wk+1 =

uk+1 − uk+1 −

k

i=1 (uk+1 , wi )wi

k

i=1 (uk+1 , wi )wi 

,

May 2, 2018 11:28

102

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 102

Mathematical Analysis for Machine Learning and Data Mining

or as wk+1 = −

uk+1 − uk+1 −

k

i=1 (uk+1 , wi )wi

k

i=1 (uk+1 , wi )wi 

.

Theorem 2.39. Let (w1 , . . . , wm ) be the sequence of vectors constructed by the Gram-Schmidt algorithm starting from the basis {u1 , . . . , um } of an m-dimensional linear space L. The set {w1 , . . . , wm } is an orthogonal basis of U and  w1 , . . . , wk =  u1 , . . . uk for 1  k  m. Proof. We begin by proving that {w1 , . . . , wk } is an orthonormal set. The argument is by induction on k  1. The base case, k = 1, is immediate. Suppose that the statement of the theorem holds for k, that is, the set {w1 , . . . , wk } is an orthonormal basis for Uk =  u1 , . . . , uk . It is clear that wk+1  = 1, so it remains to show that wk+1 ⊥ wj for 1  j  k. We have   k  1 (wk+1 , wj ) = (uk+1 , wi )wi , wj uk+1 − (uk+1 , wk+1 ) i=1 =

1 ((uk+1 , wj ) − (uk+1 , wj )(wj , wj )) = 0, (uk+1 , wk+1 )

because (wj , wj ) = 1. The equality  u1 , . . . , uk =  w1 , . . . , wk clearly holds for k = 1. Suppose that it holds for k. Since w1 , . . . , wk belong to the subspace  u1 , . . . , uk (by inductive hypothesis) it follows that wk+1 ∈  u1 , . . . , uk , uk+1 , so  w1 , . . . , wk+1 ⊆  u1 , . . . , uk . For the converse inclusion, note that wk+1 was defined such that uk+1 ∈   w1 , . . . , wk , wk+1 . Thus,  u1 , . . . , uk , uk+1 ⊆  w1 , . . . , wk , wk+1 . Corollary 2.12. Every finite-dimensional inner-product linear space L has an orthonormal basis. Proof. This follows by applying the Gram-Schmidt algorithm to a basis of the space L.  Theorem 2.40. Let V be a finite-dimensional linear space. If U is an orthonormal set of vectors, then there exists a basis T of V that consists of orthonormal vectors such that U ⊆ T . Proof. Let U = {u1 , . . . , um } be an orthonormal set of vectors in V . There is an extension of U , Z = {u1 , . . . , um , um+1 , . . . , un } to a basis

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Linear Spaces

page 103

103

of V , where n = dim(V ), by the Extension Corollary. Now, apply the Gram-Schmidt algorithm to the set U to produce an orthonormal basis W = {w1 , . . . , wn } for the entire space V . It is easy to see that wi = ui for 1  i  m, so U ⊆ W and W is the orthonormal basis of V that extends the set U .  Corollary 2.13. Let U be a subspace of an n-dimensional linear space V such that dim(U ) = m, where m < n. Then dim(U ⊥ ) = n − m. Proof.

Let u1 , . . . , um be an orthonormal basis of U , and let u1 , . . . , um , um+1 , . . . , un

be its completion to an orthonormal basis for V , which exists by Theorem 2.40. Then, um+1 , . . . , un is a basis of the orthogonal complement U ⊥ ,  so dim(U ⊥ ) = n − m. Let L be an n-dimensional inner product linear space and let {e1 , . . . , en )} be an orthonormal basis in L. Then any v ∈ L can be written as

Note that (ei , v) = be written as

n k=1

v = a1 e 1 + · · · + an e n . ak (ei , ek ) = ak , which shows that any v ∈ L can

v = (e1 , v)e1 + · · · + (en , v)en . For an arbitrary subset T of an inner product space L the set T ⊥ is defined by: T ⊥ = {v ∈ L | v ⊥ t for every t ∈ T }. Note that T ⊆ U implies U ⊥ ⊆ T ⊥ . If S, T are two subspaces of an inner product space L, then S and T are orthogonal if s ⊥ t for every s ∈ S and every t ∈ T . This is denoted as S ⊥ T. Theorem 2.41. Let L be an inner product space and let T be a subset of an inner product F-linear space L. The set T ⊥ is a subspace of L. Proof. Let x and y be two members of T . We have (x, t) = (y, t) = 0 for every t ∈ T . Therefore, for every a, b ∈ F, by the linearity of the inner product we have (ax + by, t) = a(x, t) + b(y, t) = 0, for t ∈ T , so  ax + by ∈ T ⊥ . Thus, T ⊥ is a subspace of L.

May 2, 2018 11:28

104

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 104

Mathematical Analysis for Machine Learning and Data Mining

Theorem 2.42. Let L be a finite-dimensional inner product F-linear space ⊥ and let T be a subset of L. We have  T = T ⊥ . ⊥

Proof. By a previous observation, since T ⊆ T , we have T ⊆ T ⊥ . To prove the converse inclusion, let z ∈ T ⊥ . If y ∈  T, y is a linear combination of vectors of T , y = a1 t1 +· · ·+am tm , so (y, z) = a1 (t1 , z) + · · · + am (tm , z) = 0. Therefore, z ⊥ y, which implies ⊥ ⊥  z ∈  T . This allows us to conclude that  T = T ⊥ . We refer to T ⊥ as the orthogonal complement of T . Note that T ∩ T ⊥ ⊆ {0}. If T is a subspace, then this inclusion becomes an equality, that is, T ∩ T ⊥ = {0}. Theorem 2.43. Let T be a subspace of the finite-dimensional linear space L. We have L = T ⊕ T ⊥ . Proof. We observed that T ∩ T ⊥ = 0L . Suppose that B and B  are two orthonormal bases in T and T ⊥ , respectively. The set B ∪ B  is a basis for S = T ⊕ T ⊥. Suppose that S ⊂ L. The set B ∪ B  can be extended to a orthonormal basis B ∪ B  ∪ B  for L. Note that B  ⊥ B, so B  ⊥ T , which implies B  ⊆ T ⊥ . This is impossible because B ∪ B  ∪ B  is linearly independent. Therefore, B ∪ B  is a basis for L, so L = T ⊕ T ⊥ .  Theorem 2.44. Let (L, (·, ·) be an inner product linear space, Y = {y1 , . . . , yk } be a linearly independent set in L and let T =  Y  . ⎛ ⎞ c1 ⎜ .. ⎟ For c = ⎝ . ⎠ ∈ Rk let Uc = {x ∈ L | (x, yi ) = ci for 1  i  k}. ck There exists a translation tz of L such that Uc = tz (T ⊥ ). Proof. Let u, z ∈ Uc . We have (u, yi ) = (z, yi ) = ci for 1  i  k, hence (u − z, yi ) = 0, that is, u − z ∈ T ⊥ . For a fixed z, u ∈ tz (T ⊥ ), hence Uc ⊆ tz (T ⊥ ). Conversely, if u ∈ tz (T ⊥ ), where (z, yi ) = ci for 1  i  k, then u = z + w, where w ∈ T ⊥ , which implies (u, yi ) = ci for 1  i  k,  hence u ∈ Uc . Theorem 2.45. (Pythagora’s Theorem) Let x1 , . . . , xn be a finite orthogonal set on n distinct elements in an inner product space L. We have   n n   2    xi  = xi 2 .    i=1

i=1

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Linear Spaces

page 105

105

Proof. By applying the definition of the norm induced by the inner product we have ⎞ ⎛   n n n   2     xi  = ⎝ xi , xj ⎠    i=1

i=1

=

n  n  i=1 j=1

j=1

(xi , xj ) =

n 

(xi , xi )

i=1

(because (xi , xj ) = 0 for i = j) n  = xi 2 . i=1



Definition 2.26. A subspace T of a inner product linear space is an approximating subspace if for every x ∈ L there is a unique element in T that is closest to x. Theorem 2.46. Let T be a subspace in the inner product space L. If x ∈ L and t ∈ T , then x−t ∈ T ⊥ if and only if t is the unique element of T closest to x. Proof.

Suppose that x − t ∈ T ⊥ . Then, for any u ∈ T we have x − u2 = (x − t) + (t − u)2 = x − t2 + t − u2 ,

by observing that x − t ∈ T ⊥ and t − u ∈ T and applying Pythagora’s Theorem to x − t and t − u. Therefore, we have x − u2  x − t2 , so t is the unique element of T closest to x. Conversely, suppose that t is the unique element of T closest to x and x− t ∈ T ⊥ , that is, there exists u ∈ T such that (x− t, u) = 0. This implies, of course, that u = 0L . We have x − (t + au)2 = x − t − au2 = x − t2 − 2(x − t, au) + |a|2 u2 . Since x − (t + au)2  x − t2 (by the definition of t), we have −2(x − t, au) + |a|2 u2  0 for every a ∈ F.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 106

Mathematical Analysis for Machine Learning and Data Mining

106

For a =

1 u 2 (x

− t, u) we have

1 1 (x − t, u)u) + | (x − t, u)|2 u2 u2 u2 1 1 = −2(x − t, (x − t, u)u) + | (x − t, u)|2 u2 2 u u2 |(x − t, u)|2 |(x − t, u)|2 = −2 + 2 u u2 |(x − t, u)|2 =−  0, u2

−2(x − t,

which is a contradiction.



Theorem 2.47. A subspace T of an inner product linear space L is an approximating subspace of L if and only if L = T ⊕ T ⊥ . Proof. Let T be an approximating subspace of L and let x ∈ L. By Theorem 2.46, we have x − t ∈ T ⊥ , where t is the element of T that best approximates x. If y = x − t, we can write x uniquely as x = t + y, where t ∈ T and y ∈ T ⊥ , so L = T ⊕ T ⊥ . Conversely, suppose that L = T ⊕ T ⊥ , where T is a subspace of L. Every x ∈ L can be uniquely written as x = t + y, where t ∈ T and y ∈ T ⊥ , so x − t ∈ T ⊥ . By Theorem 2.46, t is the element in T that is closest to x, so T is an approximating subspace of L.  Theorem 2.48. Any subspace T of a finite-dimensional inner product linear space L is an approximating subspace of L. Proof. Let T be a subspace of L. By Theorem 2.47 it suffices to show that L = T ⊕ T ⊥ . If T = {0L }, then T ⊥ = L and the statement is immediate. Therefore, we can assume that T = {0L }. We need to verify only that every x ∈ L can be uniquely written as a sum x = t + v, where t ∈ T and v ∈ T ⊥ . Let t1 , . . . , tm be an orthonormal basis of T , that is, a basis such that  1 if i = j, (ti , tj ) = 0 otherwise, for 1  i, j  m. Define t = (x, t1 )t1 + · · · + (x, tm )tm and v = x − t. The vector v is orthogonal to every vector ti because (v, ti ) = (x − t, ti ) = (x, ti ) − (t, ti ) = 0.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Linear Spaces

page 107

107

Therefore v ∈ T ⊥ and x has the necessary decomposition. To prove that the decomposition is unique suppose that x = s + w, where s ∈ T and w ∈ T⊥ . Since s + w = t + v we have s − t = v − w ∈ T ∩ T ⊥ = {0L }, which implies s = t and w = v.  Theorem 2.49. Let T be a subspace of an inner product space L of finite dimension. We have (T ⊥ )⊥ = T . Proof. Observe that T ⊆ (T ⊥ )⊥ . Indeed, if t ∈ T , then (t, z) = 0 for every z ∈ T ⊥ , so t ∈ (T ⊥ )⊥ . To prove the reverse inclusion, let x ∈ (T ⊥ )⊥ . Theorem 2.48 implies that we can write x = u + v, where u ∈ T and v ∈ T ⊥ , so x − u = v ∈ T ⊥ . Since T ⊆ (T ⊥ )⊥ , we have u ∈ (T ⊥ )⊥ , so x − u ∈ (T ⊥ )⊥ . Consequently, x − u ∈ T ⊥ ∩ (T ⊥ )⊥ = {0}, so x = u ∈ T . Thus, (T ⊥ )⊥ ⊆ T , which concludes the argument.  Definition 2.27. Let W = {w1 , . . . , wn } be an orthonormal set and let x ∈  W . The equality x = (x, w1 )w1 + · · · + (x, wn )wn

(2.13)

is the Fourier expansion of x with respect to the orthonormal set W . 2.6

Linear Functionals in Inner Product Spaces

A linear functional on a finite-dimensional space can be represented using an inner product, as we prove next. Theorem 2.50. Let L be an n-dimensional inner product linear space and let f : L −→ C be a linear functional on L. There exists a unique w ∈ L such that f (x) = (x, w) for x ∈ L. Proof. We have shown in Theorem 2.18 that the dual L∗ of an ndimensional space L is also n-dimensional. Starting from a basis B = {b1 , . . . , bn } in L it is possible to construct ˜ = {g1 , . . . , gn } in L∗ that consists of the linear functionals such a basis B that gi (x) = xi for every x ∈ L such that x = x1 b1 + · · · + xn bn . Let f : L −→ C be a linear functional on L. Since g1 , . . . , gn is a basis in L∗ , we have f = w1 g1 + · · · + wn gn , where w1 , . . . , wn ∈ C. This allows us to write: f (x) = w1 g1 (x) + · · · + wn gn (x) = w1 x1 + · · · + wn xn = (x, w),

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 108

Mathematical Analysis for Machine Learning and Data Mining

108

 w1  where w =

. . . wn

.

Suppose that we also have f (x) = (x, u) for x ∈ L. Then (x, u) = (x, w), or (x, u − w) = 0. Taking x = u − w we obtain u − w = 0 and, therefore, u = w, which shows the uniqueness of w.  The extension of Theorem 2.50 to infinite-dimensional spaces, known as the Riesz Theorem is a very important result of functional analysis. We present this result in Chapter 11. Theorem 2.51. Let L be a F-normed space, where F is R or C. For every u0 ∈ L − {0L }, there exists a non-trivial linear functional f : L −→ C such that f (u0 ) = u0  and f  = 1. Proof. Let S be the subspace of L generated by u0 , that is, S =  u0 and let f0 : S −→ C be given by f0 (u) = au0 , for u = au0 . It is clear that f0 (u0 ) = u0  and that |f0 (u)| = u for all u ∈  u0 . By the Hahn-Banach Theorem (see Theorem 2.19) there exists an extension f  of f0 to L such that |f (u)|  u for all u ∈ L, so f  = 1. Corollary 2.14. Let L be a F-normed space, where F is R or C. For every u0 ∈ L we have u0  = max{|f (u0 )| | f ∈ L∗ , f   1}. Proof.

 Since |f (u0 )|  f u0, the equality follows from Theorem 2.51.

Corollary 2.15. Let L be a F-normed space, where F is R or C. If f (u) = 0 for every f ∈ L∗ , then u = 0L . Proof.

This is an immediate consequence of Theorem 2.51.



Definition 2.28. Let F be R or C. The bidual of the F-linear space L is the linear space L∗∗ that consists of all linear functionals F : L∗ −→ F. The linear space L is said to be reflexive if for every F ∈ L∗∗ there exists u ∈ L such that F (f ) = f (u) for every f ∈ L∗ . Theorem 2.52. Let L be an F-normed linear space. Define φ : L −→ L∗∗ as φ(u) = F if F (f ) = f (u) for f ∈ L∗ . The mapping φ is linear and φ(u) = u for all u ∈ L. Furthermore, L is reflexive if and only if φ is bijective.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Linear Spaces

page 109

109

Proof. By Definition 2.28, L is reflexive if and only if the mapping φ is surjective. Suppose that φ(u) = F and φ(v) = G for u, v ∈ L, that is, F (f ) = f (u) and G(f ) = f (v) for every f ∈ L∗ . We have (aφ(u) + bφ(v))(f ) = aφ(u)(f ) + bφ(v)(f ) = af (u) + bf (v) = f (au + bv) (because f is a linear functional) = φ(au + bv)(f ), hence φ is a linear mapping. Furthermore, we have φ(u) = sup{f (u) | f ∈ L∗ , f   1} = u by Corollary 2.14. If φ(u) = 0 it follows that u = 0, so φ is injective. Therefore, L is reflexive if and only if φ is a bijection.  Theorem 2.53. Let (L, (·, ·)) be a real inner product linear space, S be a subspace of L, and let x ∈ L. There exists at most one vector y0 ∈ S such that x − y0   x − y for every y ∈ S. Furthermore, y0 is a unique vector in S that minimizes x − y if and only if x − y0 is orthogonal on S. Proof. Let that y0 ∈ S is such that x − y0   x − y for every y ∈ S. We claim that x − y0 is orthogonal on S, that is, x − y0 is orthogonal on every y ∈ S. Suppose that x − y0 is not orthogonal on y ∈ S, that is, (x − y0 , y) = a, where a = 0. Without loss of generality, we may assume that y = 1. If y1 = y0 + ay, we have x − y1 2 = x − y0 − ay2 = x − y0 2 − 2a(x − y0 , y) + a2 = x − y0 2 − a2 < x − y0 2 , which contradicts the minimality of x−y0 , Therefore, x−y0 is orthogonal on S. Conversely, suppose that x − y0 is orthogonal on S. Then, x − y0  is minimal and y0 is unique. For y ∈ S we have x − y2 = x − y0 2 + y0 − y2 , so x − y > x − y0  for y = y0 , which means that x − y0  is minimal  and that y0 is unique.

May 2, 2018 11:28

110

2.7

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 110

Mathematical Analysis for Machine Learning and Data Mining

Hyperplanes

Affine subspaces of a linear space L are obtained by translating subspaces of L. Definition 2.29. An affine subspace of a linear space L is a set of the form tw (U ), where U is a subspace of L and w ∈ L. A hyperplane in a non-trivial linear space L is a maximal proper affine subspace. It is clear that in a non-trivial linear space every maximal linear space is a hyperplane. If H is a hyperplane in L, we have H = L, and if U is any other affine subspace such that H ⊆ U , then U = H or U = L. Note that if H is a hyperplane obtained as a translation H = tz (S) where S is a subspace of L, then S is a maximal subspace of L. Theorem 2.54. If H is a hyperplane in the real, non-trivial linear space L, then there exists a non-trivial linear functional f on L and a number a ∈ R such that H = {x ∈ L | f (x) = a}. Conversely, if f is a non-trivial linear functional on L, then {x ∈ L | f (x) = a} is a hyperplane in L. Proof. Let H be a hyperplane in L. There exists a maximal subspace S of L such that H = w + S. If w ∈ S, then  {w} ∪ S = L. Thus, every x ∈ L can be written as x = aw + y such that a ∈ R and y ∈ S. Define f as f (x) = a for x ∈ L. This is a linear functional. Indeed, suppose that for u, v ∈ L we have u = a1 w + y1 and v = a2 w + y2 . We have f (u) = a1 and f (v) = a2 . Also, for c, d ∈ R we can write cu + dv = (ca1 + da2 ) + cy1 + d2 , which yields f (cu + dv) = ca1 + da2 = cf (u) + df (v), which proves that f is a linear functional. It is clear that H = {x ∈ L | f (x) = 1}. If w ∈ S, then H = S and we can take w1 ∈ S. We have L =  {w1 } ∪ S} and, for x = aw1 + y, we define f (x) = a. Then H = {x ∈ L | f (x) = 0}. Since H = L, the functional f is non-trivial. Conversely, let f be a non-trivial linear functional. Define the subspace S = {x ∈ L | f (x) = 0}. Let x0 ∈ L be such that f (x0 ) = 1. For every x ∈ L we have f (x − f (x)x0 ) = 0,

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Linear Spaces

page 111

111

hence x − f (x)x0 ∈ S. Therefore, L =  {x0 } ∪ S, which means that S is a proper maximal subspace of L. If a ∈ R let x1 ∈ L be such that f (x1 ) = a. Then {x ∈ L | f (x) = a} = {x ∈ L | f (x − x0 ) = 0} = x1 + S, so {x ∈ L | f (x) = a} is a hyperplane in L.



Theorem 2.55. If H is a hyperplane in the real linear space L such that 0L ∈ H. There exists a unique non-trivial linear functional f on L such that H = {x ∈ L | f (x) = 1}. Proof. By Theorem 2.54 there exists a functional f that satisfies the condition of the theorem. If g is another functional such that H = {x ∈ L | g(x) = 1} then H ⊆ {x ∈ L | f (x) − g(x) = 0} and {x ∈ L | f (x) − g(x) = 0} is a subspace of L. Since the smallest subspace that contains H is L, it follows that f = g.  A hyperplane H in a linear space L, defined by H = {x ∈ L | f (x) = a}, where f is a linear functional generates four subsets known as half-spaces shown in Table 2.1. Table 2.1 Half spaces defined by a hyperplane. Designation Definition negative closed {x ∈ L | f (x)  a} half space negative open {x ∈ L | f (x) < a} half space positive closed {x ∈ L | f (x)  a} half space positive open {x ∈ L | f (x) > a} half space

By Theorem 2.54 for every hyperplane H in Rn there exists a linear functional f on L and a number a ∈ R such that H = {x ∈ L | f (x) = a}. Applying Theorem 2.50, there exists w ∈ Rn such that f (x) = (w, x). Thus, each hyperplane H in Rn can be written as H = {x ∈ Rn | x1 w1 + · · · + xn wn = a}, where w1 , . . . , wn , a ∈ Rn . This allows us to use the alternative notation Hw,a for H. Note that it is impossible to have both a = 0n and a = 0 because, in this case, we would have H = Rn .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 112

Mathematical Analysis for Machine Learning and Data Mining

112

If x0 ∈ Hw,a , then w x0 = a, so Hw,a is also described by the equality: Hw,a = {x ∈ Rn | w (x − x0 ) = 0}. The negative closed half-space and the positive closed half-space introduced in Table 2.1 are denoted as  Hw,a = {x ∈ Rn | w x  a},  = {x ∈ Rn | w x  a}, Hw,a

respectively. Similarly, the positive and negative open half-spaces are > Hw,a = {x ∈ Rn | w x > a}, < = {x ∈ Rn | w x < a}. Hw,a

respectively. If x1 , x2 ∈ Hw,a , then w ⊥ x1 − x2 . Since x1 − x2 is located in Hw,a , it follows that w is orthogonal on any vector in Hw,a . This justifies referring to w as the normal to the hyperplane Hw,a . Observe that a hyperplane is fully determined by a vector x0 ∈ Hw,a and by w. Let x0 ∈ Rn −Hw,a , then the line passing through x0 and is orthogonal on Hw,a is described by x − x0 = λw, where λ ∈ R. Therefore, the intersection of this line with Hw,a is given by w (x0 − λw) = a, which means that λ = x = x0 +

w x0 −a w 2 w.

w x0 −a w 2 .

Thus, the intersection of with Hw,a is

Thus, the closest point in Hw,a to x0 is x = x0 −

w x0 − a w. w2

The smallest distance between x0 and a point in the hyperplane Hw,a is given by x0 − x =

|w x0 − a| . w

If we define the distance d(Hw,a , x0 ) between x0 and Hw,a as this smallest distance we have: d(Hw,a , x0 ) =

|w x0 − a| . w

(2.14)

A hyperplane Hw,a in Rn+1 is said to be vertical if wn+1 = 0; otherwise, Hw,a is said to be non-vertical.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Linear Spaces

9in x 6in

b3234-main

page 113

113

Exercises and Supplements (1) Let {x1 , . . . , xn } be a linearly independent set in the linear space L. Prove that if the set {x1 + x, . . . , xn + x} is linearly dependent, then x ∈  x1 , . . . , xn . (2) Let L be a linear space. Prove that L is infinite-dimensional if and only if there exists a sequence of vectors (xn ) in L such that {x1 , . . . , xn } is linearly independent for each n ∈ N. (3) Let L be a finite-dimensional linear space and let K be a subspace of L. Prove that if dim(K) = dim(L), then K = L. (4) Let L be an F-linear space n and let f, g1 , . . . , gn be linear functionals. Prove that f = i=1 ai gi for some a1 , . . . , an if and only if n i=1 Null(gi ) ⊆ Null(f ). n n Solution: It is clear that if f  = i=1 ai gi , then i=1 Null(gi ) ⊆ n Null(f ). Conversely, assume that i=1 Null(gi ) ⊆ Null(f ). Define the operator h : L −→ Fn as h(x) = (g1 (x), . . . , gn (x)). If h(x) = h(y), then gi (x) = gi (y), or gi (x − y) = 0 for 1  i  n, hence x − y ∈  n i=1 Null(gi ) ⊆ Null(f ). Thus, the linear functional k : h(L) −→ F given by k(g1 (x), . . . , gn (x)) = f (x) for x ∈ L is well-defined. By extending k obtain the existence of the to the entire Fn , we  nscalars a1 , . . . , an such a x . Thus, f (x) = that k(x1 , . . . , xn ) = n i i i=1 i=1 ai gi (x). (5) Let L, K be two F-linear spaces and let h : L −→ K be a linear operator. Prove that if h is injective and {x1 , . . . , xn } is a linearly independent set in L, then {h(x1 ), . . . , h(xn )} is linearly independent in K. (6) Let L be a linear space. Prove that if h : L −→ L is a linear mapping such that both Null(h) and Img(h), then L is finite-dimensional. (7) Let h1 , h2 be two linear operators on a finite-dimensional linear space L. Prove that h1 h2 is invertible if and only if both h1 and h2 are invertible. (8) Let L be a finite-dimensional F-space and let h : L −→ L be a linear operator. Prove that h is invertible is equivalent to h being injective, and also, with h being surjective. Let L1 , L2 and L be real linear spaces. A mapping φ : L1 × L2 −→ L is bilinear if the following conditions are satisfied: (i) the mapping φx1 : L2 −→ L defined by φx1 (x2 ) = φ(x1 , x2 ) is linear for every x1 ∈ L1 ; (ii) the mapping φx2 : L1 −→ L defined by φx2 (x1 ) = φ(x1 , x2 ) is linear for every x2 ∈ L2 . A duality is a quadruple (L1 , L2 , L, φ), where and φ : L1 × L2 −→ L is a bilinear mapping such that the following supplementary conditions are satisfied:

May 2, 2018 11:28

114

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 114

Mathematical Analysis for Machine Learning and Data Mining

(i) if φ(x1 , x2 ) = 0L for every x2 ∈ L2 , then x1 = 0L1; (ii) if φ(x1 , x2 ) = 0L for every x1 ∈ L1 , then x2 = 0L2 .

(9) Prove that:

 (a) (Rn , Rn , R, φ) is a duality, where φ(x, y) = n i=1 xi yi ; (b) if L is a real linear space and L is its algebraic dual (that consists of all linear functionals defined on L), then (L, L , R, φ) is a duality, where φ(x, f ) = f (x) for x ∈ L and f ∈ L .

(10) Prove that if (L1 , L2 , L, φ) is a duality, than L1 is isomorphic to a subspace of the linear space LL2 . Solution: Let F : L1 −→ LL2 be defined by F (x1 ) = φx1 for x1 ∈ L1 . It is immediate that F is a linear mapping. Suppose that F (x) = F (y) for x, y ∈ L1 , that is, φx (u) = φy (u) for u ∈ L2 , or φ(x − y, u) = 0 for u ∈ L1 . Thus, x − y = 0L , or x = y, so F is injective. This proves that L is isomorphic to F (L), a subspace of LL2 . (11) Let L1 , L2 , L be inner product real linear spaces. Prove that if φ : L1 × L2 −→ L is a bilinear mapping such that φ(x1 , x2 )  x1 x2 , then φ is a duality. (12) Consider the linear operator h : R2 −→ R2 over the linear space R2 defined as

x1 −x2 h = x2 x1 for x1 , x2 ∈ R. Prove that h has no eigenvalue. Solution: Suppose that λ were an eigenvalue and that x = were an eigenvector of h, that is:

  x1 x2

= 02

x1 −x2 =λ . x1 x2 This implies −x2 = λx1 and x1 = λx2 . At least one of x1 , x2 is not equal to 0. If x1 = 0, we have x1 (1 + λ2 ) = 0, which is contradictory, etc. (13) Let L be a linear space and let h1 , h2 : L −→ L be two linear operators. Prove that h1 h2 and h2 h1 has the same set of eigenvalues. (14) Let h : L −→ L be an invertible linear operator. Prove that λ = 0 is an eigenvalue of h if and only if λ1 is an eigenvalue of h−1 . (15) Let h : L −→ L be a linear operator such that every x ∈ L is an eigenvector of h. Prove that there exists a constant a ∈ F such that h = a1L . (16) Determine the eigenvalues and eigenvectors of the linear operator h : R2 −→ R2 defined by h(x) = (x1 + x2 , x1 + x2 ) for x ∈ R2 .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Linear Spaces

b3234-main

page 115

115

(17) Let h : L −→ L be a linear operator. Prove the following equality involving the resolvents of h: Rh,λ − Rh,μ = (λ − mu)Rh,λ Rh,μ . Solution: We have (h − λ1L )(Rh,λ − Rh,μ )(h − μ1L ) = (1L − (h − λ1L )Rh,μ )(h − μ1L ) = ((h − μ1L ) − (h − λ1L )) = (λ − μ)1L , which implies desired equality. (18) Let {u1 , . . . , un , . . .} and {v1 , . . . , vn , . . .} be two orthonormal sets in an inner product space L. Prove that (uj −vj , ui ) = (ui −vi , vj ) for i, j  1. The subsets B = {b1 , . . . , bn } and C = {c1 , . . . , cn } of Rn are reciprocal if (bi , cj ) = 1 if i = j and (bi , cj ) = 0 if i = j, for 1  i, j  n. (19) Let B = {b1 , . . . , bn } be a basis of Rn . Prove that there exists a unique reciprocal set of B. Solution: Let Ui be the subspace of Rn generated by the set B − {bi } and let Ui⊥ be its orthogonal complement. We have dim(Ui⊥ ) = 1 because dim(Ui ) = n − 1. Thus, there exists a vector t = 0 in Ui⊥ . Note that (t, bi ) = 0 because bi ∈ Ui . Define ci =

1 bi . (t, bi )

Then, (bi , ci ) = 1 and (bi , cj ) = 0 if j = i. This construction can be applied to all i, where 1  i  n and this yields a set C = {c1 , . . . , cn }, which is reciprocal to B. To prove the uniqueness of the set C, assume that D = {d1 , . . . , dn } is another reciprocal set of the basis B. Then, since (bi , cj ) = (bi , dj ), it follows that (bi , cj − dj ) = 0 for every i, j. Since cj − dj is orthogonal on all vectors of B it follows that cj − dj = 0, so cj = dj . Thus D = C. (20) If B = {b1 , . . . , bn } is a basis of Rn then the reciprocal set C of B is also a basis of Rn . (21) Let ν be a norm on Cn . Prove that there exists  a number k ∈ R such that for any vector x ∈ Cn we have ν(x)  k n i=1 |xi |. (22) Prove that if x, y, z are three vectors in Rn and ν is a norm on Rn , then ν(x − y)  ν(x − z) + ν(z − y).

May 2, 2018 11:28

116

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 116

Mathematical Analysis for Machine Learning and Data Mining

(23) Prove that for any vector norm ν on Rn we have ν(x + y)2 + ν(x − y)2  4(ν(x)2 + ν(y)2 ) for every x, y ∈ Rn . (24) Let x ∈ Rn . Prove that for every  > 0 there exists y ∈ Rn such that the components of the vector x + y are distinct and y2 < . (25) Prove that if x ∈ Rn , then x1 = max{a x | a ∈ {−1, 1}n }. (26) Let ν0 : Rn −→ R be the function defined by ν0 (x) = {i | 1  i  n, xi = 0}. Prove that ν0 is not a norm, although ν0 (x + y)  ν0 (x) + ν0 (y). (27) Let p : L −→ V be a projection of a linear normed space on a subspace V and let h be a linear operator on L. If v ∈ V , u − h(u) − b = v − h(v) − b = 0L , and 1L − ph is invertible, prove that u − v  (1L − ph)−1  u − p(u). Solution: Since v ∈ V we have p(v) = v. Therefore, from v − h(v) − b = 0L it follows that p(v − h(v) − b) = 0L , or v − ph(v) = p(b). Taking into account that u − h(u) = b we obtain p(u) − ph(u) = p(b), hence v − ph(v) = p(u) − ph(u), or u − v − ph(u − v) = u − p(u), which amounts to (1L − ph)(u − v) = u − p(u), hence u − v = (1L − ph)−1 (u − p(u)). This implies the desired inequality. (28) Let {x1 , . . . , xn } be a linearly independent space in a normed linear space L. Prove that there exists δ > 0 such that if yi − xi  < δ for 1  i  n, then the set {y1 , . . . , yn } is linearly independent.

Bibliographical Comments Among the many useful references we mention [98, 67, 132, 116, 93] and [121], a volume dedicated to applications of linear algebra in data mining. An novel, interesting approach to linear algebra that emphasizes linear operators is provided in [5, 4].

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 117

Chapter 3

Algebra of Convex Sets

3.1

Introduction

The notion of convex set that we study in this chapter is a pillar of optimization theory and is essential for the study of convex functions. After introducing convex sets in linear spaces and presenting several examples of such sets we discuss techniques for generating new convex sets starting from existing such sets. Cones are another type of subsets of linear spaces that we present in this chapter. Extreme points of convex sets are important in optimization problem and we treat then in Section 3.5. Finally, we present balanced and absorbing sets that are useful in the study of certain topological spaces. Special types of convex sets (polytopes and polyhedra) are studied in the last section. 3.2

Convex Sets and Affine Subspaces

Let L be a real linear space and let x, y ∈ L. The closed segment determined by x and y is the set [x, y] = {(1 − a)x + ay | 0  a  1}. The closed-open segment determined by x and y is the set: [x, y) = {(1 − a)x + ay | 0  a < 1}. The open-closed segment determined by x and y is the set: (x, y] = {(1 − a)x + ay | 0 < a  1}. The open segment determined by x and y is (x, y) = {(1 − a)x + ay | 0 < a < 1}. 117

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 118

Mathematical Analysis for Machine Learning and Data Mining

118

A line passing through x0 and having direction u is a set of the form

= {z ∈ L | z = x0 + tu, t ∈ R}, where x0 ∈ L and u = 0L . Suppose that x and y are two points located on the line . If z ∈ L, then the vectors z − y and y − z are collinear, so there exists t such that t 1 1 x + 1+t y. If we denote a = 1+t , z has the form y − z = t(z − x), or z = 1+t z = (1 − a)x + ay for t ∈ R. Thus, the line determined by x and y has the form

x,y = {(1 − a)x + ay | a ∈ R}. It is clear that for any x, y ∈ L we have (x, y) ⊆ [x, y), (x, y] ⊆ [x, y] ⊆ x,y . Definition 3.1. A subset C of L is convex if we have [x, y] ⊆ C for all x, y ∈ C. Note that the empty subset and every singleton {x} of L are convex. In Figure 3.1(a) we show a convex set in R2 ; the quadrilateral in Figure 3.1(b) is not convex for both x, y are inside the quadrilateral, while the segment [x, y] is not included in the quadrilateral. x3 x4 x4

x2

x x2

x

y x3

y

x1 (a) Fig. 3.1

x1 (b) Convex set (a) vs. a non-convex set (b).

Example 3.1. The set Rn0 of all vectors of Rn having non-negative components is a convex set called the non-negative orthant of Rn . Example 3.2. The convex subsets of (R, +, ·) are the intervals of R. Regular polygons are convex subsets of R2 .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Algebra of Convex Sets

page 119

119

Example 3.3. Every linear subspace T of a real linear space L is convex. Example 3.4. Let (L,  · ) be a normed linear space. An open sphere B(x0 , r) ⊆ L is convex. Indeed, suppose that x, y ∈ B(x0 , r), that is, x − x0  < r and x0 − y < r. Let a ∈ [0, 1] and let z = (1 − a)x + ay. We have x0 − z = x0 − (1 − a)x − ay = a(x0 − y) + (1 − a)(x0 − x)  ax0 − y + (1 − a)x0 − x < r, so z ∈ B(x0 , r). Similarly, a closed sphere B[x0 , r] is a convex set. Next we introduce a local variant of convexity. Definition 3.2. A non-empty subset S of a linear space L and let x ∈ S. The set S is locally convex set or star-shaped at x if for every y ∈ S we have (1 − a)x + ay ∈ S for all a ∈ [0, 1]. Every non-empty convex subset S of L is locally convex at every x ∈ S; conversely, every non-empty subset S of L that is locally convex at every x ∈ S is convex. Theorem 3.1. Let x, y, z be three distinct points in the real linear space L such that z ∈ x,y . Then, one of these points belongs to the open segment determined by the remaining two points. Proof. Since z ∈ x,y , we have z = (1 − a)x + ay for some a ∈ R (see Figure 3.2). y

 a > 1  u

  u  0 0. Then, if t < 0 and |t| is sufficiently large, we would have x + tu ∈ (R0 )n . Thus, no line is included in (R0 )n . Theorem 3.14. Let L be a real linear space and let C ∈ L be a cone. C is convex if and only if x + y ∈ C for x, y ∈ R. Proof. Let C be a convex cone. If x, y ∈ C and a ∈ (0, 1), then a1 x ∈ C 1 y ∈ C. Therefore, by convexity we have and 1−a 1 1 y ∈ C. x + y = a x + (1 − a) a 1−a Conversely, let C be a cone such that x, y ∈ C imply x + y ∈ C. For u, v ∈ C and a ∈ [0, 1] let za = au + (1 − a)v. Since C is a cone, au ∈ C and (1 − a)v ∈ C, hence za = au + (1 − a)v ∈ C. Therefore, C is convex. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Algebra of Convex Sets

9in x 6in

b3234-main

page 131

131

Example 3.10. Let U be a non-empty subset of a real linear space L. The set of all non-negative combinations of U is a convex cone that is included in every convex cone that contains U . Theorem 3.15. The intersection of any collection of cones (convex cones) in a real liner space L is a cone (a convex cone). Proof. The proof is similar to the argument of Theorem 3.8 and is omitted.  Corollary 3.4. The families of cones (convex cones) in a real linear space L is a closure system. Proof. This statement follows immediately from Theorem 3.15 by  observing that Rn itself is cone (a convex cone). We will denote the closure operator corresponding to the family of cones by Kcone ; the similar operator for convex cones will be denoted by Kccone . Theorem 3.16. Let S be a non-empty subset of a real linear space L. We have Kcone (S) = {ax | a  0 and x ∈ S}. Proof. We saw in Example 3.8 that {ax | a  0 and x ∈ S} is a cone that contains S, so Kcone (S) ⊆ {ax | a  0 and x ∈ S}. Conversely, since S ⊆ Kcone (S), if x ∈ S it follows that ax ∈ Kcone (S)  for every a  0, so {ax | a  0 and x ∈ S} ⊆ Kcone (S). Theorem 3.17. (Carath´ eodory’s Theorem for Cones) Let S be a subset of Rn and let x ∈ Kcone (S), x = 0n . Then, x is a positive linear combination of no more than n linearly independent vectors of S. Proof. Let x be a non-null vector of Kcone (S) and let m be the smallest integer such that x can be written as a positive linear combination of vectors m of S, x = i=1 ai xi . If x1 , . . . , xm were linearly dependent we would have the non-zero numbers b1 , . . . , bm ∈ R such that at least one of theses numbers is positive and m i=1 bi xi = 0, which allows us to write m  x= (ai − cbi )xi i=1

for c ∈ R. Let c0 = max{c ∈ R | ai − cbi  0 for 1  i  m}. At least one of the numbers ai − c0 bi is zero because, if this were not the case, the definition of c0 would be contradicted. This contradicts the minimality of  m. Therefore, x1 , . . . , xm are linearly independent.

May 2, 2018 11:28

132

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 132

Mathematical Analysis for Machine Learning and Data Mining

Cones define partial order relations on linear spaces. Definition 3.11. A partially ordered linear space is a pair (L, ), where L is a linear space and  is a partial order on L such that x  y and u  v imply x + u  y + v and ax  ay for every a ∈ R0 . Theorem 3.18. Let (L, ) be a partially ordered linear space. The set C = {x ∈ L | 0L  x} is a pointed convex cone. Conversely, if C is a pointed convex cone in L, the relation “” defined by x  y if y − x ∈ C is a partial order. Proof. If is immediate that C is a convex cone (by Theorem 3.14). Suppose that both x and −x belong to C, that is, 0L  x and 0L  −x. This implies x = 0L , so C is indeed a pointed cone. For the converse implication let C be a pointed convex cone. Since 0L ∈ C it follows that x  x. Further, suppose that x  y and y  x, that is, y − x ∈ C and x − y = −(y − x) ∈ C. This implies y − x = 0L , so x = y, which shows that “” is anti-symmetric. Finally, if x  y and y  z, that is, if y − x, z − y ∈ C we have y − z = (y − x) + (x − z) ∈ C because C is a convex cone.  The pointed convex cone that generates a partial order  is referred in this context as an ordering cone.

3.5

Extreme Points

Definition 3.12. Let C be a non-empty convex subset of a real linear space L. An extreme point of C is a point x ∈ C such that if x ∈ [u, v] and u, v ∈ C, then u = v = x. Theorem 3.19. Let C be a non-empty convex subset of a real linear space L. A point x ∈ C is an extreme point of C if the set C − {x} is convex. Proof. Suppose that C −{x} is a convex set for x ∈ C and that x ∈ [u, v], where u, v ∈ C. If x is distinct from both u and v, then u, v belong to the convex set C − {x}, which yields the contradiction x ∈ C − {x}. Thus, x is an extreme point of C. Conversely, suppose that x is an extreme point of C. Let u, v ∈ C −{x}, so u = x and y = x. If x ∈ [u, v], we obtain a contradiction since this implies u = v = x. Therefore [u, v] ⊆ C − {x}, so C − {x} is convex. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Algebra of Convex Sets

page 133

133

The set of extreme points of a convex set C is denoted by extr(C). Example 3.11. Let B[x0 , r] be a closed sphere of radius r in Rn . Each point x located on the circumference of this sphere, that is, each point x such that x0 − x = r is an extreme point of B[x0 , r]. Indeed, suppose that au+(1−a)v = x for some a ∈ (0, 1) and x0 −u = x0 − v = r. Then, by Supplement 6, we have u = v = x. Example 3.12. An open sphere B(x0 , r) in Rn has no extreme points for if x ∈ B(x0 , r). Indeed, let u be a vector such that u = 0n and let x1 = x+au 0 , we have and x2 = x − au, where a > 0. Observe that if a < r− x−x u x1 − x0  = x + au − x0   x − x0  + au < r and x2 − x0  = x − au − x0   x − x0  + au < r, and we have both x1 ∈ B(x0 , r) and x2 ∈ B(x0 , r). Since x = 12 x1 + 12 x2 , x is not an extreme point. Example 3.13. The extreme points of the cube [0, 1]n are all its 2n “corners” (a1 , . . . , an ) ∈ {0, 1}n. Definition 3.13. Let C be a convex set in a real linear space L. A convex subset F of C is a face of C if for every open segment (u, v) ⊆ C such that at least one of u, v is not in F we have (u, v) ∩ F = ∅. If F = C, we say that F is a proper face of C. A k-face of C is a face F of C such that dim(F ) = k. In other words, a convex subset F is a face of C if u, v ∈ C and (u, v) ∩ F = ∅ implies u ∈ F and v ∈ F , which is equivalent to [u, v] ⊆ F . Note that if F = {x} is a face of C if and only if x ∈ extr(C). An convex subset C is a face of itself. Example 3.14. Let C be a convex subset of Rn and let a ∈ Rn . The set Fa = {z ∈ C | a z  a x for every x ∈ C} is a face of C. Indeed, suppose that there exists z ∈ (u, v) such that z ∈ Fa , that is,  a z  a x for every x ∈ C. Since z ∈ (u, v), z = (1 − t)u + tv for some t ∈ (0, 1), so a z = (1 − t)a u + ta v ≤ a x for every x ∈ C. In particular, for x = u and x = v we obtain a (v − u)  0 and a (v − u)  0, so a (v − u) = 0, which implies a u = a v = az. Therefore, u, v ∈ Fa , so Fa is indeed a face of C. Any face of C of the form Fa is called an exposed face of C.

May 2, 2018 11:28

134

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 134

Mathematical Analysis for Machine Learning and Data Mining

Theorem 3.20. If F is a face of a convex set C, then F = Kaff (F ) ∩ C. k Proof. If z ∈ Kaff (F )∩C, we have x = a1 y1 +· · ·+ak yk , where i=1 ai = 1 and y1 , . . . , yk ∈ F . If all ai are non-negative, then it is immediate that  x ∈ F . Otherwise, let b = − {ai | ai < 0} and let 1  u= {ai yi | ai  0}, 1+b 1 v=− {ai yi | ai < 0}. b We have x ∈ C, v ∈ C, and b 1 x+ ∈ [x, v] ∩ F. u= 1+b 1+b Since F is a face, we have u ∈ F . Thus, Kaff (F ) ∩ C ⊆ F . The reverse inclusion is immediate.  A functional on an F-linear space L is a function f : L −→ F. A linear functional on L is a functional f on L that satisfies the condition f (ax + by) = af (x) + bf (y) for a, b ∈ F and x, y ∈ L. If F(C) is the collection of faces of a convex set C, then C ∈ F(C) and  F is a face of C. Thus, F(C) is a closure system. Example 3.14 that includes the definition of a face of a convex set in Rn is generalized in the next statement. Theorem 3.21. Let C be a convex subset of a real linear space L and let f : L −→ R be a linear functional that is not a constant on C and k = sup{f (x) | x ∈ C} is finite. If the set F = {x | f (x) = k} is non-empty, then it is a proper face of C. Proof. It is immediate that the set F is convex. If y, z ∈ F and a ∈ (0, 1) is such that ay + (1 − a)z ∈ F , it follows that af (y) + (1 − a)f (z) = k. Since f (y)  k and f (z)  k, it follows that f (y) = f (z) = k, so y, z ∈ F . This shows that F is a face of C.  As it was the case in Example 3.14, faces of a convex set that can be defined using linear functionals as shown in Theorem 3.21 are referred to as exposed faces. The hyperplane {x | f (x) = k} is called a support hyperplane of C. Theorem 3.22. Let S be a subset of Rn . The set Q(S) defined by  Q(S) = {x ∈ Rn | [x, y] ⊆ S for every y ∈ S} is a convex set.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Algebra of Convex Sets

page 135

135

Proof. Suppose that u, v ∈ Q(S). We need to prove that w = bu + (1 − b)v ∈ Q(S) for every b ∈ [0, 1]. This means that for every y ∈ S and c ∈ [0, 1] we need to show that t = cw + (1 − c)y ∈ S. Suppose initially that bc < 1 and let d = c(1−b) 1−bc . Clearly, d is defined if bc < 1 and that d ∈ [0, 1]. Let z = dv + (1 − d)y. Clearly, z ∈ vy, so z ∈ S. Note that we can write t = bcu + (1 − bc)z, which means that t ∈ uz, so t ∈ S. If bc = 1 we have both b = 1 and c = 1, so w = u and t = w, when the statement holds trivially.  Example 3.15. Let Rn×n be the linear space of square matrices of format n × n. The set of (n × n)-stochastic matrices is a convex set in Rn×n . Let A, B ∈ Rn×n be two stochastic matrices, let a ∈ [0, 1], and let C = (1 − a)A + aB. We have n n n    cij = (1 − a) aij + a bij = 1, j=1

j=1

j=1

because n 

aij =

j=1

n 

bij = 1,

j=1

so C is a stochastic matrix. Similarly, the set of doubly-stochastic matrices is convex in Rn×n . Theorem 3.23. If C is a convex subset of a real linear space L, then for any translation tz , the set tz (C) is convex. Proof. Let x and y be two elements of tz (C). There exist u and v in C such that x = u + z and y = v + z. If w is a convex combination of x and y, then there exists a ∈ [0, 1] such that w = ax + (1 − a)y = a(u + z) + (1 − a)(v + z) = au + (1 − a)v + z = tz (au + (1 − a)v) ∈ tz (C), which proves that tz (C) is convex.



We observed that permutation matrices are doubly-stochastic matrices. Therefore, any convex combination of permutation matrices is a doublystochastic matrix. An important converse statement is discussed next. Theorem 3.24. (Birkhoff-von Neumann Theorem) If A ∈ Rn×n is a doubly-stochastic matrix, then A is a convex combination of permutation matrices.

May 2, 2018 11:28

136

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 136

Mathematical Analysis for Machine Learning and Data Mining

Proof. Let A ∈ Rn×n be a doubly-stochastic matrix and let R = {r1 , . . . , rm } be the set of its rows, and C = {c1 , . . . , cm } be the set of its columns, respectively. Define the relation ρ ⊆ R × C as ρ = {(ri , cj ) | ri ∈ R, cj ∈ C, aij > 0}.  Since A is a doubly-stochastic matrix we have nj=1 aij = 1 for every i, m 1  i  m and i=1 aij = 1 for every j, 1  j  n. For a set of rows T we have  {aij | ri ∈ T, cj ∈ ρ[{ri }]} = |T |, because for each row ri the sum of the elements aij equals 1. Also, for each set of columns Z, we have  {aij | cj ∈ Z, ri ∈ ρ−1 [{cj }]} = |Z|. Therefore, |ρ[T ]| = 

 

{aij | ri ∈ T, cj ∈ C} {aij | ri ∈ T, cj ∈ ρ[{ri }]} = |T |,

which shows that we can apply Hall’s Matching Theorem to R, C and ρ. By this theorem there exists a matching f for ρ. Define the matrix P by  1 if f (ri ) = cj , pij = 0 otherwise. We claim that P is a permutation matrix. For every row i of P there exists a column cj such that pij = 1. There exists only one 1 in the j th column of P for, if pi1 j = pi2 j = 1 for i1 = i2 , it would follow that we have both f (i1 ) = j and f (i2 ) = j contradicting the fact that f is a matching. Let a = min{aij | pij = 0}. Clearly, a > 0 and a = apq for some p and q. Let B = A − aP . If B = On,n , then A is a permutation matrix. Otherwise, note that n n (i) j=1 cij = 1 − a and i=1 cij = 1 − a; (ii) 0 ≤ cij ≤ 1 − a for 1 ≤ i ≤ n and 1 ≤ j ≤ n; (iii) cpq = 0. 1 C is doubly-stochastic and we have A = Therefore, the matrix D = 1−a aP + (1 − a)D, where D has at least one more zero element than A. The equality A = aP + (1 − a)D shows that A is a convex combination of a permutation matrix and a doubly-stochastic matrix with strictly more zero components than A. The statement follows by repeatedly applying this procedure. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Algebra of Convex Sets

9in x 6in

b3234-main

page 137

137

Theorem 3.25. (Carath´ eodory’s Theorem) If U is a subset of Rn , n+1 then for every x ∈ Kconv (U ) we have x = i=1 ai xi , where xi ∈ U , ai  0 n+1 for 1  i  n + 1, and i=1 ai = 1. p+1 Proof. Consider x ∈ Kconv (U ). We can write x = i=1 ai xi , where p+1 xi ∈ U , ai  0 for 1  i  p + 1, and i=1 ai = 1. Let p be the smallest number which allows this kind of expression for x. We prove the theorem by showing that p  n. Suppose that p  n + 1. Then, the set {x1 , . . . , xp+1 } is affinely dep+1 pendent, so there exist b1 , . . . , bp+1 not all zero such that 0n = i=1 bi xi p+1 and i=1 bi = 0. Without loss of generality, we can assume bp+1 > 0 and ap+1 ai bp+1  bi for all i such that 1  i  p and bi > 0. Define   ai ap+1 ci = b i − bi bp+1 for 1  i  p. We have p p p   ap+1  ci = ai − bi = 1. bp+1 i=1 i=1 i=1 Furthermore, ci  0 for 1  i  p. Indeed, if bi  0, then ci  ai  0; if ap+1  abii for all i such that 1  i  p and bi > 0, then ci  0 because bp+1 bi > 0. Thus, we have:  p p  p    ap ci x i = ai xi = x, ai − b i x i = bp i=1 i=1 i=1 

which contradicts the choice of p.

Theorem 3.26 (Radon’s Theorem). Let P = {xi ∈ Rn | 1  i  n + 2} be a set of n + 2 points in Rn . Then, there are two disjoint subsets R and Q of P such that Kconv (R) ∩ Kconv (Q) = ∅. Proof. Since n + 2 points in Rn are affinely dependent, there exist a1 , . . . , an+2 not all equal to 0 such that n+2  ai x i = 0 (3.3) i=1

n+2

and i=1 ai = 0. Without loss of generality, we can assume that the first  k numbers are positive and the last n + 2 − k are not. Let a = ki=1 ai > 0 a and let bj = aj for 1  j  k. Similarly, let cl = − aal for k + 1  l  n + 2. Equality (3.3) can now be written as k n+2   bj xj = cl x l . j=1

l=k+1

May 2, 2018 11:28

138

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 138

Mathematical Analysis for Machine Learning and Data Mining

k n+2 Since the numbers bj and cl are non-negative and j=1 bj = l=k+1 cl = 1, it follows that Kconv ({x1 , . . . , xk }) ∩ Kconv ({xk+1 , . . . , xn+2 }) = ∅.  Theorem 3.27. (Shapley-Folkman Theorem) Let X1 , . . . , Xm be nonempty subsets of Rn and let X = X1 + · · · + Xm be their Minkowski sum. Every x ∈ Kconv (X) can be represented as a sum x = x1 + . . . + xm such that (i) xi ∈ Kconv (Xi ), and (ii) |{i | 1  i  m, xi ∈ Xi }|  m − n. m Proof. Let x = i , where yi ∈ Kconv (Xi ) and let yi = i=1 y   mi mj j=1 aij = 1 and yij ∈ Xi for 1  j  mi . j=1 aij yij , where aij > 0, Consider the vectors in Rn+m :       x y1j ymj z= , z1j = , . . . , zmj = , e1 em 1m m mi eodory’s Theorem for Cones so that z = j=1 aij zij . By Carath´ i=1 (Theorem 3.17) we can write z=

mi m  

bij zij ,

i=1 j=1

where bij  0 and at most n + m of them are positive. In particular, x=

mi m   i=1 i=1

bij xij and

mi 

bij = 1 for 1  i  m.

j=1

 i If xi = m j=1 bij yij for 1  i  m, then x = x1 + · · · + xm . Since for each i at least one of bi1 , . . . , bimi is positive and at most n + m of bij are positive, it follows that for at least m − n indices i we have bik = 1 for some k and  bij = 0 for all j = k. 3.6

Balanced and Absorbing Sets

Definition 3.14. Let L be a real linear space. A subset W of L is (i) balanced if |r|  1 implies rW ⊆ W ; (ii) symmetric if w ∈ W implies −w ∈ W ; (iii) absolutely convex if it is both convex and balanced.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Algebra of Convex Sets

b3234-main

page 139

139

The subset W is balanced if for each w ∈ W we have the segment [−w, w] ⊆ W . Clearly, every balanced set is symmetric. Example 3.16. The closed sphere B[0, 1] of a real normed linear space (L,  · ) is balanced and, therefore, is absolutely convex. Also, it is clear that the empty subset of L is balanced. Theorem 3.28. Let L be a real linear space and let U be a balanced set. The following statements hold: (i) 0L ∈ U ; (ii) U is symmetric, that is, −U = U ; (iii) aU is balanced for every a ∈ R; (iv) for x ∈ L and a ∈ R we have ax ∈ U if and only if |a|x ∈ U ; (v) if |a|  |b| for some a, b ∈ R, then aU ⊆ bU . Proof. The arguments for the first four parts are straightforward. We prove only part (v). If b = 0, then a = 0 and both aU and bU equal {0},  so we obtain the inclusion. Suppose now that b = 0. We have aU = b ab U ⊆ bU because |a|  |b|  1 and U is balanced. It is easy to see that the collection of balanced sets of a real linear space is closed with respect to arbitrary union and intersection. Definition 3.15. Let L be a real linear space. A subset U of L is absorbing if for every x ∈ L there exists δ > 0 such that |a|  δ implies ax ∈ U . Note that if U is an absorbing subset of L we have 0L ∈ U . Also, any set W that includes an absorbing set is itself absorbing. Example 3.17. The closed sphere B[0L , 1] of a real normed linear space (L,  ·) is an absorbing set because for x ∈ L, a  x implies x ∈ aB[0, 1]. For a seminorm ν defined on a real linear space L we consider the sets Bν (x0 , r) = {x ∈ K | ν(x − x0 ) < r}, Bν [x0 , r] = {x ∈ K | ν(x − x0 )  r}. These sets represent generalizations of the sets Bd (x0 , r) and Bd [x0 , r] previously introduced for metric spaces in Definition 1.49 and will be referred to as the open sphere and the closed sphere determined by ν, respectively.

May 2, 2018 11:28

140

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 140

Mathematical Analysis for Machine Learning and Data Mining

Example 3.18. Let ν be a seminorm on the linear space L. The set Bν (0, r) = {x ∈ L | ν(x) < r} is absorbing. Indeed, if ν(x) = 0, then ax ∈ Bν (0, r) for every a because ν(ax) = |a|ν(x) = 0. r , then ν(ax) = |a|ν(x) < r, that is ax ∈ If ν(x) > 0 and |a| < ν(x) Bν (0, r), which shows that Bν (0, r) is indeed, absorbing. A subset W of L is said to be absorbing in x0 if W − x0 is absorbing. By taking a = 0 it is clear that every absorbing set contains 0L . Furthermore, if W is absorbing in x0 then x0 ∈ W . Theorem 3.29. The collections of absorbing sets of a real linear space L is closed with respect to arbitrary union and finite intersection. Proof. Let {Wi | i ∈ I} be a collection of absorbing sets and let W =  i∈I Wi . Let x ∈ L and let Wi be one of the sets of the collection. Since Wi is absorbing there exists a positive number δ such that |a|  δ implies ax ∈ Wi . Since Wi ⊆ W , it follows that ax ∈ W , so W is absorbing. Let now {Zi | 1  i  n} be a finite collection of absorbing subsets and n let Z = i=1 Zi . Since each of the sets Zi is absorbing, there exist n positive numbers δ1 , . . . , δn such that |a| < δi implies axi ∈ Zi for 1  i  n. Let δ be the least of the numbers δ1 , . . . , δn . Then |a| < δ implies ax ∈ Zi for 1  i  n, that is, ax ∈ Z. Thus, Z is an absorbing set.  Definition 3.16. Let L be a real linear space and let C be a convex set such that 0L ∈ L. The Minkowski functional of C is the functional mC : ˆ 0 given by: L −→ R ˆ >0 | x ∈ rC}. mC (x) = inf{r ∈ R If C is clear from context the subscript C may be omitted. Theorem 3.30. Let L be a real linear space and let C be a convex subset of L. For k ∈ R>0 we have: (i) mC (x) < k implies x ∈ kC and mC (x) > k implies x ∈ kC; (ii) if x ∈ kC, then mC (x)  k. Proof. If mC (x) < k, taking into account the definition of mC (x) as an infimum, there exists r > 0 such that mC (x)  r < k and x ∈ rC ⊆ kC, which gives the first implication of part (i). The second implication of part (i), as well as part (ii) follow from the same definition. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Algebra of Convex Sets

b3234-main

page 141

141

If C is an absorbing set, the set {r ∈ R>0 | x ∈ rC} is non-empty for every x. Therefore, if C is convex and absorbing, mC (x) is defined and is non-negative for every x ∈ L. Example 3.19. Let C = B[0, 1] be the unit sphere centered in 0 in a real normed linear space (L,  · ). It is clear that C is convex, balanced and absorbing. The Minkowski functional of C is given by mC (x) = inf{r ∈ R>0 | x ∈ B[0, r]}. Thus, in this case we have mC (x) = x. Theorem 3.31. Let L be a real linear space and let C be a convex subset of L such that 0L ∈ C. We have: (i) mC (kx) = kmC (x) for k  0; (ii) mC (x + y)  mC (x) + mC (y). Proof.

Note that for k > 0 we have: mC (kx) = inf{r ∈ R>0 | kx ∈ hr (B(0, 1))} = inf{r ∈ R>0 | kx ∈ B(0, r)} = kmC (x).

For k = 0 the equality is immediate. To prove part (ii) consider a positive number and let s, t be two numbers such that mC (x) < s < mC (x) + and mC (y) < t < mC (y) + . Since mC xs < 1, hence xs ∈ C. Similarly, we have yt ∈ C. Since C is convex, we have t y x+y s x + = ∈ C. s+t s s+t t s+t 1 Thus, s+t mC (x + y) < 1. We saw that mC (x + y)  s + t  mC (x) +  mC (y) + 2 . Since is arbitrary we obtain the subadditivity of mC .

Theorem 3.32. Let L be a real linear space and let C be a convex subset of L such that 0L ∈ C. Then mC is a seminorm on L and BmC (0, 1) ⊆ C ⊆ BmC [0, 1]. Moreover, mC is the unique seminorm on L that satisfies this double inclusion.

May 2, 2018 11:28

142

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 142

Mathematical Analysis for Machine Learning and Data Mining

Proof. The subadditivity of mC was shown in Theorem 3.31. Let k ∈ R. If k = 0 we have   k mC (kx) = mC |k| C |k| (by the first part of Theorem 3.31)   k = |k|mC x |k| = |k|mC (x), (because C is a balanced set), which implies the positive homogeneity of mC . The double inclusion can be obtained by taking k = 1 in Theorem 3.30. Suppose now that there exists another seminorm ν on L such that Bν (0, 1) ⊆ C ⊆ Bν [0, 1]. This implies BmC (0, 1) ⊆ Bν [0, 1] and Bν (0, 1) ⊆ BmC [0, 1] due to the transitivity of inclusion. Therefore, mC (x) < 1 implies ν(x)  1 and ν(x) < 1 implies mC (x)  1. 1 1 x for > 0, then mC (z) = mC (x)+ mC (x) < 1 so ν(z) = If z = mC (x)+ 1 mC (x)+ ν(x)  1. Thus, ν(x)  mC (x) + for every positive , which implies ν(x)  mC (x). Similarly, from the fact that ν(x) < 1 implies mC (x)  1 we can show  thatmC (x)  ν(x) for x ∈ L, which shows the uniqueness of mC .

3.7

Polytopes and Polyhedra

Definition 3.17. A polytope is the convex closure of a finite set of points in Rn . A polytope P that is the convex closure of a set V of k + 1 affinely independent points is called a k-simplex or a simplex of dimension k. Definition 3.18. A polyhedron in Rn is a non-empty subset P of Rn that can be written as P = {x ∈ Rn | W  x  b}, where W = (w1 w2 · · · wm ) ∈ Rn×m and b ∈ Rm . The boundary hyperplanes of the polyhedron P are the hyperplanes wi x = bi for 1  i  m.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Algebra of Convex Sets

page 143

143

The polyhedron P defined above is the intersection on m half-spaces  bi for 1  i  m. If a boundary hyperplane Hwj ,bj of P contains P itself, we say that Hwj ,bj is a singular boundary hyperplane. A polyhedron P is a convex set. Indeed, suppose that P = {x ∈ Rn | W  x  b}, and let x, y, that is, W  x  b and W  y  b. For a ∈ [0, 1] we have

wi x

W  (ax + (1 − a)y) = aW  x + (1 − a)W  y  ab + (1 − a)b = b, so ax + (1 − a)y ∈ P . Also, it is clear that every polyhedron is a closed set. Theorem 3.33. Let P be a polyhedron in Rn defined by P = {x ∈ Rn | W  x  b}, where W = (w1 w2 · · · wm ) ∈ Rn×m and b ∈ Rm . Let I be the set of active indices, I = {i | 1  i  m, wi x = bi }. For x ∈ P we have x ∈ extr(P ) if and only if there exist n linearly independent vectors wi1 , . . . , win such that wij x = bij for 1  j  n. Proof. Without loss of generality we may assume that the first n vectors w1 , . . . , wn are linearly independent and wi x = bi for 1  i  n. Suppose that x is not an extreme point. There exists a vector h = 0n such that x + h ∈ P and x − h ∈ P . In other words, we would have wi (x + h) = bi and wi (x − h) = bi for 1  i  n, which implies wi h = 0. The linear independence of w1 , . . . , wn implies h = 0n , which contradicts the initial assumption about h. This implies that x is an extreme point. Conversely, let x be an extreme point of P and let I be the set of active indices I = {i | wi x = bi }. We prove that the set {wi | i ∈ I} is a linearly independent set of n vectors or, equivalently that  {wi | i ∈ I} = Rn (see Theorem 2.4). ⊥ Suppose that this is not the case. Then  {wi | i ∈ I} contains a vector h = 0n . Consider the segment A = [x − h, x + h]. If y ∈ A , we have y = x + ah, where |a|  . Since wi ⊥ h, y ∈ A implies wi y = bi for i ∈ I. If j ∈ I, then wj y = wj x + awj h = wj x  bj if is sufficiently small because |a|  . With this choice of we get A ⊆ P , which contradicts the assumption that x is an extreme point. Thus, {wi | i ∈ I} is linearly independent. 

May 2, 2018 11:28

144

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 144

Mathematical Analysis for Machine Learning and Data Mining

Since every extreme point of the polyhedron P defined by P = {x ∈ Rn | W  x  b}, where W = (w1 w2 · · · wm ) ∈ Rn×m corresponds to a set of n independent columns  of W , it follows that the maximum number of extreme points of P is m n . If x1 , . . . , xk , xk+1 are affinely independent points in Rm , where m  k, we denote the simplex Kconv ({x1 , . . . , xk , xk+1 }) by S[x1 , . . . , xk , xk+1 ]. The vectors x1 , . . . , xk , xk+1 are the vertices of this simplex. In Figure 3.4 we show three simplexes, S[x1 , x2 ], S[x1 , x2 , x3 ], and S[x1 , x2 , x3 , x4 ] in R3 . x4 x3 x2 x2 x1 x1

x3

x1 x2

S[x1 , x2 ]

S[x1 , x2 , x3 ] Fig. 3.4

S[x1 , x2 , x3 , x4 ]

Simplexes in

R3 .

Let S[x1 , . . . , xk+1 ] be the k-dimensional simplex generated by set of affinely independent points {x1 , . . . , xk , xk+1 } in Rm (where m  k) and let x ∈ S, which is a convex combination of x1 , . . . , xk , xk+1 . In other words, there exist a1 (x), . . . , ak (x), ak+1 (x) such that a1 (x), . . . , ak (x), ak+1 (x) ∈  [0, 1], k+1 i=1 ai (x) = 1, and x = a1 (x)x1 + · · · + ak (x)xk + ak+1 (x)xk+1 . The numbers a1 (x), . . . , ak (x), ak+1 (x) are the barycentric coordinates of x relative to the simplex S. Theorem 3.34. The barycentric coordinates of x ∈ S[x1 , . . . , xk , xk+1 ] are uniquely determined. Proof. If we have x = a1 (x)x1 + · · · + ak (x)xk + ak+1 (x)xk+1 = b1 x1 + · · · + bk xk + bk+1 xk+1 , and ai (x) = bi for some i, this implies (a1 (x) − b1 )x1 + · · · + (ak (x) − bk )xk + (ak+1 (x) − bk+1 )xk+1 = 0n , which contradicts the affine independence of x1 , . . . , xk+1 . 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Algebra of Convex Sets

page 145

145

Definition 3.19. The barycenter of the simplex S[x1 , . . . , xk , xk+1 ] ⊆ Rn is the point z=

1 (x1 + · · · + xk + xk+1 ). k+1

Definition 3.20. If x ∈ S[x1 , . . . , xk+1 ], the carrier of x is the set L(x) = {j | 1  j  k + 1 and aj (x) > 0}. The standard simplex in Rn is the simplex S[e1 , . . . , en ]. Definition 3.21. Let X = {x1 , . . . , xk , xk+1 }. A subsimplex of the simplex S[X] is a simplex of the form S[Y ], where Y ⊆ X. If S  is a subsimplex of a simplex S we write S  ≺ S. Theorem 3.35. Let X = {x1 , . . . , xk , xk+1 } be a set of affinely independent points in Rn . Each subsimplex of the form S[Y ], where Y is a non-empty subset of {x1 , . . . , xk , xk+1 } is a face of S[X]. Proof. For x ∈ S[X] let a1 (x), . . . , ak (x), ak+1 (x) be the barycentric coordinates of x. Let Y = {xj | j ∈ J}, where J ⊆ {1, . . . , k, k + 1}. Suppose that for u, v ∈ S[X] we have (u, v) ∩ S[Y ] = ∅. If u = k+1 k+1 i=1 ai (u)xi and v = i=1 ai (v)xi , there exists c ∈ (0, 1) such that cu + (1 − c)v =

k+1 

(cai (u) + (1 − c)ai (v))xi ∈ S[Y ],

i=1

which means that cai (u) + (1 − c)ai (v) = 0 for i ∈ J. Since c ∈ (0, 1) and ai (u), ai (v)  0, it follows that we have ai (u) = ai (v) = 0 for i ∈ J, so both u and v belong to S[Y ]. Thus, S[Y ] is a face of S[X].  If X = {x1 , . . . , xk , xk+1 } the face opposite to a vertex xi is the subsimplex S[X − {xi }]. This simplex is denoted oppi (S[X]). A finite set of points P in R2 is a convex polygon if no member p of P lies in the convex closure of P − {p}. Theorem 3.36. A finite set of points P in R2 is a convex polygon if and only if no member p of P lies in a two-dimensional simplex formed by three other members of P . Proof. The argument is straightforward and is left to the reader as an exercise. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 146

Mathematical Analysis for Machine Learning and Data Mining

146

Definition 3.22. Let S be a k-simplex. A triangulation of S is a collection T of distinct k-simplexes whose union is S such that any two simplexes of the collection are either disjoint or their intersection is a common face. The mesh of the triangulation T is the largest diameter of a simplex of T. Given a k-simplex S any face spanned by k of the k + 1 vertices of S is called a facet of S. Example 3.20. Figure 3.5(a) shows a triangulation of S[x1 , x2 , x3 ] that consists of the simplexes S[x1 , x3 , x4 ], S[x2 , x3 , x4 ], S[x1 , x2 , x4 ]. The simplexes shown in Figure 3.5(b) do not constitute a triangulation of S[x1 , x2 , x3 ] because the intersection of the simplexes S[x1 , x4 , x5 ] and S[x2 , x3 , x5 ] is S[x4 , x5 ], which is a face of S[x1 , x4 , x5 ] but not of S[x2 , x3 , x5 ]. x3

x3

x4

x4

x1

x1

x3 (a)

x5

x3

(b)

Fig. 3.5 (a) A triangulation of the simplex S[x1 , x2 , x3 ]; (b) a non-triangulation of S[x1 , x2 , x3 ].

Definition 3.23. Let L be a real linear space and let T, S be two subsets of L. The core of T relative to S is the set coreS (T ) = {t ∈ T | for each s ∈ S there is x ∈ (t, s) such that [t, x] ⊆ T }. The set coreL (T ) is referred to as the core of T and is denoted by core(T ). If S1 ⊆ S2 then coreS2 (T ) ⊆ coreS1 (T ) for every S1 , S2 ⊆ L.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Algebra of Convex Sets

page 147

147

Theorem 3.37. If C is a convex subset of a real linear space L, then core(C) is a convex set. Proof. Let u, v ∈ core(C). Then, for each z ∈ L there is x ∈ (u, z) and there is y ∈ (v, z) such that [u, x] ⊆ C and [v, y] ⊆ C. Let c ∈ (0, 1) and let w = cu + (1 − c)v. There exist a, b ∈ (0, 1) such that x = au + (1 − a)z, y = bv + (1 − b)z, [u, x] ⊆ C and [c, y] ⊆ C. We have     1−a 1−b 1 1 x− z + (1 − c) y− z w=c a a b b c c(1 − a) 1−c (1 − c)(1 − b) = x− z+ y− z a a b b   c(1 − a) (1 − c)(1 − b) c 1−c = x+ y− + z. a b a b This implies w+



c(1 − a) (1 − c)(1 − b) + a b

 z=

1−c c x+ y. a b

Note that the sums of coefficients in both sides of the equality is the same number S, so by dividing both members by S we obtain       c(1 − a) (1 − c)(1 − b) 1−c 1 1 c + x+ y . t= w+ z = S a b S a b Thus, t is a convex combination of w and z and, also, a convex combination of x and y. Consequently, t ∈ C and [w, t] ⊆ C, which proves that w ∈ core(C). Therefore, C is a convex set.  Definition 3.24. Let L be a real linear space and let T be a subset of L. The intrinsic core of T is the set coreKaff (T ) (T ), that is, the core of T relative to the affine space generated by T . The intrinsic core of T is denoted by icr(T ). Theorem 3.38. Let L be a real linear space and let T be a convex subset of L. We have t ∈ icr(T ) if and only if for each x ∈ T − {t} there exists y ∈ T such that t ∈ (x, y). Proof. Let t be such that for each x ∈ T − {t} there is an y ∈ T such that t ∈ (x, y). Let z ∈ Kaff (T ) and consider the line t,z and a point x ∈ t,z ∩ T such that x = t. We have x = (1 − a)t + az for some a = 0.

May 2, 2018 11:28

148

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 148

Mathematical Analysis for Machine Learning and Data Mining

By hypothesis, there exists y ∈ T such that t = (1 − c)x + cy, where c ∈ (0, 1). Thus, 1−c 1 t− x c c 1 1−c = t− ((1 − a)t + az) c c   ac − a a − ac = 1− z. t− c c

y=

If a > 0 we have x ∈ (t, z) and [t, x] ⊆ T . Otherwise, that is, if a < 0, we > 0 because c ∈ (0, 1). In this case, y ∈ (t, z) and [t, y] ⊆ T . have ac−a c Thus, in either case, t ∈ icr(T ). Suppose now that t ∈ icr(T ) and let u ∈ T . We have

t,u = at + (1 − a)u | a ∈ R = {u + a(t − u) | a ∈ R} ⊆ Kaff (T ). Let z = u + b(t − u) ∈ t,u with b > 1. Since t ∈ icr(T ), there exists y ∈ [t, z) such that we have [t, y] ⊆ T , so y = u + c(t − u), where 1 < c < b. Since t ∈ t,u (for a = 1), u ∈ t,u (for a = 0) and y ∈ t,u , it follows that t ∈ (u, y).  Example 3.21. Let S = S[x1 , . . . , xk , xk+1 ] be a k-simplex. Its intrinsic interior icr(S) consists of all points of S that have positive barycentric coordinates. In Definition3.20 we introduced the carrier L(x) of a point x of a ksimplex S[x1 , . . . , xk+1 ] in Rn . Definition 3.25. Let S[x1 , . . . , xk+1 ] be a simplex in Rn and let T be a triangulation of this simplex having the set of vertices VT . A labeling of T is a mapping : VT −→ {1, . . . , k + 1} such that (x) ∈ L(x) for every x ∈ VT . A simplex Z of T is complete if (Z) = {1, . . . , k, k + 1}. Example 3.22. An 1-simplex S[x1 , x2 ] in Rn is simply a closed segment in this space. A triangulation of this simplex is a sequence of closed intervals [zj , zj+1 ] for 1  j  p, where z1 = x1 , zj+1 = x2 and a labeling of this triangulation is a function f : {z1 , . . . , zp+1 } −→ {1, 2}, since L(zj ){1, 2}. Since the label of z1 is 1 and the label of zk+1 is 2, when we scan the labels of the vertices of the triangulation from left to right, the label must change an odd number of times. Therefore, the triangulation T contains an odd number of complete simplexes, which ensures that there exists at least

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Algebra of Convex Sets

page 149

149

one complete simplex in the triangulation T. As we shall see, a general statement that extends this result to k-simplexes is known as Sperner’s Lemma. Example 3.23. Let S[x1 , x2 , x3 ] be a 2-simplex, which is a triangle In Figure 3.6 we show a triangulation of this simplex. It is apparent that this triangulation contains seven complete simplexes. x2 2 1 1

3

2 1 x1

3

1

3 Fig. 3.6

2

2

2 3

1

3 x3

Triangulation of a 2-simplex S[x1 , x2 , x3 ].

Theorem 3.39. (Sperner’s Lemma) Let T be a triangulation of a ksimplex S. If the vertices of S have a labeling, then there exists an odd number of complete simplexes in T. Proof. The proof is by induction on k. The argument for k = 1 was given in Example 3.22. Suppose that the statement holds for simplexes of dimension less than k. Let C be the set of complete cells of T, c = |C|, and let Q be the set of cells labeled by all numbers in {1, . . . , k} such that exactly one of this color is used twice and the remainder of the colors are used once, and let q = |Q|. Let X be the set of (k − 1)-dimensional faces on the boundary of S that are labeled by {1, . . . , k}, and let Y the set of such faces inside S. Denote |X| = x and |Y | = y. Each cell in C contributes exactly one (k − 1)-dimensional face labeled by {1, . . . , k}. Each cell in Q contributes two faces labeled by {1, . . . , k}.

May 2, 2018 11:28

150

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 150

Mathematical Analysis for Machine Learning and Data Mining

Inside faces appear in two cells while boundary cells appear in one cell, so c + 2q = x + 2y.

(3.4)

On the boundary of S, the only (k − 1)-dimensional face labeled {1, . . . , k} can be on the face F whose vertices are labeled {1, . . . , k}. By the inductive hypothesis, F contains an odd number of complete (k−1)-dimensional cells, so x is odd. Since c = 2(y − q) + x, it follows that c is an odd number.  Example 3.24. The argument used in the proof of Sperner’s Lemma can be illustrated on the triangulation of the two-dimensional simplex shown in Example 3.23. The number of complete cells is c = 7, while the set of cells labeled by 1 and 2 is q = 3. On the other hand we have x = 3 and y = 5, which show that equality (3.4) is satisfied.

Exercises and Supplements (1) Prove that if C is a convex subset of a real linear space L, then for every u ∈ L the set tu (C) is convex. (2) Prove that a subset C of a real linear space L is convex if and only if aC + bC = (a + b)C for all a, b ∈ R>0 . (3) Prove that D is an affine subspace in Rn if and only if u + D = {u + x | x ∈ D} is an affine subspace for every u ∈ Rn . (4) Let S a subset of Rn such that 0n ∈ Kaff (S). Prove that dim(Kaff (S)) = dim(S) − 1. (5) Let S be a convex set in Rn such that |S|  n and  let x ∈ S. If r ∈ N such that n + 1  r  |S| prove that there exist |S|−n set of points Y , r−n Y ⊆ S, such that x ∈ Kconv (Y ). Hint: Use induction on k = |S| − n and Carath´eodory’s Theorem. (6) Let x0 , u, v, x ∈ Rn such that x = (1 − a)u + av for some a ∈ (0, 1) and d(x0 , u) = d(x0 , v) = d(x0 , x). Prove that u = v = x. Solution: Since d(x0 , u) = d(x0 , x) = r, by the cosine theorem we have x − u2 = x0 − u2 + x0 − x2 − 2x0 − ux0 − x cos α, where α = ∠(x0 − u, x0 − x). This implies x − u = 2r sin α2 . Similarly, if β = ∠(x0 − x, x0 − v), we have x − v = 2r sin β2 . Note that ∠(x0 − u, x0 − v) = α + β, so u − v = 2r sin α+β . 2

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Algebra of Convex Sets

page 151

151

Since x = (1 − a)u + av we have x − v = (1 − a)(u − v), and u − x = a(u − v), so x − v = (1 − a)u − v, and u − x = au − v. Therefore, we have sin

α+β α α+β β = (1 − a) sin . sin = a sin , 2 2 2 2

which yields α α+β α + sin = sin . 2 2 2 This, in turn implies α = 0 or β = 0. In the first case, x = v, which implies x = u = v because x = (1 − a)u + av. In the second case, x = u, which has the same consequence. sin

(7) Prove that for every set S ⊆ Rn we have S + {0} = S and S + ∅ = ∅. (8) If P ⊆ R2 is a set of five points such that no three of them are collinear, prove that P contains four points that form a convex quadrilateral. This result is known as Klein’s Theorem. Solution: Let P = {xi | 1  i  5}. If these five points form a convex polygon, then any four of them form a convex quadrilateral. If exactly one point is in the interior of a convex quadrilateral formed by the remaining four points, then the desired conclusion is reached. Suppose that none of the previous cases occur. Then, two of the points, say xp , xq , are located inside the triangle formed by the remaining points xi , xj , xk . Note that the line xp xq intersects two sides of the triangle xi xj xk , say xi xj and xi xk (see Figure 3.7). Then xp xq xk xj is a convex quadrilateral. (9) Let S be a subset of Rn . The polar of S is the set S = {y ∈ Rn | y x  1 for every x ∈ S}. Prove that for every subset S of Rn , S is a convex subset of R∗ and 0n ∈ S . (10) Let Y = {y1 , . . . , yk , yk+1 } be an affinely independent subset of Rn . Prove that the set W = {w1 , . . . , wk , wk+1 } defined as w1 = y1 , w2 = y1 + (y2 − y1 ), . . . , wk+1 = yk + (yk+1 − yk ),

May 2, 2018 11:28

152

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 152

Mathematical Analysis for Machine Learning and Data Mining

xi u @ @ @ @  @   @ u @   @  xq  @ u  @  @u  x p  xk u xj Fig. 3.7

A five-point configuration in R2 .

is affinely independent and generates the same affine subspace as the set Y . Solution: We claim that the set {w1 , . . . , wk , wk+1 } is affinely independent. Indeed suppose that one of its members, say wk+1 is an affine combination of the remaining vectors, that is, wk+1 = a1 w1 + · · · + ak wk , where

k i=1

ai = 1. Since wp = yp + (1 − )yp−1

for 2  p  k + 1, we have yk+1 + (1 − )yk = a1 y1 + a2 (y2 + (1 − )y1 ) + · · · + ak (yk + (1 − )yk−1 ) = (a1 + a2 (1 − ))y1 + (a2  + a3 − a3 )y2 + · · · + (ak−1  + ak − ak )yk−1 + ak yk . This implies yk+1 =

a1 + a2 (1 − ) a2  + a3 (1 − ) y1 + y2   ak−1  + ak (1 − ) ak  − 1 +  yk−1 + + yk , +··· +  

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Algebra of Convex Sets

page 153

153

which means that yk+1 is an affine combination of y1 , . . . , yk , which is impossible because the set Y is affinely independent. Note that each vector wp is an affine combination of yp−1 and yp for 2  p  k + 1. Thus, the affine subspace generated by {w1 , . . . , wk , wk+1 } is included in the affine subspace generated by {y1 , . . . , yk , yk+1 }. zp−1 for 2  p  k + 1 it follows Conversely, since yp = 1 zp + −1  that the affine subspace generated by {y1 , . . . , yk , yk+1 } is included in the affine subspace generated by {w1 , . . . , wk , wk+1 }. (11) Let Y = {y1 , . . . , yk , yk+1 } be an affinely independent subset of Rn . Prove that the set Z = {z1 , . . . , zk , zk+1 } defined as z1 = y1 , z2 = y1 + (y2 − y1 ), . . . , zk+1 = y1 + (yk+1 − y1 ), is affinely independent and generates the same affine subspace as the set Y = {y1 , . . . , yk , yk+1 }. Furthermore, prove that for every r > 0, there exists  > 0 such that Z ⊆ B(y1 , r). Hint: It suffices to choose 
0 consider the set Z as defined in Exercise 11. It is clear that Z is affinely independent and Z ⊆ Kaff (C). Therefore, if  is sufficiently small the simplex S[Z ] is included in B(y1 , r) and so is its barycenter. (13) Prove that the union of two convex sets is not necessarily convex. (14) Let C, D be two convex subsets of Rn . Prove that the set C + D defined by C + D = {x + y | x ∈ C, y ∈ D} is a convex subset of Rn . (15) Let h : Rn −→ Rm be a linear operator. Prove that (a) if C is a convex subset of Rn , then h(C) is a convex subset of Rm ; (b) if D is a convex subset of Rm , then h−1 (D) is a convex subset of Rn . (16) Prove that if C is a convex subset of Rn and L is a subspace of Rn , then the projection of C on L is a convex set. (17) Prove that if C is aconvex subset  of Rn and D is a convex subset of Rm ,

x

n+m then C ⊕ D = . y x ∈ C, y ∈ D is a convex subset of R

May 2, 2018 11:28

154

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 154

Mathematical Analysis for Machine Learning and Data Mining

(18) Prove that if C is a convex subset of Rn and a, b ∈ R0 , then (a + b)C = aC + bC. (19) Let S, T be two subsets of Rn . Prove that Kconv (S + T ) = Kconv (S) + Kconv (T ). (20) Prove that if S is a convex set, then aS +bS is a convex set; furthermore, prove that aS + bS = (a + b)S for every a, b  0. (21) Let A, B be two subsets of Rn such that there exists a hyperplane Hw,a > < such that A ⊆ Hw,a and B ⊆ Hw,a . Prove that: (a) Kconv (A) ∩ Kconv (B) = ∅; (b) for A − B = {x − y | x ∈ A, y ∈ B}  > and 0n ∈ Hw,0 . we have A − B ⊆ Hw,0 > for each x ∈ A, we have w x > a; similarly, Solution: Since A ⊆ Hw,a  for every y ∈ B we have w y < a. n n If u ∈ Kconv (A) we can write u = i=1 ai xi , where i=1 ai = 1, 0  ai  1 and xi ∈ A for 1  i  n. Therefore,

 

wu=w



n  i=1

 ai xi

=

n  i=1

a i w x i > a

n 

ai = a.

i=1

Similarly, if v ∈ B we obtain w v < a. Therefore, if t would belong to Kconv (A) ∩ Kconv (B) we would have the contradictory inequalities w t > a and w t < a. For the second part, let x ∈ A and y ∈ B, so w x > a and w y < a > ; (that is, −w y > −a). This implies w (x − y) > 0, so A − B ∈ Hw,0 0 clearly, 0n ∈ Hw,0 . (22) Prove that the intersection of two convex polygons in R2 having a total of n edges is either empty or is a convex polygon with at most n edges. (23) Let C = {C1 , . . . , Cm } be a collection of m convex sets in Rn , where of C that contains n + 1 m ≥ n + 1 such that if every subcollection C  sets has a non-empty intersection. Prove that C = ∅. Hint: Proceed by induction on k = m − (n + 1). Apply Radon’s theorem in the inductive step of the proof. (24) Prove that the border of a polytope is the union of its proper faces. (25) Let S[x1 , . . . , xk , xk+1 ] be a k-simplex in Rn . Prove that there exists a triangulation of this simplex whose mesh is less that a positive number . (26) Let C1 , . . . , Ck be k convex subsets of Rn , where k  n + 2. Prove that if any n + 1 of these sets have a common point, then all the sets have a common point. This fact is known as Helly’s Theorem.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Algebra of Convex Sets

page 155

155

Solution: For i ∈ {1, . . . , k} there exists xi ∈ C1 ∩ · · · ∩ Ci−1 ∩ Ci+1 ∩ · · · ∩ Ck . This results in a set {x1 , . . . , xk } of more than n + 2 vectors that are affinely dependent. By Radon’s Theorem we obtain that after a suitable renumbering we have x ∈ Kconv ({x1 , . . . , xj }) ∩ Kconv ({xj+1 , . . . , xk }) for some j, where 1  j  k − 1. Since each of the points x1 , . . . , xj belong to Cj+1 ∩ · · · ∩ Ck we have x ∈ Kconv ({x1 , . . . , xj }) ⊆ Cj+1 ∩ · · · ∩ Ck . Similarly, x ∈ Kconv ({xj+1 , . . . , xk }) ⊆ C1 ∩ · · · ∩ Cj . (27) Let C be a finite collection of convex subsets in Rn and let C be a convex subset of Rn . Prove that if any n + 1 subsets of C are intersected by some translate of C, then all sets of C are intersected by some translate of C. (28) The following result is known as the Motzkin’s Theorem. Let ℘ : R −→ R n

n−1

be the projection defined by ℘(x) =

⎛

⎞

x2 ⎝ .. ⎠, . xn

where

x ∈ Rn . Prove that if P is a polyhedron in Rn , then ℘(P ) is a polyhedron in Rn−1 . Solution: Suppose that P = {x ∈ Rn | W  x  b}, where W = (w1 w2 · · · wm ) ∈ Rn×m and b ∈ Rm . In other words, P is defined by the inequalities w1 x  b1 , . . . , wm x  bm . If w1j > 0 the inequality w1j x1 + · · · + wnj xn  bj w

implies x1  w ˜2j x2 + · · · + w ˜nj xn + ˜bj , where w ˜kj = − wkj for 2  k  n 1j b j and ˜bj = w1j . Similarly, if w1j < 0, the above inequality is equivalent ˜2j x2 + · · · + w ˜nj xn + ˜bj . Therefore, the inequalities wj x  b to x1  w where w1j = 0 are equivalent to max{w ˜2j x2 + · · · + w ˜nj xn + ˜bj | w1j < 0}  x1  min{w ˜2j x2 + · · · + w ˜nj xn + ˜bj | w1j > 0}. ⎛ Note that

⎞

x2 ⎝ .. ⎠ . xn

∈ ℘(P ) only if wij = 0 implies w2j x2 +· · ·+wnj xn < bj

May 2, 2018 11:28

156

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 156

Mathematical Analysis for Machine Learning and Data Mining

and if for w1j > 0 and w1k < 0 we have ˜nj xn + ˜bj  w ˜2k x2 + · · · + w ˜nk xn + ˜bk , w ˜2j x2 + · · · + w or, equivalently, ˜2k )x2 + · · · + (w ˜nj − w ˜nk )xn  ˜bk − ˜bj . (w ˜2j − w Thus, the set ℘(P ) is defined by a collection of inequalities involving x2 , . . . , xn , so ℘(P ) is indeed a polyhedron. (29) Prove that a cone in a linear space L is pointed if and only if x ∈ C and −x ∈ C imply x = 0L . (30) Prove that the set of cones in Rn is closed under intersection, union, and complement. (31) Prove that if C is a cone in Rn , then −C is a cone in the same space; prove that if C, D are cones in Rn , then C + D is a cone in Rn . (32) Prove that the set C ⊆ R3 defined by C = {(x, y, z) | z  0, x2 + y 2  z 2 } is a cone. (33) Let C be a convex set in Rn . Define the set R(C) = {d ∈ Rn | x + d ∈ C for every x ∈ C}. Prove that R(C) is a cone (it is known as the recession cone for C). Furthermore, prove that if C is a convex cone, then R(C) = C. (34) Let L be a linear space and let C be a cone in L such that 0L ∈ C. Define the relation ρC ⊆ L × L as ρC = {(x, y) ∈ L × L | y − x ∈ C}. Prove that: (a) ρC is reflexive and (x, y) ∈ ρC imply (ax, ay) ∈ ρC for a > 0 and (x + z, y + z) ∈ ρC ; (b) ρC is transitive if and only if C is convex; (c) ρC is antisymmetric if and only if C ∩ (−C) = {0L }. (35) Let ρ be a reflexive relation on a linear space L such that (x, y) ∈ ρ imply (ax, ay) ∈ ρ for a > 0 and (x + z, y + z) ∈ ρ. Prove that the set C = {x ∈ L | (x, 0L ) ∈ ρ} is a cone and ρ = ρC . (36) Let A ∈ Rm×n . Prove that the set CA = {x ∈ Rn | Ax  0m } is a convex cone. CA is the polyhedral cone determined by A.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Algebra of Convex Sets

page 157

157

(37) Let A ∈ Rm×n and b ∈ Rm . Prove that the non-empty polyhedron PA,b = {x ∈ Rn | Ax  b} is a polyhedral cone if and only if b = 0m . Further, prove that PA,0m is a pointed cone if rank(A) = n. Solution: It is immediate that if b = 0m , PA,0 is a cone. Conversely, suppose that PA,b is a cone and let x0 ∈ PA,b . Since 0n ∈ PA,b we have 0m  b, hence {x | Ax  0m } ⊆ PA,b . If x ∈ PA,b and Ax  0m , then there is a row aj of A such that aj x > 0 and aj (tx)  bj for every t  0 because PA,b is a cone. This leads to a contradiction. Therefore, b = 0m . If rank(A) < n there exists a non-zero vector u ∈ Rn such that Au = 0m , so the line {tu | t  0} is included in PA,0m , which implies that PA,0 is not a pointed cone. Conversely, suppose that PA,0m is not a pointed cone, that is, it contains the line {x | x = x0 + tu} with u = 0n . If aj u = 0 for some j, there exists t such that aj x0 + taj u > 0, which contradicts the inclusion  ⊆ PA,0m . Thus, Au = 0, hence rank(A) < n. (38) A Weyl pair of matrices is a pair (A, B), where A ∈ Rm×n and B ∈ Rn×p such that: {x ∈ Rn | Ax  0m } = {By | y ∈ Rp and y  0p }. Prove that for every matrix B ∈ Rn×p there exists a matrix A ∈ Rm×n such that (A, B) is a Weyl pair. Solution: Let P be the polyhedral cone P ⊆ Rp+n defined as

 

x p+n

P = ∈R

y = Bx, x  0 y



 

x p+n

∈R =

y − Bx  0, Bx − y  0, −x  0 . y

If ℘ : Rp+n −→ Rn is the function defined as ℘

  x y

= y, then ℘(P ) = C.

Since ℘(tx) = t℘(x) for t  0 and x ∈ P , C is a cone and the existence of the matrix A follows from Supplements 28 and 37. (39) Prove that a subset B of Rn is a convex cone if and only if it is closed under addition and non-negative scalar multiplication. (40) Prove that the smallest subspace of a linear space L that contains a convex cone C is C − C and the largest subspace contained in C is (−C) ∩ C. (41) Prove that if T ⊆ L then the sets T˜ = T ∪ h−1 (T ) and Tˆ = T ∩ h−1 (T ) are symmetric.

May 2, 2018 11:28

158

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 158

Mathematical Analysis for Machine Learning and Data Mining

(42) Let C be a convex subset of a real linear space L. Prove that C is a balanced set if and only if x ∈ C implies −x ∈ C. Solution: The condition is clearly necessary. To prove that it is sufficient let C be a convex set. If C is empty then C is balanced. Therefore, let C be a non-empty set and let x ∈ C. We  1. The have −x ∈ C. If r ∈ R is such that |r|  1, then 0  t = 1+r 2 convexity of C implies: rx = tx + (1 − t)(−x) ∈ C, which shows that C is balanced.

Bibliographical Comments The books [111, 23, 15, 18, 99] contain a vast amount of results in convexity theory. References that focus on geometric aspects are [69, 115].

PART II

Topology

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 161

Chapter 4

Topology

4.1

Introduction

Topology is an area of mathematics that investigates the local and the global structure of space. We present in this chapter an introduction to point-set topology. After introducing the notion of topology and topological space, we present several of the most important topological space. Then, we study closure and interior operators as they are defined by topologies, as well as some alternative techniques for defining topologies such as systems of neighborhoods and bases. The notion of compact set in a topological space is the object of a dedicated section in view of its importance as a generalization of closed and bounded sets in Euclidean spaces. We discuss the separation hierarchy of topological spaces, a device that allows topologists to impose restrictions on these spaces in order to obtain certain desired results. Locally compact spaces are topological spaces that satisfy a certain separation requirement such that every point has a local basis consisting of compact sets. A series of sections that follow concentrate on the all important notion of limit and on function continuity. We continue with connected topological space, that is, with spaces that cannot be represented as the union of two or more disjoint topological spaces. After discussing products of topological spaces we end the chapter with a study of semicontinuous functions and with epigraphs and hypographs of functions, two subjects of importance in optimization theory.

161

May 2, 2018 11:28

162

4.2

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 162

Mathematical Analysis for Machine Learning and Data Mining

Topologies

The term “topology” is used both to designate a mathematical discipline and to name the fundamental notion of this discipline, which is introduced next. Definition 4.1. A topology on a set S is a family O of subsets of S that satisfies the following conditions:  (i) for every collection C such that C ⊆ O, C ∈ O;  (ii) if D is a finite collection and D ⊆ O, then D ∈ O. The sets that belong to O are referred to as the open sets of the topology O. The pair (S, O) is referred to as a topological space. The elements of S are commonly referred to as points. The intersection of the empty collection of subsets of S belongs to O, and this intersection is S. On the other hand, the union of the empty collection (which is the empty set) belongs to O, which allows us to conclude that the empty set and the set S belong to every topology defined on the set S. Example 4.1. The pair (S, P(S)) is a topological space. The topology P(S) is known as the discrete topology. The collection {∅, S} is the indiscrete topology. Example 4.2. The pair (∅, {∅}) is a topological space as the reader can easily verify. We refer to (∅, {∅}) as the empty topological space. Example 4.3. Let O be the collection of subsets of R defined by L ∈ O if for every x ∈ L there exists ∈ R>0 such that |u − x| < implies u ∈ L. The pair (R, O) is a topological space. In other words, for every x ∈ L, there exists such that B(x, ) ⊆ L. Indeed, it is immediate that ∅ and R belong to O.  Let C be such that C ⊆ O and let x ∈ C. There exists L ∈ C such that x ∈ L and, therefore, by the definition of O, there is > 0 such that   |u − x| < implies u ∈ L. Thus, u ∈ C, so C ∈ O. Suppose now that D is a finite subcollection of O, D = {D1 , . . . , Dn },  and let x ∈ D. Since x ∈ Di for 1  i  n, there exists 1 , . . . , n such that |u − x| < i implies u ∈ Di for every i, 1  i  n. Therefore, by defining = min{ i | 1  i  n}, it follows that |x − u|  implies   u ∈ D, which proves that D ∈ O. We conclude that O is a topology on R. This topology is called the usual topology on R. Unless stated otherwise, we assume that the set of real numbers is equipped with the usual topology.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Topology

b3234-main

page 163

163

In the topological space (R, O), every open interval (a, b) with a < b is an open set. Indeed, if x ∈ (a, b) and |x − u| < , where = 12 min{|x − a|, |x − b|}, then u ∈ (a, b). A similar argument shows that the half-lines (b, +∞) and (−∞, a) are open sets for a, b ∈ R. Thus, every set (a, b) with −∞  a < b  ∞ is open. Example 4.4. Let S be an infinite set. The family of sets O = {∅} ∪ {L ∈ P(S) | S − L is finite} is a topology on S. We refer to O as the cofinite topology on S. Note that both ∅ and S belong to O. Further, if C is a subcollection of   O, then S − C = {(S − L) | L ∈ C}, which is a finite set because it is a subset of every finite set S − L, where L ∈ C. Also, if U, V ∈ O, then S − (U ∩ V ) = (S − U ) ∪ (S − V ), which shows that S − (U ∩ V ) is a finite set. Thus, U ∩ V ∈ O. Example 4.5. Let (S, ) be a partially ordered set. A subset T of S is upward closed if x ∈ T and x  y implies y ∈ T . The collection of upwards closed sets O↑ is a topology on S. It is clear that both ∅ and S belong to O↑ . Further, if {Li | i ∈ I} is a  family of upwards closed sets, then {Li | i ∈ I} is also an upwards closed  set. Indeed, suppose that x ∈ {Li | i ∈ I} and x  y. There exists Li  such that x ∈ Li and therefore y ∈ Li , which implies y ∈ {Li | i ∈ I}. Moreover, it is easy to see that any intersection of sets from O↑ belongs to O↑ , not just a finite intersection (which would suffice for O↑ to be a topology). This topology is known as the Alexandrov topology on the poset (S, ). Open sets of the topological space (R, O), where O is the usual topology on the set of real numbers have the following useful characterization. Theorem 4.1. A subset U of R is open in the topological space (R, O) if and only if it equals the union of a countable collection of disjoint open intervals. Proof. Since every open interval (finite or not) is an open set, it follows that the union of a countable collection of disjoint open intervals is open. To prove the converse, let U be an open set. Note that U can be written as a union of open intervals since for each x ∈ U there exists > 0 such that x ∈ (x − , x + ) ⊆ U . Define the relation θU on the set U by xθU y if there exist a, b ∈ R such that {x, y} ⊆ (a, b) ⊆ U . We claim that θU is an equivalence relation on U .

May 2, 2018 11:28

164

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 164

Mathematical Analysis for Machine Learning and Data Mining

Since U is open, x ∈ U implies the existence of a positive number such that {x} ⊆ (x − , x + ) ⊆ U for every x ∈ U , so θU is reflexive. The symmetry of θU is immediate. To prove its transitivity, let x, y, z ∈ U be such that xθU z and zθU y. There are a, b, c, d ∈ R such that {x, z} ⊆ (a, b) ⊆ U and {z, y} ⊆ (c, d) ⊆ U . Since z ∈ (a, b) ∩ (c, d), it follows that (a, b)∪(c, d) is an interval (e, e ) such that {x, y} ⊆ (e, e ) ⊆ U , which shows that xθU y. Thus, θU is an equivalence on U . We claim that each equivalence class [x]θU is an open interval, or a set of the form (a, +∞), or a set of the form (−∞, b). Indeed, suppose that u, v ∈ [x]θU (that is, uθU x and vθU x) and that t ∈ (u, v). We now prove that tθU x. There are two open intervals (a, b) and (c, d) such that {u, x} ⊆ (a, b) ⊆ U and {x, v} ⊆ (c, d) ⊆ U . Again, (a, b) ∪ (c, d) is an open interval (e, e ) and we have (u, v) ⊆ (e, e ) ⊆ U . Thus, if [x]θU contains two numbers u and v, it also contains the interval (u, v) determined by these numbers. To prove that [x]θU has the desired form, we shall prove that this set has no least element and no greatest element. Suppose that [x]θU has a least element y. Then, there exist a and b such that a < y < x < b and (a, b) ⊆ U . Since y is supposed to be the least element of [x]θU , if a < z < y, we have z ∈ [x]θU . This contradicts yθU z and yθU x. In a similar manner, it is possible to show that [x]θU has no largest element. Finally, we prove that the partition that corresponds to θU is countable. Select a rational number rx ∈ [x]θU ∩ Q. Since the equivalence classes [x]θU are pairwise disjoint, it follows that [x]θU = [y]θU implies rx = ry . Thus, we have an injection r : U/θU −→ Q given by r([x]θU ) = rx for x ∈ U .  Therefore, the set U/θU is countable. Example 4.6. The lower topology Ol of R consists of the sets ∅, R and all sets of the form (a, ∞) for a ∈ R. Note that (a, ∞) ∩ (b, ∞) =  (min{a, b}, ∞)} and for any set {ai | i ∈ I}, i∈I (ai , ∞) = (inf{ai | i ∈ I}, ∞), which shows that Ol is indeed a topology. Similarly, the upper topology Ou of R consists of the sets ∅, R and all sets of the form (−∞, a) for a ∈ R. The upper and lower topologies can be defined on the extended set of ˆ and sets of the ˆ consists of ∅, R, reals as follows. The lower topology Ol on rr form Ua = (a, ∞], where a ∈ R or a = −∞. Similarly, the upper topology ˆ consists of ∅, R, ˆ and sets of the form Wa = [−∞, a), where Ou on the set R a ∈ R or a = ∞.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topology

page 165

165

Definition 4.2. Let (S, O) be a topological space. A subset T of S is closed if its complement S − T is open. The collection of closed sets of (S, O) is denoted by closed(O). Theorem 4.2. The following statements hold for any topological space (S, O): (i) ∅ and S are closed sets;  (ii) for every collection C of closed sets, C is a closed set;  (iii) for every finite collection D of closed sets, D is a closed set. Proof.

This is an immediate consequence of Definition 4.1.



Example 4.7. Observe that (−∞, a) ∪ (b, +∞) is an open set in R which implies that its complement, the interval [a, b], is closed. Also, (−∞, b] and [a, ∞) are closed sets (as complements of the open sets (b, ∞) and (a, ∞), respectively). Every subset X of R of the form X = {x0 , x1 , . . . , xn , . . .} is closed. Indeed, since  (xi , xi+1 ) R − X = (−∞, x0 ) ∪ i∈N is open (as a countable union of open sets), it follows that X is closed. Definition 4.3. A topology O is finer (or stronger) than a topology O or, equivalently, O is a coarser (or weaker) than O, if O ⊆ O. Every topology on a set S is finer than the indiscrete topology on S; the discrete topology P(S) (which has the largest collection of open sets) is finer than any topology on S. Theorem 4.3. Let (S, O) be a topological space and let T be a subset of S. The collection O T defined by O T = {L ∩ T | L ∈ O} is a topology on the set T . Proof. We leave the proof of this theorem to the reader as an exercise.  Definition 4.4. If U is a subset of S, where (S, O) is a topological space, then we refer to the topological space (U, O U ) as a subspace of the topological space (S, O). To simplify notation, we refer to the subspace (U, O U ) as (U, O), or even as U .

May 2, 2018 11:28

166

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 166

Mathematical Analysis for Machine Learning and Data Mining

Theorem 4.4. Let (S, O) be a topological space and let (T, O T ) be a subspace of this space. Then, a set H is closed in (T, O T ) if and only if there exists a closed set H0 in (S, O) such that H = T ∩ H0 . Proof. Suppose that H is closed in (T, O T ). Then, the set T − H is open in this space and therefore there exists an open set L0 in (S, O) such that T −H = T ∩L0 . This is equivalent to H = T −(T ∩L0 ) = T ∩(S −L0 ). We define H0 as the closed set S − L0 . Conversely, suppose that H = T ∩ H0 , where H0 is a closed set in S. Since T − H = T ∩ (S − H0 ) and S − H0 is an open set in (S, O), it follows that T − H is open in the subspace and therefore H is closed.  4.3

Closure and Interior Operators in Topological Spaces

Theorem 4.2 implies that for every topological space (S, O) the collection closed(O) of closed sets is a closure system on S. In addition to the properties listed in Definition 1.35: (i) U ⊆ K(U ) (expansiveness), (ii) U ⊆ V implies K(U ) ⊆ K(V ) (monotonicity), and (iii) K(K(U )) = K(U ) (idempotency) for U, V ∈ P(S). the closure operator attached to a topological space satisfies the following supplementary property: K(H ∪ L) = K(H) ∪ K(L)

(4.1)

for all subsets H, L of S. Since H, L ⊆ H ∪ L, we have K(H) ⊆ K(H ∪ L) and K(L) ⊆ K(H ∪ L) due to the monotonicity of K. Therefore, K(H) ∪ K(L) ⊆ K(H ∪ L). To prove the reverse inclusion, note that the set K(H)∪K(L) is a closed set by the third part of Theorem 4.2 and H ∪L ⊆ K(H)∪K(L). Therefore, the closure of H ∪ L is a subset of K(H) ∪ K(L), so K(H ∪ L) ⊆ K(H) ∪ K(L), which implies equality (4.1). Also, note that K(∅) = ∅ because the empty set itself is closed. Note that equality (4.1) is satisfied for every H, L ∈ P(S) if and only if the union of two K-closed sets is K-closed. Indeed, suppose that equality (4.1) is satisfied, and let U and V be two K-closed sets. Since U = K(U ) and V = K(V ), it follows that U ∪ V = K(U ) ∪ K(V ) = K(U ∪ V ), which shows that U ∪ V is K-closed. Conversely, suppose that the union of two K-closed sets is K-closed. Then, K(U ) ∪ K(V ) is K-closed and contains

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topology

page 167

167

U ∪ V . Therefore, K(U ∪ V ) ⊆ K(U ) ∪ K(V ). The reverse equality follows from the monotonicity of K. Theorem 4.5. Let S be a set and let K : P(S) −→ P(S) be a closure operator that satisfies equality (4.1) for every H, L ∈ P(S) and K(∅) = ∅. The collection OK = {S − U | U ∈ CK } is a topology on S. Proof. We have K(S) = S, so both ∅ and S are K-closed sets, which implies ∅, S ∈ OK . Suppose that C = {Li | i ∈ I} ⊆ OK . Since S − Li ∈ CK , it follows    that {S − Li | i ∈ I} = S − i∈I Li ∈ CK . Thus, i∈I Li ∈ OK . Finally, suppose that D = {D1 , . . . , Dn } is a finite collection of subn sets such that D ⊆ OK . Since S − Di ∈ CK we have S − i=1 Di = n n i=1 (S − Di ) ∈ CK , hence i=1 Di ∈ OK . This proves that OK is indeed a topology.  Definition 4.5. Let (S, O) be a topological space, U be a subset of S and let x ∈ S; x is an adherent point of a set to U if x ∈ K(U ). Clearly, every member of U is adherent to U ; the converse is false. Theorem 4.6. Let (S, O) be a topological space and let U and W be two subsets of S. If U is open and U ∩ W = ∅, then U ∩ K(W ) = ∅. Proof. U ∩ W = ∅ implies W ⊆ S − U . Since U is open, the set S − U is closed, so K(W ) ⊆ K(S − U ) = S − U . Therefore, U ∩ K(W ) = ∅.  Often, we use the contrapositive of this statement: if U is an open set such that U ∩ K(W ) = ∅ for some set W , then U ∩ W = ∅. Corollary 4.1. Let (S, O) be a topological space and let T ⊆ S. Denote by KS and KT the closure operators of (S, O) and (T, O T ), respectively. For every subset W of T , we have KT (W ) = KS (W ) ∩ T . Proof. The set KS (W ) is closed in S, so KS (W ) ∩ T is closed in T by Theorem 4.4. Since W ⊆ KS (W )∩T , it follows that KT (W ) ⊆ KS (W )∩T . To prove the converse inclusion, observe that we can write KT (W ) = T ∩ H, where H is a closed set in S because KT (W ) is a closed set in T . Since W ⊆ H, it follows that KS (W ) ⊆ H, so KS (W ) ∩ T ⊆ H ∩ T =  KT (W ). Corollary 4.2. Let (S, O) be a topological space and let T ⊆ S. If U ⊆ S, then KT (U ∩ T ) ⊆ KS (U ) ∩ T .

May 2, 2018 11:28

168

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 168

Mathematical Analysis for Machine Learning and Data Mining

Proof. By applying Corollary 4.1 to the subset U ∩ T of T we have KT (U ∩ T ) = KS (U ∩ T ) ∩ T . The needed inclusion follows from the  monotonicity of KS . Definition 4.6. A set U is dense in a topological space (S, O) if K(U ) = S. A topological space is separable if there exists a countable set U that is dense in (S, O). Theorem 4.7. If T is a subspace of a separable topological space (S, O), then T itself is separable. Proof. Since S is separable, there exists a countable set U such that KS (U ) = S. On the other hand, KT (U ∩ T ) = KS (U ∩ T ) ∩ T ⊆ KS (U ) ∩ T = S ∩ T = T , which implies that the countable set U ∩ T is dense in T . Thus, T is separable.  Theorem 4.8. If T is a separable subspace of a topological space (S, O), then so is K(T ). Proof. Let U be a countable subset of T that is dense in T , that is, KT (U ) = T . We need to prove that KKS (T ) (U ) = KS (T ) to prove that U is dense in KS (T ) also. By Corollary 4.1, we have KKS (T ) (U ) = KS (U ) ∩ KS (T ) = KS (U ) due to the monotonicity of KS . Note that T = KT (U ) = KS (U ) ∩ T , so T ⊆ KS (U ), which implies KS (T ) ⊆ KS (U ). Since KS is monotonic, we have the reverse inclusion KS (U ) ⊆ KS (T ), so KS (U ) = KS (T ). This allows us to conclude that  KKS (T ) (U ) = KS (T ), so U is dense in KS (T ). Theorem 4.9. Let (S, O) be a topological space. The set U is dense in (S, O) if and only if U ∩ L = ∅ for every non-empty open set L. Proof. Suppose that U is dense, so K(U ) = S. Since K(U ∩L) = K(U ) ∩ K(L) = S ∩ K(L) = K(L), U ∩ L = ∅ would imply K(L) = K(∅) = ∅, which is a contradiction because ∅ = L ⊆ K(L). Conversely, suppose that U has a non-empty intersection with every non-empty open set L. Since K(U ) is closed, S − K(U ) is open. Observe that U ∩ (S − K(U )) = ∅, so the open set S − K(U ) must be empty. Therefore, we have K(U ) = S. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topology

page 169

169

Since ∅ is an open set in any topological space (S, O) and any union of open sets is an open set, it follows that the topology itself is an interior system on S. In addition, an interior system of open sets is closed to finite intersection. Definition 4.7 which follows is a restatement of the definition of the interior operator associated to an interior system contained by Theorem 1.31. Definition 4.7. Let (S, O) be a topological space. The interior of a set U ,  U ⊆ S, is the set I(U ) = {L ∈ O | L ⊆ U }. The interior I(U ) of a set U is the largest open set included in U , because the union of any collection of open sets is an open set. Furthermore, a set is open in a topological space if and only if it equals its interior. Theorem 4.10. Let (S, O) be a topological space and let U be a subset of S. We have K(U ) = I(U ) and K(U ) = I(U ). Proof. Since I(U ) is an open set, the set I(U ) is closed. Note that U ⊆ I(U ). Therefore, K(U ) ⊆ I(U ). Conversely, the inclusion U ⊆ K(U ) implies K(U ) ⊆ U . Since S −K(U ) is an open set included in U and I(U ) is the largest such set, it follows that S − K(U ) ⊆ I(U ), which implies I(U ) ⊆ K(U ). The second equality follows from the first equality by replacing U by its  complement U . Corollary 4.3. For every subset U of a topological space (S, O), we have I(U ) = K(U ) and K(U ) = I(U ). Proof.

Both equalities follow immediately from Theorem 4.10.



An interior operator generated by a topological space (S, O) can be defined axiomatically, as a mapping I : P(S) −→ P(S) that satisfies the conditions of Definition 1.40 and the additional condition: I(U ∩ V ) = I(U ) ∩ I(V ) for U, V ∈ P(S). Theorem 4.11. The following statements are equivalent for a topological space (S, O): (i) every countable intersection of dense open sets is a dense set; (ii) every countable union of closed sets that have an empty interior has an empty interior.

May 2, 2018 11:28

170

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 170

Mathematical Analysis for Machine Learning and Data Mining

Proof. (i) implies (ii): Suppose that (S, O) is a topological space such that every countable intersection of dense open sets is a dense set. Let H1 , . . . , Hn , . . . be a sequence of closed sets with I(Hi ) = ∅ for n  1. Then, for the open sets Li = S − Hi , we have K(Li ) = K(S − Hi ) = S − I(Hi ) = S,  so every set Li is dense. By (i), we have K( i1 Li ) = S, so ⎞ ⎛ ⎞ ⎛   Hi ⎠ I ⎝ Hi ⎠ = S − K ⎝S − i1

⎛ = S − K⎝

i1



⎞

⎛

(S − Hi )⎠ = S − K ⎝

i1



⎞ Li ⎠ = ∅,

i1

which shows that (ii) holds. (ii) implies (i): Conversely, suppose that in (S, O) every countable union of closed sets that have an empty interior has an empty interior. Let L1 , . . . , Ln , . . . be a countable collection of dense open sets. Since each of the sets Li is dense we have K(Li ) = S. Therefore, for the closed sets Hi = S − Li we have I(Hi ) = S − K(Li ) = ∅ by Corollary 4.3, so each ∞ of the sets Hi has an empty interior. By (ii) we have K ( n=1 Hi ) = ∅, so ⎞ ⎛ ⎞ ⎛   Li ⎠ K ⎝ L i ⎠ = S − I ⎝S − i1

⎛ = S − I⎝

i1



⎛

(S − Li )⎠ = S − I ⎝

i1

which shows that (i) holds.

⎞



⎞ Hi ⎠ = S,

i1



Definition 4.8. A Baire1 space is a topological space (S, O) that satisfies one of the equivalent conditions of Theorem 4.11. Theorem 4.12. Every open subspace of a Baire space (S, O) is a Baire space.

1 Ren´ e-Louis Baire was born on January 21st in Paris and died on July 5th 1932 in Chambery. He was a French mathematician who made important contributions to real analysis. Baire taught at the Universities of Montpellier and at the Faculty of Science in Dijon.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topology

page 171

171

Proof. Let U be an open set in (S, O) and let {Hn | n  1} be a collection of closed sets in U that have empty interiors. ˜ n in S such that By Theorem 4.4, for each set Hn there is a closed set H ˜ n. Hn = U ∩ H ˜ n , then V ∩ Hn = ∅. If V is a non-empty open set in S contained in H Then, V ∩ U is a non-empty open set of U , ˜n) = V ∩ H ˜ n = V, V ∩ Hn = V ∩ (U ∩ H so V is contained in Hn , which contradicts the assumption that Hn has a non-empty interior.   ˜n If n1 Hn contains a non-empty open subset W of U , then n1 H also contains W ; W is open in S because it is open in U . Since each set ˜ n has an empty interior in S, this contradicts the fact that S is a Baire H space.  Definition 4.9. A subset U of a topological space (S, O) is nowhere dense if its closure has an empty interior, that is, I(K(U )) = ∅. A subset V of S is of first category it equals a countable union of nowhere dense sets; a subset is of second category if it is not of first category. Note that U is nowhere dense in (S, O) if and only if its closure K(U ) has the same property. Theorem 4.13. The following statements are equivalent: (i) U is nowhere dense in (S, O); (ii) I(K(U )) = S; (iii) K(K(U )) = S; (iv) I(U ) is dense in S. Proof. The equivalence of these statements is an immediate consequence of the definitions.  We add now two further characterizations of Baire spaces. Theorem 4.14. For a topological space (S, O) the following are equivalent: (i) (S, O) is a Baire space;   (ii) if S = n1 Tn and each Tn is closed, then the open set n1 I(Tn ) is dense in S; (iii) non-empty open sets are not of first category. Proof. (i) implies (ii): Let (Tn ) be a sequence of closed sets with   S = n1 Tn and let G be the open set, G = n1 I(Tn ). For each n

May 2, 2018 11:28

172

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 172

Mathematical Analysis for Machine Learning and Data Mining

let Vn = Tn − I(Tn ). Note that Vn is a closed set and, furthermore, Vn is  nowhere dense. This implies that V = n1 Vn is nowhere dense. Since Vn is closed and nowhere dense, each Vn is an open dense set. By  (i), V = n1 Vn is also dense. Note that    G= Tn − I(Tn ) ⊆ (Tn − I(Tn )) = V, n1

n1

n1

so V ⊆ G. Since V is dense, so is G. (ii) implies (iii): Let G be a non-empty open set. Suppose that G is of first category, that is G can be written as a countable union G =  n1 An , where I(K(An )) = ∅. Then S = G ∪ K(A1 ) ∪ · · · ∪ K(An ) ∪ · · · is a union of closed sets, so by (ii) the open set I(G) ∪ I(K(A1 )) ∪ I(K(A2 )) ∪ · · · = I((G)) is dense in S. Since I(G) ⊆ G it follows that G is dense in S. This implies G ∩ G = ∅ which is impossible. Thus, G is not of first category. (iii) implies (i): If G is an open dense set, then G is nowhere dense. Indeed, since G is closed it suffices to show that I(G) = ∅. Since I(G) = K(G), we have I(G) = ∅ because G is dense. Assume that (Gn ) is a sequence of open dense subsets of S. Let A =  n1 Gn and assume that A∩U = ∅ for some non-empty open set U . Since S = A ∩ U = A ∪ U , we have ⎛ ⎞   Gn ⎠ ∩ U = (Gn ∩ U ). U = X ∩U = A∩U = ⎝ n1

n1

This shows that U is a first category set, which is not the case. Therefore, A is dense in S.  Definition 4.10. Let (S, O) be a topological space. The boundary of a set U , where U ∈ P(S), is the set ∂S U = K(U ) ∩ K(S − U ). If S is clear from the context, then we omit the subscript and denote the boundary of U just by ∂U . The boundary of every set is a closed set as an intersection of two closed sets. By Corollary 4.3 the boundary of a set can be expressed also in term of interiors: ∂U = (S − I(S − U )) ∩ (S − I(U )) = S − (I(S − U ) ∪ I(U )).

(4.2)

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topology

page 173

173

Note that for every set U ⊆ S, where (S, O) is a topological space we have S − K(U ) ⊆ S − U ⊆ K(S − U ). Therefore, we have S − U ⊆ K(S − U ) = (S − K(U )) ∪ ∂U.

(4.3)

Theorem 4.15. The boundary of a subset U of a topological space (S, O) consists of those elements s of S such that for every open set L that contains s we have both L ∩ U = ∅ and L ∩ (S − U ) = ∅. Proof. Let x ∈ ∂U and let L be an open set such that x ∈ L. By equality( 4.2), we have both x ∈ I(S − U ) and x ∈ I(U ). Therefore, L ⊆ S − U and L ⊆ U , which imply L ∩ U = ∅ and L ∩ (S − U ) = ∅. Conversely, suppose that, for every open set L that contains s, we have both L∩U = ∅ and L∩(S −U ) = ∅. This implies x ∈ I(U ) and s ∈ I(S −U ), so x ∈ ∂U by equality (4.2).  Theorem 4.16. Let (S, O) be a topological space, (T, O T ) be a subspace, and W be a subset of S. The boundary ∂T (W ∩ T ) of W ∩ T in the subspace T is a subset of the intersection ∂S (W ) ∩ T , where ∂S (W ) is the boundary of W in S. Proof.

By Definition 4.10, we have ∂T (W ∩ T ) = KT (W ∩ T ) ∩ KT (T − (W ∩ T )) = KT (W ∩ T ) ∩ KT (T − W ) ⊆ (KS (W ) ∩ T ) ∩ KT (T − W ) (by Corollary 4.1).

Again, by Corollary 4.1, we have KT (T − W ) = KT (T ∩ (S − W )) ⊆ KS (S − W ) ∩ T , and this allows us to write ∂T (W ∩ T ) ⊆ (KS (W ) ∩ T ) ∩ KS (S − W ) ∩ T = ∂S (W ) ∩ T, which is the desired conclusion.



The next statement relates three important sets that we defined for each subset U of a topological space (S, O). Theorem 4.17. Let (S, O) be a topological space. For every subset U of S, we have K(U ) = I(U ) ∪ ∂U .

May 2, 2018 11:28

174

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 174

Mathematical Analysis for Machine Learning and Data Mining

Proof. By equality (4.2), we have ∂U = (S − I(S − U )) ∩ (S − I(U )). Therefore, ∂U ∪ I(U ) = (S − I(S − U )) ∩ I(U ) (by Corollary 4.3) = K(U ) ∩ I(U ) (because I(U ) ⊆ K(U )) = K(U ).



Corollary 4.4. Let (S, O) be a topological space and let (T, O T ) be a subspace of (S, O). For any subset U of S, we have ∂T (U ∩ T ) ⊆ ∂S (U ). Proof. Let t ∈ ∂T (U ∩ T ). By Theorem 4.15, for every open set L ∈ O T such that t ∈ L we have both L ∩ (U ∩ T ) = ∅ and L ∩ (T − (U ∩ T )) = ∅. If L1 is an open set of (S, O) that contains S, then L1 ∩ T is an open set of (T, O T ) that contains t, so for L1 we have both (L1 ∩ T ) ∩ (U ∩ T ) = ∅ and (L1 ∩ T ) ∩ (T − (U ∩ T )) = ∅. This immediately implies L1 ∩ U = ∅  and L1 ∩ (S − U ) = ∅, that is, t ∈ ∂S (U ). By applying the notations introduced in Section 1.4 we have Oσ = O and closed(O)δ = closed(O). It is clear that Oδ is the collection that consists of all countable intersections of open sets and closed(O)σ is the collection that consists of all countable unions of closed sets. Note that Q ∈ closed(O)σ , because Q is the countable union Q =  {{x} | x ∈ qq}, and each set {x} is closed in (R, O). 4.4

Neighborhoods

Definition 4.11. Let (S, O) be a topological space. A subset U of S is a neighborhood of x ∈ S if there exists an open set T such that x ∈ T ⊆ U . The collection of neighborhoods of an element x of S is denoted by neighx (O). Equivalently, a subset U of (S, O) is a neighborhood of x ∈ S if x ∈ I(U ). Example 4.8. Let (R, O) be the topological space introduced in Example 4.2. A neighborhood of x ∈ R is a set U such that there exists > 0 such that B(x, ) ⊆ U .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topology

page 175

175

Indeed, if U ∈ neighO (x), then there is an open set T such that x ∈ T ⊆ U . Since T is an open set and x ∈ T , there exists > 0 such that B(x, ) ⊆ T ⊆ U , so B(x, ) ⊆ U . Conversely, if there exists > 0 such that B(x, ) ⊆ U , then U ∈ neighx (O) because B(x, ) is an open set that contains x and is included in U. Theorem 4.18. Let (S, O) be a topological space. A subset L of S is open if and only if L is a neighborhood of all its points. Proof. If L is open, it is immediate that L is a neighborhood of all its points. Conversely, suppose that L is a neighborhood of all its members. Then, for each x ∈ L there exists Wx ∈ O such that x ∈ Wx ⊆ L. Therefore,   {x} ⊆ Wx ⊆ L, L= 

x∈L

x∈L

 x∈L Wx . This in turn implies L ∈ O.  In other words, L ∈ O if and only if L ∈ x∈L neighx (O). Thus, if U ∈ neighx (O) there exists always an open neighborhood T of x such that x ∈ T ⊆ U. which implies L =

Theorem 4.19. Let (S, O) be a topological space. A subset T of S is closed if and only for each x ∈ T and V ∈ neighx (O) we have V ∩ T = ∅. Proof. Let T be a closed subset of S and let x ∈ T . The set S − T is open. If V is a neighborhood of x then V ∩ T = ∅ because, otherwise, we would have x ∈ V ⊆ S − T , which is a contradiction. Conversely, suppose that for each x ∈ T and V ∈ neighx (O) we have V ∩ T = ∅. We claim that in this case the set S − T is open. Indeed, let y ∈ S − T . Since y ∈ T there exists W ∈ neighy (O) such that W ∩ T = ∅, which is equivalent to W ⊆ S − T . Therefore, S − T is a neighborhood of y, so S − T is open by Theorem 4.18, which means that T is closed.  Corollary 4.5. Let O, O be two topologies on a set S. The topology O is finer than O if and only if neighx (O) ⊆ neighx (O ) for every x ∈ S. Proof.

This is an immediate consequence of Theorem 4.18.



Theorem 4.20. The following statements hold for any topological space (S, O) and x ∈ S: (i) there exists at least one set in neighx (O) for every x ∈ S; (ii) we have x ∈ W for every W ∈ neighx (O);

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 176

Mathematical Analysis for Machine Learning and Data Mining

176

(iii) if U, V ∈ neighx (O), then U ∩ V ∈ neighx (O); (iv) if U ∈ neighx (O) and U ⊆ W ⊆ S, then W ∈ neighx (O); (v) each set W ∈ neighx (O) contains a subset U ∈ neighx (O) such that U ∈ neighy (O) for each y ∈ U . Proof. Part (i) follows from the fact that S ∈ neighx (O) for each x ∈ S. If U ∈ neighx (O) there is W ∈ O such that x ∈ W ⊆ U , so x ∈ U , and we obtain part (ii). The proofs of parts (iii) and (iv) are left to the reader. For part (v) let x ∈ S and let W ∈ neighx (O). There exists an open set T such that x ∈ T ⊆ W . Since T is a neighborhood for each of its members, part (v) follows.  An alternative technique for introducing topologies starts with families of sets that satisfy characteristic properties of collections of neighborhoods. Theorem 4.21. Let S be a set such that for each x ∈ S there exists a collection Nx of subsets of S with the following properties: (i) there exists at least one set U in Nx for every x ∈ S; (ii) we have x ∈ W for every W ∈ Nx ; (iii) if U, V ∈ Nx , then U ∩ V ∈ Nx ; (iv) if U ∈ Nx and U ⊆ W ⊆ S, then W ∈ Nx ;  (v) each set W ∈ Nx contains a subset U ∈ Nx such that U ∈ y∈U Ny . There exists a unique topology O on S such that for each x ∈ S, neighx (O) = Nx . A subset L is open if and only if L ∈ Nx for each x ∈ L. Proof.

Define the family of sets O = {L | L ⊆ S, L ∈ Nx for each x ∈ L}.

It is immediate that ∅ ∈ O. Let x ∈ S. By (i) and (ii) there exists U ∈ Nx such that x ∈ U ⊆ S and by (iv), S ∈ Nx , so S ∈ O.  Let {Vi | i ∈ I} be a family of sets that belong to O and let V = i∈I Vi . If x ∈ V , there exists Vi ∈ Nx such that x ∈ Vi ⊆ V . By (iv) V ∈ Nx and, since this holds for every x ∈ V , we have V ∈ O. If V and W are two subsets of S that belong to O, we need to show that for each x ∈ V ∩ W we have V ∩ W ∈ Nx . Note that if x ∈ V ∩ W , then V ∈ Nx and W ∈ Nx , so V ∩ W ∈ Nx by (iii), which implies V ∩ W ∈ O. We conclude that O is a topology.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Topology

9in x 6in

b3234-main

page 177

177

We claim that neighx (O) = Nx for each x ∈ S. Indeed, if U is a neighborhood of x relative to the topology O, there exists an open set L ∈ O such that x ∈ L ⊆ U . By the definition of O, L ∈ Nx , which yields U ∈ Nx by (iv). Thus, neighx (O) ⊆ Nx for each x ∈ S. Conversely, let V ∈ Nx . By (v), there is a set U ∈ neighx (O) such that U ⊆ V and U ∈ Ny for every y ∈ U . By the definition of O we have U ∈ O, so V ∈ neighx (O). This shows that neighx (O) = Nx for each x ∈ S. Theorem 4.18 implies the uniqueness of the topology O.  ˆ = R ∪ {−∞, ∞} be the extended set of real numbers. Example 4.9. Let R ˆ is defined as a subset of R ˆ that includes a A neighborhood of ∞ in R ˆ is a set set (a, ∞], where a ∈ R; similarly, a neighborhood of −∞ in R that includes a set of the form [−∞, b) for some b ∈ R. The collection of neighborhoods of ∞ and −∞ are denoted by N∞ and N−∞ , respectively.  We claim that the collection a∈R Na ∪ N∞ ∪ N−∞ satisfies the condiˆ tions of Theorem 4.21, and thus is defining a unique topology on R. Suppose that U, V ∈ N∞ . Then, (a, ∞] ⊆, (b, ∞] ⊆ V for some a, b ∈ R and (max{a, b}, ∞] ⊆ U ∩ V , hence U ∩ V ∈ N∞ . Similarly, if U, V ∈ N−∞ , then U ∩ V ∈ N−∞ . ˆ it is immediate that W ∈ N−∞ ; a similar If U ∈ N∞ and U ⊆ W ⊆ R property holds for N−∞ . Let now W be a set in N∞ , so there exists a ∈ R such that (a, ∞] ⊆ W . The role of the set U of the fifth condition of Theorem 4.21 is played by the open set U = (a, ∞). A similar argument works for W ∈ N−∞ . The ˆ topology defined in Theorem 4.21 is the usual topology for R. ˆ Note that the set R is an open set in the topology introduced on R. ˆ is an open in R, ˆ then T ∩ R is open in R, so the Furthermore, if T ⊆ R ˆ usual topology on R is the trace of the topology defined on R. Theorem 4.22. The boundary ∂U of a subset U of a topological space (S, O) consists of those elements x of S such that for every V ∈ neighs (O) that contains x we have both V ∩ U = ∅ and V ∩ (S − U ) = ∅. Proof. By Theorem 4.15, x ∈ ∂T if and only if we have both L ∩ U = ∅ and L ∩ (S − U ) = ∅ for every open set that contains x. These conditions are equivalent to V ∩ U = ∅ and V ∩ (S − U ) = ∅ for each neighborhood V ∈ neighx (O) because every open set that contains x is a neighborhood of x, and every neighborhood of x contains an open set that contains x. 

May 2, 2018 11:28

178

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 178

Mathematical Analysis for Machine Learning and Data Mining

Theorem 4.23. Let (S, O) be a topological space. For every subset U of S the following statements hold: (i) ∂U = ∂(S − U ); (ii) U ∈ O if and only if ∂U ⊆ S − U ; (iii) U is closed if and only if ∂U ⊆ U . Proof. Definition 4.10, ∂U = K(U ) ∩ K(S − U ) implies immediately (i). For part (ii), if U ∈ O, S − U is a closed set, so S − U = K(S − U ), which implies ∂U = K(U ) ∩ K(S − U ) ⊆ S − U . Conversely, suppose ∂U ⊆ S −U and let x ∈ U , so x ∈ S −U . Thus, x ∈ ∂U , which means that there exists V ∈ neighx (O) that does not intersect both U and S −U . Since V intersects U , it follows that V does not intersect S − U , that is, V ⊆ U . Therefore, U is open. Finally, U is closed, if and only if S − U is open. By part (ii), this is equivalent to ∂(S − U ) ⊆ U , which by part (i) means that ∂U ⊆ U .  Theorem 4.24. Let (S, O) be a topological space and let x ∈ S. We have x ∈ K(T ) if and only if V ∩ T = ∅ for every V ∈ neighx (O). Proof. Suppose that x ∈ K(T ). If x ∈ I(T ), then x ∈ T , so the intersection of every V ∈ neighx (O) with T contains at least x, so it is non-empty. If x ∈ ∂T , then each V ∈ neighx (O) has a non-empty intersection with both T and S − T . Conversely, suppose that every V ∈ neighx (O) has a non-intersection with T . If some neighborhood is contained in T , then x ∈ I(T ) ⊆ K(T ). Otherwise, we have V ∩ T = ∅ and V ∩ (S − T ) = ∅, so x ∈ ∂T ⊆ K(T ).  Definition 4.12. Let (S, O) be a topological space and let x ∈ S. A family of neighborhoods Lx (O) of x is a local basis (or a neighborhood basis at x) if for every neighborhood V ∈ neighx (O) there exists a L ∈ Lx (O) such that x∈L⊆V. Clearly, neighx (O) itself is a local basis at x. Some authors use the term fundamental system of neighborhoods instead of local basis. Example 4.10. For the set of reals equipped with the usual topology (R, O) the collection of open intervals {(x − 1/n, x + 1/n) | n  1} is a local basis at x.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Topology

b3234-main

page 179

179

In Theorem 4.19 we characterized the points of the closure of a subset T of a topological space (S, O) as those points of S such that each of their neighborhoods has a non-empty intersection with T . Next, we introduce a class of points that satisfy a more stringent condition. Definition 4.13. Let (S, O) be a topological space and let T be a subset of S. An element t of S is an accumulation point, or a cluster point, or a limit point of T if, for every V ∈ neight (O) the set (T − {t}) ∩ V is not empty. The set of all accumulation points of a set T is the derived set of T and is denoted by T  . A element u is an isolated point of T if there exists V ∈ neighu (O) the set T ∩ V = {u}. Clearly, any accumulation point of a set T belongs to K(T ). Also, if t is an accumulation point of a set T , then it is also an accumulation point for any set that contains T . In other words, T1 ⊆ T2 implies T1 ⊆ T2 for T1 , T2 ⊆ S. Theorem 4.25. T is closed if and only if T  ⊆ T . Proof. T is closed if and only if S − T is open, which is equivalent to saying that each point x ∈ S − T has a neighborhood Wx included in S − T , or equivalently, Wx ∩ T = ∅. Thus, T is closed if and only if for x ∈ S, if every V ∈ neighx (O) has a non-empty intersection with T , then x ∈ T . In other words, every accumulation point of T belongs to T , that is,  T ⊆ T. Theorem 4.26. Let (S, O) be a topological space and let T be a subset of S. The set T ∪ T  is closed. Proof. Let x ∈ T ∪ T  and let V ∈ neighx (O). If x ∈ T , then x ∈ V ∩ T , so V ∩ (T ∪ T  ) = ∅. If x ∈ T  , then (T − {t}) ∩ V = ∅ by the definition of accumulation points, so, again V ∩ (T ∪ T  ) = ∅. Thus, in either case  V ∩ (T ∪ T  ) = ∅, which means that T ∪ T  is closed. Theorem 4.27. Let (S, O) be a topological space and let T be a subset of S. We have K(T ) = T ∪ T  . Proof. Since T ∪ T  is a closed set, it follows that K(T ) ⊆ T ∪ T  . Conversely, if x ∈ T ∪ T  it follows that every neighborhood of x intersects T , so x ∈ K(T ), which yields the desired equality. 

May 2, 2018 11:28

180

4.5

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 180

Mathematical Analysis for Machine Learning and Data Mining

Bases

Definition 4.14. Let (S, O) be a topological space. A collection B of open sets is a basis for (S, O) if every open set is a union of a subcollection of B. Example 4.11. By Theorem 4.1 a subset U of R is open in the topological space (R, O) if and only if it equals the union of a countable collection of disjoint open intervals. This shows that the collection of open intervals is a basis for the (R, O). If B is a basis for (S, O) and T is a subset of S, then the trace BT is a basis for the subspace (T, OT ).  Theorem 4.28. If B is a basis for the topological space (S, O), then B = S and for each two sets B1 , B2 ∈ B and each x ∈ B1 ∩ B2 there exists a set B ∈ B such that x ∈ B ⊆ B1 ∩ B2 .  Proof. Since S is an open set, the definition of a basis implies S = B. Then, if x ∈ B1 ∩ B2 , taking into account that B1 ∩ B2 is an open set, the last part of the theorem is immediate.  Theorem 4.29. Let S be a set. A collection B of subsets of S such that  (i) B = S, and (ii) for each two sets B1 , B2 ∈ B and each x ∈ B1 ∩ B2 there exists a set B ∈ B such that x ∈ B ⊆ B1 ∩ B2 is a basis for a topology on S. Proof. Let O be the collection of subsets of S that are unions of a subcollections of B. It is immediate that the union of any subcollection of O belongs to O. Let U, V ∈ O. If U ∩ V = ∅, then U ∩ V ∈ O because ∅ is the union of the empty subcollection of B. Suppose that x ∈ U ∪ V . There exist B1 , B2 ∈ B such that x ∈ B1 ⊆ U and x ∈ B2 ⊆ V . By hypothesis, there  exists a set Bx with x ∈ Bx ⊆ B1 ∩B2 ⊆ U ∩V . Then x∈U∩V Bx = U ∩V , so U ∩ V ∈ O. Now, an immediate argument by induction of n shows that if   U1 , . . . , Un ∈ O, then ni=1 Ui ∈ O. ˆ defined by Example 4.12. Consider the collections of subsets of R B = {(a, b) | −∞  a < b  ∞}, B−∞ = {[−∞, b) | b ∈ R}, B∞ = {(a, ∞] | a ∈ R}.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Topology

9in x 6in

b3234-main

page 181

181

ˆ We refer By Theorem 4.29 the collection B is a basis for a topology on R. to this topology as the usual topology of the extended set of reals. The restriction of this topology to R is the usual topology of reals as defined in Example 4.3. In other words, a set U is open in R if and only if U is open ˆ in R. Let O = {Oi | i ∈ I} be a family of topologies defined on a set S that contains the discrete topology P(S). We claim that O is a closure system on P(S). The first condition of Definition 1.34 is satisfied due to the definition of O. It is easy to verify that for every subfamily O of O,   O is a topology, so O is indeed a closure system. Thus, if S is a family of subsets of S, there exists the smallest topology that includes S. Theorem 4.30. The topology TOP(S) generated by a family S of subsets of S consists of unions of finite intersections of the members of S. Proof. Let E be the collection of all unions of finite intersections of the members of S. It is clear that S ⊆ E. We claim that E is a topology that contains S. Note that the intersection of the empty collection of sets in S is S, so S ∈ E; also, the union of an empty collection of finite intersections is ∅, so ∅ ∈ E.  Every U ∈ E can be written as U = {Vj | j ∈ JU }, where the sets Vj are finite intersections of sets of S. Therefore, it is immediate that any union of sets of this form belongs to E. Suppose that {Ui | i ∈ I} is a finite collection of parts of S, where  Ui = {Vj ∈ S | j ∈ Ji } and that each Vj can be written as Vj =  {Wjh ∈ S | h ∈ Hj }, where each set Hj is finite. One can prove by  induction on p = |I| that {Ui | i ∈ I} ∈ E. To simplify the presentation,  we discuss here only the case where |I| = 2. So, if Ui = {Vj ∈ S | j ∈ Ji } for i = 1, 2, we have   U1 ∩ U2 = {Vj1 ∈ S | j1 ∈ J1 } ∩ {Vj2 ∈ S | j2 ∈ J2 }  (Vj1 ∩ Vj2 ). = j1 ,j2

Since each intersection Vj1 ∩ Vj2 is in turn a finite intersection of sets of S, it follows that U1 ∩ U2 ∈ S. Thus, TOP(S) is contained in E because TOP(S) is the coarsest topology that contains S. This gives the desired conclusion. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 182

Mathematical Analysis for Machine Learning and Data Mining

182

Corollary 4.6. Let B be a collection of subsets of the set S such that for  every finite subcollection D of B, x ∈ D implies the existence of a set  B ∈ B such that x ∈ B ⊆ D. Then, TOP(B), the topology generated by B, consists of sets that are unions of subcollections of B. Proof. By Theorem 4.30, TOP(B) consists of unions of finite intersections of the members of B. Therefore, unions of sets of B belong to TOP(B).  Conversely, let U ∈ B, that is, U = {Vi | i ∈ I}, where each Vi is a finite intersection of members of B. For every x ∈ Vi , there exists a set  Bx,i ∈ B such that x ∈ Bx,i ⊆ Vi . Therefore, Vi = x∈Vi Bx,i , and this implies that U is indeed a union of sets from B.  Example 4.13. The collection of open intervals {(a, b) | a, b ∈ R and a < b} is a basis for the topological space (R, O) by Theorem 4.1. Definition 4.15. Let S be a set. A collection S of subsets of S is a sub-basis for a topology O if O = TOP(S). Theorem 4.31. The topology O generated by a sub-basis S has as a basis the collection B(S) of intersections of finite subcollections of sets in S. Proof. Note that S ∈ B(S) as the intersection of the empty subcollection of sets of S.   Let B1 = S1 and B2 = S2 , where S1 , S2 are two finite subcollections of sets in S. If x ∈ B1 ∩ B2 , then x belongs to all sets in S1 ∪ S2 . Then, if B is the intersection of all sets in S1 and S2 , we have x ∈ B ⊆ B1 ∩ B2 , so, by Theorem 4.29, B(S) is a basis of a topology O . Since S ⊆ B(S) ⊆ O , we have O ⊆ O because O is the smallest topology containing S. Finally, if W is a set in B(S), since W is the intersection of  a finite subcollection of S and S ⊆ O, we have W ∈ O. Thus, O ⊆ O. Example 4.14. The collection S = {(a, +∞) | a ∈ R}∪{(−∞, b) | b ∈ R} is a sub-basis of the usual topology on R because every member (a, b) of the basis can be written as (a, b) = (−∞, b) ∩ (a, +∞). Starting from a topology, we find a basis using the following theorem. Theorem 4.32. Let (S, O) be a topological space. If B is a collection of open subsets of S such that for every x ∈ S and every open set L ∈ O there exists a set B ∈ B such that x ∈ B ⊆ L, then B is a basis for (S, O). Proof.

This statement is an immediate consequence of Definition 4.15. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Topology

9in x 6in

b3234-main

page 183

183

Theorem 4.33. Let (S, O) be a topological space. The following statements involving a family B of subsets of S are equivalent: (i) B is a basis for (S, O); (ii) for every x ∈ S and U ∈ neighx (O), there exists B ∈ B such that x ∈ B ⊆ U; (iii) for every open set L, there is a subcollection C of B such that  L = C. Proof. (i) implies (ii): Let B be a basis for (S, O) and let U ∈ neighx (O). There exists an open set L such that x ∈ L ⊆ U . Since B is a basis, there exists a set B ∈ B such that x ∈ B ⊆ L ⊆ U , which is what we aimed to prove. (ii) implies (iii): Suppose that the second statement holds, and let L be an open set. Since L is a neighborhood for all its elements, for every x ∈ L  there exists Bx ∈ B such that {x} ⊆ Bx ⊆ L. Therefore, L = {Bx | x ∈ L}. (iii) implies (i): part (iii) implies part (i) immediately.  Corollary 4.7. Let U be a subspace of a topological space (S, O). If B is a basis of (S, O), then BU = {U ∩ B | B ∈ B} is a basis of the subspace U . Proof. Let K be an open subset in the subspace U . There is an open set L in (S, O) such that K = U ∩ L. Since B is a basis for (S, O), by the third  part of Theorem 4.33, there is a subcollection C of B such that L = C,   which implies K = {U ∩ C | C ∈ C}. Thus, BU is a basis for U . Definition 4.16. A topological space (S, O) satisfies the first axiom of countability if for every x ∈ S there is a countable local basis of neighborhoods at x. A topological space satisfies the second axiom of countability if it has a countable basis. It is clear that the second axiom of countability implies the first. Furthermore, by Corollary 4.7, every subspace of a topological space that satisfies the second axiom of countability satisfies this axiom itself. Theorem 4.34. Let (S, O) be a topological space. If (S, O) has a countable basis, then (S, O) is separable. Proof. Let {Bn | n ∈ N} be a countable basis for (S, O) and let xn be an element of Bn for n ∈ N. We claim that S = K({xn | n ∈ N}), which is equivalent to S − K({xn | n ∈ N}) = ∅.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 184

Mathematical Analysis for Machine Learning and Data Mining

184

Indeed, observe that S − K({xn | n ∈ N}) is a non-empty open set; therefore, there exists m ∈ N such that Bm ⊆ S − K({xn | n ∈ N}), so xm ∈ S − K({xn | n ∈ N}) ⊆ S − {xn | n ∈ N}, which is a contradiction.  Therefore, the countable set {xn | n ∈ N} is dense in (S, O). The notion of an open cover of a topological space is introduced next. Definition 4.17. A cover of a topological space (S, O) is a collection of  sets C such that C = S. If C is a cover of (S, O) and every set C ∈ C is open (respectively, closed), then we refer to C as an open cover (respectively, a closed cover). A subcover of an open cover C is a collection D such that D ⊆ C and  D = S. Theorem 4.35. If a topological space (S, O) satisfies the second axiom of countability, then every basis B for (S, O) contains a countable collection B0 that is a basis for (S, O). Proof. Let B = {Li | i ∈ N} be a countable basis for (S, O) and let Ci be the subcollection of B defined by Ci = {V ∈ B | V ⊆ Li } for i ∈ N. Since B is a basis for (S, O), it is clear that Ci is an open cover for Li ; that is,  Ci = Li for every i ∈ N. Since each subspace Li has a countable basis, Ci  contains a countable subcover Ci of Li . The collection B0 = {Ci | i ∈ N} is countable and is a basis for (S, O) that is included in B.  Theorem 4.36. (Lindel¨ of ’s Theorem) Let (S, O) be a topological space that satisfies the second axiom of countability. If C is an open cover of (S, O), then C contains some countable subcover of S. Proof.

Let B be a countable basis of (S, O) and let D = {B ∈ B | B ⊆ C for some C ∈ C}.

D is countable since D ⊆ B. For D ∈ D let CD ∈ C be a set such that D ⊆ CD . Note that if K = {CD | D ∈ D}, then K is countable because K ⊆ C. Let x ∈ S. There exists C ∈ C such that x ∈ C. Since B is a basis, x ∈ B ⊆ C for some B ∈ B. Then, B ∈ D, CD ∈ K and x ∈ CD , which proves that K is a countable cover contained in C.  Definition 4.18. Let (S, O) be a topological space. A subset U of S is clopen if it is both open and closed.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Topology

b3234-main

page 185

185

Clearly, in every topological space (S, O), both ∅ and S are clopen sets. Example 4.15. Let π = {Bi | i ∈ I} be a partition of a set S. The partition topology on S determined by π is the collection of subsets of S that equal an union of blocks of π, that is, are π-saturated (see Example 1.10). It is easy to verify that the collection of π-saturated sets is a topology. Moreover, since the complement of a π-saturated set is π-saturated set, it follows that all open sets of this topology are clopen. Example 4.16. Let a and d be two integers. An infinite arithmetic progression on the set Z is a set of the form Pa,d = {n ∈ Z | n = a + kd, k ∈ Z}. The number d is the difference of this progression. It is easy to see that we have Pa,d = Pa ,d for every member a of Pa,d . Also, note that P0,1 = Z. If d|d , the definition of infinite arithmetic progression implies Pa,d ⊆ Pa,d . Let Pa,d and Pb,e be two arithmetic progressions that have a non-empty intersection. This means that there is n0 ∈ Z such that n0 = a + k0 d = b + h0 e for some k0 , h0 ∈ Z. It is clear that Pa,d = Pn0 ,d and Pb,e = Pn0 ,e . If r is the least common multiple of d and e, then Pn0 ,r ⊆ Pn0 ,d ∩ Pn0 ,e = Pa,d ∩ Pb,e . By Theorem 4.29, the collection of infinite arithmetic progression is a topology basis on Z. Observe that Pa,d = Z −

d−1 

Pa+i,d .

i=1

This shows that every set of this basis is also closed as a complement of a finite union of member of the basis. Therefore, every progression Pa,d is a clopen set, which implies that every open set of this topology is clopen. This topology on Z was introduced in [61] as a tool for reproving Euclid’s result that stipulates that there exists an infinite number of prime numbers. Note that the single integers that are not divisible by a prime are −1 and 1. Suppose that there exist only a finite number of prime numbers p1 , . . . , pm . Then P0,p1 ∪ P0,p2 ∪ · · · ∪ P0,pm = Z − {−1, 1}. Since the sets P0,p1 , . . . , P0,pm are clopen, the set {−1, 1} is open. This, however is a contradiction since in this topology an open set is either empty or is infinite.

May 2, 2018 11:28

186

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 186

Mathematical Analysis for Machine Learning and Data Mining

Theorem 4.37. Let (S, O) be a topological space. A set U is clopen if and only if ∂U = ∅. Proof. Suppose that U is clopen. Then U = K(U ); moreover, S − U is also closed (because U is open) and therefore S − U = K(S − U ). Thus, K(U ) ∩ K(S − U ) = U ∩ (S − U ) = ∅, so ∂U = ∅. Conversely, suppose that ∂U = ∅. Then, since K(U ) ∩ K(S − U ) = ∅, it follows that K(U ) ⊆ S − K(S − U ). Therefore, K(U ) ⊆ S − (S − U ) = U , which implies K(U ) = U . Thus, U is closed. Furthermore, by equality (4.2), ∂U = ∅ also implies I(S − L) ∪ I(L) = S, so S − I(S − L) ⊆ I(L). By Corollary 4.3, we have K(L) ⊆ I(L), so L ⊆ I(L). Thus, L = I(L), so L is also an open set.  The definition of neighborhoods allows us to introduce the notion of convergent sequence in a topological space. Definition 4.19. Let (S, O) be a topological space. A sequence x = (xn ) ∈ Seq(S) converges to x ∈ S if for every neighborhood U ∈ neighx (O) there exists a number nU such that n  nU implies xn ∈ U . If x converges to x we say that x is a limit point for the sequence (xn ) and that x is a convergent sequence. Note that can replace neighx (O) in Definition 4.19 by a local basis of neighborhoods at x. Example 4.17. Let (S, O) be a topological space where S is an infinite set and let O be the cofinite topology introduced in Example 4.4. Let (xn ) be a sequence in S such that i = j implies xi = xj . Then, for every x ∈ S is a limit point for (xn ). Indeed, let V ∈ neighx (V ). Note that V contains an open set (which is infinite), so S − V is finite. Thus, there exists i ∈ N such that xi ∈ S − V for j  i, which implies xi ∈ V for all j  i. This example also shows that a sequence may converge to more than one element. Theorem 4.38. Let x = (x0 , . . . , xn , . . .) be a sequence in (R, O), where O is the usual topology on R. If x is an increasing (decreasing) sequence and there exists a number b ∈ R such that xn  b (xn  b, respectively), then the sequence x is convergent. Proof. Since the set {xn | n ∈ N} is bounded above, its supremum s exists by the Completeness Axiom for R. We claim that limn→∞ xn = s.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Topology

9in x 6in

b3234-main

page 187

187

Indeed, by Theorem 1.16, for every > 0 there exists n ∈ N such that s − < xn  s. Therefore, by the monotonicity of the sequence and its boundedness, we have s − < xn  s for n  n , so xn ∈ B(x, ), which proves that x converges to s. We leave it to the reader to show that any decreasing sequence in (R, O) that is bounded below is convergent.  If x is an increasing sequence and there is no upper bound for x, this means that for every b ∈ R there exists a number nb such that n  nb implies xn > b. If this is the case, we say that x is a sequence divergent to +∞ and we write limn→∞ xn = +∞. Similarly, if x is a decreasing sequence and there is no lower bound for it, this means that for every b ∈ R there exists a number nb such that n  nb implies xn < b. In this case, we say that x is a sequence divergent to −∞ and we write limn→∞ xn = −∞. Theorem 4.38 and the notion of a divergent sequence allow us to say that limn→∞ xn exists for every increasing or decreasing sequence; this limit may be a real number or ±∞ depending on the boundedness of the sequence. Theorem 4.39. Let [a0 , b0 ] ⊇ [a1 , b1 ] ⊇ · · · ⊇ [an , bn ] ⊇ · · · be a sequence of nested closed intervals of real numbers. There exists a closed interval [a, b] such that a = limn→∞ an , b = limn→∞ bn , and  [an , bn ]. [a, b] = n∈N Proof. The sequence a0 , a1 , . . . , an , . . . is clearly increasing and bounded because we have an  bm for every n, m ∈ N. Therefore, it converges to a number a ∈ R and a  bm for every m ∈ N. Similarly, b0 , b1 , . . . , bn , . . . is a decreasing sequence that is bounded below, so it converges to a number  b such that an  b for n ∈ N. Consequently, [a, b] ⊆ n∈N [an , bn ].  Conversely, let c be a number in n∈N [an , bn ]. Since c  an for n ∈ N, it follows that c  sup{an | n ∈ N}, so c  a. A similar argument shows that  c  b, so c ∈ [a, b], which implies the reverse inclusion n∈N [an , bn ] ⊆ [a, b].  Theorem 4.40. Let (S, O) be a topological space that satisfies the first axiom of countability. For T ⊆ S we have x ∈ K(T ) if and only if there exists a sequence (xn ) in T such that limn→∞ xn = x. Proof. Let x ∈ T and let {Vn | n ∈ N} be a local basis of neighborhoods  at x. Note that if Un = kn Vn , then {Un | n ∈ N} is again a local basis

May 2, 2018 11:28

188

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 188

Mathematical Analysis for Machine Learning and Data Mining

of neighborhoods at x and U0 ⊇ U1 · · · ⊇ Un ⊇ · · · and since each set Un ∩T is non-empty, there exists xn ∈ Un ∩ T . The sequence (xn ) converges to x. Conversely, suppose that (xn ) is a sequence contained in T and limn→∞ xn = x. Then, for every neighborhood V ∈ neighx (O) there exists  a number nV such that n  nV implies xn ∈ V , so x ∈ K(T ). Corollary 4.8. Let (S, O) be a topological space that satisfies the first axiom of countability. A subset U of S is closed if and only if for every sequence (xn ) in U that converges to x we have x ∈ U . Proof. Suppose that U is a closed set of (S, O) and x ∈ U . Since U = K(U ), by Theorem 4.40, there exists a sequence (xn ) in U such that limn→∞ xn = x. Conversely, if for every sequence (xn ) in U that converges to x we have x ∈ U , it follows that K(U ) ⊆ U , so K(U ) = U , so U is closed.  Corollary 4.9. Let (S, O) be a topological space that satisfies the first axiom of countability. A subset W of S is open if and only if for every x ∈ W and sequence (xn ) that converges to x there exists a number nW ∈ N such that n  nW implies xn ∈ W . Proof. Let W be an open subset in S and let x ∈ W . If (xn ) is a sequence that converges to x, since W is a neighborhood of x, the existence on nW follows immediately. Conversely, suppose that for every x ∈ W and sequence (xn ) that converges to x there exists a number nW ∈ N such that n  nW implies xn ∈ W . (yn ) be a sequence in S − W that converges to y ∈ S. Note that y cannot be an element of W because this would imply that the existence of a number n such that for n  n we would have yn ∈ W . Therefore, y ∈ S − W , so S − W is closed, which means that W is open.  Theorem 4.40 and Corollaries 4.8 and 4.9 show that sequences as adequate for describing topological spaces that satisfy the first countability axiom. However, they fail to provide descriptions for general topological spaces. A generalization of sequences that overcomes this limitation is described in Section 4.10.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Topology

4.6

9in x 6in

b3234-main

page 189

189

Compactness

Definition 4.20. Let (S, O) be a topological space. A subset T of S is  compact if for every collection C of open sets such that T ⊆ C there exists  a finite subcollection C of C such that T ⊆ C . The topological space (S, O) is compact if S itself is compact. The collection of compact subsets of a topological space (S, O) will de denoted by COMP(S, O). Example 4.18. Let (S, O) be a topological space. For x ∈ S, the singleton  {x} is a compact set for if x ∈ C, there exists a set C ∈ C such that {x} ⊆ C. Similarly, any finite subset T of S is compact. If C is a cover for a set T and C ⊆ C is a subcollection of C such that  T ⊆ C , we say that C is a subcover of T . The compactness can also be characterized in terms of neighborhoods. Theorem 4.41. Let (S, O) be a topological space and let T be a subset of S. T is compact if for any collection of neighborhoods N = {Ux | x ∈ T and Ux ∈ neighx (O)} there exists a finite subcollection of N that is a cover of T . Proof. Since every open set that contains a point x is a neighborhood of x, the condition of the theorem is clearly sufficient. Conversely, if T is compact and N is a cover of T by neighborhoods, N = {Ux | x ∈ T and Ux ∈ neighx (O)}, let Vx be an open set such that x ∈ Vx ⊆ Ux . Then {Vx | x ∈ T } is an open cover of T . The compactness of T implies the existence of a finite open cover of T , {Vx1 , . . . , Vxn }. This, in turn, implies that {Ux1 , . . . , Uxn } is a finite collection of neighborhoods contained in N that is a cover for T .  Theorem 4.42. Let (S, O) be a topological space and let T be a subset of S. T ∈ COMP(S, O) if and only if the subspace (T, O T ) is a compact space. Proof.

The argument is immediate.



Example 4.19. Every topological space (S, O) where S is a finite set is compact. If (S, O) is a topological space such that S is infinite set and O = P(S), then (S, O) is not compact because the collection {{x} | x ∈ S} is a finite cover of S that contains no finite subcover.

May 2, 2018 11:28

190

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 190

Mathematical Analysis for Machine Learning and Data Mining

Definition 4.21. A subset U of a topological space (S, O) is σ-compact if  there exists a sequence of compact sets (Cn ) in S such that U = n∈N Cn . Another useful concept is the notion of a family of sets with the finite intersection property. Definition 4.22. A collection C of subsets of a set S has the finite in tersection property (f.i.p.) if D = ∅ for every finite subcollection D of C. Theorem 4.43. The following three statements concerning a topological space (S, O) are equivalent: (i) (S, O) is compact;  (ii) if D is a family of closed subsets of S such that D = ∅, then  there exists a finite subfamily D0 of D such that D0 = ∅;  (iii) if E is a family of closed sets having the f.i.p., then E = ∅. Proof.

The argument is left to the reader.



Another characterization of compactness that is just a variant of part (iii) of Theorem 4.43 that applies to an arbitrary family of sets (not necessarily closed) is given next. Theorem 4.44. A topological space (S, O) is compact if and only if for  every family of subsets C that has the f.i.p. we have {K(C) | C ∈ C} = ∅. Proof. If for every family of subsets C that has the f.i.p. we have  {K(C) | C ∈ C} = ∅, then, in particular, if C consists of closed sets,  it follows that {C | C ∈ C} = ∅, which amounts to part (iii) of Theorem 4.43, so (S, O) is compact. Conversely, suppose that the space (S, O) is compact, which means that the property of part (iii) of Theorem 4.43 holds. Suppose that C is an arbitrary collection of subsets of S that has the f.i.p. Then, the collection of closed subsets {K(C) | C ∈ C} also has the f.i.p. because C ∈ K(C) for  every C ∈ C. Therefore, {K(C) | C ∈ C} = ∅.  Example 4.20. Let U1 ⊇ U2 ⊇ · · · be a descending sequence of non empty closed subsets of a compact space (S, O). Its intersection n1 Un  is non-empty because (S, O) is compact and kp=1 Uip = Ul = ∅, where l = min{i1 , . . . , ik } for every k  1.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Topology

9in x 6in

b3234-main

page 191

191

Example 4.21. Every closed interval [x, y] of R is a compact set. Indeed, if C is an open cover of [x, y] we can assume without loss of generality that C consists of open intervals C = {(ai , bi ) | i ∈ I}. Let ⎫ ⎧  ⎬ ⎨   (aj , bj ) for some finite J ⊆ I . K = c c ∈ [x, y] and [x, c] ⊆ ⎭ ⎩  j∈J Observe that K = ∅ because x ∈ K. Indeed, we have [x, x] = {x} and therefore [x, x] ⊆ (ai , bi ) for some i ∈ I. We claim that y  w = sup K. It is clear that w  y because y is an upper bound of [x, y] and therefore an upper bound of K. Suppose that w < y. Note that in this case there exists an open interval (ap , bp ) for some p ∈ I such that w ∈ (ap , bp ). By Theorem 1.16, for every > 0, there is z ∈ K such that sup K − < z. Choose such that < w − ap . Since the closed interval [x, z] is covered by a finite collection of open intervals [x, z] ⊆ (aj1 , bj1 )∪· · · ∪(ajr , bjr ), it follows that the interval [x, w] is covered by (aj1 , bj1 ) ∪ · · · ∪ (ajr , bjr ) ∪ (ap , bp ). This leads to a contradiction because the open interval (ap , bp ) contains numbers in K that are greater than w. So we have w = y, which shows that [x, y] can be covered by a finite family of open intervals extracted from C. In Example 4.21, we saw that every closed interval [a, b] of R is a compact set. This allows us to prove the next statement. Theorem 4.45. (Bolzano2 -Weierstrass Theorem) A bounded sequence of real numbers has a convergent subsequence. Proof. Let x = (x0 , . . . , xn , . . .) be a bounded sequence of real numbers. The boundedness of x implies the existence of a closed interval D0 = [a0 , b0 ] such that {xn | n ∈ N} ⊆ [a0 , b0 ]. 0 be the midpoint of D0 . At least one of the sets Let c = a0 +b 2 −1 −1 x ([a0 , c0 ]), x ([c0 , b0 ]) is infinite. Let [a1 , b1 ] be one of [a0 , c0 ] or [c0 , b0 ], for which x−1 ([a0 , c0 ]), x−1 ([c0 , b0 ]) is infinite. 2 Bernard P. J. N. Bolzano was born on October 5th 1781 in Prague and died in the same city on December 18th 1848. He was a mathematician, philosopher and theologian. Bolzano studied at the University of Prague, where he studied mathematics and philosophy and later, theology. Bolzano taught religion and philosophy. After his dismissal from the University in 1819 by the Austrian government due to his pacifist convictions, he spent his exile writing on philosophical and mathematical matters. Bolzano contributed to the foundation of mathematical analysis.

May 2, 2018 11:28

192

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 192

Mathematical Analysis for Machine Learning and Data Mining

Suppose that we have constructed the interval Dn = [an , bn ] having n as its midpoint such that x−1 (Dn ) is infinite. Then, Dn+1 = cn = an +b 2 [an+1 , bn+1 ] is obtained from Dn as one of the intervals [an , cn ] or [cn , bn ] that contains xn for infinitely many n. Thus, we obtain a descending sequence of closed intervals [a0 , b0 ] ⊃ [a1 , b1 ] ⊃ · · · such that each interval contains an infinite set of members  of the sequence x. By Theorem 4.39, we have [a, b] = n∈N [an , bn ], where 0 , so a = a = limn→∞ an and b = limn→∞ bn . Note that bn − an = b02−a n limn→∞ an = limn→∞ bn = b. The interval D0 contains at least one member of x, say xn0 . Since D1 contains infinitely many members of x, there exists a member xn1 of x such that n1 > n0 . Continuing in this manner, we obtain a subsequence xn0 , xn1 , . . . , xnp , . . .. Since ap  xnp  bp , it follows that the sequence  (xn0 , xn1 , . . . , xnp , . . .) converges to a. Example 4.22. The topological space (R, O) is not compact because  n1 [n, ∞) = ∅. Example 4.23. Let Rn = R × · · · × R, where the product involves n copies of R and n  1. In Example 4.13, we saw that the collection of open intervals {(a, b) | a, b ∈ R and a < b} is a basis for the topological space (R, O). Therefore, a basis of the topological space (Rn , O × · · · O) consists of parallelepipeds of the form (a1 , b1 ) × · · · × (an , bn ), where ai < bi for 1  i  n. This topological space will be denoted by (Rn , On ). Example 4.24. The open interval (0, 1)( is not compact. Indeed, it is easy ) 1 1 to see that the collection of open sets , 1 − is an open cover of n n (0, 1). However, no finite subcollection of this collection of sets is an open cover of (0, 1). Example 4.24 suggests the interest of the following definition. Definition 4.23. A subset T of a topological space (S, O) is relatively compact if K(T ) ∈ COMP(S, O). Example 4.25. The set (0, 1) is a relatively compact subset of R but not a compact one. Theorem 4.46. If (S, O) is a compact topological space, any closed subset T of S is compact. Proof. Let T be a closed subset of (S, O). We need to show that the subspace (T, O T ) is compact. Let C be an open cover of (T, O T ). Then,

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Topology

9in x 6in

b3234-main

page 193

193

C ∪ {S − T } is a open cover of (S, O). The compactness of (S, O) means that there exists a finite subcover D of (S, O) such that D ⊆ C ∪ {S − T }. It follows immediately that D − {S − T } is a finite subcover of C  for (T, O T ). Corollary 4.10. Let (S, O) be a topological space. If U is a compact set and V is a closed set in (S, O), then U ∩ V is a compact set. Proof. Note that the set U ∩ V is a closed subset of U relative to the subspace determined by U . Therefore, U ∩ V is compact in this subspace by Theorem 4.46, and therefore, it is compact in (S, O).  Bolzano-Weierstrass3 property can now be formulated in a more general context. Theorem 4.47. (Bolzano-Weierstrass Theorem) If (S, O) is a compact topological space, then, for every infinite subset U of S we have U  = ∅ (the Bolzano-Weierstrass property). Proof. Let U = {xi | i ∈ I} be an infinite subset of S. Suppose that U has no accumulation point. For every s ∈ S, there is an open set Ls such that s ∈ Ls and U ∩ (Ls − {s}) = ∅. Clearly the collection {Ls | s ∈ S} is an open cover of S, so it contains a finite subcover {Ls1 , . . . , Lsp }. Thus, S = Ls1 ∪ · · · ∪ Lsp . Note that each Lsi contains at most one element of U (which happens when si ∈ U ), which implies that U is finite. This  contradiction means that U  = ∅. 4.7

Separation Hierarchy

Separation properties of topological spaces aim to introduce conditions that ensure that it is possible to distinguish topologically between points or sets. The classes of topological spaces that we are about to present are named Ti after the German word “Trennungsaxiom” (or separation axiom), where i ∈ {0, 1, 2, 3, 4}. 3 Karl T. W. Weierstrass was born on October 31st 1815 in Ostenfelde, a Westfalian city and died on February 19th 1897 in Berlin. He studied mathematics in Bonn, then on his own, and at the Academy of M¨ unster, and taught high school. Later he taught at the Technical Institute in Berlin (a precursor of the Technical University) and at the University of Berlin. Weierstrass is known as the founder of modern analysis. He formalized the notion of continuous function, obtained convergence criteria for series and contributed to the theory of several classes of functions, and the calculus of variations.

May 2, 2018 11:28

194

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 194

Mathematical Analysis for Machine Learning and Data Mining

Definition 4.24. Let (S, O) be a topological space and let x and y be two arbitrary, distinct elements of S. This topological space is: (i) a T0 space if there exists U ∈ O such that one member of the set {x, y} belongs to U and the other to S − U ; (ii) a T1 space if there exist U, V ∈ O such that x ∈ U − V and y ∈ V − U; (iii) a T2 space or a Hausdorff space if there exist U, V ∈ O such that x ∈ U and y ∈ V and U ∩ V = ∅; (iv) a T3 space if for every closed set H and x ∈ S − H there exist U, V ∈ O such that x ∈ U and H ⊆ V and U ∩ V = ∅; (v) a T4 space if for all disjoint closed sets H, L there exist two open sets U, V ∈ O such that H ⊆ U , L ⊆ V , and U ∩ V = ∅. Example 4.26. It is clear that every T1 space is also a T0 space; however, there exists T0 spaces that are not T1 spaces. For instance, if S = {a, b} and O = {∅, {a}, {a, b}}, then (S, O) is a T0 space but not a T1 space. Theorem 4.48. A topological space (S, O) is a T1 space if and only if every singleton {x} is a closed set. Proof. Suppose that (S, O) is a T1 , space and for every y ∈ S − {x} let Uy and Vy be two open sets such as x ∈ Uy − Vy and y ∈ Vy − Uy . Then,    x ∈ y =x Uy and x ∈ y =x Vy , so y ∈ y =x Vy ⊆ S − {x}. Thus, S − {x} is an open set, so {x} is closed. Conversely, suppose that each singleton {u} is closed. Let x, y ∈ S be two distinct elements of S. Note that the sets S − {x} and S − {y} are open and x ∈ (S − {y}) − (S − {x}) and y ∈ (S − {x}) − (S − {y}), which  shows that (S, O) is a T1 -space. Theorem 4.49. (S, O) is a Hausdorff space if and only if for x, y ∈ S such that x = y, there exist U ∈ neighx (O) and V ∈ neighy (O) such that U ∩ V = ∅. Proof.

This statement is a direct consequence of Definition 4.24.



Theorem 4.50. Let (S, O) be a T4 -separated topological space. If H is a closed set and L is an open set such that H ⊆ L, then there exists an open set U such that H ⊆ U ⊆ K(U ) ⊆ L. Proof. Observe that H and S − L are two disjoint closed sets under the assumptions of the theorem. Since (S, O) is a T4 -separated topological

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Topology

b3234-main

page 195

195

space, there exist U, V ∈ O such that H ⊆ U , S − L ⊆ V and U ∩ V = ∅. This implies U ⊆ S − V ⊆ L. Since S − V is closed, we have H ⊆ U ⊆ K(U ) ⊆ K(S − V ) = S − V ⊆ L, which proves that U satisfies the conditions of the theorem.



Theorem 4.51 is in some sense a reciprocal result of Theorem 4.46, which holds in the realm of Hausdorff spaces. We need a preliminary result. Lemma 4.1. Let (S, O) be a Hausdorff topological space. If H is a compact subset of S and x ∈ H, there exist disjoint open subsets U , V of S such that x ∈ U and H ⊆ V . Proof. Since (S, O) is a Hausdorff space, for every y ∈ H there exist open neighborhoods Uy , Vy of x and y, respectively such that Uy ∩ Vy = ∅. Since  H ⊆ y∈H Vy and H is compact, it follows that there exists a finite subset   {y1 , . . . , yn } of H such that H ⊆ nk=1 Vyk . Note that nk=1 Vyk is am open n set that contains H. If Uy = k=1 Uyk , then Uy is an open neighborhood n  of x that contains x and Uy ∩ ( k=1 Vyk ) = ∅. By Lemma 4.1, if H is a compact subset of (S, O), then for every x ∈ S − H there exists an open set U with x ∈ U ⊆ S − H. Theorem 4.51. Each compact subset H of a Hausdorff space (S, O) is closed. Furthermore, if G ⊆ H, then G is compact if and only if G is a closed set in (S, O). Proof. By Lemma 4.1 the complement of H can be written as a union of open sets:  S − H = {Dx | x ∈ S − H}, so S − H is an open set, which implies that H is closed. If G is a closed set, then G is compact by Theorem 4.46. Conversely, if G is compact, then by the first part of this theorem, G is closed.  Corollary 4.11. In a Hausdorff space (S, O), each finite subset is a closed set. Proof. Since every finite subset of S is compact, the statement follows immediately from Theorem 4.51.  Theorem 4.52. Let (S, O) be a Hausdorff space and let U, W be disjoint compact subsets of S. There exist disjoint open subsets V1 , V2 of S such that U ⊆ V1 and W ⊆ V2 .

May 2, 2018 11:28

196

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 196

Mathematical Analysis for Machine Learning and Data Mining

Proof. Let x ∈ U . By Lemma 4.1 there exist disjoint open subsets Ux , Vx of S such that x ∈ Ux and W ⊆ Vx . Since U is compact, there exists n n a finite set {x1 , . . . , xn } such that U ⊆ i=1 Uxi . Define V1 = i=1 Uxi . n Let V2 = i=1 Vxi . Then, V1 and V2 are open sets, U ⊆ V1 , W ⊆ V2 and  V1 ∩ V2 = ∅. It is clear that every T2 space is a T1 space and each T1 space is a T0 space. However, this hierarchy does not hold beyond T2 . This requires the introduction of two further classes of topological spaces. Definition 4.25. A topological space (S, O) is regular if it is both a T1 and a T3 space; (S, O) is normal if it is both a T1 and a T4 space. Theorem 4.53. Every normal topological space is a regular one and every regular topological space is a T2 space. Proof. We leave the first part of the theorem to the reader. Let (S, O) be a topological space that is regular and let x and y be two distinct points in S. By Theorem 4.48, the singleton {y} is a closed set. Since (S, O) is a T3 , space, two open sets U and V exist such that x ∈ U ,  {y} ⊆ V , and U ∩ V = ∅, so (S, O) is a T2 space. Theorem 4.54. Every compact Hausdorff space (S, O) is normal. Proof. To prove that (S, O) is normal we need to show that it is a T4 space, that is, for all disjoint closed sets A, B there exist two disjoint open sets U, V ∈ O such that A ⊆ U and B ⊆ V . Suppose initially that B = {b}. For every a ∈ A we have the disjoint open sets Ua and Va such that a ∈ Ua and b ∈ Va . Since A is compact  and A ⊆ a∈A Ua , there exists a finite set {a1 , . . . , an } such that A ⊆ Ua1 ∪ · · · ∪ Uan . The statement is proven for this special basis (B = {b}) where U = Ua1 ∪ · · · ∪ Uan and V = Va1 ∩ · · · ∩ Van . For every b ∈ B, by the previous argument, there exist two disjoint open sets Ub and Vb such that A ⊆ Ub and b ∈ Vb . The set B is compact and B ⊆  {Vb | b ∈ B}, so there exist b1 , . . . , bn such that B ⊆ Vb1 ∪ · · · ∪ Vbn . The argument is completed by taking U = Ub1 ∩· · ·∩Ubn and V = Vb1 ∪· · ·∪Vbn .  Theorem 4.55. A compact Hausdorff space (S, O) is a Baire space. Proof. Let L1 , . . . , Ln , . . . be a sequence of dense open subsets of S. By  applying Theorem 4.9 we shall prove that n1 Ln is dense in S by showing that the intersection of this set with an arbitrary non-empty open set U is non-empty.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Topology

9in x 6in

b3234-main

page 197

197

We construct inductively a sequence of non-empty open sets U0 , U1 , . . . , Un , . . ., where U0 = U such that K(Un ) ⊆ Ln ∩Un−1 and K(Un ) is compact. Suppose that the non-empty open set Un−1 was constructed. Note that Ln ∩ Un−1 = ∅ because Ln is dense in S. Therefore, there exists a nonempty open set Un such that K(Un ) ⊆ Ln ∩ Un−1 and K(Un ) is compact as a closed subset of a compact space (by Theorem 4.46).  Let H = n1 K(Un ). The compactness implies that H = ∅. We have H ⊆ U and H ⊆ Ln for each n. Therefore, U has a non-empty intersection    with n1 Ln , which implies that n1 Ln is dense in S.

4.8

Locally Compact Spaces

Definition 4.26. A topological space is locally compact if it is a Hausdorff space and every point has a local basis consisting of compact sets. Example 4.27. The topological space (R, O) is not compact, as we saw in Example 4.22. However, it is locally compact because every x0 ∈ R since the family of sets {[x0 − a, x0 + a] | a ∈ R} is a local basis of compact neighborhoods. Similarly, (Rn , O) is locally compact. The next theorem states that every Hausdorff compact space is locally compact. Theorem 4.56. Let (S, O) be a Hausdorff space. The following statements are equivalent: (i) (S, O) is locally compact; (ii) for each U ∈ neighx (O) there exists an open neighborhood V of x such that K(V ) is compact and K(V ) ⊆ U ; (iii) each x ∈ S has a compact neighborhood. Proof. (i) implies (ii): Let x ∈ S and let U ∈ neighx (O). By (i) there exists a compact neighborhood W of x such that W ⊆ U . The set V = I(W ) is an open neighborhood of x. Since W is a compact subset of a Hausdorff space, W is closed. Therefore, K(V ) ⊆ K(W ) = W ⊆ U and K(V ) is compact because it is a closed subset of the compact set W . (ii) implies (iii): This implication is immediate.

May 2, 2018 11:28

198

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 198

Mathematical Analysis for Machine Learning and Data Mining

(iii) implies (i): Let U ∈ neighx (O). By (iii) there exists a compact neighborhood C of x. Consider the open neighborhood W = I(U ∩ C) of x. The compactness of C implies that C is a compact Hausdorff space, the set Y = K(W ) is a subspace of C and, therefore, it is a compact Hausdorff space. In this subspace, Y − W is closed and x ∈ Y − W , so by Lemma 4.1, there are disjoint open sets M, N such that x ∈ M , Y − W ⊆ N . Since x ∈ M ⊆ V ⊆ W , V is a neighborhood of x in Y with V ⊆ U . Since V is a closed set in the compact space Y , it follows that V is compact. Finally, Y is a neighborhood of x in S because V ⊂ W and W is open in S.  Corollary 4.12. A Hausdorff space is locally compact if each of its points has a compact neighborhood. Furthermore, every compact Hausdorff space is locally compact. Proof.

This is an immediate consequence of Theorem 4.56.



Theorem 4.57. Let (S, O) be a locally compact topological space. If W is an open or a closed set in this space, the subspace (W, O W ) is a locally compact space. Proof. Let x ∈ S and let U ∈ neighx (O W ). If W is open, any neighborhood U ∈ neighx (O W ) is also in neighx (O) in S, so there is a compact neighborhood V ∈ neighx (O) with V ⊆ U . Since V ⊆ W , V ∈ neighx (W ). If W is closed, let U0 ∈ neighx (O) such that U = U0 ∩ W . There is some compact neighborhood V0 ∈ neighx (O) with V0 ⊆ U0 . Then V = V0 ∩ W ∈ neighx (O W ) with V ⊆ U . Since W is closed in S, V0 is compact, hence V is compact.  Theorem 4.58. If a topological space (S, O) is locally compact then every neighborhood U ∈ neighx (O) of a point x includes a compact neighborhood of x. Proof. Let T be an open neighborhood of x and let W be a compact neighborhood of x. If W ⊆ T , the statement holds. Suppose, therefore, that W is not included in U and let Z = W ∩T . Since (S, O) is a Hausdorff space there exist Uz ∈ neighz (O) for each z ∈ Z and Wz , an open neighborhood of x satisfying Wz ⊆ W and Uz ∩ Wz = ∅. Since Z = W ∩ T is compact  (by Corollary 4.10), there exist z1 , . . . , zk ∈ Z such that Z ⊆ ki=1 Uzi . Let   V = ki=1 Wzi and U = ki=1 Uzi . Note that V is an open neighborhood of x. We claim that K(V ) is compact and included in T .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topology

page 199

199

The inclusion K(V ) ⊆ W implies that K(V ) is compact. Since U, V ∈ O and U ∩ V = ∅, it follows that K(V ) ∩ U = ∅. From K(V ) ∩ T = K(V ) ∩ (W ∩ T ) = K(V ) ∩ A ⊆ K(V ) ∩ U = ∅, it follows that K(V ) ∩ T = ∅, which implies that K(V ) ⊆ T . This shows that K(V ) is a compact neighborhood of x.  Corollary 4.13. Let (S, O) be a Hausdorff space. The following statements are equivalent: (i) (S, O) is locally compact; (ii) for every x ∈ S there exists a closed compact neighborhood of x; (iii) for every x ∈ S there exists a relatively compact neighborhood of x; (iv) every x ∈ S has a local basis of relatively compact neighborhoods. Proof. (i) implies (ii): Since every compact subset in a Hausdorff space is closed, if follows from Theorem 4.58, that every point of a locally compact Hausdorff space has a closed compact neighborhood. (ii) implies (iii): This implication is immediate because every closed compact set is a relatively compact set. (iii) implies (iv): By Theorem 4.58 every neighborhood of x includes a relatively compact neighborhood of x; therefore, the collection of relatively compact neighborhoods of x is a local basis at x. (iv) implies (i): Let U be a relatively compact neighborhood of an arbitrary x ∈ S. The closure K(U ) is a compact neighborhood of x, so (S, O) is locally compact.  Lemma 4.2. Let (S, O) be a locally compact space, D be an open set, and let x ∈ D. There exists an open set E such that K(E) is compact and x ∈ E ⊆ K(E) ⊆ D. Proof. Let T be a compact neighborhood of x. By Theorem 4.51 the set T is closed; furthermore, by Theorem 4.54, T is normal. Note that the sets {x} and T − D are closed in the subspace T . The normality of T implies the existence of the disjoint open sets U and V (in the topology of the subspace T ) such that x ∈ U and T − D ⊆ V . By the definition of the topology of the subspace T there exist the open sets U0 , V0 ∈ O such that U = U0 ∩ T and V = V0 ∩ T . Define the open set E as E = I(U ). We have x ∈ E. Since E ⊆ U ⊆ T , it follows that K(E) ⊆ K(T ) = T , which implies that E is compact. Finally, we have E ∩ V0 ⊆ U ∩ V0 = (T ∩ U0 ∩ V0 = U ∩ V = ∅,

May 2, 2018 11:28

200

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 200

Mathematical Analysis for Machine Learning and Data Mining

so E ⊆ X − V0 . Since S − V0 is closed, we also have K(E) ⊆ S − V0 . Since T − D ⊆ V ⊆ V0 , we obtain K(E) ⊆ T ∩ (S − V0 ) ⊆ T ∩ (S − (T − D)) = T ∩ D ⊆ D.



Theorem 4.59. Let (S, O) be a locally compact space, D be an open set, and let C be a compact set such that C ⊆ D. There exists an open set E such that K(E) is compact and C ⊆ E ⊆ K(E) ⊆ D. Proof. By Lemma 4.2, for every x ∈ C there exists an open set Ex such  that x ∈ Ex ⊆ K(Ex ) ⊆ D, where K(Ex ) is compact. Since C ⊆ x∈K Ex and C is compact, there exists a finite set {x1 , . . . , xn } such that C ⊆ n n j=1 Exj . If E is defined as E = j=1 Exj , then C ⊆ E ⊆ K(E) ⊆ K(Ex1 ) ∪ · · · ∪ K(Exn ) ⊆ D, 

which concludes the argument.

Theorem 4.60. Let (T, O) be a non-compact Hausdorff space that is locally compact, and let s0 ∈ T . Let O be the collection of subsets of the set S = T ∪ {s0 } that consists of: (i) the sets in O, and (ii) the sets of the form {s0 } ∪ (T − C), where C is a compact subset of T . The pair (S, O ) is compact Hausdorff space. Proof. Note that Let C be a collection of subsets of O . The definition of C allows us to write C = C1 ∪ C2 , where C1 ⊆ O and C2 = {{s0 } ∪ (T − Ci ), Ci is a compact subset of T, i ∈ I}.  It is clear that C1 is an open set in T . Also,   C2 = {s0 } ∪ (T − Ci ) i∈

= {s0 } ∪

 T−



 Ci

.

i∈

The set Ci are closed in (T, O) as in T as compact subsets of a Hausdorff  space (by Theorem 4.51), so the set C˜ = T − i∈ Ci is open in T . Thus,   ˜ C1 ∪ {s0 } ∪ C, C =  which proves that C ∈ O .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topology

page 201

201

Let U, V two sets in O . If U, V ∈ O, then U ∩ V ∈ O ⊆ O . Suppose now that U ∈ O and V = {s0 } ∪ (T − C), where C is a compact subset of T . As above, T − C ∈ O, so U ∩ V = U ∩ ({s0 } ∪ (T − C))} = U ∩ (T − V ) ∈ O because s0 ∈ T . Finally, if U = {s0 } ∪ (T − C1 ) and V = {s0 } ∪ (T − C2 ), where C1 , C2 are compact subsets of T , we have U ∩ V = ({s0 } ∪ (T − C1 )) ∩ ({s0 } ∪ (T − C2 )) = {s0 } ∪ ((T − C1 ) ∩ (T − C2 )) ∈ O . This argument can be extended immediately to a finite collection of subsets of O , so O is indeed a topology. To prove that (S, O ) is a Hausdorff space let x, y ∈ S be two distinct points in S. If both x, y ∈ T , then it is obviously that the Hausdorff separation is satisfied. Suppose that x ∈ T and y = s0 . Since (T, O) is a Hausdorff locally compact space there exists a neighborhood W of x in T whose closure K(W ) is compact. The set {s0 } ∪ (T − K(W )) is an open set that contains s0 and has an empty intersection with W , so (S, O ) is indeed a Hausdorff space. To prove the compactness of (S, O ) consider an arbitrary open covering {Ui | ∈ I} of S and let Ui0 be an open set such that s0 ∈ Ui0 . The set T − Ui0 is a compact subset of T . Therefore, there exists a finite covering  Ui1 , . . . , Uim of T − Ui0 , so Ui0 , Ui1 , . . . , Uim is a finite covering of S. The topological space (S, O ) is known as the Alexandrov compactification of (T, O). Note that the mapping κ : T −→ S given by κ(x) = x for x ∈ T is an injection of T in S. 4.9

Limits of Functions

Definition 4.27. Let S be a set. A non-empty collection F of non-empty subsets of S is a filter on S if U, V ∈ F implies U ∩ V ∈ F and T ∈ F and T ⊆ W implies W ∈ F. A non-empty collection B of non-empty subsets of S is a filter basis on S if U, V ∈ B implies U ∩ V ∈ B. A filter sub-basis is a non-empty collection S of non-empty subsets of S such that the intersection of any finite subcollection of S is non-empty. A filter is a filter base; the converse is not true. Note that the set of neighborhoods neighx (O) of a point x of a topological space is a filter on S and a local basis at x is a filter basis.

May 2, 2018 11:28

202

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 202

Mathematical Analysis for Machine Learning and Data Mining

Example 4.28. Let (R, O) be the set of reals equipped with the usual topology. The following collections of sets are filter bases: (i) the collection {(x0 − δ, x0 + δ) | δ > 0}; (ii) the collection {[x0 , x0 + δ) | δ > 0}; (iii) the collection {(x0 , x0 + δ) | δ > 0}; (iv) the collection {(x0 − δ, x0 ] | δ > 0}; (v) the collection {(x0 − δ, x0 ) | δ > 0}. The collections {[t, ∞) | t ∈ R} and {(−∞, t] | t ∈ R} are also filter bases. Definition 4.28. Let (T, O) be a topological space, B a filter base on a set S and let f : S −→ T be a function. A function f : S −→ T tends to ∈ T along B if for every V ∈ neigh (O) there exists B ∈ B such that f (B) ⊆ V . The element of T is the limit of f along the filter base B. Example 4.29. Suppose that (S, O ) is a topological space and that B is the filter base B = neighx0 (O ). If (T, O) is another topological space and f : S −→ T , then the limit of f along neighx0 (O ) is if for every V ∈ neigh (O) there exists W ∈ neighx0 (O ) such that x ∈ W implies f (x) ∈ V . This is denoted as limx→x0 f (x) = . Example 4.30. Suppose that S = T = R and O and O are the usual topologies on R. If B is the collection B = {(x0 − δ, x0 + δ) | δ > 0}, then is the limit along B if, for every > 0, there exists δ > 0 such that f (x0 − δ, x0 + δ) ⊆ ( − , + ). This is denoted as limx→x0 f (x) = . If B = {(x0 , x0 +δ) | δ > 0}, then f has the limit along this filter base, if for every > 0, there exists δ > 0 such that f (x0 , x0 + δ) ⊆ ( − , + ). This is denoted as limx→x0 + f (x) = and we say that f has the limit in x0 from the right. Similarly, if B = {(x0 − δ, x0 ) | δ > 0}, then f has the limit along this filter base, if for every > 0, there exists δ > 0 such that f (x0 + δ, x0 ) ⊆ ( − , + ). This is denoted as limx→x0 − f (x) = . In this case, is the limit of f in x0 from the left. We have limx→x0 f (x) = if and only if limx→x0 + f (x) = limx→x0 − f (x) = . The limits from the left or right of f are collectively known as lateral limits. Definition 4.29. Let f : X −→ R be a function, where X ⊆ R. If f is not continuous in x0 (where x0 ∈ X), we say that f is discontinuous in x0 .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topology

page 203

203

A first type discontinuity point is a discontinuity point x0 such that f has lateral limits in x0 such that at least one of them is distinct from f (x0 ). A discontinuity point of the second type is a discontinuity point for f that is not of the first kind. The jump of a function in a discontinuity point x0 is the number jump(f, x0 ) = lim f (x) − lim f (x). x→x0 +

x→x0 −

Theorem 4.61. Let f : X −→ R be a monotonic function, where X ⊆ R. All its discontinuity points are of the first kind. Proof. It is clear that a monotonic function has lateral limits in every point of its definition domain. We need to prove that these lateral limits are finite. Suppose that f is increasing. Let u, v ∈ X such that u < x < v. By monotonicity we have f (u)  f (x)  f (v). This implies f (u)  limt→x− f (t)  limt→x+ f (t)  f (v). The argument for decreasing functions is similar.  Theorem 4.62. The set of discontinuity points of a monotonic function is at most countable. Proof. Suppose initially that f : X −→ R, where X = [a, b], and f is increasing. If a < c < b we have f (a)  lim f (x) x→a+

 lim f (x)  lim f (x) x→c−

x→c+

 lim f (x)  f (b), x→b−

so the jump of f in a discontinuity point of the first kind c is less than f (b) − f (a). Let c1 < c2 < · · · < cn be n discontinuity points of f in (a, b) such that jump(f, ci )  α, where α > 0. We have f (b) − f (a)  lim f (x) − lim f (x) x→cn +

= 

n  i=1 n  i=1

x→c1 −

jump(f, xi ) +

n−1  i=1

jump(f, xi )  nα.

lim

x→xi+1 −

 f (x) − lim f (x) x→xi +

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 204

Mathematical Analysis for Machine Learning and Data Mining

204

(a) Thus, n  f (b)−f . Therefore, the number of discontinuity points where n the jump is larger than α is finite. Let C1 be the set of discontinuities of f where the jump is at least equal to 1 and for n  2 let   *  1 1  Cn = c c is a discontinuity for f and  jump(f, c)  . n n−1  Each set Cn is finite, so their union C = n1 Sn is countable as a countable union of finite sets. If I is not a closed interval bound and closed, then I can be written as a countable union of closed and bounded intervals, and the same conclusion can be easily reached. 

4.10

Nets

The notion of net is a generalization of the notion of sequence previously introduced. Definition 4.30. A directed set is a partially ordered set (I, ) such that for i, j ∈ I there exists k ∈ I such that i  k and j  k. A subset J of I is cofinal if for every i ∈ I there exists j ∈ J such that j  i. Starting from directed sets we can introduce the notion of net on a set. Definition 4.31. A I-net on a set S is a function ξ : I −→ S. For an I-net we denote ξ(i) ∈ S by xi and the net itself by (xi )i∈I . If the indexing set I is clear from context, the net will be denoted by (xi ). The poset (N, ) is a directed set. Thus, a sequence a set S is just an N-net (xn )n∈N . The notion of subnet is a generalization of the notion of subsequence. Definition 4.32. Let (I, ) and (J, ) be two directed partially ordered sets, and let ξ = (xi )i∈I , ζ = (zj )j∈J be two nets. The net J-net ζ is a subnet of the I-net ξ if there exists a function h : J −→ I such that (i) ζ = ξh, that is, zj = xh(j) for j ∈ J; (ii) for each i ∈ I there exists ji ∈ J such that j  ji implies h(j)  i. Note that for a strictly increasing function h : N −→ N and each n ∈ N there exists m ∈ N such that n  h(m) because the infinite set

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Topology

9in x 6in

b3234-main

page 205

205

{h( ) | ∈ N} cannot be contained in the finite set {0, . . . , n}. Thus, if x = (xn ) is a sequence in Seq(S) we can regard a subsequence (xh(n) ) of x as a subnet of x, where I = J = N. If (xi ) is an I-net on a poset (S, ), we say that (xi ) is an monotonic I-net if i  j implies xi  xj for i, j ∈ I; if i  j implies xi  xj for i, j ∈ I, then the net is an anti-monotonic I-net. Example 4.31. Let (S, O) be a topological space. The set of neighborhoods of x ∈ S, neighx (O) is a directed poset ordered by the partial order relation ⊇. Indeed, if U, V are two neighborhoods of x, then U ∩ V is also a neighborhood of x, U ⊇ U ∩ V and V ⊇ U ∩ V . Thus, we may consider nets of the form ξ : neighx (O) −→ S indexed neighx (O) such that ξ(V ) ∈ V for V ∈ neighx (O). We denote ξ(V ) as xV . The notions introduced next allow us to define net convergence as a generalization of sequence convergence. Definition 4.33. Let U be a subset of a set S and let (xi ) be an I-net on S. (xi ) is eventually in the set U if there exists iU ∈ I such that i  iU implies xi ∈ U . (xi ) is frequently in U if for each i ∈ I there exists a j ∈ I such that j  i and xj ∈ U . Theorem 4.63. If an I-net ξ is eventually in a set U , then any subnet ζ of ξ is eventually in U . Proof. Suppose that ζ = (zj )j∈J is a J-subnet of I-net (xi )i∈I determined by the function h : J −→ I. We have zj = xh(j) for every j ∈ J. Since ξ is eventually in U , there exists iU such that iU  i implies xi ∈ U . By Definition 4.32 there exists jU such that j  jU implies  h(j)  iU , so zj = xh(j) ∈ U . Thus, (zj ) is eventually in U . Definition 4.34. Let (S, O) be a topological space. An I-net (xi ) on S converges to x if for each V ∈ neighx (O) the net is eventually in V . In this case we say that x is a limit of the net (xi ) and we write either xi → x or limi∈I xi = x. A net (xi ) is convergent if it converges to some x ∈ S. A net (xi ) clusters at x ∈ S when it is frequently in each neighborhood V ∈ neighx (O).

May 2, 2018 11:28

206

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 206

Mathematical Analysis for Machine Learning and Data Mining

In other words, the net (xi ) clusters at x if for each V ∈ neighx (O) and i ∈ I exists a j ∈ I such that j  i and xj ∈ V . Theorem 4.64. Let (S, O) be a topological space and let T be a subset of S. A point t is an accumulation point of T if and only if there exists an I-net (xi ) in T − {t} such that xi → t. Proof. We saw that if t is an accumulation point of the subset T of the topological space (S, O) and U ∈ neight (O), then T ∩ (U − {t}) = ∅. Let (xU ) be a net such that xU ∈ T ∩ (U − {t}). Clearly, the net (xU ) converges to t. Conversely, suppose that a net (xi ) in T − {t} converges to t, then for every V ∈ neight (O) the net is eventually in V and the set T −{t} intersects each neighborhood V , which means that t is an accumulation point of T .  A helpful characterization of net convergence is given next. Theorem 4.65. Let (S, O) be a topological space having B as a basis. If (xi ) is an I-net on S we have xi −→ x if and only if for every B ∈ B such that x ∈ B the net is eventually in B. Proof. Suppose that xi → x. If B ∈ B and x ∈ B it is clear that B is a neighborhood of x, so (xi ) is eventually in B. Conversely, suppose that for every set B of the basis that contains x, (xi ) is eventually in B and let V ∈ neighx (O). There is an open set U such that x ∈ U ⊆ V and, therefore, there exists a set B ∈ B such that  x ∈ B ⊆ U ⊆ V . This shows that (xi ) is eventually in V , so xi → x. ˆ always have limits. Example 4.32. Monotonic or anti-monotonic nets on R ˆ We claim that xi → sup{xi | i ∈ Let (xi ) be a monotonic I-net on R. I}. Indeed, if s = sup{xi | i ∈ I} < ∞, then for > 0 there exists i0 ∈ I such that s −  xi0  s. Therefore, by the monotonicity of (xi )i∈I , i0  i implies s −  xi  s, which shows that limI xi = s. If s = ∞, for every r ∈ R the net (xi ) is eventually in (r, ∞), so again xi → ∞, which justifies the claim. Similarly, if (xi ) is an anti-monotonic net, xi → inf{xi | i ∈ I}. Example 4.33. Let (xn ) be an N-net in a topological space. We have xn → x if for every V ∈ neighx (O) there exists n0 such that n ≥ n0 implies xn ∈ V .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topology

page 207

207

Example 4.34. Let (S, O) be a topological space and let (xV )V ∈neighx (O) be the net introduced in Example 4.31. We have xV → x. Indeed, there exists V ∈ neighx (O) such that for every V  ⊆ V we have xV  ∈ V  ⊆ V . Theorem 4.66. Let (S, O) be a topological space and let T be a subset of S. We have x ∈ K(T ) if and only if some net in T converges to x. Proof. Suppose that there is an I-net (xi ) with xi → x. If V ∈ neighx (O), there exists i ∈ I such that xj ∈ V for all j with j  i. Since i  i, we have xi ∈ V ∩ T and, therefore, V ∩ T = ∅. This implies x ∈ K(T ) by Theorem 4.24. Conversely, let x ∈ K(T ). By Theorem 4.24 we have V ∩ T = ∅ for every V ∈ neighx (O). Let xV ∈ V ∩ T . The set (xV )V ∈neighx (O) is a net in T that converges to x.  Corollary 4.14. Let (S, O) be a topological space and let T be a subset of S. T is closed if and only if for each point t such that xi → t for some net (xi ) in T we have t ∈ T . T is open if and only if each net that converges to a point t ∈ T is eventually in T . Proof.

The corollary follows immediately from Theorem 4.66.



Theorem 4.67. Nets in Hausdorff spaces may not converge to more than one point. Proof. Suppose that (xi ) is a net in a Hausdorff space and we have both xi → x and xi → y and x = y. By Theorem 4.49 there exist U ∈ neighx (O) and V ∈ neighy (O) such that U ∩ V = ∅. There exists i ∈ I such that j  i implies xj ∈ U and there exists i ∈ I such that j  i implies xj ∈ U . Since (I, ) is a directed poset there exists k ∈ I such that k  i and k  i which implies xk ∈ U ∩ V . This contradicts the disjointness of U and V .  ˆ such that xi  yi  Theorem 4.68. Let (xi ), (yi ), (zi ) be three I-nets on R zi for i ∈ I. If lim xi = lim zi = , then lim yi = . ˆ Observe that if u, v ∈ V , then Proof. Let V be an open set of in R. [u, v] ⊆ V . Since lim xi = lim zi = , for each open set V that contains

there exist i0 and j0 such that i > i0 implies yi ∈ V and i > j0 implies zi ∈ V . Since (I, ) is a directed poset, there exits k0 ∈ I such that k0  i0 and k0  j0 . Thus, for i  k0 we have both i  i0 and i  j0 . Therefore, if i > k0 we have both xi ∈ V and zj ∈ V , which implies yi ∈ V . This shows  that yi −→ .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 208

Mathematical Analysis for Machine Learning and Data Mining

208

ˆ By extending Example 4.35. Consider now an I-net (xi )i∈I on the set R. ˆ we write limI xi = ∞ if for each a ∈ R the net is Definition 4.34 to R, eventually in (a, ∞). Similarly, limI xi = −∞ if for each a ∈ R the net is eventually in (−∞, a). ˆ For i ∈ I define the set Let (xi )i∈I be an I-net on the set R. Si = {xj | j  i}. It is clear that i  k implies Sk ⊆ Si and, therefore, inf Si  inf Sk  sup Sk  sup Si . Define the I-nets (yi ) and (zi ) as yi = inf Si and zi = sup Si for i ∈ I, so i  k implies yi  yk  zk  zi . Then, (yi ) is a monotonic I-net, while (zi ) is an anti-monotonic I-net. As we observed in Example 4.32, yi → sup{yi | i ∈ I} and zi → inf{zi | i ∈ I}. We define lim inf xi and lim sup xi as lim inf xi = lim yi = sup{yi | i ∈ I} I

= sup{inf Si | i ∈ I}, lim sup xi = lim zi = inf{zi | i ∈ I} I

= inf{sup Si | i ∈ I}. In Example 4.31 we saw that for x0 ∈ S we can define a net (xV ) indexed by neighx0 (O). In this case SV = {xU | V ⊆ U } and we have lim inf xV = sup inf{SV | V ∈ neighx0 (O)}, lim sup xV = inf sup{SV | V ∈ neighx0 (O)}. ˆ we have Theorem 4.69. For every I-net (xi ) on R lim inf xi  lim sup xi . Proof.

The theorem is a direct consequence of Theorem 1.21.



ˆ can be expressed using the sets Si . The convergence of an I-net on R Namely, we have xi → if and only if for every neighborhood V of there exists a set Si that is included in V . ˆ Theorem 4.70. Let (xi ) be an I-net on R. lim sup xi = if and only if lim xi = .

We have lim inf xi =

ˆ and a > lim inf xi , then a cannot be the Proof. Note that if a ∈ R limit of the net (xi ). Indeed, suppose that xi → a and let b such that

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Topology

b3234-main

page 209

209

lim inf xi < b < a. Thus, (b, ∞] is a neighborhood of a. For every i ∈ I there is j such that i < j such that xj < b, and so for no i ∈ I the set Si is contained in the neighborhood (b, ∞] of a, which prevents a from being a limit of (xi ). ˆ and a < lim sup xi then a cannot be the limit of the Similarly, if a ∈ R net xi . This shows that if lim inf xi < lim sup xi the limit of xi cannot exist. Thus, if lim xi exists we must have lim inf xi = lim sup xi . Conversely, suppose that lim inf xi = lim sup xi . Note that for i ∈ I we have yi  xi  zi , where yi = inf Si and zi = sup Si . Since yi → lim inf xi and zi → lim sup xi , the equality lim inf xi = lim sup xi implies xi → lim inf xi = lim sup xi , by Theorem 4.68.



In topological spaces that satisfy the first axiom of countability the characterization of the points in the closure of a set contained in Theorem 4.66 may be reformulated by replacing nets with sequences. Theorem 4.71. Let (S, O) be a topological space that satisfies the first axiom of countability and let T be a subset of S. We have x ∈ K(T ) if and only if there exists a sequence in T that converges to x. Proof. Assume that x ∈ K(T ). We have V ∩ T = ∅ for every V ∈ neighx (O). Since (S, O) satisfies the first countability axiom there exist a countable local basis of neighborhoods at x, {V0 , V1 , . . . , Vn , . . .}. We have n n k=1 Vk ∩ T = ∅. If xn ∈ k=1 Vk ∩ T , it follows that xn → x. Conversely, suppose that there exists a sequence (xn ) in T that converges to x. If V neighx (T ) there exists nV such that n  nV implies  xn ∈ V , hence xn ∈ V ∩ T . Thus, x ∈ K(T ). The next statement is a characterization of compact spaces that uses nets. Theorem 4.72. A topological space (S, O) is compact if and only if each net (xi )i∈I in S clusters at some point of S. Proof. Suppose that (S, O) is compact and let (xi )i∈I be a net in S. Define the collection A of subsets of S as A = {Ai | i ∈ I} by Ai = {xj | j ∈ I, j  i}. Let J be a non-empty and finite subset of I. Since I is directed, there ex  ists k ∈ I such that j  k for every j ∈ J, so Ak ⊆ j∈J Aj ⊆ j∈J K(Aj ). Thus, the collection {K(Ai ) | i ∈ I} has the f.i.p. so there exists

May 2, 2018 11:28

210

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 210

Mathematical Analysis for Machine Learning and Data Mining

 x ∈ {K(Ai ) | i ∈ I}. We claim that (xi ) clusters at x. Let V ∈ neighx (O) and let i ∈ I. Since x ∈ K(Ai ) we have V ∩ Ai = ∅, so xj ∈ V for some j  i. Conversely, assume that each net (xi ) clusters at some point of S. Let D be a collection of closed subsets of S that has the f.i.p.. Let I consist of all pairs (E, x) such that E is a non-empty finite subcol lection of D and x ∈ E. Define (E, x)  (E , x ) if and only if E ⊆ E . It is easy to see that I is a directed partial order for, if (E1 , x1 ), (E2 , x2 ) ∈ I, then E1 ∪ E2 is a non-empty finite subcollection of D and there ex ists x ∈ (E1 ∪ E2 ), so (E1 ∪ E2 , x) ∈ I and (E1 , x1 )  (E1 ∪ E2 , x), (E2 , x2 )  (E1 ∪ E2 , x). For i = (E, x), define xi as xi = x.  We show that if (xi ) clusters at some x, then x ∈ D. Let D be a closed set in D and let V ∈ neighx (O). If y ∈ D, then ({D}, y) ∈ I, so there exists (E, z) ∈ I such that ({D}, y)  (E, z) and z ∈ V . Since D ∈ D,   it follows that z ∈ E ⊆ D, so z ∈ V ∩ D. This shows that x ∈ D.  By Theorem 4.63, a topological space (S, O) is compact if and only if each net in S has some convergent subnet.

4.11

Continuous Functions

Definition 4.35. Let (S1 , O1 ) and (S2 , O2 ) be two topological spaces. A function f : S1 −→ S2 is continuous at x0 ∈ S if, for every V ∈ neighf (x0 ) (O2 ) there exists U ∈ neighx0 (O1 ) such that f (U ) ⊆ V . Example 4.36. Let (R, O) be the topological space introduced in Example 4.2. Choose V ∈ neighx0 (O) as V = B(f (x0 ), ). The existence of U ∈ neighx0 (O) such that f (U ) ⊆ B(f (x0 ), ) implies the existence of δ > 0 such that B(x0 , δ) ⊆ U , hence f (B(x0 , δ)) ⊆ B(f (x0 ), ). Conversely, suppose that for every > 0 there exists δ > 0 such that f (B(x0 , δ)) ⊆ B(f (x0 ), ). Let V be a neighborhood of f (x0 ). There exists > 0 such that B(f (x0 ), ) ⊆ V . We can choose U = B(x0 , δ) because f (U ) = f (B(x0 , δ)) ⊆ B(f (x0 ), ) ⊆ V. Thus, the continuity of f at x0 can be formulated in the “δ, ”-language, well-known from calculus.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Topology

9in x 6in

b3234-main

page 211

211

Definition 4.36. The function f : R −→ R is right-continuous in x0 if limx−→x0 − f (x) = f (x0 ); f is left-continuous in x0 if limx−→x0 + f (x) = f (x0 ). Definition 4.36 implies that f : R −→ R is continuous in x0 if and only if it is both right- and left-continuous. Example 4.37. Define the function f : R −→ R as  1 if x  3, f (x) = 0 otherwise. Since f (3) = 1 and limx→3+ f (x) = 1, the function is right-continuous in 3. On other hand limx→3− f (x) = 0, so the function has only lateral limits in 3. Therefore, f is not continuous in 3. Example 4.38. Let (S, O1 ) be a topological space, and let (R, O) be the topological space of real numbers equipped with the usual topology. A function f : S −→ R is continuous in x0 ∈ S if for every neighborhood V ∈ neighf (x0 ) (O) there exists U ∈ neighx0 (O1 ) such that f (U ) ⊆ V . If c < f (x0 ), the set (c, ∞) is a neighborhood of f (x0 ) in (R, O). Therefore, there exists a neighborhood U of x0 in (S, O1 ) such that f (U ) ⊆ (c, ∞). In other words, for every x ∈ U we have c < f (x). Similarly, if f (x0 ) < d, the set (−∞, d) is a neighborhood of f (x0 ) in R and therefore, there exists a neighborhood W of x0 in (S, O1 ) such that f (W ) ⊆ (−∞, d). In other words, for every x ∈ W we have f (x) < d. This discussion shows that if f is continuous in x0 , then there exists a neighborhood Y of x0 such that f is bounded in Y . Conversely, suppose that the following conditions are satisfied: (i) if c < f (x0 ) there exists a neighborhood U of x0 in (S, O1 ) such that for every x ∈ U we have c < f (x); (ii) if f (x0 ) < d there exists a neighborhood W of x0 in (S, O1 ) such that for every x ∈ W we have f (x) < d, the function f is continuous in x0 . Indeed, let x0 ∈ S and let (c, d) be an interval that includes f (x0 ). Since c < f (x0 ), there exists a neighborhood U of x0 in (S, O1 ) such that x ∈ U implies c < f (x). Similarly, there exists a neighborhood W of x0 in (S, O1 ) such that for every x ∈ W we have f (x) < d. It is clear that Z = U ∩ W is a neighborhood of x0 and x ∈ Z implies c < f (x) < d, or f (Z) ⊆ (c, d), hence f is continuous in x0 . In particular, if f : S −→ R is a continuous function in x0 and f (x0 ) > 0, then there exists a neighborhood U of x0 such that f (x) > 0 when

May 2, 2018 11:28

212

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 212

Mathematical Analysis for Machine Learning and Data Mining

x ∈ U ; this follows by taking c = 0 in the previous argument. Similarly, if f (x0 ) < 0, there exists a neighborhood V of x0 such that f (x) < 0 when x∈V. Using the characterization of real-valued continuous function defined on a topological space (S, O) discussed in Example 4.38 it is possible to establish certain closure properties of the set of these functions. Theorem 4.73. Let (S, O) be a topological space and let C(S) be the set of bounded real-valued continuous functions of the form f : S −→ R. If f, g ∈ C(S) then for every a, b ∈ R we have af + bg ∈ C(S), where (af + bg)(x) = af (x) + bg(x) for x ∈ S. Also, the product f g defined by (f g)(x) = f (x)g(x) for x ∈ S belongs to C(S). Proof. We prove only the closure of C(S) relative to the product. Suppose that f, g are continuous in x0 . Since f (x)g(x) − f (x0 )g(x0 ) = (f (x) − f (x0 ))g(x) + f (x0 )(g(x) − g(x0 )), it follows that |f (x)g(x) − f (x0 )g(x0 )|  |f (x) − f (x0 )| g(x) + |f (x0 )| |g(x) − g(x0 )|. By the continuity of f , there exists a neighborhood U of x0 in (S, O) such , where M = sup{|g(x)| | x ∈ Y }, that x ∈ U implies |f (x) − f (x0 )| < 2M where Y is a neighborhood of x0 on which g is bounded. If f (x0 ) = 0 the continuity of f g in x0 is immediate, so we may assume that f (x0 ) = 0. The continuity of g, implies the existence of a neighborhood T of x0 such that , which allows us to conclude that if x ∈ T implies |g(x) − g(x0 )| < 2|f (x 0 )| x belongs to the neighborhood U ∩ Y ∩ T of x0 , then |f (x)g(x) − f (x0 )g(x0 )| < M + |f (x0 )| = , 2M 2|f (x0 )| which implies the continuity of f g in x0 .  Note that f (U ) ⊆ V is equivalent to U ⊆ f −1 (V ). This allows us to formulate an equivalent condition for continuity. Theorem 4.74. Let (S1 , O1 ) and (S2 , O2 ) be two topological spaces. A function f : S1 −→ S2 is continuous at x0 ∈ S1 if for each neighborhood V ∈ neighf (x0 ) (O2 ) the set f −1 (V ) belongs to neighx0 (O1 ). Proof. Suppose that f is continuous at x0 ∈ S1 . Then, for every V ∈ neighf (x0 ) (O2 ) there exists U ∈ neighx0 (O1 ) such that U ⊆ f −1 (V ), which implies that f −1 (V ) is a neighborhood of x0 . Conversely, if f −1 (V ) is a neighborhood of x0 for each V ∈  neighf (x0 ) (O2 ) the continuity of f in x0 is immediate.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Topology

9in x 6in

b3234-main

page 213

213

Theorem 4.75. Let (S1 , O1 ) and (S2 , O2 ) be two topological spaces having the closure operators K1 and K2 , respectively. If f : S1 −→ S2 is continuous at x0 ∈ S1 , then x0 ∈ K1 (T ) for some subset T of S1 implies f (x0 ) ∈ K2 (f (T )). Proof. Let T be a subset of S1 , x0 ∈ T , and let V be a neighborhood of f (x0 ) in S2 . Since f is continuous at x0 , f −1 (V ) is a neighborhood of x0 in S1 . Therefore, by Theorem 4.24, f −1 (V ) ∩ T = ∅, which implies that V ∩ f (T ) = ∅. By the same Theorem 4.24, this implies f (x0 ) ∈ K2 (f (T )).  Theorem 4.76. For 1  i  3 let (Si , Oi ) be three topological spaces. If f : S1 −→ S2 is continuous at x0 , where x0 ∈ S1 and g : S2 −→ S3 is continuous at f (x0 ), then the function gf : S1 −→ S3 is continuous at x0 . Proof. Let W ∈ neighg(f (x0 ) (O3 ). Since g is continuous at f (x0 ) we have g −1 (W ) ∈ neighf (x0 ) (O2 ). The continuity of f in x0 implies, in turn, that f −1 (g −1 (W )) ∈ neighx0 (O1 ). The continuity of gf follows by observing  that (gf )−1 (W ) = f −1 (g −1 (W )). Definition 4.37. Let (S1 , O1 ) and (S2 , O2 ) be two topological spaces. A function f : S1 −→ S2 is continuous if it is continuous at each point x of S1 . Theorem 4.77. Let (S1 , O1 ) and (S2 , O2 ) be two topological spaces having the closure operators K1 and K2 , respectively, and let f : S1 −→ S2 be a function. The following statements are equivalent: (i) f is continuous; (ii) f (K1 (T )) ⊆ K2 (f (T )) for every subset T of S1 ; (iii) if L is a closed set in (S2 , O2 ), then f −1 (L) is a closed set in (S1 , O1 ); (iv) if V is an open set in (S2 , O2 ), then f −1 (V ) is an open set in (S1 , O1 ). Proof. (i) implies (ii): This implication follows from Theorem 4.75. (ii) implies (iii): Let L be a closed set in (S2 , O2 ). By (ii) we have f (K1 (f −1 (L))) ⊆ K2 (f (f −1 (L)). Taking into account that f (f −1 (L)) = L it follows that f (K1 (f −1 (L))) ⊆ K2 (L) = L, so K1 (f −1 (L)) ⊆ f −1 (L), which implies the equality K1 (f −1 (L)) = f −1 (L). This shows that f −1 (L) is a closed set. (iii) implies (iv): Let V be an open set in (S2 , O2 ). This means that S2 − V is a closed set, so by (iii), f −1 (S2 − V ) = S1 − f −1 (V ) is a closed set in (S1 , O1 ). This means that f −1 (V ) is an open set in this space.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 214

Mathematical Analysis for Machine Learning and Data Mining

214

(iv) implies (i): Let x ∈ S1 and let V ∈ neighf (x) (O2 ). There exists an open set U ∈ O2 such that f (x) ∈ U ⊆ V , so x ∈ f −1 (U ) ⊆ f −1 (V ). By (iv) f −1 (U ) is open in (S1 , O1 ), which implies that f −1 (V ) ∈ neighx (O1 ). Since f (f −1 (V )) = V , it follows that f is continuous at an arbitrary x ∈ S1 ,  that is, f is continuous in S1 . If f : S1 −→ S2 is a continuous function between the topological spaces (S1 , O1 ) and (S2 , O2 ) and O1 and O2 are topologies on S1 and S2 , respectively, such that O2 ⊆ O2 and O1 ⊆ O1 , then f is also a continuous function between the topological spaces (S1 , O1 ) and (S2 , O2 ). Therefore, any function defined on the topological space (S, P(S)) (equipped with the discrete topology) with values in an arbitrary topological space (S  , O ) is continuous; similarly, any function f : S −→ S  between a topological space (S, O) and (S  , {∅, S  }) (equipped with the discrete topology) is continuous. Theorem 4.78. Let (S1 , O1 ) and (S2 , O2 ) be two topological spaces and let S2 be a sub-basis of the topology O2 . A function f : S1 −→ S2 is continuous if and only if for set S ∈ S2 , we have f −1 (S) ∈ O1 . Proof. Let T be an open set in O2 . Then T is a union of finite in tersections of members of S2 , that is, T = {Vj | j ∈ J}, where  Vj = i∈Ij Si , each Ij is a finite set and each Si belongs to S2 . Since   f −1 (T ) = j∈J i∈Ij f −1 (Si ) and each of the sets f −1 (Si ) is open, it follows that f −1 (T ) is open, so f is continuous.  Conversely, if f is continuous each of the sets f −1 (S) is open. Theorem 4.79. Let (S1 , O1 ) and (S2 , O2 ) be two topological spaces and let B2 be a basis of the topology O2 . A function f : S1 −→ S2 is continuous if and only if for set B ∈ B2 , we have f −1 (B) ∈ O1 . Proof.

The proof is similar to the argument of Theorem 4.78.



Example 4.39. Let (S, O) be a topological space and let f : S −→ R be a continuous function, where R is equipped with the usual topology introduced in Example 4.3. Recall that the collection of open intervals of R is a basis for R, so the continuity of f is equivalent to the fact that for every a, b ∈ R, f (x) ∈ (a, b) implies that f −1 (a, b) is an open set that contains x. Function continuity can be formulated using nets. Theorem 4.80. Let (S1 , O1 ) and (S2 , O2 ) be two topological spaces. A function f : S1 −→ S2 is continuous at x ∈ S1 if and only if for each net (xi )i∈I such that xi → x we have f (xi ) → f (x).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topology

page 215

215

Proof. Let f be a function continuous at x and let (xi )i∈I be a net such that xi → x. If V ∈ neighf (x) (O2 ), then f −1 (V ) ∈ neighx (O1 ), so there exists i ∈ I such that xj ∈ f −1 (V ) for j  i. This implies f (xj ) ∈ V for j  i, so f (xi ) → f (x). Conversely, assume that for each net (xi )i∈I such that xi → x we have f (xi ) → f (x). Suppose that there exists V  ∈ neighf (x) (O2 ) such that f −1 (V  ) ∈ neighx (O1 ). For each U ∈ neighx (O1 ) there exists xU ∈ U such that xU ∈ f −1 (V  ). This yields a net (xU ) with xU → x in (S1 , O1 ), which, by hypothesis, implies f (xU ) → f (x), so f (xU ) ∈ V  for some U ∈ neighx (O1 ), so xU ∈ f −1 (V  ). This contradiction shows that f is continuous at x.  Compactness is preserved by continuous functions as we show next. Theorem 4.81. Let (S, O) and (T, O ) be two topological spaces and let f : S −→ T be a continuous function. If (S, O) is compact, then f (S) is compact in (T, O ). Proof. Let D = {Di | i ∈ I} be an open cover of f (S). Then f −1 (Di ) is an open set in (S, O) because f is continuous and the collection C = {f −1 (Di ) | i ∈ I} is an open cover of S. Since (S, O) is compact, there exists a finite subcover C1 = {f −1 (Di ) | i ∈ I1 } of S (I1 is a finite subset  of I). Since S = {f −1 (Di ) | i ∈ I1 }, we have # $ f (S) = f {f −1 (Di ) | i ∈ I1 }   = {f (f −1 (Di )) | i ∈ I1 } = {Di | i ∈ I1 }, which shows that D contains a finite subcover of f (S).



Corollary 4.15. Let (S, O) and (T, O ) be two topological spaces and let f : S −→ T be a continuous function. If U is a compact subset of S, then f (U ) is compact in (T, O ). Proof.

This statement is an immediate consequence of Theorem 4.81. 

Definition 4.38. Let (S, O) be a topological space and let f : S −→ R be a function. The support of f is the closed set supp(f ) = K({x ∈ S | f (x) = 0}). The set of continuous functions f : S −→ R such that supp(f ) is a compact set is denoted by Fc (S, O).

May 2, 2018 11:28

216

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 216

Mathematical Analysis for Machine Learning and Data Mining

Theorem 4.82. Let (S, O) and (T, O ) be two topological spaces and let f : S −→ T be a continuous function in Fc (S, O). Then, the range of f is a compact subset of T . Proof. Note that the range of f , that is, the set f (S), can be written as f (S) = f (supp(f )) ∪ {0}. Since supp(f ) is compact, the set f (supp(S)) is compact. Therefore, the range f (S) of f is compact.  Theorem 4.83. (Uryson’s Lemma for Normal Spaces) Let (S, O) be a normal topological space and let A, B be two disjoint closed sets. There exists a continuous function f : S −→ [0, 1] such that f (x) = 0 for x ∈ A and f (x) = 1 for x ∈ B. Proof. Consider the following sequence that consists of rational numbers in [0, 1] expressed as irreducible fractions: 1 1 2 1 3 1 2 3 4 1 1, 0, , , , , , , , , , , . . . . 2 3 3 4 4 5 5 5 5 6

(4.4)

We define a sequence of open subsets (Up ) of a normal topological space (S, O), where p is a member of the above sequence such that U1 = S − B and Up is defined inductively such that p < q implies K(Up ) ⊆ Uq .

(4.5)

Since A ∩ B = ∅ we have A ⊆ U1 . By Theorem 4.50 the normality of S implies the existence of an open set U0 such that A ⊆ U0 ⊆ K(U0 ) ⊆ U1 . Let Pn be the set of the first n rational numbers in the above sequence. Suppose that the open set Up is constructed for all numbers p in Pn such that p < q implies K(Up ) ⊆ Uq . If r is the (n + 1)st rational number in the sequence (4.4) and Pn+1 = Pn ∪ {r}, then there exists a largest number p and a smallest number q in Pn such that p < r < q. The sets Up and Uq are already defined and K(Up ) ⊆ Uq by the inductive hypothesis. Since (S, O) is normal, K(Up ) is a closed set and Uq is an open set, by Theorem 4.50, there exists an open set Ur such that K(Up ) ⊆ Ur ⊆ K(Ur ) ⊆ Uq . Property (4.5) holds for Pn+1 . Indeed, if p, q ∈ Pn , the property holds by inductive hypothesis. Suppose that one of the numbers p, q is r and let s ∈ Pn . Then, either s  p < r or r < q  s. In the first case, K(Us ) ⊆ K(Up ) ⊆ Ur ; in the second case, K(Ur ) ⊆ Uq ⊆ Us , so property (4.5) holds for Pn+1 . Starting from the collection {Up | p is a rational number in [0, 1]},

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Topology

b3234-main

page 217

217

we extend this notation to Q by defining  ∅ if p < 0, Up = S if p > 1. With this extended notation property (4.5) still holds. Let x ∈ S. Define the set Q(x) = {p ∈ Q | x ∈ Up }. The set Q(x) is lower bounded because x ∈ Up for p < 0. Note also, that x ∈ Up for p > 1, This allows us to define a function f : S −→ [0, 1] as f (x) = inf Q(x). We claim that f is the function whose existence is affirmed by the theorem. If x ∈ A, then x ∈ Up for every p  0, so f (x) = 0. If x ∈ B, then x ∈ Up for every p  1, so f (x) = 1. Note that (i) If x ∈ K(Ur ), then f (x)  r. Indeed, if x ∈ K(Ur ), then x ∈ Us for every s > r, so Q(x) contains all rational number greater than r and, therefore, f (x) = inf Q(x)  r. (ii) If x ∈ Ur , then f (x)  r. Indeed, since x ∈ Ur , we have x ∈ Us for s < r, so Q(x) contains no rational number less than r which implies f (x) = inf Q(x)  r. To prove the continuity of f in x0 we show that if f (x0 ) ∈ (c, d), then there exists a neighborhood U of x0 such that f (U ) ⊆ (c, d). Let p, q be rational numbers such that c < p < f (x0 ) < q < d. Let U be the open set U = Uq − K(Up ). Note that x0 ∈ U because f (x0 ) < q implies x0 ∈ Uq and f (x0 ) > p implies x0 ∈ K(Up ), as we saw previously. If x ∈ U , then x ∈ Uq ⊆ K(Uq ), so f (x)  q. Also, x ∈ K(Up ), so x ∈ Up , which implies f (x)  p. Thus, f (x) ∈ [p, q] ⊆ (c, d), so f is continuous.  Corollary 4.16. Let (S, O) be a normal topological space and let K, T be two sets such that K is compact, T is open and K ⊆ T . There exists a continuous function f : S −→ [0, 1] such that 1K  f  1T . Proof. Since the sets K and S − T are closed and disjoint, by Uryson’s Lemma (Theorem 4.83), there exists a continuous function f : S −→ [0, 1] on S such that f (x) = 0 if x ∈ S − T and f (x) = 1 if x ∈ K. This is  equivalent to 1K  f  1T .

May 2, 2018 11:28

218

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 218

Mathematical Analysis for Machine Learning and Data Mining

Note that if K1 , K2 are two compact and disjoint subsets of normal topological space (S, O), we have K1 ⊆ S − K2 , and S − K2 is open. Thus, by Corollary 4.16, there exists a continuous function f : S −→ [0, 1] such that 1K1  f  1S−K2 = 1 − 1K2 . Theorem 4.84. Any locally compact topological space is a subspace of a compact Hausdorff space. Proof.

This follows directly from Theorem 4.60.



Definition 4.39. A Hausdorff topological space (S, calo) is completely regular if for every point x ∈ S and every closed set W such that y ∈ W there exists a continuous function f : S −→ R such that f (y) = 0 and f (w) = 1 for every x ∈ W . Complete regularity of topological spaces is hereditary; in other words, if (S, O) is a completely regular topological space then any of its subspaces is also completely regular. Theorem 4.85. Any normal space is completely regular. Proof.

This follows immediately from Uryson’s Lemma (Theorem 4.83). 

Theorem 4.86. Any locally compact topological space (S, O) is completely regular. Proof. (s, O) is a subspace of a compact Hausdorff space, which is a normal space and therefore a completely regular space. Therefore, by heredity, (S, O) is completely regular. 

4.12

Homeomorphisms

Definition 4.40. Let (S, O) and (T, O ) be two topological spaces. A bijection f : S −→ T is a homeomorphism if both f and its inverse f −1 are continuous functions. If a homeomorphism exists between the topological spaces (S, O) and (S, O ), we say that these spaces are homeomorphic. Two homeomorphic topological spaces are essentially identical from a topological point of view. Example 4.40. The identity map 1S of any topological space (S, O) is a homeomorphism.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topology

page 219

219

Example 4.41. Consider the subsets [0, 1) of R and S2 (02 , 1) = {x ∈ R2 | x21 + x22 = 1}. The function f : [0, 1) −→ S2 (02 , 1) given by  f (a) =

 cos(2πa) sin(2πa)

for a ∈ [0, 1) is bijective and continuous, but it is not a homeomorphism. 1 −1 Indeed, its inverse f : S (0 , 1) −→ [0, 1) is not continuous at 2 2 0 . Ob 1 −1 1 serve that f 0 = 0. However, for a neighborhood V of 0 in S2 (02 , 1) and y ∈ V , f −1 (y) is not necessarily located in a neighborhood of 0.

[ 0

Fig. 4.1

) 1

(1, 0)

A continuous and bijective functions which is not a homeomorphism.

Theorem 4.87. A bijection f : S −→ T between two topological spaces (S, O) and (T, O ) is a homeomorphism if and only if U ∈ O is equivalent to f (U ) ∈ O . Proof. Suppose that f is a homeomorphism between (S, O) and (T, O ). If U ∈ O the continuity of f −1 implies that (f −1 )−1 (U ) = f (U ) ∈ O ; on the other hand, if f (U ) ∈ O , then, since U = f −1 (f (U )), the continuity of f yields U ∈ O. Conversely, suppose that for the bijection f : S −→ T , U ∈ O if and only if f (U ) ∈ O . Suppose that V ∈ O ; since f is a bijection, there is W ⊆ S such that V = f (W ) and W ∈ O by hypothesis. Observe that f −1 (V ) = W , so f is continuous. To prove that f −1 is continuous, note that we need to verify that (f −1 )−1 (Z) is an open set in (S, O) for any set  Z ∈ O , which is effectively the case because (f −1 )−1 (Z) = f (Z).

May 2, 2018 11:28

220

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 220

Mathematical Analysis for Machine Learning and Data Mining

Any property of (S, O) that can be expressed using the open sets of this topological space is preserved in topological spaces (T, O ) that are homeomorphic to (S, O). Therefore, such a property is said to be topological. The collection of all pairs of topological spaces that are homeomorphic is an equivalence relation on the class of topological spaces as can be easily shown. Example 4.42. All open intervals of R, bounded or not, are homeomorphic. Let (a, b) and (c, d) be two bounded intervals of R and let f : (a, b) −→ d−c (c, d) be the linear function defined by f (x) = px + q, where p = b−a and q = bc−ad b−a . It is easy to verify that f is a homeomorphism, so any two bounded intervals of R are homeomorphic; in particular, any bounded interval (a, b) is homeomorphic with (0, 1). Any two unbounded intervals (a, ∞) and (b, ∞) are homeomorphic; the mapping g(x) = ab x is a homeomorphism between these sets. Similarly, any two unbounded intervals of the form (−∞, a) and (−∞, b) are homeomorphic, and so are (a, ∞) and (−∞, b). The function h : (0, 1) −→ (0, ∞) defined by h(x) = tan πx 2 is a home2 −1 omorphism, whose inverse mapping is h (x) = π arctan x so (0, 1) is homeomorphic with (0, ∞). Finally, (−1, 1) is homeomorphic to (−∞, ∞) since the mapping h1 : (−1, 1) −→ (−∞, ∞) defined by h(x) = tan πx 2 for x ∈ (−1, 1) is a homeomorphism. Definition 4.41. Let S be a set and let F = {fα | α ∈ A, fα : S −→ Tα } be a family of functions indexed by the set A, where (Tα , Oα ) is a topological space for each α ∈ A. The weak topology on S induced by F is the topology OF generated by the collection S = {fα−1 (U ) | U ∈ Oα for α ∈ A}. This definition means that the weak topology on S induced by F has S as a sub-basis. Note that each function fα is continuous when regarded as functions between (S, OF ) and (Tα , Oα ). Furthermore, OF is the least topology on S that makes all functions fα continuous. A basis for the weak topology OF consists of finite intersections of the m (Uαk ), where Uαk ∈ Oαk . form k=1 fα−1 k Theorem 4.88. Let {(Tα , Oα ) | α ∈ A} be a collection of topological spaces, S be a set, and let F = {fα | α ∈ A, fα : S −→ Tα }

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Topology

9in x 6in

b3234-main

page 221

221

be a family of functions. Suppose that (xi )i∈I be a net in the topological space (S, OF ) equipped with the weak topology generated by F. We have xi → x in (S, OF ) if and only if fα (xi ) −→ fα (x) for each α ∈ A. Proof. Since each function fα is continuous, xi → x implies immediately fα (xi ) −→ fα (x) for each α ∈ A. Conversely, suppose that fα (xi ) −→ fα (x) for each α ∈ A and let  −1 B= m k=1 fαk (Uαk ) be a set in the basis in (S, OF ). For each k there exists iαk such that j  iαk implies fα (xj ) ∈ Uαk . Let m (Uαk ) = B, so i be such that i  iαk for 1  k  m. Then, xi ∈ k=1 fα−1 k  xi → x by Theorem 4.65. Definition 4.42. Let (S1 , O1 ) and (S2 , O2 ) be two topological spaces and let f : S1 −→ S2 be a function. The function f is open if f (L) is open for every open set L, where L ∈ O1 ; the function f is closed if f (H) is closed for every closed set H in S1 . Theorem 4.89. Let (S1 , O1 ) and (S2 , O2 ) be two topological spaces, let Ki and Ii be the closure and interior operators of the space Si for i = 1, 2, and let f : S1 −→ S2 be a function. The following statements hold: (i) f is open if and only if f (I1 (U )) ⊆ I2 (f (U )) for every U ∈ P(S1 ); (ii) f is closed if and only if K2 (f (U )) ⊆ f (K1 (U )) for every U ∈ P(S1 ); (iii) a bijection f : S1 −→ S2 is open if and only if it is closed. Proof. Part (i): Suppose that f is open. Since I1 (U ) is an open set, it follows that f (I1 (U )) is open. Note that f (I1 (U )) is included in f (U ) and, since I2 (f (U ) is the largest open set included in f (U ) it follows that f (I1 (U )) ⊆ I2 (f (U )) for every U ∈ P(S1 ). Conversely, suppose that f (I1 (U )) ⊆ I2 (f (U )) for every U ∈ P(S1 ). If U is an open set, then I1 (U ) = U , so f (U ) ⊆ I2 (f (U ), which implies f (U ) = I2 (f (U ). Therefore, f (U ) is an open set, so f is an open function. Part (ii) is has a similar argument. Part (iii): Let f be a bijection. Suppose that f is open and let U be a closed set in (S1 , O1 ). Then S1 − U is open and, therefore, the set f (S1 − U ) is open in S2 . Since f is a bijection we have f (S1 − U ) = S2 − f (U ), so f (U ) is closed. Thus, f is a closed mapping. In a similar manner we can show that if f is closed, then f is open. 

May 2, 2018 11:28

222

4.13

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 222

Mathematical Analysis for Machine Learning and Data Mining

Connected Topological Spaces

This chapter is dedicated to the formalization of the notion of an “onepiece” topological space. Theorem 4.90. Let (S, O) be a topological space. The following statements are equivalent: (i) there exists a clopen subset K of S such that K ∈ {∅, S}; (ii) there exist two non-empty open subsets L, L of S that are complementary; (iii) there exist two non-empty closed subsets H, H  of S that are complementary. Proof. (i) implies (ii): If K is clopen and K ∈ {∅, S}, then both K and ¯ are non-empty open sets. K (ii) implies (iii): Suppose that L and L are two non-empty complementary open subsets of S. Then, L and L are in the same time closed because the complements of each set is open. (iii) implies (i): If H and H  are complementary closed sets, then each of them is also open because the complements of each set is closed. Thus, both sets are clopen.  Definition 4.43. A topological space (S, O) is disconnected if it satisfies any of the equivalent conditions of Theorem 4.90. Otherwise, (S, O) is said to be connected. A subset T of a connected topological space is connected if the subspace T is connected. Theorem 4.91. Let T be a subset of S, where (S, O) is a topological space. The following statements are equivalent: (i) T is connected; (ii) there are no open sets L1 , L2 in (S, O) such that T ⊆ L1 ∪ L2 , and T ∩ L1 , and T ∩ L2 are non-empty and disjoint; (iii) there are no closed sets H1 , H2 in (S, O) such that T ⊆ H1 ∪ H2 , and T ∩ H1 and T ∩ H2 are non-empty and disjoint; (iv) there is no clopen set in (S, O) that has a non-empty intersection with T . Proof. The equivalence of the statements follows immediately from the definition of the subspace topology. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Topology

b3234-main

page 223

223

Theorem 4.92. Let C = {Ci | i ∈ I} be a family of connected subsets of a topological space (S, O). If Ci ∩ Cj = ∅ for every i, j ∈ I such that i = j,  then C is connected.  Proof. Suppose that C = C is not connected. Then C contains two complementary open subsets L and L . For every i ∈ I, the sets Ci ∩L and Ci ∩ L are complementary and open in Ci . Since each Ci is connected, we have either Ci ∩L = ∅ or Ci ∩L = ∅ for every i ∈ I. In the first case, Ci ⊆ L , while in the second, Ci ⊆ L . Thus, the collection C can be partitioned into two subcollections, C = C ∪ C , where C = {Ci ∈ C | Ci ⊆ L } and C = {Ci ∈ C | Ci ⊆ L }. Clearly, two sets Ci ∈ C and Cj ∈ C are disjoint because the sets L and L are disjoint, and this contradicts the hypothesis.  Corollary 4.17. Let (S, O) be a topological space and let x ∈ S. The  collection Cx of connected subsets of S that contain x has Kx = Cx as its largest element. Proof.

This follows immediately from Theorem 4.92.



We refer to Kx as the connected component of x. Theorem 4.93. Let T be a connected subset of a topological space (S, O), and suppose that W is a subset of S such that T ⊆ W ⊆ K(T ). Then W is connected. Proof. Suppose that W is not connected (that is, W = U ∪ U  , where U and U  are two non-empty, disjoint, and open sets in W ). There exist two open sets L, L in S such that U = W ∩ L and U  = W ∩ L . Since T ⊆ W , the sets T ∩ U and T ∩ U  are open in T , disjoint, and their union equals T . Thus, we have either T ∩ U = ∅ or T ∩ U  = ∅ because T is connected. If T ∩ U = ∅, then T ∩ L = (T ∩ W ) ∩ L = T ∩ (W ∩ L) = T ∩ U = ∅, so ¯ Since L ¯ is closed, K(T ) ⊆ L, ¯ which implies W ⊆ L, ¯ which implies T ⊆ L. U = W ∩ L = ∅. This contradicts the assumption made earlier about U . A similar contradiction follows from T ∩ U  = ∅. Thus, W is connected.  Corollary 4.18. If T is a connected subset of a topological space (S, O), then K(T ) is also connected. Proof.

This statement is a special case of Theorem 4.17.



Theorem 4.94. Let (S, O) be a topological space. The collection of all connected components of S is a partition of S that consists of closed sets.

May 2, 2018 11:28

224

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 224

Mathematical Analysis for Machine Learning and Data Mining

Proof. Corollary 4.18 implies that each connected component Kx is closed. Suppose that Kx and Ky are two connected components that are not disjoint. Then, by Theorem 4.92, Kx ∪Ky is connected. Since x ∈ Kx ∪Ky , it follows that Kx ∪ Ky ⊆ Kx because Kx is the maximal connected set  that contains x, so Ky ⊆ Kx . Similarly, Kx ⊆ Ky , so Kx = Ky . Theorem 4.95. The image of a connected topological space through a continuous function is a connected set. Proof. Let (S1 , O1 ) and (S2 , O2 ) be two topological spaces and let f : S1 −→ S2 be a continuous function, where S1 is connected. If f (S1 ) were not connected, we would have two non-empty open subsets L and L of f (S1 ) that are complementary. Then, f −1 (L) and f −1 (L ) would be two non-empty, open sets in S1 which are complementary, which contradicts  the fact that S1 is connected. A characterization of connected spaces is given next. Theorem 4.96. Let (S, O) be a topological space and let ({0, 1}, P({0, 1}) be a two-element topological space equipped with the non-discrete topology. Then, S is connected if and only if every continuous application f : S −→ {0, 1} is constant. Proof. Suppose that S is connected. Both f −1 (0) and f −1 (1) are clopen sets in S because both {0} and {1} are clopen in the discrete topology. Thus, we have either f −1 (0) = ∅ and f −1 (1) = S, or f −1 (0) = S and f −1 (1) = ∅. In the first case, f is the constant function f (x) = 1; in the second, it is the constant function f (x) = 0. Conversely, suppose that the condition is satisfied for every continuous function f : S −→ {0, 1} and suppose (S, O) is not connected. Then, there exist two non-empty disjoint open subsets L and L that are complementary. Let f = 1L be the characteristic function of L, which is continuous because both L and L are open. Thus, f is constant and this implies either L = ∅  and L = S or L = S and L = ∅, so S is connected. Example 4.43. The topological space (R, O) is connected. Suppose that K is a clopen set in R distinct from R and ∅, and let x ∈ R − K. Suppose that the set K ∩ [x, ∞) is non-empty. Then, this set is closed and bounded below and therefore has a least element u. Since K ∩ [x, ∞) = K∩(x, ∞) is also open, there exists > 0 such that (u− , u+ ) ⊆ K∩[x, ∞), which contradicts the fact that u is the least element of K ∩[x, ∞). A similar

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Topology

9in x 6in

b3234-main

page 225

225

contradiction is obtained if we assume that K ∩ (−∞, x] = ∅, so R cannot contain a clopen set distinct from R or ∅. Example 4.44. Theorem 4.96 allows us to prove that the connected subsets of R are exactly the intervals. Suppose that T is a connected subset of S but is not an interval. Then, there are three numbers x, y, z such that x < y < z, x, z ∈ T but y ∈ T . Define the function f : T −→ {0, 1} by f (u) = 0 if u < y and f (u) = 1 if y < u. Clearly, f is continuous but is not constant, and this contradicts Theorem 4.96. Thus, T must be an interval. Suppose now that T is an open interval of R. We saw that T is homeomorphic to R (see Example 4.42), so T is indeed connected. If T is an arbitrary interval, its interior I(T ) is an open interval and, since I(T ) ⊆ T ⊆ K(I(T )), it follows that T is connected. Definition 4.44. A topological space (S, O) is totally disconnected if, for every x ∈ S, the connected component of x is Kx = {x}. Example 4.45. Any topological space equipped with the discrete topology is totally disconnected. Theorem 4.97. Let (S, O) be a topological space and let T be a subset of S. If for every pair of distinct points x, y ∈ T there exist two disjoint closed sets Hx and Hy such that T ⊆ Hx ∪ Hy , x ∈ Hx , and y ∈ Hy , then T is totally disconnected. Proof. Let Kx be the connected component of x, and suppose that y ∈ Kx and y = x, that is, Kx = Ky = K. Then, K ∩ Hx and K ∩ Hy are nonempty disjoint closed sets and K = (K ∩ Hx ) ∪ (K ∩ Hy ), which contradicts the connectedness of K. Therefore, Kx = {x} for every x ∈ T and T is totally disconnected. 

4.14

Products of Topological Spaces

Theorem 4.98. Let {(Si , Oi ) | i ∈ I} be a family of topological spaces  indexed by the set I. Define on the set S = i∈I Si the collection of sets  B = { j∈J p−1 j (Lj ) | Lj ∈ Oj and J finite}. Then, B is a basis for a topology.

May 2, 2018 11:28

226

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 226

Mathematical Analysis for Machine Learning and Data Mining

  −1 Proof. Note that every set j∈J pj (Lj ) has the form i∈I−J ×  L . We need to observe only that a finite intersection of sets in j∈J j B is again a set in B. Therefore, B is a basis.   Definition 4.45. The topology TOP(B) generated on the set S = i∈I Si  by B is called the productof the topologies Oi and is denoted by i∈I Oi .   The topological space i∈I Si , i∈I Oi is the product of the collection of topological spaces {(Si , Oi ) | i ∈ I}. The product of the topologies {Oi | i ∈ I} can be generated starting  from the sub-basis S that consists of sets of the form Dj,L = {t | t ∈ i∈I | t(j) ∈ L}, where j ∈ I and L is an open set in (Sj , Oj ). It is easy to see that any set in the basis B is a finite intersection of sets of the form Dj,L . Theorem 4.99. Let {(Si , Oi ) | i ∈ I} be a collection of topological spaces.  Each projection p : i∈I Si −→ S is a continuous function for ∈ I. Moreover, the product topology is the coarsest topology on S such that projections are continuous. Proof.

Let L be an open set in (S , O ). We have   + −1 Si | t( ) ∈ L , p (L) = t ∈

which has the form

i∈I

 i∈I

Ki , where each set Ki is open because  Si if i = , Ki = L if i = ,

for i ∈ I. Thus, p−1  (L) is open and p is continuous. The proof of the second part of the theorem is left to the reader.



Theorem 4.100. Let {(Si , Oi ) | i ∈ I} be a collection of topological spaces   and let x = (xi ) ∈ i∈I Si . Then, a subset V of i∈I Si is a neighborhood  of x if and only if V contains a set of the form i∈I Vi , where Vi is a neighborhood of xi for every i ∈ I, and Vi = Si for almost all i ∈ I.  Proof. If V ∈ neighx i∈I Oi there exists a set B in the basis B of the product topology (defined in Theorem 4.98) such that x ∈ B ⊆ V and B is a set that satisfies the condition of the theorem. Conversely, suppose that {Vi | i ∈ I} is a collection such that Vi ∈ neighxi (Oi ) and J = {i ∈ I | Vi = Si } is a finite set. For each i, Vi contains  an open neighborhood Ui of xi , where Ui = Si for i ∈ J. Then, B = i∈I    is a open set with x ∈ B ⊆ i∈I Vi ⊆ V , so V ∈ neighx i∈I Oi .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topology

page 227

227

Note that if I is a finite set, I = {1, . . . , n} a neighborhood of x = (x1 , . . . , xn ) is any subset of S1 ×· · ·×Sn containing a product V1 ×· · ·×Vn , where Vi ∈ neighxi (Oi ) for 1  i  n. Let {(Si , Oi ) | i ∈ I} be a collection of topological spaces and let x =   (xi ) ∈ i∈I Si . If f : S −→ i∈I Si is a mapping, then its ith coordinate mappings is the mapping fi : S −→ Si defined by fi (x) = pi (f (x)) for x ∈ S. Theorem 4.101. Let {(Si , Oi ) | i ∈ I} be a finite collection of topological  spaces. If (S, O) is a topological space and f : S −→ i∈I Si , then f is continuous if and only if each component fi is continuous. Proof. Since fi = pi f , it is clear that the continuity of f implies the continuity of all its components.  Conversely, suppose that each fi is continuous at x0 ∈ S. If U = i∈I Ui is an open set containing f (x0 ), where Ui ∈ Oi for i ∈ I, fi−1 (Ui ) is a  neighborhood of x0 in S and, therefore, f −1 (U ) = i∈I fi−1 (Ui ) is also a neighborhood of x0 . Since every neighborhood V of f (x0 ) in S contains an open set containing f (x0 ) it follows that f −1 (V ) (which contains f −1 (U ))  is a neighborhood of x0 , so f is continuous. Theorem 4.102. Let {(Si , Oi ) | i ∈ I} be a collection of topological spaces.  If each of the spaces Si are Hausdorff spaces, then i∈I Si is a Hausdorff space.  Proof. Let x = (xi ), y = (yi ) ∈ i∈I Si . There exists i ∈ I such that xi = yi . By Theorem 4.49, there exist U ∈ neighxi (Oi ) and V ∈ neighyi (Oi ) such that U ∩ V = ∅. Thus, x and y have disjoint neighborhoods in the   product space, which implies that i∈I Si is a Hausdorff space. Theorem 4.103. If (S, O) is a Hausdorff space, then the diagonal set DS = {(x, x) | x ∈ S} is closed in the topological space (S × S, O × O). Proof. Let x, y be two distinct points in S. Then, (x, y) ∈ D × D. Since x and y are distinct and S is a Hausdorff space, by Theorem 4.49 there exist U ∈ neighx (O) and V ∈ neighy (O) such that U ∩ V = ∅. Thus, the set U × V is a neighborhood of (x, y) that does not intersect D. This implies that S × S − D is an open set, which shows that D is a closed set. 

May 2, 2018 11:28

228

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 228

Mathematical Analysis for Machine Learning and Data Mining

Corollary 4.19. Let (S, O) be a topological space, (T, O ) be a Hausdorff topological space, and let f, g : S −→ T be two continuous functions. The set EQ(f, g) = {x ∈ S | f (x) = g(x)} is a closed subset of S. Proof. Let h : S −→ S × T × T be the mapping defined by h(x) = (f (x), g(x)) for x ∈ S. It is clear that h is continuous. The set D = {(t, t) | t ∈ T } is closed by Theorem 4.103 and so is the set h−1 (D) = {x ∈ S | f (x) = g(x)}.  Corollary 4.20. Let (S, O) be a topological space, (T, O ) be a Hausdorff topological space, and let f : S −→ T be a continuous function. The set {(x, y) ∈ S × T | y = f (x)} is a closed subset of S × T . Proof. Note that the mappings φ : S × T −→ T and ψ : S × T −→ T defined by φ(x, y) = y and ψ(x, y) = f (x) are continuous. Therefore, since EQ(φ, ψ) = {(x, y) ∈ S | y = f (x)}, by Corollary 4.19, the set {(x, y) ∈ S × T | y = f (x)} is closed.  Corollary 4.21. Let (S, O) be a topological space, (T, O ) be a Hausdorff topological space, and let f, g : S −→ T be two continuous functions. If the set EQ(f, g) = {x ∈ S | f (x) = g(x)} is dense in S, then f = g. Proof. If E(f, g) is dense in S we have K(EQ(f, g)) = S. Since EQ(f, g) is closed, it follows that EQ(f, g) = S, hence f = g.   Lemma 4.3. Let C be a collection of subsets of S = i∈I Si such that C has the f.i.p. and C is maximal with this property.  We have D ∈ C for every finite subcollection D of C. Furthermore, if T ∩ C = ∅ for every C ∈ C, then T ∈ C. Proof. Let D = {D1 , . . . , Dn } be a finite subcollection of C and let D =  D = ∅. Note that the intersection of every finite subcollection of C ∪ {D} is also non-empty. The maximality of C implies D ∈ C, which proves the first part of the lemma. For the second part of the lemma, observe that the intersection of any finite subcollection of D ∪ {T } is not empty. Therefore, as above, T ∈ C.  In the proof of the next theorem we use a fundamental assumption in mathematics known as Zorn’s lemma, that states that if every chain of a partially ordered set (S, ) has an upper bound, then S has a maximal element. This is equivalent to saying that S contains a chain that is maximal with respect to set inclusion.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Topology

b3234-main

page 229

229

Theorem 4.104. (Tychonoff ’s Theorem) Let {(Si , Oi ) | i ∈ I} be a collection of topological spaces such that Si = ∅ for every i ∈ I. Then,   ( i∈I Si , i∈I Oi ) is compact if and only if each topological space (Si , Oi ) is compact for i ∈ I.  Proof. If ( i∈I Si , O) is compact, then, by Theorem 4.81, it is clear that each of the topological spaces (Si , Oi ) is compact because each projection pi is continuous. Conversely, suppose that each of the topological spaces (Si , Oi ) is compact.  Let E be a family of sets in S = i∈I Si that has the f.i.p. and let (C, ⊆) be the partially ordered set whose elements are collections of subsets of S that have the f.i.p. and contain the family E.  Let {Ci | i ∈ I} be a chain in (C, ⊆). It is easy to verify that {Ci | i ∈ I} has the f.i.p., so every chain in (C, ⊆) has an upper bound. Therefore, by Zorn’s Lemma, the poset (C, ⊆) contains a maximal collection C that has the f.i.p. and contains E. We aim to find an element   t ∈ i∈I Si that belongs to {K(C) | C ∈ C} because, in this case, the  same element belongs to {K(C) | C ∈ E} and this would imply, by Theorem 4.44, that (S, O) is compact. Let Ci be the collection of closed subsets of Si defined by Ci = {Ki (pi (C)) | C ∈ C} for i ∈ I, where Ki is the closure of the topological space (Si , Oi ). It is clear that each collection Ci has the f.i.p. in Si . Indeed, since  C has the f.i.p., if {C1 , . . . , Cn } ⊆ C and x ∈ nk=1 Ck , then pi (x) ∈ n K(pi (Ck )), so Ci has the f.i.p. Since (Si , Oi ) is compact, we have   k=1 Ci = ∅, by part (iii) of Theorem 4.43. Let ti ∈ Ci = {Ki (pi (C)) | C ∈ C} and let t ∈ S be defined by t(i) = ti for i ∈ I.  Let Dj,L = {u | u ∈ i∈I | u(j) ∈ L}, a set of the sub-basis of the product topology that contains t, defined earlier, where L is an open set in (Sj , Oj ). Since g(j) ∈ L, the set L has a non-empty intersection with every set Ki (pi (C)), where C ∈ C. On the other hand, since pi (Dj,L ) = Si for  i = j, it follows that for every i ∈ I we have pi (Dj,L )∩ C∈C Ki (pi (C)) = ∅. Therefore, pi (Dj,L ) has a non-empty intersection with every set of the form Ki (pi (C)), where C ∈ C. By the contrapositive of Theorem 4.6, this means that pi (Dj,L ) ∪ pi (C) = ∅ for every i ∈ I and C ∈ C. This in turn means that Dj,L ∪ C = ∅ for every C ∈ C. By Lemma 4.3, it follows that Dj,L ∈ C. Since every set that belongs to the basis of the product topology is a finite intersection of sets of the form Dj,L , it follows that any member of the basis

May 2, 2018 11:28

230

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 230

Mathematical Analysis for Machine Learning and Data Mining

has a non-empty intersection with every set of C. This implies that g belongs    to {K(C) | C ∈ C}, which implies the compactness of ( i∈I Si , i∈I Oi ).  Example 4.46. In Example 4.21, we have shown that every closed interval [x, y] of R where x < y is compact. By Theorem 4.104, any subset of Rn of the form [x1 , y1 ] × · · · × [xn , yn ] is compact.

4.15

Semicontinuous Functions

ˆ defined on a topological space (S, O) We saw that a function f : S −→ R is continuous in x0 ∈ S if and only if the following conditions (i) if c < f (x0 ) there exists a neighborhood U of x0 in (S, O) such that for every x ∈ U we have c < f (x); (ii) if f (x0 ) < d there exists a neighborhood W of x0 in (S, O) such that for every x ∈ W we have f (x) < d are satisfied. If only one of these conditions are satisfied, we say that f is semicontinuous in x0 . Specifically, we have the following definition. Definition 4.46. Let (S, O) be a topological space. A function f : S −→ ˆ is lower semicontinuous at x0 ∈ S if for every c < f (x0 ) there is a R neighborhood U of x0 such that for every x ∈ U we have c < f (x); The function f is upper semicontinuous at x0 ∈ S if for every d > f (x0 ) there is a neighborhood W of x0 such that for every x ∈ W we have d > f (x). The function f is lower semicontinuous (upper-semicontinuous) on S if it is lower semicontinuous (upper semicontinuous) at every point of S. Observe that the function f is lower semicontinuous at x0 if for every c < f (x0 ) the set f −1 (c, ∞] is a neighborhood of x0 . Similarly, f is upper semicontinuous at x0 if the set f −1 [−∞, d) is a neighborhood of x0 for each d > f (x0 ). Since (−f )−1 (c, ∞] = {x ∈ S | −f (x) > c} = {x ∈ S | f (x) < −c} = −1 f [−∞, −c), it follows that −f is upper semicontinuous if and only if f is lower semicontinuous. ˆ be the function Example 4.47. Let f : R −→ R  −1 if x < 0, f (x) = 1 if x  0

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topology

for x ∈ R. Observe that {x ∈ R | f (x) < d} =

⎧ ⎪ ⎪∅ ⎨

R 1.

In each case, the set {x ∈ R | f (x) < d} is open in (R, O), so f is upper semicontinuous. Example 4.48. Let x be the largest integer less or equal to x and let ˆ x be the least integer greater or equal to x. The function f : R −→ R defined by f (x) = x, is upper semicontinuous because {x ∈ R | x  t} = (−∞, t] is closed for any t ∈ R. ˆ and g : S −→ R ˆ be two functions and let Lemma 4.4. Let f : S −→ R h = f + g. For r, s ∈ R define the set Wr,s = {x ∈ S | f (x) < r} ∩ {x ∈ S | g(x) < s}. If t ∈ R we have: {x ∈ S | f (x) + g(x) < t} =



Wr,s .

r+s=t

 Proof. If x ∈ r+s=t Wr,s there exists a pair of numbers (r, s) such that r + s = t, f (x) < r, and g(x) < s. This implies f (x) + g(x) < r + s = t, so x ∈ {x ∈ S | f (x) + g(x) < t}. Conversely, suppose that f (x) + g(x) < t. We need to show that there exist r, s such that f (x) < r, g(x) < s, and r + s = t. Let r = t − g(x) and s = t − f (x). It is clear that f (x) < r and g(x) < s, so x ∈ Wr,s and,   therefore, x ∈ r+s=t Wr,s . ˆ Theorem 4.105. Let (S, O) be a topological space and let f, g : S −→ R ˆ is upper be two upper semicontinuous functions. Their sum h : S −→ R semicontinuous. Proof. Since f, g are upper semicontinuous, the sets {x ∈ S | f (x) < r} and {x ∈ S | g(x) < s} are open for each r and s. Therefore, using the notations of Lemma 4.4, each sets Wr,s is open as the intersection of two open sets. Consequently, by the same lemma, the set {x ∈ S | f (x)+g(x) < t} is open for every t, so f + g is upper semicontinuous. 

May 2, 2018 11:28

232

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 232

Mathematical Analysis for Machine Learning and Data Mining

Example 4.49. Let (S, O) be a topological space. The characteristic function 1U of a subset U of S is lower semicontinuous if and only if U is open. Indeed, since ⎧ ⎪ ⎪ ⎨S if t < 0, −1 (1U ) (t, ∞] = U if 0  t < 1, ⎪ ⎪ ⎩∅ if t  1, it follows that (1U )−1 (t, ∞] ∈ O for each t ∈ R if and only if U ∈ O. The characteristic function 1W of a set W is upper semicontinuous if and only if W is a closed set. Indeed, let W be a set such that 1W is upper semicontinuous. This means that the function −1W is lower semicontinuous and, therefore, the function 1 − 1W is lower semicontinuous. Since 1S−W = 1 − 1W , it follows that 1S−W is lower semicontinuous, so S − W is an open set, which is equivalent to W being closed. The limits superior and inferior defined for nets allow us to apply these concepts for functions having numerical values. ˆ be a function. If Let (S, O) be a topological space and let f : S −→ R x0 ∈ S and (xV ) is a net indexed by neighx0 (O) we focus our attention on the numerical net (f (xV )) indexed by the same neighx0 (O). ˆ be a real-valued function defined on the Definition 4.47. Let f : S → R topological space (S, O). The limit inferior of f in x0 and the limit superior of f in x0 are the numbers lim inf x→x0 f and lim supx→x0 f given by lim inf f = sup inf{f (V ) | V ∈ neighx0 (O)},

(4.6)

lim sup f = inf sup{f (V ) | V ∈ neighx0 (O)},

(4.7)

x→x0 x→x0

respectively. Note that if r > 0 then lim inf x→x0 af (x) = a lim inf x→x0 f (x). ˆ then Example 4.50. If f : R −→ R, lim inf f = sup inf{f (x) | |x − x0 | < r},

(4.8)

lim sup f = inf sup{f (x) | |x − x0 | < r}.

(4.9)

x→x0 x→x0

r>0

r>0

Theorem 4.106. If f, g : S −→ rr ˆ are two functions (such that the additions in the equality below are defined), then lim inf (f + g)(x)  lim inf f (x) + lim inf g(x). x→x0

x→x0

x→x0

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Topology

b3234-main

page 233

233

Proof. If lim inf x→x0 f (x) = −∞ or lim inf x→x0 g(x) = −∞ the inequality is immediate. Otherwise, if a < lim inf x→x0 f (x) and b < lim inf x→x0 g(x) there exist U, V ∈ neighx0 (O) such that inf f (U ) > a and inf g(V ) > b, so inf(f + g)(U ∩ V ) > a + b. This implies lim inf x→x0 (f + g)(x)  a + b.  ˆ be Theorem 4.107. Let (S, O) be a topological space and let f : S −→ R a function. Then, f is lower semicontinuous if and only if (xi )i∈I being a convergent net in S with xi → x implies that f (x)  lim inf f (xi ). Proof. Suppose that f is lower semicontinuous and let t ∈ R such that t < f (x). Then, the set f −1 (t, ∞] is open and x ∈ f −1 (t, ∞]. Let (xi ) be a net such that and xi → x. Since xi → x, there is some it such that i  it implies xi ∈ f −1 (t, ∞]. In other words, i  it implies f (xi ) > t. This implies lim inf f (xi )  t. Since this holds for all t < f (x), it follows that f (x)  lim inf f (xi ). Conversely, suppose that xi → x implies f (x)  lim inf f (xi ). Let T = f −1 (−∞, t], where t ∈ R. If x ∈ K(T ), by Theorem 4.66, there is a net (xi ) in T such that xi → x. By hypothesis, f (x)  lim inf f (xi ). The definition of T implies that f (xi )  t for i ∈ I, hence f (x)  t. This means that x ∈ T , so T is closed and S − T = f −1 (t, ∞] is open, which means that f is lower semicontinuous.  Theorem 4.108. Let (S, O) be a topological space and let {fj | fj : X −→ ˆ for j ∈ J} be a collection of lower semicontinuous functions. Then, the R ˆ defined by f (x) = sup{fj (x) | j ∈ J} is a lower function f : S −→ R semicontinuous function. Proof.

Note that {x ∈ S | f (x)  t} =



{x ∈ S | fj (x)  t}.

j∈J

Since each fj is lower semicontinuous, each set {x ∈ S | fj (x)  t} is closed, so {x ∈ S | f (x)  t} is closed, which allows us to conclude that f is lower semicontinuous.  Corollary 4.22. Let (S, O) be a topological space and let {fj | fj : X −→ ˆ for j ∈ J} be a collection of upper semicontinuous functions. Then, the R ˆ defined by f (x) = inf{fj (x) | j ∈ J} is an upper function f : S −→ R semicontinuous function.

May 2, 2018 11:28

234

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 234

Mathematical Analysis for Machine Learning and Data Mining

Proof. This statement follows by applying Theorem 4.108 to the collecˆ for j ∈ J}.  tion of lower semicontinuous functions {−fj | fj : X −→ R Theorem 4.109. Let (S, O) be a compact topological space and let f : ˆ be a lower semicontinuous function. The function f attains its S −→ R minimum and the non-empty set of minimizers is compact. Proof. For a ∈ f (S) let Fa = {x ∈ S | f (x)  a}. Since f is lower semicontinuous the non-empty set Fa is closed. Note that the set of minimizers of f is  Fa . M= a∈f (S)

n The collection {Fa | a ∈ f (S)} has the f.i.p. because r=1 Fai = Fmin1rn ar . Since (S, O) is compact the set M is a compact and nonempty set.  Corollary 4.23. Let (S, O) be a compact topological space and let f : S −→ ˆ be a upper semicontinuous function. The function f attains its maximum R and the non-empty set of maximizers is compact. Proof. This statement follows by applying Theorem 4.109 to the lower semicontinuous functions −f .  Using semicontinuous functions it is possible to prove a form of Uryson’s lemma for locally compact spaces. Let (S, O) be a locally compact topological space and let Fc (S, O) be the set of continuous functions f : S −→ R such that K({x ∈ S | f (x) = 0}) is a compact set. Lemma 4.5. Let (S, O) be a locally compact topological space. The set Fc (S, O) is a linear space. Proof. Note that supp(f + g) ⊆ supp(f ) ∪ supp(g) and supp(f ) ∪ supp(g) is compact, hence supp(f + g) is compact. Thus, f + g ∈ Fc (S, O). The remaining argument follows immediately.  For an open set V of a topological space (S, O) we define the set Fc,V (S, O) = {f ∈ Fc (S, O) | f (x) ∈ [0, 1] for x ∈ S and supp(f ) ⊆ V }.

(4.10)

If C is a compact set in the same topological space, we also define FC,c (S, O) = {f ∈ Fc (S, O) | f (x) ∈ [0, 1] for x ∈ S and f (x) = 1 for x ∈ C}.

(4.11)

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Topology

b3234-main

page 235

235

Theorem 4.110. (Uryson’s Lemma for locally compact spaces) Let (S, O) be a locally compact, C be a compact subset, and V be an open subset such that C ⊆ V . Then, Fc,V (S, O) ∩ FC,c (S, O) = ∅. Proof. The set Q ∩ {0, 1} is countable. Let r0 = 0 and r1 = 1 and let r2 , r3 , . . . be a list of rational numbers in (0, 1). By Theorem 4.59 there exists open sets V0 and V1 such that C ⊆ V1 ⊆ K(V1 ) ⊆ V0 ⊆ K(V0 ) ⊆ V . Suppose that n  2 and Vr1 , . . . , Vrn are such that rp < rq implies K(Vrq ) ⊆ Vrp . Let ri = max{r | 1   n and ri < rn+1 }, rj = min{r | 1   n and ri > rn+1 }. By Theorem 4.59 there exists an open set Vrn+1 such that K(Vrj ) ⊆ Vrn+1 ⊆ K(Vrn+1 ) ⊆ Vri . This process produces a collection of open sets indexed by the rational numbers r ∈ [0, 1] such that C ⊆ V1 , K(V0 ) ⊆ V , each set K(Vr ) is compact, and s > r implies K(Vs ) ⊆ Vr . For r, s ∈ Q ∩ [0, 1] define the functions fr : S −→ [0, 1] and gs : S −→ [0, 1] by  r if x ∈ Vr , fr (x) = 0 otherwise, and

 gs (x) =

1 if x ∈ K(Vs ), s

otherwise,

for x ∈ S. The functions f, g : S −→ [0, 1] are defined as f sup fr and g = inf gs . r

s

We show next that f is the function whose existence is asserted by the theorem. Example 4.49 implies that the functions fr are lower semicontinuous and gs are upper semicontinuous. By Theorem 4.108 and Corollary 4.22, f is a lower semicontinuous function and g is an upper semicontinuous function. Suppose that fr (x) > gs (x) for some x ∈ S. We have r > s, x ∈ Vr , and x ∈ K(Vs ). This leads to a contradiction because r > s implies Vr ⊆ Vs . Therefore, we have fr (x)  gs (x) for all r, s, so f  g. Suppose that there exists x such that f (x) < g(x). Let r, s ∈ Q ∩ [0, 1] such that f (x) < r < s < g(x). Since f (x) < r, we have x ∈ Vr . Since g(x) > s, we have x ∈ K(Vs ), which is a contradiction. This f = g. Note

May 2, 2018 11:28

236

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 236

Mathematical Analysis for Machine Learning and Data Mining

that the function f satisfies the conditions of the theorem. Indeed, since f = g, f is both lower and upper semicontinuous, so it is a continuous function. Furthermore, f (x) ∈ [0, 1], f (x) = 1 if x ∈ C and K({x | f (x) =  0}) ⊆ V , so f ∈ Fc,V (S, O) ∩ FC,c (S, calo). Corollary 4.24. Let (S, O) be a locally compact topological space, and let C1 , C2 be two disjoint compact subsets of S. There exists a function f ∈ Fc (S, O) such that f (x) ∈ [0, 1] for x ∈ S, f (x) = 1 for x ∈ C1 , and f (x) = 0 for x ∈ C2 . Proof. Since (S, O) is a Hausdorff space, by Theorem 4.51, C2 is closed. Thus, S − C2 is an open set and C1 ⊆ S − C2 . By Uryson’s Lemma, there exists a function f ∈ Fc,S−C2 (S, O) ∩ FC1 ,c (S, O). Therefore, f (x) = 1 for x ∈ C1 and supp(f ) ⊆ S − C2 . The last inclusion means that f (x) = 0 for  x ∈ C2 . Theorem 4.111. (Partition of Unity Theorem) Let (S, O) be a locally compact space and let V1 , . . . , Vn a collection of open sets. If C is a compact n set such that C ⊆ i=1 Vn , there exist n continuous functions h1 , . . . , hn in Fc (S, O) such that hi (x) ∈ [0, 1], supp(hi ) ⊆ Vi and h1 (x) + · · · + hn (x) = 1 for x ∈ C. Proof. Since (S, O) is a locally compact space, by Theorem 4.59, for each x ∈ C there exists a neighborhood Wx such that K(Wx ) ⊆ Vi . The compactness of C means that there exist x1 , . . . , xn such that C ⊆ Wx1 ∪  · · · ∪ Wxn . Let Hi = {K(Wxj ) | K(Wxj ) ⊆ Vi }. By Uryson’s Lemma for locally compact spaces (Theorem 4.110) there exist continuous functions gi such that g(x) ∈ [0, 1] for x ∈ S, K({x | gi (x) = 0}) is compact and included in Vi and g(x) = 1 for x ∈ Hi . Define h1 , . . . , hn as h1 = g 1 h2 = (1 − g1 )g2 .. . hn = (1 − g1 )(1 − g2 ) · · · (1 − gxn−1 )gn . Note that for each of the functions hi we have 0  hi (x)  1 for x ∈ S. Also, {x | hi (x) = 0} ⊆ {x | gi (x) = 0}, hence K({x | hi (x) = 0}) ⊆ Vi . It is easy to see that h1 + h2 + · · · + hn = 1 − (1 − g1 )(1 − g2 ) · · · (1 − gn ). Since C ⊆ H1 ∪ H2 ∪ · · · ∪ Hn , there exists i such that gi (x) = 1 if x ∈ C,  hence h1 (x) + · · · + hn (x) = 1 for x ∈ C.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topology

4.16

page 237

237

The Epigraph and the Hypograph of a Function

ˆ be Definition 4.48. Let (S, O) be a topological space and let f : S −→ R a function. Its epigraph is the set epi(f ) = {(x, y) ∈ S × R | f (x)  y}. The hypograph of f is the set hyp(f ) = {(x, y) ∈ S × R | y  f (x)}. The epigraph of a function f : R −→ R is the dotted area in R2 located above the graph of the function f and it is shown in Figure 4.2(a); the hypograph of f is the dotted area below the graph shown in Figure 4.2(b). y

y

x

(a) Fig. 4.2

x

(b)

Epigraph (a) and hypograph (b) of a function f : R −→

R.

Note that the intersection epi(f ) ∩ hyp(f ) = {(x, y) ∈ S × R | y = f (x)} is the graph of the function f . If f (x) = ∞, then (x, ∞) ∈ epi(f ). Thus, for the function f∞ defined by f∞ (x) = ∞ for x ∈ S we have epi(f∞ ) = ∅. ˆ be a Theorem 4.112. Let (S, O) be a topological space and let f : S −→ R function. Then, f is lower semicontinuous if and only if epi(f ) is a closed set in S × R. Proof. Suppose that f is a lower semicontinuous function and let (xi , ai )i−→I be a set in epi(f ) such that limI (xi , ai ) = (x, a). We have limI xi = x and limI ai = a. By the definition of epi(f ) we have f (xi )  ai for i ∈ I.

May 2, 2018 11:28

238

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 238

Mathematical Analysis for Machine Learning and Data Mining

By Theorem 4.107, if (xi )i∈I is a convergent net in S with xi → x, then f (x)  lim inf f (xi ), hence f (x)  limI ai = a. Thus, (x, a) ∈ epi(f ), so epi(f ) is closed. Conversely, suppose that epi(f ) is closed in S ×R. By a previous remark it suffices to show that the set Ut = {x ∈ S | f (x)  t} is closed for any t ∈ R. Let (xi ) be a net in Ut such that xi → x. Since f (xi )  t, we have (xi , t) ∈ epi(f ) for i ∈ I. Since epi(f ) is closed and (xi , t) → (x, t), we have  (x, t) ∈ epi(f ) so f (x)  t, which implies that Ut is closed for each t. ˆ be a Corollary 4.25. Let (S, O) be a topological space and let f : S −→ R function. Then, f is upper semicontinuous if and only if hyp(f ) is a closed set in S × R. Proof.

This follows from an application of Theorem 4.112 to −f .



The notion of level set of a function to be introduced next allows yet another characterization of semicontinuous functions. ˆ be a Definition 4.49. Let (S, O) be a topological space, f : S −→ R ˆ The level set for f at a is the set function and let a ∈ R. Lf,a = {x ∈ S | f (x)  a}. ˆ be Theorem 4.113. Let (S, O) be a topological space and let f : S −→ R a function. The following statements are equivalent: (i) f is lower semicontinuous on S; (ii) for each a ∈ R the level set Lf,a is closed; (iii) if (xi ) is a net such that xi → x then f (x)  lim inf f (xi ). Proof. (i) implies (ii): Let f be a lower semicontinuous function on S. Since Lf,a × {a} = epi(f ) ∩ (S × {a}), Lf,a × {a} is closed because S × {a} is closed in S × R. This implies that Lf,a is closed. (ii) implies (iii): Suppose that for every a ∈ R, the set Lf,a is closed and let (xi ) be a sequence such that xi → x. Since f (xi )  lim inf f (xi ), it follows that xi ∈ Lf,lim inf f (xi ) . Taking into account that Lf,lim inf f (xi ) is a closed set, it follows that x ∈ Lf,lim inf f (xi ) because xi → x. Consequently, f (x)  lim inf f (xi ). (iii) implies (i): Let (xi , yi ) be a net in epi(f ) such that xi → x and yi → y. By (iii), we have f (x)  lim inf f (xi ). Since (xi , yi ) ∈ epi(f ) we have f (xi )  yi , so lim inf f (xi )  lim inf yi = y. Thus, f (x)  y, so (x, y) ∈ epi(f ). This proves that epi(f ) is closed. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topology

page 239

239

A similar statement that involves hypographs is given next. ˆ be Theorem 4.114. Let (S, O) be a topological space and let f : S −→ R a function. The following statements are equivalent: (i) f is upper semicontinuous on S; (ii) for each a ∈ R the set L−f,a is closed relative to S; (iii) if (xi ) is a sequence such that xi → x and the limit of the sequence (f (xn )) exists, then lim sup f (xi )  f (x). Proof.

The proof is similar to the proof of Theorem 4.113.



Exercises and Supplements (1) Prove that the family of subsets {(−n, n) | n ∈ N} ∪ {∅, R} is a topology on R. (2) Let S be a set and let s0 be an element of S. Prove that the family of subsets Os0 = {L ∈ P(S) | s0 ∈ L} ∪ {∅} is a topology on S. (3) Let S be a set and let s0 be such that s0 ∈ S. Prove that: (a) the family of subsets Os0 = {L ∈ P(S ∪ {s0 }) | s0 ∈ L} ∪ {∅} is a topology on S; (b) the singleton {s0 } is compact but not closed in this topology. (4) Let (S, O) be a topological space, L be an open set in (S, O), and H be a closed set. (a) Prove that a set V is open in the subspace (L, O L ) if and only if V is open in (S, O) and V ⊆ L; (b) Prove that a set W is closed in the subspace (H, O H ) if and only if W is closed in (S, O) and W ⊆ H. (5) Let (S, O) be a topological space where O = {∅, U, V, S}, where U and V are two subsets of S. Prove that either {U, V } is a partition of S or one of the sets {U, V } is included in the other. (6) Let S be the set of subsets of R such that, for every U ∈ S, x ∈ U implies −x ∈ U . Prove that {∅} ∪ S is a topology on R. (7) Let (S, O) be a topological space, T be a set, and let f : T −→ S be a function. Prove that the collection f −1 (O) defined by f −1 (O) = {f −1 (X) | X ∈ O} is a topology on T ; furthermore, prove that f −1 (O) is the weakest topology O on T such that f is a continuous function between (T, O ) and (S, O). The topology f −1 (O) is known as the pullback topology induced by f . Solution: It is easy to verify that f −1 (O) is indeed a topology on T . Suppose that f is a continuous function between the topological spaces (T, O ) and (S, O). Then, for Y ∈ O, f −1 (Y ) ∈ O , so f −1 (O) ⊆ O .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 240

Mathematical Analysis for Machine Learning and Data Mining

240

(8) Let (S, O) be a topological space, T be a set, and let f : S −→ T be a function. Prove that the collection O = {V | V ⊆ T, f −1 (V ) ∈ O} is a topology on T ; furthermore, prove that O is the finest topology on T such that f is a continuous function between and (S, O) and (T, O ). The topology f −1 (O) is known as the push-forward topology induced by f . Solution: Since f −1 (V1 )∩f −1 (V2 ) = f −1 (V1 ∩V2 ), it follows that V1 ∈ {Vi | i ∈I} ⊆ O we have O and V2 ∈ O imply V1 ∩ V2 ∈ O . Also, if −1 −1 −1 means that f (Vi ) ∈ O. f (Vi ) ∈ O for i∈I Vi i∈I f  i ∈ I, which   This implies i∈I Vi ∈ O , so O is indeed a topology. Suppose that O1 is a topology on T such that f : S −→ T is a continuous function between the topological spaces (S, O) and (T, O1 ). Then, for every open set W ∈ O1 we have f −1 (W ) ∈ O, so W ∈ O . This shows that O1 ⊆ O , which shows that O is the finest topology on T such that f is a continuous function. Theorem 4.1 allows the introduction the notion of size of an open set in R . Let U be an open set in R, which is a union of open intervals, U = j∈J Ij (here  J may be a finite set, or the entire set {n | n  1}). Its size (U ) is defined as j∈J l(Ij ), where l(Ij ) is the length of the open interval Ij = (aj , bj ), that is, l(Ij ) = bj − aj . Since the terms of the sum are positive, the value of this sum is independent of the order of its terms. We have 0  (U )  ∞ and (∅) = 0. Note that for an open interval I = (a, b) we have (I) = l(I) = b − a. (9) Prove that the size of the open subset U =



1 n1 (n, n+ n )

of R is infinite.

(10) Prove that if U is a bounded open set and U ⊆ (a, b), then (U )  b − a.  1 , n + 21n ) in R has finite size (11) Prove that the open set W = n1 (n + 2n+1 but is not bounded. (12) Let U, V be two open sets in R such that U ⊆ V . Prove that (U )  (V ) and that (U ) + (V ) = (U ∪ V ) + (U ∩ V ). (13)  Prove that if (Un )n1 is an increasing chain of open sets in R and U = n∈N Un , then (U ) = sup{(Un ) | n  1}.

Solution: Since Un ⊆ U for n ∈ N, we have (Un )  (U ), so sup{(Un ) | n  1}  (U ).  Suppose that U = {Ij | j ∈ J}, where {Ij | j ∈ J} is a finite or infinite sequence of disjoint and open intervals and let  > 0.  If (U ) is finite, then there exists h ∈ N such that hj=1 (Ij )  (U ) − b −a

 Let η > 0 be such that η < 4h and η < j 2 j for 1  j  h. = (aj + η, bj −  η) and Kj = [aj + η, bj − η] for 1  j  k, Define Lj  and L = hj=1 Lj , K = hj=1 Kj . The set K is a compact set in R that is covered by {Un | n  1}. Thus, there exists m ∈ N such that  . 2

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topology

page 241

241

L ⊆ K ⊆ Um . If n  m, then (Un )  (Um )  (L) =

h 

(bj − aj − 2η)

j=1



h  j=1

(bj − aj ) −

  (U ) − . 2

This shows that (U ) = sup{(Un ) | n  1}. The case when (U ) = ∞ is left to the reader. (14) Let U1 , U2 be two bounded open sets in R and let K be a compact subset of R such that K ⊆ U1 ∩ U2 . Prove that (U1 ) − (U1 − K) = (U2 ) − (U2 − K). Solution: Note that U1 − K, U2 − K and (U1 ∩ U2 ) − K are open sets. It is clear that U1 ∪ (U2 − K) = U1 ∪ U2 , and U1 ∩ (U2 − K) = (U1 ∩ U2 ) − K. Therefore, (U1 ∪ U2 ) = (U1 ∪ (U2 − K)) = (U1 ) + (U2 − K) − (U1 ∩ (U2 − K) = (U1 ) + (U2 − K) − ((U1 ∩ U2 ) − K), hence (U1 ) + (U2 − K) = (U1 ∪ U2 ) + ((U1 ∩ U2 ) − K). Similarly, (U2 ) + (U1 − K) = (U1 ∪ U2 ) + ((U1 ∩ U2 ) − K), hence (U1 ) + (U2 − K) = (U2 ) + (U1 − K). (15) Let K be a compact subset of R. Supplement 14 shows that the number s(K) = (U ) − (U ∩ K) is the same for every open set in R that contains K. Prove that (a) s([a, b]) = b − a; (b) if K1 , K2 are compact subsets of R, then K1 ⊆ K2 implies s(K1 )  s(K2 ); (c) if  (Kn ) is a decreasing sequence of compact subsets of R and K = n1 Kn , then s(K) = limn→∞ s(Kn ); (d) if K1 , K2 are compact subsets of R, then s(K1 ∪K2 )+s(K1 ∩K2 ) = s(K1 ) + s(K2 ). (16) Prove that if U is a closed set in a topological space (S, O) then I(S−U ) = S − U.

May 2, 2018 11:28

242

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 242

Mathematical Analysis for Machine Learning and Data Mining

(17) Let S be a set and let H : P(S) −→ P(S) be a mapping. Prove that H is a closure operator on S if and only if U ∪ H(U ) ∪ H(H(V )) = H(U ∪ V ) − H(∅) for all U, V ∈ P(S). (18) Let (S, O) be a topological space and let I be its interior operator. Prove that  the poset of open  sets (O, ⊆) is a complete lattice, where sup L = L and inf L = I ( L) for every family of open sets L. (19) Let (S, O) be a topological space, let K be its interior operator and let K be its collection  of closed sets. Prove  that (K, ⊆) is a complete lattice, where sup L = K ( L) and inf L = L for every family of closed sets. (20) Prove that if U, V are two subsets of a topological space (S, O), then K(U ∩ V ) ⊆ K(U ) ∩ K(V ). Formulate an example where this inclusion is strict. (21) Let T be a subspace of the topological space (S, O). Let KS , IS , and ∂S be the closure, interior and boundary operators associated to S and KT , IT and ∂T the corresponding operators associated to T . Prove that (a) KT (U ) = KS (U ) ∩ T , (b) IS (U ) ⊆ IT (U ), and (c) ∂T U ⊆ ∂S U for every subset U of T . (22) Let (S, O) be a topological space and let K and I be its associated closure and interior operator, respectively. Define the mappings φ, ψ : P(S) −→ P(S) by φ(U ) = I(K(U )) and ψ(U ) = K(I(U )) for U ∈ P(S). (a) Prove that φ(U ) is an open set and ψ(U ) is a closed set for every set U ∈ P(S). (b) Prove that ψ(H) ⊆ H for every closed set H and L ⊆ φ(L) for every open set L. (c) Prove that φ(φ(U )) = φ(U ) and ψ(ψ(U )) = ψ(U ) for every U ∈ P(S). (d) Let (J1 , . . . , Jn ) be a sequence such that Ji ∈ {K, I}. Prove that there are at most seven distinct sets of the form Jn (· · · (J1 (U )) · · · ) for every set U ∈ P(S), and give an example of a topological space (S, O) and a subset U of S such that these seven sets are pairwise distinct. (23) Let (S, O) be a topological space. The subsets X and Y are said to be separated if X ∩ K(Y ) = K(X) ∩ Y = ∅. (a) Prove that X and Y are separated sets in (S, O) if and only if they are disjoint and clopen in the subspace X ∪ Y . (b) Prove that two disjoint open sets or two disjoint closed sets in (S, O) are separated.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topology

page 243

243

(24) Let (S, O) be a topological space and let X, Y be two closed sets such that I(X ∪ Y ) = ∅. Prove that at least one of the sets I(X) or I(Y ) is non-empty. Solution: Suppose that X and Y are two closed sets such that both I(X) = I(Y ) = ∅. By the second part of Theorem 4.11, we have I(X ∪ Y ) = ∅. Since this is not the case, at least one of the sets I(X) or I(Y ) must be non-empty. (25) Let (S, O) be a topological space. Prove that • ∂(∂U ) ⊆ ∂U for every U ⊆ S; • if U is open or closed, then ∂(∂U ) = ∂U ; • for every subset U of S we have ∂(∂(∂U )) = ∂(∂U ). (26) Let (S, O) is a topological space. Prove that a subset T of S is closed if and only for each x ∈ S and V ∈ neighx (O), V ∩ T = ∅ implies x ∈ T . (27) Prove that if U, V are two subsets of a topological space (S, O), then K(U ∩ V ) ⊆ K(U ) ∩ K(V ). Formulate an example where this inclusion is strict. (28) Let (S, O) be a topological space. Prove that for every subset T of S we have S = I(T ) ∪ I(S − T ) ∪ ∂S T. (29) Let (S, O) be a topological space and let K and I be its associated closure and interior operator, respectively. Define the mappings φ, ψ : P(S) −→ P(S) by φ(U ) = I(K(U )) and ψ(U ) = K(I(U )) for U ∈ P(S). (a) Prove that φ(U ) is an open set and ψ(U ) is a closed set for every set U ∈ P(S). (b) Prove that ψ(H) ⊆ H for every closed set H and L ⊆ φ(L) for every open set L. (c) Prove that φ(φ(U )) = φ(U ) and ψ(ψ(U )) = ψ(U ) for every U ∈ P(S). (d) Let (J1 , . . . , Jn ) be a sequence such that Ji ∈ {K, I}. Prove that there are at most seven distinct sets of the form Jn (· · · (J1 (U )) · · · ) for every set U ∈ P(S), and give an example of a topological space (S, O) and a subset U of S such that these seven sets are pairwise distinct. (30) Let (S, O) be a topological space, and U and U  be two subsets of S. (a) Prove that ∂(U ∪ V ) ⊆ ∂U ∪ ∂V . (b) Prove that ∂U = ∂(S − U ). (31) Let B be a basis for a topological space (S, O). Prove that if B is a collection of subsets of S such that B ⊆ B ⊆ O, then B is a basis for O. (32) Let S = {a, b, c} and let S = {{a, b}, {a, c}}. Determine the topology generated by the collection S.

May 2, 2018 11:28

244

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 244

Mathematical Analysis for Machine Learning and Data Mining

(33) Let C be the family of open intervals C = {(a, b) | a, b ∈ R and ab > 0}. Prove that: (a) Every open set L of (R, O) contains a member of C. (b) C is not a basis for the topology O.  (34) Let C be a chain of subsets of a set S such that C = S. Prove that C is the basis of a topology. (35) Let B be a basis for a topological space (S, O). Prove that if B is a collection of subsets of S such that B ⊆ B ⊆ O, then B is a basis for O. (36) Let (S, O) be a topological space, U and U  two subsets of S, and B and B two bases in the subspaces (U, O U ) and (U  , O U  ), respectively. Prove that B ∨ B is a basis in the subspace U ∪ U  . Solution: Let M be an open set in the subspace U ∪ V . By the definition of the subspace topology, there exists an open set L ∈ O such that M = L ∩ (U ∪ V ) = (L ∩ U ) ∪ (L ∩ V ), so L is the union of two open sets, L ∩ U and L ∩ U  , in the subspaces U and U  . Since  B is a basis in U , there is a subcollection B1 such that L ∩ U = B1 . Similarly, asubcollection B1 such that L ∩ U = B1 . Therefore, B contains   M = B1 ∪ B1 = B1 ∨ B1 . (37) Let S be an uncountable set and let (S, O) be the cofinite topology on S. (a) Prove that every non-finite set is dense. (b) Prove that there is no countable basis for this topological space. What does this say about Theorem 4.34? (38) Prove that if (S, O) is a topological space such that O is finite, then (S, O) is compact. (39) Let (S, O) be a compact space and let H = (H0 , H1 , . . .) be a nonincreasing sequence of non-empty and closed subsets of S. Prove that  i∈N Hi is non-empty. (40) Let U = {u ∈ R2 | u1 > 0 and u2  0 and v2  − v11 }. Prove that:

1 } u1

and V = {v ∈ R2 | v1
0 such that if x ∈ B(x0 , δ) then − 2 < f (x) − f (x0 ) < 2 . Since x0 ∈ K(S), there exists x1 ∈ S such that x1 ∈ B(x0 , δ), so −

  < f (x1 ) − f (x0 ) < 2 2

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Topology

9in x 6in

b3234-main

page 249

249

for some x1 ∈ S. Therefore, f (x1 ) > f (x0 ) − 2  r −  for every  > 0, which implies the reverse inequality. If sup{f (x) | x ∈ K(S)} = ∞ and let a ∈ R. There exists x0 ∈ K(S) such that f (x0 )  a + 1. Since f is continuous, there exists δ > 0 such that if x − x0  < δ, we have −1 < f (x) − f (x0 ) < 1. Since x0 ∈ K(S) there exists x1 ∈ S, so such that x − x0  < δ, so f (x1 ) > f (x0 ) − 1  a, so sup{f (x) | x ∈ S} = ∞. The proof of the remaining equality is left to the reader. (65) Let (S, Os0 ) be the topological space defined in Exercise 2, where s0 ∈ S. Prove that any continuous function f : S −→ R is a constant function. (66) Let (S1 , O1 ) and (S2 , O2 ) be two topological spaces and let f : S1 −→ S2 be a continuous surjective function. Prove that if (S2 , O2 ) is compact, then (S1 , O1 ) is compact. (67) Let f : R −→ R be a continuous function defined on the topological space (R, O). Prove that if f (q) = 0 for every q ∈ Q, then f (x) = 0 for every x ∈ R. (68) Let f : R −→ R be a continuous function in x0 . Prove that if f (x0 ) > 0, then there exists an open interval (a, b) such that x0 ∈ (a, b) and f (x) > 0 for every x ∈ (a, b). (69) Let (S, O) and (T, O ) be two topological spaces and let B be a basis of (T, O ). Prove that f : S −→ T is continuous if and only if f −1 (B) ∈ O for every B ∈ B . (70) Let (S, O) be a topological space and let f : S −→ R. Prove that a function is continuous at x0 ∈ S if and only if the following conditions are satisfied: (a) for every s < f (x0 ) there is U ∈ neighx0 (O) such that for every x ∈ U we have s < f (x); (b) for every t > f (x0 ) there is V ∈ neighx0 (O) such that for every x ∈ V we have t > f (x). (71) Prove that a function f : S −→ T is open if there is some basis B of the topological space (S, O) such that f (B) is open for each B ∈ B. (72) Prove that a bijection f : S −→ T is an open function if and only if it is a closed mapping. Hint: Use the identity T − f (U ) = f (S − U ) satisfied by any bijection f : S −→ T for each U ∈ P(S). (73) Prove that the function f : R −→ R defined by f (x) = x2 for x ∈ R is continuous but not open. (74) Prove that if a < b and c < d, then the subspaces [a, b] and [c, d] of (R, O) are homeomorphic. (75) Let (S, O) and (T, O ) be two topological spaces and let f : S −→ T be a bijection. Prove that

May 2, 2018 11:28

250

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 250

Mathematical Analysis for Machine Learning and Data Mining

(a) f is a closed function if and only if it is an open function; (b) f is a homeomorphism if and only if f is an open function. (76) Let (S, O) and (T, O ) be two topological spaces and let B be a basis for (S, O). Prove that if f is a homeomorphism, then f (B) is a basis for (T, O ). (77) Let (S, O) be a connected topological space and f : S −→ R be a continuous function. Prove that if x, y ∈ S, then for every r ∈ [f (x), f (y)] there is z ∈ S such that f (z) = r. (78) Let a and b be two real numbers such that a  b. Prove that if f : [a, b] −→ [a, b] is a continuous function, then there is c ∈ [a, b] such that f (c) = c. (79) Prove that a topological space (S, O) is connected if and only if ∂T = ∅ implies T ∈ {∅, S} for every T ∈ P(S). Let (S, O) be a topological space and let x and y be two elements of S. A continuous path between x and y is a continuous function f : [0, 1] −→ S such that f (0) = x and f (1) = y. We refer to x as the origin and to y as the destination of f . (S, O) is said to be arcwise connected if any two points x and y are the origin and destination of a continuous path. (80) Prove that any arcwise connected topological space is connected. (81) Prove that, for a topological space (S, O), the following statements are equivalent: (a) (S, O) is connected. (b) If S = L1 ∪ L2 and L1 ∩ L2 = ∅, where L1 and L2 are open, then L1 = ∅ or L2 = ∅. (c) If S = H1 ∪ H2 and H1 ∩ H2 = ∅, where H1 and H2 are closed, then H1 = ∅ or H2 = ∅. (d) If K is a clopen set, then K = ∅ or K = S. (82) Prove that any subspace of a totally disconnected topological space is totally disconnected, and prove that a product of totally disconnected topological spaces is totally disconnected. (83) Let (S, O) be a connected topological space and f : S −→ R be a continuous function. Prove that if x, y ∈ S, then for every r ∈ [f (x), f (y)] there is z ∈ S such that f (z) = r. (84) Prove that, for a topological space (S, O), the following statements are equivalent: (a) (S, O) is connected. (b) If S = L1 ∪ L2 and L1 ∩ L2 = ∅, where L1 and L2 are open, then L1 = ∅ or L2 = ∅.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Topology

b3234-main

page 251

251

(c) If S = H1 ∪ H2 and H1 ∩ H2 = ∅, where H1 and H2 are closed, then H1 = ∅ or H2 = ∅. (d) If K is a clopen set, then K = ∅ or K = S. (85) Prove that any subspace of a totally disconnected topological space is totally disconnected, and prove that a product of totally disconnected topological spaces is totally disconnected. (86) Prove that if (S, P(S)) and (S  , P(S  )) are two discrete topological spaces, then their product is a discrete topological space. (87) Let (S, O), (S, O ) be two topological spaces. Prove that the collection {S × L | L ∈ O } ∪ {L × S  | L ∈ O} is a sub-basis for the product topology O × O . (88) Let {(Si , Oi ) | i ∈ I}  be a collection of topological spaces. Prove that each projection pj : i∈I Si −→ Sj is a open mapping. (89) Prove that the following classes of topological spaces are closed with respect to the product of topological spaces: (a) the class of spaces that satisfy the first axiom of countability; (b) the class of spaces that satisfy the second axiom of countability; (c) the class of separable spaces. (90) Let {(Si , Oi ) | i ∈ I} be a family oftopological spaces indexed by the set I and let (xj ) be a J-net  on S = i∈I Si so that xj = (xji ) for i ∈ I. Prove that if y = (yi ) ∈ i∈I Si , then xj → y in S if and only if xji → yi in Si for every i ∈ I. (91) Let (S, O), (S, O ) be two topological spaces and let (S × S  , O × O ) be their product. (a) Prove that for all sets T, T  such that T ⊆ S and T  ⊆ S  , K(T × T  ) = K(T ) × K(T  ) and I(T × T  ) = I(T ) × I(T  ). (b) Prove that ∂(T × T  ) = (∂(T ) × K(T  )) ∪ (K(T ) × ∂T  ). (92) Let S be a subset of Rn . Prove that if S is closed, then its characteristic function 1S is upper semicontinuous on Rn ; also, if S is open, then 1S is lower semicontinuous on Rn . Solution: Let S be an open subset of Rn . Note that L1S ,a = {x ∈ S | 1S (x)  a} ⎧ ⎪ if a < 0, ⎨∅ = Rn − S if 0  a < 1, ⎪ ⎩ n R if 1  a. Thus, in every case, a level set L1S ,a is closed. By Theorem 4.113, 1S is lower semicontinuous on S. The argument for the case when S is closed is similar.

May 2, 2018 11:28

252

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 252

Mathematical Analysis for Machine Learning and Data Mining

(93) Let a and b be two real numbers such that a  b. Prove that if f : [a, b] −→ [a, b] is a continuous function, then there is c ∈ [a, b] such that f (c) = c. A function f : Rn −→ R is coercive if for any number M > 0 there exists a number rM such that x > rM implies f (x) > M . In other words, f is coercive if limx→∞ f (x) = ∞. (94) Prove that a lower semicontinuous and coercive function f : Rn −→ R attains its infimum and is bounded below. Solution: Let (xm ) be a sequence such that limm→∞ f (xm ) = inf f (x). Such a sequence exists because for every m  1 there ex1 . Since f is coercive, the seists xm such that f (xm ) < inf f + m quence (xm ) is bounded, so it contains a convergent subsequence (xmk ). Let x = limk→∞ f (xmk ). By part (iii) of Theorem 4.113 we have f (x)  limk→∞ f (xmk ) = inf f , hence f (x) = inf f . (95) Prove that the function g : R2 −→ R given by g(x) = x21 + x22 for x ∈ R2 is coercive. (96) Prove that the function h : R2 −→ R given by h(x) = (x1 − x2 )2 for x ∈ R2 is not coercive. (97) Let p be a polynomial, p(x) = a0 xn + a1 xn−1 + · · · + an , where ai ∈ R for 0  i  n. Prove that the function f : R −→ R defined by f (x) = |p(x)| is coercive. (98) Let f : Rn −→ R be a continuous function. Prove that f is coercive if and only if each level set Lf,a is compact. Solution: Since f is continuous each level set Lf,a is closed. Thus, it suffices to show that each level set is bounded. Suppose that there exists a such that Lf,a is unbounded. Then, there exists a sequence (xn ) in Lf,a such that limn→∞ xn  = ∞. The coercivity of f implies that limn→∞ f (xn ) = ∞, which contradicts the fact that f (xn )  a. Thus, each level set Lf,a is closed and bounded and is, therefore, compact. Conversely, suppose that each level set Lf,a is compact and, therefore, is bounded. To prove that f is coercive it suffices to show that for a sequence (xn ) such that limn→∞ xn  = ∞, the sequence (f (xn )) contains no bounded subsequence. Suppose that the sequence (f (xn )) contains a bounded subsequence (f (xni ))i∈I and let b be an upper bound of this sequence. Then, {xni | i ∈ I} ⊆ Lf,b , which is contradictory because no subsequence of (xn ) can be bounded. (99) Let f : Rn −→ R be a continuous function. Prove that if f is coercive, then f has a global minimum.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Topology

9in x 6in

b3234-main

page 253

253

Solution: Let Lf,a be a level set that is non-empty. Since f is coercive, Lf,a is compact. By Corollary 5.11 f attains a minimum in some x0 ∈ Lf,a . It follows that x0 is a global minimum for f .

Bibliographical Comments There are several excellent classic references on general topology ([89, 42, 47, 43, 135]). A very readable introduction to topology is [53]. Pioneering work in applying topology in data mining has been done in [109, 92]. The one-axiom for closure operators given in Exercise 17 was obtained in [110]. Supplements from 9 to 15 which offer a modality to introduce the Lebesgue measure on R originate in [63]. A modern, up-to-date introduction is [2].

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 255

Chapter 5

Metric Space Topologies

5.1

Introduction

In this chapter we study an hierarchy of topological spaces that comprises topological metric spaces, topological normed spaces, and topological inner product spaces. This hierarchy is based on the fact that metric spaces can be equipped with topologies; since norms generate metrics and inner products generate norms, increasingly specialized topologies can be defined on normed spaces and on inner product spaces. Theorem 5.1. Let (S, d) be a metric space. The collection Od defined by Od = {L ∈ P(S) | for each x ∈ L there exists > 0 such that B(x, ) ⊆ L} is a topology on the set S. Proof. We have ∅ ∈ Od because there is no x in ∅, so the condition of the definition of Od is vacuously satisfied. The set S belongs to Od because B(x, ) ⊆ S for every x ∈ S and every positive number .  If {Ui | i ∈ I} ⊆ Od and x ∈ {Ui | i ∈ I}, then x ∈ Uj for some j ∈ I. Then, there exists > 0 such that B(x, ) ⊆ Uj and therefore   B(x, ) ⊆ {Ui | i ∈ I}. Thus, {Ui | i ∈ I} ∈ Od . Finally, let U, V ∈ Od and let x ∈ U ∩V . Since U ∈ Od , there exists > 0 such that B(x, ) ⊆ U . Similarly, there exists  such that B(x,  ) ⊆ V . If 1 = min{ ,  }, then B(x, 1 ) ⊆ B(x, ) ∩ B(x,  ) ⊆ U ∩ V , so U ∩ V ∈ Od . This concludes the argument.  Theorem 5.1 implies that the collection of spheres {B(x, ) | > 0} is a local basis at x for any point x of a metric space (S, d). Definition 5.1. Let d be a metric on a set S. The topology induced by d is the family of sets Od . 255

May 2, 2018 11:28

256

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 256

Mathematical Analysis for Machine Learning and Data Mining

We refer to the pair (S, Od ) as a topological metric space. Example 5.1. The usual topology of the set of real numbers R introduced in Example 4.3 is actually induced by the metric d : R × R −→ R0 given by d(x, y) = |x − y| for x, y ∈ R. Recall that, by Theorem 4.1, every open set of this space is the union of a countable set of disjoint open intervals. The corresponding metric topology of complex numbers C has a similar definition using d(z1 , z2 ) = |z1 − z2 | for z1 , z2 ∈ C. Example 5.2. The topology on Rn induced by the metric d defined by % & & n d(x, y) = ' (xj − yj )2 j=1

for x, y ∈ R is known as the Euclidean topology on Rn . n

A topological space (S, O) is metrizable if there exists a metric d : S × S −→ R0 such that O = Od . The next statement explains the terms “open sphere” and “closed sphere,” which we have used previously. Theorem 5.2. Let (S, Od ) be a topological metric space. If t ∈ S and r > 0, then any open sphere B(t, r) is an open set and any closed sphere B[t, r] is a closed set in the topological space (S, Od ). Proof. Let x ∈ B(t, r), so d(t, x) < r. Choose such that < r − d(t, x). We claim that B(x, ) ⊆ B(t, r). Indeed, let z ∈ B(x, ). We have d(x, z) < < r − d(t, x). Therefore, d(z, t)  d(z, x) + d(x, t) < r, so z ∈ B(t, r), which implies B(x, ) ⊆ B(t, r). We conclude that B(t, r) is an open set. To show that the closed sphere B[t, r] is a closed set, we will prove that its complement S − B[t, r] = {u ∈ S | d(u, t) > r} is an open set. Let v ∈ S − B[t, r]. Now choose such that < d(v, t) − r. It is easy to see that B(v, ) ⊆ S − B[t, r], which proves that S − B[t, r] is an open set.  Corollary 5.1. The collection of all open spheres in a topological metric space (S, Od ) is a basis for this topological space. Proof.

This statement follows immediately from Theorem 5.2.



Note that every metric space (S, Od ) satisfies the first axiom of countability because for every x ∈ S the countable family of open sets {B(x, 1/n) | n  1} satisfies the requirements of Definition 4.16. This is not the case with the second axiom as we show later in Theorem 5.23.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Metric Space Topologies

page 257

257

As we have shown in Chapter 4, sequences are adequate for studying topological metric spaces. The definition of open sets in a topological metric space implies that a subset U of a topological metric space (S, Od ) is closed if and only if for every x ∈ S such that x ∈ U there is > 0 such that B(x, ) is disjoint from U . Thus, if B(x, ) ∩ U = ∅ for every > 0 and U is a closed set, then x ∈ U. Theorem 5.3. In a topological metric space (S, Od ) the closed sets are (Od )δ sets and the open sets are (Fd )σ sets, where Fd is the collection of closed sets of (S, Od ). Proof. Let U be a closed set in (S, Od ). Define Vn = {x ∈ S | d(x, y)  1 n for some  y1 ∈ U }. Clearly, each of the sets Vn is open because Vn =  B y, n . Note that if U = ∅, then Vn = ∅. It is immediate that y∈U  exists U ⊆ n1 Vn . Conversely,  1if x ∈ n1 Vn then for each n  1, there  y ∈ U such that y ∈ B x, n , which implies x ∈ U . Thus, U = n1 Vn . If V is an open set, then V = S − V is a closed set. By the first part of the theorem, there exists a countable collection of open sets (Wn ) such   that S − V = n1 Wn , which means that V = n1 (S − Wn ). Since each set S − Wn is closed, it follows that V is a union of closed sets, that is, a  (Fd )σ -set. The closure and the interior operators K and I in a topological metric space (S, Od ) are described next. Theorem 5.4. In a topological metric space (S, Od ), we have K(U ) = {x ∈ S | B(x, ) ∩ U = ∅ for every > 0} and I(U ) = {x ∈ S | B(x, ) ⊆ U for some > 0} for every U ∈ P(S). Proof. If B(x, ) ∩ U = ∅ for every > 0, then clearly B(x, ) ∩ K(U ) = ∅ for every > 0 and therefore x ∈ K(U ) by a previous observation. Now let x ∈ K(U ) and let > 0. Suppose that B(x, ) ∩ U = ∅. Then, U ⊆ S − B(x, ) and S − B(x, ) is a closed set. Therefore, K(U ) ⊆ S − B(x, ). This is a contradiction because x ∈ K(U ) and x ∈ S − B(x, ). The second part of the theorem follows from the first part and from Corollary 4.3. 

May 2, 2018 11:28

258

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 258

Mathematical Analysis for Machine Learning and Data Mining

Corollary 5.2. The subset U of the topological metric space (S, Od ) is closed if and only if B(x, ) ∩ U = ∅ for every > 0 implies x ∈ U . The border of a set U , ∂U is given by ∂U = {x ∈ S | for every > 0, B(x, ) ∩ U = ∅ and B(x, ) ∩ (S − U ) = ∅}. Proof.

This corollary follows immediately from Theorem 5.4.



Recall that the notion of diameter of a subset of a metric space was introduced in Definition 1.50. Theorem 5.5. Let T be a subset of a topological metric space (S, Od ). We have diam(K(T )) = diam(T ). Proof. Since T ⊆ K(T ), it follows immediately that diam(T )  diam(K(T )), so we have to prove only the reverse inequality. Let u, v ∈ K(T ). For every positive number , we have B(u, ) ∩ T = ∅ and B(v, ) ∩ T = ∅. Thus, there exists x, y ∈ T such that d(u, x) < and d(v, y) < . Thus, d(u, v)  d(u, x) + d(x, y) + d(y, v)  2 + diam(T ) for every , which implies d(u, v)  diam(T ) for every u, v ∈ K(T ). This yields diam(K(T ))  diam(T ).  Example 5.3. Let C be the set of complex numbers, C = {a + ib | a, b ∈ R}. A metric d : C × C −→ R0 can be defined on C as " d(u, v) = |u − v| = (u1 − v1 )2 + (u2 − v2 )2 , where u = u1 + iu2 and v = v1 + iv2 . Thus, an open sphere B(u0 , r) is given by B(u0 , r) = {z ∈ C | |z − u0 | < r}. A subset U of C is open if for every u ∈ U there is a sphere B(u, r) that is included in U . The extended set of complex numbers is obtained by adding the infinity ˆ = symbol ∞ to C and define the extended set of complex numbers C C ∪ {∞}. Note that unlike the extended set of real numbers, the extended set of complex numbers contains a unique element at infinity. As we did in the case of real numbers we add the following supplementary rules: (i) ∞ + c = c + ∞ = ∞ for c ∈ C; (ii) if c = 0, then c∞ = ∞c = ∞∞ = ∞; c = 0 and 0c = ∞. (iii) if c = 0, then ∞

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Metric Space Topologies

9in x 6in

b3234-main

page 259

259

Theorem 5.6. Let d and d be two metrics defined on a set S. The metric topology Od is finer than Od , that is, Od ⊆ O if and only if for each x ∈ S and r > 0 there exists r > 0 such that Bd (x, r) ⊆ Bd (x, r ). Proof. Suppose that Od is finer than Od . Since Bd (x, r) is an open set in Od , it follows that it is an open set in Od . This implies the existence of a sphere Bd (x, r) such that Bd (x, r) ⊆ Bd (x, r ). Conversely, suppose the condition of the theorem holds and let U  be an open set in Od . If x ∈ U  there exists a sphere Bd (x, r ) included in U  . By hypothesis, there exists r > 0 such that Bd (x, r) ⊆ Bd (x, r ), so  Bd (x, r) ⊆ U  , which means that U  ∈ O. A metric topology can be defined, as we shall see, by more than one metric. Definition 5.2. Two metrics d and d defined on a set S are topologically equivalent if the topologies Od and Od are equal. Theorem 5.7. Let d and d be two metrics defined on a set S. If there exist two numbers a, b ∈ R>0 such that a d(x, y)  d (x, y)  b d(x, y), for x, y ∈ S, then Od = Od . Proof. Let Bd (x, r) be an open sphere centered in x, defined by d. The previous inequalities imply # r$ # r$ Bd x, ⊆ Bd (x, r) ⊆ Bd x, . b a Let L ∈ Od . By Definition 5.1, for each x ∈ L there exists > 0 such that Bd (x, ) ⊆ L. Then, Bd (x, a ) ⊆ Bd (x, ) ⊆ L, which implies L ∈ Od . We  leave it to the reader to prove the reverse inclusion Od ⊆ Od . Example 5.4. By Corollary 2.8, any two Minkowski metrics dp and dq on Rn , with p, q  1 are topologically equivalent. Thus, the Euclidean topology on Rn is induced not only by d2 , but also by d1 and by d∞ . Similarly, we can examine equivalence of norms on linear spaces. If ν and ν  are norms defined on a linear space L, then the corresponding metrics d and d are equivalent if and only if there exist two numbers a, b ∈ R>0 such that a ν(x)  ν  (x)  bν  (x), for x, y ∈ L. Definition 5.3. A topological space (S, O) is metrizable if there exists a metric d on S such that O = Od .

May 2, 2018 11:28

260

5.2

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 260

Mathematical Analysis for Machine Learning and Data Mining

Sequences in Metric Spaces

Since topological metric spaces satisfy the first axiom of countability, open and closed sets in these spaces can be described in terms of sequences. As we saw in Corollaries 4.9 and 4.8, a subset T of a topological space (S, O) that satisfies the first axiom of countability is open if for every x ∈ T and sequence (xn ) that converges to x there exists a number nT ∈ N such that n  nT implies xn ∈ T . Therefore, this characterization of open sets is valid in the case of topological metric spaces. Let U be a closed set in the topological metric space (S, Od ) and let (xn ) be a sequence in U such that limn→∞ xn = x. Then, we have x ∈ U . Indeed, if this is not the case, that is, x ∈ S − U , we would have a number nT such that n  nT would imply xn ∈ S − U because S − T is an open subset of S. This contradiction means that x ∈ U . Conversely, let U be a subset of S such that for every sequence (xn ) in U such that limn→∞ xn = x, we have x ∈ U . Suppose that U is not closed, so S − U is not open. This means that there exists z ∈ S − U and a sequence (zn ) in S − U such that if limn→∞ zn = z such that for every m ∈ N there exists zn with n > m such that zn ∈ S − U , that is, zn ∈ U . This contradiction means that U is closed. Definition 5.4. A sequence x = (x0 , . . . , xn , . . .) of elements in a metric space (S, d) converges to an element x of S if for every > 0 there exists n ∈ N such that n  n implies xn ∈ B(x, ). If there exists x ∈ S such that x converges to x, we say that the sequence x is convergent. Theorem 5.8. Let (S, Od ) be a topological metric space and let x = (x0 , . . . , xn , . . .) be a sequence in Seq∞ (S). If x is convergent, then there exists a unique x such that x converges to x. Proof. Suppose that there are two distinct elements x and y of the set S that satisfy the condition of Definition 5.4. We have d(x, y) > 0. Define = d(x,y) 3 . By definition, there exists n such that n  n implies d(x, xn ) < and d(xn , y) < . By applying the triangular inequality, we obtain 2 d(x, y)  d(x, xn ) + d(xn , y) < 2 = d(x, y), 3 which is a contradiction.  If the sequence x = (x0 , . . . , xn , . . .) converges to x, this is denoted by limn→∞ xn = x or by xn → x.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Metric Space Topologies

9in x 6in

b3234-main

page 261

261

An important special case is the metric space (R, Od ), where d is the usual metric on R defined by d(x, y) = |x − y| for x, y ∈ R. Definition 5.5. A sequence of real numbers (xn ) converges to x from the left if xn  x for n ∈ N and xn → x. Similarly, (xn ) converges to x from the right if xn  x for n ∈ N and xn → x. Example 5.5. The sequence of real numbers (un ) defined by un =  1 n converges to e from the left, while the sequence (vn ) given by 1+ n n+1  converges to e from the right, as it is well-known from vn = 1 + n1 calculus.

5.3

Limits of Functions on Metric Spaces

Definition 5.6. Let (S, Od ) and (T, Oe ) be two topological metric spaces, X be subset of S, and let a an accumulation point of X. A function f : X −→ T has the limit b ∈ T in a if for every V ∈ neighb (Oe ) there exists a neighborhood U ∈ neigha (Od ) such that f (U ) ⊆ V. This is denoted by limx→a f (x) = b. Note that in order to consider the existence of a limit in a it is not necessary for f to be defined in a; it suffices for a to be an accumulation point of the definition domain of f . Theorem 5.9. Let (S, Od ) and (T, Oe ) be two topological metric spaces, X be a subset of S and let a be an accumulation point of X. If a function f : X −→ T has a limit in a, then this limit is unique. Proof. Suppose that we have limx→a f (x) = b1 and limx→a f (x) = b2. Consider the neighborhoods B(b1 , r1 ) and B(b2 , r2 ) such that r1 + r2 < e(y1 , y2 ). The open spheres B(b1 , r1 ) and B(b2 , r2 ) are disjoint. By hypothesis, there exists U1 , U2 ∈ neigha (Od ) such that f (U1 ) ⊆ B(y1 , r1 ) and f (U2 ) ⊆ B(y2 , r2 ), which implies f (U1 ) ∩ f (U2 ) = ∅. Since f (U1 ∩ U2 ) ⊆ f (U1 ) ∩ f (U2 ), it follows that f (U1 ∩ U2 ) = ∅. This contradicts the fact  that f (a) ∈ f (U1 ∩ U2 ). Example 5.6. In the special case of functions of the form f : X −→ C the definition of the limit of f in a can be formulated in an − δ language. Namely, for an accumulation point a of X, we have limx→a f (x) = b if for

May 2, 2018 11:28

262

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 262

Mathematical Analysis for Machine Learning and Data Mining

every > 0, there exists δ > 0 (which depends on both a and ) such that |x − a| < δ implies |f (x) − b| < . The same description works for real-valued limits of real-valued functions, Next we extend the notion of neighborhood to the points ∞ and −∞ ˆ of R. ˆ is a neighborhood of ∞ if it has the form Definition 5.7. A subset V of R (a, ∞) for some a ∈ R; V is a neighborhood of −∞ if V = (−∞, a) for some a ∈ R. The collection of neighborhoods of ∞ and −∞ will be denoted by N∞ and N−∞ , respectively. Using Definition 5.7, it is possible to extend the notion of limit to functions of the form f : R −→ T , where (T, Oe ) is a topological metric space. Thus, we write limx→∞ f (x) = b if for every V ∈ neighb (Oe ) there exists a ∈ R such that x > a implies f (x) ∈ V . Similarly, limx→−∞ f (x) = b if for every V ∈ neighb (Oe ) there exists a ∈ R such that x < a implies f (x) ∈ V . Definition 5.8. Let X ⊂ R, f : X −→ (S, d) be a function of a real argument ranging over a metric space (S, d), and let a be an accumulation point of the set X ∩ (−∞, a). An element b ∈ S is the left limit of f in a if for every V ∈ neighb (Od ) there exists U ∈ neigha (O) (where O is the usual topology on R) such that x ∈ U ∩ X and x < a imply f (x) ∈ V . This is denoted by limx→a− f (x) = b. Definition 5.9. Let X ⊂ R, f : X −→ (S, d) be a function of a real argument ranging over a metric space (S, d), and let a be an accumulation point of the set X ∩ (a, ∞). An element b ∈ S is the right limit of f in a if for every V ∈ neighb (Od ) there exists U ∈ neigha (O) (where O is the usual topology on R) such that x ∈ U ∩ X and x > a imply f (x) ∈ V . This is denoted by limx→a+ f (x) = b. The function f : X −→ R has the left limit b in a (where a is an accumulation point of the set X ∩ (−∞, a) and b ∈ R) if for every sequence (xn ) such that xn < a and limn→∞ xn = a we have limn→∞ f (xn ) = b.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Metric Space Topologies

9in x 6in

b3234-main

page 263

263

The function f has the right limit b in a ∈ I (where a is an accumulation point of the set X ∩ (a, ∞) and b ∈ R) if for every sequence (xn ) such that xn > a and limn→∞ xn = a we have limn→∞ f (xn ) = b. Example 5.7. Let f : R −→ R be the function defined by f (x) = n if x ∈ [n, n + 1) for x ∈ R and n ∈ N. This is the well-known floor function. Note that limx→n− f (x) = n − 1 and limx→n+ f (x) = n. Theorem 5.10. Let X be a subset of R and let f : X −→ (S, d) be a function of a real argument ranging over a metric space (S, d). The function has a limit in a (where a is an accumulation point of X in the topological space (R, O)) if and only if the lateral limits limx→a− f (x), limx→a+ f (x) exists and are equal. Proof. By Definition 5.6 it is clear if limx→a f (x) exists, then both lateral limits in a exist and are equal. Conversely, suppose that both lateral limits in a exist and are equal to b. Let V ∈ neighb (Od ). Since b = limx→a− f (x) there exists a neighborhood U1 of a such that if x ∈ U1 ∩ X and x < a, then f (x) ∈ U1 . Also, since b = limx→a+ f (x), there exists a neighborhood U2 of a such that if x ∈ U2 ∩ X and x > a, f (x) ∈ U2 . Let U = U1 ∩ U2 . We have U ∩ X ⊆ U1 ∩ X and U ∩ X ⊆ U2 ∩ X and U ∈ neigha (O). If x = a and x ∈ U ∩ X, then x ∈ U1 ∩ X and x ∈ U2 ∩ X. If x < a, then f (x) ∈ V (because x ∈ U1 ∩ X); if x > a, then we also have f (x) ∈ V . Thus, for any x = a, we have f (x) ∈ V , so limx→a f (x) = b.  Theorem 5.11. Let X be a subset of R and let f : X −→ R be a monotonic function. Then, f has lateral limits in every accumulation point of X. Proof. Suppose that f is an increasing function and let (xn ) be an arbitrary increasing sequence (xn ) such that limn→∞ xn = a and xn < a for n ∈ N. Since f is increasing, the sequence (f (xn )) is increasing and has a limit (finite or not), so the left limit of f in a exists. Similarly, for every decreasing sequence (xn ) such that limn→∞ xn = a, the sequence (f (xn )) has a limit. The argument for decreasing functions is similar.  It is possible that a function has no lateral limits in a particular point, or even in any point, as the next example shows. Example 5.8. The function f : R − {0} −→ R defined by f (x) = sin x1 for 2 . We have limn→∞ xn = 0 and x = 0 has no limit in 0. Indeed, let xn = nπ

May 2, 2018 11:28

264

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 264

Mathematical Analysis for Machine Learning and Data Mining

xn > 0 for n ∈ N. The sequence (f (xn )) is 0, 1, 0, −1, 0, 1, . . . , 2 so f has no right limit in 0. Choosing xn = − nπ we have limn→∞ xn = 0 and xn < 0; again, the sequence (f (xn )) is

0, −1, 0, 1, 0, −1, . . . , and f has no left limit in 0. 5.4

Continuity of Functions between Metric Spaces

For functions between topological metric spaces we can formulate specific characterizations of continuity. Theorem 5.12. Let (S, Od ) and (T, Oe ) be two topological metric spaces. The following statements concerning a function f : S −→ T are equivalent: (i) f is a continuous function; (ii) for every > 0 and x ∈ S there exists δ > 0 such that f (Bd (x, δ)) ⊆ Be (f (x), ). Proof. (i) implies (ii): Suppose that f is a continuous function. Since Be (f (x), ) is an open set in (T, Oe ), the set f −1 (Be (f (x), ) is an open set in (S, Od ). Clearly, x ∈ f −1 (Be (f (x), )), so by the definition of the metric topology there exists δ > 0 such that Bd (x, δ) ⊆ f −1 (Be (f (x), ), which yields f (Bd (x, δ)) ⊆ Be (f (x), ). (ii) implies (i): Let V be an open set of (T, Oe ). If f −1 (V ) is empty, then it is clearly open. Therefore, we may assume that f −1 (V ) is not empty. Let x ∈ f 1− (V ). Since f (x) ∈ V and V is open, there exists > 0 such that Be (f (x), ) ⊆ V . By part (ii) of the theorem, there exists δ > 0 such that f (Bd (x, δ)) ⊆ Be (f (x), ), which implies x ∈ Bd (x, δ) ⊆ f −1 (V ). This  means that f −1 (V ) is open, so f is continuous. Theorem 5.13. Let (S, Od ) and (T, Oe ) be two topological metric spaces, X be a subset of S, and let a ∈ S be an accumulation point of X. We have limx→a f (x) = b if for every sequence (xn ) such that limn→∞ xn = a we have limn→∞ f (xn ) = b. Proof. Suppose that limn→∞ f (xn ) = b and let > 0. By Theorem 5.12, there exists δ > 0 such that f (B(a, δ)) ⊆ B(b, ). Let x = (x0 , . . . , xn , . . .) be a sequence such that limn→∞ xn = a. Since limn→∞ xn = a, there exists

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Metric Space Topologies

b3234-main

page 265

265

nδ such that n ≥ nδ implies xn ∈ B(a, δ). Then, f (xn ) ∈ f (B(a, δ)) ⊆ B(b, ). This shows that limn→∞ f (xn ) = b. Conversely, suppose that for every sequence x = (x0 , . . . , xn , . . .) such that limn→∞ xn = a, we have limn→∞ f (xn ) = b. If f were not continuous in a, we would have an > 0 such that for all δ > 0 we would have 1 y ∈ B(a, f (y) ∈ B(f (a), ). Choosing δ = n , let yn ∈ S such that  δ)1but yn ∈ B x, n and f (yn ) ∈ B(b, ). This yields a contradiction because we  should have limn→∞ f (yn ) = b. A local continuity property is introduced next. Definition 5.10. Let (S, Od ) and (T, Oe ) be two topological metric spaces and let x ∈ S. A function f : S −→ T is continuous in x if for every > 0 there exists δ > 0 such that f (B(x, δ)) ⊆ B(f (x), ). It is clear that f is continuous if it is continuous in every x ∈ S. The definition of continuity in a point can be restated by saying that f is continuous in x if for every > 0 there is δ > 0 such that d(x, y) < δ implies e(f (x), f (y)) < . We saw that a function f : S −→ T between two topological metric spaces (S, Od ) and (T, Oe ) is continuous if and only if for every > 0 and every x ∈ S there exists δ > 0 such that f (Bd (x, δ)) ⊆ Be (f (x), ). Definition 5.11. A function f : S −→ T between two topological metric spaces (S, Od ) and (T, Oe ) is uniformly continuous if and only if for every > 0, there exists δ > 0 such that for every x ∈ S we have f (Bd (x, δ)) ⊆ Be (f (x), ). Note that the definition of uniform continuity is obtained from the second part of the characterization of continuous function by inverting the order of the expressions “for every x ∈ S” and “there exists δ > 0”. Thus, for a continuous function f from (S, Od ) and (T, Oe ), the number δ introduced in the definition of local continuity depends both on x and on . If δ is dependent only on (as in Definition 5.11) we obtain the stronger property of uniform continuity. Example 5.9. The function f : R −→ R given by f (x) = x sin x is continuous but not uniformly continuous. Indeed, let un = nπ and vn = nπ + n1 . Note that limn→∞ |un − vn | = 0, f (un ) = 0, and

May 2, 2018 11:28

266

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 266

Mathematical Analysis for Machine Learning and Data Mining

f (vn ) = (nπ + n1 ) sin(nπ + n1 ) = (nπ + n1 )(−1)n sin n1 . Therefore,   1 n 1 = π, lim |f (un ) − f (vn )| = lim nπ + sin = π lim n→∞ n→∞ n→∞ sin 1 n n n so f is not uniformly continuous. Theorem 5.14. Let (S, Od ) and (T, Oe ) be two topological metric spaces, f : S −→ T be a function, and let u = (u0 , u1 , . . .) and v = (v0 , v1 , . . .) in Seq∞ (S). The following statements are equivalent: (i) f is uniformly continuous; (ii) if limn→∞ d(un , vn ) = 0, then limn→∞ e(f (un ), f (vn )) = 0; (iii) if limn→∞ d(un , vn ) = 0, we have limk→∞ e(f (unk ), f (vnk )) = 0, where (un0 , un1 , . . .) and (vn0 , vn1 , . . .) are two arbitrary subsequences of u and v, respectively. Proof. (i) implies (ii): For > 0, there exists δ such that d(u, v) < δ implies e(f (u), f (v)) < . Therefore, if u and v are sequences as above, there exists nδ such that n > nδ implies d(un , vn ) < δ, so e(f (un ), f (vn )) < . Thus, limn→∞ e(f (un ), f (vn )) = 0. (ii) implies (iii): This implication is obvious. (iii) implies (i): Suppose that f satisfies (iii) but is not uniformly continuous. Then, there exists > 0 such that for every δ > 0 there exist u, v ∈ X such that d(u, v) < δ and e(f (u), f (v)) > . Let un , vn be such that d(un , vn ) < n1 for n  1. Then, limn→∞ d(un , vn ) = 0 but e(f (un ), f (vn )) does not converge to 0.  Next we discuss extensions of the notion of distance between points to dissimilarities between points and sets and between subsets of metric spaces. Definition 5.12. Let (S, d) be a metric space and let U be a subset of S. The distance from an element x to U is the number d(x, U ) = inf{d(x, u) | u ∈ U }. Note that if x ∈ U , then d(x, U ) = 0. Theorem 5.15. Let (S, d) be a metric space and let U be a subset of S. The function f : S −→ R given by f (x) = d(x, U ) for x ∈ S is continuous. Proof. Since d(x, z) ≤ d(x, y) + d(y, z), we have d(x, U )  d(x, y) + d(y, U ). By exchanging x and y we also have d(y, U )  d(x, y) + d(x, U ) and, together, these inequalities yield the inequality |d(x, U ) − d(y, U )|  d(x, y).

(5.1)

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Metric Space Topologies

page 267

267

Therefore, if d(x, y) < , it follows that |d(x, U ) − d(y, U )|  , which implies the continuity of d(x, U ).  Theorem 5.16. Let (S, d) be a metric space. The following statements hold: (i) d(u, V ) = 0 if and only if u ∈ K(V ), and (ii) d(u, V ) = d(u, K(V )) for every u, u ∈ S and V ⊆ S. Proof. Suppose that d(u, V ) = 0. Again, by the definition of d(u, V ), for every > 0 there exists v ∈ V such that d(u, v) < , which means that B(u, ) ∩ V = ∅. By Theorem 5.4, we have u ∈ K(V ). The converse implication is immediate, so (i) holds. To prove (ii), observe that V ⊆ K(V ) implies that d(u, K(V ))  d(u, V ), so we need to show only the reverse inequality. Let w be an arbitrary element of K(V ). By Theorem 5.4, for every > 0, B(w, ) ∩ V = ∅. Let v ∈ B(w, ) ∩ V . We have d(u, v)  d(u, w) + d(w, v)  d(u, w) + , so d(u, V )  d(u, w) + . Since this inequality holds for every , d(u, V )  d(u, w) for every w ∈ K(V ), so d(u, V )  d(u, K(V )). This allows us to conclude that d(u, V ) = d(u, K(V )).  Corollary 5.3. If V is a closed subset of a metric space (S, d) and d(u, V ) > 0, then v ∈ S − V . Proof.

This is an immediate consequence of Theorem 5.16.



Theorem 5.16 can be restated using the function dU : S −→ R0 defined by dU (x) = d(x, U ) for u ∈ S. Thus, for every subset U of S and x, y ∈ S, we have |dU (x) − dU (y)|  d(x, y), dU (x) = 0 if and only if x ∈ K(U ), and dU = dK(V ) . The function dU is continuous. ˆ 0 can be extended to the set of subsets A dissimilarity d : S × S −→ R of S by defining d(U, V ) as d(U, V ) = inf{d(u, v) | u ∈ U and v ∈ V } for U, V ∈ P(S). The resulting extension is also a dissimilarity. However, even if d is a metric, then its extension to subsets of S is not, in general, a metric on P(S) because it does not satisfy the triangular inequality. Instead, we prove that if d is a metric, then for every U, V, W we have d(U, W )  d(U, V ) + diam(V ) + d(V, W ).

May 2, 2018 11:28

268

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 268

Mathematical Analysis for Machine Learning and Data Mining

Indeed, by the definition of d(U, V ) and d(V, W ), for every > 0, there exist u ∈ U , v, v  ∈ V , and w ∈ W such that d(U, V )  d(u, v)  d(U, V ) + 2 , d(V, W )  d(v  , w)  d(V, W ) + 2 . By the triangular axiom, we have d(u, w)  d(u, v) + d(v, v  ) + d(v  , w). Hence, d(u, w)  d(U, V )+diam(V )+d(V, W )+ , which implies d(U, W )  d(U, V ) + diam(V ) + d(V, W ) + for every > 0. This yields the needed inequality. Definition 5.13. Let (S, d) be a metric space. The sets U, V ∈ P(S) are separate if d(U, V ) > 0. The notions of an open sphere and a closed sphere in a metric space (S, d) are extended to the space of subsets of S by defining the sets B(T, r) and B[T, r] as B(T, r) = {u ∈ S | d(u, T ) < r}, B[T, r] = {u ∈ S | d(u, T )  r}, for T ∈ P(S) and r  0, respectively. The next statement is a generalization of Theorem 5.2. Theorem 5.17. Let (S, Od ) be a topological metric space. For every set T , T ⊆ S, and every r > 0, B(T, r) is an open set and B[T, r] is a closed set in (S, Od ). Proof. Let u ∈ B(T, r). We have d(u, T ) < r, or, equivalently, inf{d(u, t) | t ∈ T } < r. We claim that if is a positive number such that < r2 , then B(u, ) ⊆ B(T, r). Let z ∈ B(u, ). For every v ∈ T , we have d(z, v)  d(z, u) + d(u, v) < +d(u, v). From the definition of d(u, T ) as an infimum, it follows that there exists v  ∈ T such that d(u, v  ) < d(u, V )+ 2 , so d(z, v  ) < d(u, T )+ < r+ . Since this inequality holds for every > 0, it follows that d(z, v  ) < r, so d(z, T ) < r, which proves that B(u, ) ⊆ B(T, r). Thus, B(T, r) is an open set. Suppose now that s ∈ K(B[T, r]). By part (ii) of Theorem 5.16, we have d(s, B[T, r]) = 0, so inf{d(s, w) | w ∈ B[T, r]} = 0. Therefore, for every > 0, there is w ∈ B[T, r] such that d(s, w) < . Since d(w, T )  r, it follows from the first part of Theorem 5.16 that |d(s, T ) − d(w, T )|  d(s, w) < for every > 0. This implies d(s, T ) = d(w, T ), so s ∈ B[T, r]. This allows us to conclude that B[T, r] is indeed a closed set. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Metric Space Topologies

page 269

269

To introduce a metric structure on a quotient space of a linear space we need to limit ourselves to considering only closed subspaces. Let M be a subspace of a normed linear space L and let [x], [y] ∈ L/M . The dissimilarity between the classes [x] and [y] is d([x], [y]) = inf{u − v | u ∈ [x] and v ∈ [y]}. The distance between x and [y] is d(x, [y]) = inf{x − w | w ∈ [y]}, and we have d(x, [y]) = d([x], [y]) because {u − v | u ∈ [x], v ∈ [y]} = {(x + z1 ) − (y + z2 ) | z1 , z2 ∈ M } = {x − (y − z1 + z2 ) | z1 , z2 ∈ M } = {x − (y + z) | z ∈ M } = {x − w | w ∈ M }. Note that M = [0L ]. By Theorem 5.16 if x ∈ K(M ) − M , then d(x, [0]L ) > 0. On other hand, d([x], [0L ]) = 0 even though [x] = [0L ]. Therefore, if we assume that d is a metric on L/M we must have K(M ) = M , that is, we need to consider only quotient spaces relative to closed subspaces. Definition 5.14. Let M be a subspace of a normed linear space L. The quotient norm on L/M is given by [x] = d([x], [0L ]). Theorem 5.18. (Lebesgue’s Lemma) Let (S, Od ) be a compact topological metric space and let C be an open cover of this space. There exists r ∈ R>0 such that for every subset U with diam(U ) < r there is a set L ∈ C such that U ⊆ L. Proof. Suppose that the statement is not true. Then, for every k ∈ P, there exists a subset Uk of S such that diam(Uk ) < k1 and Uk is not included in any of the sets L of C. Since (S, Od ) is compact, there exists a finite subcover {L1 , . . . , Lp } of C. Let xik be an element in Uk − Li . For every two points xik , xjk , we have d(xik , xjk )  k1 because both belong to the same set Uk . By Theorem 5.43, the compactness of S implies that any sequence xi = (xi1 , xi2 , . . .) contains a convergent subsequence. Denote by xi the limit of this subsequence, where 1  i  p. The inequality d(xik , xjk )  k1 for k  1 implies that d(xi , xj ) = 0 so xi = xj for 1  i, j  p. Let x be their common value. Then x does not belong to any of the sets Li , which contradicts the fact  that {L1 , . . . , Lp } is an open cover.

May 2, 2018 11:28

270

5.5

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 270

Mathematical Analysis for Machine Learning and Data Mining

Separation Properties of Metric Spaces

Theorem 5.19. Every topological metric space (S, Od ) is a Hausdorff space. Proof. Let x and y be two distinct elements of S, so d(x, y) > 0. Choose = d(x,y) 3 . It is clear that for the open spheres B(x, ) and B(y, ), we have x ∈ B(x, ), y ∈ B(y, ), and B(x, ) ∩ B(y, ) = ∅, so (S, Od ) is indeed a Hausdorff space.  Corollary 5.4. Every compact subset of a topological metric space is closed and bounded. Proof. By Theorem 4.51 every compact subset of a Hausdorff space is closed. Furthermore, if C is a compact subset of (S, Od ), then from the fact  n that C ⊆ {B(x, r) | x ∈ C} it follows that C ⊆ i=1 B(xi , r) for a finite  subset {x1 , . . . , xn } of C, which shows that C is bounded. Corollary 5.5. If S is a finite set and d is a metric on S, then the topology Od is the discrete topology. Proof. Let S = {x1 , . . . , xn } be a finite set. We saw that every singleton {xi } is a closed set. Therefore, every subset of S is closed as a finite union of closed sets.  Theorem 5.20. Every topological metric space (S, Od ) is a T4 space. Proof. We need to prove that for all disjoint closed sets H1 and H2 of S there exist two open disjoint sets V1 and V2 such that H1 ⊆ V1 and H2 ⊆ V2 . Let x ∈ H1 . Since H1 ∩ H2 = ∅, it follows that x ∈ H2 = K(H2 ), so #d(x, H2 ) >$ 0 by part (ii) of Theorem 5.16. By Theorem 5.17, the set

B H1 , d(x,L) 3

is an open set and so is *    d(x, L)  QH = B H1 ,  x ∈ H1 . 3

The open set QH2 is defined in a similar manner as *    d(y, H1 )  y ∈ H B H2 , QH2 = 2 .  3

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Metric Space Topologies

9in x 6in

b3234-main

page 271

271

The sets QH1 and QH2 are disjoint because t ∈ QH1 ∩QH2 implies that there is x1 ∈ H1 and x2 ∈ H2 such that d(t, x1 ) < d(x13,H2 ) and d(t, x2 ) < d(x23,H1 ) . This, in turn, would imply 2 d(x1 , H2 ) + d(x2 , H1 )  d(x1 , x2 ), d(x1 , x2 ) < 3 3 which is a contradiction. Therefore, (S, Od ) is a T4 topological space.  Corollary 5.6. Every metric space is normal. Proof. By Theorem 5.19, a metric space is a T2 space and therefore a T1 space. The statement then follows directly from Theorem 5.20.  Corollary 5.7. Let H be a closed set and L be an open set in a topological metric space (S, Od ) such that H ⊆ L. Then, there is an open set V such that H ⊆ V ⊆ K(V ) ⊆ L. Proof. The closed sets H and S − L are disjoint. Therefore, since (S, O) is normal, there exist two disjoint open sets V and W such that H ⊆ V and S − L ⊆ W . Since S − W is closed and V ⊆ S − W , it follows that K(V ) ⊆ S − W ⊆ L. Thus, we obtain H ⊆ V ⊆ K(V ) ⊆ L.  A stronger form of Theorem 5.20, where the disjointness of the open sets is replaced by the disjointness of their closures, is given next. Theorem 5.21. Let (S, Od ) be a metric space. For all disjoint closed sets H1 and H2 of S, there exist two open sets V1 and V2 such that H1 ⊆ V1 , H2 ⊆ V2 , and K(V1 ) ∩ K(V2 ) = ∅. Proof. By Theorem 5.20, we obtain the existence of the disjoint open sets QH1 and QH2 such that H1 ⊆ QH1 and H2 ⊆ QH2 . We claim that the closures of these sets are disjoint.  Suppose  that s ∈ K(QH1 )∩K(QH2 ). Then, we have B s, 12 ∩QH1 = ∅ and B s, 12 ∩ QH2 = ∅. Thus, there exist t ∈ QH1 and t ∈ QH2 such that and d(t , s) < 12 . d(t, s) < 12 As in the proof of the previous theorem, there is x1 ∈ H1 and y1 ∈ H2 such that d(t, x1 ) < d(x13,H2 ) and d(t , y1 ) < d(y13,H1 ) . Choose t and t above for = d(x1 , y1 ). This leads to a contradiction because 5 d(x1 , y1 )  d(x1 , t) + d(t, s) + d(s, t ) + d(t , y1 )  d(x1 , y1 ).  6 Corollary 5.8. Let (S, Od ) be a metric space. If x ∈ L, where L is an open subset of S, then there exists two open sets V1 and V2 in S such that x ∈ V1 , S − L ⊆ V2 , and K(V1 ) ∩ K(V2 ) = ∅.

May 2, 2018 11:28

272

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 272

Mathematical Analysis for Machine Learning and Data Mining

Proof. The statement follows by applying Theorem 5.21 to the disjoint  closed sets H1 = {x} and H2 = S − L. Recall that the Bolzano-Weierstrass property of topological spaces was introduced in Theorem 4.47. Namely, a topological space (S, O) has the Bolzano-Weierstrass property if every infinite subset T of S has at least one accumulation point, that is, if T  = ∅. For metric spaces, this property is equivalent to compactness, as we show next. Theorem 5.22. Let (S, Od ) be a topological metric space. The following three statements are equivalent: (i) (S, Od ) is compact; (ii) (S, Od ) has the Bolzano-Weierstrass property; (iii) every countable open cover of (S, Od ) contains a finite subcover. Proof. (i) implies (ii): by Theorem 4.47. (ii) implies (iii): Let {Ln | n ∈ N} be a countable open cover of S. Without loss of generality, we may assume that none of the sets Ln is n−1 included in p=1 Lp ; indeed, if this is not the case, we can discard Ln  and still have a countable open cover. Let xn ∈ Ln − n−1 p=1 Lp and let U = {xn | n ∈ N}. Since (S, Od ) has the Bolzano-Weierstrass property, we have U  = ∅, so there exists an accumulation point z of U . In every open set L that contains z, there exists xn ∈ U such that xn = z. Since {Ln | n ∈ N} is an open cover, there exists Lm such that z ∈ Lm . Suppose that the set Lm contains only a finite number of elements  xn1 , . . . , xnk , and let d = min{d(z, xni ) | 1  i  k}. Then, Lm ∩ B z, d2 is an open set that contains no elements of U with the possible exception of z, which contradicts the fact that z is an accumulation point. Thus, Lm contains an infinite subset of U , which implies that there exists xq ∈ Lm for some q > m. This contradicts the definition of the elements xn of U . r−1 We conclude that there exists a number r0 such that Lr − i=0 Li = ∅ for r  r0 , so S = L0 ∪ · · · ∪ Lr0 −1 , which proves that L0 , . . . , Lr0 −1 is a finite subcover. (iii) implies (i). Let be a positive number. Suppose that there is an infinite sequence x = (x0 , . . . , xn , . . .) such that d(xi , xj ) > for every i, j ∈ N such that i = j. Consider the open spheres B(xi , ) and the set ⎞ ⎛ $  # ⎠. C = S − K⎝ B xi , 2 i∈N

We will show that {C} ∪ {B(xi , ) | i ∈ N} is a countable open cover of S.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Metric Space Topologies

9in x 6in

b3234-main

page 273

273

  Suppose that x ∈ S − C; that is x ∈ K i∈N B xi , 2 . We have either  that x ∈ i∈N B xi , 2 or x is an accumulation point that set.  of  B x , In the first case, x ∈ i∈N B(xi , ) because i 2 ⊆ B(xi , ). If x is  an accumulation point of i∈N B xi , 2 , given any open set L such that x ∈ L, then at least one of the spheres B xi , 2 . Suppose must intersect  L that B x, 2 ∩ B xi , 2 = ∅, and let t be a point that belongs to this intersection. Then, d(x, xi ) < d(x, t) + d(t, xi ) < 2 + 2 = , so x ∈ B(xi , ). Therefore, {C}∪{B(xi , ) | i ∈ N} is a countable open cover of S. Since every countable open cover of (S, Od ) contains a finite subcover, it follows that this open cover contains a finite subcover. Observe that there exists an open sphere B(xi , ) that contains infinitely many xn because none of these elements belongs to C. Consequently, for any two of these points, the distance is less than , which contradicts the assumption we made initially about the sequence x. Choose = k1 for some k ∈ N such that k ≥ 1. Since there is no infinite sequence of points such that every two distinct points are at a distance greater than k1 , it is possible to find a finite sequence of points x = (x0 , . . . , xn−1 ) such that i = j implies d(xi , xj ) > k1 for 0  i, j  n − 1 1 and for every other point x ∈ S there exists xi such that d(xi , x)  k . 1 Define the set Lk,m,i as the open sphere B xi , m , where xi is one of the points that belongs to the sequence above determined by k and m ∈ N and m  1. The collection {Lk,m,i | m  1, 0  i  n − 1} is clearly countable. We will prove that each open set of (S, Od ) is a union of sets of the form Lk,m,i ; in other words, we will show that this family of sets is a basis for (S, Od ). Let L be an open set and let z ∈ L. Since L is open, there exists > 0 1 < 2 . By the such that z ∈ B(z, ) ⊆ L. Choose k and m such that k1 < m 1 definition of the sequence x, there is xi such that d(z, xi ) < k . We claim that   1 Lk,m,i = B xi , ⊆ L. m 1 < , it follows Let y ∈ Lk,m,i . Since d(z, y)  d(z, xi ) + d(xi , y) < k1 + m 1 1 that Lk,m,i ⊆ B(z, ) ⊆ L. Since d(y, z) < k < m , we have z ∈ Lk,m,i . This shows that L is a union of sets of the form Lk,m,i , so this family of sets is a countable open cover of S. It follows that that there exists a finite open cover of (S, Od ) because every countable open cover of (S, Od ) contains a finite subcover.  In Section 4.2, we saw that if a topological space has a countable basis, then the space is separable (Theorem 4.34) and each open cover of the

May 2, 2018 11:28

274

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 274

Mathematical Analysis for Machine Learning and Data Mining

basis contains a countable subcover (Theorem 4.36). For metric spaces, these properties are equivalent, as we show next. Theorem 5.23. Let (S, Od ) be a topological metric space. The following statements are equivalent: (i) (S, Od ) satisfies the second axiom of countability, that is, it has a countable basis; (ii) (S, Od ) is a separable; (iii) every open cover of (S, Od ) contains a countable subcover. Proof. By Theorems 4.34 and 4.36, the first statement implies (ii) and (iii). Therefore, it suffices to prove that (iii) implies (ii) and (ii) implies (i). To show that (iii) implies (ii), suppose that every open cover of(S, O d ) contains a countable subcover. The collection of open spheres {B x, n1 | x ∈ S, n ∈ N>0 } is an open cover  of S and therefore there exists a countable set Tn ⊆ S such that Cn = {B x, n1 | x ∈ Tn , n ∈ N>0 } is an open cover  of S. Let C = n1 Tn . Thus, C is a countable set. We claim that C is dense in (S, Od ). Indeed, let s ∈ S and choose n 1 such that  1n > . Since Cn is an open cover of S, there is x ∈ Tn such that s ∈ B x, n ⊆ B (x, ). Since Tn ⊆ C, it follows that C is dense in (S, Od ). Thus, (S, Od ) is separable. To prove that (ii) implies (i), let (S, Od ) be a separable space. There exists a countable set U that is dense in (S, Od ). Consider the countable collection *    1  u ∈ U, n  1 . C = B u, n  If L is an open set in (S, Od ) and x ∈ L, then there exists > 0 such that B(x, ) ⊆ L. Let n be such that n > 2 . Since U is dense in (S,Od ), we 1 know x ∈ K(U ), so there exists y ∈ S(x, ) ∩ U and x ∈ B y, n ⊆  2that  B x, n ⊆ B(x, ) ⊆ L. Thus, C is a countable basis. Theorem 5.24. Let (S, Od ) be a topological metric space. Every closed set of this space is a countable intersection of open sets, and every open set is a countable union of closed sets. Proof.

Let H be a closed set and let Un be the open set *    1    x∈F . Un = B x, n  n1   It is clear that H ⊆ n1 Un . Now let u ∈ n1 Un and let be an arbitrary positive number. For every n  1, there is an element xn ∈ H

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Metric Space Topologies

b3234-main

page 275

275

such that d(u, xn ) < n1 . Thus, if n1 < , we have xn ∈ H ∩ B(u, ), so B(u, ) ∩ H = ∅. By Corollary 5.2, it follows that u ∈ H, which proves  the reverse inclusion n1 Un ⊆ H. This shows that every closed set is a countable union of open sets. If L is an open set, then its complement is closed and, by the first part of the theorem, it is a countable intersection of open sets. Thus, L itself is a countable union of closed sets.  Definition 5.15. Let (S, Od ) be a topological metric space. A Gδ -set is a countable intersection of open sets. An Fσ -set is a countable union of open sets. Now, Theorem 5.24 can be restated by saying that every closed set of a topological metric space is a Gδ -set and every open set is an Fσ -set. For a topological space (S, O), using the notations introduced in Section 1.4, the collections Fσ and Gδ can be written as Fσ = Oσ and Gδ = closed(O)δ . The notation Fσ is suggested by the term “ferm´e” (closed, in French) and by the term “somme” (sum, in the same language); the notation Gδ originates in German: G stands for “Gebiet” (region) and δ is suggested by the word “Durchschnitt” (intersection). Theorem 5.25. Let U be a Gδ -set in the topological metric space (S, Od ). If T is a Gδ -set in the subspace U , then T is a Gδ -set in S.  Proof. Since T is a Gδ -set in the subspace U , we can write T = n∈N Ln , where each Ln is an open set in the subspace U . By the definition of the subspace topology, for each Ln there exists an open set in S such that Ln = Ln ∩ U , so    Ln = (Ln ∩ U ) = U ∩ Ln . T = n∈N

n∈N

n∈N

Since U is a countable intersection of open sets of S, the last equality shows that T is a countable intersection of open sets of S and hence a Gδ -set in S.  5.6

Completeness of Metric Spaces

Let x = (x0 , . . . , xn , . . .) be a sequence in the topological metric space (S, Od ) such that limn→∞ xn = x. If m, n > n 2 , we have d(xm , xn )  d(xm , x) + d(x, xn ) < 2 + 2 = . In other words, if x is a sequence that

May 2, 2018 11:28

276

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 276

Mathematical Analysis for Machine Learning and Data Mining

converges to x, then given a positive number we have members of the sequence closer than if we go far enough in the sequence. This suggests the following definition. Definition 5.16. A sequence x = (x0 , . . . , xn , . . .) in the topological metric space (S, Od ) is a Cauchy sequence if for every > 0 there exists n ∈ N such that m, n  n implies d(xm , xn ) < . Theorem 5.26. Every convergent sequence in a topological metric space (S, Od ) is a Cauchy sequence. Proof. Let x = (x0 , x1 , . . .) be a convergent sequence and let x = limn→∞ x. There exists n such that if n > n , then d(xn , x) < 2 . Thus, 2 2 if m, n  n = n , it follows that 2

d(xm , xn )  d(xm , x) + d(x, xn ) < which means that x is a Cauchy sequence.

+ = , 2 2 

Definition 5.17. A topological metric space is complete if every Cauchy sequence is convergent. Example 5.10. The converse of Theorem 5.26 is not true, in general, as we show next. Let ((0, 1), d) be the metric space equipped with the metric d(x, y) = 1 for n ∈ N |x − y| for x, y ∈ (0, 1). The sequence defined by xn = n+1 1 is a Cauchy sequence. Indeed, it suffices to take m, n  − 1 to obtain |xn − xm | < ; however, the sequence xn is not convergent to an element of (0, 1). This also shows that ((0, 1), Od ) is not complete. Example 5.11. The topological metric space (R, Od ), where d(x, y) = |x− y| for x, y ∈∈ R, is complete. Let x = (x0 , x1 , . . .) be a Cauchy sequence in R. For every > 0, there exists n ∈ N such that m, n  n implies |xm − xn | < . Choose m0 ∈ N such that m0  n . Thus, if n  n , then xm0 − < xn < xm0 + , which means that x is a bounded sequence. By Theorem 4.45, the sequence x contains a bounded subsequence (xi0 , xi1 , . . .) that is convergent. Let l = limk→∞ xik . It is not difficult to see that limxn xn = l, which shows that (R, Od ) is complete. Example 5.12. The space Rn is complete. Indeed, let (xn ) be a Cauchy sequence in Rn . Since |(xn )i − (xm )i |  xn − xm , each coordinate

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Metric Space Topologies

b3234-main

page 277

277

sequence ((xn )i ) is a Cauchy sequence in R. By the completeness of R, it must have a limit xi . Therefore, ⎛ ⎞ x1 ⎜ .. ⎟ x=⎝ . ⎠ xn is the limit of (xn ). Theorem 5.27. There is no clopen set in the topological space (R, O) except the empty set and the set R. Proof. Suppose that L is a clopen subset of R that is distinct from ∅ and R. Then, there exist x ∈ L and y ∈ L. Starting from x and y, we define inductively the terms of two sequences x = (x0 , . . . , xn , . . .) and y = (y0 , . . . , yn , . . .) as follows. Let x0 = x and y0 = y. Suppose that xn and yn are defined. Then, ⎧ ⎨ xn + yn if xn + yn ∈ L, 2 2 xn+1 = ⎩x otherwise, n and yn+1

⎧ ⎨ xn + yn 2 = ⎩y n

xn + yn ∈ L, 2 otherwise.

if

It is clear that {xn | n ∈ N} ⊆ L and {yn | n ∈ N} ⊆ R − L. Moreover, we have |y − x| |yn − xn | = · · · = n+1 . |yn+1 − xn+1 | = 2 2 Note that |y − x| |xn+1 − xn |  |yn − xn |  . 2n This implies that x is a Cauchy sequence and therefore there is x = limn→∞ xn ; moreover, the sequence y also converges to x, so x belongs to ∂L, which is a contradiction.  Theorem 5.28. Let x = (x0 , . . . , xn , . . .) be a Cauchy sequence in the topological metric space (S, Od ). If x contains a convergent subsequence (xp0 , . . . , xpr , . . .), then the sequence x converges and its limit is the same as the limit of the subsequence.

May 2, 2018 11:28

278

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 278

Mathematical Analysis for Machine Learning and Data Mining

Proof. Since (xn ) is a Cauchy sequence for > 0 there exists n such that d(xn , xm ) < 2 . Since (xpr ) is convergent to x there exists n such that r  n implies d(xpr , x) < 2 . Therefore, if pr  m  n we have: d(xn , x)  d(xn , xpr ) + d(xpr , x) < , so limn→∞ xn = x.



Theorem 5.29. Let (S, Od ) be a complete topological metric space. If T is a closed subset of S, then the subspace T is complete. Proof. Let T be a closed subset of S and let x = (x0 , x1 , . . .) be a Cauchy sequence in this subspace. The sequence x is a Cauchy sequence in the complete space S, so there exists x = limn→∞ xn . Since T is closed, we have x ∈ T , so T is complete. Conversely, suppose that T is complete. Let x ∈ K(T ). There exists a sequence x = (x0 , x1 , . . .) ∈ Seq∞ (T ) such that limn→∞ xn = x. Then, x is a Cauchy sequence in T , so there is a limit t of this sequence in T . The uniqueness of the limit implies x = t ∈ T , so T is a closed set.  Theorem 5.30. In a complete topological metric space (S, Od ), every descending sequence of closed sets V0 ⊃ · · · ⊃ Vn ⊃ Vn+1 ⊃ · · · such that limn→∞ diam(Vn ) = 0 has a non-empty intersection that consists of a single point of S. Proof. A sequence (xn ) in S such that xn ∈ Vn is a Cauchy sequence. Indeed, let > 0. Since limn→∞ diam(Vn ) = 0, there exists n such that if m, n > n we have xm , xn ∈ Vmin{m,n} . Since min{m, n}  n , it follows that d(xm , xn )  diam(Vmin m,n ) < . The completeness of (S, Od ) is implies that there exists x ∈ S such that limn→∞ xn = x. Note that all members of the sequence belong to Vm , with the possible exception of the  first m members. Therefore x ∈ Vm , so x ∈ n∈N Vn , which implies that  n∈N Vn = ∅.  Suppose that y ∈ n∈N Vn . Since d(y, x)  diam(Vn ) it follows that d(y, x) = 0, so y = x, which allows us to conclude that the intersection   n∈N Vn = {x}. Theorem 5.31. (Baire’s Theorem for Complete Metric Spaces) Every non-empty complete topological metric space is a Baire space. Proof. We prove that if (S, Od ) is non-empty and complete, then it satisfies the first condition of Theorem 4.11.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Metric Space Topologies

9in x 6in

b3234-main

page 279

279

Let L1 , . . . , Ln , . . . be a sequence of open subsets of S that are dense in S and let L be an open, non-empty subset of S. We construct inductively a sequence of closed sets H1 , . . . , Hn , . . . that satisfy the following conditions: (i) H1 ⊆ L0 ∩ L, (ii) Hn ⊆ Ln ∩ Hn−1 for n  2, (iii) I(Hn ) = ∅, and (iv) diam(Hn )  n1 for n  2. Since L1 is dense in S, by Theorem 4.9, L1 ∩ L = ∅, so there is a closed sphere of diameter less than 1 enclosed in L1 ∩ L. Define H1 as this closed sphere. Suppose that Hn−1 was constructed. Since I(Hn−1 ) = ∅, the open set Ln ∩ I(Hn−1 ) is not empty because Ln is dense in S. Thus, there is a closed sphere Hn included in Ln ∩ I(Hn−1 ), and therefore included in Ln ∩ Hn−1 , such that diam(Hn ) < n1 . Clearly, we have I(Hn ) = ∅. By applying Theorem 5.30 to the descending sequence of closed sets H1 , . . . , Hn , . . ., the   completeness of the space implies that n1 Hn = ∅. If s ∈ n1 Hn ,  then it is clear that x ∈ n1 Ln and x ∈ L, which means that the set  Ln has a non-empty intersection with every open set L. This implies n1  that n1 Ln is dense in S. Corollary 5.9. Let (S, Od ) be a complete topological metric space. If S =  i1 Hi is a countable union of closed sets, then there exists i such that Hi contains a open sphere B(x, ), or equivalently, I(Hi ) = ∅.  Proof. Suppose that S = i1 Hi , where each set Hi has an empty interior. Let Li = S − Hi for i  1. Each of open sets Li is dense in S because, by Theorem 4.10, K(Li ) = K(S −Hi ) = S −I(Hi ) = S. Therefore,   by Baire’s Theorem, i1 Li is dense, which means that i1 Li = ∅.    Thus, there exists x ∈ i1 Li = i1 (S − Hi ) = S − i1 Hi , which contradicts the hypothesis.  Theorem 5.32. Let (S, Od ) be a topological metric space and let T be a subset of S that is dense in this space. If the subspace (T, Od ) is complete, then (S, Od ) is complete. Proof. Let > 0 and let x be a Cauchy sequence in S. For each xn there exists yn ∈ B(xn , /3) ∩ T because T is dense in S. Since x is a Cauchy sequence, there exists n such that n, m  n implies d(xm , xn ) < /3. Therefore, taking into account that d(ym , yn )  d(ym , xm ) + d(xm , xn ) + d(xn , yn ) <

May 2, 2018 11:28

280

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 280

Mathematical Analysis for Machine Learning and Data Mining

when m, n  n , it follows that y is a Cauchy sequence. By the completeness of T , there exists t ∈ T such that limn→∞ yn = t. Thus, there exists m such that n  m implies |yn − t| < 23 , which imply |xn − t| < |xn − yn | + |yn − t| < , so limn→∞ xn = t and (S, Od ) is complete.



Let Cauchy(S, Od ) be the set of Cauchy sequences on a topological metric space (S, Od ). Define the relation ∼ on Cauchy(S, Od ) as x ∼ y if limn→∞ d(xn , yn ) = 0. It is easy to verify that ∼ is an equivalence on Cauchy(S, Od ). This allows us to consider the quotient set Cauchy(S, Od )/ ∼, where the equivalence class of the Cauchy sequence x is denoted by [x], and to introduce a metric e on Cauchy(S, Od )/ ∼ by e([x], [y]) = lim d(xn , yn ). n→∞

We need to show that e is well-defined. Let x ∈ [x] and y ∈ [y]. We have d(xn , yn )  d(xn , xn ) + d(xn , yn ) + d(yn , yn ), d(xn , yn )  d(xn , xn ) + d(xn , yn ) + d(yn , yn ), which imply |d(xn , yn ) − d(xn , yn )|  d(xn , xn ) + d(yn , yn ). Therefore, limn→∞ |d(xn , yn ) − d(xn , yn )| = 0. Since both d(xn , yn ) and d(xn , yn ) are convergent, it follows that limn→∞ d(xn , yn ) = limn→∞ d(xn , yn ), so e is well-defined. Next, we show that e is a metric on Cauchy(S, Od ). Let x = (xn ), y = (yn ), and z = (zn ) be three Cauchy sequences. We have e([x], [y]) = 0 if and only if limn→∞ d(xn , yn ) = 0, so x ∼ y, which amounts to [x] = [y]. The symmetry of e is immediate. The inequality d(xn , zn )  d(xn , yn ) + d(yn , zn ) implies lim d(xn , zn )  lim d(xn , yn ) + lim d(yn , zn ),

n→∞

n→∞

n→∞

which yields e([x], [z])  d([x], [y]) + d([y], [z]), so e is a metric. For x ∈ S consider the sequence ˜ = (x, x, . . . , x, . . .). x x]. For x, y ∈ S Define the mapping h : S −→ Cauchy(S/ ∼, Oe ) as h(x) = [˜ we have e(h(x), h(y)) = e([˜ x], [˜ y]) = lim d(x, y) = d(x, y), n→∞

so e is an isometry between (S, Od ) and Cauchy(S/ ∼, Oe ).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Metric Space Topologies

page 281

281

Theorem 5.33. Let h : S −→ Cauchy(S/ ∼, Oe ) be the isometry defined above. The set h(S) is dense in Cauchy(S, / ∼, Oe ) and the metric space Cauchy(S/ ∼, Oe ) is complete. Proof. Let [x] ∈ Cauchy(S/ ∼, Oe ) and let be a positive number. Since x = (xn ) is a Cauchy sequence, there exists n such that m, n > n implies d(xm , xn ) < 2 . Let z = xn . Then, [˜z] ∈ h(S) and e([x], [˜ z]) = lim d(xn , z) = lim d(xn , xn )  < , n→∞ n→∞ 2 which shows that [˜ z] ∈ B([x], e) ∩ h(S). Thus, h(S) is dense in S. By Theorem 5.32 it suffices to show that h(S) is complete to obtain the completeness of Cauchy(S/ ∼, Oe ). zk = (zk , zk , . . .) such that ([, zn ]) is Let [, zk ] be an element in S/ ∼, where , zn ], [z, a Cauchy sequence in Cauchy(S/ ∼, Oe ). We have d(zn , zm ) = e([, m ]) because h is an isometry, so z is a Cauchy sequence in (S, Od ). We claim that the sequence ([, zn ]) converges to [z]. Let > 0. There is a n such that k, n  n implies d(zk , zn ) < 2 . Hence, for k  n , e([z-k ], [z]) = lim d(zk , zn )  < , n→∞ 2 which shows that lim ([, zn ]) = [z].

n→∞



Theorem 5.34. Let (S, O) be a topological space, (T, Od ) be a complete metric topological space, and let C(X, Y ) be the set of continuous function from S to T equipped with the metric d∞ (f, g) = supx∈S min{d(f (x), g(x)), 1}. Then C(X, Y ) is a complete metric space. Proof. Let (fn ) be a Cauchy sequence in C(X, Y ). Then, for each x ∈ S the sequence (fn (x)) is a Cauchy sequence in (T, Od ), so we have a function f : S −→ T defined by f (x) = limn→∞ fn (x). By the definition of d∞ for each > 0 there exists n such that n > n implies d∞ (fn , f ) < . If (xi ) is a net in S that converges to x, then for n > n we have d(f (xi ), f (x))  d(f (xi ), fn (xi )) + d(fn (xi ), fn (x)) + d(fn (x), f (x))  2d∞ (fn , f ) + d(fn (xi ), fn (x))  2 + d(fn (xi ), fn (x)). Since fn is continuous we have eventually d(f (xi ), f (x)) < 3 , so f is continuous in x. Thus, C(X, Y ) is complete.  For Y = C we denote the space C(X, C) by C(X).

May 2, 2018 11:28

282

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 282

Mathematical Analysis for Machine Learning and Data Mining

Theorem 5.35. (Tietze’s Theorem) Let (S, O) be a normal topological space and let U be a closed subset of S. Each bounded continuous function f : U −→ R has an extension to a bounded continuous function defined on the entire space S. Proof. Since f is bounded, we may assume without loss of generality that f ∞ = 1. Thus, f (U ) ⊆ [−1, 1]. Let !   ! 1 1 −1 −1 ,1 . A=f −1, − ,B = f 3 3 In other words, x ∈ A if and only if −1  f (x)  − 31 and x ∈ B if and only if 13  f (x)  1. By Uryson’s Lemma, there exists a continuous function f1 : S −→ [− 13 , 13 ] such that  − 1 if x ∈ A, f1 (x) = 1 3 if x ∈ B. 3 This implies |f (x) − f1 (x)|  23 for x ∈ U . Applying the same argument to the function (f − f1 ) U produces a continuous function f2 : S −→ [− 322 , 322 ] on S such that |f2 (x)|  322 such that  2 2 |f (x) − f1 (x) − f2 (x)|  3 when x ∈ U . Thus, we build a sequence of continuous functions (fn ) defined n−1 on S such that |fn (x)|  2 3n and     n n    2   fk (x)  f (x) −   3 k=1

for x ∈ U . Define the function g : S −→ R as g(x) =

∞ 

fn (x)

n=1

for x ∈ S. We have |g(x)| 

∞  2n−1 k=1

3n

= 1,

for x ∈ S, and g U = f . The continuity of g follows from Theorem 5.34. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Metric Space Topologies

5.7

page 283

283

Pointwise and Uniform Convergence

Definition 5.18. Let X be a set and let (T, d) be a metric space. A sequence of functions (fn ), where fn : X −→ T converges pointwise to a function f : X −→ T if for every x ∈ X the sequence (fn (x)) converges to f (x). In other words, the following formula is valid: (∀ > 0)(∀x ∈ X)(∃n ,x )(n  n ,x ⇒ d(fn (x), f (x)) < ). The sequence (fn ) converges uniformly to f if lim sup d(fn (x), f (x)) = 0,

n→∞ x∈X

that is, the following first-order formula is valid: (∀ > 0)(∃n )(∀x ∈ X)(n  n ⇒ d(fn (x), f (x)) < ). Note the difference in the order of the quantifier symbols in the above formulas. In the case of the pointwise convergence the number n ,x depends both on and on x; in the case of the uniform convergence the similar number depends only on . Example 5.13. Let fn : [0, 1] −→ R be the function defined by  |nx − 1| if 0  x  n2 , fn (x) = 1 if n2 < x  1 for x ∈ [0, 1]. The graph of fn is shown in Figure 5.1. The sequence of functions (fn ) converges pointwise to the constant function f : [0, 1] −→ [0, 1] given by f (x) = 1, with the usual metric on R. Indeed, we have:  1 − |nx − 1| if 0  x  n2 , |fn (x) − f (x)| = 0 if n2 < x  1 ⎧ ⎪ if 0  x  n1 , ⎪ ⎨nx = 2 − nx if n1 < x  n2 , ⎪ ⎪ ⎩0 if 2 < x  1. n

Thus, given x and > 0, if n  2/x! we shall have |fn (x) − f (x)| < . However, the convergence of (fn ) to f is not uniform. Indeed, it is impossible to find a number n independent of x such that |fn (x)−f (x)| < because |fn (1/n) − f (1/n)| = 1 for n  1.

May 2, 2018 11:28

284

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 284

Mathematical Analysis for Machine Learning and Data Mining

y 1

x 1 n

Fig. 5.1

1 Continuous function on [0, 1].

Theorem 5.36. Let (S, Od ) and (T, Oe ) be two topological metric spaces. If (fn ) a sequence of continuous functions from S to T that converges uniformly to a function f : S −→ T then f is continuous. Proof. Let be a positive number. Since (fn ) converges uniformly to f there exists a natural number n such that n  n implies e(fn (s), f (s)) < 3 for every s ∈ S. Since fn is a continuous function there exists a δ > 0 such that d(s, s ) < δ implies e(fn (s), fn (s )) < 3 , Thus, if d(s, s ) < δ we have e(f (s), f (s ))  e(f (s), fn (s)) + e(fn (s), fn (s )) + e(fn (s ), f (s )) < , which shows that f is continuous.



Theorem 5.37. (Dini’s1 Theorem) Let (S, Od ) be a compact metric space and let (fn ) be a sequence of continuous functions fn : S −→ R for n ∈ N that converges pointwise to f : S −→ R such that fn (x)  fn+1 (x) for all x ∈ S and n ∈ N. Then (fn ) converges uniformly to f . Proof. Let gn : S −→ R be defined as gn (x) = fn (x)−f (x) for n ∈ N and x ∈ S. Then (gn ) converges pointwise to the constant function 0, gn (x)  gn+1 (x)  0 for x ∈ S and n ∈ N. Define Mn = sup{gn (x) | x ∈ S}. 1 Ullise Dini was born in Pisa, Italy, on November 14th 1845 and died on October 28th 1918. He is known for his contribution to real analysis. Dini served as rector of the Pisa University, and of the Scuola Normale Superiore and was active in Italian politics.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Metric Space Topologies

page 285

285

Define the set Tn = gn−1 (−∞, ) for some positive . Since gn is continuous, Tn is an open set. Also Tn ⊆ Tn+1 . For each x ∈ S, limn→∞ gn (x) = 0, so there exists n ∈ N such that  gn (x) < , hence x ∈ Tn . Thus, S = n∈N Tn . Since S is compact, there exists a finite subcover {Ti1 , . . . , Tim } of S with i1  · · ·  im . Since Ti1 ⊆ · · · ⊆ Tim of S, this finite subcover reduces to Tim . This means that gim (x) < for all x ∈ S. Thus, Mim  for all x ∈ S. Since Mn decreases  with n and every Mn  0, it follows that limn→∞ Mn = 0. If any of the conditions of Dini’s theorem fails the convergence of fn to f may not be uniform. Example 5.14. Let fn (x) = xn be functions defined on the non-compact set S = (0, 1). The sequence (fn ) converges pointwise to 0. However, the convergence is not uniform because sup{xn | x ∈ (0, 1)} = 1 for n ∈ N. Let S = [0, 1] be compact and let fn be defined as  1 − nx if 0  x  n1 , fn (x) = 1 if n1  x  1. It is clear that (fn ) converges pointwise to the function f given by  1 if x = 0, f (x) = 0 if 0 < x  1, which is discontinuous in 0. The convergence of (fn ) to f is not uniform because sup{fn (x) − f (x) | x ∈ [0, 1]} = 1. Theorem 5.38. Let (S, Od ) and (T, Oe ) be two topological metric spaces. Suppose that (T, Oe ) is complete. The sequence of functions (fn ) converges uniformly on S if and only if for every > 0 there exists n such that m, n  n , x ∈ S imply e(fn (x), fm (x)) < . Proof. Suppose that (fn ) converges uniformly on S and let f = limn→∞ fn . There is n /2 such that n > n /2 implies e(fn (x), f (x)) < 2 , hence m, n > n /2 imply e(fn (x), fm (x))  e(fn (x), f (x)) + e(f (x), fm (x)) < . Conversely, suppose that the condition of the theorem holds. Since (T, Oe ) is complete, the sequence (fn (x)) converges for every x. Define f (x) = limn→∞ fn (x). Let n ∈ N be such that m, n > n implies e(fm (x), fn (x)) < . For any n > n we have limm→∞ e(fm (x), fn (x)) = e(f (x), fn (x)) < for every n > n , hence (fn ) converges uniformly to f . 

May 2, 2018 11:28

286

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 286

Mathematical Analysis for Machine Learning and Data Mining

Let C(S) be the linear space of real-valued bounded continuous functions of the form f : S −→ R, where (S, O) is a topological space. For f ∈ C(S) define f ∞ = sup{|f (x)| | x ∈ S}. We claim that  · ∞ is a norm on C(S). Since f is bounded, f ∞ < ∞. If f ∞ = 0, it is clear that f (x) = 0 for every x ∈ S. Note that h = f + g implies |h(x)|  |f (x)| + g(x)  f ∞ + g∞ , hence h∞  f ∞ + g∞, that is f + g∞  f ∞ + g∞, so  · ∞ is indeed a norm. This implies that d∞ : C(S)2 −→ R defined by d∞ (f, g) = f − g∞ for f, g ∈ C(S) is a metric on C(S). We shall refer to f ∞ as the supremum norm of a function f ∈ C(S). An alternative term used for  · ∞ is the uniform convergence norm. This is motivated by the fact that if limn→∞ fn − f ∞ = 0, then fn converges uniformly to f on S. If (S, O) is compact, we have f ∞ (f ) = max{|f (x)| | x ∈ S}. Theorem 5.39. The metric space (C(S), d∞ ) is complete. Proof. Let (fn ) be a Cauchy sequence in C(S). This implies that for every > 0 there exists n such that m, n > n implies |fn (x) − fm (x)| < . Since R is complete, by Theorem 5.38 there exists a function f : S −→ R such that (fn ) converges uniformly to f . By Theorem 5.36, f is continuous. Furthermore, f is bounded because there is n1 such that |fn1 (x)−f (x)| < 1  for x ∈ S. Thus, f ∈ C(S) and limn→∞ f − fn ∞ = 0. 5.8

The Stone-Weierstrass Theorem

Theorem 4.73 shows that the set of real-valued continuous functions defined on a topological space (S, O) is an algebra. Definition 5.19. A subset U of C(S) separates points if for every x, y ∈ S such that x = y there exists a function f ∈ U such that f (u) = f (v). Example 5.15. The algebra of all polynomials defined on an interval [a, b] separates points. Indeed, let c, d ∈ [a, b] with c = d and p be the polynomial 1 (x − d) for x ∈ [a, b]. Since p(c) = 1 and p(d) = 0, the algebra p(x) = c−d of all polynomials separates points. On other hand the set of even functions on [−1, 1] is an algebra, but it does not separate points because f (−x) = f (x) for x ∈ [−1, 1].

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Metric Space Topologies

b3234-main

page 287

287

The Stone-Weierstrass theorem shows that the continuous functions of the form f : S −→ R (where (S, O) is a compact topological space) can be uniformly approximated by the functions that belong to a subalgebra U of C(S) that has certain properties. The next two lemmas are preliminaries to an elementary proof of this important result. In the next lemma we make use of Bernoulli’s inequality (Exercise 49 of Chapter 1). Lemma 5.1. Let (S, O) be a compact topological space, U be an open subset in this space, and let t0 ∈ U . If U is a subalgebra of C(S) that contains the constant functions and separates points, then there exists V ∈ neight0 (O), V ⊆ U such that for every > 0 there exists f ∈ U that satisfies the following conditions: (i) for every x ∈ S, 0  f (x)  1; (ii) f (x) < for x ∈ V ; (iii) f (x) > 1 − for x ∈ S − U . Proof. Let ca be the constant function in U whose value is a. Let t ∈ S − U . Then, since t = t0 and U separates points, there exists a function gt ∈ U such that gt (t) = gt (t0 ). Then, the function ht defined by ht = gt − gt (t0 ) · c1 belongs to U and ht (t) = gt (t) − gt (t0 ) = ht (t0 ) = 0. Define ν(f ) = sup{|f (x)| | x ∈ S}.

(5.2)

Let pt be the function given by pt = ν(h1t )2 (ht )2 . Observe that pt ∈ U, pt (t0 ) = 0, and pt (t) > 0. In addition, c0  pt  c1 . The set U (t) = {s ∈ S | pt (s) > 0} is an open neighborhood of t. Since (S, O) is compact and U is open, S −U is closed and, therefore, it is compact (by Theorem 4.46). The compactness of S − U implies the existence of a m finite number of points t1 , . . . , tm in S − U such that S − U = i=1 U (ti ).  m 1 Let p be the function defined by p(t) = m i=1 pti . Then p ∈ U, c0  p  c1 , p(t0 ) = 0 and p(t) > 0 for t ∈ S − U . Since S − U is compact, there exists δ ∈ (0, 1) such that p(t)  δ for t ∈ S − U . Let V be the set defined by V = {t ∈ S | p(t) < 2δ }. Then V is an open neighborhood of t0 and V ⊆ U . Let k be the smallest integer that is greater than 1δ . Then, k − 1  1δ , 2 hence k  1+δ δ < δ . Thus, 1 < kδ < 2.

May 2, 2018 11:28

288

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 288

Mathematical Analysis for Machine Learning and Data Mining

For n ∈ N and n  1 consider the functions qn defined by n

qn (t) = (1 − pn (t))k . We have qn ∈ U, c0  qn  c1 and qn (t0 ) = 1. For t ∈ V we have kp(t)  k δ2 < 1, so  n kδ n . qn (t)  1 − (kp(t))  1 − 2 Thus, limn→∞ qn (t) = 1 uniformly on V . For t ∈ S − U , kp(t)  kδ > 1 and, by Bernoulli’s inequality: n 1 qn (t) = n n (1 − pn (t))k k n pn (t) k p (t) n 1  (1 − pn (t))k (1 + k n pn (t)) (kp(t))n n n 1  (1 − pn (t))k (1 + pn (t))k n (kp(t)) n 1 1 = (1 − p2n (t))k  . n (kp(t)) (kδ)n Therefore, limn→∞ qn (t) = 0, and this happens uniformly on S − U . Thus, for n sufficiently large, qn has the property that c0  qn  c1 , qn (t) < on S − U and q( t) > 1 − on V . The Lemma follows by taking f = c1 − qn .  Lemma 5.2. Let (S, O) be a compact topological space, t0 ∈ S, and let U be an open set in (S, O) that contains t0 . If U is a subalgebra of C(S) that contains the constant functions and separates points, and A, B are two closed and disjoint subsets of S, then there exists f ∈ U such that: (i) 0  f (t)  1 for t ∈ S; (ii) f (t) < for t ∈ A; (iii) f (t) > 1 − for t ∈ B. Proof. Let U = S − B. Since B is closed, U is an open set. For each t ∈ A chose the open set V (t) that contains t as in Lemma 5.1. There exists a finite set of points {t1 , . . . , tm } ⊆ A such that A ⊆ m i=1 V (ti ). By the choice of V (ti ), there exist fi ∈ U for 1  i  m such that c0  fi  c1 , fi (x) < m when x ∈ V (ti ) and fi (x) > 1 − m m when x ∈ S − U = B. Then, the function f = i=1 fi belongs to U, m c0  f  c1 , f (x)  < m m when x ∈ i=1 ⊇ A and, by Bernoulli’s > 1 − on B.  inequality f (x) > 1 − m

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Metric Space Topologies

page 289

289

Theorem 5.40. (Stone-Weierstrass Theorem) If (S, O) is a compact topological space and U is a subalgebra of C(S) that satisfies the following conditions: (i) U contains the constant functions; (ii) U separates points. Then for every element f ∈ C(S) and > 0, there exists g ∈ U such that supx∈S |f (x) − g(x)| < . Proof. By replacing f by f + ν(f ), where ν was defined in equality (5.2) we can assume that f (x)  0 for x ∈ S. We can assume that < 13 . Let n be an integer such that (n − 1)  ν(f ). For j ∈ N define the sets     1 1 } and Bj = {x ∈ S | f (x)  j + }. Aj = {x ∈ S | f (x)  j − 3 3 The sets Aj and Bj are disjoint and ∅ = A0 ⊂ A1 ⊂ · · · ⊂ An = S andB0 ⊃ B1 ⊃ · · · ⊃ Bn = ∅. Lemma 5.2 implies the existence of fj ∈ U, such that 0  fj (t)  1 for t ∈ S, fj (t) < 3 for t ∈ Aj and fj (t) > 1 − n for t ∈ Bj . The function n g = i=0 fi belongs to U. For t ∈ S we have t ∈ Aj − Aj−1 for some j  1, which implies     1 4 < f (t) < j − (5.3) j− 3 3 and fi (t)
1 −

n

(5.5)

for every i  j − 2. Using inequalities (5.4) we obtain g(t) =

j−1  i=0

fi (t) +

n 

fi (t)

i=j

 j + (n − j + 1) n   1 2  j + < j + . 3

May 2, 2018 11:28

290

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 290

Mathematical Analysis for Machine Learning and Data Mining

Using inequalities (5.5) we obtain for j  2: j−2 #  $ g(t)  fi (t)  (j − 1) 1 − n i=0

  4 j− . 3 The inequality g(t) > (j − 4/3) is trivially true for j = 1. Thus,     4 1 − j− < 2 . |f (t) − g(t)|  j + 3 3 = (j − 1) − ((j − 1)/n) 2 > (j − 1) − 2 >



Stone-Weierstrass Theorem can be extended to algebras of complex continuous functions by adding a supplemental requirement. Theorem 5.41. (Complex Stone-Weierstrass Theorem) Let (S, O) be a compact topological space and U be an algebra of complex continuous functions that satisfies the following conditions: (i) U contains the constant functions; (ii) U separates points; (iii) for every f ∈ U its complex conjugate f defined as f (x) = f (x) for x ∈ S belongs to U. Then for every complex continuous function f ∈ C(S) and > 0, there exists g ∈ U such that supx∈S |f (x) − g(x)| < . Proof. Let Ur be the set of all real-valued functions on S that belong to U. If f ∈ U we have f ∈ U and since the function h = 12 (f + f ) is real-valued, we have h ∈ Ur . It is immediate that Ur is an algebra that contains the real-valued constant functions. Since U separates points, if c, d ∈ S and c = d, there exists a function

∈ U such that (c) = (d). Since U is an algebra that contains constants, the function f defined by f (x) = (x)−(c) (d)−(c) also belongs to U and, f (c) = 0 and f (d) = 1. Then h(c) = 0 and h(d) = 1, which shows that Ur also separates points. The algebra Ur satisfies the conditions of Theorem 5.40, so for every real-valued continuous function h ∈ C(S) and > 0, there exists g ∈ U such that supx∈S |h(x) − g(x)| < . If f : S −→ C is a complex valued continuous functions, then for the real-valued functions f1 = (f ) and f2 = (f ) there exist g1 , g2 ∈ Ur such that supx∈S |f1 (x) − g1 (x)| < 2 and supx∈S |f2 (x) − g2 (x)| < 2 . Define g = g1 + ig2 . We have |f (x) − g(x)| = |f1 (x) + if2 (x) − g1 (x) − ig2 (x)| = |f1 (x) − g1 (x) + i(f2 (x) − g2 (x))|  |f1 (x) − g1 (x)| + |f2 (x) − g2 (x)| < .



May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Metric Space Topologies

5.9

page 291

291

Totally Bounded Metric Spaces

Definition 5.20. An r-net for a subset T of a metric space (S, Od ) is a subset U of S such that for every x ∈ T there exists u ∈ U such that x ∈ B(u, r). A finite r-net is an r-net U such that U is a finite set. This definition allows us to introduce the notion of totally bounded or precompact set. Definition 5.21. A subset T of a metric space (S, Od ) is totally bounded (or precompact ) if, for every positive number r there exists an r-net for T . If S is totally bounded or precompact, we say that (S, Od ) is totally bounded or precompact. (S, Od ) is compact, then it is totally bounded. Next, we show that total boundedness is inherited by subsets. Theorem 5.42. If (S, Od ) is a totally bounded topological metric space and T ⊆ S, then the subspace (T, Od T ) is also totally bounded. Proof. Since (S, Od ) is totally for every r > 0 there exists  bounded, r | s = {B s , ∈ S, 1  i  n}. Let C = a finite open cover C i i r/2 2  r {B sij , 2 | 1  j  m} be a minimal subcollection of Cr/2 that consists of those open spheres that cover T ; that is,  *  # r $  1jm . B sij , T ⊆ 2   r The minimality of C implies contains an element that each set B s i j, 2  r yj of T . We have B sij , 2 ⊆ B(yj , r) and this implies that the set  {y1 , . . . , ym } is an r-net for the set T . If the subspace (T, Od T ) of (S, Od ) is totally bounded, we say that the set T is totally bounded. Example 5.16. In Example 4.21 we saw that the metric space ([0, 1], d), where d is the usual metric on [0, 1] is compact. However, the set (0, 1) is not compact, but it is totally bounded (or, precompact). The next corollary shows that there is no need to require the centers of the spheres involved in the definition of the total boundedness of a subspace to be located in the subspace.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 292

Mathematical Analysis for Machine Learning and Data Mining

292

Corollary 5.10. Let (S, Od ) be a topological metric space (not necessarily totally bounded) and let T be a subset of S. The subspace (T, Od T ) is totally bounded if and only if for every positive number r there exists a finite subcover {B(x1 , r), . . . , B(xn , r) | xi ∈ S for 1  i  n}. Proof.



The argument has been made in the proof of Theorem 5.42.

The next theorem adds two further equivalent characterizations of compact metric spaces to the ones given in Theorem 5.22. Theorem 5.43. Let (S, Od ) be a topological metric space. The following statements are equivalent. (i) (S, Od ) is compact; (ii) every sequence x ∈ Seq∞ (S) contains a convergent subsequence; (iii) (S, Od ) is totally bounded and complete. Proof. (i) implies (ii): Let (S, Od ) be a compact topological metric space and let x be a sequence in Seq∞ (S). By Theorem 5.22, (S, Od ) has the Bolzano-Weierstrass property, so the set {xn | n ∈ N} has  an accumulation point t. For every k  1, the set {xn | n ∈ N} ∩ B t, k1 contains an element xnk distinct from t. Since d(t, xnk ) < k1 for k  1, it follows that the subsequence (xn1 , xn2 , . . .) converges to t. (ii) implies (iii): Suppose that every sequence x ∈ Seq∞ (S) contains a convergent subsequence and that (S, Od ) is not totally bounded. Then, there exists a positive number r such that S cannot be covered by any collection of open spheres of radius r. Let x0 be an arbitrary element of S. Note that B(x0 , r)− S = ∅ because otherwise the B(x0 , r) would constitute an open cover for S. Let x1 be an arbitrary element in B(x0 , r) − S. Observe that d(x0 , x1 )  r. The set (B(x0 , r) ∪ B(x1 , r)) − S is not empty. Thus, for any x2 ∈ (B(x0 , r) ∪ B(x1 , r)) − S, we have d(x0 , x2 )  r and d(x0 , x1 )  r, etc. We obtain in this manner a sequence x0 , x1 , . . . , xn , . . . such that d(xi , xj )  r when i = j. Clearly, this sequence cannot contain a convergent sequence, and this contradiction shows that the space must be totally bounded. To prove that (S, Od ) is complete, consider a Cauchy sequence x = (x0 , x1 , . . . , xn , . . .). By hypothesis, this sequence contains a convergent subsequence (xn0 , xn1 , . . .). Suppose that limk→∞ xnk = l. Since x is a Cauchy sequence, there is n such that n, nk  n implies d(xn , xnk ) < 2 2 2 . The convergence of the subsequence (xn0 , xn1 , . . .) means that there exists n such that nk  n implies d(xnk , l) < 2 . Choosing nk  n , if 2

2

2

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Metric Space Topologies

b3234-main

page 293

293

n  n = n , we obtain 2

+ = , 2 2 which proves that x is convergent. Consequently, (S, Od ) is both totally bounded and complete. (iii) implies (i): Suppose that (S, Od ) is both totally bounded and complete but not compact, which means that there exists an open cover C of S that does not contain any finite subcover. Since (S, Od ) is totally bounded, there exists a 12 -net, {x11 , . . . , x1n1 }. For each of the closed spheres B[x1i , 12 ], 1  i  n1 , the trace collection CB[x1i , 12 ] is an open cover. There is a closed sphere B[x1j , 12 ] such that the open cover CB[x1j , 12 ] does not contain any finite subcover of B[x1j , 12 ] since (S, Od ) was assumed not to be compact. Let z1 = x1j . By Theorem 5.42, the closed sphere B[z1 , 12 ] is totally bounded. Thus, there exists a 212 -net {x21 , . . . , x2n2 } of B[z1 , 12 ]. There exists a closed sphere B[x2k , 212 ] such that the open cover CB[x2k , 12 ] does not contain any finite d(xn , l)  d(xn , xnk ) + d(xnk , l)
0 for which d (f (x), f (y)) = cd(x, y) for every x, y ∈ S. If the two metric spaces coincide, we refer to f as a self-similarity of (S, d). The number c is called the ratio of the similarity f and is denoted by ratio(f ). Note that an isometry is a similarity of ratio 1.

May 2, 2018 11:28

296

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 296

Mathematical Analysis for Machine Learning and Data Mining

Example 5.20. Let (R, d) be the metric space defined by d(x, y) = |x − y|. Any linear mapping (that is, any mapping of the form f (x) = ax + b for x ∈ R) is a similarity having ratio a. Definition 5.23. Let (S, d) and (T, d ) be two metric spaces. If there exists c > 0 such that d (f (x), f (y))  cd(x, y)

(5.6)

for all x, y ∈ S, then we say that f is a Lipschitz function.2 A number c that occurs in inequality (5.6) is a Lipschitz constant for f . If there exists a Lipschitz constant c < 1 for f , then f is a contraction with the contraction constant c. A function f : S −→ T is locally Lipschitz at x0 ∈ S if there exists c > 0 such that d (f (x0 ), f (y))  cd(x0 , y) for every y ∈ S. Theorem 5.47. Let (S, Od ) and (T, Od ) be two metric spaces. Every Lipschitz function f : S −→ T is uniformly continuous. Proof. Suppose that d (f (x), f (y))  cd(x, y) for x, y ∈ S and let be a positive number. Define δ = c . If z ∈ f (B(x, δ)), there exists y ∈ B(x, δ) such that z = f (y). This implies d (f (x), z) = d(f (x), f (y)) < cd(x, y) < cδ = , so z ∈ B(f (x), ). Thus, f (B(x, δ)) ⊆ B(f (x), ), which means that f is uniformly continuous.  It is easy to see, using a similar argument, that a locally Lipschitz function at x0 is continuous in x0 . Example 5.21. Let a > 0 and let f : (0, a) −→ R be the function defined by f (x) = x2 . We have |f (x) − f (y)| = |x2 − y 2 | = (x + y)|x − y|  2a|x − y|, so f is a Lipschitz function with the constant 2a. Therefore, f is uniform continuous on (0, a).

2 Rudolf Lipschitz was born near K¨ onigsberg, Germany (now Kaliningrad, Russia) on May 14th 1832 and died in Bonn, Germany on October 7th 1903. He studied mathematics at the University of K¨ onigsberg and obtained his doctorate at the University of Berlin in 1857. Lipschitz taught at universities of Berlin, Breslau and Bonn. He has important contributions in number theory, Fourier series, differential equations, and in analytical mechanics.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Metric Space Topologies

page 297

297

A function f may be uniformly continuous on a set without being a Lipschitz function as the next example shows. √ Example 5.22. Let f : (0, ∞) −→ R be the function defined by f (x) = x for x > 0. Suppose that f satisfies a Lischitz condition, that is, there exists √ √ √ √ c > 0 such that | x − y|  c|x − y|. This is equivalent to 1  c( x + y) for every x, y > 0. Choosing x = y = 9c12 leads to a contradiction. On other hand, f is uniformly continuous. Indeed, note that for every √ √ √ u, v  0 we have u + v  u + v. If |x − y| < 2 , then either x  y < x + 2 or y  x < y + 2 . The first inequality implies " √ √ √ x  y < x + 2  x + . The second implies

" √ √ y  x < y + 2  y + , √ √ so, in either case, | x− y| < , which shows that f is uniformly convergent on (0, ∞). √

A generalization of the class of Lipschitz functions is given next. Definition 5.24. Let (S, d) and (T, d ) be two metric spaces. If there exists c > 0 and a ∈ (0, 1] such that d (f (x), f (y))  cd(x, y)a for all x, y ∈ S, then we say that f is an a-H¨ older function on S. Note that if a, b ∈ (0, 1], where a  b, and f : S −→ T is a b-H¨older function, then f is also an a-H¨older function. In general, Lipschitz functions are a-H¨ older functions for a  1, which are, in turn uniformly continuous on S. Theorem 5.47 implies that every similarity is uniformly continuous. Let f : S −→ S be a function. We define inductively the functions (n) : S −→ S for n ∈ N by f f (0) (x) = x and f (n+1) (x) = f (f (n) (x)) for x ∈ S. The function f (n) is the nth iteration of the function f . Example 5.23. Let f : R −→ R be the function defined by f (x) = ax + b for x ∈ R, where a, b ∈ R and a = 1. It is easy to verify that f (n) (x) = n −1 · b for x ∈ R. an x + aa−1

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 298

Mathematical Analysis for Machine Learning and Data Mining

298

Definition 5.25. Let f : S −→ S be a function. A fixed point of f is a member x of the set S that satisfies the equality f (x) = x. Example 5.24. The function f defined in Example 5.23 has the fixed point b . x0 = 1−a Theorem 5.48. (Banach Fixed Point Theorem) Let (S, Od ) be a complete topological metric space and let f : S −→ S be a contraction on S having the contraction ratio c. There exists a unique fixed point u ∈ S for f , and for any x ∈ S we have limn→∞ f (n) (x) = u. Moreover, we have d(f (n) (x), u) 

cn d(x, f (x)) 1−c

for x ∈ S and n ∈ N. Proof. Since f is a contraction, there exists a positive number c ∈ (0, 1), such that d(f (x), f (y))  cd(x, y) for x, y ∈ S. Note that each such function has at most one fixed point. Indeed, suppose that we have both u = f (u) and v = f (v) and u = v, so d(u, v) > 0. Then, d(f (u), f (v)) = d(u, v)  cd(u, v), which is absurd because c < 1. The sequence s = (x, f (x), . . . , f (n) (x), . . .) is a Cauchy sequence. Indeed, observe that d(f (n) (x), f (n+1) (x))  c d(f (n−1) (x), f (n) (x))  · · ·  cn d(x, f (x)). For n  p, this implies d(f (n) (x), f (p) (x))  d(f (n) (x), f (n+1) (x)) + d(f (n+1) (x), f (n+2) (x)) + · · · + d(f (p−1) (x), f (p) (x))  cn d(x, f (x) + · · · + cp−1 d(x, f (x)) cn d(x, f (x)),  1−c which shows that the sequence s is indeed a Cauchy sequence. By the completeness of (S, Od ), there exists u ∈ S such that u = limn→∞ f (n) (x). The continuity of f implies u = lim f (n+1) (x) = lim f (f (n) (x)) = f (u), n→∞

so u is a fixed point of f . Since d(f (n) (x), f (p) (x)) 

n→∞

cn 1−c d(x, f (x)),

we have

lim d(f (n) (x), f (p) (x)) = d(f (n) (x), u) 

p→∞

for n ∈ N.

cn d(x, f (x)) 1−c 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Metric Space Topologies

b3234-main

page 299

299

Theorem 5.49. Let B[x0 , r] be a closed sphere in a complete metric space (S, Od ) and let f : B[x0 , r] −→ S be a contraction with a contraction constant c, c ∈ (0, 1), such that d(f (x0 ), x0 )  (1 − c)r. Then, f has a unique fixed point u in B[x0 , r] and d(f (n) (x), u)  rcn d(x, f (x0 )) for x ∈ B[x0 , r] and n ∈ N. Proof.

Let x ∈ B[x0 , r]. We have d(f (x), x0 )  d(f (x), f (x0 )) + d(f (x0 ), x0 )  cd(x, x0 ) + (1 − c)r  cr + (1 − c)r = r.

This shows that f (B[x0 , r]) ⊆ B[x0 , r], hence f is a transformation of B[x0 , r]. The statement now follows immediately from the Banach Fixed Point Theorem (Theorem 5.48).  Corollary 5.12. Let B(x0 , r) be an open sphere in a complete metric space (S, Od ). If f : B(x0 , r) −→ S is a contraction with the contraction constant c, c ∈ (0, 1), such that d(f (x0 ), x0 ) < (1 − c)r, then f has a unique fixed point in B(x0 , r). Proof. Let B(x0 , r) be an open sphere and let sphere B[x0 , r − ] be a closed sphere included in B(x0 , r). By Theorem 5.49, f has a unique fixed point u ∈ B[x0 , r − ] such that d(f (n) (x), u)  (r − )cn d(x, f (x0 )) for x ∈ B[x0 , r − ] and n ∈ N. Thus, the fixed point belongs to B(x0 , r).  Theorem 5.50. Let (S, Od ) be a complete topological metric space, f : S −→ S be a contraction on S having the contraction ratio c ∈ (0, 1) and let x ∈ S. If u is the unique fixed point of f , then d(f (x), x) . 1−c . / (x),x) Proof. Consider the closed sphere B x, d(f1−c . By Theorem 5.49 applied to the restriction of f to this closed sphere, it follows that u belongs to this sphere.  d(x, u) 

May 2, 2018 11:28

300

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 300

Mathematical Analysis for Machine Learning and Data Mining

Theorem 5.51. Let (S, Od ) be a complete topological metric space and let (T, e) be any metric space. Suppose that h : T × S −→ S is a function such that d(h(t, x), h(t, y))  cd(x, y) (uniformly relative to t ∈ T ), where c ∈ (0, 1), for t ∈ T and x, y ∈ S. Furthermore, assume that h(t, x) is continuous in t for each fixed x ∈ S. For each t ∈ T define ψ(t) to be the unique fixed point ut satisfying h(t, ut ) = ut . The mapping ψ is continuous. Proof. Let s ∈ T and let be a positive number. By the continuity of h there exists δ > 0 such that e(t, s) < δ implies d(h(t, us ), h(s, us )) < . Since h(s, us ) = us , we have d(h(t, us ), us ) < , so us is moved at most , which shows by h(t, ·). Thus, by Theorem 5.50, we have d(ut , us )  1− that ψ is continuous.  Corollary 5.13. Let (S, Od ) be a complete topological metric space, (T, e) be any metric space and let B(x, r) be an open sphere in S. Suppose that h : T × B(x, r) −→ S is a function such that d(h(t, x), h(t, y))  cd(x, y) (uniformly relative to t ∈ T ), where c ∈ (0, 1), for t ∈ T and x, y ∈ B(x, r). Furthermore, assume that h(t, x) is continuous in t for each fixed x ∈ S and that d(h(t, x), x) < (1 − c)r for every t ∈ T . There exists a continuous mapping ψ : T −→ B(x, r) such that for each t ∈ T h(t, ψ(t)) = ψ(t). Proof.

5.11

This statement follows from Theorems 5.51 and 5.50.



The Hausdorff Metric Hyperspace of Compact Subsets

Lemma 5.3. Let (S, d) be a metric space and let U and V be two subsets of S. If r ∈ R0 is such that U ⊆ B(V, r) and V ⊆ B(U, r), then we have |d(x, U ) − d(x, V )|  r for every x ∈ S. Proof. Since U ⊆ B(V, r), for every u ∈ U there is v ∈ V such that d(u, v) < r. Therefore, by the triangular inequality, it follows that for every u ∈ U there is v ∈ V such that d(x, u) < d(x, v)+r, so d(x, U ) < d(x, v)+r. Consequently, d(x, U )  d(x, V )+ r. In a similar manner, we can show that V ⊆ B(U, r) implies d(x, V )  d(x, U ) + r. Thus, |d(x, U ) − d(x, V )|  r for every x ∈ S.  Let (S, Od ) be a topological metric space. Denote by K(S, Od ) the collection of all non-empty, compact subsets of (S, Od ), and define the mapping

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Metric Space Topologies

page 301

301

δ : K(S, Od )2 −→ R0 by δ(U, V ) = inf{r ∈ R0 | U ⊆ B(V, r) and V ⊆ B(U, r)} for U, V ∈ K(S, Od ). Lemma 5.4. Let U and V be two compact subsets of a topological metric space (S, Od ). We have  * sup |d(x, U ) − d(x, V )| = max sup d(x, U ), sup d(x, V ) . x∈S

x∈V

x∈U

Proof. Let x ∈ S. There is v0 ∈ V such that d(x, v0 ) = d(x, V ) because V is a compact set. Then, the compactness of U implies that there is u0 ∈ U such that d(u0 , v0 ) = d(v0 , U ). We have d(x, U ) − d(x, V ) = d(x, U ) − d(x, v0 )  d(x, u0 ) − d(x, v0 )  d(u0 , v0 )  sup d(U, x). x∈V

Similarly, d(x, U ) − d(x, V )  supx∈U d(x, V ), which implies  * sup |d(x, U ) − d(x, V )|  max sup d(x, U ), sup d(x, V ) . x∈S

x∈V

x∈U

On the other hand, since U ⊆ S, we have sup |d(x, U ) − d(x, V )|  sup |d(x, U ) − d(x, V )| = sup d(x, V ) x∈S

x∈U

x∈U

and, similarly, supx∈S |d(x, U ) − d(x, V )|  supx∈V d(x, U ), and these inequalities prove that  * sup |d(x, U ) − d(x, V )|  max sup d(x, U ), sup d(x, V ) , x∈S

x∈V

x∈U



which concludes the argument. An equivalent useful definition of δ is given in the next theorem.

Theorem 5.52. Let (S, d) be a metric space and let U and V be two compact subsets of S. We have the equality δ(U, V ) = sup |d(x, U ) − d(x, V )|. x∈S

Proof. Observe that we have both U ⊆ B(V, supx∈U d(x, V )) and V ⊆ B(U, supx∈V d(x, U )). Therefore, we have  * δ(U, V )  max sup d(x, U ), sup d(x, V ) . x∈V

x∈U

Combining this observation with Lemma 5.4 yields the desired equality. 

May 2, 2018 11:28

302

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 302

Mathematical Analysis for Machine Learning and Data Mining

Theorem 5.53. Let (S, Od ) be a complete topological metric space. The mapping δ : K(S, Od )2 −→ R0 is a metric on K(S, Od ). Proof. It is clear that δ(U, U )  0 and that δ(U, V ) = δ(V, U ) for every U, V ∈ K(S, Od ). Suppose that δ(U, V ) = 0; that is, d(x, U ) = d(x, V ) for every x ∈ S. If x ∈ U , then d(x, U ) = 0, so d(x, V ) = 0. Since V is closed, by part (ii) of Theorem 5.16, we have x ∈ V , so U ⊆ V . The reverse inclusion can be shown in a similar manner. To prove the triangular inequality, let U, V, W ∈ K(S, Od ). Since |d(x, U ) − d(x, V )|  |d(x, U ) − d(x, V )| + |d(x, V ) − d(x, W )|, for every x ∈ S, we have sup |d(x, U ) − d(x, V )|  sup (|d(x, U ) − d(x, V )| + |d(x, V ) − d(x, W )|) x∈S

x∈S

 sup |d(x, U ) − d(x, V )| + sup |d(x, V ) − d(x, W )|, x∈S

x∈S

which implies the triangular inequality δ(U, V )  δ(U, W ) + δ(W, V ).



The metric δ is known as the Hausdorff metric, and the metric space (K(S, Od ), δ) is known as the Hausdorff metric hyperspace of (S, Od ). Theorem 5.54. If (S, Od ) is a complete topological metric space, then so is the Hausdorff metric hyperspace (K(S, Od ), δ). Proof. Let U = (U0 , U1 , . . .) be a Cauchy sequence in (K(S, Od ), δ) and  let U = K( n∈N Un ). It is clear that U consists of those elements x of S such that x = limn→∞ xn for some sequence x = (x0 , x1 , . . .), where xn ∈ Un for n ∈ N. The set U is totally bounded. Indeed, let > 0 and let n0 be such that δ(Un , Un0 )  for n  n0 . Let N be an -net for the compact set  H = nn0 Un . Clearly, H ⊆ B(N, ). Since δ(Un , Un0 )  , it follows that U ⊆ B(H, ), so U ⊆ B(N, 2 ). This shows that U is totally bounded. Since U is closed in the complete space (S, Od ), it follows that U is compact. Let be a positive number. Since U is a Cauchy sequence, there exists n 2 such that m, n  n 2 implies δ(Um , Un ) < 2 ; that is, sups∈S |d(s, Um ) − d(s, Un )| < 2 . In particular, if xm ∈ Um , then d(xm , Un ) = inf y∈Um d(x, y) < 2 , so there exists y ∈ Un such that d(xm , y) < 2 . For x ∈ U , there exists a sequence x = (x0 , x1 , . . .) such that xn ∈ Un for n ∈ N and limn→∞ xn = x. Therefore, there exists a number n such that 2

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Metric Space Topologies

9in x 6in

b3234-main

page 303

303

p  n implies d(x, xp ) < 2 . This implies d(x, y)  d(x, xp ) + d(xp , y)  2 if n  max{n 2 , n }, and therefore U ⊆ B(Un , ). 2 Let y ∈ Un . Since U is a Cauchy sequence, there exists a subsequence U = (Uk0 , Uk1 , . . .) of U such that k0 = q and δ(Ukj , Un ) < 2j for all n  kj . Define the sequence z = (z0 , z1 , . . .) by choosing zk arbitrarily for k < q, zq = y, and zk ∈ Uk for kj < k < kj+1 such that d(zk , zkj ) < 2−j . The sequence z is a Cauchy sequence in S, so there exists z = limk→∞ zk and z ∈ U . Since d(y, z) = limk→∞ d(y, zk ) < , it follows that y ∈ B(U, ). Therefore, δ(U, Un ) < , which proves that limn→∞ Un = U . We conclude  that (K(S, Od ), δ) is complete. 5.12

The Topological Space (R, O)

Let x = (x0 , . . . , xn , . . .) be a sequence of real numbers. Consider the sequence of sets (Sn ), where Sn = {xn , xn+1 , . . .} for n ∈ N. It is clear that S0 ⊇ S1 ⊇ · · · ⊆ Sn ⊇ · · · . Therefore, we have the increasing sequence of numbers inf S0  inf S1  · · ·  inf Sn  · · · and the decreasing sequence of numbers sup S0  sup S1  · · ·  sup Sn  · · · . We define lim inf x as limn→∞ inf Sn and lim sup x as limn→∞ sup Sn . The number lim inf x is referred to as the limit inferior of the sequence x, while lim sup xn is referred to as the limit superior of x. Example 5.25. Let x be the sequence defined by xn = (−1)n for n ∈ N. It is clear that sup Sn = 1 and inf Sn = −1. Therefore, lim sup x = 1 and lim inf x = −1. Theorem 5.55. For every sequence x of real numbers, we have lim inf x  lim sup x. Proof. Let Sn = {xn , xn+1 , . . .}, yn = inf Sn , and zn = sup Sn for n ∈ N. If p  n, we have Sp ⊆ Sn , so yn  yp  zp  zn , so yn  zp for every n, p such that p  n. Therefore, lim sup x = limp→∞ zp  yn for every  n ∈ N, which in turn implies lim inf x = limn→∞ yn  lim sup x. Corollary 5.14. Let x = (x0 , x1 , . . . , xn , . . .) be a sequence of real numbers. We have lim inf xn = lim sup xn = if and only if limn→∞ xn = . Proof. Suppose that lim inf xn = lim sup xn = and that it is not the case that limn→∞ xn = . This means that there exists > 0 such that, for

May 2, 2018 11:28

304

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 304

Mathematical Analysis for Machine Learning and Data Mining

every m ∈ N, n  m implies |xn − |  , which is equivalent to xn  + or xn  − . Thus, at least one of the following cases occurs: (i) there are infinitely many n such that xn  + , which implies that lim sup xn  + , or (ii) there are infinitely many n such that xn  − , which implies that lim inf xn  − . Either case contradicts the hypothesis, so limn→∞ xn = l. Conversely, suppose that limn→∞ xn = . There exists n such that n  n implies − < xn < + . Thus, sup{xn | n  n }  + , so lim sup x  + . Similarly, y −  lim inf x and the inequality

−  lim inf x  lim sup x  + , which holds for every > 0, implies lim inf xn = lim sup xn = .



The notion of limit inferior and limit superior can be extended to nets of real numbers. Let (xi )i∈I is a bounded net of R and let Si = {xp | p  i}. If i  j, j  k implies i  k, so Sj ⊆ Si . Therefore, inf Si  inf Sj  sup Sj  sup Si . Let yi = inf Si and zi = sup Si for i ∈ I. Then, i  j implies yi  yj  zj  zi , so yj  zi for every i, j such that i  j. Define lim inf(xi )i∈I = limI (yi ) and lim sup(xi )i∈I = limI (zi ). We have lim sup(xi )i∈I = lim(zi ) ≥ yi I

for every i ∈ I, which, in turn implies lim sup(xi )i∈I  limI (yi ) = lim inf(xi )i∈I . In Example 4.21, we saw that every closed interval [a, b] of R is a compact set. This allows us to prove the next statement. Theorem 5.56. Let f : (a, b) −→ R be a function that is continuous at x0 ∈ (a, b) and f (x0 ) = 0. There exists an open sphere B(x0 , r) such that f (x) has the same sign as f (x0 ) for x ∈ B(x0 , r). Proof. Suppose that f (x0 ) > 0. Since f is continuous in x0 for every positive there exists a positive δ such that x ∈ B(x0 , δ) ∩ (a, b) implies f (x0 ) − < f (x) < f (x0 ) + . If δ corresponds to = f (x2 0 ) , then f (x0 ) 3f (x0 ) < f (x) < , 2 2 so f (x) has the same sign as f (x0 ) when x ∈ B(x0 , δ).  The argument is similar when f (x0 ) < 0.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Metric Space Topologies

9in x 6in

b3234-main

page 305

305

Theorem 5.57. (Bolzano’s Theorem) Let f : [a, b] −→ R be a function continuous on [a, b] and suppose that f (a) and f (b) have opposite signs. There exists c ∈ (a, b) such that f (c) = 0. Proof. Suppose that f (a) > 0 and f (b) < 0 and let T = {x ∈ [a, b] | f (x)  0}. Since a ∈ T it follows that T = ∅ and b is an upper bound of T . If c = sup T , then a < c < b. If f (c) = 0, by Theorem 5.56 there exists a δ > 0 such that c − δ < x < c + δ implies that f (x) has the same sign as f (c). If f (c) > 0, there exists x > c such that f (x) > 0 contradicting the definition of c. If f (c) < 0, then c − δ2 is an upper bound for T , again contradicting the definition of c. Therefore, f (c) = 0.  Definition 5.26. Let S be a subset of R. A function f : S −→ R has the Darboux property on S if for any a, b ∈ S and any y between f (a) and f (b) there exists c ∈ [a, b] such that f (c) = y. Theorem 5.58. (Intermediate Value Theorem) If f : [a, b] −→ R is continuous on [a, b], then f has the Darboux property on S. Proof. Suppose that f (a) < f (b) and let y be a number such that f (a) < y < f (b). Note that for the function g(x) = f (x) − y we have g(a) < 0 and g(b) > 0. By Bolzano’s theorem, there exists c ∈ [a, b] such that g(c) = 0, which is equivalent to f (c) = y.  The converse of the intermediate value theorem is false as the next example shows. Example 5.26. Let S = [0, 1) and let f : S −→ R be the function given by  sin x1 if x > 0, f (x) = 0 if x = 0. Since the limit of f in 0 does not exist the function is not continuous on S; however, it is easy to see that the function has the Darboux property. Let (S, O) be a topological space and let C(S, O) be the set of continuous functions of the form f : S −→ R. This set is a real linear space relative to the sum defined as (f + g)(x) = f (x) + g(x) and the scalar multiplication defined as (cf )(x) = cf (x) for x ∈ S. The set Cb (S, O) of continuous and bounded functions of the form f : S −→ R is a subspace of C(S, O).

May 2, 2018 11:28

306

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 306

Mathematical Analysis for Machine Learning and Data Mining

The linear space C(S, O) is not finitely dimensional. Indeed, the infinite set of function {pn | n ∈ N} in C[a, b], where pn (x) = xn is linearly independent. A norm ν can be defined on C(S, O) as ν(f ) = sup{|f (x)| | x ∈ S}. Since |f (x)|  ν(f ) and |g(x)|  ν(g) for x ∈ S, it follows that |(f +g)(x)|  |f (x)| + |g(x)|  ν(f ) + ν(g). Thus, ν(f + g)  ν(f ) + ν(g). We leave to the reader the verification of the remaining properties of norms. If (fn ) is a sequence of functions in Cb (S, O) equipped with the norm νsup such that limn→∞ fn = f , then the metric generated by this norm is denoted by dsup . Note that the function z : S −→ R defined as z(x) = 0 for x ∈ S belongs to Cb (S, O). If f ∈ Cb (S, O), then d(z, f ) = max{|f (x)| | x ∈ S} < ∞. The next statement shows that the Cb (S, O) equipped with the topology induced by the metric dsup is complete. Theorem 5.59. Let (fn ) be a sequence of functions in Cb (S, O) such that limn→∞ fn = f (in the sense of dsup ). Then f ∈ Cb (S, O). Proof. Let be a positive number and let n be such that dsup (fn , f ) < 3 . Since each of the functions fn is continuous, there exists U ∈ neighx (S, O) such that |fn (x) − fn (y)| < 3 for every y ∈ U . Thus, if n > n we have: |f (x) − f (y)|  |f (x) − fn (x)| + |fn (x) − fn (y)| +|fn (y) − f (y) < , which implies the continuity of f . Further, since d(z, f )  d(z, fn ) +  d(fn , f ), it follows that z is a bounded function, so f ∈ Cb (S, O). ˆ we can prove a variant of Cauchy’s For nets whose set of values is R criterion that is less general but is easier to verify. ˆ has a limit (finite or infinite). Theorem 5.60. Every isotonic net of R Proof. Let (xi )i∈I be an isotonic map and let X = {xi | i ∈ I}. Suppose that there exists M = sup X and let m ∈ R such that m < M . By the definition of sup X there exists xi0 such that m < xi0  M . Since (xi ) is isotonic, j > i0 implies xj  xi0 > m, which means that limI xi = M . If X has no upper bound, then for any a > 0 there exists xi0 > a. Since  i  i0 implies xi  xi0 , we have xi > a for i  i0 , so lim xi = ∞. Every norm on a linear space generates a metric and, therefore, a metric topology. Therefore, topologies of normed linear spaces enjoy the properties previously discussed for metric spaces.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Metric Space Topologies

5.13

b3234-main

page 307

307

Series and Schauder Bases

Definition 5.27. A sequence of numbers s = (sn ) is a numerical series in R, or just a series, if there exists a sequence (xn ) in R such that sn = x0 + x1 + · · · + xn for n ∈ N. The numbers xn are the terms of the series s, while the elements sn are the partial sums of s. If the sequence (sn ) is convergent, then we say that the series s is convergent, or that the set {xn | n ∈ N} is summable. If s = limn→∞ (x0 +· · ·+xn ) we often written s = x0 + · · · + xn + · · · . Thus x0 + · · · + xn + · · · is defined as limn→∞ (x0 + · · · + xn ). If limn→∞ sn = s, then s is the sum of the series s and we write s =  n∈N xn . n The number sn = i=0 xi is the nth partial sum of the series s. If s is not convergent, then we say that it is divergent. Note that the series x0 + x1 + · · · and xk + xk+1 + · · · are convergent or divergent in the same time.  A series sn = in xi is convergent if and only if for every > 0 there exists a number n such that for n  n and p ∈ N we have: sn+p − sn  = xn+1 + · · · + xn+p   . This is a mere restatement of the completeness property of R. Note that if a series s is convergent then we have limn→∞ xn = limn→∞ (sn − sn−1 ) = 0. Theorem 5.61. If (xn ) and (yn ) are two convergent series, then (xn + yn )    is a convergent series and n∈N (xn + yn ) = n∈N xn + n∈N yn . Also,   for any a ∈ R the series (axn ) converges and n∈N axn = a n∈N xn . Proof. The statements of the theorem follow immediately from Definition 5.27.  Theorem 5.62. Let s be a convergent series having sum s. We have limn→∞ xn  = 0. Proof. For > 0 there exists a number n such that for n  n and for p ∈ N we have |sn+p − sn | = |xn+1 + · · · + xn+1 |  . In particular  |sn+1 − sn | = |xn+1 | < , which implies limn→∞ |xn | = 0.

May 2, 2018 11:28

308

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 308

Mathematical Analysis for Machine Learning and Data Mining

Note that the converse of Theorem 5.62 is false. Indeed, consider the series 1+ 21 +· · ·+ n1 +· · · , where xn = n1 . Clearly, limn→∞ |xn | = limn→∞ n1 = 0. However this series (called the harmonic series) is divergent. Indeed, note that 1 1 1 + + ··· + s2n − sn = n+1 n+2 2n 1 1 1 1 + + ··· + > , > 2n 2n 2n 2 which contradicts the convergence of (sn ). Definition 5.28. A series s = x0 + · · · + xn + · · · is absolutely convergent (or the sequence (xn ) is absolutely summable) if the series of non-negative numbers |x0 | + · · · + |xn | + · · · is convergent. Theorem 5.63. An absolutely convergent series is convergent.  Proof. Let s = xn be an absolutely convergent series. In other word, the sequence of partial sums (zn ), where zn = |x0 | + · · · + |xn | for n ∈ N is convergent. We have |sn+p − sn |  |xn+1 | + · · · + |xn+p |  zn+p − zn . The desired conclusion follows immediately.   Definition 5.29. A series s = xn is semiconvergent if it is convergent  but the series |xn | is divergent. Theorem 5.64. (Leibniz’ Theorem) Let (xn ) be a sequence of positive numbers and let s be the series x0 − x1 + x2 − x3 + · · · + x2n − x2n+1 + · · · . If x0 > x1 > · · · > xn > xn+1 > · · · and limn→∞ xn = 0, then s is a convergent series. Proof. Consider separately the sequence of even ranked partial sums s0 , s2 , s4 , . . . and s1 , s3 , s5 , . . .. Note that the first sequence is decreasing, while the second is increasing. Moreover, since s2n = x0 − x1 + x2 − · · · − x2n−1 + x2n  s1 , s2n−1 = x0 − x1 + x2 − · · · x2n−2 − x2n−1  x0 , it follows that both sequences are convergent. Furthermore, from the equality s2n = s2n−1 + x2n , it follows that limn→∞ s2n = limn→∞ s2n−1 , so the sequence of all partial sums is convergent. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Metric Space Topologies

b3234-main

page 309

309

A series of the form x0 − x1 + x2 − · · · , where xn > 0 for n ∈ N is an alternate series. Example 5.27. By Leibniz’ Theorem the alternate series 1 1 1 + ··· 1 − + − · · · + (−1)n 2 3 n+1 is convergent. However, it is not absolutely convergent because the series 1 1 1 1 + + + ···+ + ··· 2 3 n+1 is the divergent harmonic series, which shows that the series considered is semiconvergent. Thus, the reciprocal of Theorem 5.63 is false.   yn be two series with positive terms. If Theorem 5.65. Let xn and  xn  yn for n ∈ N, then the convergence of y implies the convergence   n  xn is divergent, then yn is divergent. of xn ; further, if Proof. Let sn = x0 + · · · + xn and zn = y0 + · · · + yn be the partial sums of order n. Since xn  yn , it follows that sn  zn for n ∈ N. If (sn ) is divergent, then so is (zn ). Suppose that (zn ) is convergent and let z = limn→∞ zn . Since sn  zn < z it follows that the sequence (sn ) is monotonic and has an upper  bound. Thus, (sn ) is a convergent sequence.   Theorem 5.66. Let xn and yn be two series with positive terms. If yn+1 xn+1 < xn yn   for n ∈ N, then the convergence of yn implies the convergence of xn   and the divergence of xn implies the divergence of yn . Proof.

If k =

x0 y0

The inequalities that exist by hypothesis imply x0 x1 xn > > ··· > > ··· . y0 y1 yn we have x1 < ky1 , x2 < ky2 , . . . , xn < kyn , . . . .

The desired conclusion follows immediately from Theorem 5.65.    Theorem 5.67. Let xn and yn be two series with positive terms. If xn xn 0 < lim inf  lim sup < ∞, yn yn   then both series xn and yn are either convergent or divergent.

May 2, 2018 11:28

310

Proof.

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 310

Mathematical Analysis for Machine Learning and Data Mining

Let a, b be two positive numbers such that xn xn  lim sup < b. a < lim inf yn yn

There exists a number n0 such that if n  n0 then ayn < xn < byn , which implies the statement of the theorem.  Let σ : N −→ N be a bijection and let x0 + x1 + · · · + xn + · · · be a series that is absolutely convergent. The series obtained by permuting this series using σ is the series xσ(0) + xσ(1) + · · · + xσ(n) + · · · . Theorem 5.68. If s = x0 + x1 + · · · + xn + · · · is an absolutely convergent series and σ : N −→ N be a bijection, then the series xσ(0) + xσ(1) + · · · + xσ(n) + · · · is absolutely convergent and the sums of these series are equal. Proof. Let sn = x0 + x1 + · · · + xn and let zn = |x0 | + |x1 | + · · · + |xn |. Since x0 + x1 + · · · + xn + · · · is absolutely convergent, for every > 0 there exists n such that |zn +p − zn +q | < 2 for every p, q ∈ N. If q > p this means that |xn +p+1 | + · · · + |xn +q | < 2 . Since the series x0 + x1 + · · · is convergent we may assume that we also have |sn − s| < 2 if n  n . There exists n (which depends on n ) such that if m > n , zm contains all terms of n and terms that follow these terms in the original series xn +k0 , xn +k1 , . . .. This implies |zm − sn | < 2 . Since s − zm = (s − sn ) + (sn − zm ), it follows that |s − zm | < if m > n , which shows that the series xσ(0) + xσ(1) + · · · + xσ(n) + · · · is convergent and its sum is s.   Lemma 5.5. Let xn be a semiconvergent series, and let an and −bn the sums of the positive terms and the sum of the negative terms contained in  the first n terms of the series xn , respectively. We have limn→∞ an = limn→∞ bn = ∞. Proof. Let sn = x0 + x1 + · · · + xn and let zn = |x0 | + |x1 | + · · · + |xn |. We have sn = an − bn and zn = an + bn , so an = 12 (sn + zn ) and bn = 1 2 (zn − sn ). Since limn→∞ sn is finite and limn→∞ zn = ∞, it follows that  limn→∞ an = limn→∞ bn = ∞.  Theorem 5.69. (Riemann’s3 Theorem) Let xn be a semiconvergent series. For any number r ∈ R there exists a bijection σ of N such that the 3 Georg Riemann (Sept. 17th 1826–July 20th 1866) was a very influential German mathematician who made lasting contributions to analysis, number theory, and differential geometry. He served as a Professor and Chair of Mathematics at G¨ ottingen.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Metric Space Topologies

b3234-main

page 311

311

 series xσ(n) converges to r; Furthermore, there exists a bijection σ of N  such that the series xσ(n) is divergent.  Proof. We rearrange the terms of the series xn as follows. First, we consider the shortest sequence of positive terms in the order they appear  in xn such that their sum exceeds r; then, we will continue with the  shortest sequence of negative terms in the order they appear in xn such that the sum of the terms is less than r. Then, we continue with the shortest sequence of remaining positive term until the sum exceeds r, etc. This rearrangement is possible because, as we saw in Lemma 5.5, we have limn→∞ an = limn→∞ bn = ∞. We shall prove that the series obtained in this manner is convergent and its sum is r. Let r0 , r1 , . . . , rn , . . . be the partial sums of the new series and let tn be the number of terms (positive and negative) considered in the first n rearranging operations. If n is even, then rtn > r and rtn −xtn −1 < r, hence 0 < rtn −r < xtn −1 . (Note that the term of the original series is xtn −1 because the terms of the series are numbered beginning with 0.) If n is odd then rtn < r and rtn − xtn −1 > r, hence xtn −1 < rtn − r < 0. Thus, for any n ∈ N we have |r − rtn | < |xtn −1 |. Since pn  n, we have  limn→∞ pn = ∞. The convergence of xn implies limn→∞ xtn −1 = 0, hence limn→∞ rtn = r, which proves the first part of the theorem. For the second part, we rearrange the terms of the series by choosing the least number of positive terms whose sum exceeds 1, followed by the least number of negative terms that makes the sum less than 1, then the least number of positive terms that yields a sum greater than 2, etc. In the 2n − 1st rearranging we select the least amount of positive terms that gives a sum that exceeds n; in the 2nth rearranging we select the least number of negative terms that results in a partial sum less than n. Let pn be the number of terms selected in the first 2n − 1 operations and let qn be the number of terms selected in the first 2n operations. We have as above 0 < rpn − n < xpn and 0 < n − rqn < −xqn . Recall that limn→∞ xpn = limn→∞ xqn = 0. Let a > 0 and let m ∈ N be such that for n > m we have a + 1 < n, |xpn | < 1, |xqn | < 1. It is clear that rpn > n > a. The choice of n implies a < n − 1 < rqn , so for n > m we have both rpn > a and rqn > a, which implies limn→∞ rpn = limn→∞ rqn = ∞.

May 2, 2018 11:28

312

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 312

Mathematical Analysis for Machine Learning and Data Mining

To complete the proof, let c, d be two numbers such that c < d. This time we rearrange the terms of the series by choosing first the least number of positive terms such that their sum exceeds d; then select the least number of terms such that the sum is below c. Then, add the least number of positive terms such that the sum exceeds d, etc. If pn is the number of terms considered in the nth operation, then the sequence rp1 , rp3 , rp5 , . . . tends towards d, while the sequence rp2 , rp4 , rp6 , . . . tends towards c, so the  sequence r1 , r2 , . . . cannot be convergent. The notion of series in normed linear spaces is a generalization of the notion of numerical series. The notations introduced for numerical series are extended naturally. Definition 5.30. Let (S,  · ) be a normed linear space. A sequence s = (sn ) is a series in S if there exists a sequence (xn ) in this space such that sn = x0 + x1 + · · · + xn for n ∈ N. The elements xn are the terms of the series s, while the elements sn are the partial sums of s. If the sequence (sn ) is convergent, then we say that the series s is convergent and for the limit s of this sequence we can write s = limn→∞ (x0 + · · · + xn ). This equality is often written as s = x0 + · · · + xn + · · · , which amounts to defining x0 + · · · + xn + · · · as limn→∞ (x0 + · · · + xn ). If limn→∞ sn = s, then s is the sum of the series s and we write s =  n∈N xn . n The nth partial sum of the series s is sn = i=0 xi . If s is not convergent, then we say that it is divergent. Note that the series x0 + x1 + · · · and xk + xk+1 + · · · are convergent or divergent in the same time.  Let (S,  · ) be a complete normed space. A series sn = in xi is convergent if and only if for every > 0 there exists a number n such that for n  n and for p ∈ N we have sn+p − sn  = xn+1 + · · · + xn+1   . If a series s in a normed space (S,  · ) is convergent then we have: lim xn = lim (sn − sn−1 ) = 0.

n→∞

n→∞

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Metric Space Topologies

9in x 6in

b3234-main

page 313

313

Theorem 5.70. If (xn ) and (yn ) are two convergent series in a normed  space (S,  · ), then (xn + yn ) is a convergent series and n∈N (xn + yn ) =   x + n∈N yn . Also, for any scalar a the series (axn ) converges and  n∈N n n∈N axn = a n∈N xn . Proof. The statements of the theorem follow immediately from Definition 5.30.  Theorem 5.71. Let s = x0 + x1 + · · · be a convergent series having sum s in a normed space (S,  · ). We have limn→∞ xn  = 0. Proof. For > 0 there exists a number n such that for n  n and for p ∈ N we have sn+p − sn  = xn+1 + · · · + xn+1   . In particular  sn+1 − sn  = xn+1  < , which implies limn→∞ xn  = 0. Definition 5.31. A series s = x0 +· · ·+xn +· · · in a normed space (S, ·) is absolutely convergent if the series of non-negative numbers x0  + · · · + xn  + · · · is convergent. Alternatively, we say that the sequence (xn ) is absolutely summable. Theorem 5.72. A normed space (S,  · ) is complete if and only if every absolutely convergent series s = x0 + · · · + xn + · · · is convergent. Proof. Suppose that every absolutely convergent series in S converges. Let (xn ) be a Cauchy sequence. For each j  1 there exists nj such that m, n  nj implies xn − xm   2−j . Let z0 = n0 and zj = xnj − xnj−1 for j  1. Since zj   2−j the series (zj ) is absolutely convergent, and so it  is convergent. Note that for the partial sum uj = {zi | i < j} we have uj = xnj . Since (uj ) is convergent it follows that the subsequence (xnj ) is convergent, so the sequence (xn ) is convergent by Theorem 5.28. Thus, (S,  · ) is complete. Conversely, let (S,  · ) be complete and let s = x0 + · · · + xn + · · · be n an absolutely convergent series. Let tn = i=0 xi  for n ∈ N. If m < n we have   n n       xi   xi  = tn − tm . sn − sm  =    k=m+1

k=m+1

The sequence of real numbers (tn ) is a Cauchy sequence because of the  of partial sums (sn ) is convergence of ∞ i=0 xi . Therefore, the sequence a Cauchy sequence in (S,  · ), hence the series ∞ n=0 xn is convergent. 

May 2, 2018 11:28

314

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 314

Mathematical Analysis for Machine Learning and Data Mining

Definition 5.32. Let (S,  · ) be an F-normed linear space. A sequence (en ) of distinct elements of S is a Schauder basis if every x ∈ S can be  uniquely represented as x = n∈N ai ei , where ai ∈ F for i ∈ N. Theorem 5.73. A normed linear space that has a Schauder basis is separable. Proof. Let (en ) be a Schauder basis of a normed linear space (S,  · ). n The set C that consists of elements of S of the form x = i=1 ri ei , where ri ∈ Q for 1  i  n is countable. We claim that C is dense in S.  Let x be an arbitrary element of S that can be written as x = i1 ai ei . n For xn = i=1 ai ei we have limn→∞ xn = x. For every n  1 consider a sequence of rational numbers (rn1 , . . . , rnn )  1 such that |rni − ai | < n2 e for 1  i  n. Define yn = ni=1 rni ei ∈ C i and     n n n           ai xi  =  rni ei − ai xi  yn −     i=1 i=1  i=1  n n      =  (rni − ai )ei   |rni − ai |ei    i=1



n  i=1

i=1

1 1 ei   . 2 n ei  n

Since yn − x  yn − xn  + xn − x and limn−→∞ yn − xn  = limn−→∞ xn − x = 0, it follows that limn−→∞ yn − x = 0, which concludes the argument.  Example 5.28. Let ei = (0, 0, . . . , 0, 1, 0, . . .) be an infinite sequence of real numbers that consists of 0s with the exception of the ith component which equals 1. The normed space ( 1 (R),  · ) is separable because the sequence (en ) is a Schauder basis of this space. Let (fn )n1 be a sequence of real-valued functions defined on a metric space (S,  · ). Define the sequence of partial sums (sn ) as consisting of n functions sn : S −→ R given by sn (x) = j=1 fj (x) for x ∈ S. If the sequence (sn ) converges pointwise (uniformly) to s, we say that the series  f (x) converges pointwise (uniformly) to s. In either case, we write n≥1  n s = n1 fn . It is immediate that uniform convergence of a series of functions implies pointwise convergence.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Metric Space Topologies

9in x 6in

b3234-main

page 315

315

Theorem 5.74. (Weierstrass M -test) Let (fn )n1 be a series of realvalued functions defined on a normed space (S, ·). If there exists a conver gent series n1 Mn of non-negative real numbers such that |fn (x)|  Mn  for n  1 and x ∈ S, the series n1 fn converges uniformly.  Proof. Let be a positive number. Since the series n≥1 Mn converges, p there exists n such that m, p > n implies n=m Mn < . Therefore, for   every x ∈ S and m, p > n we have | pn=m fn (x)|  pn=m |fn (x)| < ,   so the series n1 fn converges uniformly. 5.14

Equicontinuity

Definition 5.33. Let (S, d) and (T, e) be two metric spaces. A collection of functions F = {fi | i ∈ I} is equicontinuous at x0 (where x0 ∈ S) if for every > 0 there exists δ > 0 such that for every fi ∈ F, d(x, x0 ) < δ implies e(fi (x), fi (x0 )) < . F is equicontinuous if it is equicontinuous at every x0 ∈ S. Note that if F is an equicontinuous family at x0 , the positive number δ is the same for every function f that is a member of F; thus, δ depends on x0 and on . Theorem 5.75. Let (S, d) and (T, e) be two metric spaces and let F = {fn : S −→ T | n ∈ N} be a countable equicontinuous collection of functions. If (T, e) is complete and (fn (z)) converges for all z ∈ D, where D is a dense subset of (S, d), then (fn (x)) converges for all x ∈ S. Proof. Since D is dense in S for every x ∈ S and every δ > 0 there exists z ∈ D such that d(z, x) < δ. The equicontinuity of F means that there exists δ such that for every fn , d(z, x) < δ implies e(fn (z), fn (x)) < 3 for z ∈ D and x ∈ S. Since (fn (z)) converges for all z ∈ D there exists n such that n, m  n implies e(fn (z), fm (z)) < 3 . Therefore, if n, m  n we have e(fn (x), fm (x))  e(fn (x), fn (z)) + e(fn (z), fm (z)) + e(fm (z), fm (x)) < 3 = . 3 This means that (fn (x)) is a Cauchy sequence in (T, e) and, since T is  complete, it converges. Thus, (fn (x)) converges for every x. Theorem 5.76. Let (S, d) and (T, e) be two metric spaces and let F = {fn : S −→ T | n ∈ N} be a countable equicontinuous collection of functions. If

May 2, 2018 11:28

316

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 316

Mathematical Analysis for Machine Learning and Data Mining

f : S −→ T is a function such that limn→∞ fn (x) = f (x) for x ∈ S, then f is a continuous function. Furthermore, F ∪ {f } is also an equicontinuous collection of functions. Proof. For > 0 and x, y ∈ S there exists δ > 0 such that d(x, y) < δ implies e(fn (x), fn (y)) < 3 for n ∈ N because F is equicontinuous. Since limn→∞ fn (x) = f (x) there exists n such that n  n implies e(f (x), fn (x)) < 3 and e(f (y), fn (y)) < 3 . Then, n  n and d(x, y) < δ imply e(f (x), f (y))  e(f (x), fn (x)) + e(fn (x), fn (y)) + e(fn (y), f (y)  , which shows that f is a continuous function. The equicontinuity of F ∪ {f } follows from the fact that d(x, y) < δ implies e(fn (x), fn (y)) < , which in turn, yields e(f (x), f (y)) < , due to the continuity of the metric e.  A stronger property is given next. Definition 5.34. Let (S, d) and (T, e) be two metric spaces. A collection of functions F = {fi | i ∈ I} is uniformly equicontinuous if for every > 0 there exists δ > 0 such that d(x, x ) < δ implies e(fi (x), fi (x )) < for every x, x ∈ S and i ∈ I. Example 5.29. Let M be a positive number and let FM be the family of functions FM = {f : R −→ R | |f (x) − f (x )|  M |x − x | for x, x ∈ R}, where R is equipped with the metric defined by d(x, x ) = |x − x | for x, x ∈ R. FM is uniformly equicontinuous. Indeed, if > 0 it suffices to take δ= M to obtain = |f (x) − f (x )|  M |x − x | < M · M for every f ∈ FM . Theorem 5.77. Let (S, Od ) and (T, Oe ) be two topological metric spaces and let F = {fi : S −→ T | i ∈ I} be an equicontinuous family of functions. If (S, Od ) is compact, then F is uniformly equicontinuous. Proof. Let > 0. By Theorem 5.45 each function fi is uniformly continuous on S. In other words, for every i ∈ I there exists δi > 0 such that d(x, x ) < δi implies e(fi (x), fi (x )) < 2 for every x, x ∈ S.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Metric Space Topologies

page 317

317

It is clear that the collection of spheres {B(x, δi ) | x ∈ S, i ∈ I} is a cover of S. Since S is compact, it is possible to extract a finite subcollection of spheres B(x1 , δi1 ), . . . , B(xn , δin ) that is itself a cover of S. Let δ = 12 min{δi1 , . . . , δin } > 0. Suppose that d(x, x ) < δ. There exists a sphere B(xj , δij ) such that x ∈ B(xj , δij ). Let x , x ∈ S be such that d(x , x ) < δ. Since x belongs to an open sphere B(xj , δij ), we have d(xj , x ) < which, in turn implies e(f (xj ), f (x )) < 2 . We have d(xj , x )  d(xj , x ) + d(x , x )
0 there exists δ > 0 such that d(x, y) < δ implies e(f (x), f (y)) < 3 and e(fn (x), fn (y)) < 3 for all n ∈ N. The compactness of (S, Od ) implies the existence of a finite subset {u1 , . . . , um } of S such that for every x ∈ S there is uj such that x ∈ S (uj , δ). Therefore, e(f (x), f (uj )) < 3 and e(fn (x), fn (uj )) < 3 . There exists n (dependent only on ) such that n  n implies e(f (uj ), fn (uj )) < 3 . Therefore, if n > n we have e(f (x), fn (x))  e(f (x), f (uj )) + e(f (uj ), fn (uj )) + e(fn (uj ), fn (x)) < , which shows that (fn ) converges uniformly to f .



Theorem 5.79. (Arzel` a-Ascoli Theorem) Let (S, Od ) be a compact topological metric space, and let F = {fi : S −→ C | i ∈ I} be a uniformly equicontinuous family of functions that is uniformly bounded, that is, there exists c > 0 such that |fn (x)|  c for n ∈ N and x ∈ S. Then, F is totally bounded in C(S) and every sequence in F contains a uniformly convergent subsequence.

May 2, 2018 11:28

318

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 318

Mathematical Analysis for Machine Learning and Data Mining

Proof. Since F = {fi | i ∈ I} is uniformly equicontinuous, for every > 0 there exists δ > 0 such that d(x, x ) < δ implies |fi (x) − fi (x )| < 3 for every x, x ∈ S and i ∈ I. The compactness of (S, Od ) means that there exists a finite set n {x1 , . . . , xn } such that S ⊆ i=1 B(xi , δ). We associate to each function  f (x1 )  f ∈ F the vector ℘(f ) =

. . . f (xn )

∈ Cn .

Note that ℘(f ) ∈ Dn , where D = {a ∈ C | |a| < c}. Since Dn is a finite union of sets of diameter less than there exist f1 , . . . , fm such that ℘(f ) ∈ B(℘(fk ), ) for some k, 1  k  m. If f ∈ F there exists k such that |f (xi ) − fk (xi )| < 3 for 1  k  m. Every x ∈ S lies in some sphere B(xi , δ) and therefore, |f (x) − f (xi )| < 3 and fk (x) − fk (xi )| < 3 . Thus, |f (x) − fk (x)| < for every x ∈ S. Since is arbitrary F is totally bounded. Since C(S) is complete, the closure of F is compact and thus, every sequence in F contains an uniformly convergent subsequence.  Exercises and Supplements (1) Prove that every metric defined on a set S is equivalent to a bounded metric. ˆ −→ [−1, 1] be the function defined by (2) Let f : R ⎧ x ⎪ ⎨ 1+|x| f (x) = 1 ⎪ ⎩ −1

if x ∈ R, if x = ∞, if x = −∞.

Prove that ˆ and [−1, 1]; (a) f is a bijection between R 2 ˆ (b) the mapping d : (R) −→ R≥0 defined by d(x, y) = |f (x) − f (y)| ˆ; ˆ is a bounded metric on R for x, y ∈ R (c) the restriction d of d to R is equivalent to the usual distance on R. (3) Let (xn ) be a sequence in R. If a = lim inf xn prove that there may be infinitely many terms xn that are less than a; also, for every  > 0 there exist only a finite number of terms xn that are less than a−. Formulate and prove a similar statement for b = lim sup xn . (4) Let (xn ) be a sequence in R and let L be the set of all numbers that are limits of some subsequences of (xn ). Prove that:

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Metric Space Topologies

page 319

319

(a) lim inf xn = inf L and lim sup xn = sup L; (b) the set L has both a least element and a greatest element, that is, sup L ∈ L and inf L ∈ L. (5) Let (xn ) be a sequence in R and let (xni ) be a subsequence of (xn ). Prove that lim inf xn  lim inf xni  lim sup xni  lim sup xn . (6) Let (un ) and (vn ) be two sequences in R. Prove that lim sup(xn + yn )  lim sup xn + lim sup yn , lim inf(xn + yn )  lim inf xn + lim inf yn ,

where {lim sup xn , lim sup yn } = {∞, −∞} for the first inequality and {lim inf xn , lim inf yn } = {∞, −∞} for the second inequality. (7) Let (xn ) be a sequence in R. If a = lim inf xn , b = lim sup xn , prove that there exists n0 ∈ N such that n  n0 implies a < xn < b. (8) Let (S, d) be a metric space. A d-point for a function h : S −→ R is a point x0 of S such that, for every other x ∈ S we have h(x0 ) − h(x) < d(x, x0 ). Prove that: (a) If (S, d) is complete then any lower semicontinuous function f : S −→ R which is bounded below has a d-point. (b) If (S, d) is not complete, there exists a uniformly continuous function f : S −→ R which is bounded below but has no d-point. (9) Let (xm ) be a sequence in Rn . Prove that limm→∞ xm = 0n if and only if for each j, 1  j  n we have limm→∞ xm j = 0. Solution: The statement follows immediately from the inequalities: m max{|xm j | | 1  j  n}  x  

n 

|xm j |.

j=1

(10) Let (S, d) be a metric space, x0 ∈ S, and let r be a positive number. Prove that there exists a continuous function f : S −→ R such that f (x) ∈ [0, 1], f (x) = 1 for every x ∈ B(x0 , r) such that {x ∈ S | f (x) = 0} ⊆ B(x0 , 2r). Solution: Let g : R −→ R be the continuous function defined by ⎧ ⎪ if 0  t  r, ⎨1   g(t) = 1 − 2 rt − 1 if r < t  3r 2 ⎪ ⎩ 0 if t > 3r . 2

May 2, 2018 11:28

320

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 320

Mathematical Analysis for Machine Learning and Data Mining

The desired function is f (x) = g(d(x, x0 )) for x ∈ S. (11) Let (S, Od ) be a metric space and let U, V be two subsets of S such that U is compact, V is open and U ⊆ V . Prove that: (a) there exists a continuous function f : S −→ [0, 1] such that f (x) = 0 for x ∈ U and f (x) = 1 if x ∈ V ; (b) there exists a continuous function g : S −→ [0, 1] such that g(x) = 1 for x ∈ U and {x ∈ S | g(x) = 0} ⊆ V . Solution: By Corollary 5.4 U is a closed set; furthermore, the closed sets U and S − V are disjoint. Since every topological metric space is normal (by Corollary 5.6), by Uryson’s Lemma (Theorem 4.83), there exists a continuous function f : S −→ [0, 1] such that f (x) = 0 for x ∈ U and f (x) = 1 for x ∈ S − V . For the second part of the supplement take g(x) = 1 − f (x) for x ∈ S. (12) Let (S, Od ) be a metric space, {U1 , . . . , Un } be a partition of S that consists of compact sets, and let  a1 , . . . , an ∈ R. Prove that the function f : S −→ R defined as f (x) = n j=1 aj 1Uj (x) is continuous and f ∞  max{|aj | | 1  j  n}. (13) Let (S, dS ) and (T, dT ) be two metric spaces. A function f : S −→ T is bounded f (S) ⊆ BdT [y0 , r] for some y0 ∈ T . Let The set of bounded functions between these metric spaces is denoted by B(S, T ); the set of bounded and continuous functions is denoted by C(X, Y ). For f, g ∈ B(S, T ) define d : B(S, T ) × B(S, T ) −→ R0 as d(f, g) = sup{dT (f (x), g(x)) | x ∈ S}. Prove that (a) (B(S, T ), d) is a metric space; (b) if (fn ) be a sequence of functions in B(S, T ) and f ∈ B(S, T ), the sequence (fn ) converges uniformly to f if and only if fn converges to f in the metric space (B(S, T ), d); (c) the set C(S, T ) is closed in B(S, T ); (d) if (T, dT ) is complete, then C(S, T ) is a complete subspace of B(S, T ). A continuity modulus is a function ω : R0 ∪ ∞ −→ R0 ∪ ∞ such that limx↓0 ω(x) = ω(0) = 0. Let (S, dS ) and (T, dT ) be two metric spaces. A function f : S −→ T admits ω as its continuity modulus at x0 ∈ S if dT (f (x0 ), f (x))  ω(dS (x0 , x)) for every x ∈ S. The function f admits ω as its continuity modulus if dT (f (x1 ), f (x2 ))  ω(dS (x1 , x2 )) for every x1 , x2 ∈ S. Note that f is a Lipschitz function having the Lipschitz constant c if and only if it admits the function ω(t) = ct as a continuity modulus. (14) Prove that if f : S −→ T is a function between two metric spaces (S, dS ) and (T, dT ) that admits a continuity modulus ω on S that is monotone increasing, then f is uniformly continuous on S.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Metric Space Topologies

page 321

321

(15) Let (S, dS ), (T, dT ) and (U, dU ) be three metric spaces and let f : S −→ T and g : T −→ U be functions that admit continuity moduli ω1 and ω2 , respectively. Prove that gf : S −→ U admits ω2 ω1 as a continuity modulus. (16) Let (S, Od ) and (T, Oe ) be two topological metric spaces, f : S −→ T be an uniformly continuous function, and let ω : R0 ∪ ∞ −→ R0 ∪ ∞ be defined as ω(t) = sup{e(f (x), f (y)) | d(x, y) = t} for t  0. Prove that ω is the minimal continuity modulus for f .   (17) Consider the polynomials Bn,k (x) = nk xk (1 − x)n−k for n  0, 0  k  n and x ∈ [0, 1]. Prove that n Bn,k (x) = 1; (a) k=0 n (b) kBn,k (x) = nx; k=0 n (c) k2 Bn,k (x) = n(n − 1)x2 + nx = n2 x2 − nx2 + nx; k=0 n (d) (k − nx)2 Bn,k (x) = nx(1 − x); k=0 nx(1−x) . (e) k {Bn,k (x) | |k − nx|  λ}  λ2

Solution: Consider the function h(t) = (xt + 1 − x)n for t ∈ R and observe that h(1) = 1. We have   n n   n k k tk Bn,k (x), h(t) = t x (1 − x)n−k = k k=0 k=0 hence h(1) = 1 =

n k=0

Bn,k (x).

Differentiating h with respect to t yields nx(xt + 1 − x)n−1 =

n 

ktk−1 Bn,k (x)

k=0

and the second equality follows by taking t = 1. One more differentiation allows us to write n(n − 1)x2 (xt + 1 − x)n−2 =

n 

k(k − 1)tk−2 Bn,k (x),

k=0

n 2 n(n − 1)x2 = − k)Bn,k (x), which implies k=0 (k 2 k B (x) = n(n − 1)x + nx. For the fourth equality we have: n,k k=0

hence  n

2

n 

(k − nx)2 Bn,k (x) =

k=0

n 

(k2 − 2nxk + n2 x2 )Bn,k (x)

k=0

= n2 x2 − nx2 + nx − 2n2 x2 + n2 x2 = nx − nx2 = nx(1 − x).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 322

Mathematical Analysis for Machine Learning and Data Mining

322

The last equality implies nx(1 − x) =

n 

(k − nx)2 Bn,k (x)

k=0





{(k − nx)2 | |k − nx|  λ}   λ2 {Bn,k (x) | |k − nx|  λ}. k

Next we prove directly an approximation result implied by Stone-Weierstrass Theorem. (18) Let f : [0, 1] −→ R be a continuous function. Prove that for every  > 0 there exists a polynomial p on [0, 1] such that supx∈[0,1] |f (x)−p(x)| < . Solution: Since f is continuous on [0, 1], it is uniformly continuous on this interval (by Heine’s Theorem, that is, Theorem 5.45), hence there is a δ > 0 such that |x − y| < δ implies |f (x) − f (y)| < . The function f is also bounded, so |f (x)| < M for number M and for x ∈ [0, 1].  ksome  Let p be defined as p(x) = n k=0 f n Bn,k (x) for x ∈ [0, 1]. We have |f (x) − p(x)|

n

 k

=

f (x) − f

Bn,k (x) n k=0



k

=

f (x) − f

Bn,k (x) + n |x−nk|

4M δ 2

to obtain |f (x) − p(x)| < .

(19) Let (S, O) be  a topological space. A subalgebra U of C(S) vanishes nowhere if {f −1 (0) | f ∈ U} = ∅. Prove that if U contains the constant function k1 defined by k1 (x) = 1 for x ∈ S, then U vanishes nowhere. Give an example of a subalgebra U of C(S) such that the reverse implication fails. (20) Let (S, O) be a topological space. Prove that if there exists an algebra of real-valued continuous function defined on S that separates points, then (S, O) is a Hausdorff space.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Metric Space Topologies

b3234-main

page 323

323

(21) Prove that a sequence x = (xn ) in a metric space (S, d) is a Cauchy sequence if and only if the series n1 d(xn , xn+1 ) is convergent. Solution: Suppose that x is a Cauchy sequence. Then, for each  > 0 there exists n such that m, n  n implies d(xn , xm ) < ; in 1 particular, there exists n0 such  that n  n0 implies d(xn , xn+1 ) < 2n . By Theorem 5.65, the series n1 d(xn , xn+1 ) is convergent.  Conversely, suppose that the series n1 d(xn , xn+1 ) is convergent. Then, every  > 0 there exists a number n such that for n  n and p ∈ N we have d(xn , xn+1 ) + d(xn+1 , xn+2 ) + · · · + d(xn+p−1 , xn+p ) < . This, in turn implies d(xn+p , xn ) < , so (xn ) is a Cauchy sequence. (22) Let (S, O) be a normal topological space, U be a closed subset of S and let f : U −→ R be a bounded continuous function. Prove that if g : S −→ R is the extension of f to a bounded continuous function defined on the entire space S (whose existence is established by Tietze’s Theorem (Theorem 5.35), then g∞  f ∞ . Solution: In the proof of Tietze’s Theorem we made an assumption that implies f ∞  1. If this is not the case, we can apply the same argument to the function f1 = f1∞ f and obtain the existence of a function g1 such that g1 ∞  1. Then, the function g = f ∞ g1 satisfies the condition. (23) Let (S, O) be a compact metric space and let L1 , L2 be two closed sets that are disjoint. Prove that for every a1 , a2 ∈ R and every  > 0, there exists a function g such that || g ||∞ < max{a1 + a2 } +  and || g − a1 1L1 − a2 1L2 ||∞ < . (24) Let (S, O) be a compact metric space, L1 , . . . , Ln be n disjoint closed  sets, and let L = n i=1 Li . If a1 , . . . , an ∈ R, then for every  > 0 there exists a continuous function g : L −→ R such that g∞  max{|ai | |  1  i  n} and || g − n i=1 ai 1Li ||∞ < . (25) Let f : Rn −→ R be defined as f (x) = a x+b. Prove that f is a Lipschitz function. (26) Let (S, d), (T, d ), (U, d ) be three metric spaces and let f : S −→ T and g : T −→ U be two Lipschitz functions. Prove that gf is a Lipschitz function. (27) Let (xn ), (yn ), (zn ) be three sequences of real numbers such that xn  yn  zn for n ∈ N. If limn→∞ xn = limn→∞ zn = , prove that limn→∞ yn = . (28) Let (un )n1 and (vn )1 be the sequences defined by un =



n

n+1 1 1 1+ and vn = 1 + n n

May 2, 2018 11:28

324

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 324

Mathematical Analysis for Machine Learning and Data Mining

for n  1. Prove that: (a) (un )n1 is an increasing sequence, (vn )n1 is a decreasing sequence; (b) 2  un  vn  3; (c) limn→∞ un = limn→∞ vn . Solution: Observe that un can be written as a sum of n terms:      

n n 1 1 n 1 n 1 + =1+ + ··· + un = 1 + n 2 n2 1 n n nn







1 1 1 2 n−1 1 1− +··· + 1− 1− ··· 1 − . = 2+ 2! n n! n n n

Similarly, un+1 can be written as a sum n + 1 terms: un+1 = 2 +

1 2!

1−

+··· +

1 n+1

1 (n + 1)!

1−

1 n+1

1−

2 n+1

··· 1 −

n , n+1

and each of the first n terms of un+1 is at least as large as the corresponding term of un . Therefore, un  un+1 . A similar argument can be used to show that vn  vn+1 . It is clear that un  2. Note that 1 1 + ··· + 2! n! 1 1  2 + + · · · + n+1  3, 2 2

un  2 +

1 ), We leave to the reader to prove that 2  vn  3. Since vn = un (1+ n+1 the second part follows immediately.

It is immediate that both sequences are convergent. Let 1 = limn→∞ un and 2 = limn→∞ vn . It is clear that 1  2 . Since vn − un = n1 un it follows that 1 = 2 . (29) Let (x0 , x1 , . . .) be a sequence in R such that limn→∞ = x. Prove that the sequence of averages (an ) defined by

an =

n−1 1 xj n j=0

is also convergent and limn→∞ an = x. (x −x)+(x −x)+···+(x

−x

0 1 n−1 Solution: Note that an − x = Since n limj→∞ xj = x, for every  > 0 there exists n ∈ N such that n > n

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Metric Space Topologies

page 325

325

implies |xn − x| < . Consequently,  n n−1   1  |xj − x| + |xj − x| |an − x|  n j=1 n +1 

n 1 < |xj − x| + . n j=1

Since limn→∞ diately.

1 n

n j=1

|xj − x| = 0, the desired conclusion follows imme-

aro4 sense to  if the seA sequence (xn ) of complex numbers converges in Ces` to . quence of arithmetic averages (an ) converges n n 1 aro For series sn = j=1 xj define σn = n j=1 sj . The series (sn ) is Ces` summable and its sum is s if limn→∞ σn = s. (30) Let (xn ) be the sequence of defined by xn = (−1)n Prove that although aro sense. (xn ) is not convergent in the usual sense, it is convergent in Ces`  j (31) Let z = cos α + i sin α, where α = 0. Prove that the series sn = n−1 j=0 z is not convergent but it is Ces` aro summable. n−1 j z n −1 1 z = = z−1 (cos nα + sin nα − 1) it is Solution: Since j=0 z−1 clear that the series (sn ) is not convergent. On other hand, lim σn =

n→∞

1 . z−1

(32) Let L be a normed linear space. Prove that L is homeomorphic to S(0L , 1).   Hint: Let f : S(0L , 1) −→ L be the function f (x) = tan π2 x for x ∈ S(0L , 1). Prove that f is the desired homeomorphism. (33) Let L be a real normed linear space. If U is a subset of L and let V be an open subset in L, prove that the Minkowski sum U + V is an open set in L. Solution: Note that for every u ∈ U, tu (V ) = u + V is open by Theorem 6.4, and we can write U + V = u∈U tu (V ). Therefore, U + V is open. (34) Let (S,  · ) be a normed linear space, x ∈ S, U ⊆ S and let d be the metric induced by the norm on S. Prove that d(ax, U ) = |a|d(x, U ). 4 Ernesto Ces` aro, was born in Naples on March 12th and died on September 12, 1906 in Torre Annunziata. Ces` aro graduated from the University of Rome and taught ar the Sapienza University. His main contributions are in differential geometry and is known for his averaging method.

May 2, 2018 11:28

326

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 326

Mathematical Analysis for Machine Learning and Data Mining

Solution: By Definition 5.12 we have d(ax, U ) = inf{d(ax, u) | u ∈ U } = inf{ax − u | u ∈ U } 



1



= |a| inf

x − u

| u ∈ U . a Since every v ∈ U can be written as d(ax, U ) = |a|d(x, U ).

1 u a

for some u ∈ U it follows that

(35) Prove that each closed sphere B[0m , r] in the normed linear space (Rm ,  · ∞ ) is compact. Solution: Let (xn )n∈N be a sequence included in B[0m , r], that is, xn ∞  r, which means that −r  (xn )i  r for 1  i  m. Since [−r, r] is a compact subset of R it follows that there exists a subset N1 of N such that lim((xn )n∈N1 )1 = a1 exists. Then, by the same compactness property, a subset N2 of N1 such that lim((xn )n∈N2 )2 = a2 exists, etc. At the mth step of the process we have constructed a sequence am exists. of sets N1 ⊇ N2 ⊇ · · · ⊆ Nm such that lim((xn )n∈Nm )m = ⎛ ⎞ Thus, lim((xn )n∈Nm )k = ak exists for 1  k  m. If a = ⎝

a1 . ⎠ . . am

we

have lim{xn | n ∈ Nm } = a, which shows that B[0m , r] is compact. (36) Let L be a real linear space and let T be a subspace of L such that K(T ) = L. Prove that there exists a non-zero linear functional f on L such that f (x) = 0 for every x ∈ T . (37) Let (S, Od ) be a metric space. functions

Prove that the family of Lipschitz

Fc = {f : S −→ R | |f (x) − f (y)|  cd(x, y) for all x, y ∈ S} is equicontinuous. (38) Let (fn ) be a sequence of functions, where fn : [a, b] −→ R. If (fn ) converges to a function f pointwise, and the family {fn | n ∈ N} is equicontinuous, prove that the convergence to f is uniform. Extend the previous statement by replacing the interval [a, b] by a compact set in a metric space (S, d).  i (39) Let Pn be the family of polynomials of the form p(x) = n i=0 ai x for x ∈ [a, b] such that |ai |  1 for 0  i  n. Prove that Pn is uniformly bounded and equicontinuous on [a, b].

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Metric Space Topologies

9in x 6in

b3234-main

page 327

327

Bibliographical Comments Pioneering work in applying topology in data mining has been done in [109, 92].

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 329

Chapter 6

Topological Linear Spaces

6.1

Introduction

In this chapter we present the interaction between the structure of a linear space and a topology defined on such a space assuming that the topological structure is compatible with the pre-existing linear structure. Linear spaces equipped with topologies are the main objects of functional analysis. We also discuss topological properties of convex sets. 6.2

Topologies of Linear Spaces

Definition 6.1. Let F be the real or complex field. A topological F-linear space is an F-linear space (L, +, ·) equipped with a topology O such that the addition in L and the scalar multiplication are continuous operations. A topological linear space is denoted by (L, O), or just by L if there is no ambiguity. The collection of neighborhoods of a point x of a topological linear space (L, O) is denoted by neighx (L, O). If U, V are subsets of a linear space, their Minkowski sum is the set U + V = {u + v | u ∈ U, v ∈ V }. Let L be an F-linear space. The addition in L is continuous in (x, y) if for every neighborhood W of x + y there exist neighborhoods U ∈ neighx (L, O) and V ∈ neighy (L, O) such that for u ∈ U and v ∈ V we have u + v ∈ W . Similarly, scalar multiplication is continuous in (a, x) if for every neighborhood W of ax there exists U ∈ neighx (L, O) and a positive number δ such that if |b − a| < δ and u ∈ U imply bu ∈ W . 329

May 2, 2018 11:28

330

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 330

Mathematical Analysis for Machine Learning and Data Mining

For the translation tz : L −→ L generated by z ∈ L defined as tz (x) = x + z we have the following theorem. Theorem 6.1. Let L be a topological F-linear space. Every translation tz of L is a homeomorphism of L. Proof. It is immediate that every translation is a bijective mapping on L. The continuity of tz and of its inverse, t−z follows from the continuity of addition in L.  Theorem 6.2. Let (L, O) a topological linear space. If T is an open subset and R ⊆ L, the set T + R is an open subset of L. Proof. Since tr is a homeomorphism of L for every r ∈ L, tr (T ) is an open  subset of L. Therefore, taking into account that T +R = {tr (T ) | r ∈ T }, it follows that T + R is open.  For a homothety ha : L −→ L of L defined as ha (x) = ax we have the next statement: Theorem 6.3. Let L be a topological F-linear space. If a = 0, then the homothety ha is a homeomorphism of L. Proof. The continuity of ha and of its inverse h a1 follows from the continuity of scalar multiplication.  Theorem 6.4. Let L be a topological F-linear space. If U is an open subset of L, then tz (L) and ha (L) are open sets for every translation tz and every homothety ha of L. Proof. This statement is an immediate consequence of Theorems 6.1 and 6.3.  Theorem 6.5. Let (L, O) be a topological linear space. The closure K(U ) of a set U is given by  K(U ) = {U + V | V ∈ neigh0L (L, O)}. Proof. Let x ∈ K(U ). Since x+V ∈ neighx (L, O), we have (x+V )∩U =  ∅, so x ∈ U − V . Therefore, x ∈ {U − V | V ∈ neigh0L (L, O)} =  {U + V | V ∈ neigh0 (L, O)}. Suppose now that x  K(U ). There exists V ∈ neigh0 (L, O) such that  (x + V ) ∩ U = ∅, so x ∈ U − V , hence x ∈ {U + V | V ∈ neigh0L (L, O)},   which proves that {U + V | V ∈ neigh0L (L, O)} ⊆ K(U ).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Topological Linear Spaces

b3234-main

page 331

331

Theorem 6.6. Let (L, O) be a topological linear space. For any subsets U and W of L we have K(U ) + K(W ) ⊆ K(U + W ). Proof. Let u ∈ K(U ) and w ∈ K(W ). Since addition is continuous, for every V ∈ neighu+v (L, O) there exist V  ∈ neighu (L, O) and V  ∈ neighw (L, O) such that V  + V  ⊆ V . By Theorem 4.24 there exist x ∈ V  ∩U and y ∈ V  ∩W , hence x+y ∈ U +W and x+y ∈ V  +V  . Thus, every neighborhood of u + w intersects U + W , which implies u + w ∈ K(U + W ). This yields the desired inclusion.  Theorem 6.7. Let L be an F-topological linear space. If U is a subspace of L, then K(U ) is also a subspace of L. Proof. Since U is a subspace we have U + U ⊆ U and aU ⊆ U for a ∈ F. By Theorem 6.6 we have K(U ) + K(U ) ⊆ K(U + U ) ⊆ K(U ). Since ha is a homeomorphism for a = 0, aK(U ) = ha (K(U )) = K(ha (U )) ⊆ K(U ), which allows us to conclude that K(U ) is indeed a subspace.  Theorem 6.8. Let (L, O) be a topological linear space. Every maximal subspace S in L is either closed or dense in L. Proof. Since K(S) is a subspace of L and S ⊆ K(S) ⊆ L, the maximality of S leaves two alternatives: either S = K(S), which means that S is closed, or K(S) = L, which means that S is dense in L.  Corollary 6.1. Let L be a topological linear space. Every hyperplane in L is either closed or dense in L. Proof. Since a hyperplane H is the translate of a maximal subspace of L, the statement follows from Theorem 6.8.  Corollary 6.2. Every affine subset of Rn is closed. Proof.

This statement follows from Theorem 3.11 and Corollary 6.2. 

Theorem 6.9. Let (L, O) be a topological linear space. If K is a compact subset of L and C is a closed subset of L and K ∩ C = ∅, then there exists V ∈ neigh0L (L, O) such that (K + V ) ∩ (C + V ) = ∅. Proof. If K = ∅ the conclusion is immediate because K + C = ∅. Thus, we need to consider only the case when K = ∅.

May 2, 2018 11:28

332

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 332

Mathematical Analysis for Machine Learning and Data Mining

Let x ∈ K. Since x ∈ C and C is closed, there exists a neighborhood Wx of x such that Wx ∩ C = ∅. Since O is a linear topology there is a neighborhood Zx of 0L such that Wx = x+ Zx ; furthermore, by Supplement 6, there exists a symmetric neighborhood Ux of 0L such that Ux + Ux + Ux ⊆ Zx , hence x + Ux + Ux + Ux ⊆ x + Zx = Wx . Since Wx is disjoint from C, x + Ux + Ux is disjoint from C + Ux . The collection of neighborhoods {x+Ux | x ∈ K} is a cover of K. Since K is compact, there exists a finite cover of K of the form {x1 +Ux1 , . . . , xn + Uxn }. n Note that the set V = i=1 Uxi is a neighborhood of 0L . We have n n   K+V ⊆ (xi + Uxi + V ) ⊆ (yxi (xi + Uxi + Uxi ). i=1

i=1

Since no set xi + Uxi + Uxi has a non-empty intersection with C + V it follows that (K + V ) ∩ (C + V ) = ∅.  Corollary 6.3. Let (L, O) be a topological linear space. If for every x ∈ L the set {x} is closed, then (L, O) is a Hausdorff space. Proof. Let x, y ∈ L be two distinct point. The singleton K = {x} is a compact set (see Example 4.18) and {y} is a closed set. Thus, by Theorem 6.9 there exists V ∈ neigh0L (L, O) such that (x+ V )∩(y + V ) = ∅. Since x+ V and y + V are disjoint neighborhoods of x and y, respectively, it follows that there exists two open sets V1 , V2 ∈ O such that x ∈ V1 ⊆ x + V  and y ∈ V2 ⊆ y + V , which means that (L, O) is a Hausdorff space. Corollary 6.4. Let (L, O) be a topological linear space. If K is a compact subset of L and C is a closed subset of L and K ∩ C = ∅, then there an open subset U ∈ O such that 0L ∈ U and (K + U ) ∩ (C + U ) = ∅. Proof. By Theorem 6.9 there exists V ∈ neigh0L (L, O) such that (K + U ) ∩ (C + U ) = ∅. Therefore, there exists an open set U such that 0L ∈ U ⊆ V , and this implies (K + U ) ∩ (C + U ) = ∅.  A stronger form of Corollary 6.4 is given as follows. Corollary 6.5. Let (L, O) be a topological linear space. If K is a compact subset of L and C is a closed subset of L and K ∩ C = ∅, then there an open subset U ∈ O such that 0L ∈ U and K(K + U ) ∩ (C + U ) = ∅. Proof. Note that C + U is an open set and that K + U ⊆ C + U . Since C + U is closed, it follows that K(K +U ) ⊆ C + U , or K(K +U )∩(C +U ) = ∅. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Topological Linear Spaces

9in x 6in

b3234-main

page 333

333

If we have K = {0L } it follows that for any closed subset C of L, it follows that there exists an open subset U of L such that K(U )∩(C+U ) = ∅. Since C + U is an open set, C + U is closed and, therefore K(U ) ⊆ C + U . Definition 6.2. A subset U of a topological F-linear space L is bounded if there exists a neighborhood W of 0L and a ∈ F such that U ⊆ aW . The next definition is a generalization of the notion of Cauchy sequence. Definition 6.3. Let (L, O) be a topological linear space. A Cauchy net is a net (xi )i∈I such that for every U ∈ neigh0L (L, O) there exists i ∈ I such that j, k > i implies xj − xk ∈ U . (L, O) is (i) complete if every Cauchy net is convergent; (ii) quasi-complete if every bounded Cauchy net is convergent; (iii) sequentially complete if every Cauchy sequence is convergent. A net (xi )i∈I is eventually bounded if there exists a neighborhood W ∈ neigh0L (L, O) and iW ∈ I such that i  iW implies xi ∈ W . Next we show that in a topological linear the set of neighborhoods of a point x is a translation of the set of neighborhoods of 0L . Theorem 6.10. Let (L, O) be a topological linear space. If V ∈ neigh0L (L, O), where O is the topology defined on L, then tx (V ) = x + V ∈ neighx (L, O). Conversely, for every W ∈ neighx (L, O) we have W = tx (V ) = x + V , where V ∈ neigh0L (L, O). Proof. Let V ∈ neigh0L (L, O). There exists an open subset T of L such that 0L ∈ T ⊆ V . Since x + T is an open set and x ∈ x + T ⊆ x + V it follows that x + V is a neighborhood of x. If W is a neighborhood of x, following a similar argument, it follows that −x + W ∈ neigh0L (L, O) and, since W = tx (−x + W ), the second part of the theorem is proven.  Thus a topology of a linear space L is defined by the collection of neighborhoods of 0L . Theorem 6.11. A linear topological space is Hausdorff if and only if for every x ∈ L − {0L }, there exists U ∈ neigh0L (L, O) such that x ∈ U . Proof. Suppose that (L, O) is Hausdorff. Then, there exist U ∈ neigh0L (L, O) and V ∈ neighx (L, O) such that U ∩ V = ∅. In particular, x ∈ U .

May 2, 2018 11:28

334

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 334

Mathematical Analysis for Machine Learning and Data Mining

Conversely, suppose that the condition holds and let x, y ∈ L with x = y, that is, x− y = 0L . There exists U ∈ neigh0L (L, O) such that x− y ∈ U . By the continuity of subtraction there exists V ∈ neigh0L such that V −V ⊆ U . Suppose that (V + x) ∩ (V + y) = ∅. There exists z ∈ (V + x) ∩ (V + y), that is, z = u + x = v + y for some u, v ∈ V . Then, x − y = v − u ∈ V − V ⊆ U , so x − y ∈ U , which is a contradiction. Hence, (V + x) ∩ (V + y) = ∅, hence (L, O) is a Hausdorff space.  Theorem 6.12. Let (L, O) be a topological linear space. (L.O) is a Haus dorff space if and only if neigh0L = {0L }. Proof. Let x, y ∈ L be such that x − y = 0L and let W ∈ neigh0L be such that x − y ∈ W . By the continuity of subtraction, there exists V ∈ neigh0L such that V − V ⊆ W . Then x + V is disjoint from y + V . Indeed, if we would have z ∈ (x + V ) ∩ (y + V ) we could write z = x + x1 = y + y1 , where x1 , y1 ∈ V . Then, x − y = y1 − x1 ∈ V − V ⊆ W , which is a contradiction. Thus, L is a Hausdorff space. Conversely, let (L, O) be a Hausdorff topological linear space. Clearly,  neigh0L (L, O). Suppose that x = 0L also belongs to 0L ∈  neigh0L (L, O). By Theorem 6.11, there exists U ∈ neigh0L (L, O) such that x ∈ U , which is a contradiction.  Theorem 6.13. Let L be an real linear space. If S is a balanced subset of L, then so is K(S); furthermore, if 0L ∈ I(S), then I(S) is balanced. Proof. Since ha is a homeomorphism for a = 0, ha (K(S)) = K(ha (S)). Taking into account that S is balanced we have for a ∈ F and |a|  1, ha (K(S)) = K(ha (S)) ⊆ K(S), so K(S) is balanced. Again, since ha is a homeomorphism for a = 0, we have ha (I(S)) = I(ha (S)) ⊆ I(S). Note that for a = 0, aS = {0}, so we must require that 0 ∈ I(S) in order for I(S) to be balanced.  Theorem 6.14. Let L be an real linear space. If S is an absorbing subset of L, then so is K(S). Proof. Since S ⊆ K(S), it is immediate that K(S) is absorbing because every set that includes an absorbing set is itself absorbing.  Theorem 6.15. In a topological real linear space (L, O), there exists a local basis L0L of neighborhoods of 0L that satisfies the following conditions: (i) each V ∈ L0L is balanced and absorbing; (ii) for each V ∈ L0L , there exists W ∈ L0L such that W + W ⊆ V; (iii) each V ∈ L0L is closed.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topological Linear Spaces

page 335

335

Proof. For every neighborhood V ∈ neigh0L (L, O) there are a number δ > 0 and W ∈ neigh0L (L, O) such that aW ⊆ W for all a such that |a| < δ  because multiplication by scalars is continuous. Then U = {aW | |a| < δ} is a balanced neighborhood. Since vector addition is continuous in (0L , 0L ) there exist W ∈ neigh0L (L, O) such that x, y ∈ W implies x+ y ∈ V , that is, W + W ⊆ V . If x ∈ K(W ), x − W is a neighborhood of x, hence (x − W ) ∩ W = ∅, implies x ∈ W + W ⊆ V , so K(W ) ⊆ V . Since K(W ) is the closure of an balanced and absorbing set remains balanced and absorbing (by Theorems 6.13 and 6.14), this shows that there  exists a local base at 0L that satisfies the above conditions. Theorem 6.16. Let L be a real linear space and let L be a collection of subsets of L such that ∅ ∈ L, for A, B ∈ L there exists C ∈ L such that C ⊂ A ∩ B, and L satisfies the first two conditions mentioned in Theorem 6.15, that is, (i) each V ∈ L is balanced and absorbing; (ii) for each V ∈ L, there exists W ∈ L such that W + W ⊆ V . Then, there exists a linear space topology O on L such that L = neigh0L (L, O). Proof.

Define O as the collection of all subsets U such that U = {x ∈ U | x + V ⊆ U for some V ∈ L}.

It is immediate that ∅ and L belong to O and that if {Ui | i ∈ I} is a  collection of sets with the above property, then i∈I Ui enjoys the same property. Suppose that U1 , U2 are two sets in O and x ∈ U1 ∩ U2 . Since x ∈ Ui there exists Vi ∈ L such that x + Vi ⊆ Ui for i = 1, 2. By hypothesis, there exists V ∈ L such that V ⊆ V1 ∩ V2 . Since x + V1 ∩ V2 ⊆ U1 ∩ U2 , it follows that U1 ∩ U2 ∈ O, hence O is indeed a topology. For the interior I(T ) of a subset T of L relative to the topology O we have I(T ) = {x ∈ T | x + V ⊆ T for some V ∈ L}.

(6.1)

Indeed, if x ∈ I(T ), since I(T ) is open, there exists V ∈ L such that x+V ⊆ I(T ) ⊆ T , so we have I(T ) ⊆ {x ∈ T | x+V ⊆ T for some V ∈ L}. Next, we show that {x ∈ T | x + V ⊆ T for some V ∈ L} ∈ O.

May 2, 2018 11:28

336

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 336

Mathematical Analysis for Machine Learning and Data Mining

Let y ∈ T such that y + V ⊆ T for some V ∈ L. By hypothesis, there exists W ∈ L such that W + W ⊆ V . If w ∈ W , then y + w + W ⊆ y + W + W ⊆ y + V ⊆ T, hence y + w ∈ I(T ), hence the set {x ∈ T | x + V ⊆ T for some V ∈ L} is open. This implies equality (6.1). Therefore, the collection {x + V | V ∈ V ∈ L} is a local base at x. It remains to prove the continuity of vector addition and multiplication by scalars. Let x0 , y0 ∈ L and let V ∈ L. By hypothesis, there exists U ∈ L such that U +U ⊆ W . If x ∈ x0 +U and y ∈ y0 +U , then x+y ∈ x0 +y0 +U +U ⊆ x0 + y0 + V , which implies that vector addition is continuous. To prove the continuity of scalar multiplication let a0 ∈ R, x0 ∈ L, and V ∈ L. Let W ∈ L be such that W + W ⊆ V . Since W is absorbing, there exists δ > 0 such that |a|  δ implies ax ∈ W . Let n ∈ N be such that |a0 | + δ < n. If |a − a0 | < δ, then ||a| − |a0 ||  |a − a0 | < δ, which implies  a  |a | + δ   0 < 1.   n n Since W is balanced, for each a ∈ R and |a − a0 | < δ and x ∈ x0 + n1 W we have: ax = a0 x0 + (a − a0 )x0 + a(x − x0 ) a ∈ a0 x0 + W + W n ⊆ a0 x0 + W + W ⊆ a0 x0 + V, which shows that scalar multiplication is continuous.



Theorem 6.17. A Hausdorff topological F-linear space (L, O) is locally compact if and only if it is finite dimensional. Proof. If L is finite dimensional, then L is homeomorphic to Fn and therefore, it is locally compact. Let L be a Hausdorff topological F-linear space that is locally compact. Let V ∈ neigh0L (L, O) be a compact neighborhood of 0L . Note that V ⊆  {x + 0.5V | x ∈ V }, hence there exists a finite subset {x1 , . . . , xn } n of V such that V ⊆ j=1 (xj + 0.5V ) ⊆ {x1 , . . . , xn } + 0.5V . Let Y =  x1 , . . . , xn . It is clear that V ⊆ Y + 0.5V , hence 0.5V ⊆ 0.5(Y + 0.5V ) = Y + 0.52 V. Thus, V ⊆ Y +Y +0.52V = Y +0.52V , etc. This shows that V ⊆ Y +0.5nV for every n.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topological Linear Spaces

page 337

337

For x ∈ V there exists yn ∈ Y and vn ∈ 0.5n V such that x = yn + 0.5 vn . Since V is compact, there exists a subnet (vnj ) of (vn ) such that limj∈J vnj = v. Therefore, for the net (ynj ) we have n

lim ynj = lim(x − 0.5nj vnj ) = x.

j∈J

j∈J

Since Y is a closed subspace, we have x ∈ Y , that is V ⊆ Y . Since V is an absorbing set, it follows that L = Y , hence L is finite dimensional.  6.3

Topologies on Inner Product Spaces

Inner product spaces are naturally equipped with norms induced by inner products, and normed spaces, in turn, are special metric spaces, where the metrics are induced by norms. Thus, properties of topological metric spaces and normed linear spaces transfer to inner product spaces. Inner product in topological inner product spaces is continuous as we show next. Theorem 6.18. Let (xn ) and (yn ) be two sequences in a inner product space (L, (·, ·)). If lim xn = x and lim yn = y, then lim(xn , yn ) = (x, y). Proof. Since xn → x the set {xn | n ∈ N} is bounded and we have xn   m, and |(xn , yn ) − (x, y)| = |(xn , yn ) − (xn , y) + (xn , y) − (x, y)|  |(xn , yn ) − (xn , y)| + |(xn , y) − (x, y)| = |(xn , yn − y)| + |(xn − x, y)|  xn yn − y + xn − xy (by Cauchy-Schwarz Inequality)  myn − y + xn − xy, which implies that lim(xn , yn ) = (x, y).



Corollary 6.6. The norm induced by the inner product of a inner product space is continuous. Proof. This " statement is an immediate consequence of Theorem 6.18 because x = (x, x) for every x ∈ L.  Properties of orthogonal complements in inner product spaces can be sharpened when the inner product space is equipped with a topology. For example, the next statement is a strengthening of Theorem 2.41.

May 2, 2018 11:28

338

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 338

Mathematical Analysis for Machine Learning and Data Mining

Theorem 6.19. If T is a subset of a inner product space (L, (·, ·)), then T ⊥ is a closed subspace of L. Proof. We have already seen in Theorem 2.41 that T ⊥ is a subspace of L. Let (xn ) be a convergent sequence in T such that limn→∞ xn = x. By the continuity of the inner product, for t ∈ T we have (t, x) = limn→∞ (t, xn ) =  0, so x ∈ T ⊥ . Therefore, T ⊥ is a closed subspace of L. 6.4

Locally Convex Linear Spaces

Definition 6.4. A locally convex linear space is a topological linear space L such that every x ∈ L has a local basis of convex neighborhoods. Let ν be a seminorm on the linear space L. We saw that the collection of open spheres of the form Bν (x0 , r) is a local basis in x0 for the topology defined by ν. Theorem 6.20. Let L be a linear space and let N be a collection of seminorms on L. The collection T of all finite intersections of the form m Bi1 ,...,im (x, r) = k=1 Bνik (x, r) is a local basis at 0L for a linear space topology ON . The topology ON consists of those subsets U of L such that for every x ∈ U there exists T ∈ T such that x + T ⊆ U . If I = {i1 , . . . , ik }, we denote the set Bi1 ,...,im (x, r) = BI (x, r). With this notation we have

m k=1

Bνik (x, r) by

BI (x, r) ∩ BJ (x, r) = BI∪J (x, r). Proof. It is immediate that ∅ and L are open sets. Also, if U ⊆ ON , then  U is an open set. Let U1 , U2 be two open sets and let x ∈ U1 ∩ U2 . There exists a finite intersection of the form BJ (x, r) included in U1 and a finite intersection of the form BK (x, s) included in U2 . Therefore, x ∈ BJ∪K (x, r) ⊆ U1 ∩ U2 , so U is closed with respect to finite intersections and is, therefore, a topology. Next, we show that the addition and the scalar multiplication are continuous in this topology.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topological Linear Spaces

page 339

339

To prove that addition of elements of L is continuous it suffices to show that given U ∈ T and x, y ∈ L there are U1 , U2 ∈ T such that (x + U1 ) + (y + U2 ) ⊆ (x + y) + U . For each seminorm νi we have Bνi (0L , r1 ) + Bνi (0L , r2 ) ⊆ Bνi (0L , r1 + r2 ), by the triangular inequality for νi . Therefore, (x + Bνi (0L , r1 )) + (y + Bνi (0L , r2 )) ⊆ x + y + Bνi (0L , r1 + r2 ). Thus, for W = Bνi1 (0L , r1 ) ∩ · · · ∩ Bνik (0L , rk ) we can take U1 = U2 = Bνi1 (0L , r1 /2) ∩ · · · ∩ Bνik (0L , rk /2). Thus, addition in L is continuous. To prove the continuity of multiplication by scalars, we need to show that given a ∈ R, x ∈ L and U ∈ T there is δ > 0 and V ∈ T such that (a + B(0, δ))(x + U ) ⊆ ax + U . Suppose initially that U = Bνi (0L , r). If |a − a | < δ and x − x ∈ B(0L , r) we have νi (ax − a x ) = νi ((a − a )x + a (x − x ))  νi ((a − a )x) + νi (a (x − x )) = |a − a |νi (x) + |a |νi (x − x )  |a − a |νi (x) + (|a| + δ)νi (x − x )  δ(νi (x) + |a| + δ). Thus, if δ(νi (x)+ |a|+ δ) < r, the desired continuity of scalar multiplication follows. These observations can be readily extended to finite intersections as needed.  Note that the local basis at x ∈ L defined above consists of convex sets. For this reason, topological spaces defined in this manner are said to be locally convex. Definition 6.5. Let N = {νi | i ∈ I} be a collection of seminorms defined on a linear space L The collection N is separating if for every x ∈ L − {0L } there exists νi ∈ N such that νi (x) = 0. Theorem 6.21. Let L be a linear space and let N be a collection of seminorms on L. The locally convex topological space (L, ON ) is a Hausdorff space if and only if N is a separate collection of seminorms. Proof. Suppose that (L, ON ) is a Hausdorff space. Suppose that for x = 0L we would have ν(x) = 0 for every ν ∈ N. This would imply that x ∈ BI (0L , r), so x belongs to all neighborhoods of 0L , which contradicts the separation property of Hausdorff spaces.

May 2, 2018 11:28

340

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 340

Mathematical Analysis for Machine Learning and Data Mining

If x and y are two distinct elements of L, we have x − y = 0L , so there exists a seminorm ν ∈ N such that ν(x0 − y0 ) = d > 0. This is why Bν (x, d/3) ∩ Bν (y, d/3) = ∅. Indeed, if we would have z ∈ Bν (x, d/3) ∩ Bν (y, d/3) this would imply d = ν(x − y)  ν(x − z) + ν(z − y) < 2d/3, which is a contradiction.  Let (L,  · ) be a normed linear topological space and let V be a neighborhood of 0L . There exists an open set included in V and therefore, there is an open sphere B(0L , r) included in V . Thus, every neighborhood of 0L contains an open, convex, and balanced neighborhood C of 0L . By Theorem 3.32, the Minkowski functional mC is a seminorm. Let L be a locally convex topological space. The topology of L is defined by the collection of seminorms defined by a local basis at 0L that consists of convex and balanced open sets that contain 0L .

6.5

Continuous Linear Operators

The set of linear operators between two F-linear spaces L, K is denoted by L(L, K); if L and K are equipped with topologies then the set of continuous linear operators between these spaces is denoted by Lc (L, K). If L = K we write L(L) and Lc (L) for these sets, respectively. Then L(L) is an algebra relative to operator sum, product of a scalar with an operator, and operator composition. Theorem 6.22. Let (L, O) and (K, O ) be two topological linear spaces having the zero elements 0L and 0K , respectively. A linear operator h : L −→ K is continuous in x ∈ L if and only if it is continuous in 0L . Proof. Let h : L −→ K be a linear operator that is continuous in x0 and let V ∈ neigh0K (O ). Then h(x0 ) + V ∈ neighh(x0 ) (O ). By the continuity of h in x0 there exists U ∈ neighx0 (L, O) such that x ∈ U implies h(x) ∈ h(x0 ) + V . Since −x0 + U ∈ neigh0L (L, O) and x ∈ −x0 + U , we have x + x0 ∈ U . Therefore, by the linearity of h, we have h(x + x0 ) = h(x) + h(x0 ) ∈ h(x0 ) + V , that is, h(x) ∈ V . This yields the continuity of h in 0L . Conversely, suppose that h is continuous in 0L . Let x ∈ L and let V ∈ neighh(x0 ) (O ). Note that −h(x0 ) + V is a neighborhood of 0K . The continuity of h in 0L implies that there is U ∈ neigh0L (L, O) such that x ∈ U implies h(x) ∈ −h(x0 ) + V .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Topological Linear Spaces

b3234-main

page 341

341

Since x0 + U is a neighborhood of x0 and x ∈ x0 + U (equivalent to x − x0 ∈ U ), we have h(x − x0 ) ∈ −h(x0 ) + V , which is equivalent to  h(x) ∈ V by the linearity of h. Thus h is continuous in x0 . Corollary 6.7. Let (L, O) and (K, O ) be two topological linear spaces. A linear operator h : L −→ K is either continuous everywhere in L or is not continuous in any x ∈ L. Proof.

This is a direct consequence of Theorem 6.22.



Theorem 6.23. The set Lc (L) of continuous linear operators defined on a topological linear space L is an algebra relative to operator sum, product of a scalar with an operator, and operator composition. Proof. This follows immediately from the corresponding results for linear operators and for continuous functions.  For a subspace U of a linear space L the mapping hU : L −→ L/U is a surjective linear operator (see, for example [121]). In the presence of a topology on L we can prove a stronger result. Recall that the push-forward topology was introduced in Supplement 8 of Chapter 4. Theorem 6.24. Let U be a subspace of the topological linear space (L, O) and let hU : L −→ L/H. If L/U is equipped with the push-forward topology induced by hU , then hU : L −→ L/U is an open mapping. Proof. If X is an open subset in L and U is a subset of L, then the set  X + U = u∈U tu (X) is an open set as a union of open sets. The set hU (X)  is open in L/H because h−1 U (hU (X)) = X + U .

6.6

Linear Operators on Normed Linear Spaces

Next we discuss linear operators between topological linear spaces whose topologies are induced by norms. Theorem 6.25. Let (L,  · ) and (K,  · ) be two normed linear spaces. A linear operator h : L −→ K is continuous if and only it is bounded, that is, there exists a number c > 0 such that h(x)  cx for all x ∈ L. Proof. Suppose that h is continuous, and therefore, continuous at 0. There exists δ > 0 such that x  δ implies h(x)  1. If x  1, then δx  δ, so h(δx)  1, which implies h(x)  1δ , which shows that h is bounded.

May 2, 2018 11:28

342

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 342

Mathematical Analysis for Machine Learning and Data Mining

Conversely, suppose that h is bounded, that is, x  1impliesh(x)         c. Then, if x  c , we have  c x  1, which yields h c x   c, so h(x)  . Thus, h is continuous at 0, which implies its continuity.  Corollary 6.8. A functional f : L −→ R is continuous if and only if it is bounded. Proof.

This is an immediate consequence of Theorem 6.25.



The set of bounded linear operators between the normed spaces (L,  ·) and (K,  · ) will be denoted by B(L, K). Definition 6.6. Let (L,  · ) and (K,  · ) be two normed linear spaces and let h : L −→ K be a linear and continuous operator between these spaces. The norm of h is the number h given by h = sup{h(x) | x ∈ B[0L , 1]}. The existence of the number h follows from the fact that for x = 1 we have h(x)  c, where c was introduced in Definition 6.6. The homogeneity of linear operators implies that the norm of a linear and continuous operator h is also given by *  h(x)  (6.2) h = sup x = 0L . x Note that h is the least number c such that h(x)  cx. The number h is indeed a norm on Lc (L, K), the linear space of continuous linear operators between L and K. If h = 0, the inequality h(x)  hx implies h(x) = 0 for every x ∈ L, hence h is the zero operator. If h1 , h2 ∈ Lc (L, K) we can write (h1 + h2 )(x) = h1 (x) + h2 (x)  h1 (x) + h2 (x)  h1 x + h2 x = (h1  + h2 )x, hence h1 + h2 ∈ Lc (L, K) and h1 + h2   h1  + h2 . Furthermore, if h ∈ Lc (L, K), then (ah)(x) = ah(x) = |a|h(x)  (|a|h)x, which means that ah ∈ Lc (L, K) and ah  |a|h. We conclude that  ·  is indeed a norm on Lc (L, K).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topological Linear Spaces

page 343

343

Definition 6.7. A sequence of linear operators (hn ) in Lc (L, K) converges uniformly to an operator h ∈ Lc (L, K) if limn→∞ hn − h = 0. If limn→∞ hn (x) = h(x) for every x ∈ L, we say that (hn ) converges to h pointwise. Since hn (x) − h(x) = (hn − h)(x)  hn − h x, it is clear that uniform convergence implies pointwise convergence. Theorem 6.26. If K is a complete linear space, then Lc (L, K) is complete. Proof. Suppose that (hn ) is a Cauchy sequence, that is, for every > 0 there exists n such that m, n  n implies hn − hm  < . Since hn (x) − hm (x) = (hn − hm )(x)  hn − hm  x, then (hn (x)) is a Cauchy sequence, hence there exists limn→∞ hn (x). If we denote this limit by h(x), this defines a linear operator h. To prove that is bounded, note that hn − hn  < for m, n  n implies hn (x) − hm (x)  x for m, n  n . For m → ∞ we obtain hn (x) − h(x)  x for n  n , hence hn − h ∈ Lc (L, K). Since h = hn − (hn − h), it follows that  h ∈ Lc (L, K) and limn→∞ hn = h. Theorem 6.27. Let h, g : L −→ L be two linear operators on a normed linear space L. We have gh  gh. Proof.

For x ∈ L we have gh(x) = g(f (x))  gf (x)  gf x.

Since gh is the least number c such that gh(x)  cx, it follows that gh(x)  gh.  Corollary 6.9. Let h : L −→ L be a linear operators on a normed linear space L. We have hn   (h)n for n ∈ N. Proof.

This follows immediately from Theorem 6.27.



Observe that if h : L −→ K is a linear bounded operator, and, therefore, a continuous operator, the unit sphere B[0L , 1] is mapped by h to a bounded set. Operators that map the unit sphere to a set whose closure is compact posses a property stronger than continuity, because this property implies their continuity. This is formalized in the next definition.

May 2, 2018 11:28

344

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 344

Mathematical Analysis for Machine Learning and Data Mining

Definition 6.8. Let (L,  · ) and (K,  · ) be two normed linear spaces. A linear operator h : L −→ K is compact if it maps B[0L , 1] into a set whose closure is compact. Since f, g ∈ Lc (L, K) imply f + g ∈ Lc (L, K) and af ∈ Lc (L, K) it follows that Lc (L, K) is a linear subspace of the linear space of linear operators between L and K. Theorem 6.28. Let (hn ) be a sequence of operators in Lc (L, K), where (L,  · ) and (K,  · ) are two normed linear spaces. The sequence (hn ) converges in norm to h ∈ Lc (L, K) if and only if for any bounded subset X of L the sequence (hn (x)) converges uniformly to h(x). Proof. Suppose that (hn ) converges in norm to h, that is, limn→∞ hn − h = 0 and let X be a bounded subset of L such that x  r. For > 0 there exists n ∈ N such that n  n implies hn − h  r . Therefore, for x ∈ X and n  n we have hn (x) − h(x) = (hn − h)(x)  hn − h · x < , which means that (hn ) converges uniformly on X. Conversely, suppose that (hn ) converges uniformly on every bounded subset X of L. In particular, this takes place on the sphere B[0L , 1]. Therefore, for 2 there is n ∈ N such that n  n implies hn (x) − h(x)  2 for  x  1. Therefore, hn − h  2  , which concludes the proof. The notion of pointwise convergence introduced in Definition 5.18 is applicable to linear operators between normed linear spaces. Namely, a sequence of linear operators (hn ) in L(L, K) (where L and K be two normed F–linear spaces) is pointwise convergent if for every x ∈ L the sequence (hn (x)) is convergent in K. If (hn ) is a pointwise convergent sequence of linear operators, it is possible to define a operator h : L −→ K as h(x) = limn→∞ f (x). The mapping h is linear because h(ax + by) = lim hn (ax + by) = lim (ahn (x) + bhn (y)) = ah(x) + bh(y) n→∞

n→∞

for a, b ∈ F and x, y ∈ L. Theorem 6.29. If (hn ) is a sequence of linear operators in L(L, K) that converges in norm to a linear operator h : L −→ K, that is, limn→∞ hn − h = 0, then (hn ) converges pointwise to h, that is, limn→∞ hn (x) = h(x) for every x ∈ L.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topological Linear Spaces

Proof.

page 345

345

The statement follows from hn (x) − h(x) = (hn − h)(x)  hn − hx.



Theorem 6.30. Let L and K be two normed F-linear spaces. and let h : L −→ K be a linear operator. There exists a linear and continuous left inverse h−1 of h if and only if there exists M > 0 such that h(x)  M x 1 . for every x ∈ L. In this case, h−1   M Proof. Suppose that g is a linear and continuous left inverse of h. By Theorem 6.25, g is bounded, so there exists k > 0 such that g(y)  ky for y ∈ K. If y = h(x), then x = g(h(x))  kh(x), so h(x)  k1 x, which shows that the inequality of the theorem is satisfied with M = k1 . Conversely, suppose that h(x)  M x for every x ∈ L. Note that if x ∈ Null(h) we have h(x) = 0, which implies x = 0, that is, x = 0L . Thus, h is injective, hence there exists a left inverse g : Img(h) −→ L. Note that for y ∈ Img(h) such that y = h(x) we have g(y) = x 

1 1 h(x) = y, M M

which shows that g is continuous.



Theorem 6.31. Let L be a normed space and let h : L −→ L be a linear continuous operator on L. If 1L is the identity operator on L and h  r < 1, then 1L − h has an continuous left inverse operator (1L − h)−1 and 1 (1L − h)−1   1− h . Proof.

We have

(1L − h)x = x − h(x)  x − h(x)  x − rx = (1 − r)x. By Theorem 6.30, 1L − h has a continuous left inverse operator (1L − h)−1 1 and (1L − h)−1   1− h .  Corollary 6.10. Let L and K be two normed F-linear spaces and let h : L −→ K be a surjective linear operator. The spaces L and K are homeomorphic if and only if there exists m, M ∈ F such that mx  h(x)  M x for every x ∈ L. Proof.

This statement is a consequence of Theorem 6.30.



Theorem 6.32. Let L be a real finite-dimensional normed linear space. If U is a closed and bounded subset of L, then U is a compact set.

May 2, 2018 11:28

346

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 346

Mathematical Analysis for Machine Learning and Data Mining

Proof. Let {x1 , . . . , xm } be a basis in L. Define f : Rm −→ L as f (a) = a1 x1 + · · · + am xm for a ∈ Rm . We have   m m       (ai − bi )xi   |ai − bi |xi  f (a) − f (b) =    i=1

 max |ai − bi | 1im

i=1

m 

xi  = a − b∞

i=1

m 

xi .

i=1

Let W = f −1 (U ) ⊆ Rm . By Theorem 4.77, W is a closed set. We claim that W is bounded. Indeed, note that the set {a ∈ Rm | a∞ = 1} is a closed and bounded subset of Rm , and therefore, this set is compact. Thus, f achieves μ = inf{f (a) | a∞ = 1} in some b ∈ Rm . In other words, there exists b ∈ Rm with b∞ = 1 such that m f (b) = i=1 bi xi = μ. Clearly, we have b = 0m . This implies f (b) = 0 m because, otherwise we would have i=1 bi xi = 0 and this would imply bi = 0 for 1  i  m by the linear independence if x1 , . . . , xm . Therefore, f (b) = 0, which allows to conclude that μ = f (b) > 0. Since U is bounded we have x   c for  x ∈ U .  1  m Let a ∈ R . If a = 0m , we have  a a = 1. Therefore,      1   a   μ, f   a∞ or f (a)  μa∞ . This inequality also holds for a = 0m , so it holds for a ∈ Rm . For a ∈ W we have f (a) ∈ U , hence a∞  μc and μa∞  f (a)  c, which implies that W is bounded.



Theorem 6.33. A real finite-dimensional normed linear space is complete. Proof. Let L be a real finite-dimensional normed linear space and let (xn ) be a Cauchy sequence in L. Let n1 ∈ N be a number such that i, j  n1 implies xi − xj  < 1. Thus, i  n1 implies xi   xi − xn1  + xn1   1 + xn1 . Therefore, xi   1 + x1  + · · · + xn1  = k for i ∈ N. We conclude that the set {xi | i ∈ N} ⊆ B[0, k], so {xi | i ∈ N} is bounded. By Theorem 6.32 the set B[0, k] is compact. Thus, (xn ) must contain a convergent subsequence (xni ). By Theorem 5.28 the sequence (xn ) converges and its limit is the same as the limit of the subsequence. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topological Linear Spaces

page 347

347

Theorem 6.34. Every finite-dimensional subspace of a normed linear space is closed. Proof. Let U be a finite-dimensional subspace of a normed linear space L and let (un ) be a sequence in U such that limn→∞ un = u. Thus, (un ) is a Cauchy sequence and, by the completeness of U established in Theorem 6.33, we have u ∈ U . This shows that U is closed.  Theorem 6.35. (Riesz’ Lemma) Let L be a normed linear space and let U be a proper and closed subspace of L. If r ∈ (0, 1) there exists y ∈ L such that y = 1 and inf{y − u | u ∈ U }  r. Proof. If z ∈ L − U we have d(z, U ) > 0 by Theorem 5.16 because U = K(U ). Let a = inf{z − u | u ∈ U } > 0. Since ar > a, there exists u1 ∈ U such that z − u1  < ar . Then, we have for all u ∈ U :   z − u 1   1 − u = z − (u1 + z − u1 u)  z − u1  z − u1  r  a = r, a because u1 + z − u1 u ∈ U . Thus y =

z−u1 z−u1

is the desired element.



The next statement offers a characterization of finite-dimensional normed linear spaces. Theorem 6.36. Let L be a normed linear space. L is finite-dimensional if and only if its unit ball B[0, 1] is compact. Proof. If L is finite-dimensional, taking into account that B[0, 1] is bounded and closed, it follows that B[0, 1] is compact by Theorem 6.32. Conversely, suppose that B[0, 1] is compact but L is not finitedimensional. Consider a sequence (xn ) defined inductively as follows. Let x0 be such that x0  = 1. Suppose that we defined x0 , x1 , . . . , xn and let Un be the subspace generated by these elements. Since Un is finitedimensional it follows that it is closed. By Riesz’ Lemma we select xn+1 such that xn+1  = 1 and d(xn+1 , Un ) > 12 . Then, xn − xi   12 for i < n and, therefore, (xn ) cannot have a convergent subsequence, which contradicts the compactness of B[0, 1].  The notion of separability of topological spaces is applicable to normed linear space.

May 2, 2018 11:28

348

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 348

Mathematical Analysis for Machine Learning and Data Mining

Example 6.1. Any finite dimensional normed real linear space L is separable. Indeed, if B = {e1 , . . . , em } is a basis in L, then for every x ∈ L we m have x = i=1 ai ei . Each of the numbers ai is the limit of a sequence of m rational numbers ai = limn→∞ rni . If xn = i=1 rni ei , then it is easy to see that x = limn→∞ xn , which shows that the set of elements of L with rational coefficients in dense in L. Therefore, L is separable.

6.7

Topological Aspects of Convex Sets

Theorem 6.37. Let C be a convex set in a real linear space L. Both K(C) and I(C) are convex sets. Proof. The convexity of C implies (1 − a)C + aC ⊆ C for a ∈ [0, 1]. Since ha is a homeomorphism of L we have (1 − a)K(C) ⊆ K((1 − a)C) and aK(C) ⊆ K(aC). Therefore, (1−a)K(C)+aK(C) ⊆ K((1−a)C)+K(aC) ⊆ K((1−a)C +aC)) ⊆ K(C), which implies that K(C) is convex. If u, w ∈ I(C), there exist U, W ∈ neigh0L (L, O) such that u + U ⊆ C and v + V ⊆ C. The convexity of C implies (1 − a)(u + U ) + a(v + V ) = (1 − a)u + av + (1 − a)U + aV ⊆ C, 

so I(C) is convex.

Theorem 6.38. Let C be a non-empty convex set in a normed real linear space L such that I(C) = ∅. If x ∈ I(C) and y ∈ K(C), then [x, y) ⊆ I(C). Proof. Since x ∈ I(C), there exists r0 > 0 such that B(x, r0 ) ⊆ C. To show that [x, y) ⊆ I(C) it suffices to prove that for every z ∈ (x, y) ⊆ C there exists r > 0 such that B(z, r) ⊆ C. Since z ∈ (x, y), there exists a ∈ (0, 1) such that z = (1 − a)x + ay. Since y ∈ K(C), for every r1 > 0, we have B(y, r1 ) ∩ C = ∅, so there exists w1 ∈ C such that y − w1  < r1 . If r < 12 ((1 − a)r0 + ar1 ), we claim that B(z, r) ⊆ C. Indeed, let w ∈ B(z, r). We have z − w < r If w2 is defined such that w is a convex combination of w1 and w2 , that is, w = (1 − a)w2 + aw1 , then x − w2 = x −

a 1 w− w1 , 1−a 1−a

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topological Linear Spaces

so

page 349

349

    1 a   x − w2  = x − w+ w1   1−a 1 − a     1  a 1 a   =  z− y− w+ w1   1 − a 1−a 1−a 1 − a 

1 a z − w + y − w1  1−a 1−a a 1 r+ r1 .  1−a 1−a Since r < 12 ((1 − a)r0 + ar1 ), we have a 1 1 3 a r+ r1 < r0 + r1 . 1−a 1−a 2 21−a Choosing r1 such that r1 < 1−a 3a r0 , it follows that x − w2  < r0 , so w2 ∈ B(x, r0 ) ⊆ C, hence w2 ∈ C. Since w is a convex combination of two elements of C, if follows that w ∈ C, so B(z, r) ⊆ C, which shows that z ∈ I(C), that is, [x, y) ⊆ I(C).  

Theorem 6.39. If C is a convex subset of a normed real linear space L such that I(C) = ∅, then K(I(C)) = K(C) and I(K(C)) = I(C). Proof. Note that, since I(C) ⊆ C, we have K(I(C)) ⊆ K(C). Conversely, let x ∈ K(C). Since I(C) = ∅, there exists y ∈ I(C) such that za = (1 − a)x + ay ∈ I(C). Since lima→1 za = x, it follows that x ∈ K(I(C)), so K(C) ⊆ K(I(C)). For the second equality observe that C ⊆ K(C) implies I(C) ⊆ I(K(C)). Let u ∈ I(K(C)). There exists > 0 such that B(u, ) ⊆ K(C). Let . v ∈ I(C), where v = u and let y = (1 + d)u − dv, where d = 2 u−v We have y − u = du − v  , 2 it follows that y ∈ K(C). Since d 1 y+ v, u= 1+d 1+d y ∈ K(C) and v ∈ I(C), it follows that u ∈ I(C) by Theorem 6.38, so I(K(C)) ⊆ I(C).  n+1 Theorem 6.40. Let Tn+1 = {(t1 , . . . , tn+1 ) | i=1 ti = 1, ti  0 for 1  i  n + 1}. The topological subspace (Tn+1 , On ) of the topological space (Rn+1 , On+1 ) is homeomorphic to any simplex S[x1 , . . . , xn , xn+1 ] in Rn , where the vectors x1 , . . . , xn , xn+1 are in general position.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 350

Mathematical Analysis for Machine Learning and Data Mining

350

Proof.

Let f : Tn+1 −→ S[x1 , . . . , xn , xn+1 ] be the function defined by f (t1 , . . . , tn , tn+1 ) = t1 x1 + · · · + tn xn + tn+1 xn+1

for

⎛ ⎜ ⎜ t=⎜ ⎝

t1 .. . tn

⎞ ⎟ ⎟ ⎟, ⎠

tn+1

n+1

where i=1 ti = 1 and ti  0. The uniqueness of the barycentric coordinates (Theorem 3.34 implies that f is a bijection. Observe that we have f (t) = (x1 · · · xn xn+1 )t = Xt,  where X = (x1 · · · xn xn+1 ) ∈ R

n×(n+1)

and t =

t1 . . . tk , tk+1 )

 . Thus, f is

also continuous as a linear mapping. The inverse of f is also continuous. Indeed, note that if t1 x1 + · · · + tn xn + tn+1 xn+1 = y, then t1 (x1 − xn+1 ) + · · · + tn (xn − xn+1 ) = y − xn . Since x1 , . . . , xn , xn+1 are in general position, the vectors x1 − xn+1 , . . . , xn − xn+1 are linearly independent and the matrix V = (x1 − xn+1 · · · xn − xn+1 ) is invertible. This yields ⎛ ⎞ t1 ⎜ .. ⎟ −1 ⎝ . ⎠ = V (y − xn+1 ). tn Therefore,

⎛ ⎜ ⎜ ⎜ ⎝

t1 .. . tn

⎞  ⎟  In 0n ⎟ (V  (y − xn+1 ) 1), ⎟= −1n 1 ⎠

tn+1 which implies that the inverse of f is also continuous.



We shall refer to the homeomorphism f defined in Theorem 6.40 as the canonical homeomorphism Tn+1 and S[x1 , . . . , xn , xn+1 ]. The existence of this homeomorphism between Tn+1 and S[x1 , . . . , xn , xn+1 ] allows us to transfer topological properties to simplexes. An example is given in the next corollary. Corollary 6.11. Every simplex S[x1 , . . . , xn , xn+1 ] in Rn is a compact set.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topological Linear Spaces

page 351

351

Proof. Since Tn+1 is a closed and bounded set of Rn , it follows that Tn+1 is compact. The property of S[x1 , . . . , xn , xn+1 ] follows from the previous theorem.  Theorem 6.41. The set {x ∈ S[x1 , . . . , xn , xn+1 ] | ai (x) > 0 for 1  i  n + 1} is an open set in S[x1 , . . . , xn , xn+1 ]. Furthermore, x ∈ ∂S[x1 , . . . , xn , xn+1 ] if and only if ai (x) = 0 for some i, 1  i  n + 1. Proof. Consider the open sets Ui = {t ∈ Rn+1 | ti > 0}, where 1  i  n + 1. Note that {t ∈ Tn+1 | ti > 0 for 1  i  n + 1} = Tn+1 ∩

n+1 

Ui ,

i=1

which implies that {t ∈ Tn+1 | ti > 0 for 1  i  n + 1} is an open set in Tn+1 . Its image through the canonical homeomorphism f is the set {x ∈ S[x1 , . . . , xn , xn+1 ] | ai (x) > 0 for 1  i  n + 1}, so this latest set is open in S[x1 , . . . , xn , xn+1 ]. Note that Tn+1 ⊆ H1n ,1 . The canonical homeomorphism f can be extended to the hyperplane H1n ,1 as f (t) = t1 x1 + · · · + tn xn + tn+1 xn+1 ; the points in ∂Tn+1 (which must have some ti equal to 0) are mapped by this extension into points in ∂S[x1 , . . . , xn , xn+1 ]. If x ∈ ∂S[x1 , . . . , xn , xn+1 ], any open set that contains x intersects Rn+1 −S[x1 , . . . , xn , xn+1 ], so it contains points in f (H1n+1 ,1 ) with negative  coordinates. This implies that some tj is 0. Theorem 6.42. Every point located in I(S[x1 , . . . , xn , xn+1 ]) belongs to the interior of a unique subsimplex of S[x1 , . . . , xn , xn+1 ]. Proof. If x ∈ I(S[x1 , . . . , xk , xk+1 ]) then we can write x = ai1 (x)xi1 + · · · + aim (x)xim , where {i1 , . . . , im } is the set of subscripts that correspond to non-zero barycentric coordinates. Since these coordinates are uniquely determined by x, it follows that x belongs to the interior of a unique sub simplex of S[x1 , . . . , xn , xn+1 ]. 6.8

The Relative Interior

Consider the uni-dimensional simplex S in R2 generated by the points x1 = # $ # $ 1 0 0 and x2 = 1 . The interior I(S) is empty since no two-dimensional

May 2, 2018 11:28

352

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 352

Mathematical Analysis for Machine Learning and Data Mining

sphere centered in a point of S is included in S. However, there exists a continuous bijection h between the interval # [0,$1] (which has a non-empty r interior in R1 ) and S, defined by h(r) = 1 − r , and this suggests that the notion of interior of a set should be adapted to accommodate this situation. This can be achieved by relaxing the defining condition for interior points of C that requires for x ∈ I(C) the existence of a positive number r such that B(x, r) ∩ C = ∅. Instead, we define a point x to be relatively interior to C if the intersection with C of each line through x contains an open interval about x. Equivalently, x is relatively interior to C if given y ∈ C, there is an r > 0 such that x + ty ∈ C for all t with |t| < r. Definition 6.9. Let C be a convex set, C ⊆ Rn . The relative interior RI(C) of C consists of all relatively interior points of C: RI(C) = {x ∈ C | B(x, r) ∩ Kaff (C) ⊆ C for some r > 0}. A set S is relatively open is S = RI(S). It is clear that I(C) ⊆ RI(C) ⊆ C for each convex set C. Example 6.2. Let C = {x0 } be an one-point subset of Rn . It is immediate that I(C) = ∅, since there is no open sphere B(x0 , r) with r > 0 included in C. On the other hand, RI(C) = {x0 }. Example 6.3. Let C = {x ∈ R3 | x21 +x22  1, x3 = 0}. We have I(C) = ∅. However, since Kaff (C) = {(x1 , x2 , 0) | x1 , x2 ∈ R}, the relative interior of C is RI(C) = {x ∈ R3 | x21 + x22 < 1, x3 = 0}. If C is a convex subset of Rn that contains a non-empty open set D, then RI(C) = ∅. Indeed, in this case, for x ∈ D there exists r > 0 such that B(x, r) ⊆ D. Therefore, B(x, r) ∩ Kaff (C) ⊆ D ∩ Kaff (C) ⊆ C ∩ Kaff (C) = C, which implies x ∈ C. Thus, RI(C) is non-empty. This observation is applied in the next example. Example 6.4. Let S[x1 , . . . , xk , xk+1 ] be the k-dimensional simplex in Rn generated by the affinely independent points x1 , . . . , xk , xk+1 . Thus, Kaff consists of those points x that are affine combinations of the form x = a1 x1 + · · · + ak xk + ak+1 xk+1 ,

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topological Linear Spaces

page 353

353

where a1 + · · · + ak + ak+1 = 1 and the coefficients a1 , . . . , ak , ak+1 are uniquely determined by x. Note that S[x1 , . . . , xk , xk+1 ] = Kconv ({x1 , . . . , xk , xk+1 }) ⊆ Kaff ({x1 , . . . , xk , xk+1 }). Let h : Kaff (S) −→ Rk+1 be the affine and continuous mapping defined as

⎞ a1 ⎜ .. ⎟ ⎟ ⎜ h(x) = ⎜ . ⎟ ⎝ ak ⎠ ⎛

ak+1 for x ∈ Kaff (S). Define the Ki as Ki = R × · · · × R>0 × · · · R, where the ith component of each vector of Ki is positive, for 1  i  k + 1. Each set Ki is open, and therefore, each set h−1 (Ki ) is an open non-empty k+1 subset of Kaff (S). This implies that i=1 h−1 (Ki ) is open. Furthermore, we have k+1 

h−1 (Ki ) = {a1 x1 + · · · + ak xk + ak+1 xk+1 | ai > 0 for 1  i  k + 1}

i=1

⊆ Kconv ({x1 , . . . , xk , xk+1 }) = S[x1 , . . . , xk , xk+1 ]. Thus, Kaff ({x1 , . . . , xk , xk+1 }) contains a non-empty open set, so the relative interior of S[x1 , . . . , xk , xk+1 ] is non-empty. In the special case of uni- and bi-dimensional simplexes we have S[x1 , x2 ] ⊆ S[x1 , x2 , x3 ]. Note, however, that RI(S[x1 , x2 ]) = {(1 − a)x1 + ax2 | a ∈ (0, 1)}, while RI(S[x1 , x2 , x3 ]) = {(1 − a − b)x1 + ax2 + bx3 | a, b ∈ (0, 1)}. This shows that the relative interior operation is not monotonic. Indeed, we actually have: RI(S[x1 , x2 ]) ∩ RI(S[x1 , x2 , x3 ]) = ∅. Theorem 6.43. If C is a non-empty convex subset of Rn , then RI(C) = ∅.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 354

Mathematical Analysis for Machine Learning and Data Mining

354

Proof. Let m be the dimension of Kaff (C). There is an affinely independent set {x1 , . . . , xm+1 } of points from C, where m is maximal with this property. The set S = Kconv ({x1 , . . . , xm+1 }) is a simplex contained in C. As we saw in Example 6.4, S has a non-empty relative interior. Since Kaff (S) ⊆ Kaff (C) and dim(Kaff (S)) = m = dim(Kaff (C)), it follows that  Kaff (S) = Kaff (C). Thus, S has a non-empty relative interior. Theorem 6.44. Let C be a non-empty convex set in Rn . If x ∈ RI(C) and y ∈ K(C), then [x, y) ⊆ RI(C). Proof. Suppose initially that y ∈ C. Since x ∈ RI(C), there exists r > 0 such that such that B(x, r) ∩ Kaff (C) ⊆ C. Let xa = (1 − a)x + ay for 0  a < 1, so xa ∈ [x, y), and let z ∈ B(xa , (1 − a)r) ∩ Kaff (C). 1 1 (z − xa ). We have w − x = 1−a z − xa  < r, so Define w = x + 1−a w ∈ B(x, r). Also, w = x+

1 a 1 (z − (1 − a)x − ay) = z− y ∈ Kaff (C), 1−a 1−a 1−a

so w ∈ B(x, r) ∩ Kaff (C), which shows that z is a convex combination of some point of w ∈ B(x, r) ∩ Kaff (C) ⊆ C and of y. Since C is convex, xa ∈ RI(C). Suppose now that y ∈ K(C) − C. Consider a sequence (yk ) of elements of C such that limk→∞ yk = y and let yk,a = (1 − a)x + ayk . Let r be a positive number such that B(x, r) ∩ Kaff (C) ⊂ C. Then, B(yk,a , (1 − a)r) ∩ Kaff (C) ⊆ C for all k. Since limk→∞ yk,a = ya , if k ) ⊆ B(yk,a , (1 − a)r). Therefore, is sufficiently large, we have B(ya , (1−a)r 2 (1−a)r  B(ya , 2 ) ∩ Kaff (C) ⊆ C, so ya ∈ RI(C). Corollary 6.12. If C is a convex subset of Rn , then RI(C) is a convex set. Proof. RI(C).

This follows immediately from Theorem 6.44 by taking y ∈ 

Theorem 6.45. (Prolongation Theorem) Let C be a non-empty convex subset of Rn . We have z ∈ RI(C) if and only if for every x ∈ C, there exists a > 1 such that (1 − a)x + az ∈ C. Proof. Let za = (1 − a)x + az for a ∈ R. It is clear that za ∈ Kaff {x, z}. If a = 0 we have za = x and if a = 1, z1 = z. Therefore, if a > 1, za is

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topological Linear Spaces

page 355

355

located on the prolongation of the segment [x, z] (which explains the name given to this theorem). Suppose that z ∈ RI(C). Since z − za  = (1 − a)(z − x) = |1 − a|z − x, r it suffices to take |1 − a| < z−x , to obtain za ∈ B(z, r). Since za ∈ Kaff [x, z] ⊆ Kaff (C), it follows that za ∈ C. Conversely, suppose that for every x ∈ C, there exists a > 1 such that (1 − a)x + az ∈ C. Since RI(C) = ∅, there exits x ∈ RI(C). By hypothesis, there exists ya = (1 − a)x + az with a > 1 such that ya ∈ C. Then,

z= where

a−1 1 a , a

1 a−1 x+ , a a

∈ (0, 1). This implies z ∈ RI(C).



Theorem 6.46. Let C be a convex subset of Rn . The relative interior RI(C) is a relatively open subset of Rn . Proof. We need to show that RI(RI(C)) = RI(C). Suppose that x ∈ RI(RI(C)). Then for every u ∈ RI(C) there exist v ∈ RI(C) and a ∈ [0, 1) such that x = (1 − a)u + av. Also, since u, v ∈ RI(C), then for every s, t ∈ C, there exist p, q ∈ C and μ, ν ∈ [0, 1) such that u = (1 − μ)s + μp and v = (1 − ν)t + νq. This allows us to write x = (1 − a)u + av = (1 − a)((1 − μ)s + μp) + a((1 − ν)t + νq) = (1 − a)(1 − μ)s + (1 − a)μp + a(1 − ν)t + aνq, which shows that x can be expressed as a convex combination of s, p, t, and q.  Definition 6.10. Let C be a convex set in Rn . The relative border of C is the set ∂ r (C) = K(C) − RI(C). Theorem 6.47. Let C be a non-empty, convex, and closed subset of a complete normed real linear space L. For each x ∈ L there is a unique t ∈ C that is closest to x. Proof. Let d = d(x, C) and let (yn ) be a sequence in C such that yn − x = d. We claim that (yn ) is a Cauchy sequence.

May 2, 2018 11:28

356

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 356

Mathematical Analysis for Machine Learning and Data Mining

  Note that 12 yn + 12 ym ∈ C because C is convex. Therefore,  12 yn + 12 ym −

  x  d. We have

yn − ym 2 = (yn − x) − (ym − x)2 = 2yn − x2 + 2ym − x2 − yn + ym − 2x2 (by the Parallelogram Equality) 2  y + y   n m − x = 2yn − x2 + 2ym − x2 − 4 2  2yn − x2 + 2ym − x2 − 4d2 , which shows that (yn ) is indeed a Cauchy sequence. Due to the completeness of L there exists y = limn→∞ yn . By the continuity of the norm, x − y = x − lim yn  = lim x − yn  = d. n→∞

n→∞

Suppose that both u, v ∈ C are such that x − u = x − v = d. If limn→∞ yn = u and limm→∞ ym = v in the inequality yn − ym 2  2yn − x2 + 2ym − x2 − 4d2 , it follows that u − v2  0, which implies u = v.



Corollary 6.13. If T is a closed subspace of a complete normed real linear space L, then T is an approximating subspace of L and L = T  T ⊥ . Proof. The statement follows immediately from Theorem 6.47 by observing that every subspace is a non-empty convex set. The second part of the corollary follows from Theorem 2.46.  6.9

Separation of Convex Sets

Definition 6.11. Let L be a real linear space and let f be a linear functional defined on L. A hyperplane H = {x ∈ L | f (x) = a} separates two subsets C, D of L if f (x)  a for every x ∈ C and f (x)  a for every x ∈ D. If a separating hyperplane H exists for two subsets C, D of L we say that C, D are separable.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Topological Linear Spaces

9in x 6in

b3234-main

page 357

357

In other words, C and D are separated by a hyperplane H if C and D are located in distinct closed half-spaces associated to H. The sets C and D are linearly separable if there exists a hyperplane that separates them. Definition 6.12. The subsets C and D of a real linear space L are strictly separated by a hyperplane H = {x ∈ L | f (x) = a} if we have either f (x) > a > f (y) for x ∈ C and y ∈ D, or f (y) > a > f (x) for x ∈ C and y ∈ D. The sets C and D are strictly linearly separable if there exists a hyperplane that strictly separates them. Theorem 6.48. (Geometric Version of Hahn-Banach Theorem) Let C be a convex subset of a real normed linear space L such that I(C) = ∅ and let V be an affine subspace such that V ∩ I(C) = ∅. There exists a closed hyperplane H in L and a linear functional ∈ L∗ such that (i) V ⊆ H and H ∩ I(C) = ∅, and (ii) there exists c ∈ R such that (x) = c for all x ∈ V and (x) < c for all x ∈ I(C). Proof. Suppose initially that 0L ∈ I(C) and let S be the subspace generated by V , S = V . Thus, V is a hyperplane in S and, by Theorem 2.55 there exists a linear functional f on S such that V = {x ∈ S | f (x) = 1}. Since V ∩ I(C) = ∅ we have f (x) = 1  mC (x) for x ∈ V . Since f is linear, if a > 0 we have f (ax) = a  mC (ax) for x ∈ V ; if a < 0, f (ax)  0  mC (ax), so we have f (ax)  mC (ax) for every a. By Hahn-Banach Theorem (Theorem 2.19) there exists an extension

of f to L such that (x)  mC (x) for x ∈ L. Define H as the hyperplane H = {x | (x) = 1}. Since (x)  mC (x) for x ∈ L and mC is continuous, is continuous,

(x) < 1 for x ∈ I(C), so H is the desired closed hyperplane. If 0L ∈ I(C) let x0 ∈ I(C). Then, the previous argument applied to the  set C − x0 yields the same conclusion. Definition 6.13. Let C be a convex set in a linear space L. A hyperplane H in L is said to be a supporting hyperplane of C if the following conditions are satisfied: (i) H is closed; (ii) C is included in one of the closed half-spaces determined by H; (iii) H ∩ K(C) = ∅.

May 2, 2018 11:28

358

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 358

Mathematical Analysis for Machine Learning and Data Mining

Theorem 6.49. Let C be a convex set in a linear space L. If I(C) = ∅ and x0 ∈ ∂C, then there exists a supporting hyperplane H of C such that x0 ∈ H. Proof. Let V be affine subspace V = {x0 }. Since x0 ∈ ∂C it is clear that {x0 } ∩ I(C) = ∅. By Theorem 6.48 there exists a closed hyperplane H in L and a linear functional F ∈ L∗ such that (i) x0 ∈ H and H ∩ I(C) = ∅, and (ii) there exists c ∈ R such that F (x0 ) = c for all and F (x) < c for all x ∈ I(C). Furthermore, C ⊆ {x ∈ L | F (x)  c}.  Note that the support hyperplane is not unique, in general. The next statement establishes that under certain conditions two convex subsets of a real linear space are separated by a hyperplane. It was proven in [46]. Theorem 6.50. (Separation Theorem) Let L be a real linear space and let C1 , C2 be two non-empty convex sets in L such that I(C1 ) = ∅ and C2 ∩ I(C1 ) = ∅. There exists a closed hyperplane H separating C1 and C2 . In other words, there exists a linear functional f ∈ L∗ such that sup{f (x) | x ∈ C1 }  inf{f (x) | x ∈ C2 }, which means that C1 and C2 are located in distinct half-spaces determined by H. Proof. Let C = C1 − C2 . We have I(C) = ∅ and 0L ∈ C. By Theorem 6.49 there exists a linear functional f in L∗ that is not equal to 0 such that f (x)  0 for x ∈ C. Therefore, f (x1 )  f (x2 ) for x1 ∈ C1 and x2 ∈ C2 , which implies the existence of c ∈ R such that sup{f (x1 ) | x1 ∈ C1 }  c  inf{f (x2 ) | x2 ∈ C2 }. The desired hyperplane is H = {x ∈ L | f (x) = c}.



Corollary 6.14. Let C1 , C2 be two disjoint subsets of a locally convex linear topological space. If C1 is open, then C1 and C2 are separable, that is, there exists a linear functional f ∈ L∗ such that sup{f (x) | x ∈ C1 }  inf{f (x) | x ∈ C2 }, Theorem 6.51. Let L be a linear normed space. For every x0 ∈ L there exists a linear functional : L −→ R such that   = 1 and (x0 ) = x0 .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topological Linear Spaces

page 359

359

Proof. For x0 = 0L the existence of is immediate. So let x0 = 0L and let C1 = B[0L , x0 ] and C2 = {x0 }. Since I(C1 )∩C2 = ∅, by Theorem 6.50 there exists a linear functional 1 and a ∈ R such that 1 (s)  a  1 (x) for all s ∈ C1 . Define = 11 1 . We have   = 1 and (s)  (x) for all x ∈ C1 . Then, x = x sup{| (y)| | y  1} = sup{| (x y)| | y  1} = sup{| (s)| | s ∈ C1 } = sup{ (s) | s ∈ C1 }  (x). 1 0 Since   = 1 we have sup |(y)| y | y = 0L = 1, which implies (y)  y for all y ∈ L. This, in turn yields (x)  x, hence (x) = x.



Proof. Since C1 is open, I(C1 ) = C1 = ∅. Therefore, by Theorem 6.50,  the sets C1 , C2 are separable. Theorem 6.52. Let C be an open convex set in a real linear space L and let f : L −→ R be a non-zero linear continuous functional. Then, the set f (C) is an open subset of R. Proof. If t ∈ f (C), then there exists y ∈ C such that f (y) = t. Let x0 ∈ L be such that f (x0 ) = 0. The set U0 = {t ∈ R | y + tx0 ∈ C} contains 0 and is open in R because t ∈ U0 implies t ∈ f (x1 0 ) (C − y), hence U0 = f (x1 0 ) (C − y). Since f (y) + U0 is an open set that contains t = f (y) and is included in f (C) it follows that f (C) is open.  Corollary 6.15. Let C1 , C2 be two disjoint open subsets of a locally convex linear topological space. The sets C1 , C2 are strictly separable. Proof. By Corollary 6.14, the sets C1 , C2 are separable, so there exists a non-zero linear functional such that sup{f (x) | x ∈ C1 }  a and a  inf{f (x) | x ∈ C2 }. By Theorem 6.52, f (C1 ) ⊆ (−∞, a) and f (C2 ) ⊆  (a, ∞), hence C1 , C2 are strictly separable. Theorem 6.53. Let C1 , C2 be two disjoint closed subsets of a locally convex linear topological space L. If C2 is compact, then there exist disjoint open convex sets D1 , D2 in L such that C1 ⊆ D1 and C2 ⊆ D2 . Proof. Let C = C1 − C2 and let (xi )i∈I be a net in C that converges to x. Then xi = yi − zi , where (yi )i∈I is a net in C1 and (zi )i∈I is a net in C2 . Since C2 is compact, there exists a subnet (zij )j∈J that is convergent, zij → z. Since the subnet (xij )j∈J converges to x, the net (yij )j∈J converges to x + z ∈ C1 because C1 is closed. Therefore, y − z ∈ C, hence C is closed.

May 2, 2018 11:28

360

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 360

Mathematical Analysis for Machine Learning and Data Mining

Since C1 , C2 are disjoint, 0L ∈ C, hence there exists a balanced and convex neighborhood W of 0L such that W ∩ C = ∅. The sets U = C1 + 0.5C1 and  V = C2 + 0.5W , then U ∩ V = ∅ and U, V are open and convex. Corollary 6.16. Let C1 , C2 be two disjoint closed subsets of a locally convex linear topological space L. If C2 is compact, then C1 , C2 are strictly separable. Proof.

This follows from Theorem 6.53.



Theorem 6.54. Let C be a closed and convex subset of a real normed linear space L. If x ∈ L − C, there exists a closed half-space U in L such that C ⊆ U and x ∈ U . Proof. Since C is a closed set, d({x}, K) > 0. By Theorem 6.50 applied to the sets C and B[x, d2 ], there exists a hyperplane separating these sets. Then U is the half-space determined by this hyperplane that contains C.  Corollary 6.17. Let C be a closed and convex subset of a real normed linear space L. Then C equals the intersection of all closed half-spaces that contain C. Proof. It is clear that C is included into the intersection T of all closed half-spaces that contain C. Suppose that this inclusion is strict, that is, there x ∈ T − C. By Theorem 6.54 there exists a half-space U such that x ∈ U and C ⊆ U . This leads to a contradiction, hence C = T .  Next, we discuss the separation of convex subsets of Rn . Let C, D be two disjoint subsets of Rn . If w x  a  w y for all x ∈ C and y ∈ D, it follows that sup{w x | x ∈ C}  inf{w y | y ∈ D}, for some w = 0. It is easy to see that this inequality is sufficient for the existence of a hyperplane that separates C and D. Lemma 6.1. Let C be a non-empty and closed convex set, C ⊆ Rn and let x0 ∈ C. There exists a unique point u ∈ C such that u − x0  is the minimal distance from x0 to C. Proof. Let μ = min{x − x0  | x ∈ C}. There exists a sequence of elements in C, (zn ) such that limn→∞ zn − x0  = μ. By the law of the m − x0 2 . parallelogram, zk − zm 2 = 2zk − x0 2 + 2zm − x0 2 − 4 xk +x 2

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topological Linear Spaces

page 361

361

m Since C is convex, we have xk +x ∈ C; the definition of μ implies that 2  x + x 2  k  m − x0   μ2 ,  2 so

zk − zm 2  2zk − x0 2 + 2zm − x0 2 − 4μ2 . Since limn→∞ zn − x0  = μ, for every > 0, there exists n such that k, m > n imply zk − x0  < μ and zm − x0  < μ . Therefore, if k, m > n , it follows that zk − zm 2  4μ2 ( 2 − 1). Thus, (zn ) is a Cauchy sequence. If limn→∞ zn = u, then u ∈ C because C is a closed set. Suppose v ∈ C with v = v and v − x0  = u − x0 . Since C is convex, w = 12 (u + v) ∈ C and we have    1  1 1    (u + v) − x0   u − x0  + v − x0  = μ.  2  2 2     If  12 (u + v) − x0  < μ, the definition of μ is violated. Therefore, we have    1     (u + v) − x0  = μ,  2  which implies u − x0 = k(v − x0 ) for some k ∈ R. This, in turn, implies |k| = 1. If k = 1 we would have u − x0 = v − x0 , so u = v, which is a contradiction. Therefore, a = 1 and this implies x0 = 12 (u + v) ∈ C, which is again a contradiction. This implies that u is indeed unique.  The point u whose existence and uniqueness is established by Lemma 6.1 (see Figure 6.1) is the C-proximal point to x0 . Lemma 6.2. Let C be a non-empty and closed convex set, C ⊆ Rn and let x0 ∈ C. Then u ∈ C is the C-proximal point to x0 if and only if for all x ∈ C we have (x − u) (u − x0 )  0. Proof.

Let x ∈ C. Since x − x0 2 = x − u + u − x0 2 = x − u2 + u − x0 2 + 2(x − u) (u − x0 ),

May 2, 2018 11:28

362

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 362

Mathematical Analysis for Machine Learning and Data Mining

x x u x

 x

C

3

x0 Fig. 6.1

u is the proximal point to x0 in the convex set C.

u − x0 2  0 and (x − u) (u − x0 )  0, it follows that x − x0   x − u, which means that u is the closest point in C to x0 , and the condition of the lemma is sufficient. Conversely, suppose that u is the proximal point in C to x0 , that is, x − x0   x0 − u for x ∈ C. If t is positive and sufficiently small, then u + t(x − u) ∈ C because x ∈ C. Consequently, x0 − u − t(x − u)2  x0 − u2 . Since x0 − u − t(x − u)2 = x0 − u2 − 2t(x0 − u) (x − u) + t2 x − u2 , it follows that −2t(x0 − u) (x − u) + t2 x − u2  0, for t > 0 and t sufficiently small. This is equivalent to 2(x − u) (u − x0 ) + tx − u2  0, which holds for t > 0 and t sufficiently small. This implies (x−u) (u−x0 )  0.  Theorem 6.55. Let C be a non-empty and closed convex set, C ⊆ Rn and let x0 ∈ C. There exists w ∈ Rn − {0n } and a ∈ R such that a < w x0 and a  w x for x ∈ C. Proof. By Lemmas 6.1 and 6.2 there exists a unique proximal point to x0 , u ∈ C, such that (x − u) (u − x0 )  0

(6.3)

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topological Linear Spaces

page 363

363

for every x ∈ C. Let w = x0 − u. The inequality (6.3) is equivalent to x w  u w for x ∈ C and we can write w2 = x0 − u2 = x0 (x0 − u) − u (x0 − u) = x0 w − u w = w  x 0 − u w  w x0 − x w, which implies x w  w x0 − w2 for every x ∈ C. Therefore, if a = sup{w x | x ∈ C} we have a  w x0 − w2 . Since w = 0n , it follows that a < w x0 . The inequality w x  a for x ∈ C follows from the definition of a.



A variation of the previous theorem, where C is just a convex set (not necessarily closed) is given next. Theorem 6.56. Let C be a non-empty convex set, C ⊆ Rn and let x0 ∈ ∂C. There exists w ∈ Rn − {0n } such that w (x − x0 )  0 for x ∈ K(C). Proof. Since x0 ∈ ∂C, there exists a sequence (zm ) such that zm ∈ K(C) and limm→∞ zm = x0 . By Theorem 6.55 for each m ∈ N there exists wm ∈ Rn − {0n } such that wm zm > wm x for each x ∈ K(C). Without loss of generality we may assume that wm  = 1. Since the sequence (wm ) is bounded, it contains a convergent subsequence wip such that limp→∞ wip = w and we have wip zip > wip x for each x ∈ K(C).  Taking p → ∞ we obtain w x0 > w x for x ∈ K(C). Theorem 6.57. Let C be a non-empty convex set, C ⊆ Rn and let x0 ∈ I(C). There exists w ∈ Rn − {0n } such that w (x − x0 )  0 for x ∈ K(C). Proof. If x ∈ K(C), the statement follows from Theorem 6.55. Otherwise, x0 ∈ K(C) − C ⊆ ∂C, so x0 ∈ ∂C and the statement is a consequence of Theorem 6.56.  Corollary 6.18. Let C be a non-empty convex set, C ⊆ Rn and let x0 ∈ ˜  (x − x0 )  0 for x ∈ K(C). ˜ ∈ Rn − {0n } such that w I(C). There exists w ˜ as w ˜ = −w, where w is the vector whose existence was Proof. Define w ˜  (x − x0 )  0 ˜ ∈ Rn − {0n } and w established in Theorem 6.57. Clearly, w for x ∈ K(C). 

May 2, 2018 11:28

364

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 364

Mathematical Analysis for Machine Learning and Data Mining

The existence of a supporting hyperplane for a non-empty convex set and a point located on its border can be shown directly for Rn . Theorem 6.58. Let C ⊆ Rn be a non-empty convex set and let x0 ∈ ∂C. There exists a supporting hyperplane of C at x0 . Proof. Since x0 ∈ ∂C, there exists a sequence (zn ) of elements of Rn − C such that limn→∞ zn = x0 . By Theorem 6.55 for each zn there exists wn such that wn zn > a and wn x  a for x ∈ C. Without loss of generality we may assume that wn  = 1. Since the sequence (wn ) is bounded, it contains a convergent subsequence (wim ) such that limm→∞ wim = w. For this subsequence we have w zim > a and w x  a. Taking m → ∞ we obtain w x0 > a and w x  a for all x ∈ C, which means that Hw,a is  a support plane of C at x0 . Theorem 6.59. Let S, T be two non-empty convex subsets of Rn that are disjoint. There exists w ∈ Rn − {0n } such that inf{w s | s ∈ S}  sup{w t | t ∈ T }. Proof.

It is easy to see that the set S − T defined by S − T = {s − t | s ∈ S and t ∈ T }

is convex. Furthermore 0n ∈ S − T because the sets S and T are disjoint. Thus, there exists in S − T a proximal point w to 0n , for which we have (x − w) w  0 for every x ∈ S − T , that is, (s − t − w) w  0, which is equivalent to s w  t w + w2 for s ∈ S and t ∈ T . This implies the inequality of the theorem.



Corollary 6.19. For any two non-empty convex subsets that are disjoint, there exists a non-zero vector w ∈ Rn such that inf{w s | s ∈ S}  sup{w t | t ∈ K(T )}. Proof. This statement follows from Theorem 6.59 and from Supplement 64 of Chapter 4.  Definition 6.14. A subset C of Rn is bounded from below if inf{xj | x ∈ C} > −∞ for every j, 1  j ≤ n.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Topological Linear Spaces

b3234-main

page 365

365

Example 6.5. The set (Rn )0 and any of its subsets are bounded from below. Theorem 6.60. Let C be a closed and convex subset of Rn that is bounded from below. Then every supporting hyperplane of C contains an extreme point of C. Proof. Let x0 ∈ ∂C. By Theorem 6.58 there exists a supporting hyperplane Hw,a of C at x0 . Define Z = Hw,a ∩ C. Note that Z = ∅ because x0 ∈ Hw,a and x0 ∈ C. Every extreme point z of Z is also an extreme point of C, which is equivalent to showing that if z is not an extreme point of C, then it cannot be an extreme point of Z. Let z is a point in C that is not an extreme point of this set. If z ∈ Z, then z cannot be an extreme point of Z. If z ∈ Z we have z ∈ C. We may assume that C ⊆ {x | w x  a}. Since z is not an extreme point in C, there exist x1 , x2 ∈ C with x1 = x2 such that z = (1 − t)x1 + tx2 for some t ∈ (0, 1). Since w z = a = (1 − t)w x1 + tw x2 and C ⊆ {x | w x  a} it follows that w x1 = a = w x2 , hence x1 , x2 ∈ Z. Thus, z is not an extreme point of Z. We show next that Z contains an extreme point. Define ⎛ 1⎞ z1 ⎜z21 ⎟ ⎜ ⎟ z1 = ⎜ . ⎟ ⎝ .. ⎠ zn1 such that z11 = inf{z1 | z ∈ Z}. Since Z ⊆ C and C is bounded from below, z11 is well-defined. If z1 is unique, we will show next that it is an extreme point. If z1 is not unique, we can define z2 such that ⎛ 2⎞ z1 ⎜z22 ⎟ ⎜ ⎟ z2 = ⎜ . ⎟ ⎝ .. ⎠ zn2 such that z12 = z11 and

⎛ 1 ⎞ ⎫  z1 ⎪  ⎪ ⎪ ⎜ z2 ⎟ ⎬ ⎜ ⎟ 2 z2 = inf z2 ⎜ . ⎟ ∈ Z . ⎝ .. ⎠ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩  ⎭ z ⎧ ⎪ ⎪ ⎪ ⎨

n

May 2, 2018 11:28

366

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 366

Mathematical Analysis for Machine Learning and Data Mining

Since C ⊆ Rn this process will stop at most after n steps. Let zj be the point where the process stops. For all k, 1  k  j, we have ⎧  ⎫ ⎛ ⎞  t1 ⎪ ⎪ ⎪ ⎪  ⎪ ⎪ ⎜ t2 ⎟ ⎨  ⎬ ⎜ ⎟  j j zk = inf zk z = ⎜ . ⎟ , zi = zi for 1  i  k − 1 .  ⎪ ⎪ ⎝ .. ⎠ ⎪ ⎪ ⎪ ⎪ ⎩  ⎭ t n

Suppose zj is not an extreme point of Z. Then, there exist two distinct points u, v ∈ Z such that zj = (1 − t)u + tv for some t ∈ (0, 1). Thus, zij = (1 − t)ui + tvi for 1  i  n. For 1  i  j, since zij is the infimum, zij  ui and zij  vi , which implies zij = ui = vi for 1  i  j, which  contradicts the uniqueness of zj . 6.10

Theorems of Alternatives

Separation results have two important consequences for optimization theory, namely Farkas’ and Gordan’s alternative theorems. Theorem 6.61. (Farkas’ Theorem) Let A ∈ Rm×n and let c ∈ Rn . Exactly one of the following linear systems has a solution: (i) Ax  0m and c x > 0; (ii) A y = c and y  0m . Proof. If the second system has a solution, then A y = c and y  0m for some y ∈ Rm . Suppose that x is a solution of the first system. Then, c x = y Ax  0, which contradicts the inequality c x > 0. Thus, if the second system has a solution, the first system has no solution. Suppose now that the second system has no solution. Note that the set S = {x ∈ Rn | x = A y, y  0m } is a closed convex set. Furthermore, c ∈ S because, otherwise, c would be a solution of the second system. Thus, by Theorem 6.55, there exists w ∈ Rn − {0n } and a ∈ R such that w c > a and w x  a for x ∈ S. In particular, since 0n ∈ S we have a  0 and, therefore, w c > 0. Also, for y  0m we have a  w A y = y Aw. Since y can be made arbitrarily large we must have Aw  0m . Then w is a solution of the first system.  Theorem 6.62. (Gordan’s Alternative Theorem) Let A ∈ Rm×n be a matrix. Exactly one of the following linear systems has a solution: (i) Ax < 0m for x ∈ Rn ; (ii) A y = 0n and y ≥ 0m for y ∈ Rm .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topological Linear Spaces

page 367

367

Proof. Let A be a matrix such that the first system, Ax < 0m has a solution x0 . Suppose that a solution y0 of the second system exists. Since Ax0 < 0m and y0 ≥ 0m (which implies that at least one component of y0 is positive) it follows that y0 Ax0 < 0, which is equivalent to x0 A y < 0. This contradicts the assumption that A y = 0n . Thus, the second system cannot have a solution if the first has one. Suppose now that the first system has no solution and consider the non-empty convex subsets S, T of Rm defined by S = {s ∈ Rm | s = Ax, x ∈ Rn } and T = {t ∈ Rm | t < 0m }. These sets are disjoint by the previous supposition. Then, by Theorem 6.59, there exists w = 0m such that w As  w t for s ∈ S and t ∈ K(T ). This implies that w  0m because otherwise the components of t that correspond to a negative component of w could be made arbitrarily negative (and large in absolute value) and this would contradict the above inequality. Thus, w ≥ 0m . Since 0m ∈ K(T ), we also have w As  0 for every s ∈ Rm . In particular, for s = −A w we obtain w A(−A w) = −A w2 = 0, so A w = 0m , which means that the second system has a solution.  The next definition comes from [27] where a study of consistency of linear inequality systems was developed. Definition 6.15. A linear system is consistent or inconsistent according as solution of the system do or do not exist. n Example 6.6. An inequality j=1 aj xj + b > 0 is inconsistent only when a1 = · · · = an = 0 and b  0. Definition 6.16. A system of inequalities is minimally inconsistent or irreducibly inconsistent if removing one of its conditions renders the system consistent. A single inequality is minimally inconsistent if it is inconsistent. Lemma 6.3. Let { i | 1  i  k} be a set of k linear functionals on a real linear space L. The set Lx = {y ∈ Rk | yi = i (x) for 1  i  k} is a linear subspace of Rk ; furthermore, Lx is a proper subspace of Rk for every x ∈ L if and only if the set { i | 1  i  k} is linearly dependent.

May 2, 2018 11:28

368

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 368

Mathematical Analysis for Machine Learning and Data Mining

Proof. It is immediate that Lx is a linear subspace due to the linearity of the functionals { i | 1  i  k}. The subspace Lx is non-trivial if and only if there exists a non-zero vector w ∈ Rk such that w ⊥ z for every k z ∈ Lx , that is, i=1 wi i (x) = 0 for every x ∈ L, which is equivalent to  the linear dependence of the set { i | 1  i  k}. Lemma 6.4. Let L be a linear space, let : L −→ R be a linear functional, and let xt = (1 − t)a + tb be a point on the segment [a, b] ⊆ L, where −(a) t ∈ [0, 1]. If (a) < 0 and (b)  0, then t < (b)−(a) implies (xt ) < 0. Proof. The inequality (xt ) < 0 is equivalent to (1 − t) (a) + t (b) < 0, −(a) . In other words, if xt is which, in turn, is equivalent to t < (b)−(a)  sufficiently close to a, (xt ) is negative. The next result is a general alternative theorem. We follow the presentation of [71]. Theorem 6.63. (Motzkin’s Transposition Theorem) Let L be a real linear space, and let { i | i ∈ I}, { j | j ∈ J}, and { k | k ∈ K}, be three families of real-valued linear functionals defined on L, where I, J and K are finite sets. Then exactly one if the linear systems

i (x) < 0 for i ∈ I, j (x)  0 for j ∈ J, and k (x) = 0 for k ∈ K, and

(6.4)



  yi i + j∈J zj j + k∈K uk k = 0, y  0n , y = 0n , and z  0q , i∈I

(6.5)

where |I| = n, |J| = p, and |K| = q, is consistent. Proof. Suppose that system (6.4) has a solution x. If system (6.5) has a solution y, z, u, then ⎛ ⎞    ⎝ yi i + zj j + uk k ⎠ (x) = 0. i∈I

j∈J

k∈K

On other hand,     yi i (x) + zj j (x) + uk k (x)  yi i (x) < 0. i∈I

j∈J

k∈K

i∈I

This contradiction shows that system (6.5) is inconsistent. Conversely, if system (6.4) is inconsistent, then system (6.5) has a solution.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 369

Topological Linear Spaces

369

Without loss of generality we may take K = ∅, because each equality

k (x) = 0 can be replaced by a pair of inequalities k (x)  0 and − k (x)  0. Also, we may assume that the system

i (x) < 0 for i ∈ I, j (x)  0 for j ∈ J

(6.6)

is minimally inconsistent. We may also assume that there exists only one strict inequality in the system (6.6), that is, |I| = 1. Indeed, suppose that |I| > 1 and let p ∈ I. Consider the system obtained by replacing all inequalities i (x) < 0 by

i (x)  0 for i = p:

p (x) < 0, i (x)  0 for i ∈ I − {p}, j (x)  0 for j ∈ J.

(6.7)

Since system (6.4) is minimally inconsistent, there exists a ∈ L such that

p (a) < 0, i (a)  0 for i ∈ I − {p}, j (a)  0 for j ∈ J. Since system (6.6) is minimally inconsistent, there exists b ∈ L such that

p (b)  0, i (b) < 0 for i ∈ I − {p}, j (b)  0 for j ∈ J. By Lemma 6.4 this implies that there exists a point c that is sufficiently close to a that satisfies system (6.6) which is a contradiction. This proves that system (6.7) is minimally inconsistent because system (6.4) is minimally inconsistent. The minimal inconsistency of the system

(x) < 0, j (x)  0 for j ∈ J

(6.8)

implies that for any two distinct r, k ∈ J, there exist a, b ∈ L such that

(a) < 0, r (a) > 0, k (a)  0, j (a)  0 for j ∈ J − {r, k},

(b) < 0, r (b)  0, k (b) > 0, j (a)  0 for j ∈ J − {r, k}. This implies the existence of c ∈ [a, b] such that k (c) = 0. Thus, c satisfies the inequalities

(c) < 0, r (c) > 0, k (c) = 0, j (c) < 0 for j ∈ J − {r, k}, where r (c) > 0 follows from the fact that system (6.8) is inconsistent. Consider two consistent systems like the previous one,

(x) < 0, r (x) > 0, k1 (x) = 0, k2 (x)  0,

j (x)  0 for j ∈ J − {r, k1 , k2 },

(x) < 0, r (x)  0, k1 (x) > 0, k2 (x) > 0,

j (x)  0 for j ∈ J − {r, k1 , k2 }.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 370

Mathematical Analysis for Machine Learning and Data Mining

370

that yield a consistent system

(x) < 0, r (x) > 0, k1 (x) = 0, k2 (x) = 0,

j (x)  0 for j ∈ J − {r, k1 , k2 }. By repeatedly applying this argument and replacing one of the non-strict inequalities, we obtain a consistent system

(x) < 0, k (x) > 0, j (x) = 0 for j ∈ J − {k}.

(6.9)

Since system (6.8) is inconsistent, the set of linear functional { } ∪ { j | j ∈ J} is linearly dependent, which implies the existence of y and zj for  j ∈ J such that y + {zj j | j ∈ J} = 0. At least two multipliers are non-zero and we may assume that zr > 0 for some r ∈ J. If a ∈ L satisfies system (6.9) for k = r, then y (a) = −zr r (a) < 0. Since (a) < 0 we have y > 0. For an arbitrary s ∈ J and a b ∈ L that satisfies system (6.9) for k = s, then μs s (b) = −y (b) > 0 and since s (b) > 0, we have μs > 0, which completes the argument. 

6.11

The Contingent Cone

Definition 6.17. Let L be a normed real linear space and let S be a subset of L. If z ∈ K(S), then h ∈ L is a tangent vector to S at z if there is a sequence (xn ) in S such that limn→∞ xn = x and a sequence (an ) ∈ Seq(R0 ) such that h = limn→∞ an (xn − z). The set T (S, z) of all tangent vector to S at z is the contingent cone to S at z. It is easy to verify that T (S, z) is indeed a cone. Theorem 6.64. Let L be a normed real linear space and let S be a nonempty subset of L. If S is local convex at z ∈ S, then Kcone (S − {z}) ⊆ T (S, z). Proof. Suppose that S is local convex at z ∈ S. Let x ∈ S be an arbitrary 1 element and let xn = z +  n (x1 − z). 1 It is clear that limn→∞ xn = z. Furthermore, since xn = 1 − n z + n x and S is locally convex at z it follows that xn ∈ S. Note that n(xn − z) = x − z, so x − z is a tangent vector to S for every x ∈ S, which implies S − {z} ⊆ T (S, z). Since T (S, z) is a cone, it follows  that Kcone (S − {z}) ⊆ T (S, z).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topological Linear Spaces

page 371

371

Theorem 6.65. Let L be a normed real linear space and let S be a nonempty subset of L. For every z ∈ S we have T (S, z) ⊆ K(Kcone (S − {z})). Proof.

This statement follows immediately from Definition 6.17.



Corollary 6.20. Let L be a normed real linear space and let S be a nonempty subset of L. If S is local convex at z ∈ S, then Kcone (S − {z}) ⊆ T (S, z). Proof.

The corollary follows from Theorems 6.64 and 6.65.



Theorem 6.66. Let L be a normed real linear space and let S be a nonempty subset of L. For every z ∈ S the contingent cone T (S, z) is closed. Proof. Let (hn ) be a sequence of vectors in T (S, z) such that limn→∞ hn = h. For each hn there exists a sequence (xn,m ) such that limm→∞ xn,m = z and a sequence of scalars (an,m ) such that hn = limm→∞ an,m (xn,m − z). Therefore, for each n ∈ N there exists M (n) ∈ N such that if m  M (n) we have xn,m − z  n1 and an,m (xn,m − z)  n1 . Let yn = xn,n and bn = an,n . It is clear that limn→∞ yn = z and bn (yn − z) − h = an,n (xn,n − z) − h  an,n (xn,n − z) − hn  + hn − h 1  + hn − h, n which means that h = limn→∞ bn (yn − z), so h ∈ T (S, z). This shows that T (S, z) is closed.  Corollary 6.21. Let L be a normed real linear space and let S be a nonempty subset of L. If S is locally convex at z, then T (S, z) = K(Kcone (S − {z})). Proof. Recall that T (S, z) ⊆ K(Kcone (S − {z})) by Theorem 6.65. Since Kcone (S − {z}) ⊆ T (S, z) by Corollary 6.20, it follows that K(Kcone (S − {z})) ⊆ T (S, z) because T (S, z) is closed. This completes the argument.



Theorem 6.67. The contingent cone T (S, z) of a non-empty convex subset S of a real linear space L is convex for every z.

May 2, 2018 11:28

372

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 372

Mathematical Analysis for Machine Learning and Data Mining

Proof. Let h, k ∈ T (S, z). We have h = limn→∞ an (xn − z) and k = limn→∞ bn (yn − z), where limn→∞ xn = limn→∞ yn for some sequences of real numbers (an ) and (bn ). Let cn = an + bn and an xn + bn yn zn = cn an bn = xn + yn an + b n an + b n define the sequences (cn ) and (zn ). Note that zn ∈ S because S is convex. Since an bn (xn − z) + (yn − z) + z, zn = cn cn it follows that limn→∞ zn = z. The desired conclusion follows from the fact that h + k = lim an (xn − z) + lim bn (yn − z) n→∞

n→∞

= lim an xn + bn yn − cn z) n→∞

= lim cn (zn − z). n→∞



Recall that a face of a convex set C is a convex subset of C such that (u, v) ⊆ C and {u, v} ∩ (C − F ) = ∅ implies (u, v) ⊆ C − F ; the face F of C is proper if F = C. Theorem 6.68. Let C be a convex set in a real linear space L. A face F of C is included in ∂C. If L is locally convex and I(C) = ∅ then any x ∈ C ∩ ∂C belongs to a proper face. Proof.

Let x ∈ F and let y ∈ C − F . We have: [0, 1] ⊆ {a | (1 − a)x + ay ∈ C}.

However, if a < 0, z = (1 − a)x + ay ∈ C, because otherwise we would have 1 −a z+ y 1−a 1−a and this would mean that x ∈ (z, y), where we have z ∈ C, y ∈ C − F and x ∈ F , which contradicts Definition 3.13. 1 Define zn = n+1 n x − n y. We have limn→∞n = x, hence x is a limit point of a sequence of points not in C. This shows that x ∈ K(C) ∩ K(L − C), so x ∈ ∂C. x=

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topological Linear Spaces

page 373

373

Suppose now that x ∈ C ∩ ∂C. Let D = I(C). Since D is open, by Corollary 6.14 there exists a continuous, non-zero linear functional such that c = sup{f (y) | y ∈ D}  (x). Since x ∈ C, (x) = c. Since D is open,

(D) is an open set by Theorem 6.52, so the hyperplane H = {y | (y) = c} is disjoint from D. Thus, x belongs to the proper face H ∩ C. 

6.12

Extreme Points and Krein-Milman Theorem

The notion of supporting set of a convex set generalizes the notion of extreme point. Definition 6.18. Let C be a convex subset of a real linear space L. A set S is a supporting set of C if the following conditions are satisfied: (i) S ⊆ C; (ii) S is closed and convex, and (iii) if u, v ∈ C and I([u, v]) ∩ S = ∅, then [u, v] ⊆ S. Any extreme point x of a convex set can be regarded as a supporting set Sx = {x}. Moreover, if a supporting set for C consists of one point, S = {x}, then x is an extreme point of C. Lemma 6.5. Let C be a convex subset of a real linear space L. The intersection of a family of supporting sets of C is a supporting set of C. Proof.

The lemma is an immediate consequence of Definition 6.18.



Lemma 6.6. Let C be a convex subset of a real linear space L and let S be a supporting set of C. If T is a supporting set of S, then T is a supporting set of C. Proof. It is clear that T ⊆ S ⊆ C and that T is convex and closed by Theorem 4.4. Suppose now that if u, v ∈ C and x ∈ I([u, v]) ∩ T . Then x ∈ I([u, v]) ∩ S, hence [u, v] ⊆ S. Since T is a support set for S, it follows  that [u, v] ⊆ T . Thus, T is a supporting set of C. Lemma 6.7. Let C be a compact and convex subset of a real linear space L and let f be a continuous linear functional on L such that m = max{f (x) | x ∈ C}. The set S = {x ∈ C | f (x) = m} is a supporting hyperplane for C. Proof. Suppose that x, y ∈ S, that is, f (x) = f (y) = m. Then, f ((1 − a)x + ay) = m for a ∈ [0, 1], so S is convex. If [x, y] ⊆ C, z ∈ I([x, y]), and

May 2, 2018 11:28

374

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 374

Mathematical Analysis for Machine Learning and Data Mining

f (z) = m, then m = f ((1 − a)x + ay) and since f (x)  m and f (y)  we must have f (x) = f (y) = m. This implies [x, y] ⊆ S.  Lemma 6.8. Let L be a locally convex linear topological space and let U be a closed convex set such that 0L ∈ U . There exists a continuous linear functional f on L such that inf{f (u) | u ∈ U } > 0. Proof. Since 0L ∈ U , there exists an open set V such that 0L ∈ V and U ∩ V = ∅. Let W = V ∩ (−V ). Then, W is an open convex set that contains 0L , −W = W , and W ∩ U = ∅. Since 0L ∈ RI(W ), there exists a non-zero linear functional f such that sup{f (u) | u ∈ W }  inf{f (u) | u ∈ U } = a. Thus, f (u)  a for every u ∈ W and, since u ∈ W implies −u ∈ W , we also have −f (u)  a for u ∈ W , hence |f (u)|  a for x ∈ W . Therefore, for any c > 0 we have |f (u)| < c for u ∈ ac W . Since W1 = ac W is an open set containing 0L it follows that f is continuous at 0L . To show that a > 0, note that there exists u0 such that f (u0 ) > 0 because f is a non-zero functional. Since 0L ∈ RI(W ), there exists t > 0 so that tu ∈ W . This implies 0 < tf (u) = f (tu)  a.  Lemma 6.9. Let L be a locally convex linear topological space and let U be a closed convex set such that x0 ∈ U . There exists a continuous linear functional f on L such that inf{f (x) | x ∈ U } > f (x0 ). Proof. Since x0 ∈ U , it follows that 0L ∈ t−x0 (U ) and it is clear that t−x0 (U ) is a closed convex set. Thus Lemma 6.8 is applicable, and it implies that there exists a continuous linear functional f on L such that inf{f (z) | z ∈ t−x0 (U )} > 0. Since z ∈ t−x0 (U ), it follows that x = z + x0 ∈ U , hence  inf{f (x) | x ∈ U } > f (x0 ). Krein and Milman theorem is a generalization of a result of Carath´eodory (see Theorem 3.25) to locally convex linear topological spaces. Theorem 6.69. (Krein-Milman Theorem) Let L be a locally convex Hausdorff linear topological space and let C be a non-empty compact and convex subset of L. We have: C = K(Kconv (extr(C))).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Topological Linear Spaces

9in x 6in

b3234-main

page 375

375

Proof. We saw that the intersection of supporting sets for C is a supporting set for C and that a support set of a support set of C is itself a support set for C. The collection S of supporting sets of C is a partially ordered set (relative to set inclusion). Therefore, by Hausdorff maximality principle (Theorem 1.23) there exists a maximal chain C of supporting sets for C. Since C is compact, the intersection T of all non-empty supporting sets of C is non-empty and, also, is a supporting set for C. It is also a minimal nonempty supporting set of C, because if T would contain a strictly smaller supporting set, this would contradict the maximality of the chain C. A minimal supporting M set may contain only one point and such a point must be an extreme point. Indeed, if a supporting set S contains two distinct points x and y, then there exists a continuous linear functional f such that f (x) > f (y). Then, the subset of S where f attains its maximum is a supporting subset of S and, therefore of C. Since C is compact, it is a non-empty supporting set of C that does not contain y. If a supporting set consists of one point x, x must be extreme. Thus, every non-empty supporting set contains an extreme point. The maximum of a continuous linear functional f on C is equal to the maximum on the set extr(C) because the subset of C where a linear functional assumes its maximum is a non-empty supporting set. Let D = K(Kconv (extr(C))). Since extr(C) ⊆ C, it follows that Kconv (extr(C))) ⊆ C because C is convex. Thus, D = K(Kconv (extr(C))) ⊆ K(C) = C. Suppose that x ∈ D. By Lemma 6.9 there is a continuous linear functional f such that f (x) > max{f (y) | y ∈ D} = max{f (y) | y ∈ C}. Thus, x ∈ C, and C ⊆ D, which implies C = D. 

Exercises and Supplements (1) Prove that a subset U of a real topological linear space L is open if and only tz (U ) is open for every z ∈ L. (2) Prove that in a topological linear space (L, O) for every U ∈ neigh0L (L, O) there exists a balanced neighborhood W ∈ neigh0L (L, O) such that W ⊆ U . Solution: The continuity of scalar multiplication implies that there exists > 0 and V ∈ neigh0L (L, O) such that rV ⊆ U if |r| < . Define W = {rV | |r| < }. It is clear that W ∈ neigh0L (L, O), W is balanced and W ⊆ U .

May 2, 2018 11:28

376

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 376

Mathematical Analysis for Machine Learning and Data Mining

(3) Prove that in a topological linear space (L, O) for every convex neighborhood U ∈ neigh0L (L, O) there exists a convex balanced neighborhood W ∈ neigh0L (L, O) such that W ⊆ U .  Solution: The set Z = {rU | |r|  1} is convex as an intersection of convex sets. set. Indeed, for |t|  1 we  Moreover, Z is a balanced  have tZ = {trU | |r|  1} ⊆ {rU | |r|  1} = Z. Thus, by Theorem 6.13, I(Z) is a balanced set. By Supplement 2 there exists a balanced neighborhood W such that W ⊆ U . We have 1r W = W if |r| = 1 because W is balanced, hence W ⊆ rU , which implies W ⊆ Z. This implies I(Z) ∈ neigh0L (L, O); also I(Z) is convex by Theorem 6.37, and, as we saw above, a balanced set. (4) Let L be a real topological linear space. If U ⊆ R is a closed set of scalars  such that 0 ∈ U and T is a closed subset of L such that 0L  T , then {ha (T ) | a ∈ U } is a closed set. (5) Let U, V be two compact subsets of a real topological linear space L. Prove that U + V is compact. (6) Let L be a topological linear space. Prove that: (a) if V ∈ neigh0L (L, O), then Vˆ = V ∩ h−1 (V ) is a symmetric neighborhood of 0L ; (b) if W ∈ neigh0L (L, O) there exists a symmetric neighborhood U of 0L such that U + U ⊆ W ; (c) if W ∈ neigh0L (L, O) for each k  2 there exist symmetric neighborhoods U1 , U2 , . . . , Uk of 0L such that U1 + U2 + · · · + Uk ⊆ W . Solution: The first part is immediate. By the continuity of addition in L there exist V1 , V2 ∈ neigh0L (L, O) such that V1 + V2 ⊆ W . Then, U = Vˆ1 ∩ Vˆ2 is a symmetric neighborhood of 0L and U + U ⊆ W . For the last part by applying m times the second statement one could construct a sequence U1 , U2 , . . . , U2m of symmetric neighborhoods of 0L such that U1 + U2 + · · · + U2m ⊆ W . Choosing m such that k  2m we obtain U1 + · · · + Uk ⊆ U1 + · · · + Uk + · · · + U2m ⊆ W . (7) Prove that a subset U of a linear space L is bounded if and only if for every sequence (xn ) in U we have limn→∞ n1 xn = 0. (8) Let f : L −→ R be a linear functional defined on the real topological linear space L such that f is not identically 0 on L. Prove that the following statements are equivalent: (a) f is continuous; (b) the set {x ∈ L | f (x) = 0} is closed; (c) the set {x ∈ L | f (x) = 0} is not dense in L. (9) Let (L, O) be  a real linear space. Prove that for every subset T of L we have K(T ) = {T + V | v ∈ neigh0L (L, O)}. Solution: We have x ∈ K(T ) if and only if x + V ∩ T = ∅ for every V ∈ neigh0L (L, O). This is equivalent to x ∈ T − V . Since −V ∈

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topological Linear Spaces

page 377

377

neigh0L (L, O) if and only if V ∈ neigh0L (L, O), the desired equality follows. (10) Let (L, O) be a topological F-linear space. Prove that if U is an open set in L and W is a subset of L, then K(W ) + U = W + U . Solution: The inclusion W + U ⊆ K(W ) + U is immediate. To prove the reverse inclusion let y ∈ K(W ) + U . We have y = x + u, where x ∈ K(W ) and u ∈ U . There exists V ∈ neigh0L (L, O) such that u + V ⊆ U because U is an open set. Since x ∈ K(W ), taking into account that x − V ∈ neighx (L, O), there exists some z ∈ W ∩ (x − V ). Then, y = x + u = z + u + (x − z) ∈ z + u + V ⊆ U + V . (11) Let (L, O) be a topological F-linear space. Prove that if U is a compact subset of L and W is a closed subset of L, then U + W is closed. Solution: By Theorem 4.63, since U is compact if and only if each net in S has some convergent subnet. Suppose that a net in U clusters to x and let (xi ) be a subnet in U such that xi → u. Let (yi ) be a net in W such that xi +yi → z. Since yi = (xi +yi )−xi → z − x = y and W is closed, we have y ∈ W , so z = x + y ∈ U + W , which implies that U + W is closed by Corollary 4.14. (12) Let (L, O) be a Hausdorff topological linear space. Prove that every finite dimensional subspace of L is closed. (13) Corollary 6.1 shows that a hyperplane H in a topological linear space L is either closed or dense in L. If H = {x ∈ L | f (x) = 0}, where f is a linear functional, prove that H is closed if and only if f is continuous, and H is dense if and only if f is discontinuous. (14) Prove that a product of locally convex linear spaces is locally convex. (15) Let L be a linear space. Prove that the product topology on RL is a locally convex Hausdorff topology. (16) Let f be a linear functional defined on a normed real linear space (L, ·). Prove that: (a) f is continuous if and only if f −1 (0) is a closed subspace; (b) we have f  = d(0L1,M ) , where Mf = f −1 (1). f

In the special case, when f = 0 we have Mf = ∅ and d(0L , Mf ) = ∞. Solution: It is clear that if f is continuous, then f −1 (0) is a closed subspace of L. Conversely, suppose that f −1 (0) is a closed subspace and let x0 ∈ L − f −1 (0) such that f (x0 ) = 1. Since x0 ∈ f −1 (0) we have d(x0 , f −1 (0)) > 0. Every x ∈ L can be uniquely written as x = ax0 + z

May 2, 2018 11:28

378

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 378

Mathematical Analysis for Machine Learning and Data Mining

with z in the subspace f −1 (0) and x = d(x, 0L ) (because d is induced by the norm)  d(x, f −1 (0)) (because 0L ∈ f −1 (0)) = |a|d(x0 , f −1 (0)) and f (x) = a, hence |f (x)|  When f is continuous we have f  = sup x =0L

=

1 x, d(x0 ,f −1 (0)

hence f is continuous.

1 |f (x)| = x inf x | f (x) =0

x |f (x)|

1 1 . = inf x | f (x)=1 d(0L , Mf )

(17) Let (L,  · ) be a normed real linear space. If M is a closed proper linear subspace of L and x0 ∈ L − M , prove that: (a) there exists a linear functional f with f  = 1 such that f (y) = 0 for every y ∈ M and f (x0 ) = d(x0 , M ); (b) for each x0 ∈ L − {0L } there exists a linear functional f with f  = 1 and f (x0 ) = x0 ; (c) if U is a closed subspace of L such that every linear functional g such that g(u) = 0 for every u ∈ U is 0, then U is dense in L. Solution: Note that the linear subspace M1 generated by M and x0 consists of linear combinations of the form y + ax0 , where y ∈ M and a ∈ R. Define the linear functional f1 on M as f1 (y + ax0 ) = a. Clearly, f1 (y) = 0 for every y ∈ M and f1 (x0 ) = 1. Note that f1−1 (1) = {y + x0 | y ∈ M }. This implies that f1  = 1 . By Hahn-Banach theorem, f1 can be extended to a linear funcd(x0 ,M ) 1 , which shows that tional f˜ on the entire space L with f˜ = d(x0 ,M )

f = d(x0 , M )f˜ is the needed functional. The last two parts follow immediately from the first. (18) A subset S of Rn is midpoint convex if x, y ∈ S implies 12 (x + y) ∈ S. Prove that if S is closed and mid-point convex, then it is convex. Solution: Let x, y ∈ S and let z be a vector in the segment [x, y]. Construct the sequences of vectors (xn ) and (yn ) as x0 = x, y0 = y, xn+1 =

1 1 (xn + z), and yn+1 = (yn + z). 2 2

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Topological Linear Spaces

b3234-main

page 379

379

If zn = 12 (xn + yn ) for n  0 then zn − z  21n x − y. Therefore, limn→∞ zn = z. By mid-point convexity we have zi ∈ S and because S is closed, we have z ∈ S. Thus, C is indeed convex. (19) Prove that the set of rational numbers Q is mid-point convex but not convex. (20) If U is a compact subset of Rn , prove that its convex closure Kconv (U ) is a compact set. Solution: Let x ∈  Kconv (U ). By Carath´eodory’s Theorem (Theon+1 rem 3.25) we have x = i=1 ai ui , where ui ∈ U , ai  0 for 1  i  n+1, n+1 and i=1 ai = 1. (xp ) be a sequence in Kconv ). Each xp can be written as xp = Let (U n+1 n+1 j−1 cp,j u(p,j) , where cp,j  0, j=1 cp,j = 1, and u(p,j) ∈ U . ⎞ ⎛ Note that the vectors ⎝

cp,1 . . .

⎠ belong to a simplex in Rn+1 which

cp,n+1

is compact (by Corollary 6.11). Since U is compact too, there exists a sequence p1 , p2 , . . . in N such that (cpj ,1 ) and (upj ,n+1 ) are convergent subsequences of the sequences (cp,j ) and (up,j ). Suppose that limj→∞ cpj ,k = ck and limj→∞ upj ,k = uk for 1  k  n + 1. Therefore, the sequence (xpj ) is a convergent subsequence of (xp ), hence Kconv (U ) is compact. (21) Let C be a compact subset of Rn . Prove that 0n ∈ Kconv (C) if and only the set {h ∈ Rn | x h < 0} is empty. Solution: If 0n ∈ Kconv (C) then 0n is a convex combination n n 0n = i=1 ai xi for some xi ∈ C. Then, i=1 ai (xi , h) = 0 and this contradicts the fact that xi h < 0. This implies that {h ∈ Rn | x h < 0} is empty. Conversely, suppose that 0n ∈ Kconv (C). Since Kconv (C) is compact by Supplement 20, there exists h ∈ Rn such that x h < 0 for all x ∈ Kconv (C), hence x h < 0 for x ∈ C. (22) Show that Farkas’ Theorem (Theorem 6.61) follows from Supplement 26. Solution: Let A ∈ Rm×n be a matrix and let b ∈ Rn be a vector. Since C = {A y | y ∈ Rm , y  0m } is a closed convex cone in Rn , by Supplement 26 we have C = (C ∗ )∗ . By Supplement 25 the polar cone C ∗ is given by C ∗ = {p ∈ Rn | Ap  0m }. Thus, we have c ∈ (C ∗ )∗ if and only if c x  0 for all x ∈ C ∗ , that is, Ax  0m implies c x  0. Therefore, A x  0m implies c x  0 is equivalent to the existence of x ∈ Rn such that c = Ax and x  0n .

May 2, 2018 11:28

380

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 380

Mathematical Analysis for Machine Learning and Data Mining

This allows us to conclude that A x  0m implies c x  0 is equivalent to the existence of x ∈ Rn such that c = Ax and x  0n , which is the statement of Theorem 6.61. (23) Let P ⊆ Rn be a polytope and let {Fi | 1  m  m} be the set of its faces. Prove that if Hwi ,ai is a hyperplane  that supports P such that > for 1  i  m, then P = m Fi = P ∩ Hw i=1 Hwi ,ai . i ,ai Let T be a subset of Rn . Define the set T ∗ as  {p ∈ Rn | p x  0 for all x ∈ T } T∗ = n R

if T = ∅, if T = ∅.

(24) Prove that for every subsets T, T1 , T2 of Rn we have: (a) T ∗ is a closed convex cone with vertex 0n for every T ⊆ Rn . The set T ∗ is known as the polar cone of T . (b) T1 ⊆ T2 implies (T2 )∗ ⊆ (T1 )∗ ; (c) T ⊆ (T ∗ )∗ . (25) Let A ∈ Rm×n and let C = {A y | y  0m }. Prove that C is a closed and convex cone and that C ∗ = {x ∈ Rn | Ax  0m }. Solution: We leave it to the reader to show that C is a closed and convex cone. If z ∈ C ∗ we have z A y  0, or y Az  0 for every y  0. This is equivalent to Az  0m , which gives the desired equality for C ∗ . (26) Prove that if C is a convex and closed cone in Rn having the vertex 0n , then C = (C ∗ )∗ . Solution: By Part (b) of Exercise 24 we have C ⊆ (C ∗ )∗ ; this leaves us to prove the reverse inclusion. Suppose that x ∈ (C ∗ )∗ but x ∈ C. By the Separation Theorem (Theorem 6.55) there exists w ∈ Rn − {0n } such that w y  a for every y ∈ C and w x > a. Since 0n ∈ C, it follows that 0  a, that is, that a is a non-negative number, so w x > 0. ˜ ∈C We claim that w ∈ C ∗ . If this is not the case, there exists a y ˜ > 0. Therefore, choosing k an arbitrarily large number, such that w y y) can be made arbitrarily large, which contradicts the the number w (k˜ fact that w y  a for all y ∈ C. Therefore, we have w ∈ C ∗ . Since

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Topological Linear Spaces

page 381

381

x ∈ (C ∗ )∗ , w x  0, which contradicts the fact that w x > 0. Thus, x ∈ C. (27) Prove that if C is an open set in Rn , then Kconv (C) is open. (28) Let U be a subset of Rn . Prove that: (a) We have Kconv (K(U )) ⊆ K(Kconv (U )); (b) if U is bounded, then Kconv (K(U )) = K(Kconv (U )); (c) if U is both bounded and closed, then Kconv(U ) is a closed set. (29) Give an example of a non-bounded and closed subset U of Rn such that Kconv (U ) is not closed. (30) Let C be a convex subset of Rn . Prove that: (a) RI(C) is a convex subset of Rn ; (b) K(RI(C)) = K(C); (c) RI(K(C)) = RI(C) = RI(RI(C)). (31) Let C be a convex subset of Rn . Prove that Kaff (C) = Kaff (K(C)) = Kaff (RI(C)).

Bibliographical Comments The standard reference for topological linear space is [90]. The books [111, 23, 15, 18, 99] contain a vast amount of results in convexity theory. References that focus on geometric aspects are [69, 115]. We followed [85] in the presentation of contingent cones.

PART III

Measure and Integration

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 385

Chapter 7

Measurable Spaces and Measures

7.1

Introduction

Measure theory is an area of mathematical analysis that seeks to formalize a generalization of the notion of body volume. The relevant mathematical structures (algebras and σ-algebras of sets) have been presented in Chapter 1. In this chapter we discuss measurable spaces, Borel sets in topological spaces and several classes of functions that are naturally related to the notion of measure. Then, we focus on outer measures, on Lebesgue measures on Rn , and on signed and complex measures. The chapter concludes with some applications of measure theory to probabilities.

7.2

Measurable Spaces

Definition 7.1. A measurable space is a pair (S, E), where S is a non-empty set and E is a σ-algebra of subsets of S. The sets that belong to E are referred to as measurable sets. For a measurable space (S, E) and a set T ∈ E we denote E[T ] = {U ∈ E | U ⊆ T }. It is immediate to verify that E[T ] is a σ-algebra for every T ∈ E. If (S, E) is a measurable space then Eσ ⊆ E. Moreover, by part (ii) of Theorem 1.12, we also have Eδ ⊆ E. Definition 7.2. Let (S, E) be a measurable space. An E-partition (or a measurable partition of (S, E)) is a partition π ∈ PART(S) such that B ∈ E for every B ∈ π. 385

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 386

Mathematical Analysis for Machine Learning and Data Mining

386

The set of finite E-partitions of a set T ∈ E is denoted by PARTE (T ). If π is a E-measurable partition, then Eπ ⊆ E. Definition 7.3. Let {(Si , Ei ) | i ∈ I} be a family of measurable spaces  and let pi : i∈I Si −→ Si be the ith projection.  The product σ-algebra on the set i∈I Si is the collection of subsets of  form p−1 i (Ei ), where Ei ∈ Ei for i ∈ I. i∈I Si generated by the sets of the  This σ-algebra will be denoted by i∈I Ei .   The pair ( i∈I Si , i∈I Ei ) is the product measurable space of the family {(Si , Ei ) | i ∈ I} of measurable spaces. Theorem 7.1. Let I be a countable set and let {(Si , Ei ) | i ∈ I} be a countable family of measurable spaces indexed by I. Then, the product σ  algebra i∈I Ei is generated by the sets of the form i∈I {Ei ∈ Ei | i ∈ I}.  Proof. If Ei ∈ Ei we have p−1 i (Ei ) = j∈I Cj , where  Sj if j = i, Cj = Ei if j = i. Therefore, we have

+ i∈I

Ei =



p−1 i (Ei )

i∈I

and the theorem follows because the previous equality involves a countable intersection.  Theorem 7.2. Let {(Si , Ei ) | i ∈ I} be a family of measurable spaces indexed by the set I, where each σ-algebra Ei is generated by a collection Ci , that is, Ei = Kσ-alg (Ci ). Then, we have +  Kσ-alg {p−1 Ei . i (Ci ) | Ci ∈ Ci , i ∈ I} = i∈I

Proof.

Let C be the collection of subsets of

 i∈I

Si defined by

C = {p−1 i (Ci ) | Ci ∈ Ci , i ∈ I}.  It is clear that Kσ-alg (C) ⊆ i∈I Ei . For each i ∈ I the collection {C | C ∈ Ei and p−1 i (C) ∈ Kσ-alg (C)} is a σ-algebra on Si that contains Ci and, therefore Ei . In other words, p−1 i (C) ∈   Kσ-alg (C) for all C ∈ Ei , i ∈ I, and, therefore i∈I Ei ⊆ Kσ-alg (C).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 387

387

Corollary 7.1. Let {(Si , Ei ) | i ∈ I} be a family of measurable spaces indexed by the set I, where each σ-algebra Ei is generated by a collection Ci . If I is countable and Si ∈ Ci for i ∈ I, then  Kσ-alg

+

 Ci | Ci ∈ Ci , i ∈ I

+

=

i∈I

Proof.

Ei .

i∈I

The argument is based on the observation that +  {Ci ∈ Ci , i ∈ I} = p−1 i (Ci ) i∈I

i∈I



and on the fact that I is countable.

Definition 7.4. Let S, T be two sets and let U be a subset of S and V be a subset of T . The set U × V is a rectangle on S × T . Example 7.1. If (S, E), (T, E ) are two measurable spaces. Denote by p1 : S × T −→ S, p2 : S × T −→ T the projection function. Then, by Definition 7.3, the product σ-algebra on S × T , denoted here by E × E , is −1 generated by sets of the form p−1 1 (U ) = U × T and p2 (V ) = S × V . By Theorem 7.1, the same σ-algebra is generated by rectangles of the form U × V , where U ∈ E and V ∈ E . Theorem 7.3. Let f : S −→ T be a function. If E is a σ-algebra on the set T , then {f −1 (V ) | V ∈ E } is a σ-algebra on S. Proof. Suppose that U = f −1 (V ) for some V ∈ E . Since S − U = f −1 (T − V ) by Theorem 1.6, the first condition of the definition of σalgebras is satisfied. If {Ai | i ∈ N} is a countable collection of subsets of S such that Ai = f −1 (Vi ) where Vi ∈ E , then  i∈N

Ai =

 i∈N

⎛ f −1 (Vi ) = f −1 ⎝



⎞ Vi ⎠ .

i∈N

 Since i∈N Vi ∈ E , the second condition of the definition of σ-algebras is satisfied. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

b3234-main

page 388

Mathematical Analysis for Machine Learning and Data Mining

388

7.3

9in x 6in

Borel Sets

Definition 7.5. Let (S, O) be a topological space. The Borel sets of (S, O) are the sets in Kσ-alg (O). The σ-algebra of Borel sets Kσ-alg (O) is denoted by B(S, O), or just by B(S) if the topology O is clear from context. Theorem 7.4. Let (S, O) be a topological space. We have  (i) if U0 , U1 , . . . , Un , . . . are open subsets of S, then n∈N Un is a Borel set; (ii) all closed sets are Borel sets;  (iii) if V0 , V1 , . . . , Vn , . . . are closed sets, then n∈N Vn is a Borel set. Proof.

These statements follow immediately from Definition 7.5.



Example 7.2. We identify several families of Borel subsets of the topological space (R, O). Every open interval (a, b) and every set (a, ∞) or (−∞, a) is a Borel set for a, b ∈ R, because they are open sets. The closed intervals of the form [a, b] are Borel sets because they are closed sets in the topological space. Since [a, b) = (−∞, b) − (−∞, a) it follows that the half-open intervals of this form are also Borel sets. For every a ∈ R we have {a} ∈ B(R, O) because {a} = [a, b) − (a, b) for every b ∈ R such that b > a. Therefore, every countable subset {an | n ∈ N} of R is a Borel set. Theorem 7.5. The Borel σ-algebra B(R, O) equals the σ-algebra generated by any of the following families of subsets of R: (i) all open intervals (a, b), where a < b; (ii) all closed intervals [a, b], where a  b; (iii) all half-open intervals [a, b), where a < b; (iv) all rays of the form (a, ∞); (v) all rays of the form (−∞, b); (vi) all rays of the form [a, ∞); (vii) all rays of the form (−∞, b]. Proof. Since every open set in R is a countable union of open intervals it follows that the family of Borel sets of R coincides with the σ-algebra B(R, O) generated by the family of open intervals (a, b), where a < b. Let B be the Borel σ-algebra generated by the family of half-open intervals [a, b), where a < b. Note that a half-open interval [a, b) can be

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

written as [a, b) =

∞ 

page 389

389

(a − 1/n, b),

n=1

which implies that [a, b) ∈ B for all a, b ∈ R with a < b. Therefore, B ⊆ B. Conversely, observe that ∞  [a + 1/n, b), (a, b) = n=1

which implies that (a, b) ∈ B and, therefore, B ⊆ B . This allows us to conclude that B = B . We leave the proofs of the remaining parts to the reader.  The next theorem shows that the Borel sets of a subspace T of a topological space S are the intersections of the Borel sets of S with T . Theorem 7.6. Let T be a non-empty subset of a topological space (S, O) equipped with the topology OT = O T . We have B(T, OT ) = {U ∩ T | U ∈ B(S, O)}. Proof.

By Corollary 1.3 we have B(T, OT ) = Kσ-alg (B(S, O) T ) = Kσ-alg (B(S, O)) T = B(S, O) T = {U ∩ T | U ∈ B(S, O)}.



Corollary 7.2. The collection of Borel subsets of R is given by B(R) = ˆ  , where R ˆ is equipped with the topology defined in Example 4.9. (B(R)) R Proof.

This is an immediate consequence of Theorem 7.6.



ˆ O) of Borel sets on the extended set of Theorem 7.7. The collection B(R, ˆ real numbers R equipped with the topology introduced in Example 4.9 equals ˜ generated by all intervals of the form (a, ∞], where a ∈ R. the σ-algebra E ˜ for a ∈ R ∪ ˆ − (a, ∞] = [−∞, a], we have [−∞, a] ∈ E Proof. Since R ˜ ˜ {−∞}. Thus, (a, ∞] ∩ [−∞, b] = (a, b] ∈ E, so E contains the σ-algebra ˜ contains {∞} = containing sets of the form (a, b], which is B(R, O). Also, E   ˜ n∈N (n, ∞] and {−∞} = n∈N [−∞, n]. This implies that E contains the ˜ ˆ smallest σ-algebra containing B(R, O) ∪ {−∞, ∞}, so B(R, O) ⊆ E.  ˆ O), the smallest σ-algebra Since (a, ∞] = n∈N (a, n) ∪ {∞} ∈ B(R, containing sets of the form (a, ∞] must be included in any other σ-algebra ˜ ⊆ B(R, ˜ = B(R, ˆ O). We conclude that E ˆ O).  containing these sets, so E

May 2, 2018 11:28

390

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 390

Mathematical Analysis for Machine Learning and Data Mining

ˆ O) is generated by sets of the form A similar argument shows that B(R, [a, ∞], [−∞, a], or [−∞, a), where a ∈ R. We can restrict a, b in the equivalent descriptions of the σ-algebra B(R) of Borel sets of R given in Theorem 7.5 to belong to the set Q of rational numbers. We show here, as an example, that B(R) equals the σ-algebra Kσ-alg (D), where D is the collection of the sets of the form (−∞, a] with a ∈ Q. Since each set (−∞, a] is closed and, therefore, Borelian, we have Kσ-alg (D) ⊆ B(R). To prove the reverse inclusion let (a, b) be an open interval in R and let an = a + n1 and bn = b − n1 for n  1. We have ∞ ∞   (an , bn ] = ((−∞, bn ] ∩ (−∞, an ]), (a, b) = n=1

n=1

which show that (a, b) ∈ Kσ-alg (D). This implies the reverse inclusion B(R) ⊆ Kσ-alg (D), so B(R) = Kσ-alg (D). Theorem 7.8. Let Cn be the closed sets of the topological space (Rn , O). We have B(Rn ) = Kσ-alg (O) = Kσ-alg (Cn ) = Kσ-alg (COMP(Rn , O)). Proof. Since compact subsets of Rn are closed, we have COMP(Rn , O) ⊆ Cn , hence Kσ-alg (COMP(Rn , O)) ⊆ Kσ-alg (Cn ). If C ∈ Cn and k ∈ N, the set Ck = C ∩ B[0n , k] is closed and  bounded, so Ck ∈ COMP(Rn , O). Since C = k∈N Ck , it follows that C ∈ Kσ-alg (COMP(Rn , O)), so Cn ⊆ Kσ-alg (COMP(Rn , O)), which, in turn yields Kσ-alg (Cn ) ⊆ Kσ-alg (COMP(Rn , O)). Thus, Kσ-alg (Cn ) = Kσ-alg (COMP(Rn , O)). Since Kσ-alg (O) = Kσ-alg (Cn ), the statement of the theorem follows.  Let a, b ∈ Rn . The closed interval of Rn determined by a, b is the set K = [a1 , b1 ] × · · · × [an , bn ]. Similarly, the open interval, open-closed, and closed-open intervals of Rn are the sets: I = (a1 , b1 ) × · · · × (an , bn ), G = (a1 , b1 ] × · · · × (an , bn ], H = [a1 , b1 ) × · · · × [an , bn ), respectively. It is immediate to verify that K, I, G and H are π-systems. An analogue of Theorem 7.5 for Rn is given next: Theorem 7.9. The σ-algebra of Borel subsets of (Rn , O) is generated by each of the following collections of sets:

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

(i) (ii) (iii) (iv)

page 391

391

the collection I of all open intervals of Rn ; the collection H of all closed-open intervals of Rn ; the collection Ir of all open intervals of Rn with rational endpoints; the collection Hr of all closed-open intervals of Rn with rational endpoints.

Proof. Since Ir ⊆ I ⊆ O, we have Kσ-alg (Ir ) ⊆ Kσ-alg (I) ⊆ Kσ-alg (O) = B(Rn ). Let U be an open set in Rn . We claim that U=

 {I ∈ Ir | I ⊆ U }.

 It is clear that {I ∈ Ir | I ⊆ U } ⊆ U . To prove the converse inclusion let x ∈ U . There exists a cube with rational endpoints that contains x and is included in U . Since the set of all such cubes is countable, we have U ⊆ Kσ-alg (Ir ), so Kσ-alg (O) ⊆ Kσ-alg (Ir ). Thus, Kσ-alg (Ir ) = Kσ-alg (I) = Kσ-alg (O) = B(Rn ). If H ∈ H we can write     1 1 H = [a1 , b1 ) × · · · × [an , bn ) = a1 − , b 1 × · · · × an − , b n , k k k1

which implies H ⊆ Kσ-alg (I); in particular, this also implies Hr ⊆ Kσ-alg (Ir ). These inclusions imply Kσ-alg (H) ⊆ Kσ-alg (I) and Kσ-alg (Hr ) ⊆ Kσ-alg (Ir ). Every I ∈ I can be written as: I = (a1 , b1 ) × · · · × (an , bn ) =

 k1

1 a1 + , d1 k



 1 × · · · × an + , dn , k

which yields I ⊆ Kσ-alg (H), and also, Ir ⊆ Kσ-alg (Hr ). Therefore, Kσ-alg (I) ⊆ Kσ-alg (H) and Kσ-alg (Ir ) ⊆ Kσ-alg (Hr ). This allows us to conclude that Kσ-alg (I) = Kσ-alg (H) = Kσ-alg (Ir ) = Kσ-alg (Hr ) = B(Rn ).



May 2, 2018 11:28

Mathematical Analysis for Machine Learning

b3234-main

page 392

Mathematical Analysis for Machine Learning and Data Mining

392

7.4

9in x 6in

Measurable Functions

The notion of measurable functions is similar to the notion of continuous function is topology and is used in the definition of the Lebesgue integral that we present in the next chapter. We begin with a study of simple functions as a first step in the construction of the Legesgue integral. Definition 7.6. A simple function on a set S is a function f : S −→ R that has a finite range. The set of simple functions on the set S is denoted by SF(S); also, the set of non-negative simple functions on the same set is denoted by SF+ (S). Simple functions are linear combinations of characteristic functions. Theorem 7.10. Let f ∈ SF(S) such that Ran(f ) = {y1 , . . . , yn } ⊆ R.  Then, f = ni=1 yi 1f −1 (yi ) . Let x ∈ S. If f (x) = yj , then  1 if = j, 1f −1 (y ) (x) = 0 otherwise. n = yj , which shows that f (x) Thus, i=1 yi 1f −1 (yi ) (x) n −1 (x). y 1 i=1 i f (yi ) Proof.

= 

Theorem 7.11. Let f1 , . . . , fk be k simple functions defined on a set S. If g : Rk −→ R is an arbitrary function, then g(f1 , . . . , fk ) is a simple function on S and we have: mk m1   ··· g(y1p1 , . . . , ykpk )1f −1 (y1p )∩···∩f −1 (ykp ) (x) g(f1 , . . . , fk )(x) = p1 =1

1

pk =1

k

1

k

for every x ∈ S, where Ran(fi ) = {yi1 , . . . , yimi } for 1  i  k. Proof. The function g(f1 , . . . , fk ) is a simple function because it has a finite range. Moreover, if Ran(fi ) = {yi1 , . . . , yimi }, then the values of g(f1 , . . . , fk ) have the form g(y1p1 , . . . , ykpk ) and g(f1 , . . . , fk ) can be written as: g(f1 , . . . , fk )(x) m1 mk   = ··· g(y1p1 , . . . , ykpk )1f −1 (y1p ) (x) · · · 1f −1 (ykp ) (x) =

p1 =1 m1  p1 =1

for x ∈ S.

pk =1

···



1

k

1

pk = 1mk g(y1p1 , . . . , ykpk )1f −1 (y1p 1

k

−1 1 )∩···∩fk (ykpk )

(x) 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 393

393

Corollary 7.3. If fi ∈ SF(S) for 1  i  k then max{f1 (x), . . . , fk (x)}, min{f1 (x), . . . , fk (x)}, f1 (x) + · · · + fk (x), f1 (x) · . . . · fk (x) belong to SF(S). Proof.

The statement follows immediately from Theorem 7.11.



Definition 7.7. Let f : S −→ R be a simple function such that Ran(f ) = {y1 , . . . , yn }. A finite partition of S is related to f if for every block B of π there exists y ∈ Ran(f ) such that f (x) = y for every x ∈ B. It is easy to see that the largest partition in (PART(S), ) related to a simple f whose range is {y1 , . . . , yn } is {f −1 (y1 ), . . . , f −1 (yn )}. Definition 7.8. Let (S, D) and (T, E) be two measurable spaces. A function f : S −→ T is said to be measurable if f −1 (V ) ∈ D for every V ∈ E. Example 7.3. Every constant function f : S −→ T between two measurable spaces (S, D) and (T, E) is measurable because f −1 (V ) is either ∅ or S for every V ∈ E. Lemma 7.1. Let (S, E) be a measurable space and let f : S −→ S  be a function. The collection E1 = {W ∈ P(S  ) | f −1 (W ) ∈ E} is a σ-algebra on S  . Proof. If f −1 (W ) ∈ E, then f −1 (S  − W ) = f −1 (S  ) − f −1 (W ) = S − f −1 (W ) ∈ E, hence S  − W ∈ E1 . Let {Wn | n ∈ N} be a countable collection of subsets of S  such that   −1 −1 (Wn ) ∈ E, it f (Wn ) ∈ E for n ∈ N. Since f −1 n∈N Wn = n∈N f   follows that n∈N Wn ∈ E1 , so E1 is indeed a σ-algebra on S  . Theorem 7.12. Let (S, E) and (S  , E ) be two measurable spaces and let C a collection of generators for E . A function f : S −→ S  is measurable if and only if f −1 (C  ) ∈ E for every C  ∈ C . Proof. The necessity of the condition is immediate. Suppose now that f −1 (C  ) ∈ E for every C  ∈ C , where Kσ-alg (C ) = E . We have C ⊆ E1 , where E1 = {W ∈ P(S  ) | f −1 (W ) ∈ E} is a σalgebra by Lemma 7.1. Therefore, E = Kσ-alg (C ) ⊆ Kσ-alg (E1 ) = E1 , and this means that the pre-images of each set in E under f belong to E. Therefore, f is measurable. 

May 2, 2018 11:28

394

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 394

Mathematical Analysis for Machine Learning and Data Mining

Given a measurable space (S, E) and a function f : S −→ S  , E1 is the least σ-algebra that can be defined on S  such that the function f : S −→ S  is measurable. Theorem 7.13. Let (S, O), (S  , O ) be two topological spaces and let f : S −→ S  be a continuous function. Then, f is measurable relative to the measurable spaces (S, B(S)) and (S  , B(S  )), where B(S) and B(S  ) are the collection of Borel sets in (S, O) and (S  , O ), respectively. Proof. By Lemma 7.1 the collection of sets E1 = {W ∈ P(S  ) | f −1 (W ) ∈ B(S)} is a σ-algebra on S  . Since f is continuous, E1 contains every open set in O , so the σ-algebra of Borel sets B(T ) that is generated by O is contained in E1 . Thus, for every Borel set U in S  , f −1 (U ) ∈ B(S), which implies that f is indeed measurable.  We refer to a measurable function between (S, B(S)) and (T, B(T )) as a Borel measurable function. Thus, Theorem 7.13 establishes that a continuous function between two topological spaces is a Borel measurable function. Example 7.4. Not every measurable function is continuous. Indeed, let f : R −→ R be the function defined as  1 if 0  x  1, f (x) = 0 otherwise. It is clear that f is a measurable function between (R, B(R)) and itself; however f is not continuous. We focus now on functions defined on a measurable space and ranging ˆ the extended set of reals (equipped with the σ-algebra of Borel over R, ˆ We will refer to measurable functions between the measurable subsets of R). ˆ B(R) ˆ as measurable functions. spaces (S, E) and (R, ˆ be Theorem 7.14. Let (S, E) be a measurable space and let f, g : S −→ R two measurable functions. If U ∈ E, then the sets {x ∈ U | f (x) < g(y}, {x ∈ U | f (x)  g(y}, and {x ∈ U | f (x) = g(y} belong to E. Proof. We have f (x) < g(x) if and only if there exists a rational number r such that f (x) < r < g(x). This allows us to write: {x ∈ U | f (x) < g(x)}  ({x ∈ U | f (x) < r} ∩ {x ∈ U | r < g(x)}) . = r∈Q

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 395

395

Since f and g are measurable, we have {x ∈ U | f (x) < r} ∈ E and {x ∈ U | r < g(x)} ∈ E, so {x ∈ U | f (x) < r} ∩ {x ∈ U | r < g(x)} ∈ E, which implies that {x ∈ U | f (x) < g(x)} ∈ E as a countable union of sets in E. Obviously, we also have {x ∈ U | g(x) < f (x)} belongs to E. Since {x ∈ U | f (x)  g(x)} = U − {x ∈ U | g(x) < f (x)}, it follows that {x ∈ U | f (x)  g(x)} ∈ E. The equality {x ∈ U | f (x) = g(x)} = {x ∈ U | f (x)  g(x)} − {x ∈ U | f (x) < g(x)} implies that {x ∈ U | f (x) = g(x)} ∈ E.



Theorem 7.15. Let (S, E) be a measurable space and let f = (fn ) be a seˆ B(R)). ˆ quence of measurable functions between (S, E) and (R, The functions hn = sup{f1 , . . . , fn } and gn = inf{f1 , . . . , fn } are measurable. Also, the functions p and q defined by p(x) = sup{fi (x) | i  1} and q(x) = inf{fi (x) | i  1} are measurable, as well as the functions l, u given by u(x) = lim sup fn (x) and l(x) = lim inf fn (x) for x ∈ S. Proof.

Since {x ∈ S | hn (x) > t} = {x ∈ S | gn (x) < t} =

n 

{x ∈ S | fi (x) > t},

i=1 n 

{x ∈ S | fi (x) < t},

i=1

the measurability of fi for 1  i  n implies that hn and gn are measurable. For the functions p and q we have ∞  {x ∈ S | fi (x) > t}, {x ∈ S | p(x) > t} = {x ∈ S | q(x) < t} =

i=1 ∞ 

{x ∈ S | fi (x) < t},

i=1

which prove that p and q are measurable. Finally, note that u(x) = inf n supkn fk (x) and l(x) = supn inf kn fk (x) for x ∈ S. The measurability of u and k follows from previous parts of this theorem.  Corollary 7.4. Let (S, E) be a measurable space and let f = (fn ) be a sequence of measurable functions between (S, E) and (R, B(R)). If the sequence f converges pointwise to a function f , then f is measurable.

May 2, 2018 11:28

396

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 396

Mathematical Analysis for Machine Learning and Data Mining

Proof. If limn→∞ fn (x) = f (x), then f (x) = lim sup fn (x) = lim inf fn (x) for x ∈ S and the measurability of f follows immediately from Theorem 7.14.  Lemma 7.2. Let (S, E) be a measurable space. The following statements ˆ are equivalent: that concern a function f : S −→ R −1 (i) for each t ∈ R the set f ([−∞, t]) ∈ E; (ii) for each t ∈ R the set f −1 ([−∞, t)) ∈ E; (iii) for each t ∈ R the set f −1 ([t, ∞]) ∈ E; (iv) for each t ∈ R the set f −1 ((t, ∞]) ∈ E. Proof.

Since f −1 ([−∞, t)) =



f −1 ((−∞, t − 1/n]),

n∈N

it follows that (i) implies (ii). Note that f −1 ([t, ∞]) = S − f −1 ([−∞, t)), which shows that (ii) implies (iii). The fact that (iii) implies (iv) follows from the fact that  f −1 ([t + 1/n, ∞]). f −1 ((t, ∞]) = n∈N

Finally, (iv) implies (i) because f −1 ([−∞, t]) = S − f −1 ((t, ∞]).



Theorem 7.16. Let (S, E) be a measurable space. The following statements ˆ are equivalent: that concern a function f : S −→ R (i) any of the equivalent conditions of Lemma 7.2 hold for f ; ˆ f −1 (B) ∈ E; (ii) for each Borel subset B of R, ˆ (iii) for each open subset U of R, f −1 (U ) ∈ E; ˆ f −1 (V ) ∈ E. (iv) for each closed subset V of R, Proof. (i) implies (ii): Suppose that f satisfies any of the equivaˆ is generated by lent conditions of Lemma 7.2. Since the Borel algebra B(R) −1 the sets of the form (t, ∞] it follows that f (B) ∈ E for each Borel subset ˆ B of R. ˆ and any closed (ii) implies (iii) and (iv): Any open subset of R ˆ are members of B(R), ˆ so the implications follow immediately. subset of R ˆ (iii) implies (i) because the set [−∞, t) is an open set in R. ˆ (iv) implies (i) because the set [−∞, t] is a closed set in R. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 397

397

ˆ Corollary 7.5. Let (S, E) be a measurable space. A function f : S −→ R ˆ B(R) ˆ if and is measurable (considered as a function between (S, E) and (R, only if it satisfies any of the equivalent conditions of Theorem 7.16. Proof.

This statement is follows immediately from Theorem 7.16.



Corollary 7.6. Let (S, E) be a measurable space. A simple function f : ˆ is measurable if for every y ∈ Ran(f ), f −1 (y) ∈ E. In particular, S −→ R for T ∈ E, the characteristic function 1T is a measurable simple function −1 because 1−1 T (1) = T and 1T (0) = S − T . Furthermore, f is a linear combination of characteristic functions of sets in E. Proof.

These statements follow from Theorems 7.10 and 7.16.



It is easy to verify that if f is a measurable function between (S, E) and (T, E ) and g is a measurable function between (T, E ) and (U, E ), then gf is a measurable function between (S, E) and (U, E ). Theorem 7.17. Let J be an interval of R and let f : J −→ R be a function that has a countable set of discontinuities of the first kind on J. Then, f is a measurable function between the measurable spaces (J, B(J)) and (R, B(R)). Proof. It suffices to show that for every c ∈ R, the set f −1 (−∞, c) is a Borel set. Let U be the open set U = I(f −1 (−∞, c)); since U is open, it is B(J)-measurable. Define V as V = f −1 (−∞, c) − U = f −1 (−∞, c) − I(f −1 (−∞, c)). We shall prove that every x ∈ V is a discontinuity point of the first kind for f . Since x ∈ V , it follows that x ∈ U = I(f −1 (−∞, c)). Note that x ∈ V implies f (x) < c. By Theorem 5.4, for every r > 0, there exists y ∈ B(x, r) such that y ∈ f −1 (−∞, c) (which amounts to f (y)  c). Since f (x) < c, x is a discontinuity point of the first kind for f . Therefore, V is at most countable because f has a countable set of discontinuities of the first kind. Thus, V is a Borel set. Since f −1 (−∞, c) = U ∪ V it follows that f −1 (−∞, c) is a Borel set.  Corollary 7.7. Let J be an interval of R and let f : J −→ R be a monotonic function. Then, f is a measurable function between the measurable spaces (J, B(J)) and (R, B(R)). Proof.

This follows from Theorems 4.61 and 7.17.



The notion of measurability can be extended to complex-valued functions.

May 2, 2018 11:28

398

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 398

Mathematical Analysis for Machine Learning and Data Mining

Definition 7.9. Let (S, E) be a measurable space and let f : S −→ C be a complex-valued function, where f (x) = u(x) + iv(x) for x ∈ S, where u and v are real-valued functions. The function f is measurable if and only if both u and v are measurable.

7.5

Measures and Measure Spaces

Definition 7.10. Let E be a σ-algebra of subsets of a set S. A measure is ˆ 0 that satisfies the following conditions: a function m : E −→ R (1) m(∅) = 0; (2) for every countable collection {U0 , U1 , . . .} of sets in E that are  pairwise disjoint such that n∈N Un ∈ E we have: ⎛ ⎞   m⎝ Un ⎠ = m(Un ) n∈N

n∈N

(the countable additivity property). If the σ-algebra E is replaced by an algebra and the countable additive property by simple additivity, then we obtain the notion of premeasure. Note that a premeasure on an algebra E is supposed to satisfy the same conditions as a measure; however, in the case of a premeasure, the countable  union n∈N may not belong to E. A measure space is a triple (S, E, m), where (S, E) is a measurable space and m is a measure. Example 7.5. If (S, E) is a measurable space the mapping m : E −→ R given by m(U ) = 0 for every U ∈ E is a measure. Example 7.6. Let S be a set and let s be a fixed element of S. Define the ˆ 0 by mapping δs : P(S) −→ R  1 if s ∈ U δs (U ) = = 1U (s), 0 otherwise for U ∈ P(S). It is easy to verify that δs is a measure defined on P(S). Indeed, we have δs (∅) = 0. If U0 , U1 , . . . is a countable collection of pairwise disjoint

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 399

Measurable Spaces and Measures

399

sets, then s may belong to at most one of these If there is a set Ui sets.    such that s ∈ Ui , s ∈ n∈NUi , so δs n∈N Un = n∈N δs (Un ) = 1. If no   such set Ui exists, then δs n∈N Un = n∈N δs (Un ) = 0. In either case, the countable additivity property is satisfied. The measure δs is known as the Dirac measure concentrated as s. Example 7.7. Let S be a finite set and let E = P(S). The mapping m : P(S) −→ R given by m(U ) = |U | is a measure on P(S), as can be  verified immediately. Note that m(T ) = {δu (T ) | u ∈ S}. This measure can be extended to arbitrary sets S by defining  |T | if T is finite, m(T ) = ∞ otherwise, for T ∈ P(S). We refer to m as the counting measure on P(S). Theorem 7.18. Let (S, E, m) be a measure space, (T, E ) be a measurable space and let f : S −→ T be a measurable function. The mapping m : E −→ R defined by m (V ) = m(f −1 (V )) for V ∈ E is a measure on (T, E ). Proof. If V = ∅, f −1 (∅) = ∅, so m (∅) = 0. If {Vn | n ∈ N} is a family of pairwise disjoint sets in E we have ⎛ ⎞ ⎛ ⎛ ⎞⎞   Un ⎠ = m ⎝f −1 ⎝ Un ⎠⎠ m ⎝ n∈N

⎛ = m⎝

n∈N

 n∈N

⎞

f −1 Un ⎠ =



m(f −1 (Un )) =

n∈N



m (Un ),

n∈N

which shows that m is indeed a measure on E .



The measure m introduced in Theorem 7.18 is the image measure of m under f . For a finite collection of n pairwise disjoint sets {U0 , U1 , . . . , Un−1 } in a measure space (S, E, m) we have the finite additivity property: m(U0 ∪ U1 ∪ · · · ∪ Un−1 ) =

n−1 

m(U ).

(7.1)

i=0

Observe that in a measure space (S, E, m), if U, V ∈ E and U ⊆ V , then V = U ∪ (V − U ), so by the finite additivity property,

May 2, 2018 11:28

400

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 400

Mathematical Analysis for Machine Learning and Data Mining

m(V ) = m(U ) + m(V − U )  m(U ). This shows that U ⊆ V implies m(U )  m(V ) (the monotonicity of measures). Definition 7.11. Let (S, E, m) be a measure space. If m(S) is finite (which implies that m(U ) is finite for all U ∈ E) we say that m is a finite measure.  If S equals a countable union of sets Un in E, that is, S = n∈N Un such that each m(Un ) is finite, then m is a σ-finite measure. The measure m is semi-finite if for each U ∈ E with m(U ) = ∞, there exists V ∈ E such that V ⊆ U and 0 < m(V ) < ∞.  If U ∈ E and S = n∈N Un such that each m(Un ) is finite, then U = n∈N (U ∩ Un ), where U ∩ Un ∈ E and m(U ∩ Un ) is finite. Every finite measure is clearly σ-finite. Also, every σ-finite measure is semi-finite. Indeed, let m be a σ-finite measure and let W ∈ E be a subset  of S such that m(W ) = ∞. Since m is σ-finite, W = n∈N Wn , where Wn ∈ E and m(Wn ) is finite for n ∈ N. There exists at least one set Wn such that m(Wn ) > 0 because, otherwise, we would have m(W ) = 0. This shows that m is semi-finite. 

Example 7.8. Let (S, P(S)) be a measure space and let f : S −→ [0, ∞]. Define mf : P(S) −→ [0, ∞] as  mf (U ) = sup {f (x) | x ∈ T }. T ⊆U,T finite Then mf is a semi-finite measure if and only if f (x) < ∞ for every x ∈ S. Furthermore, mf is σ-finite if and only if mf is semi-finite and the set {x | f (x) > 0} is countable. Observe that if f (x) = 1, then mf is the counting measure introduced in Example 7.7. Let (S, E, m) be a measure space and let X, Y ∈ E. Since X ∪ Y = X ∪ (Y − X), Y = (Y − X) ∪ (Y ∩ X) and the pairs of sets X, (Y − X) and (Y − X), (Y ∩ X) are disjoint, we can write m(X ∪ Y ) = m(X) + m(Y − X) = m(X) + m(Y ) − m(X ∩ Y ).

(7.2)

m(X ∪ Y ) + m(X ∩ Y ) = m(X) + m(Y )

(7.3)

The resulting equality:

for X, Y ∈ E is known as the modularity property of measures.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Measurable Spaces and Measures

b3234-main

page 401

401

If the sets Ui of a countable collection {U0 , U1 , . . .} of sets in E are not disjoint, then instead of the additivity property we have the subadditive inequality: ⎞ ⎛   Un ⎠  m(Un ). (7.4) m⎝ n∈N

n∈N

By Theorem 1.33 there exists a family of sets {Vi ∈ E | i ∈ N} ⊆ E that  are pairwise disjoint such that Vi ⊆ Ui for i ∈ N and {Vi | i ∈ N} =  {Ui | i ∈ N}. This implies m(Vi )  m(Ui ) and ⎛ ⎞ ⎛ ⎞   m⎝ Un ⎠ = m ⎝ Vn ⎠ n∈N

=

 n∈N

n∈N

m(Vn ) 



m(Un ),

n∈N

which gives the desired subadditivity property of measures. Definition 7.12. Let (S, E, m) be a measure space. A null set is a subset U of S such that U ∈ E and m(U ) = 0. A countable union of null sets in a measure space (S, E, m) is a null set due to the subadditivity of measures. Note that if U is a null set in (S, E, m) and V ⊆ U , it does not follow that V ∈ E. However, if V ∈ E, then, of course V is a null set. Definition 7.13. A property of the elements of a measure space (S, E, m) is a measurable subset R of S. If m(S − R) = 0 we say that the property R holds m-almost everywhere, or that R holds m-a.e. If m is clear from context we refer to R as an a.e. property. Definition 7.14. The measure space (S, E, m) is complete (and m is a complete measure) if W ∈ E, m(W ) = 0, U ⊆ W imply U ∈ E. Definition 7.15. Let (S, E, m) be a measure space. The completion of (S, E, m) is the collection Em of subsets of S that consists of those subsets T of S such that there exist U, V ∈ E such that U ⊆ T ⊆ V , and m(V −U ) = 0.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 402

Mathematical Analysis for Machine Learning and Data Mining

402

If U, V ∈ E are two sets in E that satisfy the conditions of Definition 7.15, we have m(U ) = m(V ). Moreover, if Z ⊆ T and Z ∈ E, then m(Z)  m(U ) = m(V ). Therefore, sup{m(Z) | Z ∈ E and Z ⊆ T }  m(V ). Actually, we have the equality sup{m(Z) | Z ∈ E and Z ⊆ T } = m(V ) because m(U ) is one of the numbers that occurs in the set of the left side of the previous inequality. We infer that the value m(V ) = m(U ) depends only on the set T and the measure m. ˆ 0 These considerations allow us to define a function m ˜ : Em −→ R where m(T ˜ ) is the common value of m(U ) and m(V ), when these sets are introduced as above. The function m ˜ is the completion of m. Theorem 7.19. Let (S, E, m) be a measure space. The collection Em is a σ-algebra on S that includes E, m ˜ is a measure on Em that is complete, and m ˜ E = m. Proof. Let T ∈ E. By taking U = T and V = T , it follows that E ⊆ Em . Since U ⊆ T ⊆ V for some U, V ∈ E, we also have V ⊆ T ⊆ U and m(U − V ) = 0. Therefore, T ∈ Em . Suppose that (Tn ) is a sequence of sets in Em . Let Un and Vn be two sets in E such that Un ⊆ Tn ⊆ Vn and m(Vn − Un ) = 0 for n ∈ N. Then      Vn , and n∈N Un and n∈N Vn belong to E, n∈N Un ⊆ n∈N Tn ⊆ ⎛ ⎞ ⎞ n∈N ⎛    m⎝ Vn − Un ⎠  m ⎝ (Vn − Un )⎠ n∈N

n∈N





n∈N

m(Vn − Un ) = 0,

n∈N



Tn ∈ Em . Thus, Em is a σ-algebra on S that includes E. It is clear that m ˜ is an extension of m because, if T ∈ E, we obtain m(T ) = m(T ˜ ) by taking U = T = V . We need to prove that m ˜ is countable additive. Let (Tn ) be a sequence of pairwise disjoint sets and let Un , Vn be sets in E such that Un ⊆ Tn ⊆ Vn , and m(Vn − Un ) = 0. It is immediate that the sequence (Un ) also consists of disjoint sets. Therefore, ⎛ ⎞ ⎛ ⎞     m ˜⎝ Tn ⎠ = m ⎝ Un ⎠ = m(En ) = m(E ˜ n ), so

n∈N

n∈N

so m ˜ is a measure.

n∈N

n∈N

n∈N

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 403

403

To show that m ˜ is a complete measure let W ∈ Em be a subset of S such that m(W ˜ ) = 0 and let R ⊆ W . Since W ∈ Em , there exist U, V ∈ E, such that U ⊆ W ⊆ V and m(U ) = m(V ). It is clear that  m(U ) = m(W ) = m(V ) = 0, so W ∈ Em . ˆ Theorem 7.20. Let (S, E) be a complete measurable space. If f : S −→ R ˆ is a measurable function and f = g a.e., where g : S −→ R then g is a measurable function. Proof. Let D be the subset of S defined by D = {x ∈ S | f (x) = g(x)}. By hypothesis, we have m(D) = 0. Since {x ∈ S | g(x) > t} = ({x ∈ S | f (x) > t} ∪ {x ∈ D | g(x) > t}) − {x ∈ D | g(x)  t}, taking into account that {x ∈ S | f (x) > t} is measurable because f is measurable, it follows that the sets {x ∈ D | g(x) > t} and {x ∈ D | g(x)  t} are measurable as subsets of D. This implies the measurability of g.  Definition 7.16. Let (S, E1 , m1 ) and (S, E2 , m2 ) be two measure spaces on a set S. (S, E2 , m2 ) is an extension of (S, E1 , m1 ) if E1 ⊆ E2 and m2 E1 = m1 . The fact that (S, E2 , m2 ) is an extension of (S, E1 , m1 ) is denoted by (S, E1 , m1 )  (S, E2 , m2 ). Theorem 7.21. Let (S, E, m) be a measure space. There exists a unique measure space (S, E , m ) that is an extension of (S, E, m), is complete, and for any other complete extension (S, E , m ) then (S, E , m )  (S, E , m ). Proof.

The collection E defined as

E = {U ∪ L | U ∈ E, L ⊆ M for some M ∈ E with m(M ) = 0} is a σ-algebra. Note that ∅ ∈ E , so E is a non-empty collection. Suppose that T ∈ E , so T = U ∪ L, where U ∈ E and L ⊆ M for some M ∈ E with m(M ) = 0. We have: T = U ∩ L = U ∩ (M ∩ L) = U ∩ (M ∪ (M ∩ L) = (U ∩ M ) ∪ (M ∩ U ∩ L) = U ∪ M ∪ (M − (U ∪ L)), and, since U ∪ M ∈ E and (M − (U ∪ L)) ⊆ M , it follows that T ∈ E .

May 2, 2018 11:28

404

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 404

Mathematical Analysis for Machine Learning and Data Mining

Suppose now that T1 , T2 , . . . belong to E , that is, Tj = Uj ∪ Lj , where Uj ∈ E and Lj ⊆ Mj for some Mj ∈ E and m(Mj ) = 0.     This implies j1 Tj = j1 Uj ∪ j1 Mj , where j1 Uj ∈ E and     M . Since j1 Mj ∈ E and m( j1 Mj ) = 0, it folj1 Lj ⊆  j1 j  lows that j1 Tj ∈ E . This shows that E is indeed, a σ-algebra. For T ∈ E define m (T ) as m (T ) = m(U ), where T = U ∪ L, L ⊆ M for a set M ∈ E with m(M ) = 0. Then, m is well-defined. Indeed, suppose that T = U1 ∪ L1 such that L1 ⊆ M1 for a set M1 ∈ E with m(M1 ) = 0. Since T = U ∪ L = U1 ∪ L1 we have U ⊆ T = U1 ∪ L1 , so m(U )  m(u1 ) + m(L1 ) = m(U1 ); reversing the roles of U and U1 we have m(U1 )  m(U ), hence m(U1 ) = m(U ), which shows that m (T ) is well-defined. We claim that m is a measure on E . It is immediate that m (∅) = 0. Let {T0 , T1 , . . .} be a countable collection of pairwise disjoint sets in E . There exist U1 , U2 , . . . in E and L1 , L2 , . . . such that Tj = Uj ∪ Lj and Lj ⊆ Mj , where Mj ∈ E and m(Mj ) = 0 for j  1. Note that the sets Uj are also      pairwise disjoint. Since j1 Tj = j1 Uj ∪ j1 Lj , j1 Lj ⊆ j1 Mj ,  and m( j1 Mj ) = 0, it follows that ⎛ m ⎝

 j1

⎞

⎛

Tj ⎠ = m ⎝



j1

⎞ Uj ⎠ =

 j1

m(Uj ) =



m (Tj ).

j1

To prove that m is complete let T ∈ E with m (T ) = 0 and let Z be a subset of T . We have T = U ∪ L, where U ∈ E and L ⊆ M , where m(M ) = 0. Note that m(U ) = m (T ) = 0. For M  = U ∪ M we have m(M  )  m(U ) + m(M ) = 0. By taking Z = U  ∪ L , where U  = ∅ ∈ E and L = Z ⊆ M  we have Z ∈ E . It is immediate that E ⊆ E and that U ∈ E implies m(U ) = m (U ), so (S, E , m ) is indeed a complete extension of (S, E, m). This extension is unique for if (S, E , m ) is another complete extension of S, cale, m) and T ∈ E we have T = U ∪ L, where U ∈ E, L ⊆ M for some M ∈ E with m(M ) = 0. This implies U, M ∈ E and m (M ) = m(M ) = 0. Since m is complete, we obtain L ∈ E , hence T ∈ E . Also, m (U )  m (T )   m (U ) + m (L) = m (U ), hence m (T ) = m (U ) = m(U ) = m (T ). The smallest complete extension of a measure space (S, E, m) whose existence is proven in Theorem 7.21 is called the completion of (S, E, m).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 405

405

Lemma 7.3. Let (S, E, m) and (S, E, m ) be two measure spaces on S that have the same collection of measurable sets E and m(S) = m (S). The collection D = {C ∈ E | m(C) = m (C)} is a Dynkin system. Proof. By hypothesis, S ∈ D, so the first condition of Definition 1.45 is satisfied. Suppose that U, V ∈ D, that is, m(U ) = m (U ), m(V ) = m (V ) and U ⊆ V . Then, m(V − U ) = m(V ) − m(U ) = m (V ) − m (U ) = m (V − U ), hence V − U ∈ D. Finally, let T = (T0 , T1 , . . .) is an increasing sequence of subsets of S such that m(Ti ) = m (Ti ) for i ∈ N. We have ⎞ ⎞ ⎛ ⎛   m⎝ Ti ⎠ = lim m(Ti ) = lim m(Ti ) = m ⎝ Ti ⎠ , so



i∈N

i∈N Ti ∈ D.

n→∞

n→∞

i∈N



Theorem 7.22. Let (S, E, m) and (S, E, m ) be two measure spaces on S that have the same collection of measurable sets E, where E = Kσ-alg (C) for some π-system C of subsets of S. If m and m are finite measures such that (i) m(S) = m (S), and (ii) m(C) = m (C) for every C ∈ C, then m = m . Proof. Let D = {C ∈ E | m(C) = m (C)}. We prove the theorem by showing that E ⊆ D. By hypothesis, C ⊆ D, so KDyn (C) ⊆ KDyn (D) = D, taking into account that D is a Dynkin system (by Lemma 7.3). Since C is a π-system we have Kσ-alg (C) = KDyn (C) by Theorem 1.42. Since E = Kσ-alg (C), it follows that E ⊆ D.  Corollary 7.8. Let (S, E, m) and (S, E, m ) be two measure spaces on S that have the same collection of measurable sets E, where E = Kσ-alg (C) for some π-system C of subsets of S. If the following conditions: (i) m(C) = m (C) for C ∈ C; (ii) there exists a increasing sequence of sets in C, C = (Cn ), such that   n∈N Cn = S, where m(Ci ) and m (Ci ) are finite for i ∈ N,  then m = m .

May 2, 2018 11:28

406

Proof.

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 406

Mathematical Analysis for Machine Learning and Data Mining

For n ∈ N define the measures mn and mn as mn (T ) = m(T ∩ Cn ) and mn (T ) = m (T ∩ Cn )

for T ∈ E and n ∈ N. By Theorem 7.22 we have mn = mn . Therefore, m(T ) = lim mn (T ) = lim mn (T ) = m (T ) n→∞

n→∞

for T ∈ E, so m = m .



The modularity property of measures can be extended to the inclusionexclusion equality given next. Theorem 7.23. (The Inclusion-Exclusion Equality) Let m be a measure defined on a ring of subsets E of a set S. If U1 , . . . , Un are n members of E and n  2, then  m

n 

 Ui

=

i=1

n 

m(Ui ) −

i=1

+



m(Ui1 ∩ Ui2 )

i1 a. Therefore, x ∈ nk {x ∈ S | fn (x) > a}. Conversely, suppose that there exists m  k such that fm (x) > a. This implies immediately that (supnk fn )(x) > a, which justifies the first equality of the lemma. We omit the proof of the second equality.  Theorem 7.30. Let (S, E) be a measurable space and let f = (f0 , f1 , . . . , fn , . . .) be a sequence of real-valued functions, fi : S −→ R. If the functions fn are measurable, then so are the functions supn fn , inf n fn , lim supn→∞ fn and lim inf n→∞ fn . Proof. The conclusion follows immediately from Lemma 7.4 and from the equalities lim sup fn = inf sup fm n→∞

n1 mn

and lim inf fn = sup inf fm . n→∞

n1 mn



May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 414

Mathematical Analysis for Machine Learning and Data Mining

414

Corollary 7.15. Let (S, E) be a measurable space and let f = (f0 , f1 , . . . , fn , . . .) be a sequence of real-valued functions, fi : S −→ R such that limn→∞ fn (x) ˆ exists for x ∈ S. If the functions fn are measurable, then f : S −→ R defined by f (x) = limn→∞ fn (x) for x ∈ S is a measurable function. Proof.

This statement is an immediate consequence of Theorem 7.30. 

We extend the notion of measurability to functions with complex values. Definition 7.18. Let (S, O) be a measurable space. A function f : S −→ C is measurable if assuming that f (x) = u(x) + iv(x), u and v are both realvalued measurable functions. Note that √ if f : S −→ C is measurable, then |f | is also measurable because |f | = u2 + v 2 . Let (S, O) be a topological space and let f : S −→ C be a complexvalued function. Its support is the set supp(f ) = K({x ∈ S | f (x) = 0}). The set of continuous complex-valued functions whose support is compact is denoted by Cc (S). Note that Cc (S) is a linear space. Recall that a locally compact topological space is automatically a Hausdorff space (see Definition 4.26). Theorem 7.31. (Uniqueness of Measures Theorem) Let (S, E) be a measurable space such that E = Kσ-alg (G) such that (i) if G, H ∈ G, then G ∩ H ∈ G, and (ii) there exists a increasing sequence of sets (Gn ) in G such that  n∈N Gn = S. If m1 , m2 are two measures such that m1 (G) = m2 (G) for every G ∈ G and both m1 (Gn ) and m2 (Gn ) are finite for n ∈ N, then m1 (U ) = m2 (U ) for every U ∈ E. Proof.

For each of the sets Gn let Dn be the collection Dn = {U ∈ E | m1 (Gn ∩ U ) = m2 (Gn ∩ U )}.

We show that each Dn is a Dynkin system by proving that Dn satisfies the conditions of Theorem 1.41. Note that S ∈ Dn because m1 (G) = m2 (G) for each set G ∈ G. If U ∈ Dn , then m1 (Gn ∩ U ) = m1 (Gn ) − m1 (Gn ∩ U ) = m2 (Gn ) − m2 (Gn ∩ U ) = m2 (Gn ∩ U ), hence U ∈ Dn .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 415

415

Finally, let (Up ) be a sequence of pairwise disjoint sets in Dn . We have: ⎛ ⎞ ⎛ ⎞   m1 ⎝Gn ∩ U p ⎠ = m1 ⎝ (Gn ∩ Up )⎠ p∈N

=

p∈N



m1 (Gn ∩ Up ) =

p∈N



m2 (Gn ∩ Up )

p∈N

⎛



= m2 ⎝

⎛

⎞

(Gn ∩ Up )⎠ = m2 ⎝Gn ∩

p∈N



⎞ Up ⎠ ,

p∈N

 hence p∈N Up ∈ Dn . Since G is a π-system, it follows that KDyn (G) = Kσ-alg (G) by Theorem 1.42. Thus, G ⊆ Dn implies Kσ-alg (G) ⊆ Kσ-alg (G) ⊆ Dn for n ∈ N. Since E = Kσ-alg (G) ⊆ Dn ⊆ E, it follows that E = Dn for all n ∈ N, hence m1 (Gn ∩ U ) = m2 (Gn ∩ U ) for n ∈ N and U ∈ E. By the continuity property of measures we have: m1 (U ) = lim m1 (Gn ∩ U ) = lim m2 (Gn ∩ U ) = m2 (U ) n→∞

n→∞

for every U ∈ E which concludes the argument.



The behavior of measures with respect to limits of sequences of sets is discussed next. Theorem 7.32. (Measure Continuity Theorem) Let (S, E, m) be a measure space and let (U0 , U1 , . . .) be a sequence of sets. (i) if (U0 , U1 , . . .) is an increasing sequence, then m(lim Un ) = lim m(Un ) (continuity from below); (ii) if (U0 , U1 , . . .) is a decreasing sequence and there exists a set Ui such that m(Ui ) is finite, then m(lim Un ) = lim m(Un ) (continuity from above). Proof. Suppose that U0 ⊂ U1 ⊂ · · · is an increasing sequence, so  m(lim Un ) = m( n Un ). By Theorem 1.33 there exists a sequence V0 ⊂ V1 ⊂ · · · of disjoint sets   in E such that Un = Vn and V0 = U0 , and Vn = Un − Vn−1 for n  1. Then,     Vn = m(V0 ) + m(Vn ) m(lim Un ) = m n

n1



= lim

n→∞

m(V0 ) +

n  i=1



m(Vi )

May 2, 2018 11:28

416

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 416

Mathematical Analysis for Machine Learning and Data Mining





= lim m V0 ∪ n→∞

n 

 Vi

i=1

= lim m(Ui ). n→∞

Suppose now that U0 ⊃ U1 ⊃ · · · is a decreasing sequence of sets, so  m(lim Un ) = m( n Un ). Also, suppose that m(U0 ) is finite. Define the sequence of sets W0 , W1 , . . . by Wn = U0 − Un for n ∈ N.  Since this sequence is increasing, we have m( n∈N Wn ) = lim m(Wn ) by the first part of the theorem. Thus, we can write: ⎞ ⎛  Wn ⎠ = lim m(Wn ) = m(U0 ) − lim m(Un ). m⎝ n∈N

Since

⎛ m⎝



⎞

⎛

Wn ⎠ = m ⎝

n∈N

⎛



⎞ (U0 − Un )⎠

n∈N

= m ⎝U0 −



⎞ Un )⎠

n∈N

⎛

= m(U0 ) − m ⎝



⎞ Un )⎠ ,

n∈N

it follows that

⎛ m(lim Un ) = m ⎝



⎞ Un )⎠ = lim m(Un ),

n∈N

because U0 is finite.



Theorem 7.33. (Borel-Cantelli Lemma) Let (S, E, m) be a measure  space. If S = (S0 , S1 , . . .) is a sequence of sets such that i m(Si ) < ∞, then m(lim sup S) = 0. ∞ Proof. Let Tp = i=p Si for p ∈ N. By the subadditivity of m we have ∞ therefore limp→∞ m(Tp ) = 0 because of the m(Tp )  i=p m(Si ) and,  convergence of the series i m(Si ). ∞ ∞ ∞ = Since lim sup S = p=0 i=p Si p=0 Tp , it follows that m(lim sup S)  m(Tp ) for every p ∈ N, so m(lim sup S)  inf p m(Tp ) = 0, which implies m(lim sup S) = 0. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Measurable Spaces and Measures

7.6

b3234-main

page 417

417

Outer Measures

Definition 7.19. Let S be a set. An outer measure on S is a function ˆ 0 such that: μ : P(S) −→ R (i) μ(∅) = 0; (ii) μ is countably subadditive, that is, ⎞ ⎛   {μ(Un ) | n ∈ N} Un ⎠  μ⎝ n∈N

for every countable family {Un ∈ P(S) | n ∈ N} of subsets of S; (iii) μ is monotonic, that is, if U, V ⊆ S such that U ⊆ V , then μ(U )  μ(V ). Clearly, every measure on S is an outer measure on the same set. The reverse is false, as the next example shows. Example 7.10. Let S = {s0 , s1 , s2 } ˆ 0 by P(S) −→ R ⎧ ⎪ ⎪ ⎨0 μ(X) = 2 ⎪ ⎪ ⎩1

be a set. Define the function μ : if X = ∅, if X = S, otherwise.

For the disjoint sets X0 = {s0 } and X1 = {s1 } we have μ(X1 ∪ X2 ) = 1 and μ(X1 ) + μ(X2 ) = 2, so μ is not a measure. To show that μ is an outer measure we need to verify only is countable subadditivity for the other properties are immediate. Let {Un ∈ P(S) | n ∈ N} be a countable family of subsets of S. Three cases need to be  considered depending on the set U = n∈N Un : (i) If U = ∅, then Un = ∅ for every n ∈ N and the subadditivity is immediate. (ii) If U = S, then μ(U ) = 2. If there is a set Un such that Un = S, the subadditivity follows. If none of sets Un equals S, there must be least two non-empty distinct sets Uj and Uk among the members of the family {Un | n ∈ N}. For these sets μ(Uj ) = μ(Uk ) = 1 and the subadditivity follows. (iii) If U is neither ∅ nor S, then μ(U ) = 1. Clearly, at least one of the sets Un is non-empty which implies the subadditivity.

May 2, 2018 11:28

418

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 418

Mathematical Analysis for Machine Learning and Data Mining

Example 7.11. For U ⊆ R define the collection CU of sequences of open ∞ intervals of R of the form ((an , bn ))n1 such that U ⊆ n=1 (an , bn ). Let ˆ 0 be defined as: μL : P(R) −→ R ⎧ ⎫ ⎨ ⎬ (bn − an ) | ((an , bn ))n1 ∈ CU . μL (U ) = inf ⎩ ⎭ n1

We claim that μL is an outer measure on R. It is easy to see that μL (∅) = 0. If CU = ∅ define μL (U ) = ∞. Let U, V ⊆ R be such that U ⊆ V . Then CV ⊆ CU and this implies μL (U )  μL (V ). To prove the countable subadditivity of μL let (Un )n1 be a count able family of subsets of R. If n1 μL (Un ) = ∞, the subadditivity fol lows immediately. Suppose that n1 μL (Un ) is finite and let > 0. Let ((ajn , bjn ))n1 ∈ CUj and  (bjn − ajn ) < μL (Uj ) + j . 2 n1

If ((aj , bj ))j1 is a sequence constructed by amalgamating the sequences  ((ajn , bjn ))n1 (see Supplement 13 of Chapter 1), it follows that j1 Uj ⊆  j1 (aj , bj ) and     (bj − aj ) < μL (Uj ) + j 2 j1 j1  = μL (Uj ) + , j1

  hence μL ( j1 Uj )  j1 μL (Uj ). Thus, μL is indeed an outer measure. This measure is referred to as the Lebesgue outer measure on R. Example 7.12. Example 7.11 can be generalized to a collection C = {C1 , . . . , Cn , . . .} of subsets of a set S such that ∅ ∈ C, and a function ˆ 0 such that φ(∅) = 0. φ : C −→ R For U ∈ P(S) define ⎫ ⎧ ⎨  ⎬ μφ (U ) = inf φ(Cj ) | C1 , . . . , Cj , . . . ∈ C and U ⊆ Cj . ⎭ ⎩ j1

j1

If inf ∅ is defined as ∞ then μφ is an outer measure on S. Suppose that U ⊆ V ⊆ S. If there is no sequence (Cn ) in C such that  V ⊆ n1 Cn , then μφ (V ) = ∞, and we have the inequality μφ  μφ (V ).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 419

419

Otherwise, every cover of V is also a cover of U , which again, implies μφ (U )  μφ (V ).    If n1 μφ (Un ) = ∞, then μφ ( n1 Un )  n1 μφ (Un ). Therefore,  we may assume that n1 μφ (Un ) is finite, hence μφ (Un ) is finite for n  1. By the definition of μφ (Un ), for every > 0, there exist Cn1 , Cn2 , . . . in C   such that Un ⊆ p1 Cnp and p1 φ(Cnp )  μφ (Un ) + 2 n . This implies    n≥1 Un ⊆ n1 p1 Cnp , hence ⎛ ⎞ #   $  (μφ (Un ) + n < μφ ⎝ Un ⎠  φ(Cnp ) < μφ (Un ) + , 2 n1

n1 p1

n1

n1

  which implies μφ ( n1 Un )  n1 μφ (Un ). An outer measure on a set defines a class of subsets introduced next. ˆ 0 be an outer measure on S. Definition 7.20. Let μ : P(S) −→ R A subset T of S is μ-measurable if μ(H) = μ(H ∩ T ) + μ(H ∩ T¯) for every set H ∈ P(S). Lemma 7.5. Let S be a set and let μ be an outer measure on a set S. A set T is μ-measurable if and only if μ(H)  μ(H ∩ T ) + μ(H ∩ T¯ ) for every H ∈ P(S) such that μ(H) < ∞. Proof. The necessity of the condition is obvious. Suppose, therefore, that the condition is satisfied. Since μ is subadditive, we have μ(H)  μ(H ∩ T ) + μ(H ∩ T¯), which implies μ(H) = μ(H ∩ T ) + μ(H ∩ T¯ ).



ˆ 0 be an outer measure on S and let Example 7.13. Let μ : P(S) −→ R T be a subset of S such that μ(T ) = 0. The set T is μ-measurable. Indeed, let H ∈ P(S). Since H ∩ T ⊆ T we have μ(H ∩ T )  μ(T ) = 0, so μ(H ∩ T ) = 0. Also, H ∩ T¯ ⊆ H, hence μ(H ∩ T¯ )  μ(H). This allows us to conclude that μ(H)  μ(H ∩ T¯ ) + μ(H ∩ T ), which implies that T is μ-measurable by Lemma 7.5. It follows that the sets ∅ and S are μ-measurable for any outer measure μ. Also, for any outer measure μ on a set S, the measure space (S, Eμ , μ) is complete.

May 2, 2018 11:28

420

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 420

Mathematical Analysis for Machine Learning and Data Mining

Theorem 7.34. Let S be a set and let μ be an outer measure on S. If μ(U ) = 0, then U is μ-measurable. Proof. Let T be a subset of S. Since T ∩ U ⊆ U , we have μ(T ∩ U )  μ(U ) = 0. Also, T ∩ U ⊆ T , hence μ(T )  μ(T ∩ olU ) = μ(T ∩ olU ) + μ(T ∩ U ), hence U is measurable.  Theorem 7.35. (Carath´ eodory Outer Measure Theorem) Let μ be an outer measure on a set S. The collection of μ-measurable sets is a σalgebra Eμ on S. The restriction mμ = μ Eμ to the σ-algebra Eμ is a measure and (S, Eμ , mμ ) is a complete measure space. Proof. If U is a μ-subset we have μ(H) = μ(H ∩ U ) + μ(H ∩ U ). Since this inequality remains the same when we exchange U with U , it follows that U is μ-measurable, so the collection of μ-measurable set is closed with respect to complementation. Next, we show that the set of μ-measurable sets is closed with respect to finite unions. Let T1 , . . . , Tn be a sequence of μ-measurable sets. We n prove, by induction on n ∈ N that j=1 Tj is μ-measurable. The base step, n = 0, is immediate. Suppose that w is μ-measurable and let H ∈ P(S). By the inductive hypothesis we have: ⎞ ⎞ ⎛ ⎛ n n   Tj ⎠ + μ ⎝H ∩ Tj ⎠ μ(H) = μ ⎝H ∩ ⎛ = μ ⎝H ∩

j=1 n 

⎞

j=1

⎛

Tj ⎠ + μ ⎝H ∩

j=1

n 

⎞ Tj ⎠

j=1

for every H ∈ P(S). Substituting H ∩ Tn+1 for H and H ∩ Tn+1 in the above equality yields ⎞ ⎛ ⎞ ⎛ n n   μ(H ∩ Tn+1 ) = μ ⎝(H ∩ Tn+1 ) ∩ Tj ⎠ + μ ⎝(H ∩ Tn+1 ) ∩ Tj ⎠ ⎛ μ(H ∩ Tn+1 ) = μ ⎝(H ∩ Tn+1 ) ∩

j=1 n 

⎞

⎛

Tj ⎠ + μ ⎝(H ∩ Tn+1 ) ∩

j=1

Note that (Tn+1 ∩ Tj ) ∪ (Tn+1 ∩ Tj ) ∪ (Tn+1 ∩ Tj ) = Tn+1 ∪ Tj

j=1 n  j=1

⎞ Tj ⎠ .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 421

421

for 1  j  n. Therefore, ⎞ ⎛ ⎞ ⎛ n+1 n n    Tj = ⎝(H ∩ Tn+1 ) ∩ Tj ⎠ ∪ ⎝(H ∩ Tn+1 ) ∩ Tj ⎠ H∩ j=1

j=1

⎛

n 

∪ ⎝(H ∩ Tn+1 ) ∩

j=1

⎞ Tj ⎠ ,

j=1

hence

⎛

n 

μ ⎝H ∪ ⎛

⎞ Tj ⎠

j=1

 μ ⎝(H ∩ Tn+1 ) ∩

n 

⎞

Tj ⎠ + μ ⎝(H ∩ Tn+1 ) ∩

j=1

⎛

+μ ⎝(H ∩ Tn+1 ) ∩

⎛

n 

n 

⎞ Tj ⎠

j=1

⎞ Tj ⎠ .

j=1

The measurability of Tn+1 implies μ(H) = μ(H ∩ Tn+1 ) + μ(H ∩ Tn+1 ) ⎛ ⎞ ⎛ ⎞ n n    μ ⎝(H ∩ Tn+1 ) ∩ Tj ⎠ + μ ⎝(H ∩ Tn+1 ) ∩ Tj ⎠ j=1

⎛

+μ ⎝(H ∩ Tn+1 ) ∩ ⎛  μ ⎝H ∪

n+1 

⎞

n 

⎞

Tj ⎠ + μ ⎝(H ∩ Tn+1 ) ∩

j=1

⎛

Tj ⎠ + μ ⎝H ∩

j=1

j=1

⎛

n+1 

⎞

n 

⎞ Tj ⎠

j=1

Tj ⎠ ,

j=1

n which proves that j=1 Tj is μ-measurable for n ∈ N. Thus, the collection of μ-measurable sets is an algebra, and, therefore is also closed under finite intersections and set difference. If U0 , U1 are two disjoint μ-measurable sets, then μ(H ∩ (U0 ∪ U1 )) = μ(H ∩ U0 ) + μ(H ∩ U1 ), for every H. Again, an inductive argument allows us to show that if T0 , . . . , Tn are pairwise disjoint μ-measurable sets then   n n   Ui = μ(H ∩ Ui ). (7.7) μ H∩ i=0

i=0

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

422

9in x 6in

b3234-main

page 422

Mathematical Analysis for Machine Learning and Data Mining

n Define Wn = i=0 Ti . We have seen that Wn is μ-measurable for every n ∈ N. Thus, we have μ(H) = μ(H ∩ Wn ) + μ(H ∩ Wn )   n   Ti + μ(H ∩ Wn ) =μ H∩ 



μ H∩ where W =

 i0

i=0 n 

 Ti

+ μ(H ∩ W ),

i=0

Ti . By equality (7.7) we have n  μ(H)  μ(H ∩ Ti ) + μ(H ∩ W ),

(7.8)

i0

for every n ∈ N. Therefore, μ(H) 

∞ 

μ(H ∩ Ti ) + μ(H ∩ W ),

i0

hence μ(H)  μ(H ∩ W ) + μ(H ∩ W ). By Lemma 7.5, the set W is μmeasurable. Note also that we have shown that n  μ(H ∩ Ti ) + μ(H ∩ W ) = μ(H ∩ W ) + μ(H ∩ W ). (7.9) μ(H) = i0

Suppose now that the sets T0 , T1 , . . . are not disjoint. Consider the sequence of pairwise disjoint sets V0 , V1 , . . . defined by: V0 = T0 , n−1  Ti , Vn = Tn − i=0

for n  1. The measurability of each set Vn is immediate and, by the    previous argument, n∈N Vn is μ-measurable. Since n∈N Vn = n∈N Tn ,  it follows that n∈N Tn is μ-measurable. We conclude that the collection of μ-measurable sets is actually a σ-algebra. Let T0 , T1 , . . . a sequence of sets in Eμ that are pairwise disjoint. Choosing H = W in equality (7.9) we have: n  μ(Ti ), μ(W ) = i0

which proves that mμ = μ Eμ is indeed a measure. Let W ∈ Eμ such that mμ (W ) = 0 and let U ⊆ W . Note that μ(U )  μ(W ) = mμ (W ) = 0. For every T ⊆ S we have μ(T ∩ U ) + μ(T ∩ U )  μ(U )+ μ(T ) = μ(T ), so U ∈ Eμ , which shows that (S, Eμ , mμ ) is a complete measure space. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Measurable Spaces and Measures

b3234-main

page 423

423

ˆ 0 If μφ is the outer measure that is generated by a mapping φ : C −→ R such that φ(∅) = 0 (as discussed in Example 7.12), we will denote by mφ the measure mμφ . Corollary 7.16. Let μ be an outer measure and let U0 , U1 , . . . be a sequence of μ-measurable sets. Then, both lim inf Un and lim sup Un are μ-measurable sets. Proof.

This statement follows immediately from Theorems 7.35 and 7.52. 

Definition 7.21. Let μL be the Lebesgue outer measure on R introduced in Example 7.11. A Lebesgue measurable set is a subset U of R that is μL -measurable. ˆ is a Lebesgue measurable set. Theorem 7.36. Every Borel set B ∈ B(R) Proof. The argument is based on the fact that the σ-algebra of Borel sets of reals is generated by intervals of the form (−∞, b], as we saw in Theorem 7.5. It would suffice to show that each such interval is μL -measurable to show that B(R) ⊆ EμL . We need to prove that for every subset T of R such that μL (T ) is finite we have: μL (T )  μL (T ∩ (−∞, a]) + μL (T ∩ (a, ∞)).  Let ((an , bn ))n1 be a sequence of open interval such that T ⊆ n1 (an , bn )  and n1 (bn − an ) < μL (T ) + . Observe that the sets (an , bn )∩(−∞, b] and (an , bn )∩(b, ∞) are disjoint. Define the intervals (sn , tn ) and (un , vn ) such that the following conditions are satisfied: (i) (an , bn ) ∩ (−∞, a] ⊆ (sn , tn ), (ii) (an , bn ) ∩ (a, ∞) ⊆ (un , vn ), and (iii) tn − sn + vn − un  bn − an + 2 n . Since  (sn , tn ), T ∩ (−∞, a] ⊆ n1

T ∩ (a, ∞) ⊆



(un , vn ),

n1

it follows that μL (T ∩ (−∞, a]) 



(tn − sn ),

n1

μL (T ∩ (a, ∞)) 



(vn − un ).

n1

May 2, 2018 11:28

424

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 424

Mathematical Analysis for Machine Learning and Data Mining

Taking into account   that (tn − sn + vn − un )  (bn − an ) + , n1

n1

it follows that  (bn − an ) + < μL (T ) + 2 . μL (T ∩ (−∞, a]) + μL (T ∩ (a, ∞))  n1

Since this holds for every positive it follows that μL (T ∩ (−∞, a]) + μL (T ∩ (a, ∞))  μL (T ), so (−∞, a] is μL -measurable.



Corollary 7.17. Every open set and every closed set in R is measurable. Proof. Since (−∞, a] = R − (a, ∞), each set (−∞, a] is measurable.  In turn, this implies that (−∞, b) = n1 (−∞, b − 1/n] is measurable. Therefore, each open interval (a, b) is measurable because (a, b) = (−∞, b)∩ (a, ∞). Since each open set is a countable union of open intervals, it follows that open sets are measurable.  We revisit the construction of an outer measure discussed in Examˆ 0 ple 7.12 to better relate the outer measure μ to the function φ : C −→ R that generated the measure. Example 7.12 shows that a premeasure on S induces an outer measure on S. Theorem 7.37. Let φ be a premeasure on an algebra E of subsets of a set S and let μφ is the⎧outer measure defined by: ⎫ ⎨  ⎬ φ(Uj ) | Uj ∈ E for j  1 and U ⊆ Uj . μφ (U ) = inf ⎭ ⎩ j1

j1

For every T ∈ E we have μφ (T ) = φ(T ) and every set in E is μφ -measurable. Proof. Note that μφ (T )  φ(T ) for all T ∈ E because {T } is a cover for T.  Suppose T ∈ E and T ⊆ j1 Uj . Let {Vj | j  1} be the collection of  disjoint members of E defined by Vj = T ∩ (Uj − k 0 there exists a sequence {Wj | j  1} of sets in E such that  H ⊂j1 Wj and j1 φ(Wj )  μφ (H)+ . The additivity of φ on E implies:   φ(Wj ∩ T ) + φ(Wj ∩ T ) μφ (H) +  j1

j1

 μφ (H ∩ T ) + μφ (H ∩ T ),

(7.10)

because {Wj ∩ T | j  1} is a cover of H ∩ T by sets in E and {Wj ∩ T | j  1} is a cover of H ∩ T by sets in E. Since inequality (7.10) holds for every positive we have μφ (H)  μφ (H ∩ T ) + μφ (H ∩ T ), which implies  that T is μφ -measurable. Theorem 7.38. Let φ be a premeasure defined on an algebra E and let E = Kσ-alg (E) be the σ-algebra generated by E. There exists a measure mφ on E whose restriction to E equals φ, namely, the restriction of the outer measure μφ to E . Proof.

This statement follows from Theorem 7.37.



Theorem 7.39. (Carath´ eodory Extension Theorem) Let E be an algebra of subsets of a set S, E = Kσ-alg (E) be the σ-algebra generated by E, φ be a premeasure on E, and let mφ be the measure on E whose restriction to E equals φ. ˜ ) = mφ (T ) for T ∈ E, If m ˜ is another measure on E such that m(T then for all U ∈ E we have m(U ˜ )  mφ (U ); also if mφ (U ) is finite, then m(U ˜ ) = mφ (U ). If there exists a countable cover of S, {Uj | j  1, Uj ∈ E and mφ (Uj ) < ∞}, then m(U ˜ )  mφ (U ) for all U ∈ E .  Proof. Let T ∈ E such that T ⊆ j1 Uj , where Uj ∈ E. We have   ˜ j) = ˜ )  mφ (T ) for m(T ˜ )  j1 m(U j1 φ(Uj ), which implies m(T every T ∈ E . Let T ∈ E with mφ (T ) < ∞. There exists a cover of T , {Uj | j  1}  such that j1 mφ (Uj ) < mφ (T ) + . n Define the increasing sequence of sets (Wn ) by Wn = j=1 Uj and let  ˜ ) = limn→∞ m(W ˜ W = n1 Wn . We have m(W n ) = limn→∞ mφ (Wn ) = mφ (W ). Suppose now that T ∈ E and mφ (T ) is finite. Let be a positive number  and let {Uj | j  1} be a cover of T such that j1 mφ (Uj ) < mφ (T ) + . ˜ −T ) < . This implies mφ (W )  mφ (T )+ , so mφ (W −T ) < , hence m(W

May 2, 2018 11:28

426

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 426

Mathematical Analysis for Machine Learning and Data Mining

Since m(W ˜ ) = mφ (W ), we have mφ (T )  mφ (W ) = m(W ˜ ) = m(T ˜ ) + m(W ˜ − T )  m(T ˜ ) + , ˜ ) because mφ (T ) is finite, hence for every > 0, hence mφ (T )  m(T ˜ ). mφ (T ) = m(T If {Uj | Uj ∈ E, j  1 and mφ (Uj ) < ∞}, we may assume that the collection {U1 , U2 , . . .} is a partition of S. If T ∈ E , T can be written  ˜ j ), then as the disjoint union T = j1 T ∩ Uj . Since mφ (Uj ) = m(U ˜ ).  mφ (T ) = m(T Definition 7.22. An outer measure on a set S is regular if given any subset T of S, there exists a μ-measurable set U such that T ⊆ U and μ(T ) = μ(U ). The regularity of an outer measure ensures that it is possible to determine the outer measure of any subset T of S by considering only the subsets in Eμ . Theorem 7.40. Let S be a set and let (S0 , S1 , . . .) a sequence of subsets of S. If μ is a regular outer measure on S, then $ # μ lim inf Sn  lim inf μ(Sn ). n

n

Proof. Since μ is regular, for each n ∈ N there exists a μmeasurable set Un such that Sn ⊆ Un and μ(Sn ) = μ(Un ). Then, lim inf n μ(Sn ) lim inf n μ(Un ). Since lim inf n μ(Un ) is measurable (by Corollary 7.16) we have: $ # $ # μ lim inf Sn  μ lim inf Un  lim inf μ(Un ) = lim inf μ(Sn )). n n n n  Corollary 7.18. Let μ be an outer measure on a set S. If S = (S0 , S1 , . . .) is an expanding sequence of subsets of S, then μ(limn Sn ) = limn μ(Sn ).  Proof. Since S is an expanding sequence limn Sn = n Sn , so μ(limn Sn )  μ(Sn ) for n ∈ N, so μ(limn Sn )  limn μ(Sn ). On the other hand, Theorem 7.40 implies μ(lim Sn )  lim μ(Sn ), which gives the desired equality.  For finite regular outer measures the measurability condition can be simplified as shown next. Theorem 7.41. Let μ a regular outer measure on a set S such that μ(S) is finite. A subset T of S is measurable if and only if μ(S) = μ(T ) + μ(T ), where T = S − T .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 427

427

Proof. The condition is clearly necessary. To prove its sufficiency, let T be a subset of S such that μ(S) = μ(T ) + μ(T ). By Lemma 7.5, to prove that T is measurable it suffices to show that if H is a set with μ(H) < ∞, then μ(H)  μ(H ∩ T ) + μ(H ∩ T ). The regularity of μ implies the existence of a μ-measurable set K such that H ⊆ K and μ(H) = μ(K). Since K is measurable, we have μ(H) = μ(H ∩ K) + μ(H ∩ K), μ(H) = μ(H ∩ K) + μ(H ∩ K). This implies μ(S) = μ(T ) + μ(T ) = μ(T ∩ K) + μ(T ∩ K) + μ(T ∩ K) + μ(T ∩ K)  μ(K) + μ(K) = μ(S). Thus, μ(T ∩ K) + μ(T ∩ K) + μ(T ∩ K) + μ(T ∩ K) = μ(K) + μ(K) = μ(S). Since μ(K)  μ(T ∩ K) + μ(T ∩ K), it follows that μ(K ∩ T ) + μ(K ∩ T )  μ(K). Since H ∩ T ⊆ K ∩ T and H ∩ T ⊆ K ∩ T , we have μ(H ∩ T ) + μ(H ∩ T )  μ(K) = μ(H), which shows that T is indeed, μ-measurable.  7.7

The Lebesgue Measure on Rn

ˆ n , I = (a1 , b1 ) × Let K = [a1 , b1 ] × · · · × [an , bn ] be a closed interval of R · · · × (an , bn ), an open interval, G = (a1 , b1 ] × · · · × (an , bn ] an open-closed interval, and H = [a1 , b1 )× · · ·× [an , bn ) a closed-open interval of Rn , where ˆ a1 , . . . , an , b1 , . . . , bn ∈ R. The collections of closed, open, open-closed and closed-open intervals of Rn are denoted by Kn , In , Gn and Hn , respectively. If U is any of the above intervals its volume is: vol(U ) =

n +

(bj − aj ).

j=1

Theorem 7.42. The set E of finite unions of disjoint open-closed intervals is an algebra of sets. Proof. Let G = (a1 , b1 ] × · · · × (an , bn ] and G = (a1 , b1 ] × · · · × (an , bn ] ˆ n. be two open-closed intervals in R

May 2, 2018 11:28

428

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 428

Mathematical Analysis for Machine Learning and Data Mining

We begin by showing that E is closed with respect to finite intersections. We have G∩G = ∅ if there exists j, 1  j  n such that (an , bn ]∩(an , bn ] = ∅. In this case, G ∩ G ∈ E. Suppose now that G ∩ G = ∅. We have (aj , bj ] ∩ (aj ∩ bj ] = ∅ for 1  j  n, which is possible if and only if max{aj , aj }  min{bj , bj }; in this case (aj , bj ] ∩ (aj ∩ bj ] = (max{aj , aj }, min{bj , bj }] and G ∩ G =

n +

(max{aj , aj }, min{bj , bj }],

j=1

so E is closed with respect to finite intersections. To show that E is closed with respect to complements consider an openˆ − (aj , bj ] = closed interval G = (a1 , b1 ] × · · · × (an , bn ]. Note that R (−∞, aj ) ∪ (bj , ∞]. By applying the second equality of Supplement 7 of Chapter 1 we have: ˆ n − ((a1 , b1 ] × (a2 , b2 ] × · · · × (an , bn ]) R ˆ n−1 ) ˆ − (a1 , b1 ]) × R = ((R ˆ − (a2 , b2 ]) × R ˆ n−2 ) ∪ ((a1 , b1 ] × (R n−k

ˆ − (ak , bk ]) × R ˆ ) ∪ ((a1 , b1 ] × · · · × (ak−1 , bk−1 ] × (R ˆ − (an , bn ]). ∪ ((a1 , b1 ] × (a2 , b2 ] × · · · × (an−1 , bn−1 ] × (R ˆ Since R−(a j , bj ] = (−∞, a]∪(bj , ∞] for 1  j  n, by applying the distributivity of set product with respect to union it follows that the complement ˆ n − ((a1 , b1 ] × (a2 , b2 ] × · · · × (an , bn ]) is an union of open-closed of G, R intervals, so it belongs to E. If E ∈ E is the union of disjoint open-closed intervals in Gn , E =  G1 ∪ · · · ∪ Gm . Then E = m j=1 Gj . Each complement Gj belongs to E as we saw above, and since E is closed with respect to finite intersections, E ∈ E so E is closed with respect to complementation. Finally, since E is closed with respect to finite intersection and complementation, it follows that E is closed with respect to finite unions, so it is an algebra of sets.  Corollary 7.19. If G, G1 , . . . , Gm are pairwise disjoint open-closed intervals, there exist pairwise disjoint open-closed intervals G1 , . . . , Gq such that m G − j=1 Gj = G1 ∪ · · · ∪ Gq . Proof. This follows from the fact that an algebra of sets is closed with respect to set difference. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Measurable Spaces and Measures

b3234-main

page 429

429

Corollary 7.20. If G1 , . . . , Gm are open-closed intervals then there exist pairwise disjoint open-closed intervals G1 , . . . , Gq such that G1 ∪· · ·∪Gm = G1 ∪ · · · ∪ Gq . Proof. Let E be the algebra of finite unions of disjoint open-closed inˆ n . Consider the sequence of open-closed interval (G1 , . . . , Gm ). tervals in R We have shown that the complement Gj of any of these intervals belongs to E.  For each x ∈ nj=1 there is a least j such that x ∈ Gj . Thus, x ∈ Gj , where Gj = Gj ∩ G1 ∪ · · · ∪ Gj−1 ). Note that if j = k, then Gj ∩ Ck = ∅, the sets G1 , G2 , . . . , Gm belong to E, and that G1 ∪ · · · ∪ Gm = G1 ∪ · · · ∪ Gq .  Lemma 7.6. Let G ∈ Gn be a closed-open interval, G = (a1 , b1 ] × · · · × (an , bn ]. Suppose that for each interval (aj , bj ] there is a subdivision Δj = m m {a0j , a1j , . . . , aj j } such that aj = a0j < a1j < · · · < aj j = bj . Let Gi1 i2 ···in be the open-closed interval given by Gi1 i2 ···in = (ai11 −1 , ai11 ] × · · · × (ainn −1 , ainn ]. Then,  vol(G) = {vol(Gi1 i2 ···in ) | 1  i1  m1 , . . . , 1  in  mn }. Proof. Note that  {vol(Gi1 i2 ···in ) | 1  i1  m1 , . . . , 1  in  mn }  = {(ai11 − ai11 −1 ) · · · (ainn − ainn−1 ) | 1  i1  m1 , . . . , 1  in  mn }   {(ainn − anin −1 ) | 1  i1  m1 } = {(ai11 − ai11 −1 ) | 1  i1  m1 } · · · (by the distributivity property) = (b1 − a1 ) · · · (bn − an ) = vol(G), which concludes the argument.



Theorem 7.43. Let G = (a1 , b1 ] × · · · × (an , bn ]. If π = {G1 , . . . , G } is a partition of G that consists of open-closed intervals in Gn , then vol(G) =  k=1 vol(Gk ). Proof. The idea of the proof is to show that both G and each of the sets G1 , . . . , G can be decomposed into a union of pairwise open-closed intervals. Let pj : Rn −→ R be the j th projection defined by pj (x) = xj for x ∈ Rn , where 1  j  n. Consider the partition m −1 m πj = {(a0j , a1j ], . . . , (aj j , aj j ]},

May 2, 2018 11:28

430

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 430

Mathematical Analysis for Machine Learning and Data Mining

of the interval (aj , bj ] generated by the collection {pj (G1 ), . . . , pj (G )} as m defined in Supplement 6 of Chapter 1. We have a0j = aj and aj j = bj . Each set pj (Gk ) is πj -saturated, that is, each set pj (Gk ) is an union of a collection Gkj of rjk one-dimensional open-closed intervals Ghjk , where 1  h  rjk , 

rjk

pj (Gk ) =

Ghjk .

h=1

Therefore, by Lemma 7.6, we have Gk =

n +

pj (Gk ) =

j=1

n r jk + 

Ghjk

j=1 h=1

for 1  j . Thus, we have G=

  k=1

Gk =

jk  + n r  

Ghjk

k=1 j=1 h=1

Ghjk

and the intervals are pairwise disjoint. An application of Lemma 7.6 leads to the desired conclusion.  Theorem 7.44. Let G1 , . . . , Gm be m pairwise disjoint open-closed interˆ n , then ˆ n . If m Gj ⊆ G, where G is an open-closed interval in R vals in R j=1 vol(G1 ) + · · · + vol(Gn )  vol(G). Proof. By Theorem 7.42, G1 ∪· · ·∪Gm belongs to the algebra E of unions m of families of disjoint open-closed intervals. Therefore, G− j=1 Gj belongs to the same algebra and, thus it can be written as a union of disjoint openclosed intervals G1 , . . . , Gp . Thus, G = G1 ∪ · · · ∪ Gm ∪ G1 ∪ · · · ∪ Gp ,  m vol(Gj ) + pk=1 vol(Gk ), by Theorem 7.43. Thus, hence vol(G) = j=1 m  j=1 vol(Gj )  vol(G). ˆ n . If G ⊆ Theorem 7.45. Let G1 , . . . , Gm be m open-closed intervals in R m n ˆ j=1 Gj , where G is an open-closed interval in R , then vol(G)  vol(G1 )+ · · · + vol(Gn ). m m Proof. Since G ⊆ j=1 Gj , we have G = j=1 Gj , where Gj = G ∩ Gj are open-closed intervals. G can be written as a union of disjoint members of the algebra E of unions of disjoint open-closed intervals: ⎞ ⎛ m−1  Gj ⎠ . G = G1 ∪ (G2 − G1 ) ∪ · · · ∪ ⎝Gm − j=1

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

Thus,

page 431

431

⎛ vol(G) = vol(G1 ) + vol(G2 − G1 ) + · · · + vol ⎝Gm −

m−1 

⎞ Gj ⎠

j=1



m 

vol(Gj ). 

j=1 n

ˆ , where Ij = Theorem 7.46. Let I1 , . . . , Im be m open intervals in R (cj1 , dj1 )×(cjn , djn ) for 1  j  m. If K = [a1 , b1 ]×· · ·×[an , bn ] is a closed ˆ n such that K ⊆ m Ij , then vol(K)  vol(I1 )+· · ·+vol(Im ). interval in R j=1 Proof. Consider the open-closed intervals G = (a1 , b1 ]×· · ·×(an , bn ] and Gj = (cj1 , dj1 ] × (cjn , djn ] for 1  j  m. We have: G ⊆ K ⊆ I1 ∪ · · · ∪ Im ⊆ G1 ∪ · · · ∪ Gm , hence vol(K) = vol(G)  vol(G1 ) + · · · + vol(Gm ) (by Theorem 7.45) = vol(I1 ) + · · · + vol(Im ).



Theorem 7.37 allows the introduction of an outer measure on R as ⎫ ⎧ ⎨  ⎬ μn (U ) = inf vol(Ij ) | Ij ∈ I(Rn ) for j  1 and U ⊆ Ij ⎭ ⎩ n

j1

j1

for U ⊆ R . The collection Eμn of μn -measurable sets defines a complete measure space (Rn , Eμn , mμn ). n

Definition 7.23. The collection Eμn is the σ-algebra Ln of Lebesgue measurable sets in Rn and mμn is the n-dimensional Lebesgue measure on Rn . Since the dimension of the ambient space Rn is usually clear from context, we use the notation mL for the n-dimensional Lebesgue measure mμn . Theorem 7.47. For every type of interval U (open, closed, open-closed, or closed-open) we have mL (U ) = vol(U ); also, each interval is a mL measurable set. Proof. Let K be a closed interval, K = [a1 , b1 ] × · · · × [an , bn ]. For every > 0 we have; K ⊆ (a1 − , b1 + ) × · · · × (an − , bn + ).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 432

Mathematical Analysis for Machine Learning and Data Mining

432

Therefore, by the definition of μn we have: μn (K)  vol((a1 − , b1 + ) × · · · × (an − , bn + )) =

n +

(bj − aj + 2 )

j=1

for every > 0. Therefore, μn (K)  vol(K). Let {Ij | j ∈ J} be a cover of K by open intervals. Since K is compact, there exists a finite subcollection of this collection of open intervals such that K ⊆ Ij1 ∪ · · · ∪ Ijk , hence vol(K)  vol(Ij1 ) + · · · + vol(Ijk )  vol(Ij ).  j∈J

This implies vol(K)  μn (K) (since μn (K) was defined as an infimum). Therefore, vol(K) = μn (K). Let W be an interval delimited by a1 , b1 , . . . , an , bn and let K1 , K2 be K1 = [a1 − , b1 + ] × · · · × [an − , bn + ], K2 = [a1 + , b1 − ] × · · · × [an + , bn − ]. Since K2 ⊆ W ⊆ K1 , it follows that μn (K2 )  μn (W )  μn (K1 ) for every > 0. Thus, μn (W ) = vol(W ). Next, we show that every open-closed interval G = (a1 , b1 ]×· · ·×(an , bn ] is μn -measurable. Let I = (c1 , d1 ) × · · · × (cn , dn ) and let GI = (c1 , d1 ] × · · · × (cn , dn ]. Then, μn (G ∩ I)  μn (G ∩ GI ) = vol(G ∩ GI ) and μn (G ∩ I)  μn (G ∩ GI ). By Corollary 7.19 we have GI ∩ G = GI − G = G1 ∪ · · · ∪ Gk for some pairwise disjoint open-closed intervals G1 , . . . , Gk . This implies μn (I ∪ G)  μn (G1 ) + · · · + μn (Gk ) = vol(G1 ) + · · · + vol(Gk ) = vol(GI ) = vol(I). We may conclude that μn (I ∩ G) + μn (I ∩ G)  vol(I). ˆ n with μn (E) < ∞. Consider a cover of E that Let E be a subset of R   consists of open intervals, E ⊆ j≥1 Ij such that j1 vol(Ij ) < μn (E)+ . If G is an open-closed interval, then μn (E ∩ G) + μn (E ∩ G)    μn (Ij ∩ G) + μn (Ij ∩ G) j1

=

 j1

j1

(μn (Ij ∩ G) + μn (Ij ∩ G)) 

 j1

vol(Ij ) < μn (E) + .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 433

433

Therefore, μn (E ∩ G) + μn (E ∩ G)  μn (E), hence G is μn -measurable, that is, G ∈ Ln . For any interval V that has an edge that consists of a single point we have μn (V ) = 0 and, therefore, by the completeness of μn , V is a Lebesgue measurable set. Since every open, closed, or closed-open interval differs from an open-closed interval G with the same sides by finitely many intervals of measure 0, any such interval is also a Lebesgue measurable set.  n n n ˆ ˆ The Lebesgue measure mL on R is σ-finite because R = k=1 [−k, k]n and mL ([−k, k]n ) = μn ([−k, k]n ) = vol([−k, k]n ) = (2k)n for k ∈ N and ˆ n ) is not finite. k  1. On the other hand, mL (R Theorem 7.48. Every Borel set in Rn is a Lebesgue measurable set. ˆ n ) is generated by the set of Proof. This follows from the fact that B(R intervals and each interval is a Lebesgue measurable set.  The next two theorems offer characterizations of μn -measurable sets ˆ n. relative to certain topologically defined subsets of R ˆ n , O) be the usual topological space on R ˆ n . A subset Theorem 7.49. Let (R n ˆ is Lebesgue measurable if and only if there exists A ∈ Oδ such that U of R U ⊆ A and μn (A − U ) = 0. Proof. Suppose that there exists A ∈ Oδ such that U ⊆ A and μn (A − U ) = 0. By the completeness of mL the set A − U is Lebesgue measurable and, therefore U = A − (A − U ) is Lebesgue measurable. Conversely, suppose that U is Lebesgue measurable. There exists a sequence (V1 , V2 , . . .) of Lebesgue measurable sets such that μn (Vj ) < ∞  ˆn =  and R j1 Vj . Therefore, U = j1 (U ∩ Vj ) and μn (U ∩ Vj ) < ∞ for j  1. The definition of μn as an infimum implies a collection of open intervals   (k,l) (k,l) k,l and j1 vol(Ij )  μn (Uk ) + 21k l . Ij such that Uk ⊆ j1 Ij  (k,l) Let W (k,l) = j1 Ij . Then, Uk ⊆ W k,l and μn (W (k,l) )  μn (Uk ) +  1 . Therefore, μn (W (k,l) − Uk ) < l21k . If Z (l) = k1 W (k,l) , then Z (l) is 2k l  open, U ⊆ Z (l) , and Z (l) − U ⊆ k1 (W (k,l) − Uk ). This implies   1 1 μn (Z (l) − U )  μn (W (k,l) − Uk ) < = . k l2 l k1 k1  If A = l1 Z (l) , we have U ⊆ A and μn (A − U )  μn (Z (l) − U ) < 1l for  every l  1, hence μn (A − U ) = 0.

May 2, 2018 11:28

434

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 434

Mathematical Analysis for Machine Learning and Data Mining

ˆ n , O) be the usual topological space on R ˆ n. A Theorem 7.50. Let (R ˆ n is Lebesgue measurable if and only if there exists C ∈ subset U of R n ˆ , O)σ such that C ⊆ U and μn (U − C) = 0. COMP(R Proof. Suppose that B is a union of compact sets such that B ⊆ U and μn (U − B) = 0. The completeness of mL implies that U − B is Lebesgue measurable. Since U = B ∪ (U − B), it follows that U is Lebesgue measurable. Conversely, suppose that U is Lebesgue measurable. Since U is Lebesgue measurable, by Theorem 7.48 there exists A ∈ Oδ such that U ⊆ A and μn (A − U ) = 0. The set C = A is a countable union of closed sets,  C = j1 Tj . The sets Tj,k = Tj ∩ [−k, k]n are compact because they are   ˆ n , O)σ . Tj,k belongs to COMP(R closed and bounded and C = j1

k1

 n

ˆ is almost equal to a Borel set. Corollary 7.21. Every Lebesgue set in R Proof.



This follows from Theorems 7.48 and 7.49. n

ˆ 0 is a measure such that m(G) = ˆ ) −→ R Theorem 7.51. If m : B(R vol(G) for every open-closed interval G, then m = mL . Proof. Let G is an unbounded open-closed interval and let (G1 , G2 , . . .) be an increasing sequence of bounded open-closed intervals such that  j1 Gj = G. Since limn→∞ m(Gi ) = m(G), it follows that m(G) = ∞. If G1 , . . . , Gm are pairwise disjoint open-closed intervals we have ⎛ ⎞ ⎛ ⎞ m m m m     m⎝ Gj ⎠ = m(Gj ) = mL (Gj ) = mL ⎝ Gj ⎠ , j=1

j=1

j=1

j=1

hence m and mL are equal on the algebra that consists of unions of pairwise disjoint open-closed intervals. Since these unions generate the σ-algebra of  Borel sets, we have m = mL . n

ˆ , Ln , mL ) is the completion of the Theorem 7.52. The measure space (R n n ˆ ˆ measure space (R , B(R ), mL ). Proof. Since every Borel subset of Rn is a Lebesgue measurable set we ˆ n ⊆ Ln . have B(R Let U be a Lebesgue measurable set. By Theorem 7.49 there exists an ˆ n ) such that U ⊆ A and μn (A − U ) = 0. Note that A is a A ∈ Oδ ⊆ B(R Borel set.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Measurable Spaces and Measures

b3234-main

page 435

435

n

ˆ , O)σ such that C ⊆ U By Theorem 7.49 there exists C ∈ COMP(R and μn (U − C) = 0 and, again C is a Borel set. Note that m(A − C)  m(A − U ) + m(U − C) = 0, so m(A − C) = 0, ˆ n ).  which shows that Ln is indeed the completion of B(R ˆ n with Next we examine the behavior of the Lebesgue measures on R respect to linear and affine transformations. ˆ n transforms every interval (open, closed, Note that a translation hx of R open-closed, or closed-open) into an interval of the same nature with the same volume. A homothety ha transforms any interval into an interval of the same nature; the volume this time changes. Indeed, let G = (a1 , b1 ] × (an , bn ] be an open-closed interval. Since ha (G) = (aa1 , ab1 ] × (aan , abn ], we have vol(ha (G)) = an vol(G). This is the homogeneity property of volume. The image of G under reflection is a closed open interval r(G) = [−b1 , −a1 ) × · · · [−bn , an ) and vol(r(G)) = vol(G). ˆ n are invariant under We conclude that the volumes of intervals in R translations and reflections; with respect to homotheties, the volume is a homogenous function of degree n. ˆ n is closed Theorem 7.53. The set Ln of Lebesgue measurable sets in R with respect to translations, reflections and homotheties. Furthermore, mL (tx (U )) = mL (U ), mL (s(U )) = mL (U ), and μL (ha (U )) = an m( U ) for every Lebesgue measurable set U , x ∈ Rn and a > 0. ˆ n and let {I1 , I2 , . . .} be a collection of Proof. Let U be a subset of R  open intervals such that U ⊆ j1 Ij . It follows that  tx (U ) ⊆ tx (Ij ), j1

hence μn (tx (U )) 

 j1

vol(tx (Ij )) =



vol(Ij ),

j1

hence μn (tx (U ))  μn (U ). Therefore, μn (U ) = μn (t−x (tx (U )))  μn (tx (U )), which implies μn (tx (U )) = μn (U ).

May 2, 2018 11:28

436

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 436

Mathematical Analysis for Machine Learning and Data Mining

  Since r(U ) ⊆ j1 r(Ij ), we have μn (r(U ))  j1 vol(r(Ij )) = vol(I ), which implies μ (r(U ))  μ (U ). By replacing U with r(U ) j n n j1 in the last inequality we obtain μn (U )  μn (r(U )), so μn (U ) = μn (r(U )).  Since ha (U ) ⊆ j1 ha (Ij ), we have   μn (ha (U ))  vol(ha (Ij )) = an vol(Ij ),



j1

j1

hence μn (ha (U ))  an μn (U ). Substituting ha (U ) for U and previous inequality yields μn (h a1 (ha (U ))) 

1 a

for a in

1 μn (ha (U )), an

which amounts to an μn (U )  μn (ha (U )). Thus, an μn (U ) = μn (ha (U )). ˆ n , by Corollary 1.2, we have Since tx , ha and r are bijections on R tx (S) = tx (S), ha (S) = ha (S), and r(S) = r(S), ˆ n. for every subset S of R ˆ n . We have Let T ∈ Ln and let U be a subset of R μn (U ∩ tx (T )) + μn (U ∩ tx (T )) = μn (U ∩ tx (T )) + μn (U ∩ tx (T )) = μn (U ), which implies that tx (T ) ∈ Ln . Similar arguments show that ha (T ) ∈ Ln and r(T ) ∈ Ln .



ˆ n −→ R ˆ n be a invertible linear operator. If Theorem 7.54. Let h : R T ∈ Ln , then h(T ) ∈ Ln and mL (h(T )) = | det(Ah )|mL (T ). Proof. An injective linear transformation can be expressed as a composition of three types of linear transforms whose matrices are T (i)↔(j) , T a(i) with a = 0, and T (i)+1(j) (see [121]). Since ⎛ ⎞ ⎛ ⎞ x1 x1 ⎜ . ⎟ ⎜ . ⎟ ⎜ .. ⎟ ⎜ .. ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ x ⎟ ⎜x ⎟ ⎜ i⎟ ⎜ j⎟ ⎜ ⎟ ⎜ . ⎟ (i)↔(j) ⎜ .. ⎟ ⎜ ⎟ T ⎜ . ⎟ = ⎜ .. ⎟ , ⎜ ⎟ ⎜ ⎟ ⎜ xj ⎟ ⎜ xi ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ .. ⎟ ⎜ .. ⎟ ⎝ . ⎠ ⎝ . ⎠ xn xn

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 437

437

we have hT (i)↔(j) (G) = (a1 , b1 ] × · · · × (aj , bj ] × · · · × (ai , bi ] × · · · × (an , bn ], hence hT (i)↔(j) (G) is an open-closed interval with the same volume as G and we have vol(hT (i)↔(j) (G)) = | det(AhT (i)↔(j) )|vol(G) because det(AhT (i)↔(j) ) = −1. The effect of hT a(i) on x is

⎞ ⎛ ⎞ x1 x1 ⎜ .. ⎟ ⎜ .. ⎟ ⎜ . ⎟ ⎜ . ⎟ ⎜ ⎟ ⎜ ⎟ ⎟ ⎜ ⎟ T a(i) ⎜ ⎜ xi ⎟ = ⎜axi ⎟ . ⎜ . ⎟ ⎜ . ⎟ ⎝ .. ⎠ ⎝ .. ⎠ ⎛

xn

xn

We have for a > 0: hT a(i) (G) = (a1 , b1 ] × · · · × (aai , abi ] × · · · (an , bn ], and for a < 0, hT a(i) (G) = (a1 , b1 ] × · · · × (abi , aai ] × · · · (an , bn ], hence hT a(i) (G) is an open-closed interval, vol(hT a(i) (G)) = |a|vol(G), and vol(hT a(i) (G)) = | det(AhT a(i) )|vol(G) because det(AhT (i)↔(j) ) = a. Finally, for T (i)+1(j) we have ⎛

⎞ ⎛ ⎞ x1 x1 ⎜ . ⎟ ⎜ . ⎟ ⎜ .. ⎟ ⎜ .. ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ x ⎟ ⎜x + x ⎟ ⎜ i⎟ ⎜ i j⎟ ⎜ . ⎟ ⎜ . ⎟ ⎟ ⎜ ⎟ T (i)+1(j) ⎜ ⎜ .. ⎟ = ⎜ .. ⎟ . ⎜ ⎟ ⎜ ⎟ ⎜ xj ⎟ ⎜ xj ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ .. ⎟ ⎜ .. ⎟ ⎝ . ⎠ ⎝ . ⎠ xn xn

Therefore, if y ∈ G = hT (i)+1(j) (G) we have yk ∈ (ak , bk ] for k = i and yi − yj ∈ (ai , bi ]. Consider the open-closed interval G = (a1 , b1 ] × · · · × (ai−1 , bi−1 ] × (ai + aj , bi + aj ] × (ai+1 , bi+1 ] × · · · (an , bn ]

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 438

Mathematical Analysis for Machine Learning and Data Mining

438

and the sets n

ˆ | yk ∈ (ak , bk ) if k =  i and ai + aj < yi  ai + yj }, M  = {y ∈ R n  ˆ | yk ∈ (ak , bk ) if k =  i and bi + aj < yi  bi + yj }. M = {y ∈ R It is clear that the sets G , G , M  and M  are Borel sets and, therefore, are Lebesgue measurable. Note that M  = tu (M  ), where u = (bi − ai )ei . Therefore, by Theorem 7.53 we have mL (M  ) = mL (M  ). The sets G and M  are disjoint for the defining conditions of these sets imply yi −yj > ai and yi −yj  ai which cannot be satisfied simultaneously. Similarly, the sets G and M  are disjoint because their defining conditions imply yi  bi + aj and yi > bi + aj , respectively, and these conditions are incompatible. On other hand we have G ∪ M  = G ∪ M  . Indeed, if y ∈ G ∪ M  then ai < yi − yj  bi or ai + aj < yi  ai + yj . If y ∈ G ∪ M  we have, then ai + aj < yi  bi + aj or bi + aj < yi  bi + yj . Since ai < yi  bi and aj < yj  bj the two systems of inequalities are equivalent, so G ∪ M  = G ∪ M  . This implies mL (G ) + mL (M  ) = mL (G ) + mL (M  ), hence mL (hT (i)+1(j) (G)) = mL (G ) = mL (G) = | det(AT (i)+1(j) )|mL (G). We have shown that for every elementary transformation h we have mL (h(G)) = | det(Ah )|mL (G) for any open-closed interval G. If I is an open interval, it is immediate that h(I) is a Borel set. For I = (a1 , b1 ) × · · · × (an , bn ) consider the open-closed intervals G1 = (a1 , b1 ] × · · · × (an , bn ], G 2 = (a1 , b1 − ] × · · · × (an , bn − ]. Since G 2 ⊆ I ⊆ G1 , we have h(G 2 ) ⊆ h(I) ⊆ h(G1 ), so | det(Ah )|mL (G2 )  mL (h(I))  | det(Ah )|mL (G1 ). Taking → 0 we have mL (h(I)) = | det(Ah )|mL (I) for every open interval I.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 439

439

ˆ n and suppose Let h be an elementary linear transformation. If U ⊆ R  that {I1 , I2 , . . .} is a cover of U . We have h(I) ⊆ j1 h(Ij ), hence   mL (h(Ij )) = | det(Ah )| mL (Ij ). μn (h(U ))  j1

j1

This implies μn (h(U ))  | det(Ah )|μn (U ). Since every linear transformation h is a superposition of elementary transformations, h = h1 h2 · · · hq , we have μn (h(U ))  | det(Ah1 )| · · · | det(Ahq )|μn (U ) = | det(Ah )|μn (U ) ˆ n . Since h is invertible we also have for every U ⊆ R μn (U )  | det(Ah−1 )|μn (h(U )) =

1 μn (h(U ), | det(Ah )|

that is, μn (h(U ))  | det(Ah )|μn (U ). Thus, we have μn (h(U )) = | det(Ah )|μn (U ). Let now T ∈ Ln . We claim that if h is an invertible linear operator, ˆ n . We have: then h(T ) ∈ Ln . Indeed, let U be a subset of R μn (U ∩ h(T )) + μn (U ∩ h(T )) = μn (h(h−1 (U )) ∩ h(T )) + μn (h(h−1 (U )) ∩ h(T )) = μn (h(h−1 (U ∩ T ) + μn (h(h−1 (U ) ∩ T )) = | det(Ah )|(μn (h−1 (U ∩ T )) + μn (h−1 (U ) ∩ T ) = | det(Ah )|μn (h−1 (U )) = μn (U ), hence h(T ) ∈ Ln . Furthermore, mL (h(T )) = μn (h(T )) = |det(Ah )|μn (T ) = |det(Ah )|mL (T ).



Theorem 7.55. A subspace V of Rn with dim(V ) < n is Lebesgue measurable and mL (V ) = 0. Proof. Let {v1 , . . . , vm } be a basis in V , where m  n − 1. This basis can be extended to a basis of Rn (see [121]), {v1 , . . . , vm , vm+1 , . . . , vn }. If x ∈ Rn we have x = x1 v1 + · · · + xn vn and the function g : Rn −→ Rn defined by ⎛ ⎞ x1 ⎜ .. ⎟ g(x) = ⎝ . ⎠ xn

May 2, 2018 11:28

440

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 440

Mathematical Analysis for Machine Learning and Data Mining

is invertible and

    v∈V . 0n−m 

 g(V ) =

v

For r ∈ N define the m-dimensional cube Kr as Kr = [−r, r] × · · · × [−r, r] m × {0} × {0} · · · × {0}. 2 34 5  Clearly, Kr ∈ Ln and mL (Kr ) = 0. Since g(V ) ⊆ r1 Kr , by the completeness of the Lebesgue measure, it follows that g(V ) ∈ Ln and  mL (g(V )) = 0. ˆ n −→ R ˆ n be a linear operator such Theorem 7.56. Let h : R that det(Ah ) = 0. Then, Img(h) is a Lebesgue measurable set with mL (Img(h)) = 0. Proof. If det(Ah ) = 0, h is not invertible and V = Img(h) is a subspace that is at most (n − 1)-dimensional. Thus, by Theorem 7.55, Img(h) is a  Lebesgue measurable set with mL (Img(h)) = 0. The remainder of this section focuses on null sets in (R, B(R), mL ). We introduced the notion of null set in a measure space (S, E, m) in Definition 7.12. In the specific case of (R, L, mL ) we have the following alternative characterization: Theorem 7.57. A Lebesgue measurable subset U of R is a null set in (R, L, mL ) if for every > 0 there exists a countable collection of intervals   (ai , bi ) for i ∈ N such that U ⊆ i∈N (ai , bi ) and i∈N (ai , bi ) < . Proof. Suppose that U is a null set in (R, B(R), mL ), hence U is mL measurable and mL (U ) = 0. We have ⎧ ⎫ ⎨ ⎬ (bn − an ) | ((an , bn ))n1 ∈ CU = 0, mL (U ) = μL (U ) = inf ⎩ ⎭ n1

hence for every > 0, there exists a collection of intervals {(an , bn ) | n  1} such that   (ai , bi ) and

(ai , bi ) < . U⊆ i∈N i∈N Conversely, suppose that for a set U ∈ L and every > 0 there exists a  countable collection of intervals (ai , bi ) for i ∈ N such that U ⊆ i∈N (ai , bi )   Then, mL (U )  and i∈N (ai , bi ) < . i∈N (ai , bi ) < , hence  mL (U ) = 0.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 441

441

By a previous observation the open intervals can be replaced by closed intervals or by semi-closed (or semi-open) intervals without affecting the definition of null set. Example 7.14. Every one-element subset {x} of R is a null set because we have {x} ⊆ (x − 2 , x + 2 ) for every > 0. Theorem 7.58. A countable union of null subsets of R is a null set. Proof. Let {Un | n ∈ N} be a countable collection of null sets and let be a positive number. For a set Un there exists a countable collection of   intervals In = {Inm | m ∈ N} such that Un ⊆ In and m∈N (Inm ) < 2n+1 . The total length of all intervals can be bounded by   

(Inm ) < = . 2n+1 n∈N m∈N n∈N   Since n∈N In is a countable collection of intervals such that {Un | n ∈     N} ⊆ n∈N In , it follows that {Un | n ∈ N} is a null set. Corollary 7.22. The set of rational numbers Q is a null set. Proof. Since Q is a countable set, we can regard it as the countable union of singleton sets {q} for q ∈ Q. The statement follows immediately from Theorem 7.58.  Null sets that are uncountable also exist. We introduce a special uncountable subset of the set of real numbers that is a null set. Let vn : {0, 1}n −→ N be the function defined by vn (b0 , b1 , . . . , bn−1 ) = 2n−1 b0 + · · · + 2bn−2 + bn−1 for every sequence (b0 , . . . , bn ) ∈ {0, 1}n. Clearly, vn (b0 , . . . , bn−2 , bn−1 ) yields the number designated by the binary sequence (b0 , . . . , bn−2 , bn−1 ). For example, v3 (110) = 22 · 1 + 21 · 1 + 0 = 6. Similarly, let wn : {0, 1, 2}n −→ N be the function defined by wn (b0 , b1 , . . . , bn−1 ) = 3n−1 b0 + · · · + 3bn−2 + bn−1 for every sequence (b0 , . . . , bn ) ∈ {0, 1, 2}n. Then, wn (b0 , . . . , bn−2 , bn−1 ) is the number designated by the ternary sequence (b0 , . . . , bn−2 , bn−1 ). For example, w3 (110) = 32 · 1 + 31 · 1 + 0 = 12. Consider a sequence of subsets of R, E 0 , E 1 , . . ., where E 0 = [0, 1] and 1 E is obtained from E 0 by removing the middle third (1/3, 2/3) of E 0 .

May 2, 2018 11:28

442

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 442

Mathematical Analysis for Machine Learning and Data Mining

If the remaining closed intervals are E01 and E11 , then E 1 is defined by E 1 = E01 ∪ E11 . By removing the middle intervals from the sets E01 and E11 , four new 2 2 2 2 2 2 2 , E01 , E10 , E11 are created. Let E 2 = E00 ∪ E01 ∪ E10 ∪ closed intervals E00 2 E11 . E n is constructed from E n−1 by removing 2n−1 disjoint middle third intervals from E n−1 (see Figure 7.2). Namely, if Ein0 ···in−1 is an interval of the set E n , by removing the middle third of this interval, we generate two and Ein+1 . closed intervals Ein+1 0 ···in−1 0 0 ···in−1 1 In general, En is the union of 2n closed intervals  {Ein0 ,...,in−1 | (i0 , . . . , in−1 ) ∈ {0, 1}n}, En = i0 ,...,in−1

for n  0. An argument by induction on n ∈ N shows that Ein0 ···in−1

! 2wn (i0 , . . . , in−1 ) 2wn (i0 , . . . , in−1 ) + 1 = , . 3n 3n

Indeed, the equality above holds for n = 0. Suppose that it holds for n, and denote by a and b the endpoints of the interval Ein0 ···in−1 ; that is, 2wn (i0 , . . . , in−1 ) , 3n 2wn (i0 , . . . , in−1 ) + 1 b= . 3n By the inductive hypothesis, the points that divide Ein0 ···in−1 are a=

2a + b 6wn (i0 , . . . , in−1 ) + 1 = 3 3n+1 2wn+1 (i0 , . . . , in−1 , 0) + 1 = 3n+1 and 6wn (i0 , . . . , in−1 ) + 2 a + 2b = 3 3n+1 2wn+1 (i0 , . . . , in−1 , 1) = . 3n+1 Thus, the remaining left third of Ein0 ···in−1 is

! 2wn (i0 , . . . , in−1 ) 2wn+1 (i0 , . . . , in−1 , 0) + 1 , 3n 3n+1 ! 2wn+1 (i0 , . . . , in−1 , 0) 2wn+1 (i0 , . . . , in−1 , 0) + 1 = , , 3n+1 3n+1

Ein+1 = 0 ···in−1 0

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 443

443

while the remaining right third is

! 2wn+1 (i0 , . . . , in−1 , 1) 2wn (i0 , . . . , in−1 ) + 1 , 3n+1 3n ! 2wn+1 (i0 , . . . , in−1 , 1) 2wn+1 (i0 , . . . , in−1 , 1) + 1 , , = 3n+1 3n+1

Ein+1 = 0 ···in−1 1

which concludes the inductive argument. 0

1

E01

2 E00

E11

2 E01

2 E10

E0

E1 2 E11

E2

E3 .. . Fig. 7.2

Construction of the Cantor dust.

Each number x located in the leftmost third E01 = [0, 1/3] of the set E0 = [0, 1] can be expressed in base 3 as a number of the form x = 0.0d2 d3 · · · ; the number 1/3, the right extreme of this interval, can be written either as x = 0.1 or x = 0.022 · · · . We adopt the second representation which allows us to say that all numbers in the rightmost third E11 = [2/3, 1] of E 0 have the form 0.2d2 d3 · · · in the base 3. 2 2 2 2 , E01 , E10 , E11 obtained The argument applies again to the intervals E00 1 2 from the set E . Every number x in the interval Eij can be written in base 3 as x = 0.i j  · · · , where i = 2i and j  = 2j. The Cantor set is the intersection  C = {E n | n  0}. Let us evaluate the total length of the intervals of which a set of the form En consists. There are 2n intervals of the form Ein0 ···in−1 , and the length of each of these intervals is 31n . Therefore, the length of E n is (2/3)n , so this

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 444

Mathematical Analysis for Machine Learning and Data Mining

444

length tends toward 0 when n tends towards infinity. This implies that the Cantor set is a null set. Despite its sparsity, surprisingly, the Cantor set is equinumerous with the interval [0, 1], so it is uncountable. To prove this fact, observe that the Cantor set consists of the numbers x that can be expressed as x=

∞  an , 3n n=1

where an ∈ {0, 2} for n  1. For example, 1/4 is a member of this set because 1/4 can be expressed in base 3 as 0.020202 · · · . Define the function g : C −→ [0, 1] by g(x) = y if x = 0.a1 a2 · · · (in base 3), where ai ∈ {0, 2} for i  1 and y = 0.b1 b2 · · · (in base 2), where bi = ai /2 for i  1. It is easy to see that this is a bijection between C and [0, 1], which shows that these sets are equinumerous. We now examine the behavior of the sets ! 2wn (i0 , . . . , in−1 ) 2wn (i0 , . . . , in−1 ) + 1 , Ein0 ···in−1 = 3n 3n relative to two mappings f0 , f1 : [0, 1] −→ [0, 1] defined by f0 (x) =

x x+2 and f1 (x) = 3 3

for x ∈ [0, 1]. Note that f0 (Ein0 ···in−1 )

! 2wn (i0 , . . . , in−1 ) 2wn (i0 , . . . , in−1 ) + 1 = , 3n+1 3n+1 ! 2wn+1 (0i0 , . . . , in−1 ) 2wn+1 (0i0 , . . . , in−1 ) + 1 = , 3n+1 3n+1 n+1 = E0i . 0 ···in−1

Similarly, n+1 . f1 (Ein0 ···in−1 ) = E1i 0 ···in−1

Thus, in general, we have fi (Ein0 ···in−1 ) = Eiin+1 for i ∈ {0, 1}. 0 ···in−1 This allows us to conclude that E n+1 = f0 (E n ) ∪ f 1 (E n ) for n ∈ N.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 445

445

Since both f0 and f1 are injective, it follows that   En = E n+1 C= n1

=



n0

[f0 (E ) ∪ f1 (E n )] n

n0

⎛



=⎝

⎞

f0 (E n )⎠ ∪ ⎝

n0

⎛

= f0 ⎝

⎛

 n0

⎞



⎞ f1 (E n )⎠

n0

⎛

E n ⎠ ∪ f1 ⎝



⎞ En⎠ .

n0

2 In Figure 7.3 we show how sets of the form Eij are mapped into sets of 3 the form Eijk by f0 (represented by plain arrows) and f1 (represented by dashed arrows). 2 E00

f0 / 3 E000

2 E01

 3 E001

2 E10     9  9 z z 3 3 3 3 E010 E011 E100 E101

Fig. 7.3

2 E11

j 3 E110

f1 N 3 E111

2 into sets E 3 . Mapping sets Eij ijk

Definition 7.24. Let (S, E, m) be a measure space and let f : S −→ R be a function. An upper bound for f is a number a ∈ R such that f (x)  a for x ∈ S. A lower bound for f is a number b ∈ R such that b  f (x) for x ∈ S. Equivalently, a is an upper found for f if f −1 (a, ∞) = ∅. Similarly, b is a lower bound for f if f −1 (−∞, b) = ∅. Let Uf the set of upper bounds for f and let Lf the set of lower bounds for the same function. Definition 7.25. The supremum of f is the number sup f defined by  inf Uf if Uf = ∅, sup f = ∞ if Uf = ∅. The infimum of f is the number inf f defined by  sup Lf if Lf = ∅, inf f = −∞ if Lf = ∅.

May 2, 2018 11:28

446

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 446

Mathematical Analysis for Machine Learning and Data Mining

If the function f is measurable, then we can define the essential supremum and the essential infimum of f that are similar to sup f and inf f . Definition 7.26. Let (S, E, m) be a measure space and let f : S −→ R be a measurable function. An essential upper bound of f is a number a such that m(f −1 (a, ∞)) = 0. An essential lower bound of f is a number b such that m(f −1 (−∞, b)) = 0. The sets of essential upper bounds and essential lower bounds for f are denoted by Ufess and Less f , respectively. The essential supremum for f is ess sup f given by  inf Ufess if Ufess = ∅, ess sup f = ∞ otherwise. The essential infimum for f is the number ess inf f given by  if Less sup Less f f = ∅, ess inf f = −∞ otherwise. Note that if the function f is modified on a null set, then ess sup f and ess inf f do not change. Example 7.15. Le f : R −→ R be the function given by:  2 if x ∈ Q, f (x) = arctan x if x ∈ R − Q. It is immediate that sup f = 2, ess sup f =

π 2

and inf f = ess inf f = − π2 .

Note that ess sup f  sup |f |. Theorem 7.59. Let (I, LI , mL ) be a measure space where I is an interval of R and let f : I −→ R be a continuous function. Then ess sup f = supI |f |. Proof. Since ess sup f  supI |f | we need to prove only that supI |f |  ess sup f . If c ∈ R and c < supI |f | there exists x0 ∈ I such that c < |f (x0 )|. Since f is continuous, there exists δ > 0 such that (x0 − δ, x0 + δ) ∩ I ⊆ {x ∈ I | |f (x)| > c}, hence 0 < mL ((x0 − δ, x0 + δ) ∩ I)  mL ({x ∈ I | |f (x)| > c}), hence c  sup essf . This implies supI |f |  ess sup f .



May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 447

447

Theorem 7.60. Let (S, E, m) be a measure space. The collection E∗ defined as E∗ = {T ∈ P(S) | U ⊆ T ⊆ W where U, W ∈ E and m(W − U ) = 0}. is a σ-algebra; furthermore, if T ∈ E∗ and m∗ (T ) equals m(U ), then (S, E∗ , m∗ ) is a measure space. Proof. It is clear that E ⊆ E∗ and S ∈ E∗ . The measure m∗ is well-defined. Indeed, suppose that we have both U ⊆ T ⊆ W and U1 ⊆ T ⊆ W1 , where U, U1 , W, W1 ∈ E and m(W1 − U1 ) = m(W − U ) = 0. Since U − U1 ⊆ T − U1 ⊆ W1 − U1 , we have m(U − U1 ) = 0, hence μ(U ) = m(U ∩U1 ). Similarly, m(U1 ) = m(U ∩U1 ), so m(U1 ) = m(U ). If U ⊆ T ⊆ W , then S −W ⊆ S −T ⊆ S −U . Since (S −U )−(S −W ) = W − U , it follows that S − T ∈ E∗ . Suppose that Un ⊆ Tn ⊆ Wn , where Un , Wn ∈ E and m(Wn − Un ) = 0    for n ∈ N. If U = n∈N Un , W = n∈N Wn , and T = n∈N Tn , then U ⊆ T ⊆ Wn , U, W ∈ E and   Un = (W − Un ), W −U =W − n∈N

n∈N

so m(W − U ) = 0 because a countable union of sets of measure 0 has measure 0. Thus, T ∈ E∗ . If the sets Tn are disjoint, the same is true for the sets Un , hence   ∗ m (T ) = m(U ) = n∈N m(Un ) = n∈N m∗ (Tn ), hence m∗ is a measure  on E∗ . Theorem 7.61. (Regularity Theorem for Measures on Rn ) Let (Rn , B(Rn ), m) be a measure space defined on Rn such that if U ∈ B(Rn ) is bounded then m(U ) < ∞. If U ∈ B(Rn ) and > 0, there exists a closed set C and an open set V such that C ⊆ U ⊆ V and m(V − C) < . If U ∈ B(Rn ) and m(U ) < ∞, then m(U ) = sup{m(K) | K is compact and K ⊆ U }. Proof. Consider the open-closed interval G = (a1 , b1 ] × · · · × (an , bn ]. The set Ik = (a1 , b1 + k1 ) × · · · × (an , bn + k1 ) is open and limk→∞ Ik = G. Since m(I1 ) is finite, by the Measure Continuity Theorem (Theorem 7.32) it follows that m(Ik − G) < when k is sufficiently large, which proves the first part of the theorem for open-closed intervals. We saw that open-closed intervals form a semi-ring (by Supplement 35 of Chapter 1). For an arbitrary set U ∈ B(Rn ) there exist open-closed

May 2, 2018 11:28

448

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 448

Mathematical Analysis for Machine Learning and Data Mining

  intervals Gk such that U ⊆ k Gk and m ( k Gk − G) < . Choose open  sets Vk such that Vk ⊆ Gk and m(Gk − Vk ) < 2 k . Then V = k Vk is open and m(V − U ) < 2 . This proves the first part of the theorem which concerns open sets. The case of the closed sets can be shown by taken the complementary sets. Since m(U ) < ∞, m(U − U0 ) < for some bounded subset U0 of S and, then, by the first part that m(U − K) < for a closed subset of U0 . Since  K is bounded and closed in Rn , it is compact. Definition 7.27. A collection of intervals C of R is a Vitali1 cover of a subset X of R if for every > 0 and x ∈ X there exists I ∈ C such that x ∈ I and mL (I) < . Theorem 7.62. (Vitali Theorem) Let X be a non-empty subset of R and let K be a Vitali cover of X that consists of non-trivial closed intervals (that is, of intervals that do not consist of one point). For every > 0 there exists a finite or infinite sequence of disjoint intervals (K1 , . . . , Kn , . . .) in  K such that μL (X − i1 Ki ) < . Proof. Suppose initially that X is bounded. Let U be a bounded open set such that X ⊆ U and let K0 be the collection of closed intervals of K that are included in U . It is clear that K0 is a Vitali covering of X. Define s1 = sup{mL (K) | K ∈ K0 }. We proceed to construct inductively a sequence of closed intervals (Kn ) in K0 . Since X = ∅ we have 0 < s1 < ∞ and we can select an interval K1 ∈ K0 that satisfies mL (K1 ) > s21 . n If X ⊆ j=1 Kj , the construction is completed. Otherwise, there exists n n x ∈ X − j=1 Kj and, therefore, since j=1 Kj is closed, and K0 is a Vitali covering of X, there exists an interval in K0 that contains x and is disjoint n from j=1 Kj . Define ⎧  ⎫  n ⎨ ⎬   Kj = ∅ . sn+1 = sup mL (K)  K ∈ K0 and K ∩ ⎩ ⎭  j=1

1 Giuseppe Vitali (26th of August 1875–29th of February 1932) was an Italian mathematician who worked in mathematical analysis. Vitali studied at the Scuola Normale Superiore in Pisa and graduated to the University of Pisa in 1899. After teaching in secondary schools he became a professor of calculus at the University of Modena in 1923. He also taught at the Universities of Padua and Bologna.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 449

449

We have 0 < sn+1 < ∞ and we can choose an interval Kn+1 such that mL (Kn+1 ) >

n  sn+1 and Kn+1 ∩ Kj = ∅. 2 j=1

 If this process ends in p steps, then X ⊆ pj=1 Kj and (K1 , . . . , Kp ) is the required sequence. Otherwise, that is, if the process does not terminate ∞ in a finite number of steps, the series j=1 mL (Kj ) must be convergent because the intervals K1 , . . . , Kn , . . . are disjoint and included in the bounded set U . Therefore, limn→∞ mL (Kn ) = 0 and, consequently, limn→∞ sn = 0. For each n let Kˆn be the interval in K with the same center as Kn such ˆ n ) = 5 mL (Kn ) (see Figure 7.4). Since mL (K ˆ n ) = 5mL (Kn ), that mL (K ∞ ˆ the series j=1 mL (Kn ) is also convergent. I

Kn ˆn K Fig. 7.4

When I ∩ Kn = ∅ and mL (Kn ) >

1 m (I) 2 L

ˆ n. then I ⊆ K

  Let x ∈ X − pj=1 Kj . Since pj=1 Kj is closed and and K0 is a Vitali cover of X, there are intervals in K0 that contain x and are disjoint from p j=1 Kj . Let I be such an interval. Since mL (I) < sk+1 holds for each k k such that I ∩ n=1 Kn = ∅, and limk→∞ sk = 0, if follows that I ∩ k j=1 Kj = ∅ when k is sufficiently large. Let k0 be the smallest k such that  I ∩ kj=1 Kj = ∅. Then mL (I)  sk0 and, therefore, mL (I)  2mL (Kk0 ) because sk0 /2  mL (Kk0 ). Since I ∩ Kk0 = ∅, the definition of the intervals ˆ n implies that I ⊆ K ˆ k0 . Since I is disjoint from p Kj , it follow that K j=1 ˆ n . Since x was an arbitrary ˆ k0 ⊆ ∞ K k0  p + 1. Thus, x ∈ I ⊆ K n=N +1 p element of X − n=1 Kn , it follows that X−

p  n=1

Kn ⊆

∞ 

ˆn K

n=p+1

for every positive integer p. Therefore,     p ∞ ∞    ˆ n ). μL X − K n  μL X − Kn  mL (K n=1

n=1

n=p+1

May 2, 2018 11:28

450

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 450

Mathematical Analysis for Machine Learning and Data Mining

∞ ˆ n ) implies that Since the convergence of the series n=1 mL (K ∞ ˆ n ) = 0, it follows that μL (X − ∞ Kn ) = 0, so limp→∞ n=p+1 mL (K n=1 (Kn ) is the required sequence. This concludes the argument when X is bounded. When X is not bounded, for each m such that Xm = X ∩ (−m, m) = ∅, we can apply the previous argument and find a sequence of disjoint intervals  Kkm that belong to K such that j Kjm ⊆ (−m, m) and contains almost every point in X ∩ (−k, k). Merging these sequences results in a sequence for X. 

7.8

Measures on Topological Spaces

We examine the relationships that exists between a topological space and a measure space defined on the same set. The following definition comprises the main concepts: Definition 7.28. Let (S, O) be a Hausdorff topological space, B(S) be the σ-algebra of Borel sets, and let (S, B(S, O), m) be a measure space. The measure m is: (i) a Borel measure if m(C) < ∞ for every C ∈ COMP(S, O); (ii) a locally finite measure if for every x ∈ S there exists U ∈ neighx (O) such that m(U ) < ∞; (iii) a inner regular measure if for every U ∈ B(S, O) we have: m(U ) = sup{m(C) | C ∈ COMP(S, O), C ⊆ U }; (iv) an outer regular measure if for every U ∈ B(S, O) we have: m(U ) = inf{m(D) | D ∈ O, U ⊆ D}. Theorem 7.63. Every locally finite measure is a Borel measure. Proof. Let C ∈ COMP(S, O). Each x ∈ C has an open neighborhood Vx with m(Vx ) < ∞. Since C is compact there exists a finite family n n Vx1 , . . . , Vxn such that C ⊆ j=1 Vxj . Therefore, m(C)  m( j=1 Vxj )  n  j=1 m(Vxj ) < ∞. Definition 7.29. Let (S, O) be a Hausdorff topological space. A measure defined on B(S, O) is a Radon measure if it is locally finite and inner regular. Theorem 7.64. Let (S, O) be a Hausdorff topological space. If every point has a countable neighborhood basis, every inner regular Borel measure m is also locally finite, and therefore, is a Radon measure.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Measurable Spaces and Measures

b3234-main

page 451

451

Proof. Suppose that m is not local finite. This means that there exists x ∈ S such that m(V ) = ∞ for every open neighborhood V of x. Since x has a countable neighborhood basis (Vn )n1 that consists of open sets, n we have a countable neighborhood basis (Vn ) such that Vn = j=1 Vn such  that m(Vn ) = ∞ for n  1 and n1 Vn = {x}. Since m(Vn ) = ∞ and m is inner regular, there exists a compact set Cn such that Cn ⊆ Vn such that m(Cn ) > n for every n  1. Let C =  {x} ∪ n1 Cn . The set C is compact. Indeed, suppose that C is an open cover of C. There exists some U ∈ C such that x ∈ U . Since {Vn | n  1} is a neighborhood basis at x, there exists n0 such that Vn 0 ⊆ U for some n0  1. Then, Cn ⊆ Vn ⊆ Vn 0 ⊆ U for n  n0 . Since C1 ∪ · · · ∪ Cn0 is a compact subset of C, it is covered by finitely many sets in C. These sets together with U constitute a finite covering of C. Thus, we have m(C) < ∞ because m is a Borel measure, and, since Cn ⊆ C, m(C)  m(Cn ) > n for every n  1, which is a contradiction.  Let (S, O) be a topological space and let C(S, R) be the set of continuous functions from S to R. Denote by Fcb (S, R) the subset of C(S, R) that consists of continuous bounded functions. Note that for any measurable space (S, E) the following hold: (i) the function φ : C(S, R) −→ Fcb (S, R) defined by φ(f ) = arctan f is injective; (ii) φ(f ) is continuous if and only if f is continuous; (iii) φ(f ) is measurable as a function between (S, E) and (R, B(R)) if and only if f is measurable as a function between the same measurable spaces. The second point follows immediately from the equalities φ(f ) = arctan f and f = tan φ. To prove the third point let B be a Borel set in R. We have φ(f )−1 (B) = f −1 ({tan(x) | x ∈ B}) and f −1 (B) = (φ(f ))−1 ({arctan(x) | x ∈ B}). Note that both {tan(x) | x ∈ B} and {arctan(x) | x ∈ B} are Borel sets (by Theorem 7.13), so the equivalence of the measurability of f and φ(f ) follows. Definition 7.30. Let (S, O) be a topological space. The σ-algebra of Baire sets is the smallest σ-algebra BA(S) such that all functions f ∈ Fcb (S, R) are measurable as functions between the measurable spaces (S, BA(S)) and (R, B(R)).

May 2, 2018 11:28

452

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 452

Mathematical Analysis for Machine Learning and Data Mining

We have BA(S) ⊆ B(S). Theorem 7.65. In a topological metric space (S, Od ) we have BA(S) = B(S). Proof. Let V be a closed set in (S, O) and let f : S −→ R be the function defined by f (x) = d(x, V ) = inf{d(x, v) | v ∈ V }. The inequality (5.1), |d(x, V ) − d(y, V )|  d(x, y), implies the continuity of f ; moreover, V = f −1 ({0}), so V is a Baire set. Since all closed subsets of (S, Od ) are Baire set, it follows that all Borel sets are Baire sets and, so B(S) ⊆ BA(S). Coupled with the reverse inclusion stated above, we have BA(S) = B(S).  Definition 7.31. Let (S, O) be a topological space and let (S, E, m) be a measure space on the same set S. A subset U ∈ E is regular if m(U ) = sup{m(C) | C ∈ COMP(S, O) ∩ E and C ⊆ U }. The measure m is tight if m(S) = sup{m(C) | C ∈ COMP(S, O) ∩ E}. If U ∈ E is regular, then m(C)  m(U ) for any compact set included in U ; also, for every > 0 there exists a compact set C such that m(U ) −  m(C)  m(U ). Theorem 7.66. Let (S, O) be a Hausdorff topological space and let (S, E, m) be a measure space on the same set S, where m is finite and tight. The collection F = {F ∈ E | both F and S − F are regular} is a σ-algebra. Proof. Since m is tight, S is regular; also, we have ∅ ∈ F, hence S ∈ F. Let U1 , U2 , . . . , Un , . . . be a countable collection of sets in F and let  U = n∈N Un . For each n  1, both Un and S − Un are regular sets, so m(Un ) = sup{m(C) | C ∈ C(S, O) ∩ E and C ⊆ Un } and m(S − Un ) = sup{m(D) | D ∈ C(S, O) ∩ E and D ⊆ S − Un }.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Measurable Spaces and Measures

b3234-main

page 453

453

Let > 0. For n  1 there exist disjoint compact sets Cn and Dn such that Cn ⊆ Un and Dn ⊆ S − Un , m(Un − Cn ) < n , and m((S − Un ) − Dn ) < n . 2 2 There exists m such that: ⎛ ⎞  m⎝ Cn ⎠ > m(U ) − . 2 

1nm

The set C = nm Cn is compact, C ⊆ U and m(C)  m(U ) − .   The set D = n∈N Dn is compact, m((S − U ) − D)  n1 2 n = . Thus, C and S − C are regular, so F is a σ-algebra.  7.9

Measures in Metric Spaces

In this section we discuss the interaction between metrics and measures defined on metric spaces. Recall that the dissimilarities between subsets of a metric space was introduced in Section 5.4. Definition 7.32. Let (S, d) be a metric space. A Carath´eodory outer ˆ 0 such measure on (S, d) is an outer measure on S, μ : P(S) −→ R that for every two subsets U, V of S such that d(U, V ) > 0 we have μ(U ∪ V ) = μ(U ) + μ(V ). Theorem 7.67. Let (S, d) be a metric space. An outer measure μ on S is a Carath´eodory outer measure if and only if every closed set of (S, Od ) is μ-measurable. Proof. Suppose that every closed set is μ-measurable and let U, V be two subsets of S such that d(U, V ) > 0. Consider the closed set K(B(U, r)), where r = d(u,v) 2 . Clearly, we have U ⊆ K(B(U, r)) and V ⊆ S − K(B(U, r)). Since K(B(U, r)) is a μ-measurable set, we have: μ(U ∪V ) = μ((U ∪V )∩K(B(U, r)))+μ((U ∪V )∩K(B(U, r))) = μ(U )+μ(V ), so μ is a Carath´eodory outer measure. Conversely, suppose that μ is a Carath´eodory outer measure, that is, d(U, V ) > 0 implies μ(U ∪ V ) = μ(U ) + μ(V ).

May 2, 2018 11:28

454

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 454

Mathematical Analysis for Machine Learning and Data Mining

Let U be an open set, L be a subset of U , and let L1 , L2 , . . . be a sequence of sets defined by:     1  , Ln = t ∈ Ld(t, K(U ))   n for n  1. Observe that L1 , L2 , . . . is an increasing sequence of sets, so the sequence μ(L1 ), μ(L2 ), . . . is increasing. Therefore, limn→∞ μ(Li ) exists and limn→∞ μ(Li )  μ(L). We claim that limn→∞ μ(Li ) = μ(L).  Since every set Ln is a subset of L, it follows that n1 Ln ⊆ L. Let t ∈ L ⊆ U . Since U is an open set, there exists > 0 such that B(t, ) ⊆ U , so d(t, K(U ))  n1 if n > 1 . Thus, for sufficiently large values of n, we have   t ∈ Ln , so L ⊆ n1 Ln . This shows that L = n1 Ln . Consider the sequence of sets Mn = Ln+1 − Ln for n  1. Clearly, we can write: ∞ ∞ ∞    Mk = L2n ∪ M2p ∪ M2p+1 , L = L2n ∪ p=n

k=2n

p=n

so μ(L)  μ(L2n ) + If both series lim

n→∞

∞ p=1 ∞ 

∞  p=n

μ(M2p ) and

μ(M2p ) + ∞ p=1

μ(M2p+1 ),

p=n

μ(M2p+1 ) are convergent, then

μ(M2p ) = 0 and lim

p=n

∞ 

n→∞

∞ 

μ(M2p+1 ) = 0,

p=n

and so μ(L)  limn→∞ μ(L2n ). ∞ If the series p=n μ(M2p ) is divergent, let t ∈ M2p ⊆ L2p+1 . If z ∈ 1 by the definition of L2p+1 . Let y ∈ M2p+2 ⊆ K(U ), then d(t, z)  2p+1 L2p+3 . We have 1 1 > d(y, z) > , 2p + 2 2p + 3 so d(t, y)  t(t, z) − d(y, z) 

1 1 − , 2p + 1 2p + 2

which means that d(M2p , M2p+2 ) > 0 for p  1. Since μ is a Carath´eodory outer measure, we have:  n  n   μ(M2p ) = μ M2p  μ(L2n ). p=1

p=1

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Measurable Spaces and Measures

b3234-main

page 455

455

This implies limn→∞ μ(Ln ) = limn→∞ μ(L2n ) = ∞, so we have in all cases limn→∞ μ(An ) = μ(L). Let F be a closed set in (S, Od ) and let V be an arbitrary set. The set V ∪ K(F ) is contained in the set K(F ) = F , so, by the previous argument, there exists a sequence of sets Ln such that d(Ln , F )  n1 for each n and limn→∞ μ(Ln ) = μ(V ∩ K(F )). Consequently, μ(V )  μ((V ∩ F ) ∪ Ln ) = μ(V ∪ F ) + μ(Ln ). Taking the limit we obtain μ(V )  μ(V ∩ F ) + μ(V ∩ K(F )), which proves that F is μ-measurable.  Corollary 7.23. Let (S, d) be a metric space. An outer measure on S is a Carath´eodory outer measure if and only if every Borel subset of S is μ-measurable. Proof.

This statement is an immediate consequence of Theorem 7.67. 

Let (S, d) be a metric space and let C be a countable collection of subsets of S. Define Cr = {C ∈ C | diam(C) < r} and assume that for every x ∈ S and r > 0 there exists C ∈ Cr such that ˆ 0 be a function and let μφ,r be the outer measure x ∈ C. Let φ : C → R constructed using the method described in Example 7.12. This construction yields an outer measure that is not necessarily a Carath´eodory outer measure. When r decreases, μφ,r increases. This allows us to define μ ˆφ = lim μφ,r . r→0

We shall prove that the measure μ ˆφ is a Carath´eodory outer measure. Since each measure μφ,r is an outer measure, it follows immediately that μ ˆφ is an outer measure. Theorem 7.68. Let (S, d) be a metric space, C be a countable collection of ˆ 0 . Then, μ ˆφ is a Carath´eodory outer measure. subsets of S, and f : C → R Proof. Let U, V be two subsets of S such that d(U, V ) > 0. We need to ˆφ (U ) + μ ˆφ (V ). show only that μ ˆφ (U ∪ V )  μ Choose r such that 0 < r < d(U, V ) and let D be an open cover of U ∪ V that consists of sets of Cr . Each set of D can intersect at most one of the set U and V . Therefore, D is a disjoint union of two collections, D = DU ∪ DV , where DU is an open cover for U and DV is an open cover

May 2, 2018 11:28

456

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 456

Mathematical Analysis for Machine Learning and Data Mining

for V . Then,    {φ(D) | D ∈ DV } {φ(D) | D ∈ D} = {φ(D) | D ∈ DU } +  μφ,r (U ) + μφ,r (V ). This implies μφ,r (U ∪ V )  μφ,r (U ) + μφ,r (V ), which yields μ ˆφ (U ∪ V )  ˆφ (V ) by taking the limit for r → 0.  μ ˆφ (U ) + μ 7.10

Signed and Complex Measures

The notion of measure can be extended by allowing a measure to range over the extended set of real numbers. Definition 7.33. Let (S, E) be a measurable space. A signed measure is a ˆ that satisfies the following conditions: function m : E −→ R (i) m(∅) = 0; (ii) Ran(m) contains only one of the values ∞ and −∞; (iii) for every countable collection {U0 , U1 , . . .} of sets in E that are pairwise disjoint one of the following cases may occur:   (a) if m( n∈N ) is finite, then the series n∈N m(Ui ) is absolutely convergent and ⎞ ⎛   Un ⎠ = m(Un ); m⎝ n∈N

n∈N

  (b) if m( n∈N ) is not finite, then the series n∈N m(Un ) is divergent (the additivity property of signed measures). We refer to the triple (S, E, m) as a signed measure space. The set of signed, real-valued measures over a measurable space (S, E) is denoted by RM(S, cale). The set RM(S, E) is a real linear space relative the addition of real-valued measures and multiplication by scalars. Example 7.16. Let (S, E) be a measurable space and let m1 , m2 be two measures defined on E such that at least one of these measures is finite. ˆ given by m(U ) = m1 (U ) − m2 (U ) for Then, the function m : E −→ R U ∈ E is a signed measure. The requirement that at least one of m1 , m2 is finite ensures that m is well-defined.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 457

457

Definition 7.34. Let (S, E, m) be a signed measure space. A set U ∈ E is positive if for every set W such that W ∈ E and W ⊆ U we have m(W )  0. Similarly, U ∈ E is negative if for every set W such that W ∈ E and W ⊆ U we have m(W )  0. A set U ∈ E is an m-null for a signed measure m if it is both positive and negative. Both positivity and negativity are hereditary properties; in other words, if a set is positive (negative), then so is any of its measurable subsets. Note that the requirement that a set is m-null, where m is a signed measure, is more stringent than the usual definition of null sets for measures. A set U is m-null if and only if for each of its measurable subsets W we have m(W ) = 0. It is clear that for every m-null set U we have m(U ) = 0. However, the inverse is not true: a set of measure 0 is not necessarily an m-null set in the sense of Definition 7.34 as the next example shows. Example 7.17. Let (S, P(S)) be a measurable space and let x1 and x2 be two distinct elements of S. Consider the Dirac measures δx1 and δx2 defined on P(S) (as introduced in Example 7.6). It is easy to verify that ˆ defined by m(U ) = δx1 (U ) − δx2 (U ) is a signed measure on m : P(S) −→ R P(S), so (S, P(S), m) is a signed measure space. Then, we have m({x1 , x2 }) = δx1 ({x1 , x2 }) − δx2 ({x1 , x2 }) = 1 − 1 = 0. However, {x1 , x2 } is not an m-null set because for its subsets {x1 } and {x2 } we have m({x1 }) = 1 and m({x2 }) = −1. Theorem 7.69. Let (S, E, m) a signed measure space and let U =  (U0 , . . . , Un , . . .) be a sequence of positive (negative) sets. Then, U = n∈N is a positive (negative) set. Proof. Let W be a subset of U . (V0 , . . . , Vn , . . .) as

Define the sequence of sets V =

Vn = W ∩ Un ∩

n−1 

Ui ,

i=0

for n  0. Any set Vn is a measurable subset of Un and, therefore, m(Vn )  0.  Furthermore, since the sets Vn are pairwise disjoint and W = n∈N Vn , it  follows that m(W ) = n∈N m(Vn )  0, so U is indeed a positive set. The argument for the union of negative set is similar. 

May 2, 2018 11:28

458

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 458

Mathematical Analysis for Machine Learning and Data Mining

Theorem 7.70. Let (S, E, m) a signed measure space and let U ∈ E be a set such that 0 < m(U ) < ∞. Then, U contains a positive subset T such that m(T ) > 0. Proof. If U itself is positive, then we can take T = U . Therefore, we need to consider only the case when U is not positive, that is, the case when U contains some subsets of negative measure. We construct inductively a sequence of sets V = (V0 , V1 , . . . , Vn , . . .) as follows. By the assumption made above there exists a subset V0 of U such that m(V0 ) < 0. Let n0 be the least natural number such that m(V0 )  − n10 . Suppose that we constructed the sets V0 , . . . , Vi−1 . Let ni be the least natural number such that there exists a set Wi ∈ E such that Wi ⊆ U − i−1 1 j=1 Wj and m(Wi )< − ni . ∞ Define T = U − i=0 Wi . We have U =T ∪

∞ 

Wi ,

i=0

and the sets and since the sets Wi are are pairwise disjoint, this implies m(U ) = m(T ) +

∞ 

m(Wi ).

i=0

∞ The finiteness of m(U ) means that the series i=0 m(Wi ) is absolutely  1 convergent. Since n1i < |m(Wi )|, the series ∞ i=0 ni is convergent which, in turn, implies limi→∞ ni = ∞. Further, we have m(T ) > 0. To show that T is a positive set let be a positive number. We shall prove that T contains no measurable subset Z such that m(Z) < − . Suppose that such a set Z exists. Choose k such that 1 < nk − 1

and observe that T ⊆ U − have

k−1 i=0

Wi . If such a set Z would exist we would

m(Z) < − < −

1 , nk − 1

and this would contradict the definition of nk as the least natural number  such that there exists a subset of U − k−1 j=0 Wj whose measure is less than 1 − nk . Since this is true for every > 0, it follows that T contains no subsets of negative measure, so T is positive. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Measurable Spaces and Measures

b3234-main

page 459

459

Theorem 7.71 (Hahn Decomposition Theorem). Let (S, E, m) be a signed measure space. There exists a two-block partition of S, π = {B+ , B− } such that B+ ∈ E is a positive set for m and B− ∈ E is a negative set. Proof.

Suppose that ∞ is not in the range of m. Define M = sup{m(U ) | U ∈ E and m(U )  0}.

We have M  0 because m(∅) = 0. Let U = (U0 , . . . , Un , . . .) be a sequence of sets in E such that  We claim that limn→∞ m(Un ) = M and let U = ∞ n=0 m(Ui ) ∈ E. m(U ) = M . By Theorem 7.69, U is a positive set, so m(U )  M . Since U − Un is a subset of U , it follows that m(U − Un )  0. Consequently, m(U ) = m(Un ) + m(U − Un )  m(Un ), which implies m(U )  m(Un ) for n ∈ N. Therefore, m(U ) = M , which justifies the previous claim. In addition, this means that M < ∞. Consider now the set V = S − U . We claim now that V is a negative set. Suppose that V is not negative. Then it contains a positive set W . Since W is disjoint from U , it follows that U ∪ W is a positive set, so M  m(U ∪ W ) = m(U ) + m(W ) > M + m(W ), which is impossible. Thus, V is a negative set. The partition π is obtained now as B+ = U and B− = V .  The two-block partition π whose existence was established in Theorem 7.71 is known as a Hahn decomposition of the signed measure space (S, E, m). The Hahn decomposition is not unique as the next example shows. Example 7.18. Let (S, P(S), m) be the signed measure space introduced in Example 7.17, where we have the strict inclusion {x1 , x2 } ⊂ S. The range of m is the set {−1, 0, 1}. Define B+ = S − {x2 } and B− = {x2 }. The partition {B+ , B− } is a  = {x1 } Hahn decomposition for (S, P(S), m). On the other hand, if B+    and B− = S − {x1 }, then {B+ , B− } is a distinct Hahn decomposition of this signed measure space. Definition 7.35. Let (S, E) be a measurable space and let m0 , m1 : E −→ ˆ 0 be two measures. The measures m0 and m1 are mutually singular if R there exists a subset T ∈ E such that m0 (T ) = m1 (S − T ) = 0. This is denoted by m0 ⊥ m1 .

May 2, 2018 11:28

460

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 460

Mathematical Analysis for Machine Learning and Data Mining

Theorem 7.72. (Jordan’s Decomposition Theorem) A Hahn decomposition of a signed measure space (S, E, m) generates a unique pair of mutually singular measures such that m is the difference of these mutually singular measures. Any two mutually singular measures on (S, E, m) generate a Hahn decomposition of a signed measure space. Proof. Let π = {B+ , B− } be a Hahn decomposition of (S, E, m). Define the measures m+ (U ) = m(U ∩ B+ ), m− (U ) = −m(U ∩ B− ), for U ∈ E. Observe that the measures m+ and m− are mutually singular because m+ (B− ) = 0 and m− (S − B− ) = m− (B+ ) = 0, which shows that the role of the set T can be played by the set B− . Moreover, we have m(U ) = m(U ∩ B+ ) + m(U ∩ B− ) = m+ (U ) − m− (U ), for every U ∈ E. This shows that we can decompose any signed measure into the difference of two measures. Conversely, suppose that q0 , q1 are two mutually singular measures on the measurable space (S, E) such that q0 (T ) = q1 (S − T ) = 0 and let ˆ be the signed measure q(U ) = q0 (U ) − q1 (U ) for U ∈ E. We q : E −→ R claim that {S − T, T } is a Hahn decomposition of the signed measure space (S, E, q). Note that q(T ) = −q1 (T )  0 and q(S − T ) = q0 (S − T )  0. Suppose that Z is a subset of T . Since q0 and q1 are measures, we have q0 (Z)  q0 (T ) = 0 and q1 (Z)  q1 (T ). Therefore, q0 (Z) = 0 and q(Z) = q0 (Z) − q1 (Z)  0, so T is a negative set. If Y ⊆ S −T , then q1 (Y )  q1 (S −T ) = 0, so q1 (Y ) = q0 (Y ) − q1 (Y )  0, which proves that S − T is a positive set. Thus, {S − T, T } is indeed a Hahn decomposition of the signed measure space (S, E, q). Moreover, q0 (U ) = q0 (U ∩T )+q0 (U ∩(S−T ) = q0 (U ∩(S−T )) = q(U ∩(S−T )) = q+ (U ) and q1 (U ) = q1 (U ∩ T ) + q1 (U ∩ (S − T )) = q1 (U ∩ T ) = −q(U ∩ T ) = q− (U ).  The decomposition of a signed measure m as a difference of two mutually singular measures is known as the Jordan decomposition of the signed measure m.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 461

461

Definition 7.36. The measure |m| defined by m(U ) = m+ (U ) + m− (U ) is called the absolute value or the total variation of m. A signed measure m is finite (σ-finite) if |m| is finite (σ-finite). Theorem 7.73. Let (S, E, m) be a signed measure space. The following statements hold for a set T ∈ E:  (i) |m|(T ) = sup{ nk=1 |m(Tj )| | {T1 , . . . , Tn } ∈ PARTE (T )}; (ii) sup{|m(U )| | U ∈ E[T ]}  |m|(T )  2 sup{|m(U )| | U ∈ E[T ]}; (iii) for every positive measure m1 such that |m(T )|  m1 (T ) we have |m|  m1 . Proof.

Let {T1 , . . . , Tn } ∈ PARTE (T )}. We have |m|(T ) = 

n  j=1 n 

|m|(Tj ) =

n 

(m− (Tj ) + m+ (Tj ))

j=1

(m− (Tj ) − m+ (Tj )) =

j=1

n 

|m(Tj )|.

(7.11)

j=1

If {B+ , B− } is a Hahn decomposition theorem of S, {T ∩ B+ , T ∩ B− } ∈ PARTE (T ) and |m(T ∩ B+ )| + |m(T ∩ B− )| = m(T ∩ B+ ) − m(T ∩ B− ) = m+ (T ) + m− (T ) = |m|(T ). Therefore, |m|(T ) 

n   |m(Tj )| | {T1 , . . . , Tn } ∈ PARTE (T )}. { k=1

This and inequality (7.11) imply the equality of the first part. If T ∈ E and U ∈ E[T ] we have {U, T − U } ∈ PARTE (T ) and therefore, |m|(T )  |m(U )| + |m(T − U )|  |m(U )|. Consequently, sup{|m(U )| | U ∈ E[T ]}  |m|(T ). If {B+ , B− } is a Hahn decomposition for S we have |m|(T ) = m+ (T ) + m− (T ) = m(T ∩ B+ ) − m(T ∩ B− ) = |m(T ∩ B+ )| + |m(T ∩ B− )|  2 sup{|m(U )| | U ∈ E[T ]}, which proves the second part of the theorem. Since sup{|m(U )| | U ∈ E[T ]}  |m|(T ), we have |m| is a positive measure with |m(T )|  |m|(T ) for T ∈ E.

May 2, 2018 11:28

462

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 462

Mathematical Analysis for Machine Learning and Data Mining

Let m1 be a positive measure m1 such that |m(T )|  m1 (T ) for every T ∈ E and let {B+ , B− } be a Hahn decomposition of S relative to m. Then, |m|(U ) = m(U ∩ B+ ) − m(U ∩ B− ) = |m(U ∩ B+ )| + |m(U ∩ B− )|  m1 (U ∩ B+ ) + m1 (U ∩ B− ) = m1 (U ), for every U ∈ E, hence |m|  m1 .



Theorem 7.74. Let (S, E, m) be a signed measure space. We have m(T ) = 0 for some T ∈ E (that is, T is an m-null set) if and only if |m|(T ) = 0. Proof. Note that T is a m-null set if and only if m(U ) = 0 for every U ∈ E[T ], which means that sup{m(U ) | U ∈ E[T ]} = 0. By Theorem 7.73, |m|(T ) = 0.  The next definition further extends the notion of measure to include measures whose values are complex numbers. Definition 7.37. Let (S, E) be a measurable space. A complex measure on E is a complex function m : E −→ C such that (i) m(∅) = 0, and ∞ ∞ (ii) m( j=1 Ej ) = j=1 m(Ej ) for every infinite sequence (En ) of disjoint sets in E. Note that a complex measure has no infinite values. Thus, a real measure can be regarded as a complex measure only if its values are finite. Jordan’s Decomposition Theorem can be naturally extended to complex measures. Suppose that m is a complex measure on S. We can write m(E) = m1 (E)+im2 (E), where m1 , m2 are signed measures on S. Further, by Jordan’s decomposition theorem there exists mutually singular measures m1 , m1 and m2 , m2 such that m1 = m1 − m1 and m2 = m2 − m2 . This allows us to write m(E) = m1 − m1 + i(m2 − m2 ), where m1 − m1 and m2 − m2 are signed measures. For a complex measure on a measurable space (S, E) let |m| : E −→ R be the real-valued function defined by:  n  |m(Ei )| | {E1 , . . . , En } |m|(E) = sup i=1



is a partition of E with Ei ∈ E .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 463

463

Theorem 7.75. For every measure space (S, E, m), where m is a complex measure m : E −→ C, the mapping |m| : E −→ R is a finite measure on (S, E). Proof. Observe that |m|(∅) = 0. Since m is a complex measure on S, as we shown before, can write m(E) = m1 (E) + im2 (E), where m1 , m2 are signed measures on S. Furthermore, there exist mutually singular measures m1 , m1 and m2 , m2 such that m1 = m1 − m1 and m2 = m2 − m2 , hence m(E) = m1 (E) − m1 (E) + i(m2 (E) − m2 (E)). Let C, D ∈ E, where C∩D = ∅, and let {B1 , . . . , Bn } be a finite partition of C ∪ D, where Bj ∈ E for 1  j  n. We have n 

|m(Bj )| 

j=1

n 

|m(Bj ∩ C)| +

j=1

n 

|m(Bj ∩ D)|

j=1

 |m|(C) + |m|(D). Since |m|(C ∪ D) = sup{

n 

|m(Bj )| | {B1 , . . . , Bn } is a partition of C ∪ D },

j=1

it follows that |m|(C ∪ D)  |m|(C) + |m|(D). To prove the reverse inequality, let {B1 , . . . , Bp } be a finite partition of C and let {B1 , . . . , Bq } be a partition of D. Since C and D are disjoint, {B1 , . . . , Bp , B1 , . . . , Bq } is a partition of C ∪ D. Thus, ⎧ ⎫ p q ⎨ ⎬  |m|(C ∪ D)  sup |m(Bj )| + |m(Bj )| ⎩ ⎭ j=1 j=1 ⎫ ⎧ p ⎬ ⎨ |m(Bj )| | {B1 , . . . , Bp } is a partition of C = sup ⎭ ⎩ j=1 ⎫ ⎧ q ⎬ ⎨ |m(Bj )| | {B1 , . . . , Bq } is a partition of D + sup ⎭ ⎩ j=1

= |m|(C) + |m|(D). Therefore |m|(C ∪ D) = |m|(C) + |m|(D), so m is finitely additive. By Jordan’s decomposition theorem we have |m|(E)  m1 (E) + m1 (E) + m2 (E) + m2 (E),

May 2, 2018 11:28

464

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 464

Mathematical Analysis for Machine Learning and Data Mining

which implies that |m| is finite. If (Un )n1 is a decreasing sequence of measurable sets such that  Un = ∅, then for m ∈ {m1 , m1 , m2 , m2 } we have limn→∞ m (Un ) = 0, hence limn→∞ |m|(Un ) = 0. Thus, by Supplement 46, |m| is countably additive.  For a complex measure m the measure |m| is known as the variation of m. The number |m|(S) is known as the total variation of m and is denoted by m. Let CM(S, E) be the collection of all complex measures on the measure space (S, E). The set CM(S, E) is a complex linear space relative the addition of complex-valued measures and multiplication by scalars. Furthermore, the total variation is a norm over this space. If m1 , m2 ∈ CM(S, E) the sum m1 + m2 and product with a scalar c ∈ C are defined as (m1 + m2 )(U ) = m1 (U ) + m2 (U ), (am1 )(U ) = am1 (U ) for each U ∈ E. The set CM(S, E) becomes a complex linear space and this space can be equipped with the norm  ·  defined as m = |m|(S). 7.11

Probability Spaces

Definition 7.38. A probability space is a measure space (Ω, E, P ), where P : E −→ [0, 1] is a measure such that P (Ω) = 1. We will refer to P as a probability measure or, simply, as a probability. In the context of probability spaces we refer to the subsets of Ω that belong to E as events. Example 7.19. Probability spaces formalize the notion of sets of experiments. Consider, for instance, throwing a coin. There are two possible outcomes, head or tail, denoted by h and t. If we define Ω = {h, t}, E = P(Ω) and P (h) = P (t) = 12 we obtain the probability space that describes the coin throwing experiment. Let (Ω, E, P ) and (Ω, E, Q) be two probability spaces that share the same Ω and σ-algebra of events E that is generated by a π-system C. If P (A) = Q(A) for all A ∈ C, then P (A) = Q(A) for all A ∈ E. This follows from Corollary 7.8 because both P and Q are finite measures. If A is an event of a probability space (Ω, E, P ) and P (A) = 1 we say that A occurs almost surely.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Measurable Spaces and Measures

b3234-main

page 465

465

Theorem 7.76. Let (Ω, E, P ) be a probability space and let B ∈ E be an event such that P (B) > 0. Define the probability conditioned by the event B, P (·|B) : E B −→ [0, 1] as P (A ∩ B) . P (A|B) = P (B) The triple (B, E B , P (·|B)) is a probability space. Proof. Let {An | n ∈ N} be a sequence of pairwise disjoint events in (Ω, E, P ). We have: ⎞ ⎛     ⎠ P s∈N An ∩ B ⎝ P An B = P (B) n∈N  P s∈N (An ∩ B) = P (B)   P (An ∩ B) = = n∈N P (An |B), P (B) n∈N

so P is countably additive. Since P (B|B) = 1, (B, E B , P (·|B)) is a probability space.  Let E1 = {B ∈ E | P (B) > 0}. The function P (·|·) : E × E −→ [0, 1]  given by P (A|B) = P P(A∩B) (B) , where A ∈ E and B ∈ E is the conditional probability defined by P . Theorem 7.77. (Total Probability Theorem) Let {Bn | n ∈ N} be a family of pairwise disjoint events in the probability space (Ω, E, P ) such  that n∈N Bn = Ω and P (Bn ) = 0 for n ∈ N. We have  P (A) = P (A|Bn )P (Bn ). n∈N

Proof. Note that the family of events {A ∩ Bn | n ∈ N} is a partition of A. This allows us to write ⎞ ⎛  Bn ⎠ A = A∩Ω = A∩⎝ =



n∈N

(A ∩ Bn ).

n∈N

Since the events of the form A ∩ Bn are pairwise disjoint, it follows that   P (A) = P (A ∩ Bn ) = P (A|Bn )P (Bn ). n∈N

n∈N



May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 466

Mathematical Analysis for Machine Learning and Data Mining

466

Definition 7.39. Let (Ω, E, P ) be a probability space and let A, B ∈ E. The events A and B are independent if P (A ∩ B) = P (A)P (B). Note that if A, B are events and P (B) = 0, then A, B are independent if P (A) = P (A|B). If A ∩ B = ∅, then A and B are independent if at least of them equals ∅. If A ⊆ B, then A, B are independent if A = ∅ or B = Ω. Example 7.20. Consider the probability space ([0, 1], B([0, 1]), P ), where P is obtained by defining P ([a, b]) = b − a. If 0  a  c  b  d  1, the independence condition for the events [a, b] and [c, d] P ([a, b] ∩ [c, d]) = P ([a, b]) · P ([c, d]) amounts to (b − c) = (b − a)(d − c). The concept of independent events can be extended to any finite collection of events. Definition 7.40. Let (Ω, E, P ) be a probability space and let A1 , . . . , An be events in E. These events are independent if for every k such that 2  k  n we have the equalities P (Ai1 ∩ Ai2 ∩ · · · ∩ Aik ) = P (Ai1 )P (Ai2 ) · · · P (Aik ). Example 7.21. Three events A1 , A2 , A3 are independent if each of the following equalities P (A1 ∩ A2 ) = P (A1 )P (A2 ),

(7.12)

P (A2 ∩ A3 ) = P (A2 )P (A3 ),

(7.13)

P (A1 ∩ A3 ) = P (A1 )P (A3 ),

(7.14)

P (A1 ∩ A2 ∩ A3 ) = P (A1 )P (A2 )P (A3 ).

(7.15)

It interesting to note that the satisfaction of equalities (7.12) - (7.14) is independent of the satisfaction of equality (7.15). Consider, for example, the events A1 = [a, b], A2 = [c, d], and A3 = [c, e] ∪ [k, h], where a < c < b < e < d < k < h. If (b − c) = (b − a)(d − c), A1 and A2 are independent. Since A1 ∩ A3 = [c, b] and A2 ∩ A3 = [c, e], A1 and A3 are independent if (b − c) = (b − a)[(e − c) + (h − k)]; also, A2 and A3 are independent if (e − c) = (d−c)[(e−c)+(h−k)]. The first two conditions imply d = e+h−k. Thus, the

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 467

467

events A1 , A2 , A3 are pairwise independent if and only if b−c = (b−a)(d−c), e − c = (d − c)2 , and d = e + h − k. Note that equality (7.15) amounts to b − c = (b − a)(d − c)[(e − c) + h − k]. In view of the fact that d = e + hk this equality becomes b − c = (d − c)2 .

(7.16)

If (d − c) = (d − c)2 , then equality (7.16) cannot be satisfied. Thus, also A1 , A2 , A3 are pairwise independent, the collection {A1 , A2 , A3 } is not independent. The σ-algebra generated by an event A, σ(A) = {∅, A, Ω − A, Ω} was introduced in Example 1.25. If A, B are independent events in the probability space, then any pair of events in {A, Ω − A} × {B, Ω − B} is independent. Indeed, note that P (A ∩ (Ω − B)) = P (A − (A ∩ B)) = P (A) − P (A ∩ B) = P (A) − P (A)P (B) = P (A)(1 − P (B)) = P (A)P (Ω − B), so A and Ω − B are independent events. Similar computations for the remaining pairs of {A, ω − A} × {B, Ω − B} lead to the conclusion that these pairs of events are independent. Thus, the notion of independence can be defined for σ-algebras. Definition 7.41. The collection of σ-algebras {Ei | 1  i  n} on Ω, where (Ω, Ei , P ) are probability spaces is independent if every set of n events A1 , . . . , An where Ai ∈ Ei for 1  i  n is independent. In the language of probabilities a measurable function between a probability space (Ω, E, P ) and (R, B(R), mL ) is known as a random variable.2 We denote such random variables using capital letters X, Y, . . .. Since random variables are special measurable functions, previous facts relevant to such functions apply to random variables. For instance, if X : Ω −→ R is a random variable defined on a probability space (Ω, E, P ), the σ-algebra generated by X is the collection EX = {X −1 (B) | B ∈ B(R)}, as we have shown in Theorem 7.3. 2 A random variable is clearly a function rather than a variable; however, the term “variable” was adopted broadly in probability theory, and we will continue to use it in this context.

May 2, 2018 11:28

468

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 468

Mathematical Analysis for Machine Learning and Data Mining

Example 7.22. The constant function X : E −→ R defined as X(ω) = c is a random variable for any probability space (Ω, E, P ) because  Ω if c ∈ B, −1 X (B) = ∅ otherwise, and the fact that {∅, Ω} ⊆ E. Thus, EX = {Ω, ∅}. Example 7.23. Suppose now that Ran(X) = {a, b}. The σ-algebra EX consists of {Ω, ∅, X −1(a), X −1 (b)}. Let X, Y are two random variable defined on a probability space (Ω, E, P ). Since |X − Y | is a random variable, |X − Y |−1 ({0}) ∈ E. If P (|X − Y |−1 ({0}) = 1, the random variables are equal almost surely. In this case, we say that X and Y are versions of each other. Definition 7.42. Let (Ω, E, P ) be a probability space and let X, Y be random variables defined on that space. The random variables X and Y are independent if the σ-algebras EX and EY they generate respectively are independent. Example 7.24. Let ([0, 1], B([0, 1]), mL ) be a probability space and let a, b ∈ [0, 1], where a < b. The σ-algebra generated by the random variable X[ a, b] = 1[a,b] is EX = {[0, 1], ∅, [a, b], [0, a) ∪ (b, 1]}. If c, d ∈ [0, 1] and c < d, consider the random variable X[ c, d] = 1[c,d]. If a < b  c < d, mL ([a, b] ∩ [c, d]) = 0 and X[ a, b], Xc,d are not independent because mL ([a, b]) = b − a > 0, mL ([c, d]) = d − c, and mL ([a, b] ∩ [c, d]) = 0. Thus, it is necessary for independence to have [a, b] ∩ [c, d] = ∅. If one of the interval is included in the other, it is easily seen that the independence condition is violated. Thus, the to obtain independence, the intersection [a, b] ∩ [c, d] must be a non-empty interval [p, q], and we must have (b − a)(d − c) = (q − p). Note that if a  c  b  q, then p = c and q = b, hence we must have (b − a)(d − c) = b − c. By Exercise 48, this condition suffices for the independence of X[a,b] and X[c,d]. The alternative, c  a  d  b implies (b − a)(d − c) = d − a. Definition 7.43. A statement on a probability space (Ω, E, P ) is a radom variable S : Ω −→ {0, 1}. If S(ω) = 1 we say that the statement holds on ω.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 469

469

A statement is true almost surely (a.s.) if FS = {ω ∈ Ω | S(ω) = 1} belongs to E and P (FS ) = 1. Theorem 7.78. Let X : Ω −→ R be a random variable on the probability space (Ω, E, P ). If h : R −→ R is a Borel measurable function, then Y = hX is a random variable on the same probability space. Proof. The statement follows from the equality Y −1 (U ) = X −1 (f −1 (U )) that holds for every Borel subset U of R.  Theorem 7.79. Let X1 , X2 , . . . be random variables on a probability space (Ω, E, P ). The following are random variables on the same probability space: (i) max{X1 , X2 } and min{X1 , X2 }; (ii) supn Xn and inf n Xn ; (iii) lim supn→∞ Xn and lim inf n→∞ Xn and Proof.

Observe that {ω ∈ Ω | max{X1 , X2 }(ω)  x} = {ω ∈ Ω | X1 (ω)  x} ∩ {ω ∈ Ω | X1 (ω)  x}, {ω ∈ Ω | min{X1 , X2 }(ω)  x} = {ω ∈ Ω | X1 (ω)  x} ∪ {ω ∈ Ω | X1 (ω)  x}

ˆ which proves the first part. for every x ∈ R, The second part follows from the fact that  {ω ∈ Ω | Xn  x} ∈ E {ω ∈ Ω | (sup Xn )(ω)  x} = n∈N

and {ω ∈ Ω | (inf Xn )(ω)  x} =



{ω ∈ Ω | Xn  x} ∈ E.

n∈N

Finally, since lim sup Xn = inf sup Xn n→∞

m nm

and lim inf Xn = sup inf Xn , n→∞

the last part follows.

m nm



Corollary 7.24. If (Xn (ω)) converges for almost every ω, then limn→∞ Xn is a random variable.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 470

Mathematical Analysis for Machine Learning and Data Mining

470

Proof. This statement follows from the fact that limn→∞ Xn lim supn→∞ Xn = lim inf n→∞ Xn .

= 

The conditional probability can be extended to involve an event in a probability space (Ω, E, P ) and a σ-algebra G included in F. Fix A ∈ F and define a measure m on G be m(B) = P (B ∩A) for B ∈ G. If P (B) = 0 then m(B) = 0, hence m # P . By Radon-Nikodym theorem (Theorem 8.58), there exists a G-measurable function f that is integrable 6 relative to P such that P (A ∩ B) = m(B) = B f dP . The function f is a random variable, is referred to as the conditional probability of A on the σ-algebra G, and will denoted by P (A|G). Thus, we can write 7 P (A|G) dP (7.17) P (A ∩ B) = B

for B ∈ G. Example 7.25. Let (Ω, E, P ) be a probability space and let π = {B1 , B2 , . . .} be a partition of Ω that consists of sets in E. If G is the σ-algebra generated by the sets in π, the function f : Ω −→ R defined by f (ω) =

P (A ∩ Bi if ω ∈ Bi P (Bi

is measurable relative to the σ-algebra G. If G ∈ G, then G is a union of   blocks of π, G = j∈J Bij , hence P (AG) = |∈J P(A|B| )P(B| ). Example 7.26. Let (Ω, E, P ) be a probability space. For A ∈ E we have P (A|E) = 1A with probability 1 because 1A satisfies equality (7.17). If G = {∅, Ω}, then P (A|E) = P (A) with probability 1 because every G-measurable function must be a constant.

Exercises and Supplements (1) Let (S, E) and (T, F) be two measurable spaces such that S ∩ T = ∅. Prove that the collection E ∨ F is a σ-algebra on the set S ∪ T . Furthermore, we have Kσ -alg (E ∪ F) = E ∨ F. Solution: If B ∈ E ∨ F, we have B = C ∪ D, where C ∈ E and D ∈ F. Then, (S ∪ T ) − B = (S − C) ∪ (T − D), S − C ∈ E because E is a σ-algebra, and T − D ∈ F, hence (S ∪ T ) − B ∈ E ∨ F. Thus, E ∨ F is closed with respect to complement.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 471

471

Let now {Bn | n ∈ N} be a countable family of sets in E ∨ F. We can write Bn = Cn ∪ Dn , where Cn ∈ E and Dn ∈ F for n ∈ N. Since  n∈N

Bn =

 n∈N

⎛ (Cn ∪ Dn ) = ⎝

 n∈N

⎞

⎛

Cn ⎠ ∪ ⎝

⎞



Dn ⎠ ,

n∈N

it follows that E ∨ P(A) is indeed a σ-algebra because  n∈N Dn ∈ F.



n∈N

Cn ∈ E and

The equality Kσ -alg (E ∪ F) = E ∨ F follows immediately. (2) Let (S, E) be a measurable space and let (Un ) be a sequence of subsets in E. Prove that there exists  a sequence  (Vn ) of pairwise disjoint subsets of S such that Vn ⊆ Un and n Vn = n Un . (3) Let F and O be the collection of closed and open set of a topological metric space. Consider the sequences un = (σ, δ, σ, · · · ) and vn = (δ, σ, δ, · · · ) of length n that consist of alternating symbols σ and δ, and the collections Fun = (· · · ((Fσ )δ )σ · · · ) and Ovn = (· · · ((Oδ )σ )δ · · · ). (a) Prove that Fun−1 ⊆ Fun , Ovn−1 ⊆ Ovn , Fun−1 ⊆ Ovn , and Ovn−1 ⊆ Fun , for every n  1. (b) Prove that for every n ∈ N the collections Fun and Ovn consist of Borel sets. (4) A σ-algebra on S is separable if it is generated by a countable collection of subsets of S. Prove that the σ-algebra of Borel subsets of R is separable. (5) Let (S, E) be a measurable space and let f, g : S −→ R be two simple measurable functions. Prove that if a ∈ R the set {x ∈ S | af (x)  g(x)} belongs to E. Solution: It is easy to see that g − af is a simple measurable function and {x ∈ S | af (x)  g(x)} = (g − af )−1 (R0 ), the statement follows. (6) Let {(Si , Ei ) | i ∈ I} be a family of measurable spaces, (S, E) be a measurable space and let {fi | i ∈ I} be a collection of functions fi : for i ∈ I. Let E be the σ-algebra on S generated by the S −→ Si  collection i∈I fi−1 (Ei ). Let (S0 , E0 ) be a measurable space and let f : S0 −→ S be a function. Prove that f is measurable (as a function between the measurable spaces (S0 , E0 ) and (S, E)) if and only if each function gi = fi f is a measurable function between the measurable spaces (S0 , E0 ) and (Si , Ei ) (see diagram below).

May 2, 2018 11:28

472

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 472

Mathematical Analysis for Machine Learning and Data Mining

S

fi

- Si 

6 gi

f S0

Solution: Since the composition of two measurable mappings is measurable, the condition is necessary. Conversely, suppose that each function gi = fi f is measurable and let U ∈ E.  Since E is the σ-algebra generated by the collection i∈I fi−1 (Ei ), to prove that f is measurable it would suffice to show that f −1 (D) ∈ E0 for every D of the form D = fi−1 (E), where E ∈ fi−1 (Ei ). Since fi f is measurable, it follows (fi f )−1 (E) ∈ E0 . Note that f −1 (D) = f −1 (fi−1 (E)) = (fi f )−1 (E) ∈ E0 , which implies that f is measurable. (7) Prove that if f : R −→ R is a monotonic function, then f is measurable ˆ )). ˆ , B (R relative to (R (8) Let f = (f0 , f1 , . . .) be a sequence of measurable real-valued functions defined on the measurable space S = (S, E). The sets





U = x ∈ S lim f (x) exists and is finite

n→∞

 



U+ = x ∈ S lim f (x) = ∞

n→∞

 



U− = x ∈ S lim f (x) = −∞

n→∞ 

belong to E. ˆ is a measur(9) Let (S, E) be a measurable space. Prove that if f : S −→ R able function, then the set Mf,t = {x ∈ S | f (x) = t} ˆ. is measurable for every t ∈ R

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 473

473

Solution: Since Lf,t = {x ∈ S | f (x)  t} ∩ {x ∈ S | f (x)  t}, the statement follows. (10) Let {B(xi , ri )| 1  i  N } be a finite collection of open spheres in Rn and let W = N i=1 B(xi , ri ). Prove that there exists a set I ⊆ {1, . . . , N } so that  i , ri ) | i ∈ I} are disjoint,  the spheres of the collection n{B(x W ⊆ i∈I B(xi , 3ri ) and mL (W )  3 i∈I m(B(xi , ri )). Solution: Suppose without loss of generality that r1  r2  · · ·  rN . Let i1 = 1. Discard all B(xi , ri ) that intersect B(i1 , r1 ) and let B(xi2 , ri2 ) the first of the remaining spheres, etc. The process halts after a finite number of steps and yields I = {i1 , i2 , . . .}. The spheres {B(xi , ri ) | i ∈ I} are clearly disjoint. Each discarded B(xi , ri ) is a subset of of B(xi , 3ri ) for some i ∈ I because if r   r and B(x , r  ) ∩ B(x, r), then B(x , r  ) ⊆ B(x, 3r). This proves the second claim. Finally, the third claim follows from the second because mL (B(x, 3r))  3n mL (B(x, r)). (11) Let (S, E, m) be a measure space and let (fn )n1 be a sequence of functions, where fn : S −→ T , where (T, d) is a metric space. Suppose that A ∈ E is a set with m(A) < ∞ such that limn→∞ fn = f a.e. on A. Prove that for every  > 0 there exists a subset B of A such that m(B) <  and (fn ) converges uniformly to f on the set A − B.  Solution: Let Un,k = mn {x ∈ A | |fm (x) − f (x)|  1/k}. For a fixed k, the sequence (Un,k ) is clearly decreasing.  If x ∈ A is such that limn→∞ fn (x) = f (x), then x ∈ n1 En,k , which   = 0. Since m(A) < ∞, by the continuity implies that m n1 Un,k from above (Theorem 7.32), it follows that foreach k there exists a number nk such that m(Unk ,k ) < 2k . If B = k1 Unk ,k , on the set A − B the sequence (fn ) converges uniformly to f and m(B) 

 k1

m(Enk ,k )
})  }, is finite. Prove that λ = l(f ) if and only if the following conditions: (a) mL ({x | |f (x)| > λ})  λ, and (b) mL ({x | |f (x)| > λ − δ}) > λ − δ for each δ > 0 are satisfied.

May 2, 2018 11:28

474

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 474

Mathematical Analysis for Machine Learning and Data Mining

Solution: Suppose that λ = l(f ). By the definition of infimum the second condition is met. Again, by definition, mL ({x | |f (x)| > l(f ) + })  l(f ) +  for each  > 0. When  → 0, the set {x | |f (x)| > l(f ) + } expands toward the limit set {x | |f (x)| > l(f )}, hence mL ({x | |f (x)| > λ})  λ due to the continuity property of measures. Conversely, let λ satisfy the two requirements. By definition, λ − δ < l(f )  λ for each δ > 0. This is possible only if λ = l(f ). (13) Prove that l(f )  0. Solution: Note that the inequality mL ({x | |f (x)| > })   can be satisfied only by non-negative values of  since the values of mL are non-negative. Thus, l(f )  0. (14) Prove that l(f ) = 0 if and only if f = 0 almost everywhere. Solution: By Supplements 12 and 13, a necessary and sufficient condition for l(f ) = 0 is mL ({x | |f (x)| > 0}) = 0, which means that f = 0 almost everywhere. (15) Let f, g : R −→ R be Lebesgue measurable functions. Prove that l(f + g)  l(f ) + l(g). Solution: If x is such that |f (x) + g(x)| > l(f ) + l(g), then at least one of the inequalities |f (x)| > l(f ), |g(x)| > l(g) must be satisfied. Therefore, mL ({x | |f (x) + g(x)| > l(f ) + l(g)})  mL ({x | |f (x)| > l(f )}) + mL ({x | |g(x)| > l(g)})  l(f ) + l(g), by Supplement 1. Therefore, l(f + g)  l(f ) + l(g). (16) Let f : R −→ R be a Lebesgue measurable function. Prove that if |a|  1, then l(af )  |a|l(f ). Solution: By Supplement 1, if |a| > 1, we have mL ({x | |af (x)| > |a|l(f )}) = mL ({x | |f (x)| > l(f )})  l(f )  |a|l(f ), which implies l(af )  |a|l(f ).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 475

475

(17) Let M be the set of Lebesgue measurable functions f : R −→ R such that l(f ) exists and is finite. Prove that if (fn ) is a sequence of functions in M such that limm,n→∞ l(fm − fn ) = 0, then there exists a function f ∈ M such that limn→∞ l(fn − f ) = 0. Solution: Let (fn ) be a sequence of functions such that limm,n→∞ l(fm − fn ) = 0. We seek to determine a subsequence (fnk ) such that l(fm − fn ) < 2−k when m  nk and n  nk , where k  1. By Supplement 1 we have mL ({x | |fnk +1 (x) − fnk (x)| > 2−k })  2−k .  −j }. It is clear that Let Ek = ∞ j=k {x | |fnk +1 (x) − fnk (x)| > 2 1−k Ek+1 ⊆ Ek and that mL (Ek )  2 . The series fn1 + (fn2 − fn1 ) + (fn3 − fn2 ) + · · · converges uniformly over any set R − Ek and, therefore, it converges for almost all x to some function f . If x ∈ Rr −Ek we have |f (x)−fnk (x)|  21−k , hence mL ({x | |f (x) − fnk (x)| > 21−k })  mL (Ek )  21−k , which implies l(f − fnk )  21−k . By Supplement 15, lim sup l(f − fn )  l(f − fn ) + lim sup l(fn − fnk )  21−k + 2−k , which is possible only if limn→∞ l(f − fn ) = 0. Since f − fn ∈ M and fn ∈ M , it follows that f ∈ M . (18) Let M r be the set of Lebesgue measurable functions of the form f : Rr −→ R, where l(f ) exists and is finite. Prove that ν : M r −→ R defined as ν(f ) = l(f ) is a seminorm; consequently, d : M × M −→ R0 defined by d(f, g) = l(f − g) = inf{ | mL ({x | |f (x) − g(x)| > })  } is a semimetric on M . Two functions f, g are close relative to this semimetric if the measure of the set on which they differ signficantly is small. (19) Let (S, E) be a measurable space and let f : S −→ R be a bounded function. Prove that for every  > 0 there are simple functions h and g defined on S such that h (x)  f (x)  g (x) and 0  h (x) − g (x) < . This statement is known as the simple approximation lemma. Solution: Since f is bounded, Ran(f ) is contained in some interval (c, d). Let (y0 , y1 , . . . , yn ) be a sequence of numbers in (c, d) such that

May 2, 2018 11:28

476

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 476

Mathematical Analysis for Machine Learning and Data Mining

c = y0 < y1 < · · · < yn = d and yk − yk−1 < . Define Ik = [yk−1 , yk ) and Ek = f −1 (Ik ) for 1  k  n. Since f is a measurable function, each set Ek is a measurable set. Define the simple functions h and g to be h =

n 

yk−1 1Ek and g =

k=1

n 

yk 1Ek .

k=1

Note that h (x)  f (x)  g (x) and 0  h (x) − g (x) < . (20) Prove Corollary 7.14 starting with the simple approximation lemma. ˆ be a function. Prove that f + = f 1 ˆ (21) Let f : S −→ R and f − = R0 . −f 1R ˆ 0

(22) Let S be a set and let E be a σ-algebra. Define the function m : E −→ ˆ 0 by R  |U | if U is finite, m(U ) = ∞ otherwise for U ∈ P(S). Prove that m is a measure. ˆ 0 is continuous if for every ascending se(23) A function m : P(S) −→ R   quence of subsets of S, (Un ), we have limn→∞ m(Un ) = m n∈N Un .

Prove that the second condition in the definition of a measure m : S −→ ˆ 0 is equivalent to finite additivity and the continuity property. R

(24) Let μ be an outer measure on the set S. Prove that |μ(U ) − μ(V )|  μ(U ⊕ V ) for every U, V ∈ P(S) such that μ(U ) < ∞ or μ(V ) < ∞. ˆ 0 be an outer measure such that Ran(μ) = {0, ∞}. (25) Let μ : P(S) −→ R Prove that every subset of S is μ-measurable. (26) Let C be a collection of sets such that ∅ ∈ C, every subset of a set in C also belongs  to C, and for every sequence (S0 , S1 , . . . , Si , . . .) of sets in C we have i∈N Si ∈ C. Prove that ˆ 0 defined by (a) the function μ : P(S) −→ R  μ(U ) =

0 1

if U ∈ C, otherwise

is an outer measure. ˆ 0 defined by (b) The function μ∞ : P(S) −→ R μ∞ (U ) = is an outer measure.

0 ∞

if U ∈ C, otherwise

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Measurable Spaces and Measures

b3234-main

page 477

477

(27) Let μ be an outer measure on a set S. Prove that if one of the subsets U, V of S is μ-measurable, then μ(U ∪ V ) + μ(U ∩ V ) = μ(V ) + μ(V ). ˆ 0 such that (28) Let S be a semiring of subsets of a set S and let m : S −→ R m(∅) = 0, m is finitely additive and countable subadditive. Prove that m can be extended to a measure on Kσ -alg (S). Solution: Note that m is monotonic.  Indeed, if U, V ∈ S and V ⊆ U , taking into account that U − V = U, where U is a finite collection U of pairwise disjoint sets in S, it follows  that m(U )  m(V ), by the finite additivity of m. Define μ(U ) = inf n m(Un ), where the infimum extends to all covers of U by sets in S The argument of Example 7.12 shows that m can be extended to an outer measure μ on S such that the sets in S are μ-measurable.   to S for n ∈ N and U ⊆ n Un , then m(U )  If U and Un belong n m(U ∩ Un )  n m(Un ). Therefore, m(U )  μ(U ). Since μ(U )  m(U ) because U ∈ S, we have m(U ) = μ(U ) for U ∈ S. Since S ⊆ Kσ -alg (S) ⊆ Eμ and μ is σ-additive on Eμ , it follows that μ is σ-additive on Kσ -alg (S). (29) Let (S, E, m) be a measure space such that m is σ-finite. Suppose that S is a semi-ring on S such that Kσ -alg (S) = E. Prove that: (a) if U ∈ E and  > 0 there existsa sequence of pairwise disjoint sets V1 , V2 , . . . in S such that U ⊆ n Vn and  m



 Vn − U

< ;

(7.18)

n

(b) if  m(U ) is finite there exists a sequence as above such that  m n Vn ⊕ U < . Solution: By Supplement 28, m can be extended to an outer measure μ on Kσ -alg (S) = E. Thus, m(T ) = μ(T ) for every T ∈ S and E ⊆ Eμ . Suppose that U ∈ E and m(U ) < ∞. There  exists a sequence of S such that m V  sets (V1 , V2 , . . .) in k k k m(Vk ) < m(T ) + ,   < . The sequence (V1 , V2 , . . .) can be which implies m n Vn − U replaced  by a sequence of pairwise sequence of disjoint sets of the form Vk ∩ k−1 j=1 Vj . If U ∈ E and m(U ) = μ(U ) = ∞, since m  is σ-finite, there exists a sequence of sets (Wp ) in S such that S = p Wp and m(W  p ) < ∞. Consequently, there exist Zpq in S such that U ∩ Wp ⊆ q Zpq and   < 2m . The sequence (Zpq ) constitute a sem q Zpq − (U ∩ Wp ) quence of sets in S that satisfies Condition (7.18). As before, the sequence (Zpq ) can be replaced with a sequence of pairwise disjoint sets.

May 2, 2018 11:28

478

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 478

Mathematical Analysis for Machine Learning and Data Mining

The second part follows immediately from the first. (30) Let S be a semi-ring on a set S such that S is a countable union of sets from S such that (S, E, m1 ) and (S, E, m2 ) are measure spaces such that E = Kσ -alg (S) and both m1 (S) and m2 (S) are finite. Prove that: (a) if m1 (T )  m2 (T ) for each T ∈ S, then m1 (U )  m2 (U ) for every U ∈ E; (b) if m1 (T ) = m2 (T ) for each T ∈ S, then m1 (U ) = m2 (U ) for every U ∈ E. Solution: By Supplement 29,  let (Vk ) be a sequence of disjoint sets  V in S such that U ⊆ n and n k m2 (Vk ) < m2 (U ) + . Then, m1 (U )    m (V )  m (V ) < m (V 1 2 2 k k k ) + . k k The second part is an immediate consequence of the first. (31) Let G, G be two extended open-closed intervals of Rn . Prove that G∩G is an extended open-closed interval of Rn . Solution: Let G = (a1 , b1 ] × · · · × (an , bn ] and G = (a1 , b1 ] × · · · × be two extended open-closed intervals. We have x ∈ G ∩ G if and only if max{ai , ai } < xi  min{bi , bi } for 1  i  n, hence

(an , bn ]

n

G ∩ G =

(max{ai , ai }, min{bi , bi }]. j=1

(32) Let G = (a1 , b1 ]×· · ·×(an , bn ] be an extended open-closed interval in Rn . Prove that Rn − G is a union of at most 2n disjoint extended open-closed intervals. Solution: For an extended open-closed interval (a, b] of R consider the disjoint extended open-closed intervals (a, b]l = (−∞, a] and (a, b]r = (b, ∞]. Then, R − (a, b] = (a, b]l ∪ (a, b]r . If x ∈ G at least one of the inequalities ai < xi  bi must fail, so R −G n

= =

n 

(a1 , b1 ] × · · · × (ai−1 , bi−1 ] × ((ai , bi ]l ∪ (ai , bi ]r ) × · · · × (an , bn ]

i=1 n 

(a1 , b1 ] × · · · × (ai−1 , bi−1 ] × (ai , bi ]l × · · · × (an , bn ]

i=1 n 

∪

(a1 , b1 ] × · · · × ×(ai , bi ]r × · · · × (an , bn ]

i=1

and some of the 2n pairwise disjoint sets that occur on the right side of the equality may be empty.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 479

479

(33) Let S be a bounded Borel subset  of R and let I be a collection of open intervals in R such that S ⊆ I. Prove  that I contains a finite disjoint subcollection {I1 , . . . , Ik } such that ki=1 mL (Ik )  16 mL (S). Solution: By Theorem 7.61 S contains a compact subset K such that mL (K)  12 mL (S). Let I0 be a subcollection of I that covers K and let I0 ∈ I0 be an interval of maximal length in I0 . Let I1 the collection obtained from I0 by discarding I0 and all intervals that intersect it. Repeat the process; to obtain I2 discard from I1 an interval I1 of maximal length and all intervals that intersect it, etc., until no more intervals in I0 are left. The intervals Ii are disjoint, and each interval Ii together with the intervals that intersect it is included in an interval Ji with the same midpoint as Ii and with mL (Ji ) = 3mL (Ii ). Thus, the family of intervals   mL (Ji ) {Ji } covers K and ki=1 mL (Ii ) =  m(K)  mL6(S) . 3 3 (34) Let G = (a1 , b1 ] × · · · × (an , bn ] and G = (c1 , d1 ] × · · · × (cn , dn ] be two extended open-closed intervals in Rn . Prove that G − G is a union of at most 2n disjoint extended open-closed intervals. Solution: Note that G − G = G ∩ (Rn − G). By Supplement 32, R −G is is a union of at most 2n disjoint extended open-closed intervals, Rn − G = G1 ∪ · · · ∪ Gp , hence n

G − G = G ∩ (Rn − G) = G ∩ (G1 ∪ · · · ∪ Gp ) =

p 

(G ∩ Gk ).

k=1

By Supplement 31, each set G ∩Gk is an extended open-closed interval of Rn ; moreover, the sets G ∩ G1 , . . . , G ∩ Gk are pairwise disjoint because G1 , . . . , Gk are pairwise disjoint. (35) Let E be the collection of subsets of Rn that are unions of finite collections of pairwise disjoint open-closed extended intervals. Prove that E is an algebra of subsets of Rn . p   Solution: If U, V ∈ E we have U = m j=1 Gj and V = k=1 Gk , where G1 , . . . , Gm are pairwise disjoint open-closed extended intervals and G1 , . . . , Gp are sets that satisfy the same description. Then, U ∩V =

p m  

Gj ∩ Gk

j=1 k=1

and Gj ∩Gk are extended open-closed intervals that are pairwise disjoint, so E is closed under finite intersections. Supplement 32 shows that the complement of an extended open-closed interval is an union of at most 2n disjoint extended open-closed intervals. Since E is closed to finite intersections it follows that E is indeed an algebra.

May 2, 2018 11:28

480

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 480

Mathematical Analysis for Machine Learning and Data Mining

(36) Prove that if μL is the Lebesgue outer measure on R and F is a finite subset of R, then μL (F ) = 0. (37) Let a, a1 , · · · , an be vectors in Rn . The parallelepiped determined by these vectors is the set P = {x ∈ Rn | x = a + α1 a1 + · · · + αn an , 0  αj  1 for 1  j  n}. Prove that: (a) if K1 = [0, 1]n is the n-dimensional unit cube, then there exists a linear transformation h on Rn such P = tb (h(K1 )), where the columns of Ah are a1 , . . . , an ; (b) P is a Lebesgue set and mL (P ) = | det(Ah )|. Solution: Let h : Rn −→ Rn be the linear operator defined as h(x) =

n 

xi bi

i=1

for x ∈ Rn . The equality P = tb (h(K1 )) follows immediately. P ∈ Ln by Theorem 7.53 and mL (P ) = mL (tb (h(K1 ))) = mL (h(K1 )) = | det(Ah )|mL (K1 ) = | det(Ah )|. (38) Let (S, E, m) be a measure space such that E = Kσ -alg (A), where m : ˆ 0 is a measure that is σ-finite on A. E −→ R Prove that: (a) for every U ∈ E we have m(U ) = inf{m(V ) | U ⊆ V and V ∈ Aσ }; (b) if U ∈ E and  > 0 then there exist V ∈ Aσ such that U ⊆ V and m(V − U ) < . ˆ 0 be defined as: Solution: Let m : P(S) −→ R m (L) = inf{m(M ) | L ⊆ M and M ∈ Aσ }. Suppose initially that m is finite, that is m(S) < ∞. Define the collection F = {V ∈ E | m (V ) = m(V )}. It is immediate that A ⊆ F ⊆ E. We claim that F is a monotone collection.  Let (Vn ) be an ascending chain of sets in F such that n∈N Vn = V .  Note that m (Vn ) = m(Vn ) for n ∈ N by the definition of F. Since m (Vn ) is defined by an infimum, for every  > 0 there exists Un ∈ Aσ such that Vn ⊆ Un and m(Un )  m(Vn ) + 2n , or m(Un − Vn )  2n .  If U = n∈N Un ∈ Aσ , we have V ⊆ U and ⎛ m(U − V ) = m ⎝ 

 n1



⎞ (Un − V )⎠ 

n∈N

m(Un − Vn ) 

 n∈nn

m(Un − V )

  = 2, 2n n1

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 481

481

hence m (V )  m(U ) < m(V ) + 2. Therefore, B ∈ F. The argument for a descending chain is similar, so F is a monotone collection, which means that F = Kmon (F). Since A ⊆ F ⊆ E, it follows that Kmon (A) ⊆ Kmon (F) ⊆ Kmon (E). By Theorem 1.39 we have E = Kσ -alg (A) = Kmon (A), which implies E ⊆ F ⊆ E, hence F = E, which, in turn, shows that m = m on E.  If m is σ-finite let (Sn ) be a sequence of subsets of S such that S = n∈N Sn and m(Sn ) < ∞. Define the finite measure mn on E as mn (U ) = m(U ∩ Sn ) for U ∈ E. By the previous argument, for every U ∈ E and  > 0 there exists Vn inAσ  such that U ⊆ Vn and m((Vn ∩Sn )−(U ∩Sn )) = m n (Vn −U )  2n . Since Sn ∈ Aσ we have Vn ∩ Sn ∈ Aσ and the set V = n∈N (Vn ∩ Sn ) ∈ Aσ . Furthermore, U ⊆ V and  m(V − U )  m((Vn ∩ Sn ) − U ) n∈N





m((Vn ∩ Sn ) − (U ∩ Sn ))

n∈N



  = 2, 2n

n∈N

which implies m(U )  m(V )  m(U ) + 2, so m(U ) = inf{m(V ) | U ⊆ V and V ∈ Aσ }. (39) Let (S, E, m) be a measure space, where m is a finite signed measure. Prove that m+ (T ) = sup{m(U ) | U ∈ E and U ⊆ T } and m− (T ) = − inf{m(U ) | U ∈ E and U ⊆ T }. Solution: For T ∈ E, T ⊆ U , and any Hahn decomposition {B+ , B− } we have: m(T ) = m+ (T ) − m− (T )  m+ (T )  m+ (U ) = m(U ∩ B+ ) ⊆ m(U ), by the definition of m+ given in Theorem 7.72. Similarly, m(T ) = m+ (T ) − m− (T )  m+ (T )  −m− (U ) = −m(U ∩ B− ).

(40) This supplement shows that the Jordan decomposition of a signed measure has a minimal character. Let (S, E, m) be a measure space, where m is a finite signed measure such that m(T ) = m1 (T ) − m2 (T ) for T ∈ E, where m1 and m2 are non-negative measures. Prove that m1  m+ and m2  m− . Solution: Assume that for some T ∈ E we have m1 < m+ . Therefore, m1 (T ∩ B+ ) + m1 (T ∩ B− ) < m+ (T ∩ B+ ) + m+ (T ∩ B− ). It follows that m1 (T ∩B+ ) < m+ (T ∩B+ ) because m+ (T ∩B− ) = 0 and m1 (T ∩B− )  0.

May 2, 2018 11:28

482

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 482

Mathematical Analysis for Machine Learning and Data Mining

The inequality m1 (T ∩ B+ ) < m+ (T ∩ B+ ) implies m2 (T ∩ B+ ) < m− (T ∩ B+ ) = 0, which contradicts the fact that m2 is non-negative. (41) Let (S, E) be a measurable space and let m, m1 , m2 be complex measures on this space. Prove that: (a) |m1 + m − 2|  |m1 | + |m2 |; (b) |cm| = |c| |m| for c ∈ C; (c) max{(m), (m)}  |m|  |(m)| + |(m)|. (42) Let (S, E) be a measurable space and let m be a complex measure on this space. Prove that the following statements are equivalent: (a) U is a null set for m; (b) U is a null set for both (m) and (m); (c) U is a null set for |m|. (43) Let (S, E, m) be a measure space, where m is a complex measure. Prove that m1 , m2 : E −→ R defined by m1 (U ) = (m(U )) and m2 (U ) = (m(U )) are finite signed measures. (44) Let m, m be signed measures. Prove that the following are equivalent: (a) (b) (c) (d)

m ⊥ m ; |m| ⊥ m ; m ⊥ |m |; |m| ⊥ |m |.

Solution: Suppose that m ⊥ m , that is, there exists T ∈ E such that m(T ) = m (S − T ) = 0. We have |m|(T ) = 0, so T is an |m|-null set and |m| ⊥ m . Suppose now that |m| ⊥ m , so that |m|(V ) = m (S − V ) = 0 for some V ∈ E. Since m+  |m| and m−  |m|, T is both m+ -null and m− -null. Therefore, m+ ⊥ m and m− ⊥ m . Finally, suppose that m+ ⊥ m and m− ⊥ m . Then, there are two subsets X, Y in S such that m+ (X) = m (S − X) and m− (Y ) = m (S − Y ). Note that (S − X) ∪ Y is m -null because every subset of (S − X) ∪ Y = (S − X) ∪ (Y − (S − X)) is the disjoint union of two m -null sets. Furthermore, S − ((S − X) ∪ (S − Y )) = X ∩ Y , which is both m+ - and m− -null, hence m-null. This shows that m ⊥ m . (45) Let m1 , m2 be positive measures such that at least one of the measures m1 , m2 is finite and let m = m1 − m2 be a signed measure on a set S. Prove that m1  m+ and m2  m− . (46) Let (S, E) be a measurable space. Prove that if m : E −→ R is a finitely additive function such  that for every decreasing sequence of sets (Un )n1 where Un ∈ E and n1 Un = ∅ we have limn→∞ m(Un ) = 0, then m is a measure.  Solution: Define Wn = Un − Un+1 for n  1. Then, Un = kn Wk ∞ k ∞ for n ∈ N. Since j=1 Wj = j=1 Wj + j=k+1 Wj , the finite additivity

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Measurable Spaces and Measures

page 483

483

of m implies

m

∞ 

 Wj

 =m

j=1

k 

 Wj

+ m(Uk+1 )

j=1

=

k 

m(Wj ) + m(Uk+1 ).

j=1

Since limk→∞ m(Uk+1 ) = 0, we have m



∞ j=1

  Wj = ∞ j=1 m(Wj ).

(47) Let (S, E) be a measurable space and let m be a complex measure on this space. Prove that for every real-valued measure m on the same measure space such that |m(E)|  m (E) for E ∈ E, we have |m|(E)  m (E) for E ∈ E. In other words, prove that the variation |m| of m is the smallest of all real measures m such that |m(E)|  m (E). Solution: Let m be a measure space such that |m(E)|  m (E) for E ∈ E. For E ∈ E we have:  |m|(E) = sup

n  i=1

|m(Ei )| | {E1 , . . . , En } 

is a partition of E with Ei ∈ E   sup

n  i=1

m (Ei ) | {E1 , . . . , En } 

is a partition of E with Ei ∈ E = m (E).

(48) Prove that if A, B are two independent events, then the pairs of events (A, B), (A, B), and (A, B) are independent. Let A be a Lebesgue measurable subset of R. A function f : R −→ R is measurable Lebesgue on A if f −1 (−∞, a) ∈ L for every a ∈ A. (49) Prove that the characteristic function 1B of a subset B of R is measurable Lebesgue on R if and only if B ∈ L. Solution: Suppose that 1B is measurable Lebesgue on R. Then, B = R − B = 1−1 B (−∞, 1/2) ∈ L, hence B ∈ L. Conversely, if B ∈ L,

May 2, 2018 11:28

484

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 484

Mathematical Analysis for Machine Learning and Data Mining

then

⎧ ⎪ ⎨∅ (−∞, b) = 1−1 B B ⎪ ⎩ R

if b  0, if 0 < b  1, if 1 < b

for every b ∈ R, hence 1B is measurable Lebesgue on R.

Bibliographical Comments Our presentation of product measure space follows [32]. The results in Supplements 12-18 belong to [11].

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 485

Chapter 8

Integration

8.1

Introduction

The Lebesgue integral is the cornerstone of integration theory and offers a far-reaching generalization of the Riemann integral that is applicable to a broad class of function. This chapter begins with the gradual introduction of the Lebesgue integral starting with simple measurable real functions and extending the integral to measurable non-negative functions, and ending with the integral of real-valued measurable functions. A further extension to complex-valued functions follows. Fundamental results of integration theory such as the monotone convergence theorem, Fatou’s lemma, the dominated convergence theorem, etc., which have frequent applications in practice are also discussed. We review the basics of Riemann integration and analyze the relationship between the class of Lebesgue-integrable functions and the class of Riemann integrable function. This is especially interesting because, although the class of Lebesgue-integrable functions is broader than the class of Riemann-integrable functions, the value of a Riemann integral is easier to compute and, in many cases, the value of these integrals are the same. The notions of absolute continuity of measures and the Radon-Nikodym theorem that leads to the notion of Radon-Nikodym derivative and to the Lebesgue decomposition theorem are presented. Finally, we discuss properties of Lp spaces and certain aspects of probability theory.

8.2

The Lebesgue Integral

The construction of the integral is done in several stages. We begin by defining the integral for simple measurable functions, then extend this definition 485

May 2, 2018 11:28

486

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 486

Mathematical Analysis for Machine Learning and Data Mining

to non-negative measurable function. After introducing the notion of integrable real-valued function, the notion of integral for arbitrary real-valued measurable functions is introduced. Finally, we present the construction of the integral for complex-valued functions. 8.2.1

The Integral of Simple Measurable Functions

Recall that the set of simple, non-negative functions was denoted by SF+ (S). Let (S, E, m) be a measure space and let f ∈ SF+ (S) be a nonnegative simple measurable function between the measurable spaces (S, E) and (R, B(R)). By Theorem 7.10 the function f can be written as n 

f (x) =

yi 1f −1 (yi ) (x),

i=1

for x ∈ S, where Ran(f ) = {y1 , . . . , yn } ⊆ R0 . We have f −1 (yi ) ∈ E because f is a measurable function. Definition 8.1. Let (S, E, m) be a measure space and let f ∈ SF+ (S) be a n simple measurable function given by f (x) = i=1 6yi 1f −1 (yi ) (x) for x ∈ S. The Lebesgue integral of f on S is the number S f dm defined as 7 n  f dm = yi m(f −1 (yi )), S

i=1

where we assume that 0 · ∞ = 0 in order to accommodate the case when m(f −1 (yi )) = ∞. Note that if yi = 0, the contribution of the term yi m(f −1 (yi )) is zero even when m(f −1 (yi )) = ∞, due to the convention established earlier that 0 · ∞ = 0. Example 8.1. Let (S, E, m) be a measure space and let U ∈ E. The characteristic function 1U belongs to SF+ and we have 7 1U dm = m(U ). S

Example 8.2. Let (S, E, m) be a measure space and let δs0 be a Dirac n measure. If f ∈ SF+ (S), where f (x) = i=1 yi If −1 (yi ) (x), then 7 n  f dδs0 = yi δs0 (f −1 (yi )) = yi S

if f (s0 ) = yi . In other words,

6

i=1

S

f dδs0 = f (s0 ).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 487

487

As Theorem 7.25 shows, we could use any partition6of a set S related to a simple measurable function f ∈ SF+ (S) to define S f dm. In other words, if π = {B1 , . . . , B } is any partition of S related to f , then 7   f dm = yk m(Bk ), (8.1) S

k=1

where f (x) = yk for every x ∈ Bk for 1  k  . Theorem 8.1. Let (S, E, m) be a measure space and let f, g ∈ SF+ (S) be such that f (x)  g(x) for every x ∈ S. We have: 7 7 f dm  g dm. S

S

Proof. Let π = {B1 , . . . , Bn } and σ = {C1 , . . . , Cm } be two partitions of S related to f and g, respectively. Then, the partition π ∧ σ that consists of the non-empty sets of the form Bi ∩ Cj is related to both f and g and we have 7 n  m  f dm = {uij m(Bi ∩ Cj ) | Bi ∩ Cj = ∅}, S

i=1 j=1

7 g dm = S

n  m 

{vij m(Bi ∩ Cj ) | Bi ∩ Cj = ∅},

i=1 j=1

where uij = f (x) and vij = g(x) 6if x ∈ Bi ∩6Cj . Since 0  uij  vij whenever Bi ∩ Cj = ∅, the inequality S f dm  S g dm follows immediately.  Let (S, E, m) be a measure space. The integral of a non-negative simple measurable function f relative to a subset T of S that belongs to E is defined by 7 7 f dm = T

f 1T dm. S

If f is a simple non-negative measurable function, then so is f 1T . Furthermore, the function f 1T is zero outside the set T . n for x ∈ S and T is a subset of S that If f (x) = i=1 yi If −1 (yi ) (x)  belongs to E, then f 1T (x) = ni=1 yi 1f −1 (yi )∩T (x) for x ∈ S. Therefore, we have 7 7 n  f dm = f IT dm = yi m(f −1 (yi ) ∩ T ). (8.2) T

S

i=1

Theorem 8.2. Let (S, E, m) be a measure space and let U, V be two disjoint subsets of S. If f ∈ SF+ (S) then 7 7 7 f dm = f dm + f dm. U∪V

U

V

May 2, 2018 11:28

488

Proof.

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 488

Mathematical Analysis for Machine Learning and Data Mining

Suppose that Ran(f ) = {y1 , . . . , yn }. We have m(f −1 (yi ) ∩ (U ∪ V )) = m(f −1 (yi ) ∩ U ) + m(f −1 (yi ) ∩ V ),

because the sets U and V are disjoint. Therefore, we have 7 n  f dm = yi m(f −1 (yi ) ∩ (U ∪ V )) U∪V

i=1

=

n 

m(f −1 (yi ) ∩ U ) +

i=1

f dm +

=

m(f −1 (yi ) ∩ V )

i=1

7

7

n 

f dm,

U

V



which concludes the argument.

Theorem 8.3. Let (S, E, m) be a measure space and let f, g ∈ SF+ (S). Then, 7 7 7 (af + bg) dm = a f dm + b g dm, S

S

S

for every a, b ∈ R0 . Proof. Let π = {B1 , . . . , Bn } and σ = {C1 , . . . , Cm } be two measurable partitions of S related to f and g, respectively. Suppose that the infimum of these two partitions in (PART(S), ) is τ = π ∧ σ, where τ = {D1 , . . . , Dr }. The partition τ is related to both f and g and, therefore, there exist u1 , . . . , ur and v1 , . . . , vr in R such that f (x) = ui and g(x) = vi for every x ∈ Di and 1  i  r. Furthermore, τ is related to af + bg as well because (af + bg)(x) = aui + bvi for every x ∈ Di . Thus, 7 r  (af + bg) dm = (aui + bvi )m(Di ) S

i=1 r 

=a

ui m(Di ) + b

i=1

7 =a

S

which is the desired equality.

7

f dm + b

r 

vi m(Di )

i=1

g dm, S



Corollary 8.1. Let (S, E, m) be a measure space and 6let f ∈ SF+ (S). Then, ˆ 0 defined by mf (U ) = f dm is a measure the mapping mf : E −→ R U on E.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 489

Integration

Proof. It is clear that additivity:

6 ∅

⎛

mf ⎝

f dm = 0. Thus, we need to prove only the ⎞



Un ⎠ =

n∈N

that is,

7 

489

f dm = N Un

n∈



mf (Un ),

n∈N

7 n∈N

f dm

(8.3)

Un

for every countable collection {U0 , U1 , . . .} of sets in E that are pairwise disjoint. Consider the case when f is the characteristic function of a set W ∈ E, that is, f = 1W . In this case, equality (8.3) amounts to ⎞ ⎛   m ⎝W ∩ Un ⎠ = m(W ∩ Un ), n∈N

n∈N

which holds by the distributivity of the intersection over the countable union and the additivity of measures. Since every simple non-negative function is a linear combination of char acteristic functions, the additivity of mf follows immediately. Example 8.3. Note that for x ∈ R the interval [x − 1, x) contains exactly one integer k, denoted by x. Consider the simple function measurable f : [0, 3] −→ R given by f (x) = x2 . The range of this function is the set {0, 1, 4, 9} and we have f −1 (0) = [0, 1), f −1 (1) = [1, 2), f −1 (4) = [2, 3), f −1 (9) = {3}. Thus, if m is the Lebesgue measure we have 7 f dm = 0 · m([0, 1)) + 1 · m([1, 2)) + 4 · m([2, 3)) + 9 · m({3}) [0,3]

= 0 · 1 + 1 · 1 + 4 · 1 + 9 · 0 = 5. Theorem 8.4. Let (S, E, m) be a measure space 6 and let f, g ∈ SF+ (S) such 6 that f (x)  g(x) for x ∈ S. Then, S f dm  S g dm.   Proof. Let f (x) = ni=1 yi 1f −1 (yi ) (x) and g(x) = m j=1 zj 1f −1 (zj ) (x) for x ∈ S. If f −1 (yi ) ∩ g −1 (zj ) = ∅, then for x ∈ f −1 (yi ) ∩ g −1 (zj ) we have

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 490

Mathematical Analysis for Machine Learning and Data Mining

490

yi = f (x)  g(x) = zj . This allows us to write 7 n  f dm = yi m(f −1 (yi )) S

i=1

=

n  m 

yi m(f −1 (yi ) ∩ g −1 (zj ))

i=1 j=1

(because only terms with f −1 (yi ) ∩ g −1 (zj ) = ∅ contribute) n  m   zj m(f −1 (yi ) ∩ g −1 (zj )) i=1 j=1

=

m  j=1

zj m(f

−1

7 (zj )) =

g dm. S



Theorem 8.5. Let (S, E, m) be a measure space and let f ∈ SF+ (S). If  (U 6 n ) is an increasing 6sequence of subsets of S with n1 Un = S, then f 1U dm = limn→∞ S f 1Un dm. S n Proof. Since f ∈ SF+ (S) we have f (x) = i=1 yi 1f −1 (yi ) (x). Then n n −1 )(x) = y 1 (x)1 (x) = y 1 (f 1 U i U i f (yi ) f −1 (yi )∩U (x). Therefore, i=1 6 i=1 n −1 f 1 dm = y m(f (y ) ∩ U ). U i i=1 i S −1 −1 (y (yi ) ∩ U ). Therefore, Note 6that limn→∞ m(f i ) ∩ Un ) = m(f 6  limn→∞ S f 1Un dm = S f 1U dm. Theorem 8.6. Let f = (fn )n1 be a sequence on non-negative simple measurable functions defined on the measure space (S, E, m) such that fn (x)  fn+1 (x) for all x ∈ S and n  1. Also, let f be a non-negative simple measurable function on the same space. The following statements hold: 6 f (x)  f (x) for x ∈ S, then lim (i) if lim n→∞ n n→∞ S fn dm  6 S f dm; 6 (ii) if 6 limn→∞ fn (x)  f (x) for x ∈ S, then limn→∞ S fn dm  f dm. S 6 6 Proof. Theorem 8.4 implies that S fn dm  S fn+1 dm for6 n  1 and therefore limn→∞ fn dm 6exists in [0, 6∞]. By the same theorem, S fn dm  6 f dm, hence limn→∞ S fn dm  S f dm, which proves part (i). S To prove part (ii), suppose that limn→∞ fn (x)  f (x) for x ∈ S. By Supplement 5 of Chapter 7 the set Sn,a = {x ∈ S | af (x)  fn (x)} is measurable for every a ∈ [0, 1). The sequence of sets (Sn,a )n1 is increasing.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 491

491

 We claim that n1 Sn,a = S. Suppose that this is not the case, that is,  there exists x ∈ S such that x ∈ n1 Sn,a . Then af (x) > fn (x) for n  1, hence af (x)  f (x) > 0, which contradicts the assumption that a ∈ [0, 1). We have af (x)1Sn,a (x)  f (x) for x ∈ S. Therefore, 7 7 a f dm = af dm S S 7 af 1Sn,a dm = lim n→∞

S

(by Theorem 8.4) 7  lim fn dm n→∞

S

(by Theorem 8.5). Since this 6 inequality holds for a ∈ [0, 1) it follows that limn→∞ S fn dm.

6 S

f dm  

Theorem 8.7. Let (S, E) be a measure space and let (fn )n1 and (gn )n1 be two increasing sequences of functions 6 in SF+ (S). If lim 6 n→∞ fn (x) = limn→∞ gn (x) for x ∈ S, then limn→∞ S fn dm = limn→∞ S gn dm. Proof. We have fp  limn→∞ gn (x) 6 for x ∈ S and p 6 1. Therefore, the second part of Theorem 6 8.6 implies S fp dm 6  limn→∞ S gn dm for every p  1, hence limn→∞ S fn dm  limn→∞ S gn dm. The reverse inequality has a similar proof.  8.2.2

The Integral of Non-negative Measurable Functions

The extension of integral to non-negative measurable functions is based on the fact (shown in Theorem 7.26) that a non-negative measurable function is the pointwise limit of a sequence of measurable simple non-negative functions. Definition 8.2. Let (S, E, m) be a measure space, f : S −→ [0, ∞] be a non-negative measurable function. The Lebesgue integral of f on S, denoted 6 by S f dm, is defined as: 7 7 f dm = lim fn dm, S

n→∞

S

where (fn ) is any sequence of non-negative simple functions such that limn→∞ fn = f .

May 2, 2018 11:28

492

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 492

Mathematical Analysis for Machine Learning and Data Mining

The non-negative measurable function f is m-integrable, or just inte6 grable if S f dm is finite. 6 By Theorem 8.7, S f dm is well-defined because its value is independent of the choice made for the sequence of non-negative simple functions that converges to f . Theorem 8.8. Let (S, E, m) be a measure space, f, g : S −→ [0, ∞] be non-negative measurable functions. We have: 7 7 7 (f + g) dm = f dm + g dm S S S 7 7 af dm = a f dm S

S

for every a  0. Proof. Let (fn )n1 and (gn )n1 be two sequences of non-negative measurable simple functions such that limn→∞ fn = f and limn→∞ gn = g. Then, (fn + gn )n1 is a sequence of non-negative measurable simple functions that converges to f + g, hence 7 7 (f + g) dm = lim (fn + gn ) dm n→∞ S S  7 7 = lim fn dm + gn dm n→∞

S

S

= (by Theorem 8.3) 7 7 = lim fn dm + lim gn dm n→∞ S n→∞ S 7 7 = f dm + g dm. S

S

Since a  0, (afn )n1 is a sequence of non-negative measurable simple functions such that limn→∞ afn = af . By Theorem 8.3, 7 7 af dm = lim afn dm n→∞ S S 7 = lim a fn dm n→∞ 7 7S = a lim fn dm = a f dm. n→∞ S  S Theorem 8.9. Let (S, E, m) be a measure space, and let f and g be two non-negative measurable functions such that f (x)  g(x) for x ∈ S. We 6 6 have S f dm  S g dm.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 493

493

Proof. Let (fn ) and (gn ) be two sequences of non-negative increasing functions such that limn→∞ fn = f and limn→∞ gn = g. Since fn  f  g = lim gn , n→∞

we have

7 S

7

7 fn dm  lim

n→∞

gn dm = S

g dm, S

by the second part6 of Theorem 8.6. The inequality 6 implies S f dm  S g dm.

6 S

fn dm 

6 S

g dm 

ˆ 0 Theorem 8.10. Let (S, E, m) be a measure space, and let f, g : S −→ R be two non-negative measurable functions. The following statements hold: 6 (i) we have S f dm = 0 if and only if f = 0 a.e.; 6 6 (ii) if f = g a.e., then S f dm = S g dm. 8 9 6 Proof. Suppose that S f dm = 0 and let Un = f −1 n1 , ∞ for n  1. We have n1 1Un  f , which implies 7 7 1 1 m(Un ) = 1Un dm  f dm = 0 n S n S by Theorem 8.9. Thus, m(Un ) = 0 for n  1. Since {x ∈ S | f (x) = 0} =  n1 Un , it follows that {x ∈ S | f (x) = 0} = 0, so f = 0 a.e. Conversely, suppose that f = 0 a.e. Let (fn ) be an increasing sequence of non-negative measurable 6 simple functions such that limn→∞6 fn = f . We have fn = 0 a.e., hence S fn dm = 0 for n  1, which implies S f dm = 0. For the second part let V = {x ∈ S | f (x) = g(x)} ∈ E. There exists W ∈ E such that S − V ⊆ W and m(W ) = 0. Let Y = S − W ⊆ V . The functions f 1Y and g1Y are measurable and f 1Y = g1Y . Also, note that f 1W = 0 a.e and g1W = 0 a.e. on S. 6 6 The first part implies S f 1W dm = S g1W dm = 0. Theorem 8.8 implies: 7 7 f dm = (f 1W + f 1Y ) dm S 7S 7 = f 1W dm = g1W dm S 7S 7 = (g1W + g1Y ) dm = g dm.  S S

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 494

Mathematical Analysis for Machine Learning and Data Mining

494

The next theorem shows that pointwise convergence of a sequence of non-negative functions allows us to permute integration and limit. Theorem 8.11. (The Monotone Convergence Theorem) Let f be a non-decreasing sequence of non-negative and measurable functions, f = (fn ) ˆ 0 is the such that limn→∞ fn (x) exists for every x ∈ S. If f : S −→ R function defined by f (x) = limn→∞ fn (x) for x ∈ S, then f is a nonnegative measurable function and 7 7 f dm = lim fn dm. S

n→∞

S

Proof. Since f is a nondecreasing sequence, limn→∞ fn (x) exists, finite or not. Therefore, by Corollary 7.15, the function f is measurable. By the 6 6 hypothesis of the theorem, S fn dm  S f dm. 6 The sequence v = (v0 , . . . , vn , . . .) given by vn = S fn dm for 6 ˆ 0 be v = limn→∞ vn  f dm, so n ∈ N is increasing. Let v ∈ R S 6 6 limn→∞ S fn dm  S f dm. 6 6 To prove the reverse inequality, S f dm  limn→∞ S fn dm, let h be a simple non-negative function such that h  f . For α ∈ (0, 1) and n ∈ N consider the sets Unα = {x ∈ S | fn (x)  αh(x)}. We claim that the sequence of sets U = (U0α , . . . , Unα , . . .) is nondecreasing and  S= Unα . n∈N

Thus, limn→∞ Unα = S and limn→∞ IUnα (x) = 1 for x ∈ S and α ∈ (0, 1). Let x ∈ Unα . We have fn (x)  αh(x) and, since fn+1 (x)  fn (x), it α α . Thus, Unα ⊆ Un+1 , so U is indeed a non-decreasing follows that x ∈ Un+1 sequence of sets. Let x be an element of S. If we would have fn (x) < αh(x) for every n ∈ N this would imply that f (x) = limn→∞ fn (x) < αh(x) < h(x), which contradicts the definition of h. Therefore, there exists n ∈ N such that fn (x)  αh(x), so x ∈ Unα , which justifies our claim. We have 7 7 7 fn dm  fn dm  α h dm = αmh (Unα ). S

Thus,

α Un

α Un

7 fn dm  lim αmh (Unα ) = αmh

lim

n→∞

S

n→∞

#

$ lim Unα = αmh (S),

n→∞

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 495

495

by Theorem 7.32. Therefore, 7 fn dm  lim αmh (S), lim n→∞

S

α→1

for every non-negative simple function h such that h  f . We conclude 6 6  that limn→∞ S fn dm  S dm. Theorem 8.12. (Fatou’s Lemma) Let (S, E, m) be a measure space and let f be a sequence of non-negative measurable functions, f = (f0 , f1 , . . . , fn , . . .) such that fi : S −→ R0 . We have the inequality 7 7 lim inf fn dm  lim inf fn dm. S

n→∞

n→∞

S

6 Proof. If k  j, then inf f  f . Therefore, nk n j S inf nk fn dm  6 6 6 S fj dm, which implies S inf nk fn dm  inf kj S fj dm. The sequence of functions (inf nk fn ) is nondecreasing and lim inf fn = lim inf fn . n→∞

k→∞ nk

By the Monotone Convergence Theorem we have 7 7 7 lim inf fn dm = lim inf fn  lim inf fj dm. S

n→∞

k→∞

j→∞

S nk

S



Example 8.4. The inequality in Fatou’s Lemma may be strict. Consider, for example the sequence of functions f = (f0 , f1 , . . . , fn , . . .), where  1 if n  x  n + 1, fn (x) = 0 otherwise. 6 It is clear that lim inf n→∞ f6n = limn→∞ fn = 0, so R lim inf n→∞ fn dm = Therefore, 0. On another hand, R fn dm = 1 for n ∈ N. 6 lim inf n→∞ R fn dm = 1. Lemma 8.1. Let (S, E, m) be a measure space and let f be a sequence of non-negative measurable functions, f = (f1 , . . . , fn , . . .). We have 7  ∞ ∞ 7  fj dm = fj dm. S j=1

j=1

S

Proof. Since f consists of non-negative functions, the sequence of func tions g = (g0 , g1 , . . . , gn , . . .) given by gn = nj=1 fj is non-decreasing and ∞ limn→∞ gn = j=1 fj .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 496

Mathematical Analysis for Machine Learning and Data Mining

496

The Monotone Convergence Theorem (Theorem 8.11) implies 7 7 lim gn dm = lim gn dm. S n→∞

n→∞

S

By the definition of (gn ) this means that ⎛ ⎞ 7 7  n ∞  ⎝ fj dm = lim fj ⎠ dm S j=1

n→∞

= lim

n→∞

S n 7  j=1

j=1

fj dm =

S

∞ 7  j=1

fj dm,

S



which concludes the argument. Note that both sides of the equality of Lemma 8.1 may be infinite.

Theorem 8.13. Let (S, E, m) be a complete measure space, E ∈ E be a set of finite measure and let f : S −→ R be a bounded function. We have * 7   inf h dmh ∈ SF(E) and h  f E 7 *   = sup g dmg ∈ SF(E) and g  f E

if and only if f is measurable. Proof. Suppose that f (x) < M when x ∈ E. Since f is measurable, for −n  k  n the sets  *  (k − 1)M kM  < f (x)  Ek = x ∈ E  n n k=n form a partition of the set E, so m(E) = k=−n m(Ek ). Let hn , gn be the simple functions defined by: n M  hn (x) = k1Ek (x), n gn (x) =

M n

k=−n n 

(k − 1)1Ek (x).

k=−n

We have gn (x)  f (x)  hn (x). Therefore, 7 * 7 k=n M  inf h dm | h ∈ SF(E), h  f  hn dm = km(Ek ) n E E k=−n

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

and

*

7 g dm | g ∈ SF(E), g  f

sup

497

7 

E

gn dm = E

Consequently,

page 497

k=n M  (k − 1)m(Ek ). n k=−n

7

*

0  inf

h dm | h ∈ SF(E), h  f E 7 * − sup g dm | g ∈ SF(E), g  f E

n M  M m(E),  m(Ek ) = n n k=−n

for n  1, which implies the equality of the theorem. Conversely, suppose that 7 *   inf h dmh ∈ SF(E) and h  f E * 7   h dmg ∈ SF(E) and g  f . = sup E

For every n ∈ N there 6 are simple 6 functions gn and hn such that gn (x)  f (x)  hn (x) and S hn dm − S gn dm < n1 . Therefore, the functions p = sup hn and q = inf gn are measurable and p  f  g ∗ . Observe that *   1 x ∈ S | p(x) < q(x) − {x ∈ S | p(x) < q(x)} = . m m1

Since 

*  * 1 1 ⊆ x ∈ S | hn (x) < gn (x) − , x ∈ S | p(x) < q(x) − m m ) ( 1 < m for every n and m x ∈ S | hn (x) < gn (x) − m n for every n  1 } = 0 and p = f 1. Therefore, we have m {x ∈ S | hn (x) < gn (x) − m almost everywhere. Thus, f is measurable.  ˆ Theorem 8.14. Let (S, E, m) be6 a measure6 space and let f, g : S 6 −→ R0 . If 6 f (x)  g(x) for x ∈ S and S f dm, S g dm exist, then S f dm  S g dm.

Proof. It is easy to see that f (x)  g(x) for x ∈ S implies f +6(x)  g + (x) − − + for x ∈ and 6 +f (x)  g6 (x) 6 S.− We have the inequalities S f dm  − dm and S f dm  S g dm. and all four previously mentioned Sg integrals are finite due to 7 the integrability 7 7 of f and7g. Therefore, 7 7 f + dm −

f dm = S

S

f − dm 

S

g + dm −

S

g − dm =

S

g dm. S



May 2, 2018 11:28

498

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 498

Mathematical Analysis for Machine Learning and Data Mining

Theorem 8.15. Let f be a non-negative function defined6on the measure space (S, E, m). The function f is zero a.e. if and only if S f dm = 0. Proof. Suppose that f is zero a.e.. If h is a simple function such that 6 0  h  f , then h = 0 6almost everywhere. Thus, S h dm = 0 for all such simple function, hence6 S f dm = 0. Suppose now that S f dm = 0 and let T = {x ∈ S | f (x) > 0}. Let      1 1  −1 ,∞ . Tn = x ∈ S   x < ∞ = f n n  The sequence (Tn ) is increasing and T = {Tn | 1  n  ∞}. simple function hn = n1 1Tn . We have hn  f and 6 6 Consider the 1 R hn dm = n m(Tn )  R f dm = 0, hence m(Tn ) = 0 for all n. By Theorem 7.32 it follows that m(T ) = 0, so f is 0 a.e.  ˆ 0 Theorem 8.16. Let (S, E, m) be a measure space and let f, g : S −→ R be two measurable functions defined in 6 6 on S. 6If f  g almost everywhere f dm, g dm exist, then f dm  the sense of the measure m and S S S 6 g dm. S Proof. Let U = {x ∈ S | f6(x)  g(x)}. 6Then S − U is a null set. The inequality f 1U 6g1U implies S f 1U dm  S g1U dm. Since S − U is null, 6 f 1 dm = S−U g1A dm = 0, so A S−U 7 7 7 7 f dm = f dm + f dm = f dm S 7S−U 7 U 7 7U g dm = g dm + g dm = g dm.   U

U

S−U

S

ˆ 0 Corollary 8.2. Let (S, E, m) be a measure space and let f, g : S −→ R be measurable functions defined on S. If f = g a.e., then 6 two non-negative 6 f dm = g dm. S S Proof.

This follows immediately from Theorem 8.16.



ˆ 0 is an Theorem 8.17. Let (S, E, m) be a measure space. If f : S −→ R integrable function then f is finite almost everywhere. Proof. Let Y be the set Y = {x ∈ R | f (y) = ∞}. For the simple 6 = nI we have g  f . Since gn dm = nm(A), we have function g n Y n 6 6 sup gn dm = ∞, it follows that f dm = ∞, which contradicts the fact that f is integrable. Thus, m(Y ) = 0, so f is finite almost everywhere. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Integration

b3234-main

page 499

499

Theorem 8.18. (Mean Value Theorem for Lebesgue Integrals) Let ˆ 0 is an integrable (S, E, m) be a measure space and U ∈ E. If f : S −→ R function and U ∈ E, then 7 f dm  m(U ) sup{f (x) | x ∈ U }. m(U ) inf{f (x) | x ∈ U }  U

Proof. Let (S, E, m) be a measure space and let g and h be the simple functions defined on S by g = inf{f (x) | x ∈ S}IU and h = sup{f (x) | x ∈ S}IU , then 7 7 g dm = inf{f (x) | x ∈ S} IU dm = inf{f (x) | x ∈ S}m(U ), 7U 7U h dm = sup{f (x) | x ∈ S} IU dm = inf{f (x) | x ∈ S}m(U ). U

U

6 6 6 Since U g dm ≤ U f dm ≤ U h dm, the double inequality of the theorem follows immediately.  Theorem 8.19. Let (S, E, m) 6 be a measure space and let f : S −→ R0 be a non-negative function. If f dm = 0, then f (x) = 0 almost everywhere. Proof. For n  1, define Un = {x ∈ S | f (x)  n1 }. Clearly, Un is   U. measurable and so is U = n1 Un . We have f (x) = 0 if x ∈ Let hn be the simple function defined by ⎧ ⎨ 1 if f (x)  1 , n hn (x) = n ⎩0 otherwise.

6 6 Clearly, we have hn  f , so hn dm  f dm = 0, so hn  f , so 6 hn dm = 0, which implies n1 m(Un ) = 0 for n ∈ N. Since Un is a monotonic sequence, m(U ) = lim m(Un ) = 0, and f is zero outside the null set U .  ˆ 0 be Definition 8.3. Let (S, E, m) be a measure space and let f : S −→ R a measurable non-negative function. The integral on a measurable subset U ∈ E of f is defined as 7 7 f dm = f 1U dm. U

S

Theorem 8.20. Let (S, E, m) be a measure space, and let U and V two disjoint subsets of S such that U, V ∈ E. If f is a non-negative measurable function, then 7 7 7 f dm = f dm + f dm. U∪V

U

V

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 500

Mathematical Analysis for Machine Learning and Data Mining

500

6 Proof. By Definition 8.2, U∪V6f dm is the supremum of the set that consists of numbers of the form U∪V h dm, h is a non-negative simple measurable 6 that h(x) 6  f (x) for x ∈ U ∪ V . By Theorem 8.2 6 function such we have U∪V h dm = U h dm + V h dm, which implies 7 7 7 h dm  f dm + f dm. U∪V

Therefore,

U

7

V

7 f dm 

U∪V

7 f dm +

U

f dm. V

To prove the converse inequality, let h1 , h2 be two simple functions on S such that h1 (x) = 0 if x ∈ U , h2 (x) = 0 if x ∈ V , and max{h1 (x), h2 (x)}  f (x) for x ∈ U ∪ V . Define the non-negative simple function h : S −→ R as ⎧ ⎪ ⎪ ⎨h1 (x) if x ∈ U, h(x) = h2 (x) if x ∈ U, ⎪ ⎪ ⎩0 if x ∈ S − (U ∪ V ). Clearly, we have h(x)  f (x) for x ∈ S. We have 7 7 7 7 7 h1 dm + h2 dm = h dm + h dm = U

V

Consequently,

U

7

V

7

U∪V

V

f dm. U∪V

7 f dm 

f dm + U

7 h dm 

f dm, U∪V

which concludes the argument.



Theorem 8.21. Let (S, E, m) be a measure space, and let f be a nonnegative measurable function. If U and V are two subsets of S such that 6 6 U ⊆ V , then U f dm  V f dm. Proof. Note that U ⊆ 6V implies I6U (x)  IV (x)6 for x ∈ S. 6Therefore, f IU  f IV . This implies U f dm = S f IU dm  S f IV dm = V f dm.  8.2.3

The Integral of Real-Valued Measurable Functions

The definition of Lebesgue integral is extended now to real-valued functions ˆ that range over the extended set of real numbers R. Definition 8.4. Let (S, E, m) be a measure space. A measurable func6 + ˆ 6tion −f : S −→ R is m-integrable, or just integrable if both S f dm and f dm are finite. S

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Integration

b3234-main

page 501

501

6 + 6 − If at least one of the 6 integrals S f dm or S f dm is finite, then we say that the integral S f dm exists is defined as: 7 7 7 f dm = f + dm − f − dm. S

6

S

S

Note that the integral S f dm may exist (and be ∞ or −∞) even if the function f is not integrable. The set of all real-valued integrable functions on the measure space (S, E, m) is denoted by L1 (S, E, m), or just L1 (S) when the measure space is understood from context. ˆ Theorem 8.22. Let (S, E, m) be a measure space. A function f : S −→ R is integrable if and only if |f | is integrable. 6 6 Proof. Suppose that f is integrable, so S f + dm and S6f − dm are Since6|f | = f + (x) + f − (x) for x ∈ S, it follows that S |f | dm = 6finite. + f dm + S f − dm, so |f | is integrable. S 6 |f |+ dm and suppose that |f | is integrable, that is both S 6 Conversely, 6 − + |f | dm are finite. Since |f | = |f |,6 if follows that S |f | dm is finite. S 6 This implies that both S f + dm and S f − dm are finite, which implies that f is integrable.  ˆ be a Theorem 8.23. Let (S,6E, m) be a measure space and f : S 6−→ R f dm is defined, and c ∈ R, then cf dm is measurable function. If S 6S 6 defined and S cf dm = c S f dm. 6 6 If f is integrable, then cf is also integrable and, again, S cf dm = c S f dm. Proof. If c = −1, then cf = −f , so (cf )+ = f − and (−f )− = f + . Therefore, 7 7 7 7 7 + − − f dm = f dm − f dm = (−f ) dm − (−f )+ dm S S S S S 7  7 =− (−f )+ dm − (−f )− dm S S 7 = − (−f ) dm. S

Thus,

7

7 (−f ) dm = −

S

f dm. S

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 502

Mathematical Analysis for Machine Learning and Data Mining

502

If c  0 we need to consider c = 0 and c6> 0. In the first 6 6 two subcases: subcase, we have cf = 0, so S cf dm = S 0 dm = 0 = 0 · S f dm, because we adopted the convention that 0∞ = 0(−∞) = 0. + + − − 6 Suppose now that c > 0. We have (cf ) = cf , (cf ) = cf . Thus, if S cf dm exists we have 7 7 7 7 7 + − + cf dm = (cf ) dm − (cf ) dm = cf dm − cf − dm. (8.4) S

S

S

S

S

+

Since f  0, by Theorem 7.26, there exists a non-decreasing selimit is quence of non-negative simple measurable functions (fn ) whose 6 6 + lim cf (x) = cf (x). Since cf dm = c f dm, we f + . Thus, n→∞ n n n S 6 S 6 6 6 have S cf + dm = c S f + dm. Similarly, we have S cf − dm = c S f − dm and the desired result follows from equality (8.4). For the remaining case, c < 0, we have 7 7 7 7 cf dm = (−1)(cf ) dm = − (−c)f dm = −(−c) f dm S

S

S

S

by the previous arguments. Thus, in all cases, the desired conclusion follows. then |f | is integrable (by Theorem6 8.22), so 6 If f is integrable, 6 |cf | dm = |c| |f | dm < ∞, so cf is also integrable and S cf dm = S S 6  c S f dm by the first part of the theorem. ˆ Theorem 8.24. Let (S, E, m) be a measure space and 6 R be 6 let f, g : S −→ two measurable functions such6 that f  g and both S f dm and S g dm 6 are defined. Then S f dm  S g dm. Proof.

Since f (x)  g(x) we have f + (x) = max{f (x), 0}  max{g(x), 0} = g + (x),

g − (x) = − min{g(x), 0}  − min{f (x), 0} = f − (x), 6 6 6 6 which implies S f + dm  S g + dm and S g − dm  S f − dm. In turn, this yields 7 7 7 7 7 7 f dm = f + dm − f − dm  g + dm − g − dm = g dm. S

S

S

S

S

S



ˆ be an integrable function on the measure Theorem 8.25. Let f : S −→ R space (S, E, m). We have: 7  7    f dm  |f | dm.  

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 503

503

Proof. By Theorem 8.22, 6 6 6 |f | is integrable. Since −|f |  f  |f |, we have − |f | dm  f dm  |f | dm. This implies the desired inequality.  ˆ be Theorem 8.26. Let (S, E, m) be a measure space and let f : S −→ R a measurable function that is integrable. Then f (x) ∈ R a.e. and the set U = {x ∈ S | f (x) = 0} is of σ-finite measure m. 6 Proof. Since f is integrable, we have S |f | dm is finite by Theorem 8.22. Let T = {x ∈ S | f (x) = ∞}. If r > 0 we have r1T (x)  |f |(x) for x ∈ S, hence 7 7 r1T dm  |f | dm < ∞. r m(T ) = 1 r

6

S

S

Therefore, m(T )  |f | dm for every r, so m(T ) = 0. S Let U = {x ∈ S | f (x) = 0} and Un = {x ∈ S | |f (x)|   Note that U = n1 Un . We have n1 1Un (x)  |f (x)|, hence 7 7 1 1 m(Un ) = 1Un dm  |f | dm < ∞. n S n S

1 n}

for n  1.

Thus, m(An ) is finite for every n, hence U is of σ-finite measure m.



ˆ be a Corollary 8.3. Let (S, E, m) be a measure space and let f : S −→ R measurable function that 6is integrable. 6 There exists a function g : S −→ R such that f = g a.e. and S g dm = S f dm. Proof. It suffices to define g = f 1S−T , where T = {x ∈ S | f (x) = ∞} is the set defined in the proof of Theorem 8.26.  ˆ be Theorem 8.27. Let (S, E, m) be a measure space and let f, g : S −→ R 6 two functions such that f = g almost everywhere. If S f dm is defined, 6 then S g dm is defined and the two integrals are equal. 6 + + + − − = g and f = g a.e., so f dm = Proof. If f = g a.e., then f S 6 + 6 − 6 − 6S g + dm and S f dm = S6g +dm by Corollary 8.2. Furthermore, 6 f dm is finite if 6and only if S g dm is finite and, similarly S f − dm S is finite if and only if S g − dm is finite. The conclusion follows immediately.  ˆ are two functions such that there exists x such that If f, g : S −→ R {f (x), g(x)} = {−∞, ∞}} we will assume that f (x) + g(x) = 0. ˆ Theorem 8.28. Let (S, E, m) be a measure space6 and let f, g : S −→ R 6 be two measurable functions. If both S f dm and S g dm are defined and

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

S

b3234-main

page 504

Mathematical Analysis for Machine Learning and Data Mining

504

(6

9in x 6in

f dm,

6 S

) 6 g dm =  {−∞, ∞}, then S (f + g) dm is defined and 7 7 7 (f + g) dm = f dm + g dm. S

S

(8.5)

S

If both f, g are integrable, then so is f + g and equality (8.5) holds. 6 6 Proof. Assume that S f dm and S g dm are defined. Therefore, out of the four values 7 7 7 7 f + dm, f − dm, g + dm, g − dm, S6 S S 6 S 6 6 − at most S f + dm,6 S g + dm or S f − dm, g dm may be infinite. S 6 Suppose that S f − dm < ∞ and S g − dm < ∞. Then, if T = {x ∈ S | f (x) = −∞ and g(x) = −∞}, m(S − T ) = 0. Define the measurable functions φ, ψ : S −→ (−∞, ∞] as φ = f 1T and ψ = g1T . We have φ = f and ψ = g almost everywhere on S and φ + ψ = f + g almost everywhere on S. 6 By Theorem φ(x)6+ ψ(x)  φ− (x) + ψ − (x), so S (φ + 6 − 7.28 6we have ψ) dm  S φ dm + S ψ − dm, so S (φ + ψ) dm is defined. Equality (7.6) and Theorem 8.8 7 7 7 imply (φ + ψ)+ dm +

S

7

φ− dm +

S

ψ − dm

S

7 7 (φ + ψ)− dm + φ+ dm + ψ + dm. S S S 6 6 6 Since S (φ + ψ)− dm, S φ− dm, and S ψ − dm are finite we have 7 7 7 + (φ + ψ) dm = (φ + ψ) dm − (φ + ψ)− dm S S 7S 7 7 7 + + = φ dm + ψ dm − φ− dm − ψ − dm S S S S 7 7 = f dm + g dm. S S 6 argument can be used for the case when S f + dm < ∞ and 6A similar + S g dm < ∞. In either case, the equality of the theorem follows immediately.  =

Theorem 8.29. Let (S, E, m) be a measure space, U, V ∈ 6 E such that U ∩ ˆ be a measurable function. If V = ∅, and let f : S −→ R 6 6 6 U∪V f dm 6 exists, 6 f dm and f dm exist and f dm = f dm+ then both U U∪V V f dm. 6 V 6 U 6 6 If U f dm6 and V f 6dm and are defined, then U∪V f dm exists and U∪V f dm = U f dm + V f dm. Proof. This theorem follows immediately from the corresponding result for non-negative functions (Theorem 8.20). 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

8.2.4

page 505

505

The Integral of Complex-Valued Measurable Functions

ˆ be a Definition 8.5. Let (S, E, m) be a measure space and let f : S −→ C measurable function. The function f is integrable on S with respect to m 6 if S |f | dm is finite. The set of all complex-valued integrable functions on the measure space (S, E, m) is L1 (S, E, m); the ambiguity introduced by this notation (the same as the one used for real-valued integrable functions) will be resolved in the context of the specific situation. Theorem 8.30. Let (S, E, m) be a measure space and let f ∈ L1 (S, E, m) be a complex-valued integrable function. Then f (x) ∈ C a.e., and the set {x ∈ S | f (x) = 0} is σ-finite relative to m. Proof.

This statement follows by applying Theorem 8.26 to |f |.



ˆ be an integrable Let (S, E, m) be a measure space and let f : S −→ C function and let Df = {x ∈ S | f (x) ∈ C}. We have:  f (x) if x = ∞, (f 1Df )(x) = 0 if x = ∞. Thus, the value of (f 1Df ) is defined for all x ∈ S, including those x for which f (x) = ∞. Additionally, we have |(f 1Df )|  |f 1Df |  |f | and |(f 1Df )|  |f 1Df |  |f |, which shows that for an integrable complex-valued function f both (f 1Df ) and (f 1Df ) are integrable real-valued functions. As follows from the next definition, the integral of a complex-valued function is defined only if f is integrable. Its value is a finite complex number. ˆ be Definition 8.6. Let (S, E, m) be a measure space and let f : S −→ C 6an integrable function and let Df = {x ∈ S | f (x) ∈ C}. The integral f dm is defined as: S 7 7 7 f dm = (f 1Df ) dm + i (f 1Df ) dm. S

S

S

ˆ be Theorem 8.31. Let (S, E, m) be a measure space and let f : S −→ C an integrable function. an integrable function g : S −→ C such 6 6 There exists that f = g a.e. and S f dm = S g dm.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 506

Mathematical Analysis for Machine Learning and Data Mining

506

Proof. Note that f = f 1Df a.e. because {x ∈ S | f (x) = (f 1Df )(x)} = {x ∈ S | f (x) = 0} and m({x ∈ S | f (x) = 0}) = 0. Thus, if we take  g = f 1Df the requirements of the theorem are satisfied. ˆ be a Corollary 8.4. Let (S, E, m) be a measure space and let f : S −→ C measurable function that 6is integrable. 6 There exists a function g : S −→ C such that f = g a.e. and S g dm = S f dm. Proof.



The function g is g = f 1Df .

ˆ Theorem 8.10 can be extended to functions that range over C. ˆ Theorem 8.32. Let (S, E, m) be a measure space, and let f, g : S −→ C be two complex-valued measurable functions. If f = g a.e. then, if one of 6 6 6 6S f dm or S g dm is defined, the other is also defined and S f dm = g dm. S Proof. If 6f is integrable and f = g a.e., then |f | = |g| a.e., hence 6 |f | dm = S |g| dm < ∞, so g is also integrable. S By Theorem 8.31, there exist integrable functions f1 , g1 :6S −→ C such 6 6that f = f61 a.e., and g = g1 a.e. such that S f dm = S f1 dm and S g dm = S g1 dm. Since f = g a.e., f1 = g16 a.e., hence (f 6 1 ) = (g1 ) ). Theorem 8.10 implies (f ) dm = a.e. 6and (f1 ) = (g 1 1 S (g1 ) dm 6 6 S 6  and S (f1 ) dm = S (g1 ) dm. Therefore, S f dm = S g dm. ˆ be Theorem 8.33. Let (S, E, m) be a measure space, and let f : S −→ C a measurable function. The following statements are equivalent: 6 (i) S |f | dm = 0; (ii) f = 0 a.e. on S; 6 (iii) S f 1E dm = 0 for every E ∈ E. 6 Proof. (i) implies (ii): If S |f | dm = 0, we have |f | = 0 a.e. on S by Theorem 8.10. (ii) implies (iii): If f = 0 a.e. on S then f 1E = 0 a.e. on S for 6 all E ∈ E, so S f 1E dm = 0 for every E 6∈ E. (iii) implies (i): Suppose that S f 1E dm = 0 for every E ∈ E. 6 For E = S we have S f dm = 0, so f is integrable. Let g : S 6 −→ C such that f = g a.e. For every E ∈ E we have f 1E = g1E and S g1E dm = 6 f 1 E dm = 0. This yields S 7  7 7 (g)1E dm = (g1E ) dm =  g1E dm = 0, S

S

S

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 507

507

and, therefore (g) = 0 a.e. on S. Similarly, (g) = 0 a.e., so g = 0 a.e. on S. Therefore, f = 0 a.e. on S.  ˆ Theorem 8.34. Let (S, E, m) be a measure space and let 6 f, g : S −→ C be two functions such that f = g almost everywhere. If S f dm is defined, 6 then S g dm is defined and the two integrals are equal. Proof. If f is integrable, 6 then |f | 6= |g| a.e. and, since |f |, |g| are nonnegative, it follows that S |f | dm = S |g| dm, hence g is also integrable. By 6 C such that f = φ a.e., g = ψ 6 Corollary 68.4, there exist6 φ, ψ : S −→ Since φ =6 ψ a.e. we a.e., S f dm = S φ dm, and S g dm = S ψ dm. 6 (φ) dm = S (ψ) dm have6(φ) = (ψ) a.e., and (φ) = (ψ), hence S 6 and S (φ) dm = S (ψ) dm, which implies 7 7 7 7 f dm = φ dm = (φ) dm + i (φ) dm S S S 7S 7 7 = (ψ) dm + i (ψ) dm = g dm.  S S S ˆ be Theorem 8.35. Let (S, E, m) be a measure space, and let f : S −→ C a measurable function. If f is integrable and c ∈ C, then λf is integrable 6 6 and S cf dm = c S f dm. Proof. Let φ : S −→6 C such that6 φ(x) = f (x)6almost everywhere on 6 S. Then cφ = cf a.e., S cφ dm = S cf dm 6and S φ dm6= S f dm by Theorem 8.32. Thus, it suffices to prove that S cφ dm = c S φ dm. Suppose that c = a + ib, where a = (c) and b = (c). Since cφ(x) = (a + ib)((φ)(x) + i(φ)(x)) = (a(φ)(x) − b(φ)(x)) + i(b(φ)(x) + a(φ)(x)), we have, by Theorem 8.23: 7 7 7 (cφ) dm = a (φ) dm − b (φ) dm, 7S 7S 7S (cφ) dm = a (φ) dm + b (φ) dm. S

S

S

These equalities imply 7 7 7 7 cφ dm = c (φ) dm + ic (φ) dm = c φ dm. S

S

S

S



May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 508

Mathematical Analysis for Machine Learning and Data Mining

508

An analog of Theorem 8.28 for complex measurable function is given next. ˆ be two integrable functions on the Theorem 8.36. Let f, g : S −→ C measure space (S, E, m) ranging over the extended set of complex numbers ˆ Then f + g is also integrable and C. 7 7 7 (f + g) dm = f dm + g dm. S

S

S

Proof. If both f and g are integrable, by Theorem 8.31 there exist two integrable functions f1 , g1 : S −→ C such that f = f1 a.e., g = g1 a.e., 6 6 6 6 f dm = S f1 dm, and S g dm = S g1 dm. Therefore,6 f + g = f1 + g1 S a.e., and it 6suffices to show 6that f1 + g1 is integrable and 6S (f1 + g1 ) dm = 6 6 f dm + g dm. Since |f + g | dm  |f | dm + S |g1 | dm, hence 1 S 1 S 1 S 1 S 1 f1 + g1 is integrable. By Theorem 8.28 we have: 7 7 7 (f1 + g1 ) dm = (f1 ) dm + (g1 ) dm, S S S 7 7 7 (f1 + g1 ) dm = (f1 ) dm + (g1 ) dm, hence 8.3

6

S (f1

S

+ g1 ) dm =

6 S

S

f1 dm +

6 S

S



g1 dm.

The Dominated Convergence Theorem

Theorem 8.37. (The Dominated Convergence Theorem) Let (S, E, m) be a measure space and let f = (f0 , f1 , . . .) be a sequence of functions such that |fi | < g a.e., where g is an integrable function. If f (x) = limn→∞ fn (x) for every x ∈ S, then f is integrable and 7 7 lim fn dm = f dm. n→∞

S

S

Proof. Suppose initially that the functions fn are non-negative. By Fatou’s Lemma (Theorem 8.12) we have 7 7 7 f dm = lim inf fn dm  lim inf fn dm. S

S

n→∞

Thus, it suffices to show that lim supn→∞

6 S

n→∞

fn dm 

S

6 S

f dm.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 509

509

An application of Fatou’s Lemma to the sequence of non-negative functions (g − f1 , g − f2 , . . .) yields 7 7 lim (g − fn ) dm  lim inf (g − fn ) dm. S n→∞

n→∞

Note that 7

7

7

lim (g − fn ) dm =

7

(g − f ) dm =

S n→∞

Also,

S

S

g dm − S

f dm. S

 7 7 (g − fn ) dm = lim inf g dm − fn dm n→∞ S S S 7 = g dm − lim sup fn dm.

7 lim inf

n→∞

n→∞

S

Thus, we have 7

7 7 g dm − f dm  g dm − lim sup fn dm. n→∞ S S 6 6 6 S Since g is integrable, S g dm is finite, so lim supn→∞ S fn dm  S f dm, which gives the needed equality for non-negative functions. If f0 , f1 , . . . are general measurable functions (that is, not necessarily non-negative), since |fn |  g, it follows that 0  fn (x) + g(x)  2g(x) for x ∈ S. Then, the sequence of non-negative functions fn + g is dominated by the integrable function 2g, and we apply the first part of the argument, so 7 7 (fn + g) dm =

lim

n→∞

S

(f + g) dm, S

which gives the equality in the general case.



Theorem 8.38. Let (S, E, m) be a measure space. 6 If (fn ) is a sequence  |f | dm < of measurable functions on S such that 6 ∞, then  n1 S n  f (x) exists a.e. on S. If f = n1 fn a.e., then S f dm = n1 n f dm. n n1 ˆ 0 be the function defined by g =  Proof. Let g : S −→ 6R n1 |fn |. 6  Lemma 8.1 implies that S g dm = n1 S |fn | dm < ∞, hence g < ∞  a.e. on S. Therefore, the series n1 fn (x) converges absolutely, hence it converges a.e. on S. n  Let sn = m1 fm . Then, m=1 fm be a partial sum of the series limn→∞ sn = f a.e. on S and |sn |  g. The Dominated Convergence Theorem implies 7 7 lim sn dm = f dm.  n→∞ S

S

May 2, 2018 11:28

510

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 510

Mathematical Analysis for Machine Learning and Data Mining

Theorem 8.39. (Beppo Levi’s Theorem) Let (S, E, m) be a measure space and let f = (f0 , f61 , . . . , fn , . . .) be a sequence of measurable functions. n  If the series n∈N S |fn | dm is convergent, then n=1 fn (x) converges n a.e., its sum f (x) = n=1 fn (x) is integrable, and 7 7 f dm = fn dm. S

Proof. and

The function h(x) =



S

n∈N n∈N

7 h dm = S

|fn (x)| is non-negative, measurable,

7 n∈N

|fn | dm,

S

6  by Lemma 8.1. Since n∈N S |fn | dm is finite, h is integrable, which implies that h is finite almost Define the function φ as ⎧ everywhere. ⎪ fn (x) if f (x) exists, ⎨ φ(x) = n∈N n∈N ⎪ ⎩0 otherwise.  Note that the set on which n∈N fn (x) is null.  We have | kn fk (x)|  φ(x). By the Dominated Convergence Theorem (Theorem 8.37) we obtain 7 7 7   fk dm = lim fk dm f dm = lim n→∞

= lim

n→∞

7 kn

which concludes the argument.

n→∞

kn

fk dm =

 7

kn

fk dm,

k∈nn



ˆ and g : S −→ R ˆ be two measurable Theorem 8.40. Let f : S −→ R functions relative6to the measure 6 space (S, E, m) such that f (x) = g(x) a.e. Then, if one of S f dm or S g dm exists the other exists and they are equal. Proof. We need to deal only with the case when both f and g are nonnegative. If this is not the case, the same conclusion can be reached by decomposing f and g into their positive and negative components. Let A = {x ∈ S | f (x) =6g(x)}. By hypothesis, 6 m(A) = 0. If hn = n1A we have S hn dm = 0, so S IA dm = 0 because limn→∞ hn = IA , where IA is the indicator function of A (introduced in Definition 1.14. 6 6 Note 6 that we6 have both f  g+IA6 and g  f6+IA , so S f dm  S g dm  and S g dm  S f dm. Therefore, S f dm = S g dm.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 511

511

Theorem 8.41. (Markov’s Inequality) Let (S, E, m) be a measure space, f : S −→ R0 be a measurable function, and let a be a positive number. We have: 7 1 m({x ∈ S | f (x)  a})  f dm. a S Proof.

Since f (x)  0, we have for T = {x ∈ S | f (x)  a}: 7 7 f dm  f dm  am(T ), S

T



which implies the desired inequality.

Let f : S −→ R0 be6 a measurable function on the measure space (S, E, m) and let E(f ) = S f dm. Define the measurable function g : S −→ R by g(x) = (f − E(f ))2 . The variance of f , var(f ) is the number E(g). By Markov’s inequality applied to g and to a = (kvar(f ))2 we have m({x ∈ S | (f (x) − E(f ))2  (kvar(f ))2 }) 

1 E(g), (kvar(f ))2

which now becomes m({x ∈ S | |f (x) − E(f )|  kvar(f )}) 

1 . k2

(8.6)

This inequality is known as Chebyshev’s inequality. Corollary 8.5. Let f : S −→ R be an integrable function defined on the measure space (S, E, m). The set {x ∈ S | f (x) = 0} is σ-finite relative to m. Proof.

By applying Markov’s inequality to the function |f | we have 7 1 (8.7) m({x ∈ S | |f (x)|  })  n |f | dm, n S

which means that each set {x ∈ S | |f (x)|  statement follows by observing that

1 n}

has a finite measure. The

{x ∈ S | f (x) = 0} = {x ∈ S | |f (x)| > 0} *  1 = x ∈ S | |f (x)|  . n n1



ˆ Corollary 8.6. Let f : S −→ 6 R be an integrable function defined on the measure space (S, E, m). If S |f | dm = 0, then f (x) = 0 a.e. on S.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 512

Mathematical Analysis for Machine Learning and Data Mining

512

Proof. By inequality (8.7), the hypothesis implies m({x ∈ S | |f (x)|  1 n }) = 0 for n  1. Furthermore, since      1  , x ∈ S |f (x)|  {x ∈ S | f (x) = 0} =  n n1

by the subadditivity of m it follows that m({x ∈ S | f (x) = 0}) = 0, so f (x) = 0 a.e. on S.  ˆ be an integrable function defined on the Corollary 8.7. Let f : S −→ R measure space (S, E, m). Then |f (x)| < ∞ a.e. on S. Proof.

An application of Markov’s Inequality yields 7 1 m({x ∈ S | |f (x)|  n})  |f | dm n S

for each n  1. Therefore, m({x ∈ S | |f (x)| = ∞})  m({x ∈ S | |f (x)|  n}) 7 1  |f | dm n S for every n  1, which implies m({x ∈ S | |f (x)| = ∞}) = 0, a statement equivalent to what we aim to prove.  ˆ be Corollary 8.8. Let (S, E, m) be a measure space and let f : S −→ R ˆ a measurable function on this space ranging in R. Then f is integrable if and only if there exists a function g : S −→ R that is integrable such that f (x) = g(x) a.e. Proof. If there exists a function g : S −→ R that is integrable such that f (x) = g(x) a.e., then f is integrable by Theorem 8.40. Conversely, suppose that f is integrable and let A = {x ∈ S | |f (x)| = ∞}. By Corollary 8.7, m(A) = 0, and f0 = f 1A equals f a.e. The function  f0 : S −→ R is integrable. 8.4

Functions of Bounded Variation

In Chapter 4 we have shown that for monotonic functions all discontinuity points are of first kind, that is, the function has lateral limits in such points (see Theorem 4.61). Next, we show that if f : [a, b] −→ R is an increasing function, then f is differentiable a.e.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 513

513

Let f : [a, b] and let x ∈ (a, b). The derivatives of f in x are D+ f (x) = lim suph↓0 D− f (x) = lim suph↑0

f (x+h)−f (x) , h f (x+h)−f (x) , h +

D+ f (x) = lim inf h↓0 D− f (x) = lim inf h↑0

f (x+h)−f (x) , h f (x+h)−f (x) . h

It is immediate that D f (x)  D+ f (x) and D− f (x)  D− f (x). Definition 8.7. A function f : [a, b] is differentiable in x ∈ (a, b) if D+ f (x) = D+ f (x) = D− f (x) = D− f (x). In this case the common value of these derivatives is denoted by f  (x). If D+ f (x) = D+ f (x), we say that f has a right derivative in x and we denote their common value by f  (x+); if D− f (x) = D− f (x), we say that f has a left derivative and their common value is denoted by f  (x−). If f is continuous on [a, b] and one of its derivatives is non-negative on (a, b), then f is non-decreasing on [a, b]. For example, if D+ f (x) > 0 for (x) > 0, which means that x ∈ (a, b), we have lim suph↓0 f (x+h)−f h f (x + h) − f (x) | 0 < h < r} > 0, h by Example 4.50. This implies f (x + h)  f (x), so f is non-decreasing on (a, b) and, by continuity, on [a, b]. inf sup{

r>0

Theorem 8.42. Let f : [a, b] −→ R be an increasing function. The set where any two derivatives of f are distinct have measure 0. Proof.

Let E = {x ∈ [a, b] | D+ f (x) > D− f (x)}.

For u, v ∈ Q define Eu,v = {x | D+ f (x) > u > v > D− f (x)}. Clearly, E is the countable union of sets Eu,v , hence it suffices to prove that μL (Eu,v ) = 0. For > 0 there exists an open set U such that mL (U ) < μL (Eu,v ) + . For each x ∈ Eu,v there exists [x−h, x] ⊆ U such that f (x)−f (x−h) < vh. By Vitali’s Theorem (Theorem 7.62) there exists a collection of such intervals [x1 − h1 , x1 ], . . . , [xN − hN , xN ] whose interiors cover a subset A of Eu,v and f (xn ) − f (xn − hn ) < such that μL (A) > μL (Eu,v ) − . This implies N 

(f (xn ) − f (xn − hn )) < v

n=1

N 

hn < vmL (U ) < v(μ(Eu,v ) + ).

n=1

For y ∈ A there exists an interval (y, y+k) that is contained in some interval [xn − hn , xn ] such that f (y + k) − f (y) > uk. By Vitali’s Theorem, there

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 514

Mathematical Analysis for Machine Learning and Data Mining

514

 exists a collection (y1 , y1 +k1 ), . . . , (yM , yM +kM ) such that M i=1 (yi , yi +ki ) contains a subset C of A with μL (C) > mL (Eu,v ) − 2 . This implies M 

(f (yi + ki ) − f (yi )) > u

i=1

M 

ki > u(mL (Eu,v ) − 2 ).

i=1

Since each interval (yi , yi + ki ) is contained in some interval (xj − hj , xj ) and f is increasing, we have  {f (yi + ki ) − f (yi )) | (yi , yi + ki ) ⊆ (xn − hn , xn )}  f (xn ) − f (xn − hn ). Therefore, N 

(f (xn ) − f (xn − hn )) 

n=1

M 

(f (yi + ki ) − f (yi )),

i=1

hence v(mL (Eu,v)+ )  u(mL (Eu,v)−2 ) for every positive , that is, vmL (Eu,v )  umL Eu,v . Since u > v, this implies mL (Eu,v ) = 0. Thus, (x) is defined a.e. and that f is differentiable if g is g(x) = limh→0 f (x+h)−f h finite.  Corollary 8.9. Let f : [a,6b] −→ R be an increasing function. The derivab tive f  is measurable and a f  dmL  f (b) − f (a). Proof. By Theorem 8.42 f is differentiable a.e. Let g(x) = 1 f (x+ n )−f (x) f (x+h)−f (x) and let gn (x) = . Define f (x) = f (b) for limh→0 1 h n x  b. We have gn (x)  0 because f is increasing and limn→∞ gn (x) = g(x) a.e., so g is measurable. By Fatou’s Lemma (Theorem 8.12) we have 7 b 7 b g dmL = lim gn dmL a

a n→∞

7

 lim inf

7

b

a 7 = lim inf n  = lim inf

b

(f (x + 1/n) − f (x)) dmL 

gn d;L = lim inf n

a 7 a+1/n

b+1/n

f dmL −

b

dmL a

7 f (b) − n

a+1/n

f dmL

  f (b) − f (a).

a

Therefore, g is integrable and finite a.e. Thus, f is differentiable and g = f  a.e.  Definition 8.8. Let [a, b] be an interval on R; a subdivision of [a, b] is a finite set Δ = {x0 , x1 , . . . , xn } such that a = x0 < a1 < · · · < xn = b. The members of Δ are the subdivision points of [a, b].

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 515

515

We denote ν(Δ) = max{ai − ai−1 | 1  i  n}. The set of subdivisions of [a, b] is denoted by SUBD[a, b]. A subdivision Δ = {a0 , a1 , . . . , am } of [a, b] is a refinement of a subdivision Δ = {a0 , a1 , . . . , an } of the same interval if Δ ⊆ Δ . This is denoted by Δ  Δ . It is clear that Δ  Δ implies ν(Δ )  ν(Δ). Definition 8.9. Let f : [a, b] −→ R be a real-valued function and let Δ = {x0 , x1 , . . . , xn } ∈ SUBD[a, b]. The variation of f on the interval [a, b] with the subdivision Δ is the b number Va (f, Δ) defined as n 

Vba (f, Δ) =

|f (xi ) − f (xi−1 )|.

i=1

The positive variation of f with the subdivision Δ is +

Vba (f, Δ) =

n 

max{f (xi ) − f (xi−1 ), 0}.

i=1

Similarly, the negative variation of f with the subdivision Δ is −

Vba (f, Δ) =

n 

− min{f (xi ) − f (xi−1 ), 0}.

i=1

It is easy to verify that max{r, 0} − min{r, 0} = |r| and max{r, 0} + min{r, 0} = r for every r ∈ R. Therefore, we have +

−

Vba (f, Δ)+ Vba (f, Δ) = Vba (f, Δ), + b a

− b a

V (f, Δ)− V (f, Δ) =

n 

(f (xi ) − fxi−1 ) = f (b) − f (a),

(8.8) (8.9)

i=1

and these equalities imply +

1 2

Vba (f, Δ) = (Vba (f, Δ) + f (b) − f (a)),

(8.10)

−

1 2 Equalities (8.10) and (8.11) imply

Vba (f, Δ) = (Vba (f, Δ) − f (b) + f (a)). +

−

Vba (f, Δ)+ Vba (f, Δ) = Vba (f, Δ).

(8.11)

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 516

Mathematical Analysis for Machine Learning and Data Mining

516

Note that if Δ, Δ ∈ SUBD[a, b] and Δ ⊆ Δ , then +

+

−

−

Vba (f, Δ) 

Vba (f, Δ ), Vba (f, Δ) Vba (f, Δ ), and Vba (f, Δ) Vba (f, Δ ). Definition 8.10. Let f : [a, b] −→ R be a real-valued function. The varib ation of f on the interval [a, b] is the number Va f defined as

Vba f = sup{Vba (f, Δ) | Δ ∈ SUBD[a, b]}. The positive variation of f on [a, b] is +

+

Vba f = sup{Vba (f, Δ) | Δ ∈ SUBD[a, b]}. Similarly, the negative variation of f on [a, b] is the number −

−

Vba f = sup{Vba (f, Δ) | Δ ∈ SUBD[a, b]}. b The function is of bounded variation over [a, b] if Va f is finite. From equalities (8.8) and (8.9) it follows that +

1 2

(8.12)

1 2

(8.13)

Vba f = (Vba f + f (b) − f (a)), −

Vba f = (Vba f − f (b) + f (a)), and

+ b a

− b a

V f + V f = Vba f .

Theorem 8.43. A function f : [a, b] −→ R is of bounded variation if and only it equals the difference of two monotone increasing real-valued functions on [a, b]. Proof.

Let f be a function of bounded variation. Define the functions + x

− x

g, h : [a, b] −→ R as g(x) =Va f and h(x) =Va f . Both g and h are b monotone increasing. Since Va f is finite, both g and h are finite. Conversely, if f = g − h, where g, h are monotone increasing functions on [a, b], then for any Δ = {x0 , x1 , . . . , xn } ∈ SUBD[a, b] we have n  Vba (f, Δ) = |f (xi ) − f (xi−1 )| i=1



n 

(g(xi ) − g(xi−1 )) +

i=1

n 

(h(xi ) − h(xi−1 ))

i=1

= g(b) − g(a) + h(b) − h(a), hence

V

b af

 g(b) − g(a) + h(b) − h(a).



May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Integration

b3234-main

page 517

517

Definition 8.11. A function f : [a, b] −→ R is absolutely continuous on [a, b] if for each > 0 there exists δ > 0 such that for every finite collection of n pairwise disjoint intervals {(ui , vi ) | 1  i  n} such that i=1 (vi −ui ) < δ n we have i=1 |f (vi ) − f (ui )| < . Theorem 8.44. If a function f : [a, b] −→ R is absolutely continuous then it of bounded variation on [a, b]. Proof. Let Δ be a subdivision of [a, b] and let > 0. There exists δ > 0 b and a refinement Δ of Δ such that ν(Δ ) < δ, which implies Va (f, Δ ) < . b b  Thus, Va (f, Δ) < for any Δ, hence Va f < . 8.5

Riemann Integral vs. Lebesgue Integral

We present the fundamentals of Riemann integration, a topic that is usually presented in basic calculus. As we shall see, the class of Lebesgue-integrable functions is broader than the class of Riemann-integrable functions; in many cases, the value of the integrals are the same. However, computing effectively the Lebesgue integral is difficult and Riemann integration benefits from the multitude of integration techniques developed in classical analysis. Therefore, it is important to elucidate the relationships that exists between these types of integration. Recall that the notion of subdivision of an interval was introduced in Definition 8.8. If f : [a, b] is a bounded function on [a, b] and Δ = {a0 , a1 , . . . , an } ∈ SUBD[a, b] define mi = inf{f (x) | x ∈ [ai−1 , ai ]}, and Mi = sup{f (x) | x ∈ [ai−1 , ai ]} for 1  i  n. The lower Darboux sum and upper Darboux sum for f and Δ are given by n n   mi (ai − ai−1 ) and S(f, Δ) = Mi (ai − ai−1 ), s(f, Δ) = i=1

i=1

respectively. These sums coincide with the Lebesgue integrals of the simple n n functions Lf,Δ = i=1 mi 1[ai−1 ,ai ] and Uf,Δ = i=1 Mi 1[ai−1 ,ai ] , respectively, that is, 7 7 s(f, Δ) = Lf,Δ dm and S(f, Δ) = Uf,Δ dm. [a,b]

[a,b]

For a subdivision Δ of [a, b] let Ξ = {ξ1 , . . . , ξn } be a set of numbers such that ξi ∈ [ai−1 , ai ] for 1  i  n. The Riemann sum that corresponds

May 2, 2018 11:28

518

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 518

Mathematical Analysis for Machine Learning and Data Mining

to Ξ is the number σ(f, Δ, Ξ) given by σ(f, Δ, Ξ) =

n 

f (ξi )(ai − ai−1 ).

(8.14)

i=1

If m = inf{f (x) | x ∈ [a, b]} and M = sup{f (x) | x ∈ [a, b]}, then we have m(b − a)  s(f, Δ)  σ(f, Δ, Ξ)  S(f, Δ)  M (b − a). Indeed, we have m  mi  f (ξi )  Mi  m for every ξ ∈ [ai−1 , ai ], which implies immediately the desired inequality. It is easy to verify that s(f, Δ) = inf σ(f, Δ, Ξ) and S(f, Δ) = sup σ(f, Δ, Ξ). Ξ

Ξ

Moreover, we can prove a stronger claim. Theorem 8.45. Let Δ, Δ ∈ SUBD[a, b] such that Δ  Δ . For any bounded function f on [a, b] we have s(f, Δ)  s(f, Δ )  S(f, Δ )  S(f, Δ). Proof. Let Δ = {a0 , a1 , . . . , an } be a subdivision of [a, b] and let Δ be a subdivision of the same interval such that Δ < Δ and Δ is given by Δ = {a0 , a1 , . . . , ai , c, ai+1 , . . . , an }, where a0 < a1 < · · · < ai < c < ai+1 < · · · < an . For mi = inf{f (x) | x ∈ [ai−1 , ai ]}, mi = inf{f (x) | x ∈ [ai−1 , c]}, and mi = inf{f (x) | x ∈ [c, ai ]} we obviously have mi  mi and mi  m . Therefore, mi (ai − ai−1 )  mi (c − ai−1 ) + mi (ai − c). Thus, s(f, Δ)  s(f, Δ ) and this inequality clearly holds when Δ has more division points in the interval [ai−1 , ai ]. This leads to the inequality s(f, Δ)  s(f, Δ ). The inequality S(f, Δ )  S(f, Δ) can be shown in a similar manner noting that if Mi = sup{f (x) | x ∈ [ai−1 , ai ]}, Mi = sup{f (x) | x ∈ [ai−1 , c]}, and Mi = sup{f (x) | x ∈ [c, ai ]}, we have Mi  Mi and Mi  Mi . Since the inequality s(f, Δ )  S(f, Δ ) obviously holds, the lemma is proven.  Theorem 8.46. Let Δ1 , Δ2 ∈ SUBD[a, b]. For any bounded function f on [a, b] we have s(f, Δ1 )  S(f, Δ2 ).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 519

519

ˆ be the subdivision of [a, b] whose set of points is the union Proof. Let Δ ˆ and Δ2  Δ. ˆ of the sets of points of Δ1 and Δ2 . Clearly, we have Δ1  Δ Therefore, by Theorem 8.45 we have ˆ  S(f, Δ) ˆ  S(f, Δ1 ), and s(f, Δ1 )  s(f, Δ) ˆ ˆ  S(f, Δ2 ). s(f, Δ2 )  s(f, Δ)  S(f, Δ) These inequalities give the desired inequality.



Theorem 8.47. For any bounded function f on [a, b] we have: sup{s(f, Δ) | Δ ∈ SUBD(Δ)}  inf{S(f, Δ) | Δ ∈ SUBD(Δ)}. Proof. Let Δ0 be a subdivision of [a, b]. For any Δ ∈ SUBD[a, b] we have s(f, Δ)  S(f, Δ0 ), so S(f, Δ0 ) is an upper bound of the set {s(f, Δ) | Δ ∈ SUBD(Δ)}. Therefore, sup{s(f, Δ) | Δ ∈ SUBD(Δ)} exists and sup{s(f, Δ) | Δ ∈ SUBD(Δ)}  S(f, Δ0 ). Since this inequality holds for every Δ0 , it follows that sup{s(f, Δ) | Δ ∈ SUBD(Δ)}  inf{S(f, Δ) | Δ ∈ SUBD(Δ)}.



The lower Riemann integral of f on [a, b] is defined as 7 b f dx = sup{s(f, Δ)|Δ ∈ SUBD([a, b])}. a

Similarly, the upper Riemann integral is 7 b f dx = inf{S(f, Δ)|Δ ∈ SUBD([a, b])}. a

Theorem 8.47 means that for every Δ ∈ SUBD([a, b]) we have 7 b 7 b s(f, Δ)  f dx  f dx  S(f, Δ). a

a

Definition 8.12. A function f : [a, b] −→ R is Riemann integrable if 6b 6b a f dx = a f dx. Example 8.5. The function f : [0, 1] −→ R defined by  1 if x ∈ Q ∩ [0, 1], f (x) = 0 otherwise,

May 2, 2018 11:28

520

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 520

Mathematical Analysis for Machine Learning and Data Mining

61 61 is not integrable Riemann because 0 f dx = 0 while 0 f dx = 1. This is inconvenient because f is 0 a.e. (the set of rational numbers in [0, 1] is a null set by Corollary 7.22). However, f is Lebesgue integrable on [0, 1]. Since the set Q1 = {x ∈ Q ∩ [0, 1]} is countable, we can write Q1 = {q1 , q2 , . . .}. Let # $ In = qn − n+1 , qn + n+1 2 2 be an open interval of length 2n centered in qn for n  1. Thus, Q1 ⊆ S ,  where S = n1 In is a Lebesgue measurable set. Note that 1Q  1S . 1 Since 7  1S dmL  < , mL (S ) 2n [0,1] 6

it follows that [0,1] f dmL  6 implies [0,1] f dmL = 0.

n1

6 [0,1]

1S dmL < , for every > 0, which

The next statement shows that the Lebesgue integral is a generalization of the Riemann integral. Theorem 8.48. Let f : [a, b] −→ R be a bounded function defined 6 b on [a, b]. If f is Riemann integrable on [a, b], then it is measurable, and a f (x) dx = 6b f dm. a Proof.

We have the obvious inequalities 7 b f dx = sup{Lf,Δ|Δ ∈ SUBD([a, b])} a

7  inf{Uf,Δ |Δ ∈ SUBD([a, b])} =

b

f dx. a

6b 6b Since f is Riemann integrable, we have a f dx = a f dx, which implies the Lebesgue integrability of f and the equality of the theorem.  Theorem 8.49. Let f : [a, b] −→ R be a continuous function. Then f is Lebesgue integrable on [a, b]. Furthermore, the function F : [a, b] −→ R 6 defined by F (x) = [a,x] f (x) dm is differentiable on (a, b) and its derivative is F  (x) = f (x) for x ∈ (a, b). Proof. Since continuous functions are measurable and f is bounded on 6 [a, b], the integral [a,b] |f | dm exists.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 521

521

Let x and h be such that a < x < x + h < b. We have 7 F (x + h) − F (x) = f dm. [x,x+h]

Let m = inf{f (t) | t ∈ [x, x + h]} and M = sup{f (t) | t ∈ [x, x + h]}. Both m and M are attained (by Theorem 4.109), so there exist t1 , t2 ∈ [x, x + h] such that m = f (t1 ) and M = f (t2 ). By the Mean 6 Value Theorem for Lebesgue Integrals (Theorem 8.18) we have f (t1 )  h1 [x,x+h] f dm  f (t2 ). Therefore, F (x + h) − F (x)  f (t2 ). h The Intermediate Value Theorem (Theorem 5.58) implies the existence of (x) . Taking limh→0 we obtain θ ∈ [0, 1] such that f (x + θh) = F (x+h)−F h   that f (x) = F (x). f (t1 ) 

Let (Δn ) be a sequence of subdivisions of the interval [a, b]. Note that  the set of all subdivision points n∈N Δn is a countable set and, therefore, a null set. Theorem 8.50. Let f : [a, b] −→ R be a bounded functions and let (Δn ) a sequence of subdivisions of [a, b] with limn→∞ ν(Δn ) = 0.  For x ∈ [a, b] − n∈N Δn define l(x) = sup Lf,Δn (x) and u(x) = inf Uf,Δn (x). The function f is continuous in x if and only if u(x) = f (x) = l(x). Proof. Suppose that f is continuous on [a, b]. Since x is not a subdivision point, we have Lf,Δ0 (x)  Lf,Δ1 (x)  · · ·  f (x)  · · ·  Uf,Δ1 (x)  Uf,Δ0 . By Theorem 7.15 both l = limn→∞ Lf,Δn and u = limn→∞ are measurable functions.  Theorem 8.51. Let f : [a, b] −→ R be a bounded function. The, f is Riemann-integrable if and only if f is a.e. continuous relative to the Lebesgue measure on [a, b]. If f is Riemann-integrable, then f is Lebesgue-integrable and the two integrals are equal. Proof. Suppose that f is Riemann-integrable. There exists a chain of subdivisions Δ1  · · ·  Δn  · · · such that S(f, Δn ) − s(f, Δn ) < n1 for n  1.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 522

Mathematical Analysis for Machine Learning and Data Mining

522

For each subdivision Δn define the simple functions gΔn and h Δn on [a, b] such that gΔn (a) = hΔn (a) = f (a) and gΔn (x) = inf{f (x) | x ∈ [ai−1 , ai ]} if x ∈ (ai−1 , ai ] and hΔn (x) = sup{f (x) | x ∈ [ai−1 , ai ]} if x ∈ (ai−1 , ai ]. Since f is bounded, the terms of the sequences (gΔn ) and (hΔn ) are bounded. Furthermore, (gΔn ) is an non-decreasing sequence of functions, (hΔn ) is an non-increasing sequence of simple functions, gΔn  f  hΔn , and 7 7 gΔn dmL = s(f, Δn ), hΔn dm = S(f, Δn ). [a,b]

[a,b]

The functions g = limn→∞ gΔn and h = limn→∞ hΔn are measurable, 7 7 g dmL = lim s(f, Δn ) and h dmL = lim S(f, Δn ) n→∞

[a,b]

n→∞

[a,b]

by the Dominated Convergence Theorem (Theorem 8.37). Therefore, 7 7 b 7 g dmL = h dmL = f dx. 6

[a,b]

[a,b]

a

Consequently, [a,b] (h − g) dmL = 0. Since h − g  0, it follows that g = h a.e. in [a, b] by Corollary 8.6.  If g(x) = f (x) and x ∈ [a, b] − n1 Δn , then f is continuous x, and therefore, it is a.e. continuous relative to the Lebesgue measure on [a, b]. Since g(x)  f (x)  h(x), it follows that f (x) = g(x) a.e. Thus, f is Lebesgue integrable by Theorem 8.40 and the Riemann integral equals the Lebesgue integral. Suppose that f is continuous a.e. in [a, b]. Let now Δn be the subdivision of [a, b] which divides [a, b] is equal subintervals of length b−a n . At each x where f is continuous we have lim gΔn = lim hΔn = f (x),

n→∞

n→∞

hence there equalities hold a.e. on [a, b]. Thus, we have limn→∞ (hΔn − gΔn ) = 0 a.e. on [a, b]. Since 7 7 gΔn dm = s(f, Δn ), hΔn dm = S(f, Δn ), [a,b]

[a,b]

it follows that limn→∞ (S(f, Δn ) − s(f, Δn )) = 0, which implies that f is Riemann integrable. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 523

523

ˆ such Definition 8.13. Let f : (a, b) −→ R be a function, where a, b ∈ R that f is Riemann integrable on all intervals [c, d] included in (a, 6c 6t 6 b b). If both limt→a,ta t f (x) dx and limt→b,tb c f (x) dx exist, then a f (x) dx is called a convergent improper integral; its value is defined as: 7 c 7 t 7 b f (x) dx = lim f (x) dx + lim f (x) dx. t→a,ta

a

t→b,tb

t

c

Note that the choice of c is immaterial to the convergence or the value of the convergent improper integral. If f is integrable over [a, b], then the value of the improper integral exists and is equal to the Riemann integral. 6∞ Example 8.6. Consider the Riemann integral −∞ e−|x| dx. Choosing c = 0 and taking into account that limt→−∞ et = 0 we have 7 0 7 0 e−|x| dx = lim ex dx = lim ex |0t = 1, lim t→−∞

lim

hence

6∞ −∞

t→∞

e

−|x|

t→−∞

t

7

0

t

e−|x| dx = lim

t→∞

7 0

t t

t→−∞

e−x dx = lim −e−x |t0 = 1,

dx = 2, which shows that

6∞ −∞

t→∞

e−|x| dx is convergent.

If f 6: [a, ∞) −→ R is Riemann integrable on every interval [a, b], and t lim f (x) dx exists, then we have the one-sided improper integral 6t 6 ∞t→∞ a 6∞ f (x) dx defined as a 6 f (x) dx = limt→∞ a f (x) dx. Similarly, the onea n sided improper integral −∞ f (x) dx exists if f (x) is Riemann integrable 6b 6n on [a, b] and lima→−∞ a f (x) dx exists. In this case, −∞ f (x) dx = 6b lima→−∞ a f (x) dx. Example 8.7. Consider the continuous function f : R −→ R defined by ⎧ ⎨ sin x if x = 0, x f (x) = ⎩1 otherwise 6t and the integral 0 f (x) dx. Integrating by parts yields t 7 7 t t − cos x  − cos x f (x) dx = − dx  − x  x2 a a a 7 t cos x cos a cos t − − = dx a t x2 a 6t 6t x dx|  a x12 dx, and the fact for t  a. Since limt→∞ cost t = 0, | a cos 2 x 6t 6t x that a x12 dx is convergent, it follows that a cos x2 dx is convergent.

May 2, 2018 11:28

524

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 524

Mathematical Analysis for Machine Learning and Data Mining

Note that    7 (n+1)π  n 7 (k+1)π      sin x   sin x     dx =  dx  x   x  π k=1 kπ n 7 π n 7 π   | sin(t + kπ)| | sin t| = dt = dt t + kπ t + kπ k=1 0 k=1 0 7 π n n  2 1 1 ,  | sin t| dt = (k + 1)π 0 π k+1 k=1 k=1 6∞ which shows that 0 | sinx x | dx is not convergent, because the harmonic series is not convergent. This, in turn, implies that the function f is not Lebesgue 6 ∞ sin x integrable (by Theorem 8.22). However, the Riemann integral x dx exists. 0 Example 8.7 shows that the existence of improper Riemann integrals does not necessarily entail the existence of the corresponding Lebesgue integral. However, if f is non-negative this implication holds as we show next. Theorem 8.52. If f : R −→ R is Riemann-integrable on R and f (x)  0 for x ∈ R, then f is Lebesgue-integrable on the same set and 7 7 f (x)dx = f dm. R

R

Proof. Let (fn ) be the sequence of functions defined by fn = f 1[−n,n] for n  1. Observe that for every x ∈ R we have fn (x)  fn+1 (x) for n  1, so (fn ) is a monotonic sequence and 6limn→∞ fn = f .6 Since each n fn is Riemann integrable on [−n, n] we have [−n,n] fn dm = −n f dx, so 6 6n fn ∈ L1 (R) for n  1. By definition, R f dx = limn→∞ −n f dx. By (Theorem 8.11) we have 7 7 f dm = lim fn dm R

n→∞

R

(by the Monotone Convergence Theorem) 7 7 = lim fn dx = f dx, n→∞

1

R

hence f ∈ L (R) and proof is complete.

R



In general, we use Lebesgue integrals. When integrating continuous functions, the use of Lebesgue integrals causes no problem since, in this case, the Riemann and Lebesgue integrals coincide.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

8.6

page 525

525

The Radon-Nikodym Theorem

Definition 8.14. Let (S, E, m) be a measure space and let m be a signed measure defined on E. The measure m is absolutely continuous with respect to the measure m if m(T ) = 0 implies m (T ) = 0 for every T ∈ E. This is denoted by m # m. We have m # m if and only if |m |(T ) = 0 for every T ∈ E with m(T ) = 0. Example 8.8. Let (S, E) be a measurable space and let m, m be two measures defined on E such that m (U )  m(U ) for every U ∈ E. Then, m # m. Theorem 8.53. Let m be a signed finite measures on the measurable space (S, E). We have m # m if and only if for every positive number there exists a positive number δ such that if T ∈ E, then m(T ) < δ implies |m |(T ) < . Proof. To establish that the condition is sufficient let T ∈ E be a set such that m(T ) = 0. Then m(T ) < δ, hence |m |(T ) < for every positive . Therefore, |m |(T ) = 0, hence m # m. To establish that the condition is necessary, suppose that m # m. and the , δ-condition is not true. Then, there exists a positive number and a sequence of sets (T1 , T2 , . . .) in E such that m(Tn ) < 21n and m (Tn )  for ∞  every n  1. Define T = lim sup Tn = n=1 in Ti . We have ⎞ ⎛ ∞    1 1 Ti ⎠  m(Ti ) < = n−1 , m(T )  m ⎝ i 2 2 i=n in

in

for every n  1, which implies m(T ) = 0, hence |m |(T ) = 0  Since m is finite, it follows that |m | is finite, hence |m |( i≥n Ti ) < ∞.  Since the sequence ( i≥n Tn ) is increasing, by Theorem 7.32 we have ⎛ ⎛ ⎞ ⎞   |m |(T ) = |m | ⎝lim Ti ⎠ = lim |m | ⎝ Ti ⎠ > , in

in 

which contradicts the absolute continuity of m with respect to m.



Theorem 8.54. Let (S, E, m) be a measure space and let m be a signed measure defined on E. The following statements are equivalent: (i) m # m; (ii) |m | # m; (iii) m+ # m and m− # m.

May 2, 2018 11:28

526

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 526

Mathematical Analysis for Machine Learning and Data Mining

Proof. (i) implies (ii): Since the absolute value of |m | is the same as |m |, this implication follows immediately. (ii) implies (iii): |m | # m implies m+ +m− # m. Since m+ , m− and m are positive measures, we obtain (iii). (iii) implies (i): Suppose that m(T ) = 0. We have m+ # m and  m− # m, hence m+ (t) = m− (T ) = 0. Thus, |m |(T ) = 0 and m # m.  Lemma 8.2. Let (S, E, m) be a measure space such that m(S) is finite and let m be a measure defined on E such that m (T )  m(T ) for every T ∈ E. For a E-partition π of S, π = {L1 , . . . , Lp } define the simple function fπ : S −→ R by ⎧  ⎨ m (Li ) if x ∈ L and m(L ) > 0, i i fπ (x) = m(Li ) ⎩ 0 otherwise. If π1 and π2 are two E-partitions on S7such that π2  π1 , then 7 S

fπ22 dm 

S

fπ21 dm.

Proof. It is easy to see 6 that 0  fπ (x)  1 for all x ∈ S. We have m (Li ) = Li fπ dm for every Li ∈ π; furthermore, if I ⊆  {1, . . . , n} and K = i∈I Li , then  m (Li ) m (K) = i∈I

   m (Li )  = m(Li )i ∈ I, m(Li ) > 0  m(Li ) i∈I  7  7    = fπ dmi ∈ I, m(Li ) > 0 = fπ dm.  Li K 



Suppose that π1 = {B1 , . . . , Bn } and π2 = {C1 , . . . , Cm }. Since π2  π1 , for each block Bi of π1 there exists a family of blocks {Cj | j ∈ Ji } such  that Bi = {Cj | j ∈ Ji }. If m(Cj ) = 0, then m (Cj ) = 0. Consequently, we have    7    m (Cj )    m(Cj )j ∈ Ji , m(Cj )>0 fπ1 dm = m (Bi ) = m (Cj ) =  m(Cj ) Bi j∈Ji   7    fπ2 dmj ∈ Ji =  Cj  7  7    = fπ2 dmj ∈ Ji = fπ2 dm.  Cj Bi

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 527

527

Let Bi be a block of the partition π1 such that m(Bi ) > 0. We can write 7 7 m (Bi ) fπ1 fπ2 dm = fπ dm m(Bi ) Bi 2 Bi (because fπ1 (x) is constant on Bi ) 7 (m (Bi ))2 fπ2 dm = m(Bi ) Bi 2 7 7   m (Bi ) dm = fπ21 dm, = m(B ) i Bi Bi 6 which implies Bi fπ1 (fπ1 − fπ2 ) dm = 0 for every block Bi of π1 . Consequently, 7 n 7  fπ1 (fπ1 − fπ2 ) dm = fπ1 (fπ1 − fπ2 ) dm = 0. S

Bi

i=1

Therefore, 7 7 (fπ1 − fπ2 )2 dm = (fπ21 − 2fπ1 fπ2 + fπ22 ) dm S 7S = (−fπ21 + 2fπ21 − 2fπ1 fπ2 + fπ22 ) dm 7S = (−fπ21 + 2fπ21 − 2fπ1 fπ2 + fπ22 ) dm 7S = (−fπ21 + 2fπ1 (fπ1 − fπ2 ) + fπ22 ) dm 7S 7 2 2 = (fπ2 − fπ1 ) dm + 2 fπ1 (fπ1 − fπ2 ) dm 7S 7 S 7 2 2 2 = (fπ2 − fπ1 ) dm = fπ2 dm − fπ21 dm, S

which shows that 7

2

S

7

Since

6

S (fπ1

7

fπ21 dm.

(8.15)

− fπ2 )2 dm  0, the desired conclusion follows.



(fπ1 − fπ2 ) dm = S

S

fπ22

S

dm − S

Lemma 8.3. Let (S, E, m) be a measure space such that m(S) is finite. If m1 is a finite measure defined on E such that m1 (T )  m(T ) for every T ∈ E, then there exists a non-negative measurable function f : S −→ R 6 such that m1 (T ) = T f dm for every set T ∈ E.

May 2, 2018 11:28

528

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 528

Mathematical Analysis for Machine Learning and Data Mining

Proof. (6Without loss of generality )we may assume that m(S) = 1. Let I = sup S fπ2 dm | π ∈ PARTfin (S) . Since fπ (x) ∈ [0, 1] for all x ∈ S, it follows that 0  I  1. By the definition of the supremum, for every n  1, there exists a partition πn ∈ PARTfin (S) such that 7 1 I− n < fπ2 dm. 4 S  Define the finite partition σn as σn = nl=1 πl . Since σn+1  σn ≤ πn , by Lemma 8.2 we have 7 7 7 1 I− n < fπ2n dm  fσ2n dm  fσ2n+1 dm  I. (8.16) 4 S S S Using equality (8.15) we have 7 7 7 1 2 2 (fσn+1 − fσn ) dm = fσn+1 dm − fσ2n dm < n . 4 S S S By the Cauchy-Schwartz inequality applied to the functions f = fσn+1 −fσn and g = 1 we obtain 7 1 |fσn+1 − fσn | dm < n . 2 S 6 Since S |fσn+1 − fσn | dm is finite, by Beppo Levi’s Theorem, the telescopic  series n1 (fσn+1 − fσn ) is m-convergent a.e., so the function f given by series  (fσn+1 − fσn ) f = fσ1 + n1

is defined a.e. with respect to m. On the m-null set we set f (x) = 0. We claim that the function f satisfies the conditions of the theorem. Consider the two-block E-partition θ = {T, S − T } of S and let ζn = 6 σn ∧ θ. We have m1 (T ) = T fζn dm because T is a ζn -saturated set. Moreover, we have 7 7 1 2 fσn dm  fζ2n dm  I. I− n < 4 S S 6 Therefore, as before, we have S (fζn −fσn )2 dm < 41n . By Cauchy-Schwartz inequality applied to the functions fζn − fσn and IT we have  7    (fζn − fσn ) dm ≤ |fζn − fσn | < 1 .   2n T Observe that 7 7 7 m1 (T ) = fζn dm = (fζn − fσn ) dm + fσn dm T T T 6 for every 6 n  1. 6 Since limn→∞ T (fζn − fσn ) dm = 0, and Theorem limn→∞ T fσn dm = T f dm by the Dominated Convergence 6  (Theorem 8.37) because 0  fσn  1, we have m1 (T ) = T f dm.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 529

529

Corollary 8.10. Let (S, E, m) be a measure space such that m(S) is finite and m(S) > 0. If m1 is a finite measure defined on E such that m1 (T )  m(T ) for every T ∈ E, then there6 exists a non-negative measurable function f : S −→ R such that m1 (T ) = T f dm for every set T ∈ E. Proof.

Consider the measure space (S, E, m1 ), where m1 (T ) =

m(T ) m(S) .

1 (T ) . Clearly, we have m1 (T )  m1 (T ) for every Define m1 by m1 (T ) = mm(S) T ∈ E. By Lemma 68.3, there exists a non-negative measurable function f such that 6m1 (T ) = T f dm1 for every set T ∈ E. This implies immediately  m1 (T ) = T f dm for every set T ∈ E.

Theorem 8.55. Let (S, E, m) be a measure space such that m(S) is finite and m(S) > 0 and let m1 is a finite measure defined on E such that m1 (T )  m(T ) for every6 T ∈ E. If f1 is a measurable function f : S −→ R such that m1 (T ) = T f1 dm for every set T ∈ 6 every non-negative 6 E, then for measurable function g : S −→ R we have S g dm1 = S gf1 dm. that g = IW , where W is a set in E. Then, 6Proof. Suppose initially 6 g dm = m (W ) = f 1 1 S W 1 dm. If g is a simple function, then, by Theorem 7.10, g can be written as a linear combination of indicator functions of sets W1 , . . . , Wn in E, n  yi IWi , g= i=1

which implies 7 7 n n   g dm1 = yi m1 (Wi ) = yi S

=

i=1 n  i=1

7 yi

i=1

IWi f1 dm = S

f1 dm

Wi

7  n S

 yi IWi

7 f1 dm =

gf1 dm. S

i=1

By Theorem 7.26, if g is a non-negative measurable function, then g is the limit of a non-decreasing sequence of simple measurable func6 , . . . , g , . . .). By the previous argument, we have g dm tions (g 1 n n 1 = S 6 g f dm. Since 0  f (x)  1, the sequence g f (x) increases to gf 16 n 1 1 (x), S n 1 6 so limn→∞ S gn f1 dm = S gf1 dm. Therefore, 7 7 7 g dm1 = lim gn dm1 = lim gn f1 dm n→∞ S n→∞ S S 7 7 = lim gn f1 dm = gf1 dm. S n→∞

S

Finally, if g is not non-negative the result follows from a separate application to the functions g + and g − . 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 530

Mathematical Analysis for Machine Learning and Data Mining

530

By relaxing the requirement imposed on the finite measures m and m1 in Lemma 8.3 we obtain the next statement. Theorem 8.56. Let (S, E, m) be a measure space such that m is finite and m(S) > 0. If m1 is a finite measure defined on E such that m1 # m, then there6 exists a non-negative measurable function f : S −→ R such that m1 (T ) = T f dm for every set T ∈ E. Proof. Let m2 be the measure defined by m2 = m + m1 . We have both m(T ) ≤ m2 (T ) and m1 (T )  m2 (T ) for every T ∈ E. Therefore, by Corollary 8.10, there exist two measurable functions f and f1 such that 7 7 m(T ) = f dm2 and m1 (T ) = f1 dm2 . T

T

Consider the sets 6 U = {x ∈ S | f (x) > 0} and V = {x ∈ S | f (x) = 0}. We have m(V ) = V f dm = 0, so m1 (V ) = 0 because m1 # m, so both m and m1 are null on V . (x) · IU (x). If T ⊆ U , then m1 (T ) = Define the function h by h(x) = ff1(x) 6 6 6 f dm2 = T hf dm2 = S hf IT dm2 . By Theorem 8.55 applied to the T 1 function g = hIT we have 7 7 hIT dm = hIT f dm2 , so m1 (T ) =

6

S

hIT dm = S

S

6 T



h dm.

A more general result can be obtained by allowing m to be σ-finite. Theorem 8.57. Let (S, E, m) be a measure space such that m is σ-finite and m(S) > 0. If m1 is a finite measure defined on E such that m1 # m, then there6 exists a non-negative measurable function f : S −→ R such that m1 (T ) = T f dm for every set T ∈ E. Proof. Let (Sn ) be a increasing sequence of sets in E such that   n∈N Sn = S and m(Sn ) < ∞ for n ∈ N. Let mn , mn be defined as    mn (B) = m(B ∩ Sn ) and mn (B) = m (B ∩ Sn ). Since mn # mn , by Theorem 8.56, there exists a sequence 6of functions (fn ) that are positive and mn -integrable such that mn (B) = B fn dmn for n ∈ N and B ⊆ Sn . Note that mn (B) = m(B) and mn (B) = m (B) for every B ∈ E and B ⊆ Sn , and mn (B) = mn (B) = 0 for every B ⊆ S − Sn . Thus, if B ∈ E and B ⊆ Sn ⊆ Sn+1 we have 7 7 7 (fn+1 − fn ) dm = fn+1 dm − fn dm B

B

B

= mn+1 (B) − mn (B) = m (B) − m (B) = 0.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 531

531

Thus, fn+1 = fn a.e. on Bn . Also, and mn -integrable function is mintegrable, hence fn ∈ L1 (S). Now we can define a function f : S −→ R as  f (x) = fn (x) if x ∈ Bn . We have f −1 (−∞, a) = n∈N fn−1 (−∞, a), hence f is positive and measurable. By the upward continuity of measures we obtain 7 7 7 f dm = lim f dm = lim fn dm n→∞

B

= In particular,

6 S

B∩Sn lim m (B ∩ n→∞ n

n→∞

B∩Sn

Sn ) = lim mn (B ∩ Sn ) = m (B). n→∞



f dm = m (S) < ∞, hence f ∈ L1 (S).



Finally, the next statement extends Theorem 8.57 by allowing m1 to be a real measure. Theorem 8.58. (The Radon-Nikodym Theorem) Let (S, E, m) be a measure space such that m is σ-finite and m(S) > 0. If m1 is a real finite measure defined on E such that m1 # m, then there 6exists a non-negative measurable function f : S −→ R such that m1 (T ) = T f dm for every set T ∈ E. The function f is uniquely determined up to a μ-null set. Proof. Since m1 is a real and finite, by Jordan’s Decomposition Theorem, we can write m1 = m1+ − m1− . It is immediate that m1+ # m and 6m1− # m. By Theorem68.57, there exist g, h ∈ L1 (S) such that m1+ (T ) = T g dm and m1− (T ) = T h dm. The function f = g − h satisfies the condition of the theorem. 6 6 1 6 If f1 , f2 ∈ L (S) such that m1 (T ) = T f1 dm = T f2 dm, we have (f − f2 ) dm = 0, hence f1 = f2 a.e.  T 1 Definition 8.15. Let (S, E, m) be a measure space such that m is σ-finite and m(S) > 0. If m1 is a real finite measure defined on E such that 6 m1 # m, 1 the function f ∈ L (S) that satisfies the equality m1 (T ) = T f dm for every set T ∈ E is the Radon-Nikodym derivative of m relative to m . 1 We will denote f by dm dm . ˆ 0 Lemma 8.4. Let (S, E, m) be a measure space6 and let f, g : S −→ R 6 be measurable functions. We have S g dmf = S gf dm, where mf is the indefinite integral of f relative to m. Proof.

Suppose initially that g = 1T for some T ∈ E. We have 7 7 7 1T dmf = mf (T ) = f dm = 1T f dm, S

T

S

which show that the equality of the lemma holds for characteristic functions. This can be extended, by linearity to measurable non-negative simple

May 2, 2018 11:28

532

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 532

Mathematical Analysis for Machine Learning and Data Mining

functions, and, then, by the Monotone Convergence Theorem, to measurable functions.  ˆ be Theorem 8.59. Let (S, E, m) be 6a measure space and let f : S −→ R a measurable function such that S f dm is defined. Consider the signed measure mf , that is, the indefinite integral of f relative to m. ˆ is integrable over S with respect A measurable function g : S −→ R to 6 mf if and6 only if gf is integrable over S relative to m, in which case S g dmf = S gf dm. 6 6 Proof. By Lemma 8.4 we have S |g| dm|f | = S |gf | dm. This equality shows that g is integrable over S with respect to m|f | if and only if gf is integrable over S with respect to m. The equality of the theorem follows now from Lemma 8.4.  Theorem 8.60. Let (S, E, m) be a measure space, where m is a σ-finite measure. If m1 , m2 are σ-finite measures on (S, E) such that m1 # m, m2 # m, m1 + m2 is defined, then m1 + m2 # m and d(m1 + m2 ) dm1 dm2 = + (a.e.). ddm ddm ddm Proof.

We have 7 d(m1 + m2 ) dm = (m1 + m2 )(T ) ddm T 7 7 dm1 dm2 dm + dm = T dm T dm   7 dm1 dm2 = + dm, dm dm T which implies the conclusion.



Theorem 8.61. Let (S, E, m) be a measure space, where m is a σ-finite measure. If m1 is a σ-finite measures on (S, E) such that m1 # m, then for every k ∈ R we have: dm1 dkm1 =k (a.e.). ddm ddm 6 dm1 6  1 Proof. We have (km1 )(T ) = k T dm dm = T k dm dm for all T ∈ E, dm dkm1 dm1  so ddm = k ddm a.e. Theorem 8.62. (Chain Rule for Radon-Nikodym Derivatives) Let m1 , m, m be σ-finite signed measures such that m1 # m and m # m. Then m1 # m and dm1 dm dm1 = a.e. with respect to m. dm dm dm

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Integration

b3234-main

page 533

533

Proof. If T ∈ E is such that m(T ) = 0, then m (T ) = 0, hence m1 (T ) = 0. Therefore, m1 # m. Theorem 8.59 implies that 7 7 dm1 dm1 dm  dm dm = m1 (T ) =   T dm T dm dm for every T ∈ E, which gives the conclusion of the theorem.



Theorem 8.63. (Lebesgue Decomposition Theorem) Let (S, E, m) be a measure space such that m is finite and m(S) > 0. If m ˜ is a finite measure defined on E then there exist two measures m0 and m1 on E such that m0 ⊥ m ˜ = m0 + m1 . and m1 is absolutely continuous with respect to m such that m Proof. Since both m and m ˜ are finite measures, so is the measure m = m + m. ˜ By Radon-Nikodym Theorem, there exist non-negative 6 measurable functions6 f, g such that for each U ∈ E, we have m(U ) = U f dm and ˜ are absolutely continuous relative m(U ˜ ) = U g dm because both m and m to m . Let V = {x ∈ S | f (x) > 0} and W = {x ∈ S | f (x) = 0}. Then S is the disjoint union of V and W , where m(W ) = 0. ˜ ∩ W ). We have 6m0 (V ) = 0, so m0 ⊥ m. Define m0 by m0 (U ) = m(U ˜ ∩ V ) = U∩V g dm. Let m1 be defined by m1 (U ) = m(U We have m ˜ = m0 + m1 and we need to show only6 that m1 # m. Let T be a set such that m(T ) = 0. Then 0 = m(T ) = T f dm and f = 0 a.e. relative to m . Since f > 0 on U ∩ V we have m (U ∩ V ) = 0. Thus, ˜ V = 0, which concludes the argument.  m(U ˜ ∩ V ) = 0, so m1 (U ) = U ∩ 8.7

Integration on Products of Measure Spaces

Let (S, E) and (T, E ) be two measurable spaces. For a set E ⊆ S × T , s ∈ S, and t ∈ T , subset Es of T and the subset E t of S are defined as: Es = {t ∈ T | (s, t) ∈ E}, E t = {s ∈ S | (s, t) ∈ E}. The sets Es and E t are referred to as the s-section and the t-section of E, respectively. Let E = U × V be a rectangle in S × T . We have   V if s ∈ U, U if t ∈ V, and (U × V )t = (U × V )s = ∅ if s ∈ U ∅ if t ∈ V for s ∈ S and t ∈ T , respectively.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 534

Mathematical Analysis for Machine Learning and Data Mining

534

Is immediate that (E)s = Es and     Ei = (Ei )s , i∈I

s

i∈I

for any collection {Ei | i ∈ I} of subsets of S × T . If f : S × T −→ Z, then the section fs is the function fs : T −→ Z that depends on the argument t and is given by fs (t) = f (s, t); similarly, the section f t is the function f t that depends on s and is given by f t (s) = f (s, t) for (s, t) ∈ S × T . For every D ⊆ S × T we have the equalities (fs )−1 (D) = (f −1 (D))s , t −1

(f )

(D) = (f

−1

t

(D)) .

(8.17) (8.18)

Theorem 8.64. Let (S, E) and (T, E ) be two measurable spaces and let (S × T, E × E ) be their product. If E ∈ E × E , then Es ∈ E and E t ∈ E for s ∈ S and t ∈ T . ˆ is E × E -measurable, then each section fs is E If f : S × T −→ R measurable and each f t is E-measurable. Proof.

Let F be the collection of subsets of S × T defined by F = {E ⊆ S × T | for all s ∈ S, Es ∈ E }.

F contains all rectangles of the form U × V such that U ∈ E and V ∈ E . In particular, S × T ∈ F. Since F is closed under complementation and countable unions, it follows that F is a σ-algebra. Therefore, E × E ⊆ F and Es ∈ E , whenever E ∈ E × E . Similarly, E t ∈ E whenever E ∈ E × E . This proves the first part of the theorem. The second part of the theorem is an immediate consequence of equalities (8.17) and (8.18).  If (S, E, m) and (T, E , m ) are two σ-finite measure spaces then m ((U × V )s ) = m (V )1U (s),

(8.19)

m((U × V ) ) = m(U )1V (t)

(8.20)

t

for s ∈ S and t ∈ T . Theorem 8.65. Let (S, E, m) and (T, E , m ) be two σ-finite measure spaces ˆ and gE : T −→ R ˆ defined and let E ∈ E × E . The functions fE : S −→ R  t by fE (s) = m (Es ) and gE (t) = m(E ) are Borel-measurable.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Integration

b3234-main

page 535

535

Proof. Suppose initially that the measure m is finite. We claim that the collection F = {E ∈ E × E | fE is measurable} is a Dynkin system. The definitions of the functions fE and gE mean that equalities (8.19) and (8.20) can be written as fU×V (s) = m (V )1U (s), gU×V (t) = m(U )1V (t), 

for U ∈ E, V ∈ E , s ∈ S and t ∈ T . Therefore, U × V ∈ F and, in particular, S × T ∈ F. If P, Q ⊆ S × T and P ⊆ Q, we have m ((Q − P )s ) = m (Qs ) − m (Ps ), or equivalently, fQ−P = fQ − fP . Therefore, under these assumptions, P, Q ∈ F and P ⊆ Q imply Q − P ∈ F. Also, if (En ) is an increasing sequence of subsets in E × E and E =  n∈N En , then fE (s) = m (Es ) = lim fEn (s). n→∞

Thus, F is indeed a Dynkin system. Let now U1 , U2 ∈ E and V1 , V2 ∈ E . Since (U1 × V1 ) ∩ (U2 × V2 ) = (U1 ∩ U2 ) × (V1 ∩ V2 ), it follows that F is also a π-system and, therefore, F is a σ-algebra by Theorem 1.43. Since the set of rectangles U × V with U ∈ E and V ∈ E is included in F, it follows that F = E × E , so fE is measurable for every E ∈ E × E . Suppose now that m is σ-finite. Let (Dn ) be a sequence of pairwise  disjoint sets in E such that n∈N Dn = T and m (Dn ) is finite for n ∈ N. Define a finite measure mn (B) = m(B ∩ Dn ) for B ∈ E . By the first part of the argument, the function fE,n defined by fE,n (s) = mn (Es ∩ Dn ) is E measurable. Since fE (s) = n∈N fE,n (s), it follows that fE is measurable.  The treatment for gE is similar. Suppose that (S, E, m) and (T, E , m ) are two measure spaces and h : ˆ Then, for each s ∈ S there exists a function h(s, ·) : T −→ R, ˆ S × T −→ R.  such h(s, ·)(t) = h(s, t). If the function h(s, ·) is m -integrable, its m 6 6 integral is denoted by T h(s, ·) dm or by T h(s, t) dm (t). Similarly, for ˆ If h(·, t) is m-integrable, each t ∈ T there exists a function h(·, t) : S −→6 R. 6 its m-integral is denoted by S h(·, t) dm or by S h(s, t) dm(s). The symbol

May 2, 2018 11:28

536

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 536

Mathematical Analysis for Machine Learning and Data Mining

6 6 s in S h(s, t) dm(s) or the symbol t in T h(s, t) dm (t) designates the component that is involved in the integration process. Theorem 8.66. Let (S, E, m) and (T, E , m ) be two σ-finite measure spaces. There is a unique measure m × m on the σ-algebra E × E such that (m × m )(A × B) = m(A)m (B) for A ∈ E and B ∈ E . If E ∈ E × E  we have 7 7  m(Es ) dm(s) = m(E t ) dm (t). (m × m )(E) = S

T

ˆ and Proof. Theorem 8.65 implies that the functions fE : S −→ R  t ˆ gE : T −→ R defined by fE (s) = m (Es ) and gE (t) = m(E ) are Borelmeasurable. This allows us to define the functions (m × m )1 and (m × m )2 on E × E as 7  m(E t ) dm (t), (m × m )1 (E) = 7T  (m × m )2 (E) = m (Es ) dm(s). S

It is clear that (m × m )1 (∅) = (m × m )2 (∅) = 0. Let (En ) be a sequence of pairwise disjoint sets in E × E and let E = t ∈ T , (Ent ) is a sequence of pairwise disjoint sets in E such n∈N En . For  t  t that E = En and m(E t ) = n∈N m(Ent ). By Beppo Levi’s Theorem (Theorem 8.39), we have: 7 m(E t ) dm (t), (m × m )1 (E) = T 7 = m (Ent ) dm (t),



n∈N

=



T

(m × m )1 (En ),

n∈N 

which means that (m × m )1 is σ-additive. Similarly, (m × m )2 is σ-additive. Furthermore, (m × m)1 (U × V ) = m(U )m (V ) = (m × m )2 (U × V ). for U ∈ E and V ∈ E . Thus, (m × m )1 and (m × m )2 are measures on E × E . The uniqueness of m×m follows from Corollary 7.8. Thus, (m×m )1 = (m × m )2 , which yields the equality of the theorem. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 537

537

The assumption of σ-finiteness for m and m is essential. The next two theorems establish sufficient conditions for the existence of 6 integrals of the form S×T f d(m × m ) as iterated integrals of the sections of the function f . The first result, known as Tonelli’s Theorem shows that this is always possible for non-negative functions (and possibly obtain ∞ as a value). The second result known as Fubini’s Theorem applies to more general real-valued functions but requires the iterated integrals to be finite. Theorem 8.67. (Tonelli’s1 Theorem) Let (S, E, m) and (T, E , m ) be ˆ 0 be an (S × T )two σ-finite measure spaces and let f : S × T −→ R measurable function. 6  ˆ 0 defined by φ(s) = (t) is E The function φ : S −→ R T fs dm 6 t ˆ 0 defined as ψ(t) = f dm(s) measurable and the function ψ : T −→ R S

is E-measurable. Furthermore, we have:   7 7 7 7 7 f d(m×m ) = f t dm(s) dm (t) = fs dm (t) dm(s) S×T

T

S

S

T



Proof. Let E ∈ E × E . The sections (1E )s and (1E )t of its characteristic function 1E are equal to the characteristic functions 1Es and 1E t , respectively. Thus, for every s ∈ S and t ∈ T we have: 7 7 1Es dm (t) = m (Es ) and 1E t dm(s) = m(E t ). T

S

Furthermore, 7 7 S

6 6

T

  7 7 1Es dm (t) dm(s) = 1E t dm(s) dm (t) S 7T = 1E d(m × m ) S×T

6 because 1 dm(s) = m (Es ) dm(s) = (m × m )(E) = E s dm (t) S T S 6 1 d(m × m ) and S×T E  7 7 7  1E t dm(s) dm (t) = m(E t ) dm(t) = (m × m )(E) T S T 7 = 1E d(m × m ). 

S×T 1 Leonida

Tonelli was born on April 19th 1885 in Gallipoli, Apulia, Italy and died on March 12th 1946 in Pisa. Tonelli studied at the University of Bologna with Cesare Arzel` a and Salvatore Pincherle, where he obtained his doctorate in 1907. He taught at the Universities of Cagliari, Parma, Bologna and, after 1930 ar the University of Pisa. Tonelli has been a member of Accademia Nazionale dei Lincei. His contribution are in the calculus of variations, integration theory, and is considered one of the founders of the modern theory of functions of real variables.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 538

Mathematical Analysis for Machine Learning and Data Mining

538

This shows that the theorem holds for characteristic functions, and, therefore, for simple measurable functions on E × E . This, in turn, imply that the theorem holds for arbitrary non-negative measurable functions.  If positivity and measurability of f is replaced by m × m -integrability we obtain a variant of the previous result. Theorem 8.68. (Fubini’s2 Theorem) Let (S, E, m) and (T, E , m ) be ˆ be an (S×T )-measurable two σ-finite measure spaces and let f : S×T −→ R  function that is m × m -integrable. The section fs is m -integrable a.e. for s ∈ S and f t is m-integrable a.e. for t ∈ T . Furthermore, the functions If : S −→ R and Jf : T −→ R defined by: ⎧7 ⎨ fs dm if fs is m -integrable, If (s) = T ⎩ 0 otherwise and

⎧7 ⎨ f t dm(s) Jf (t) = S ⎩ 0

if f t is m-integrable, otherwise

belong to L1 (S, E, m) and L1 (T, E , m ) respectively and we have: 7 7 7 f d(m × m ) = If dm = Jf dm . S×T

S

T

Proof. The section fs and its positive and its negative part (f + )s , (f − )s ˆ and ψ : of f , are measurable and, therefore, φ : S6 −→ R 6 + the functions  − ˆ S −→ R defined by φ(s) = (f )s dm and ψ(s) = (f )s dm are E-measurable and m-integrable. Thus, by Theorem 8.17, they are finite almost everywhere in the sense of the measure m. Let Z = {s ∈ S | φ(s) = ∞ or ψ(s) = ∞}. 2 Guido

Fubini was born on January 19th 1879 in Venice, Italy. In 1896 Fubini began his studies at Scuola Normale Superiore di Pisa where he studies with Ulisse Dini and Luigi Bianchi. He defended his doctoral thesis in 1900 and began teaching at the University of Catania in Sicily, moved to the University of Genoa, and in 1908 he moved to the Politecnico in Turin, and then the University of Turin. After Mussolini adopted racial policies in 1939, Fubini, who was Jewish, accepted an invitation by Princeton University to teach there; he died in New York City on June 6th 1943 in New York. Fubini’s contributions are in functional analysis, complex analysis, and the calculus of variations.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 539

539

We have Z ∈ E and

7 ⎧7 ⎨ (f + )s dm − (f − )s dm If (s) = T T ⎩ 0

if s ∈ Z, if s ∈ Z.

Therefore, If is integrable. By Tonneli’s Theorem (Theorem 8.67) we have: 7 7 7 f d(m × m ) = f + d(m × m ) − f − d(m × m )   7 7 7 7 +  +  = (f )s dm dm(s) − (f )s dm dm(s) 7 = If dm. 

Similar arguments work for f t and Jf .

Lebesgue measurable functions need to be defined a.e. Two functions which are equal a.e. will be identified. Example 8.9. Note that 7 t e−ax sin x dx = 0

Since

7 t 7 0

0

∞

|e

−ax

1  1 − e−at (a sin t + cos t) . 2 1+a 

sin x| da

7 dx = 0

t

| sin x|

1 dx  t < ∞. x

Fubini’s Theorem can be applied for x ∈ (0, t) and a ∈ (0, ∞) and this yields: 7 ∞  7 t 7 t sin x dx = sin x e−ax da dx x 0 0 0  7 ∞ 7 t −ax = e sin x dx da 0 0 7 ∞ 7 ∞ −at 1 e = da − (a sin t + cos t) da 2 1 + a 1 + a2 0 0 7 ∞ −at π e = − (a sin t + cos t) da. 2 1 + a2 0 If we apply the change of variable a = st the last integral becomes 7 ∞ 7 ∞ −st2 e s sin t + t cos t (a sin t + cos t) da = e−s ds = 0, 2 1+s s 2 + t2 0 0 6∞ hence 0 sinx x dx = π2 .

May 2, 2018 11:28

540

8.8

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 540

Mathematical Analysis for Machine Learning and Data Mining

The Riesz-Markov-Kakutani Theorem

In this section we present a Riesz’s representation result for a class of linear functionals defined on a certain linear space of continuous functions. This theorem is known also as the Riesz-Markov3-Kakutani4 theorem or the RMK theorem, and is named after the discover of the result and the mathematicians who gave important extensions of this result. We follow the outline of the proof given in [114]. The proof contains a series of lemmas that use consistent notations introduced gradually. Recall that for a topological space (S, O) we defined Fc (S, O) as the set of continuous functions f : S −→ R such that supp(f ) is a compact set. Also, for an open set V and a compact set C we defined the families of functions: Fc,V (S, O) = {f ∈ Fc (S, O) | f (x) ∈ [0, 1] for x ∈ S and supp(f ) ⊆ V }, FC,c (S, O) = {f ∈ Fc (S, O) | f (x) ∈ [0, 1] for x ∈ S and f (x) = 1 for x ∈ C}. Definition 8.16. A positive linear functional is a linear functional defined on Fc (S, O) such that f  0 implies (f ) > 0. Lemma 8.5. Let (S, O) be a locally compact topological space and let be a positive linear functional on the linear space Fc (S, O). If E is a σ-algebra E on S such that B(S) ⊆ E, then there exists at most one measure m on E such that: (i) m(C) is finite for 6 every compact subset C of S; (ii) we have (f ) = S f dm for f ∈ Fc (S, O); (iii) for every E ∈ E we have 3 A. A. Markov was born on June 14th , 1856 and died on July 20th was a Russian mathematician best known for his work on stochastic processes. A primary subject of his research later became known as Markov chains and Markov processes. Markov studied at Petersburg University, where among his professors was Chebyshev. Markov his got his doctorate in 1885 and became a professor in 1894. He extended Riesz’ Theoremto the case of bounded positive real functional. 4 Shizuo Kakutani was born on August 28, 1911 in Osaka and died on August 17, 2004 in New Haven, Connecticut. Kakutani was a Japanese-American mathematician, best known for his contributions to functional analysis. Kakutani attended Tohoku University in Sendai and spent two years at the Institute for Advanced Study in Princeton. He received his Ph.D. in 1941 from Osaka University. After the war he returned to the Institute for Advanced Study in 1948, and was appointed a professor at Yale in 1949. He extended Riesz’ theorem to locally compact spaces.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 541

541

m(E) = inf{m(V ) | E ⊆ V, V ∈ O}; (iv) we have m(E) = sup{m(C) | C ⊆ E, C is compact}, where E is an open set, or a member of E with m(E) < ∞. Proof. Let m1 and m2 be two measures such that m1 (C) = m2 (C) for all compact subsets of S that belong to E. Let C be a compact subset such that C ∈ E and let > 0. Since m2 (C) = inf{m2 (V ) | C ⊆ V, V ∈ O} and m2 (C) is finite, it follows that there exists an open set V such that m2 (V ) < m2 (C) + . By Uryson’s Lemma for locally compact spaces (Theorem 4.110) there exists a continuous function f ∈ Fc,V (S, calo) ∩ FC,c (S, O). Therefore, 7 7 m1 (C) = 1C dm1  f dm1 = (f ) S S 7 7 = f dm2  1V dm2 = m2 (V ) < m2 (C) + , S

S

hence m1 (C)  m2 (C). By swapping m1 and m2 we obtain the reverse inequality, hence m1 and m2 are equal on all compact subsets of E. By  (iv) we have m1 (E) = m2 (E) for every E ∈ E, so m1 = m2 . Let (S, O) be a locally compact topological space and let be a positive linear functional on Fc (S, O). For V ∈ O define m (V ) as (8.21) m (V ) = sup{ (f ) | f ∈ Fc,V (S, O)}. If V1 ⊆ V2 we have m (V1 )  m (V2 ). The definition of m is extended to P(S) as (8.22) m (E) = inf{m (V ) | E ⊆ V, V ∈ O} for every subset E of S. ˜ be the set of all subsets E of S such that m (E) is finite and Let E m (E) = sup{m (C) | C ⊆ E and C is compact}. Define E as ˜ for every compact subset C}. E = {E | E ⊆ S, E ∩ C ∈ E (8.23) Lemma 8.6. The function m defined by equality (8.22) is monotonic and ˜ m (E) = 0 implies E ∈ E. Suppose that E1 ⊆ E2 . Then, {V ∈ O | E2 ⊆ V } ⊆ {V ∈ O | E1 ⊆ V }, which implies inf{m (V ) ∈ O | E1 ⊆ V }  {m (V ) ∈ O | E2 ⊆ V }, which shows that m is monotonic. ˜ and E ∈ E. If m (E) = 0 it is immediate that E ∈ E

Proof.



May 2, 2018 11:28

542

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 542

Mathematical Analysis for Machine Learning and Data Mining

The positive linear functional is monotonic because for f, g ∈ Fc (S, O) such that f  g we have g − f  0, hence (g) = (f ) + (g − f )  (f ). Note that m was defined for all subsets of S. This allows us to formulate the following preliminary result: Lemma 8.7. The function m defined by equality (8.22) is subadditive ∞ on P(S), that is, if E1 , . . . , En , . . . are subsets of S, then m ( i=1 Ei )  ∞ i=1 m (Ei ). Proof. We prove by induction on n  2 that if V1 , . . . , Vn are open sets, n and g is a function such that 0  g(x)  1 and supp(g) ⊆ i=1 Vi , then n

(g)  i=1 m (Vi ). For the base case, n = 2, let V1 , V2 ∈ O and let g be a function such that 0  g(x)  1 and supp(g) ⊆ V1 ∪V2 . An application of the Partition of Unity Theorem (Theorem 4.111) to the compact set supp(g) implies the existence of two continuous functions h1 , h2 in Fc (S, O) such that h1 (x), h2 (x) ∈ [0, 1], supp(h1 ) ⊆ V1 , supp(h2 ) ⊆ V2 , and h1 (x) + h2 (x) = 1 for x ∈ supp(g). Therefore, g = h1 · g + h2 · g, hence supp(h1 · g) ⊆ V1 , supp(h2 · g) ⊆ V2 and, therefore, (g) = (h1 · g) + (h2 · g)  m (V1 ) + m (V2 ). Since this inequality holds for every g such that g(x) ∈ [0, 1] and supp(g) ⊆ V1 ∪ V2 it follows that m (V1 ∪ V2 )  m (V1 ) + m (V2 ). If there exists Ei such that m (Ei ) = ∞, the inequality of the lemma obviously holds. Suppose therefore that m (E1 ), . . . , m (En ) are finite. Since m (E) = inf{m (V ) | E ⊆ V and V ∈ O}, there exist open sets V1 , . . . , Vn such that m (Vi ) < m (Ei ) + 2 i for 1  i  n. Let V be the open set ∞ V = n=1 Vi and let f be a function such that f (x) ∈ [0, 1] for x ∈ S and supp(f ) ⊆ V . Since supp(f ) is a compact set, there exists n such that supp(f ) ⊆ V1 ∪ · · · ∪ Vn . By the induction hypothesis, ∞  m (Vi ) + .

(f )  m (V1 ∪ · · · ∪ Vn )  m (V1 ) + · · · + m (Vn )  i=1

Since this holds for every f such that f (x) ∈ [0, 1] and supp(f ) ⊆ V and ∞ n=1 Ei ⊆ V , it follows that ∞  ∞   m Ei  m (V )  m (Ei ) + , i=1

i=1

which completes the argument.



Lemma 8.8. Let C be a compact subset of a locally compact topological ˜ and space (S, O). We have C ∈ E m (C) = inf{ (f ) | f ∈ FC,c (S, O)}.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 543

543

Proof. Let f ∈ FC,c (S, O)} and let a ∈ (0, 1). The set Va = {x | f (x) > a} = f −1 (a, ∞) is open and C ⊆ Va because f (x) = 1 for x ∈ C. If g ∈ Fc,Va , then ag  f , hence  1 m (C)  m (Va ) = { (g) | g ∈ Fc,Va (S, O)}  (f ). a ˜ When a → 1 we have m (C)  (f ), hence m (C) is finite. Thus, C ∈ E. For > 0 there exists an open set V such that C ⊆ V and m (V ) < m (C) + . By Uryson’s Lemma for locally compact spaces, there exists f in FC,c (S, O) ∩ Fc,V (S, O). Thus, (f )  m (V ) < m (C) + . This gives the equality of the lemma.  Let f ∈ Fc (S, O). Since supp(f ) is a compact set, by the last lemma we have: m (supp(f )) = inf{ (f ) | f ∈ Fsupp(f ),c (S, O).

(8.24)

Lemma 8.9. Let (S, O) be a locally compact topological space, be a positive linear functional on Fc (S, O). For every open set V we have: m (V ) = sup{m (C) | C ⊆ V and Cis compact}. Proof. The extended definition of m in equality (8.21), for a compact set C we have m (C) = inf{m (V ) | E ⊆ V, V ∈ O}. Thus, for every compact set C included in an open set V we have m (C)  m (V ). This entails the inequality sup{m (C) | C ⊆ V andCis compact}  m (V ), hence m (V ) is an upper bound of the set {m (C) | C ⊆ V and C is compact}. To prove the reverse inequality, we need to show that there exists a compact set C included in V such that m (V )  m (C). Let a be a number such that a < m (V ) = sup{ (f ) | f ∈ Fc,V (S, O)}. To prove that m (V ) = sup{m (C) | C ⊆ V and C is compact}, by Supplement 16 of Chapter 1 it suffices to show that a  sup{m (C) | C ⊆ V and Cis compact}. By the definition of m (V ) there exists f ∈ Fc,V (S, O) such that a
0 there exists g ∈ FC1 ∪C2 ,c (S, O) such that (g) < m (C1 ∪ C2 ) + . Note that f g ∈ FC1 ,c (S, O) and (1 − f )g ∈ FC2 ,c (S, O). Since is linear we have m (C1 ) + m (C2 )  (f g) + (g − f g) = (g) < m (C1 ∪ C2 ) + . By Lemma 8.7, we have m (C1 ∪ C2 ) = m (C1 ) + m (C2 ). If m (E) is infinite, the equality follows from Lemma 8.7. Suppose now that m (E) is finite and let be a positive number. Since ˜ there are compact subsets Di of Ei with m (Di ) > m (Ei ) − i . If Ei ∈ E 2 Ci = D1 ∪ · · · ∪ Di we have n n   m (Di ) > m (Ei ) − . m (E)  m (Cn ) = i=1 i=1  Since this holds for every n ∈ N and > 0, m (E)  ∞ n=1 m (En ) and the equality follows from Lemma 8.7. If m (E) is finite and > 0, there exists N such that m (E) < N ˜  i=1 m (Ei ) + , so m (E)  m (CN ) + 2 , hence E ∈ E.

˜ and > 0. There exist a compact set C and an Lemma 8.11. Let E ∈ E open set V such that C ⊆ E ⊆ V and m (V − C) < . ˜ m (E) = sup{m (C) | C is compact and C ⊆ E} Proof. Since E ∈ E, and m (E) is finite. Therefore, there exists a compact set C such that m (E) < m (C) + 2 . Since we have m (E) = inf{m (V ) | V ∈ O and E ⊆ V }, there exists an open set V such that m (V ) − 2 < m (E). ˜ by Lemma 8.9. Therefore, Since V −C is an open set, we have V −C ∈ E m (C) + m (V − C) = m (V ) < m (C) + , by Lemma 8.8.



May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 545

545

˜ then A − B, A ∪ B, and A ∩ B belong to E. ˜ Lemma 8.12. If A, B ∈ E, Proof. By Lemma 8.11, there exists compact sets C1 , C2 and open sets V1 , V2 such that C1 ⊆ A ⊆ V1 , m (V1 − C1 ) < , and C2 ⊆ B ⊆ V2 , m (V2 − C2 ) < . Since A − B ⊆ V1 − C2 ⊆ (V1 − C1 ) ∪ (C1 − V2 ) ∪ (V2 − C2 ), we have, by the subadditivity of m shown in Lemma 8.7: m (A − B)  + m (C1 − V2 ) + . ˜ by Lemma 8.9. Since C1 −V2 is a compact subset of A−B we have A−B ∈ E ˜ ˜ Since A ∪ B = (A − B) ∪ B, A ∪ B ∈ E by Lemma 8.10. A ∩ B ∈ E because A ∩ B = A − (A − B).  Lemma 8.13. Let (S, O) be a topological space. The collection E defined by equality (8.23) is a σ-algebra such that B(S, O) ⊆ E. ˜ for every compact subset C of S. Proof. Let E ∈ E. Then E ∩ C ∈ E This implies (S − E) ∩ C = C − (E ∩ C), ˜ because C ∈ E ˜ by Lemma 8.8. hence S − E is the difference of two sets in E ˜ Therefore, (S − E) ∩ C ∈ E, hence E ∈ E implies E = S − E ∈ E.  Suppose that (Ai )i1 is a sequence of sets in E and let A = i1 Ai . Define the sequence of sets (Bi )i1 as B1 = A1 ∩ C, Bn = (An ∩ c) − (B1 ∪ · · · ∪ Bn−1 ) for n  2. ˜ by Lemma 8.12. Also, The sets in the sequence (Bi )i1 are disjoint sets in E  ˜ A ∩ C = n1 Bn . Therefore, A ∩ C ∈ E by Lemma 8.10, hence A ∈ E. ˜ so H ∈ E. If H is closed, then H ∩ C is compact, hence H ∩ C ∈ E, Therefore S ∈ E. Since E is a σ-algebra in S that contains all closed subsets of S, it follows that E contains B(S, O).  ˜ equals the collection of sets E in E such Lemma 8.14. The collection E that m (E) is finite. ˜ By Lemma 8.8, for each compact set C we have Proof. Let E ∈ E. ˜ ˜ C ∈ E, so E ∩ C ∈ E by Lemma 8.12. Therefore, E ∈ E.

May 2, 2018 11:28

546

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 546

Mathematical Analysis for Machine Learning and Data Mining

Suppose now that E ∈ E and m (E) is finite. There exists an open set V such that C ⊆ V and m (V ) is finite. By Lemmas 8.9 and 8.11, there exists a compact set C such that C ⊆ V and m (V − C) < . Since ˜ for > 0 there is a compact set C1 such that C1 ⊆ E ∩ C with E ∩ C ∈ E, m (E ∩ C) < m (C1 ) + . Since E ⊆ (E ∩ C) ∪ (V − K), it follows that ˜  mell (E)  mell (E ∩ C) + mell (V − C) < mell (C1 ) + 2 , hence E ∈ E. Theorem 8.69. (The Riesz-Markov-Kakutani Theorem) Let (S, O) be a locally compact space, and let be a positive linear functional on Fc (S, O). There exists a measurable space (S, E) such that B(S, O) ⊆ E and a unique positive measure m on E such that 6 (i) (f ) = S f dm ; (ii) m (C) is finite for every compact subset of S; (iii) for every E ∈ E we have m (E) = inf{m (V ) | E ⊆ V and V ∈ O}; (iv) for every E ∈ E such that m (E) is finite, and for every E that is open we have m (E) = sup{m (C) | C ⊆ E and C is compact }; (v) the measure space (S, E, m ) is complete. Proof. Note that m is a measure on E by Lemmas 8.10 and 8.14. To prove (i) it suffices to prove that for real-valued functions we have (f )  6 f dm . Indeed, if this is the case, we also have S 7 − (f ) = (−f ) 

7 (−f ) dm = −

S

f dm , S

6 6 f dm . Therefore, if

(f )  f dm holds, then which implies

(f )   S S 6

(f ) = S f dm . Let f ∈ Fc (S, O) and let [a, b] be an interval such that supp(f ) ⊆ [a, b]. Choose y0 , y1 , . . . , yn such that y − 0 = a, y0 < y1 < · · · < yn−1 < yn , and yn = b, and let Ei = {x | yi−1 < f (x)  yi } ∩ supp(f ) for 1  i  n − 1. Since f is continuous, f is Borel measurable, so the sets E1 , . . . , En are disjoint Borel sets that partition supp(f ). Let Vi be open sets such that Ei ⊆ Vi and m (Vi ) < m (Ei ) + n and f (x) < yi + for x ∈ Vi and for 1  i  n. By the Partition of Unity Theorem, there are hi ∈ Fc,Vi (S, O) such   that hi = 1, hence f = i hi f . By Lemma 8.8 we have m (supp(f )) <  

( i hi ) = i (hi ).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 547

547

Since hi f  (yi + )hi and yi − < f (x) when x ∈ Ei , we have

(f ) = =

n  i=1 n 

(hi f ) 

n 

(yi + ) (hi )

i=1

(|a| + y + i + ) (hi ) − |a|

i=1 n 

n 

(hi )

i=1

# $ − |a|m (supp(f )) (|a| + yi + ) m (Ei ) + n i=1  n  n   = (yi − )m (Ei ) + 2 m (supp(f )) + (|a| + yi + n i=1 i=1 7  f dm + (2m (supp(f )) + |a| + b + ), 

S

which implies the desired inequality. ˜ It is immediate that m is monotone and that m (E) = 0 implies E ∈ E and, therefore E ∈ E. This shows that condition (v) is satisfied. By Lemma 8.8, part (ii) holds. Part (iii) follows from the definition of m for open set given in equality (8.21) and its extension to P(S) given in equality (8.22). The argument for part (iv) is given by Lemma 8.10. Lemma 8.5 establishes the uniqueness of a measure for which conditions (i)-(v) are satisfied.  8.9

Integration Relative to Signed Measures and Complex Measures

Definition 8.17. Let (S, 6 E.m) be a measure space, where m is a signed ¯ measure. The 6integral S f dm 6 of a −measurable function f : S −→ R is + defined if both S f dm and S f dm are defined and at least one of then is neither ∞ or −∞. If this is the case, 7 7 7 f dm = f dm+ − f dm− . S

The function f is integrable if

6

S

S

f dm is finite. S

Theorem 8.70. Let (S, E.m) be a measure space, where m is a signed measure. The following statements concerning a measurable function f : ˆ are equivalent: S −→ R (i) f is integrable with respect to m; (ii) f is integrable with respect to both m+ and m− ; (iii) f is integrable with respect to |m|.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 548

Mathematical Analysis for Machine Learning and Data Mining

548

6 6 6 Proof. Note that S f dm is finite if and only if S f dm+ and S f dm− are finite, so (i) is equivalent to (ii). 6 statement is equivalent to the finiteness of S |f | dm+ and 6 The second 6 −  S |f | dm , or with S |f | d|m| < ∞, which is equivalent to (iii). Theorem 8.71. Let m1 , m2 be two measures on the measurable space (S, E) non-negative measurable function f : S −→ such that m1 6 m2 . For every 6 ˆ 0 we have R f dm  f dm 1 2. S S Proof. Suppose that f is a non-negative simple measurable func6 n n −1 −1 y 1 . Then, f dm (yi )  tion f = i f 6 (yi ) 1 = i=1 i=1 yi m1 (f S n −1 y m (f (y ) = f dm . i 2 i=1 i 2 S If f is a non-negative measurable function and (fn ) is a sequence of measurable non-negative simple functions such that limn→∞ fn = f , the Monotone Convergence Theorem implies the desired inequality.  Theorem 8.72. Let (S, E, m) be a measure space and let f : S −→ C be a for x ∈ S. We have 6 6complex-valued function such 6that f (x) = u(x) + iv(x) |f | dm < ∞ if and only if |f | dm < ∞ and |f | dm < ∞. S S S " |f (x)| = u2 (x) 6+ v 2 (x)  |u(x)| + |v(x)|, so 6Proof. Observe that 6 S |f | dm 6< ∞ and S 6|f | dm < ∞ implies S |f | dm 6 6 < ∞. |f | dm  |u| dm and |f | dm  Since S S S |v| dm, it is clear that 6 6S 6 |f | dm < ∞ implies |f | dm < ∞ and |f | dm < ∞.  S S S Definition 8.18. Let (S, E, m) be a measure space and let6 f : S −→ C be a complex-valued function. The6 function f is integrable if S |f | 6dm < ∞. 6 If f = u + iv, the integral S f dm is the complex number S u dm + i S v dm. If (S, E, m) is a measure space, where m is a complex measure and f : S −→ C is a complex bounded function defined on S, the Jordan 6 decomposition of m6 can be used to define S f dm. Namely, if m = m1 − m1 + i(m2 − m2 ), S f dm is defined as  7 7 7 7 7     f dm = f dm1 − f dm1 + i f dm2 − f dm2 . S

S

S

S

S

Theorem 8.73. Let B(S, E, m) the set of complex-valued bounded and measurable functions on (S, E, m) and let f ∞ = ess sup{|f (x)| | x ∈ S}

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 549

549

for f ∈ B(S, E, m). We have:  7      f dm  f ∞ m.   Proof. Let f be a simple measurable function on S with f (S) = {a1 , . . . , an }. If Ai = f −1 (ai ) for 1  i  n, then 7    n n n           aj m(Aj )  |aj ||m(Aj )|  f ∞ |m(Aj )|,  f dm =   S    j=1

j=1

j=1

6

hence | f dm|  f ∞ m. Since every bounded function f is the limit of a sequence of simple functions that converges uniformly to f (see Corollary 7.14), the desired conclusion follows.  8.10

Indefinite Integral of a Function

ˆ Theorem 8.74. Let (S, E, m) be a measure space and f : S 6 −→ R be a 6 measurable function. If the integral S f dm exists, then S f 1U dm is defined for every U ∈ E. Proof.6 Observe that (f 1U )+ = f + 1U  f + and6 (f 1U )− = f6− 1U  f − . + − Since S f dm S f dm or S6 f dm is 6 of+the integrals 6 6 exists,+at least one finite. Since S (f 1U ) dm =6 S f dm and 6 S (f 1U )− dm = S f − dm, + − it follows that 6 at least one of S (f 1U ) dm, S (f 1U ) dm is finite, which means that S f 1U dm exists.  ˆ be a Definition 8.19. Let (S, E, m) be a measure space and f : S −→ R measurable function. The indefinite integral of f with respect to m is the ˆ defined by function φ : E −→ R 7 φ(U ) = f 1U dm S

for U ∈ E. If f  0 a.e. on S, then φ is a measure; we denote it by f m. ˆ be a Theorem 8.75. Let6 (S, E, m) be a measure space and f : S −→ R 6function such that S f dm is defined. If U ∈ E and m(U ) = 0, then f dm = 0. U

May 2, 2018 11:28

550

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 550

Mathematical Analysis for Machine Learning and Data Mining

Proof. This statement follows from the fact that f 1U = 0 almost everywhere on S.  ˆ be Theorem 8.76. Let 6(S, E, m) be a measure space and f : S −→ R a function such that S f dm is defined. If (U1 , U2 , . . .) is a sequence of  pairwise disjoint sets in E and U = n1 Un , then 7 7 f dm = f dm. U

n1

Un

 Proof. Since the sets U1 , U2 , . . . are pairwise disjoint, 1U 6= n1 1Un ,  This, in turn, implies U f dm = hence 6f 1U = n1 f 1Un .   n1 Un f dm. ˆ be Theorem 8.77. Let 6(S, E, m) be a measure space and f : S −→ R a function such that S f dm is defined. If (U1 ,6U2 , . . .) is an increasing 6  sequence of sets and U = n1 Un , then limn→∞ Un f dm = U f dm.  of sets6 and U = n1 Un such If6 (U1 , U2 , . . .) is a decreasing sequence 6 that U1 f dm is finite, then limn→∞ Un f dm = U f dm.  Proof. For an increasing sequence of sets (U1 , U2 , . . .) and U = n1 Un , the set U can be written as the union of a pairwise disjoint sequence, U = U1 ∪ (U2 − U1 ) ∪ (U3 − U2 ) ∪ · · · . By Theorem 8.76, 7 7 7 f dm = f dm + f dm U

U1

n≥2

7 =

f dm + lim U1

7

= lim

n→∞

n→∞

Un −Un−1 n 7 

m=2

f dm

Um −Um−1

f dm. Un

For a decreasing sequence of sets (U1 , U2 , . . .) we have U1 − U = ∞ sequence 6of sets. By n=1 (U1 − Un ), where (U1 − Un ) is an increasing 6 the first part of the theorem we have lim 6 6 n→∞ U1 −Un f dm = U1 −U f dm. 6 Since U1 −U f dm + U f dm = 6 U1 f dm, taking into6 account that 6 f dm is6 finite, it follows that 6U1 6 | U f dm| is6 finite and6 U1 −U f dm = f dm − f dm. Similarly, f dm = U1 f dm − Un f dm, hence U1 U1 −Un 6 U 6 6 limn→∞ (6 U1 f dm − U6n f dm) = 0 because | U1 f dm| is finite. This yields  limn→∞ Un fn dm = U f dm. ˆ be Theorem 8.78. Let (S, E, m) be 6a measure space and let f : S −→ R a measurable function such that S f dm is defined. Then, the function

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Integration

ˆ defined by mf (U ) = mf : E −→ R measurable space (S, E).

6 U

9in x 6in

b3234-main

page 551

551

f dm is a signed measure on the

Proof. Since (f6 1U )+ = f + 1U6  f + and (f 1U6)− = f − 1U  f − , either 6 + + − − S (f 1 6U ) dm  S f < ∞ or S (f 1U ) 6dm  S f < ∞, which means that S f 1U dm is defined6 and, therefore U f dm is defined. 6 We claim that either U f dm > −∞ for every U ∈ E, or U f dm < ∞ for every U ∈ E. 6 6 6 Suppose that 6 S f − dm < ∞. Then, S (f 1U )− dm  6S f − dm < ∞, 6 hence6 U f dm = S f 1U dm > −∞ for U ∈ E. Similarly, if S f + dm < ∞, then U f dm < ∞ for all U ∈ E. The conclusion then follows from the last part of Theorem 8.77.  Let (S, E, m) be a measure space and let f : S −→ R0 be a nonnegative function that is not necessarily integrable relative to m. Define 6 the measure mf on mf (U ) = U f dm. Note that mf (U ) = 0 implies m(U ) = 0; mf is finite if and only if f is integrable. The measure mf is said to have f as density with respect to m. Note that if mf = mg , then f = g almost everywhere. Theorem 8.79. Let (S, E, m) be a measure space, f : S −→ R0 be a non-negative function that is6 not necessarily integrable relative to m, and 6 let h : S −→ R0 . We have h dmf = hf dm. A function h : S −→ R, not necessarily non-negative, is integrable with with respect to m. In this case, respect to m6f if and only 6if hf is integrable 6 6 h dmf = hf dm and U h dmf = U hf dm. 6 for U ∈ E, then h dm Proof. If h = 1 U 6 6 6 f = mf (U ), 6so the equality h dmf = hf dm amounts to mf (U ) = 1U f dm = U f dm, which clearly holds. Thus, by linearity, the first equality of the theorem holds for simple non-negative functions, hence by passing to limit the equality holds for non-negative measurable functions. If h is not non-negative, the first equality applied to |h| implies that h is integrable with respect to mf if and only hf is integrable with respect to m. If h is integrable, the first equality follows by applying this argument  to h+ and h− . Substituting h by h1U gives the second equality. 8.11

Convergence in Measure

We have seen (Corollary 7.4) that if a sequence of measurable f = (fn )n1 converges pointwise to a function f , then f is measurable. The same holds when f converges to f a.e.

May 2, 2018 11:28

552

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 552

Mathematical Analysis for Machine Learning and Data Mining

Theorem 8.80. Let (S, E, m) be a complete measure space and let f = (f1 , f2 , . . .) be a sequence of measurable functions such that limn→∞ fn (x) = f (x) a.e. Then, f is a measurable function. Proof.

This statement follows immediately from Theorem 7.15.



Definition 8.20. Let (S, E, m) be a measure space and let f = (fn ) be a ˆ sequence of measurable functions that are finite a.e, where fn : S −→ R. ˆ The sequence f converges in measure to the measurable function f : S −→ R that is finite a.e. if for every > 0, limn→∞ m({x ∈ S | |fn (x) − f (x)|  }) = 0. Note that the definition of convergence in measure makes no use of a norm or a metric. Theorem 8.81. Let (S, E, m) be a measure space and let f = (fn ) be a ˆ sequence of measurable functions that are finite a.e, where fn : S −→ R. If f converges in measure to f and f = g a.e., then the sequence f also converges to g in measure. Proof.

For > 0 we have {x ∈ S | |fn (x) − f (x)| < } − {x ∈ S | |fn (x) − g(x)| < } ⊆ {x ∈ S | f (x) = g(x)},

hence m({x ∈ S | |fn (x) − f (x)| < }) − m({x ∈ S | |fn (x) − g(x)| < })  m({x ∈ S | f (x) = g(x)}) = 0. Therefore, limn→∞ m({x ∈ S | |fn (x) − f (x)| < })  limn→∞ m({x ∈ S | |fn (x) − g(x)| < }). By reversing the roles of f and g we obtain limn→∞ m({x ∈ S | |fn (x) − f (x)| < }) = limn→∞ m({x ∈ S | |fn (x) − g(x)| < }). Therefore, f = (fn )n1 converges in measure to f if and only if it converges in measure to g.  Theorem 8.82. Let (S, E, m) be a measure space and let f = (fn ) be a ˆ sequence of measurable functions that are finite a.e, where fn : S −→ R. The sequence f converges in measure to f if for every > 0, there exists n ∈ N such that n > n implies m({x ∈ S | |fn (x) − f (x)|  }) < . Proof. Suppose that f converges in measure to f , that is, limn→∞ m({x ∈ S | |fn (x) − f (x)|  }) = 0 for every > 0. For every n0  1 there exists n1 such that n  n1 implies m({x ∈ S | |fn (x) − f (x)|  })  n10 . Choose n0 such that n10 < . Then, m({x ∈ S | |fn (x) − f (x)|  })  . Conversely, suppose that for every > 0, there exists n ∈ N such that n > n implies m({x ∈ S | |fn (x) − f (x)|  }) < . Choose m such that

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 553

553

1 < m . Then n > n m1 implies m({x ∈ S | |fn (x) − f (x)|  }) < m({x ∈ 1 1 }) < m , hence limn→∞ m({x ∈ S | |fn (x) − f (x)|  S | |fn (x) − f (x)|  m }) = 0. 

Let (S, E, m) be a measure space and let f, g : S −→ R0 be two measurable functions. We have: {x ∈ S | f (x) + g(x)  2 } ⊆ {x ∈ S | f (x)  } ∪ {x ∈ S | g(x)  }, which implies m({x ∈ S | f (x) + g(x)  2 })  m({x ∈ S | f (x)  }) + m({x ∈ S | g(x)  }). Theorem 8.83. Let (S, E, m) be a measure space and let f = (fn ) be a sequence of measurable functions that converges in measure to both f and g. Then, f = g a.e. Proof.

Let > 0. Since

    m({x ∈ S | |f (x) − g(x)|  })  m x ∈ S |f (x) − fn (x)|   2      +m x ∈ S |fn (x) − g(x)|  ,  2 

and limn→∞ m({x ∈ S | |f (x) − fn (x)|  2 }) = limn→∞ m({x ∈ S | |f (x)−fn (x)|  2 }) = 0, it follows that m({x ∈ S | |f (x)−g(x)|  }) = 0. Thus, for n  1, the set Un = {x ∈ S | |f (x) − g(x)|  n1 } is a null set,  hence U = n1 Un is a null set. Thus, for x ∈ U we have |f (x)−g(x)| < n1 for all n  1, hence f = g a.e.  Definition 8.21. Let (S, E, m) be a measure space and let f = (fn ) be a ˆ sequence of measurable functions that are finite a.e., where fn : S −→ R. The sequence f is a Cauchy sequence in measure if every > 0 there exists n ∈ N, such that m, n  n implies m({x ∈ S | |fn (x) − fm (x)|}) < . Theorem 8.84. (Riesz’ Theorem) Let (S, E, m) be a measure space. If f = (fn ) is a Cauchy sequence in measure, then f has an almost everywhere uniformly convergent subsequence. Proof. Suppose that f = (fn ) is a Cauchy sequence in measure, which means that for every > 0 there exists n ∈ N, such that m, n  n implies m({x ∈ S | |fn (x) − fm (x)|}) < .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 554

Mathematical Analysis for Machine Learning and Data Mining

554

This generates a sequence of numbers n0 , n1 , . . . , nm , . . . defined as follows: • for = 1 there exists k0 such that k > k0 implies m(|fk − fk0 |  1) < 1; • for = 12 there exists k1 > k0 such that k > k1 implies m(|fk − fk1 |  2−1 ) < 2−1 ; • ···; • for = 21n there exists kn > kn−1 such that k > kn implies m(|fk − fk1 |  2−n ) < 2−n ; • ··· The last statement implies m(|fkn+1 − fkn |  2−n ) < 2−n .  −i Let Bn = ∞ i=n (|fki+1 − fki |  2 ). Note that m(Bn ) 

∞ 

m(|fki+1 − fki |  2−i )

i=n

< ∞

∞ 

2−i = 2−(n−1) .

i=n

If B = n=1 Bn , then m(B)  m(Bn ) < 2−(n−1) , hence m(B) = 0.  For every x ∈ S − B = ∞ n=1 (S − Bn ) there exists n0 such that x ∈ S − Bn0 . Therefore, if n  n0 , |fkn+1 − fkn | < 2−n . Then, for every n > m  n0 we have |fkn (x) − fkm (x)|  |fkn (x) − fkn−1 (x)| + · · · +  |fkm+1 (x) − fkm (x)| < 2−(n−1) + · · · + 2−m < 2−(m−1) . Thus, (fkn ) is a Cauchy sequence, hence there the limit limn→∞ fkn (x) in R. Let f : S −→ R be the measurable function defined as  limn→∞ fkn (x) if x ∈ S − B, f (x) = 0 if x ∈ B. For every > 0 there exists n0 such that 2−(n0 −1) < . Then, m(Bn0 ) < 2−(n0 −1) < . We claim that (fkn ) converges almost uniformly to f on S − Bn0 . ∞ If x ∈ S − Bn0 = n=n0 (|fkn+1 − fkn |  2−n we have |fkn+1 (x) − fkn (x)| < 2−n when n  n0 . Then, as above, n > m  n0 implies fkn (x) − fkm (x)| < 2−(m−1) . Note that x ∈ S−Bn0 ⊆ S−B, hence limn→∞ fkn (x) = f (x). If we take n → ∞ we obtain |f (x) − fkm (x)| < 2−(m−1) for every  m  n0 and x ∈ S − Bn0 , which yields the desired conclusion.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 555

555

Theorem 8.85. Let (S, E, m) be a measure space and let f = (f1 , . . . , fn , . . .) be a Cauchy sequence in measure. If f contains a subsequence (fnj ) that converges in measure to a function f , then f converges in measure to the same f . Proof. Define the sequence of non-negative functions h = (hn ) as hn = |fn − f | for n  1. Since (fn ) is a Cauchy sequence in measure, for every > 0 there is n ∈ N such that m, n  n implies m({x ∈ S | |fn (x) − fm (x)|}) < . Since |hn (x) − hm (x)| = ||fn − f | − |fm − f ||  |fn − fm | < , the sequence (hn ) is a again a Cauchy sequence and (hnj ) is a subsequence of (hn ). Since (hnj ) converges in measure to 0 there exists k ∈ N such that k > k m({x ∈ S | hnj  }) < for j  k . Let p  k. Since np  n we have np  k0 , hence m({x ∈ S | hn > 2 })  m({x ∈ S | |hn − hnp |  }) + m({x ∈ S | hnp  }) < 2 , hence hn converges to 0 in measure. Thus, fn converges in measure to f .  Theorem 8.86. (Completeness of Convergence in Measure) Let (S, E, m) be a measure space and let f = (f1 , . . . , fn , . . .) be a Cauchy sequence in measure. Then f converges in measure. Proof. Since f is a Cauchy sequence, there is a subsequence (fnj ) constructed as in the proof of Theorem 8.84 such that for gj = fnj for j  1 and Sj = {x ∈ S | |gj+1 (x) − gj (x)|  21j } we have m(Sj )  2−j . Let   ∞ 1 1 Sj . We have m(Hk )  ∞ Hk = ∞ j = 2k−1 . Let W = j=k j=k k=1 Hk = 2 ∞ ∞ S = lim sup S . Since W ⊆ H for every k, we have m(W ) = 0. j k k=1 j=k j Let x ∈ W . Then, x ∈ Hk for some k and, therefore, x ∈ Sj for j  k. Therefore, for k  j  m we have |gj (x) − gm (x)| 

m−1  i=1

|gi+1 (x) − gi (x)| 

m−1  i=1

1 1  j−1 . 2i 2

The sequence of real numbers (gj (x)) is a Cauchy sequence and, therefore, it is convergent. Let f : S −→ R be the function defined as  limj→∞ gj (x) if the limit exists, f (x) = 0 otherwise,

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 556

Mathematical Analysis for Machine Learning and Data Mining

556

then f is measurable and, since the limit exists for every x ∈ W , it follows that (gj ) converges pointwise to f almost everywhere. 1 for m  j. Therefore, If x ∈ Hj , then |gj (x) − gm (x)|  2j−1 |gj (x) − f (x)| = lim |gj (x) − gm (x)|  m→∞

which implies {x ∈ S | |gj (x) − f (x)| 

1 2j−1 }

1 2j−1

,

⊆ Hk . Therefore

1 1 })  m(Hk )  j−1 . 2j−1 2 Therefore, (gj ) converges in measure to f . By Theorem 8.85 if follows that  (fn ) converges in measure to f . m({x ∈ S | |gj (x) − f (x)| 

8.12

Lp and Lp Spaces

Let (S, E, m) be a measure space. Denote by Lp (S, E, m) the of all 6 set p ˆ measurable complex-valued function f : S −→ C such that S |f | dm < ∞. If (S, E, m) is understood from context we write just Lp instead of Lp (S, E, m). ˆ The same is used for the set of real-valued function f : S −→ R 6 notation p such that S |f | dm < ∞. ˆ we have Observe that by Theorem 8.22 for a function f : S −→ C p p f ∈ L if and only if |f | ∈ L1 . Theorem 8.87. For every measure space (S, E, m) the set of complexvalued functions Lp (S, E, m) is a linear space. Proof. Let f, g ∈ Lp (S, E, m). Both f and g are finite a.e. on S, hence f + g is defined a.e on S. If a, b are two complex numbers we have |a| + |b|  2 max{|a|, |b|}, hence (|a| + |b|)p  2p max{|a|p , |b|p }  2p (|a|p + |b|p ). Thus, |(f + g)(x)|p  2p (|f (x)|p + |g(x)|p ) a.e. on S and, therefore  7 7 7 |(f + g)(x)|p dm  2p |f (x)|p dm + |g(x)|p dm < ∞, S

S

S

hence f + g ∈ L (S, E, m). Also, we have 7 7 |af |p dm = |a|p |f |p dm < ∞, p

S

hence af ∈ L (S, E, m). p

S



May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Integration

b3234-main

page 557

557

For f ∈ Lp and p  1 let νp (f ) be the number 7  p1 p |f | dm . νp (f ) = S

It is immediate that νp (af ) = |a|νp (f ) for every a ∈ R. Theorem 8.88. (H¨ older’s Inequality for Functions) Let (S, E, m) be a measure space and let f ∈ Lp (S, E, m) and g ∈ Lq (S, E, m) be two functions, where p, q are two numbers such that 1p + 1q = 1 and p, q > 1. We have ν1 (f g)  νp (f )νq (g). Proof. Observe that if νp (f ) = 0, by6 Theorem 8.15, |f |p = 0 a.e., so the function f g is 0 a.e., which implies S f g dm = 0. The same holds if νq (g) = 0, so in either case the inequality of the theorem holds. Therefore, without loss of generality we may assume that νp (f ) = 0 and νq (g) = 0. Suppose initially that νp (f ) = νq (f ) = 1. By Lemma 2.1, if p, q ∈ R − {0, 1} are two numbers such that 1p + 1q = 1 and p > 1, then, for every a, b ∈ R0 , we have |f (x)g(x)|  By integrating we have

|g(x)|q |f (x)|p + . p q

7 |f g| dm 7 7 1 1  |f |p dm + |g|q dm p S q S 1 1 = + = 1. p q

ν1 (f g) =

S

If νp (f ) = 1 or νq (f ) = 1 we apply the inequality obtained above to the functions f˜ = νp1(f ) f and g˜ = νq1(g) g. Since ν1 (f˜g˜)  1, it follows that  ν1 (f g)  νp (f )νq (f ). p . Therefore, limp→1,p>1 q = ∞. The equality p1 + 1q = 1 implies q = p−1 We show that H¨older’s Inequality can be extended to the case when p = 1 and q = ∞ by adopting the special definitions

L∞ (S, E, m) = {g | g is bounded a.e} and ν∞ (g) = inf{a ∈ R | |g(x)|  a a.e}.

May 2, 2018 11:28

558

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 558

Mathematical Analysis for Machine Learning and Data Mining

For this special case let ν∞ (g) = a0  0 and note that f g  f a0 a.e. Therefore, ν1 (f g)  ν1 (f a0 ) = a0 ν1 (f ) = ν1 (f )ν∞ (g), which shows that H¨older’s Inequality holds for p = 1 and q = ∞. Theorem 8.89. For every f ∈ L∞ (S, E, m) we have ν∞ (f ) = ess sup |f |, and {x ∈ R | f > ν∞ (f )} is a null set. Proof.

This follows immediately from the definitions.



Corollary 8.11. (Cauchy-Schwarz Inequality) Let (S, E, m) be a measure space and let f, g ∈ L2 (S, E, m) be two functions. We have ν1 (f g)  ν2 (f )ν2 (g). Proof. q = 2.

The desired inequality follows from Theorem 8.88 by taking p = 

Theorem 8.90. (Minkowski’s Inequality for Functions) Let (S, E, m) be a measure space and let f, g ∈ Lp (S, E, m) be two functions, where p  1. We have νp (f + g)  νp (f ) + νp (q). Proof. hence

For p = 1, observe that |(f + g)(x)|  |f (x)| + |g(x)| for x ∈ S, 7

7

7

|f + g| dm  S

|f | dm + S

|g| dm, S

which gives the desired inequality for p = 1. Suppose now that 1 < p < ∞. Since |f + g|p  (|f | + |g|)p  2p max{|f |p + |g|p }  2p (|f |p + |g|p ), we get that |f + g|p ∈ L1 , or |f + g| ∈ Lp . Furthermore, we have 7 7 p p (νp (f + g)) = |f + g| dm = |f + g| · |f + g|p−1 dm S S 7 7  |f | · |f + g|p−1 dm + |g| · |f + g|p−1 dm S

S

 νp (f )νq (|f + g|p−1 ) + νp (g)νq (|f + g|p−1 )

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 559

559

by H¨ older inequality. This implies (νp (f + g))p  νp (f ) + νp (g). νq (|f + g|p−1 )

(8.25)

Taking into account that p = (p − 1)q, it follows that 7 νq (|f + g|

p−1

)=

 q |f + g|p−1 dm

S

7

|f + g| dm p

=

 1q

 1q

p

= (νp (f + g)) q .

S

Substituting this expression in equality (8.25) leads to p

νp (f + g)p− q  νp (f ) + νq (g). Since p − pq = 1, Minkowski’s inequality follows. For p = ∞ let k0 , k1 be two constants such that |f (x)|  k0 (a.e.) and |g(x)|  k1 (a.e.). Then, |f (x) + g(x)|  k0 + k1 (a.e.), so f + g ∈ L∞ . This implies ν∞ (f + g)  ν∞ (f ) + ν∞ (g) by an application of the definition of  ν∞ . The previous results imply that νp is a seminorm on Lp . However, νp is not a norm because νp (f ) = 0 only implies that f = 0 a.e., but not f = 0. Theorem 8.91. Let (S, E, m) be a measure space. The measurable simple functions in Lp (S, E, m), where 1  p  ∞, form a dense set in this space. Proof. Consider the case when 1  p < ∞. Let f ∈ Lp (S, E, m). By Theorem 7.26 there exist two non-decreasing sequences of non-negative simple measurable functions (fˆn ) and (f˘n ) such that f + = limn→∞ fˆn and f − = limn→∞ f˘n . The sequence (fn ) defined by fn = fˆn − f˘n consists of simple functions such that |fn |  |f |. Therefore, fn ∈ Lp (S, E, m). Since |fn (x) − f (x)|  |f (x)| and limn→∞ |fn (x) − f (x)| = 0 for every x ∈ S, by the Dominated Convergence Theorem (Theorem 8.37) applied to the functions |fn − f | we obtain limn→∞ fn − f p = 0, which concludes the proof when 1  p < ∞. For p = ∞, let f ∈ L∞ . Then, −f ∞  f (x)  f ∞ . If > 0 consider the numbers a0 < a1 < · · · < ak such that am+1 − am <  for 1  m  k − 1 and k−1 m=1 (am , am+1 ] = [−f ∞ , f ∞ ]. The function k−1 f = m=1 ai 1(am ,am+1 is a simple measurable function and f − f ∞  , which gives the desired argument. 

May 2, 2018 11:28

560

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 560

Mathematical Analysis for Machine Learning and Data Mining

A special type of simple functions is introduced next. Definition 8.22. A step function on the closed interval [a, b] is a function f : [a, b] −→ R such that: (i) there exist a0 , a1 , . . . , an ∈ [a, b] where a = a0 < a1 < · · · < an = b; (ii) f is constant on each interval (ai−1 , ai ). Each step function is a simple measurable function between the measure spaces ([a, b], B([a, b])) and (R, B(R)). Lemma 8.15. Let U be a Borel subset of [a, b]. For every > 0 there exists a step function f on the measure space ([a, b], B([a, b]), mL ) such that 1U − f p < . Proof. If U is a Borel subset of R then there exists a collection of open   intervals (an , bn ) such that U ⊆ (an , bn ) and n∈N (bn −an ) < mL (U )+ .  Let n be a positive integer such that n>n (bn − an ) < .   Let G = [a, b] ∩ n 0. By Lemma 8.15, for every > 0 there exists a step function g on the measure space ([a, b], B([a, b]), mL ) such that 1U − gp < . If f is a Borel simple function on [a, b] and Ran(f ) = {y1 , . . . , yk }, then  f (x) = ki=1 yi 1f −1 (yi ) . Each of the sets f −1 (yi ) is a Borel subset of [a, b]

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 561

561

and 1f −1 (yi ) can be approximated by a step function gi ; in other words, for k every  > 0 we have 1f −1 (yi ) − gi p <  . It is clear that i=1 yi gi is a step function. Let  = k |y | . We have i=1

i

    k k k            yi gi  =  yi 1f −1 (yi ) − yi gi  f −     i=1

i=1

p



k 

i=1

p

|yi |1f −1 (yi ) − gi p  

i=1

k 

|yi |  ,

i=1



which completes the argument.

Lemma 8.16. Let f : [a, b] −→ R be a step function such that M = sup{f (x) | x ∈ [a, b]}. For every δ > 0 there exists a continuous function g : [a, b] −→ R such that sup{g(x) | x ∈ [a, b]} = M and mL ({x ∈ [a, b] | f (x) = g(x)}) < δ. Proof. Since f is a step function there exist a0 , a1 , . . . , an ∈ [a, b] and y0 , . . . , yn−1 ∈ R such that a = a0 < a1 < · · · < an = b and f (x) = yi for x ∈ (ai , ai+1 ) for 0  i  n − 1. Let > 0 be a number such that < min{ ai+12−ai | 1  i  n − 1}. Define the continuous function g as: ⎧ y0 (x − a0 ) if a0  x  a0 + , ⎪ ⎪ ⎪ ⎪ ⎪ ⎨yi−1 if ai−1 +  x  ai − , g(x) = yi − yi−1 ⎪ yi−1 + (x − ai + ) if ai −  x  ai + , ⎪ ⎪ 2 ⎪y ⎪ ⎩ n−1 (a − x) if an −  x  an . n We have: {x ∈ [a, b] | f (x) = g(x)} = [a0 , a0 + ]

n−1 

[ai − , ai + ] ∪ [an − , an ]

i=1

= 2 (n + 1). Note that sup{g(x) | x ∈ [a, b]} = M . Choosing < mL ({x ∈ [a, b] | f (x) = g(x)}) < δ.

δ 2(n+1)

we have 

Theorem 8.93. The subspace of Lp ([a, b]) determined by the continuous functions defined on [a, b] is dense in Lp ([a, b]) for 1  p < ∞.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 562

Mathematical Analysis for Machine Learning and Data Mining

562

Proof. Each continuous function belongs to Lp ([a, b]). Since the step functions form a dense subset of Lp ([a, b]) it suffices to show that for every > 0 and every step function f on [a, b] there is a continuous function g such that f − gp < . If g is constructed as in Lemma 8.16 we have 7 |f − g|p dmL  (2M )p mL ({x ∈ [a, b] | f (x) = g(x)}) < (2M )p δ. [a,b]

Since δ is arbitrary we obtain the desired conclusion.



Theorem 8.94. Let (S, E, m) be a measure space. If there exists a countable collection B = {Bk | k ∈ N} such that for every > 0 and for every set A ∈ E there exists Bk ∈ B such that m(A ⊕ Bk ) < , then the space Lp (S, E, m) is separable. Since m(A ⊕ Bk ) < , we have 1 1 1A − 1Bk p = (m(A ⊕ Bk )) p < p . It follows that each m-integrable function can be approximated in Lp (S, E, m) with any degree of accuracy by a simple function of the form  f (x) = nk=1 yk 1Bk (x). Therefore, we have a countable collection F of such functions dense in the set of m-integrable functions, which means that F is  dense in Lp (S, E, m).

Proof.

Definition 8.23. Two functions f, g ∈ Lp (S, E, m) are equivalent if {x ∈ S | f (x) = g(x)} is a null set. If f, g are equivalent we denote this by f ∼ g. It easy to verify that this is indeed an equivalence relation. We denote by [f ]p the equivalence class of f ∈ Lp (S, E, m); the quotient space Lp (S, E, m)/∼ is denoted by Lp (S, E, m), or just by Lp when there is no risk of confusion. Then Lp is a normed space, with the norm given by [f ]p = inf{νp (h) | h ∈ Lp , h ∼ f }. Actually, [f ]p = νp (f ). 6 p ˆ and Observe that if f : S −→ R S |f | dm < ∞, then m({x ∈ S | f (x) = ∞} = m({x ∈ S | |f (x)|p = ∞}) ⎛ ⎞  = m ⎝ {x ∈ S | |f (x)|p > j ⎠ j∈N

= lim m ({x ∈ S | |f (x)|p > j}) j→∞ 7  lim |f |p dm, j→∞

by Markov’s Inequality.

S

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 563

563

Theorem 8.95. (Riesz-Fischer Theorem) Let (S, E, m) be a measure space. For each p, 1  p  ∞ the space Lp (S, E, m) is complete. Proof. By Theorem 5.72 it suffices to show that every absolutely convergent series in Lp (S, E, m) is convergent. Suppose initially that 1  p < ∞. Let f = (fn ) be a sequence of functions in Lp (S, E, m) that is absolutely convergent. In other words, ∞ n=0 fn p < ∞. Let g : S −→ [0, ∞] be the function p p  n ∞   |fk (x)| = lim |fk (x)| . g(x) = n→∞

k=0

k=0

By Minkowski’s inequality we have  n  p  p1 7  n n        |fk | dm =  |fk |  fk p   S k=0

k=0

p

k=0

for every n ∈ N. Therefore, by the Monotone Convergence Theorem (theorem 8.11), we have p ∞ p 7  7 n  g dm = lim |fk | dm  fk p , S

n→∞

S

k=0

k=0

hence g is integrable. Thus, by Theorem 8.17, g is finite a.e., and the series f0 (x) + f1 (x) + · · · is absolutely convergent and, therefore, convergent a.e. ˆ be the function defined by Let f : S −→ R ⎧∞  ⎪ ⎨ fk (x) if g(x) < ∞, f (x) = k=0 ⎪ ⎩ 0 otherwise. Then, f is measurable and |f |p  g, so f ∈ Lp (S, E, m). Note that   n     fk (x) − f (x) = 0 lim  n→∞   k=0 n holds a.e.; also, | k=0 fk (x) − f (x)|p  g(x) holds a.e. By the Dominated Convergence Theorem (theorem 8.37),  n       fk − f  = 0. lim  n→∞   k=0

p

May 2, 2018 11:28

564

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 564

Mathematical Analysis for Machine Learning and Data Mining

Thus, the absolutely convergent series f is convergent, and we conclude that Lp (S, E, m) is complete. For the case p = ∞ let f be a sequence of functions in L∞ (S, E, m) that ∞ is absolutely convergent, that is, k=0 fk ∞ < ∞. Define the null set Uk = {x ∈ S | |fk (x)| > fk ∞ } and let U = ∞ ∞ k=0 Uk . The series k=0 fk (x) converges at each x ∈ S − U , and the function f given by ⎧∞  ⎪ ⎨ fk (x) if g(x) < ∞, f (x) = k=0 ⎪ ⎩ 0 otherwise.  is bounded and measurable. Since k∈N Uk is null, we have   n ∞       f − f  fk ∞  k    k=0

∞

k=n+1

for each n. Therefore,   n ∞       lim f − fk   lim fk ∞ = 0. n→∞   n→∞ k=0

k=n+1

∞

Again, this shows that L (S, E, m) is complete.



Let c1 ∈ Fc (S, O) be the constant function defined by c1 (x) = 1 for x ∈ S. If is a bounded positive functional on Fc (S, O) and f ∈ Fc (S, O) is such that f 1  1, then | (f )|  (|f |)  (c1 ), hence  1 = (c1 ). Theorem 8.96. Let L be a linear space of bounded real-valued functions defined on a set S, such that (i) the constant function c1 belongs to L; (ii) if f, g ∈ L, then the functions f ∨ g and f ∧ g given by (f ∨ g)(x) = max{f (x), g(x)} and (f ∧g)(x) = min{f (x), g(x)}, respectively, for x ∈ S belong to L. If is a bounded linear functional on L, there exist two positive linear functionals + and − such that = + − − , and   =

+ (c1 ) + − (c1 ). Proof.

For a non-negative function f ∈ L let + be defined as

+ (f ) = sup{ (h) | 0  h  f }.

We have + (f )  0 and (f )  + (f ). Note that a  0 implies + (af ) = a + (f ). We claim that for two non-negative functions f, g ∈ L we have + (f + g) = + (f ) + + (g).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 565

565

If 0  h1  f and 0  h2  g, then 0  h1 + h2  f + g, which implies

+ (f ) + + (g)  + (f + g). If 0  h  f + g, then 0  h ∧ f  f , hence 0  h − (h ∧ f )  g and, therefore,

(h) = (h ∧ f ) + (h − (h ∧ f ))  + (f ) + + (g). By the definition of + , this implies + (f + g)  + (f ) + − (g), hence

+ (f + g) = + (f ) + − (g). Note that the previous considerations are developed for non-negative functions in L. We need to extend these observations to the entire function space L. Let f ∈ L be an arbitrary function in L. Since the functions of this space are bounded there exist two non-negative constants k1 , k2 such that both f + k1 and f + k2 are non-negative. We have

+ (f + k1 + k2 ) = + (f + k1 ) + + (k2 ) = + (f + k2 ) + + (k1 ), hence + (f + k1 ) − + (k1 ) = + (f + k2 ) − + (k2 ). Therefore, the value of

+ (f + k) − + (k) is independent of k. Now, the value of + (f ) is extended to the entire space L by choosing k such that f + k is non-negative and defining + (f ) as + (f +k)− + (k). Now we have + (f +g) = + (f )+ − (g) and + (af ) = a + (f ) for every f, g ∈ L and a  0. Furthermore, since

+ (−f ) + + (f ) = + (0) = 0, we have + (−f ) = − + (f ), hence + (af ) = a + (f ) for a ∈ R, so + is a linear functional on L. Since = + − , and

(f )  + (f ), it follows that − is a positive linear functional. Note that     +  +  −  = + (c1 ) + − (c1 ). If f ∈ L is a function such that 0  f (x)  1, then |2f (x) − 1|  1, and    (2f − 1) = 2 (f ) − (c1 ). Therefore,    2 +( c1 ) − (c1 ) = 

+ (c1 ) + − (c1 ), hence   = + (c1 ) + − (c1 ). 8.13

Fourier Transforms of Measures

Define the function s : R0 −→ R as: 7 t sin x s(t) = dx. x 0 We have shown in Example 8.9 that limt→∞ s(t) = that 7 t sin tθ dt = sign(θ)s(t|θ|) t 0 for θ ∈ R.

π 2.

It is easy to verify

(8.26)

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 566

Mathematical Analysis for Machine Learning and Data Mining

566

Definition 8.24. Let m be a finite measure on Rn . The Fourier transform of m is the function m ˆ : Rn −→ Rn defined as: 7 ei(t,x) dmx . m(t) ˆ = Rn

Recall that the sphere B∞ (0n , r) in (Rn , d∞ ) is: B∞ (0n , r) = {x ∈ Rn | −r < xk < r for 1  k  n}. In the next theorem we prove that the Fourier transform of a measure m defines the values of the measure m on open-closed intervals of Rn whose border is a null set. Since the family of open-closed intervals in Rn is a semi-ring of sets that generates the σ-algebra B(R) (by Supplement 30 of Chapter 7), it follows that two measures that have the same Fourier transforms must be equal. Theorem 8.97. (The Inversion Theorem) Let G = (a1 , b1 ]×·×(an , bn ] be a bounded open-close interval and let m be a measure on Rn such that m(∂G) = 0. We have: 7 n + 1 e−itk ak − e−itk bk m(t) ˆ dt, m(G) = lim n r→∞ (2π) itk B∞ (0n ,r) k=1

where dt denotes dt1 · · · dtn . Proof.

We have 7

n + e−itk ak − e−itk bk m(t) ˆ dt itk B∞ (0n ,r) k=1 7  7 n + e−itk ak − e−itk bk 1 i(t,x) e dm dt = x (2π)n B∞ (0n ,r) itk Rn k=1  7 7 n + e−itk ak − e−itk bk i(t,x) 1 e dt dmx . = (2π)n Rn itk B∞ (0n ,r)

1 (2π)n

k=1

Next, the inner integral can be evaluated using Fubini’s theorem as: 7 n + e−itk ak − e−itk bk i(t,x) e dt itk B∞ (0n ,r) k=1 7 n + e−itk ak +itk xk − e−itk bk +itk xk = dt itk B∞ (0n ,r) k=1

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

7

7

r

= −r n +

=

··· 7

−r

k=1

The integral 7

6r −r

r

r

page 567

567

n + eitk (xk −ak ) − eitk (xk −bk ) dt itk

−r k=1

eitk (xk −ak ) − eitk (xk −bk ) dtk . itk

eitk (xk −ak −eitk (xk −bk ) itk

dtk can be rewritten as

eitk (xk −ak ) − eitk (xk −bk ) dtk itk −r 7 0 itk (xk −ak ) 7 r itk (xk −ak ) e − eitk (xk −bk ) e − eitk (xk −bk ) = dtk + dtk itk itk −r 0 7 0 −itk (xk −ak ) 7 r itk (xk −ak ) e − e−itk (xk −bk ) e − eitk (xk −bk ) = + itk itk r 0 (by changing the variable t in the first integral to −t) 7 0 −itk (xk −bk ) −e−itk (xk −ak ) (eitk (xk −ak ) − eitk (xk −bk )+e itk ) = r 7 r sin tk (xk − ak ) − sin tk (xk − bk ) =2 dtk tk 0 = 2(sign(xk − ak )s(t · |xk − ak |) − sign(xk − bk )s(t · |xk − bk |)). r

This allows us to write 7

n + e−itk ak − e−itk bk m(t) ˆ dt itk B∞ (0n ,r) k=1 7 n + 1 n = 2 (sign(xk − ak )s(t · |xk − ak |) (2π)n Rn

1 (2π)n

k=1

−sign(xk − bk )s(t · |xk − bk |))dmx 7 + n 1 = n (sign(xk − ak )s(t · |xk − ak |) π Rn k=1

−sign(xk − bk )s(t · |xk − bk |))dmx n 7 1 + ∞ = n (sign(xk − ak )s(t · |xk − ak |) π −∞ k=1

−sign(xk − bk )s(t · |xk − bk |))dxk .

May 2, 2018 11:28

568

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 568

Mathematical Analysis for Machine Learning and Data Mining

Let γa,b,t (x) = π1 (sign(x − a)s(t · |x − a|) − sign(x − b)s(t · |x − b|)). The function γa,b,t is bounded and limt→∞ γa,b,t (x) = ξa,b (x), where ⎧ ⎪ 0 if x < a, ⎪ ⎪ ⎪ ⎪ 1 ⎪ ⎪ ⎨ 2 if x = a, ξa,b (x) = 1 if a < x < b, ⎪ ⎪ 1 ⎪ ⎪ if x = b, ⎪ 2 ⎪ ⎪ ⎩0 if x > b. 6∞ Therefore, −∞ ξa,b (x) dx = m((a, b]) + 12 m({a}) + 12 m({b}) = m((a, b]) because m({a}) = m({b}) = 0. Consequently, 7 n + 1 e−itk ak − e−itk bk m(t) ˆ dt, lim n r→∞ (2π) itk B∞ (0n ,r) k=1

=

n +

m((ak , bk ]) = m(G). 

k=1

Corollary 8.12. (Cram´ er-Wold Theorem) The values of a finite measure m on Rn on every half-space of Rn determine completely the measure m.  be a half-space of Rn . If m is a measure on Rn , the Proof. Let Hw,a measure of this half-space is m(Hw,a ) = m(f −1 (−∞, a)), where f : Rn −→ R is the linear functional defined by f (x) = 6(w, x) for x ∈ Rn . The Fourier transform of the measure mf −1 on R is R eisa d(mf −1 ). Applying a transformation of variables we obtain: 7 7 isa −1 e d(mf ) = eis(t,x) dmx = m(st). ˆ

R

6

Rn

In particular, m(t) ˆ = R eia d(mf −1 ). Thus, the values of m on half-spaces  in Rn determine completely the measure m. Corollary 8.13. The values of a finite signed measure m on Rn on every half-space of Rn determine completely the signed measure m. Proof. Let m = m+ − m− be the Jordan decomposition of m as a difference of two measures (established in Theorem 7.72). The measure m determines uniquely the measures m+ and m− and these measures are, in turn determined uniquely by their values on half-spaces. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Integration

8.14

b3234-main

page 569

569

Lebesgue-Stieltjes Measures and Integrals

We start from the fact that the set F of unions of finite collections of disjoint open-closed intervals of R is an algebra of sets. Theorem 8.98. Let F : R −→ R be an increasing and right-continuous function. The function φF : F −→ R defined by ⎛ φF ⎝

n  j=1

⎞ (aj , bj ]⎠ =

n 

(F (bj ) − F (aj ))

j=1

is a premeasure on F. Proof. A sequence of open-closed intervals ((c1 , d1 ], . . . , (cm , dm ]) is connected if dk = ck+1 for 1  k  m − 1. For such a sequence m F In this situation we refer to we have φF ((c, d]) = j=1 φ ((cj , dj ]). ((c1 , d1 ], . . . , (cm , dm ]) as a sequence subordinated to (c, d]. Note that in the definition of φF the contribution of an open-closed interval can be replaced with the sum of the contributions of any subordinated sequence of open-closed intervals without changing the valued of φF . Thus, φF is well-defined. This also shows that φF is finitely additive. We need to verify that if {Un | n  1} is a sequence of pairwise disjoint   sets in F such that T = n1 Un ∈ F, then φF (T ) = n1 φF (Un ). Since T ∈ F, T is a finite union of pairwise disjoint open-closed intervals, m m F T = p=1 Wp , we have φF (T ) = p=1 φ (Wp ). Therefore, the family {Un | n  1} can be partitioned into a finite number of disjoint subfamilies  W1 , . . . , Wm such that Wi = {U | U ∈ Wi } for 1  i  m. n Suppose that Wi = {Uk | k ∈ N}. Since φF (Wi )  φF ( k=1 Uk ) =  n ∞ F F F k=1 φ (UK ) for n  1, it follows that φ (Wi )  k=1 φ (Uk ). Conversely, suppose that Wi = (a, b], where a, b ∈ R and that Uk = (ak , bk ) for k  1. Since F is right-continuous, for every > 0 there exists δ such that F (a + δ) − F (a) < and for each k  1, there exists δk such that F (bk + δk ) − F (bk ) < 2 k .   Since (a, b] = k1 (ak , bk ), we have [a + δ, b] = k1 (ak , bk + δj ). The compactness of [a+δ, b] implies that there exists a finite subcover of [a+δ, b] by intervals of the form (ak , bk +δj ). Keeping only those open intervals that are maximal we obtain the intervals (a1 , b1 + δ1 ), . . . , (aq , bq + δq ) that cover

May 2, 2018 11:28

570

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 570

Mathematical Analysis for Machine Learning and Data Mining

[a + δ, b], where bk + δk ∈ (ak+1 , bk+1 + δk+1 ) for 1  k  q − 1. We have φF ((a, b]) = F (b) − F (a) < F (b) − F (a + δ) + (because F (a + δ) − F (a) < )  F (bn + δq ) − F (a1 ) + = F (bn + δq ) − F (aq ) +  F (bn + δq ) − F (aq ) +

q−1  k=1 q−1 

(F (aj+1 − F (aj )) + (F (bj + δj ) − F (aj )) +

k=1 q 

 − F (a )) + < φF ((aj , bj ]) + 2 . j j 2 j=1 j1  F F Since is arbitrary, this implies φ  j1 φ ((aj , bj ]). If a = −∞, then the intervals (aj , bj + δj ) cover [−t, b] for every finite  t, which imply F (b) − F (−t)  j1 φF ((aj , bj ]) + 2 , as above.  If b = ∞, F (t) − F (a)  j1 φF ((aj , bj ]) + 2 . Since this holds for arbitrary and t, we get the reverse inequality. 
0, ⎪ ⎨m((0, x]) F (x) = 0 if x = 0, ⎪ ⎪ ⎩−m((x, 0]) if x < 0, is an increasing and right-continuous function and m = mF . Proof. Let φF be the premeasure on the algebra F of unions of finite collections of disjoint open-closed intervals of R. If F −G is a constant, then it is immediate that φF = φG . Moreover, φF is σ-finite. The first part of the theorem follows from Carath´eodory Extension Theorem (Theorem 7.39). The monotonicity of m implies the monotonicity of F ; also, and the continuity properties of mF (Theorem 7.32) imply the right continuity of F . Since m = mF on F, the equality of these measures on B(R) follows. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Integration

9in x 6in

b3234-main

page 571

571

The measure mF introduced above is the Lebesgue-Stieltjes measure defined by F . Example 8.10. For F (x) = x, mF is the usual Lebesgue measure on R. If F is defined as  1 if x  x0 , F (x) = 0 if x < x0 , then the Lebesgue-Stieltjes measure is the Dirac measure concentrated at x0 , δx0 . Definition 8.25. A function F : R −→ R is absolutely continuous on an interval [a, b] if for every > 0 there exists δ > 0 such that for every finite collection of disjoint open intervals {(xj , yj ) | (xj , yj ) ⊆ [a, b] for 1  j  n n n} with j=1 (yj − xj ) < δ we have j=1 |F (xj ) − F (yj )| < . Example 8.11. Let h ∈ L1 ([a, b]) 6 x be a finite function. Then the function fh : R −→ R defined by fh (x) = a h dmL is absolutely continuous. Indeed, let {(xj , yj ) | (xj , yj ) ⊆ [a, b] for 1  j  n} be a finite collection and let n E = j=1 (xj , yj ). We have:   7 n n  7 yj n 7 yj       |fh (yj ) − fh (xj )| = h dmL   |h| dmL = |h| dmL .   xj  E j=1 j=1 j=1 xj 6 Since h ∈ L1 ([a, b]), the measure m(E) = E |h| dmL is absolutely continuous with respect to mL and, therefore, by Theorem 8.53, there exists δ > 0 such that mL (E) < δ implies m(E) < . If the total length n j=1 (yj − xj ) < δ, then m(E) < . Theorem 8.100. Let F : R −→ R be a monotone and absolutely continuous on R and let mF be the Lebesgue-Stieltjes measure generated by F . Then every Lebesgue measurable set is mF -measurable and mF # mL . Proof. Let B be a Borel set such that mL (B) = 0. We claim that mF (B) = 0. There exists a sequence of open sets U1 ⊇ U2 ⊇ · · · ⊇ B such that mL (U1 ) < δ and thus, mL (Uj ) < δ for all j ∈ N. Also, limn→∞ mF (Uj ) = mF (B). By Theorem 4.1 an open set Uj can be written as a union of disjoint  open intervals Uj = k∈N (akj , bkj ); therefore, n n   |mF (akj , bkj )|  |F (bkj ) − F (akj )|  . k=1

k=1

When n → ∞ we obtain |mF (Uj )| < , hence |mF (B)|  . Since is  arbitrary, mF (B) = 0, hence mF # mL .

May 2, 2018 11:28

572

8.15

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 572

Mathematical Analysis for Machine Learning and Data Mining

Distributions of Random Variables

Definition 8.26. Let X : Ω −→ R be a6 random variable on a probability space (Ω, E, P ). The Lebesgue integral Ω X dP is the expectation of X or the mean value of X and it is denoted by E(X). Markov’s inequality (Theorem 8.41) applied to a random variables X becomes P (X  a) 

1 E(X). a

6 The number Ω (X − E(X))2 dP is the variance of X and is denoted by var(X). Chebyshev’s Inequality (inequality (8.6)) can be written as: P (|X − E(X)| > k

"

var(X)) 

1 . k2

Definition 8.27. The distribution of a random variable X : Ω −→ R is the ˆ defined by: measure PX on B(R) PX (U ) = P (X −1 (U )) = P ({ω ∈ Ω | X(ω) ∈ U }, for every Borel subset U of R. The triple (R, B(R), PX ) is itself a probability space referred to as the probability space induced by X. Instead of writing PX (U ) we may write P (X ∈ U ) for any Borel subset U of R. Example 8.12. A discrete random variable is a random variable X : Ω −→ R that is a measurable function between a probability space (Ω, E, P ) and (R, B(R), mL ) such that Ran(X) = {x1 , . . . , xn , . . .} is a countable set. The partition of Ω related to X is {X −1 (x1 ), . . . , X −1 (xn ), . . .}. The number pj = P (X −1 (xj ) is the point mass in xj .  Note that j≥1 pj = 1. Definition 8.28. Let X, Y be two random variables on the measure space (Ω, E, P ). X and Y are equal in distribution if PX = PY . X and Y are pointwise equal if P ({ω ∈ Ω | X(ω) = Y (ω)} = 1. If X, Y are pointwise equal, then they are equal in distribution. The reverse is false as the next example shows.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 573

573

Example 8.13. Let (Ω, E, P ) be the probability space introduced in Example 7.19 that describes the coin throwing experiment. Define X, Y as   1 if ω = h, 1 if ω = t, X(ω) = and Y (ω) = 0 if ω = t 0 if ω = h. X and Y are equal on distribution; however, X(ω) = Y (ω) for every ω ∈ Ω = {h, t}. X : Ω −→ R is a random variable on the probability space (Ω, E, P ) if and only if for every Borel set A, we have X −1 (A) ∈ E. Since the Borel σ-algebra B(R) is generated by sets of the form (−∞, x] (by Theorem 7.5) it suffices to have X −1 ((−∞, x]) = {ω ∈ Ω | X(ω)  x} ∈ B(R) for every x ∈ R for X to be a random variable. Definition 8.29. Let X : Ω −→ R be a random variable on the probability space (Ω, E, P ). The distribution function of X is the function FX : R −→ [0, 1] defined by FX (x) = P ({ω | X(ω)  x}) for x ∈ R. To simplify notation we will denote P ({ω | X(ω)  x}) by P (X  x). Example 8.14. The distribution function of a discrete random variable X  with Ran(X) = {x1 , . . . , xn , . . .} is FX (x) = {pj | xj  x}. Definition 8.30. A random variable X is (i) continuous if FX is continuous; (ii) absolutely continuous if there exists a non-negative 6 b Lebesgue integrable function f such that FX (b) − FX (a) = a f dmL ; (iii) singular if F  exists and F  (x) = 0 a.e. Example 8.15. If X is a discrete random variable, then X is singular. Theorem 8.101. Let FX : R −→ [0, 1] be a distribution function of a random variable X. Then FX is non-decreasing function that is monotonic, 0  FX (x)  1, FX is right-continuous, limx→−∞ FX (x) = 0, and limx→∞ FX (x) = 1. Proof. FX is clearly monotonic and 0  FX (x)  1. Let (un ) be a decreasing sequence in R such that limn→∞ un = x0 . The sequence of events (Un ) defined by Un = {ω ∈ Ω | X(ω)  un } is  decreasing and n∈N Un = U , where U = {ω ∈ Ω | X(ω)  x}. Therefore, by the continuity from above shown in Theorem 7.32, FX (x0 ) = P (lim Un ) = lim P (Un ) = lim FX (un ), n→∞

n→∞

May 2, 2018 11:28

574

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 574

Mathematical Analysis for Machine Learning and Data Mining

which shows that FX (x0 ) = lim x → x0 +FX (x), that is, FX is right-continuous. By the continuity from below, it follows that if limx→x0 − FX (x) = P ({ω ∈ Ω | X(ω < x0 }. It is clear that  limx→−∞ FX (x) = 0, and limx→∞ FX (x) = 1. Corollary 8.14. A distribution function FX of a random variable X has at most a countable set of discontinuity points. Proof.

Since

jump(FX , x0 ) = lim FX (x) − lim FX (x) = P ({ω ∈ Ω | X(ω) = x0 }). x→x0 +

x→x0 −

Therefore, by Theorem 4.62, FX can have at most a countable set of discontinuity points.  Theorem 8.102. If F : R −→ [0, 1] is non-decreasing function that is monotonic, 0  F (x)  1, F is right-continuous, limx→−∞ F (x) = 0, and limx→∞ FX (x) = 1, then there exists a random variable X such that F = FX . Proof. Consider the probability space ([0, 1], B([0, 1], P ), where P is the Lebesgue measure mL restricted to [0, 1]. Suppose initially that F is continuous and strictly increasing. Thus, F is invertible. Define the random variable X as X(a) = F −1 (a) for a ∈ [0, 1]. Since F −1 is increasing, X is measurable. We have P ({a ∈ [0, 1] | X(a)  x}) = mL ({a ∈ [0, 1] | F −1 (a)  x}) = mL ({a ∈ [0, 1] | a  F (x)}) = F (x). If F is not strictly increasing let h be the function defined as h(a) = inf{x | a  F (x)}. Note that the set {x | a  F (x)} has the form [h(a), ∞) because F is right-continuous. This implies h(a)  x if and only if a  F (x). Thus, if X is defined as X(a) = h(a) for 0 < a < 1, then X is a random variable and mL ({a | X(a)  x}) = F (x), hence  mL (a, b] = F (b) − F (a). Definition 8.31. An absolutely continuous random variable X on a probability space (Ω, E, P ) has a probability density function (pdf) f if f : such that for every U ∈ B(R) we R −→ R0 is a measurable function 6 have P ({ω ∈ Ω | X(ω) ∈ U }) = U f (x) dx. If6 f is a random variable density of a random variable X, then f (x)  0 and R f (x) dx = 1. It is clear that the probability density function is not

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Integration

b3234-main

page 575

575

unique for a random variable. Indeed if f is a density for X and f = g a.e., then g is also a density for X. If FX is a distribution functions having the pdf f , we have 7 x FX (x) = PX (X  x) = f (t) dt. −∞

Example 8.16. Let a, b ∈ R be such that a < b. A random variable X whose range is [a, b] has a uniform distribution if its pdf is ⎧ ⎨ 1 if x ∈ [a, b], f (x) = b − a ⎩0 otherwise. The corresponding distribution function is ⎧ 0 ⎪ 7 x ⎨x − a FX (x) = f (t) dt = ⎪ −∞ ⎩b−a 1

if x < a, if a  x < b, if b  x.

Uniformly distributed random variable play an important role highlighted by the following statement: Theorem 8.103. Let X be a random variable that has a continuous distribution function FX . The random variable Y = FX (X) has a uniform distribution on [0, 1]. Proof. Since 0  FX (x)  1 if follows that P (Y < 0) = P (Y > 1) = 0. Let y ∈ (0, 1). If FX is strictly increasing there is a unique x0 such that FX (x0 ) = y. Then FY (y) = P (Y  y) = P (X  x0 ) = FX (x0 ) = y for y ∈ (0, 1), which means that Y is uniformly distributed in [0, 1]. If FX is not strictly increasing, then x0 can be arbitrarily chosen such that  FX (x0 ) = y. Theorem 8.103 is useful for generating random variables with a prescribed distribution function starting with a random variable distributed uniformly. Indeed, if we seek to generate values of a random variable X with an invertible distribution function FX , taking into account that −1 (Y ). Y = FX (X) is uniformly distributed on [0, 1] we obtain X = FX Example 8.17. The random variable X has a normal distribution with parameters m, σ if it has the pdf x−m 2 1 fm,σ (x) = √ e−( σ ) σ 2π for x ∈ R. Note that f (x) > 0 for x ∈ R.

May 2, 2018 11:28

576

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 576

Mathematical Analysis for Machine Learning and Data Mining

6 To show that f is a pdf we need to verify that R fm,σ (x) dx = 1. Let y = x−m σ . We have: 7 7 1 1 2 √ e− 2 y dy. fm,σ (x) dx = 2π R R 6 − 1 y2 Define I = R e 2 dy. Then, we can write 7 7 7 2 2 1 2 − 12 y 2 − 12 z 2 e dy e dz = e− 2 (y +z ) dy dz. I = R

R

R

Changing the variables to polar coordinates y = r cos θ and z = r sin θ we obtain 7 2π 7 ∞ r2 e− 2 r dr dθ = 2π, I2 = 0

0

which shows that fm,σ is indeed a pdf. The class of random variables with parameters m, σ is denoted by N(m, σ). If X belongs to this class we will write X ∼ N(m, σ). If X : Ω −→ R is random variable and h : R −→ R is a Borel measurable function, then Y = hX is a random variable by Theorem 7.78. Suppose that h is a monotonic and invertible function having the inverse g. The distribution of Y is PY (U ) = P (hX ∈ U ) = P (X ∈ h−1 (U )).

(8.27)

The distribution function of Y is: FY (t) = P (Y  t) = P (hX  t) = P (X  g(t)) = FX (g(t)). Thus, FY (t) = FX (g(t)) for t ∈ R.  = f , then the density of If FX has a density that allow us to write FX     FY is FY (t) = FX (g(t))g (t) = f (g(t))g (t) for t ∈ R. Example 8.18. Suppose that h(t) = at + b for t ∈ R, where a > 0. Then  h is monotonic and invertible and for Y = aX + b we have FY (t) = FX t−b a . If FX has a density, then the density of FY is   1  t−b  FY (t) = FX . a a For example, if X ∼ N(m, σ), then FY (t) = Y ∈ N(am + b, aσ).

aσ

1 √

e− 2π

(t−(am+b))2 a2 σ2

, hence

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Integration

8.16

9in x 6in

b3234-main

page 577

577

Random Vectors

Definition 8.32. An n-dimensional random vector is a measurable function X from a probability space (Ω, E, P ) into (Rn , B(Rn )). The distribution of X is the measure PX on Rn defined by PX (U ) = P ({ω ∈ Ω | X(ω) ∈ U }) for U ∈ B(Rn ). Let pj : Rn −→ R be the j th projection, pj : Rn −→ R, where 1  j  n. The random variable Xj = pj X is the j th component of X. The joint distribution function of X is the function FX : Rn −→ R defined by ⎛ ⎞ n  FX (x1 , . . . , xn ) = P ⎝ {ω ∈ Ω | Xj  xj }⎠ . j=1

n Note that FX (x) = P ( i=1 (Xi  xi )) for x ∈ Rn . The marginal distributions of the distribution function FX are the distribution functions FXi . Let X : Ω −→ Rn be a random vector and let h : Rn −→ Rk is a measurable function between (Rn , B(Rn ), mL ) and (Rm , B(Rm ), mL ). Then, using an argument similar to the one of Theorem 7.78, it follows that Y = hX is an m-dimensional random vector. Definition 8.33. Let X1 , . . . , Xn be n random variables on a probability space (Ω, E, P ). X1 , . . . , Xn are independent if for any sets U1 , . . . , Un ∈ B(R) we have   n n  + Xi ∈ U i = P (Xi ∈ Ui ). P i=1

i=1

Theorem 8.104. The random variables X1 , . . . , Xn are independent if n FX (x1 , . . . , xn ) = i=1 FXi (xi ). Proof. The necessity of the condition of the theorem follows by taking Ui = (−∞, xi ] for 1  n in Definition 8.33. The sufficiency of follows from Theorem 7.22 and the fact that sets of the form (−∞, x] form a π-system that generates B(R).  Let X1 , . . . , Xn be n independent random variables. If FXi has the density fi for 1  i  n, then Fubini’s Theorem implies that 6 xn 6 x1 · · · −∞ f1 · · · fn dmL = FX1 (x1 ) · · · FXn (xn ), so the distribution func−∞ tion FX of the random vector X having components X1 , . . . , Xn has the density fX given by fX (x) = fX1 (x1 ) · · · fXn (xn ) for x ∈ Rn .

May 2, 2018 11:28

578

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 578

Mathematical Analysis for Machine Learning and Data Mining

Theorem 8.105. The random variables X1 , X2 on the probability space (Ω, E, P ) are independent if and only if E(f (X1 )g(X2 )) = E(f (X1 ))E(g(X2 )) for all Borel measurable bounded functions f, g. Proof. We first show that the condition is sufficient. If we choose f = 1B1 and g = 1B2 , where B1 , B2 are two Borel sets, we have 7 1B1 ×B2 (X1 (ω), X2 (ω)) dP P ((X1 ∈ B1 ) ∩ (X2 ∈ B2 )) = 7Ω = 1B1 (X1 (ω))1B2 (X2 (ω)) dP 7Ω 7 = 1B1 (X1 (ω)) dP 1B2 (X2 (ω)) dP Ω

Ω

= P (X1 ∈ B1 )P (X2 ∈ B2 ), which show that X1 , X2 are indeed independent. To prove that the necessity suppose that X1 , X2 are independent. The equality of the theorem holds for the Borel sets B1 , B2 , f = 1B1 , g = 1B2 as we saw. If f, g are simple measurable functions, then f and g are linear combinations of characteristic functions of sets in E, f=

n 

bi 1Bi , g =

i=1

This allows us to write

E(f (X1 )g(X2 )) = E ⎝

n 

bi 1Bi

i=1

=

=

cj 1 C j .

j=1

⎛

=

m 

n  i=1 n  i=1 n  i=1

b i cj b i cj

m 

⎞ cj 1 C j ⎠

j=1 m  j=1 m 

E(1Bi (X1 )1Cj (X2 )) E(1Bi (X1 ))E(1Cj (X2 ))

j=1

bi E(1Bi (X1 ))

m 

E(1Cj (X2 ))

j=1

= E(f (X1 ))E(g(X2 )). Consider now two measurable bounded functions. Since every bounded measurable function f is the limit of a sequence of simple measurable functions, by the Dominated Convergence Theorem the result follows. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 579

579

Definition 8.34. Let X1 , X2 be two random variables. The correlation coefficient of X1 and X2 is the number: cov(X1 , X2 ) ρX1 ,X2 = " . var(X1 )var(X2 ) We have |ρX1 ,X2 |  1. The random variables X1 , X2 are uncorrelated if ρX1 ,X2 = 0. Theorem 8.106. Let X1 , X2 be two random variables. If X1 , X2 are independent, then they are uncorrelated, that is, cov(X1 , X2 ) = 0. Proof. Let f, g be the functions defined by f (x) = x − E(X1 ) and g(x) = x − E(X2 ) in Theorem 8.105. Since X1 , X2 are independent we have cov(X1 , X2 ) = E(f (X1 )g(X2 )) = E((X1 − E(X1 ))(X2 − E(X2 ))) = E(X1 − E(X1 ))E(X2 − E(X2 )) = 0 because E(X1 − E(X1 )) = E(X2 − E(X2 )) = 0.



Definition 8.35. Let X be a random vector ranging in Rn . The mean vector of X is the vector m = E(X) ∈ Rn having the components mj = E(Xj ) for 1  j  n. The covariance matrix of X is the matrix cov(X), where (cov(X))ij = E((Xi − E(Xi ))(Xj − E(Xj ))). Observe that

 cov(X)ij =

var(Xi )

if i = j,

cov(Xi , Xj ) if i = j.

Theorem 8.107. The covariance matrix of a random vector is positive semi-definite. Proof. Let X be a random vector ranging in Rn . Since cov(Xi , Xj ) = cov(Xj , Xi ) for 1  i, j  n, it follows that cov(X) is symmetric. Moreover, we have  yi cov(Xi , Xj )yj y cov(X)y = i

j

⎛

= cov ⎝ where Z =



j yj Xj .

 i

yi Xi ,



⎞ yj Xj ⎠ = var(Z)  0,

j



If rank(cov(X)) = n then X is a non-degenerate n-dimensional random vector.

May 2, 2018 11:28

580

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 580

Mathematical Analysis for Machine Learning and Data Mining

˜ =X− If X is an n-dimensional random vector, the random vector X ˜ = 0. E(X) is said to be centered. This is justified by the fact that E(X)  ˜ ˜ The covariance matrix of X can now be written as cov(X) = E(XX ). Theorem 8.108. Let (ω, E, P ) be a probability space, X be an integrable random variable, and let G be a σ-algebra on Ω such that G ⊆ E. There exists an integrable random variable Y : Ω −→ R on (Ω, G, P ) such that 6 6 Y dP = X dP for every G ∈ G. G G Proof. 6 Suppose initially that X is non-negative. Define the measure m(G) = G X dP . Since X is integrable, m is finite and m # P . By RadonNikodym Theorem6(Theorem 8.58) there exists a G-measurable function f such that m(G) = G f dP . Then, we can define Y = f . If X is not non-negative, the same argument can be applied separately to X + and X − . If Y + and Y − are the resulting random variables, Y can be defined as Y = Y + − Y − .  The random variable Y constructed in the proof of Theorem 8.108 is not unique (there could be several such random variables that may differ on G-null sets). If Y1 and Y2 are two random variables that satisfy the condition of Theorem 8.108, then P (Y1 = Y2 ) = 1. Definition 8.36. Let (ω, E, P ) be a probability space, X be an integrable random variable and let G be a σ-algebra on Ω such that G ⊆ E. A conditional expectation of X generated by the σ-algebra G is a random variable Y introduced as in Theorem 8.108. With the notation introduced here we 7have: 7 E(X|G) dP = X dP for every G ∈ G.

G

G

Example 8.19. Let (ω, E, P ) be a probability space, X be an integrable random variable, and let G = {∅, Ω}. We have E(X|G) = E(X). If X is G-measurable, then for 6every Borel set B ∈ B(R) we have X −1 ∈ 6 G. Therefore, G E(X|G) dP = G X dP for every G ∈ G and we have E(X|G) = X almost surely. Example 8.20. If X −1 (B(R)) and G are independent, then E(X|G) = E(X). Indeed, note that {1G | G ∈ G} are independent random variables, hence 7 7 X dP = E(1G X) = E(X)E(1G ) = E(X) dP, G

G

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 581

581

hence E(X) a version of E(X|G). Since E(X) is constant, the equality holds. Example 8.21. Let (Ω, E, P ) be a probability space, π = {Bi | i ∈ I} be a countable partition of Ω that consists of members of E, and let G = Eπ (see Theorem 1.34). The random variable Y = E(X|G) has the value ai 6 over Bi , hence ai P (Bi ) = Bi X dP . Thus, if P (Bi ) > 0 and ω ∈ Bi we have 7 1 X dP. Y (ω) = P (Bi ) Bi If P (Bi ) = 0, the value of Y (ω) is constant on Bi (but arbitrary). Definition 8.37. Let (Ω, E, P ) be a probability space, Y be an integrable random variable on this space and let X be another random variable on the same space. Define the conditional expectation E(Y |X) as E(Y |EX ). Conditional expectation is linear, that is, E(aX + bY |G) = aE(X|G) + bE(Y |G) for a, b ∈ R. The random variable aE(X|G) + bE(Y |G) is Gmeasurable. If A ∈ G, then by the linearity of the integral we have 7 7 7 (aE(X|G) + bE(Y |G)) dP = a E(X|G) dP + b E(Y |G) dP A A 7A 7 7 = a X dP + b Y dP = (aX + bY ) dP. A

A

A

If X  Y , then E(X|G)  E(Y |G). Indeed, we have 7 7 7 7 E(X|G) dP = X dP  Y dP = E(Y |G), A

A

A

A

which implies P (E(X|G) − E(Y |G) > 0) = 0, which yields the conclusion. Theorem 8.109. (The Tower Property) Let (Ω, E, P ) be a probability space, and let G1 , G2 be two σ-algebras on Ω such that G1 ⊆ G2 ⊆ E. If X is a random variable such that E(X) < ∞ then E(E(X|G2 )|G1 ) = E(X|G1 ). 6 6 Proof. We have 6 G2 E(X|G2 ) dP = G2 X dP for G2 ∈ G2 and 6 G1 E(X|G1 ) dP = G1 X dP for G1 ∈ G1 ⊆ G2 . Therefore, for G1 ∈ G1 we obtain 7 7 E(X|G2 ) dP = G1

E(X|G1 ) dP. G1

Thus, E(X|G1 ) satisfies the condition defining the conditional expectation of E(X|G2 ) with respect to G1 , and the equality of the theorem follows. 

May 2, 2018 11:28

582

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 582

Mathematical Analysis for Machine Learning and Data Mining

Corollary 8.15. Let (Ω, E, P ) be a probability space, and let X, Y be two random variables defined on this space, where X is integrable (that is, E(X) < ∞). We have E(E(X|Y )) = E(X). Proof. If G1 = {∅, Ω} and G2 = EY we have G1 ⊆ G2 . The equality of theorem 8.109 amounts to the equality to be shown here.  Exercises and Supplements (1) Let f : Rr −→ R be a Lebesgue measurable function such that l(f ) = inf{ | mL ({x | |f (x)| > })  } is finite. Prove that: (a) We have λ = l(f ) if and only if i. mL ({x | |f (x)| > λ})  λ, and ii. mL ({x | |f (x)| > λ − δ}) > λ − δ for each δ > 0. Solution: Suppose that λ = l(f ). By the definition of l(f ) we have mL ({x | |f (x)| > l(f ) − δ}) > l(f ) − δ for each δ > 0, so the second condition of the first part is satisfied by λ. By the same reason, mL ({x | f (x) > l(f ) + })  l(f ) +  for each  > 0. As  → 0, the set {x | f (x) > l(f ) + } expands towards {x | f (x) > l(f )}. Thus,   lim mL ({x | f (x) > l(f ) + }) = mL lim {x | f (x) > l(f ) + } .

→0

→0

We conclude that mL ({x | |f (x)| > l(f )})  l(f ), so the first condition is satisfied. Conversely, suppose that both conditions are satisfied by λ. By the definition of infimum we have λ − δ < l(f )  λ for each δ > 0, which is possible only if λ = l(f ). ˆ 0 . Prove that (2) Let (S, E, m) be a measure space and let f : S −→ R !  ! f dm = sup h dm | h ∈ SF+ (S) | h(x)  f (x) for x ∈ S . S

S

ˆ be two measurable (3) Let (S, E, m) be a measure space and let f, g : S −→ R functions such that f (x) = g(x) a.e. " " If one of the integrals S f dm or S g dm exists, then both exist and are equal.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 583

Integration

583

Solution: Suppose initially that both f and g are non-negative. Define Δ(f, g) = {x ∈ S | f (x) = g(x)} and observe that m(Δ(f, g)) = 0. ˆ as Define h : S −→ R  ∞ if x ∈ Δ(f, g), h(x) = 0 if x ∈ Δ(f, g). Note that we have both f  g + h and g  f + h.

(8.28)

Let (hn ) be given by hn = n1Δ(f,g) for n ∈ N. " " the sequence of functions h dm = lim h dm = 0. Inequalities (8.28) We have n→∞ n S S " " " " " imply " f dm  S g dm and S g dm  S f dm, hence S f dm = S g dm. S If the non-negativity of f and g is dropped, the same result can be obtained by applying the previous argument to the positive and negative parts of these functions. (4) Let f be a non-decreasing sequence of non-negative functions, f = (fn ) such that limn→∞ fn (x) = f (x) a.e., where f is a measurable function. Prove that ! ! f dm = lim fn dm T

n→∞

T

for all measurable T . Solution: Let A = {x | limn→∞ fn (x) = f (x)}. We have A is null. If   fn (x) if x ∈ A, f (x) if x ∈ A, gn (x) = and g(x) = 0 if x ∈ A, 0 if x ∈ A, then !

!

!

gn dm =

fn dm + !T ∩A

T

=

0 dm !T ∩A

fn dm + T ∩A

! fn dm =

fn dm.

T ∩A

Thus, limn→∞ gn (x) = g(x) everywhere, so limn→∞ by the Monotone Convergence Theorem.

T

" T

gn dm =

" T

g dm

(5) Let (S, E, m) be a measure space and let f : S −→ R be " a Borel measurable function. Prove that if U ∈ E is a null set, then U f dm = 0. Hint: Prove the statement starting with simple measurable functions. (6) Let (S, E, m) be a measure space and let f = (fn ) be a sequence of funcˆ tions, where " n  1. Suppose that limn→∞ fn (x) = " fn : S −→ R0 for f (x) and" S f dm = "limn→∞ S fn dm < ∞. Prove that if U ∈ E, then limn→∞ U fn dm = U f dm.

May 2, 2018 11:28

584

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 584

Mathematical Analysis for Machine Learning and Data Mining

Solution: Let U ∈ E. We have: !

! f 1U dm = lim inf fn 1U dm n→∞ !  lim inf fn 1E dm

!

f dm = U

n→∞

(by Fatou’s Lemma (Theorem 8.12)) ! = lim inf fn dm. n→∞

U

" " Similarly,"we have U "f dm  lim"inf n→∞ U fn"dm. Since 1U" + 1U = 1, we " have U f dm = f dm − U f dm and U fn dm = fn dm − f dm, which implies U n !

!

!

f dm −

f dm =

f dm

!

! fn dm − fn dm  lim inf n→∞ U ! U ! fn dm. = f dm − lim sup

U

U

n→∞

U

Therefore, !

! n∞

hence

" U

!

fn dm 

lim sup U

f dm = limn→∞

f dm  lim inf U

" U

n→∞

fn dm, U

fn dm.

(7) Let (P, P(P), m) be a measure space defined on the set of positive natural numbers P, where m is the counting measure. Let f = (fn )n1 be the sequence of functions defined by fn = 1{n} . Prove that: (a) if g : P " −→ P is a function such that fn (x)  g(x) for every x ∈ P, then P g dm = ∞; (b) each function fn is measurable; " " (c) verify that limn→∞ pp fn dm = 1 and P limn→∞ fn dm = 0; why is the dominated convergence theorem not applicable? " Hint: We have limn→∞ fn = 0, so P limn→∞ fn dm = 0; on other " " hand, P fn dm = 1 for every n  1, hence limn→∞ pp fn dm = 1. (8) Prove that an absolutely continuous function on [a, b] is uniformly continuous on [a, b]. Also, prove that if f is a Lipschitz function on [a, b], then f is absolutely continuous on this interval. (9) Prove that if f, g : [a, b] −→ R are absolutely continuous on [a, b], then so are f + g and f g.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Integration

b3234-main

page 585

585

(10) Prove that if h is Riemann integrable on [a, b], then the function f defined as ! x f (x) =

h(x) dx a

for a  x  b is absolutely continuous on [a, b]. (11) Define the relation “∼” on the set of measures on a set S as m ∼ m if m  m and m  m. Prove that “∼” is an equivalence relation. ˆ 0 (12) Let (S, E, m) be a measure space and let U ∈ E. Define mU : E −→ R as mU (T ) = m(T ∩ U ) for every T ∈ E. Prove that mU is a measure on E. Furthermore, mT  mV if and only if m(T − V ) = 0. (13) Let m, m be two σ-finite measure on (S, E). If m  m and m  m, dm dm prove that dm  dm = 1 a.e. on S.

dm

= (14) Let m be a σ-finite measure on (S, E). Prove that m  |m| and d|m| 1 a.e. on S. (15) Let (N, P(N), m) be the measure space defined by m(A)" = |A| for every A ⊆ N. For a function f : N −→ R show that N f dm as   + (f (n))− . If this difference is not defined (because n∈N (f (n)) − n∈N both sums are ∞) the integral does not exist. (16) Let (fn ) be a sequence of functions in L2 (S, E, m) such that limn→∞ fn − f 2 = 0. Prove that for every function g ∈ L2 (S, E, m) we have limn→∞ (fn , g) = (f, g). (17) Let f : R>0 −→ R>0 be defined as f (x) = ex1√x for x > 0. Prove that f ∈ L1 (R>0 , B(R>0 ), mL ) but f ∈ L2 (R>0 , B(R>0 ), mL ). (18) Prove that if a, b  0, then (a + b)p  2p−1 (ap + bp ) for p  1. Use this fact to show that a sequence (fn ) in Lp (R>0 ) converges to f ∈ Lp (R>0 ) relative to the norm  · p if and only if limn→∞ fn p = f p . (19) Let (fn )1 be sequence in Lp (S, E, m) such that limn→∞ fn = f in the sense of the norm  · p . Prove that the sequence fn converges to f in measure. Solution: This fact follows from the Chebyshev’s Inequality applied to the function |fn − f |. (20) Let (fn )1 be a Cauchy sequence in Lp (S, E, m). Prove that (fn )1 is a Cauchy sequence in measure, that is, for every  > 0 we have limm,n→∞ m({x ∈ S | |fn (x) − fm (x)| > }) = 0. (21) Let S, T be two sets and let E be a subset of S × T . Prove that (1E )s = 1Es and (1E )t = 1E t for every s ∈ S and t ∈ T . (22) Let S, T be two sets and let f : S −→ T . Prove that for every subset V of T we have 1f −1 (V ) = 1V f .

May 2, 2018 11:28

586

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 586

Mathematical Analysis for Machine Learning and Data Mining

Let S be a set and let f : S −→ {0, 1} be a function. The set defined by f is Tf = {x ∈ S | f (x) = 1}. It is clear that f = 1Tf . If z : S −→ {0, 1} is the constant function defined by z(x) = 0, the set Tz is the empty set. The set of characteristic functions of the collection S of subsets of S is denoted by IS . (23) Let f, g ∈ IP(S) . Prove that: (a) if f g = z, then f + g ∈ IP(S) ; (b) 1S − f = 1S−Tf . (24) Prove that S is a semiring of sets if and only if the following conditions are satisfied by IS : (a) z ∈ IS ; (b) f, g ∈ IS implies f g ∈ IS ; (c) for every f, g ∈ IS there exists a finite collection {h1 , . . . , hp } ⊆ IS such that for every i, j, 1  i, j  p, i = j implies hi hj = z and f (IS − g) = h1 + · · · + hp . (25) Let f : S −→ R be a simple function on S such that Ran(f ) = {y1 , . . . , yn }. If T ⊂ S, and g(x) = f (x)IT (x) for x ∈ S, prove that Ran(g) = {y ∈ Ran(f ) | f −1 (y) ∩ T = ∅} ∪ {0}. Also, prove that if y ∈ Ran(g), then g −1 (y) = f −1 (y) ∩ T . (σ-algebras) on a set S such (26) Let (E0 , E1 , . . .) be a sequence of algebras  that En ⊆ En+1 for n ∈ N. Prove that n∈N En is an algebra (a σalgebra, respectively) on S. (27) Let S be an infinite set. (a) Prove that the collection of all subsets U of S such that U is finite or S − U is finite is an algebra but not a σ-algebra. (b) Prove that the collection E of all subsets of S defined by E = {U ∈ P(S) | U is countable or S − U is countable} is a σ-algebra on S. (c) Suppose that S is uncountable. Prove that there exists a subset W of S that does not belong to the σ-algebra E defined above. Prove that the σ-algebra E is not closed with respect to arbitrary unions. (28) Prove that any σ-algebra E contains the empty set; further, prove that if s = (S0 , S1 , . . .) is a sequence of sets of E, then both lim inf s and lim sup s belong to E. (29) Let A ⊆ N be a set. Define the function dA : N1 −→ R as dA (n) =

|A ∩ {1, . . . , n}| n

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 587

587

for n  1. The upper asymptotic density of A is d(A) = lim sup dA (n), while the lower asymptotic density of A is d(A) = lim inf dA (n). The set A has asymptotic density d(A) if d(A) = d(A) = d(A). (a) Prove that if A has asymptotic density d(A), then N −A has asymptotic density 1 − d(A). (b) Compute the asymptotic density of the set A3 = {3k | k ∈ N}. (c) Prove that for any finite subset A of N, d(A) = 0. (d) Compute the asymptotic density of the set Pk = {nk | n ∈ N}. 2m , . . . , 22m+1 − 1} be the set of numbers whose (e) Let A = ∞ m=0 {2 binary expression contains a number of odd digits. Prove that A does not have an asymptotic density. (f) Let C be a subset of N and let B = {bp | p ∈ N} be a set such that  3p if p ∈ C, bp = 3p + 1 otherwise. Prove that B has asymptotic density equal to 1/3 regardless of whether C has an asymptotic density and that A3 ∩ B = {3m | m ∈ C}. (g) Prove that the collection of sets that have an asymptotic density is not an algebra. Hint for Part (e): Note that each set of the form {22m , . . . , 22m+1 − 1} contains 22m elements and that A ∩ {1, . . . n} =

p 

{22m , . . . , 22m+1 − 1},

m=0

where p is the largest number such that 22p+1 − 1  n. Thus, p = p 2p+2 2m  and the set A ∩ {1, . . . n} contains = 2 3 −1  12 log 2 n+1 m=0 2 2 numbers. If n has the form n = 22m+1 , then p = m and the set A contains 2m+2 2 −1 elements. Then 3 lim

m→∞

2 22m+2 − 1 = . 3 · 22m+1 3

On the other hand, if n = 22m+2 , then 22p+1  22m+2 + 1, so again p = m and 1 22m+2 − 1 = . lim m→∞ 3 · 2m+2 3 We conclude that dA = dA , so A has no asymptotic density. (30) Let x, y, a1 , b1 , . . . , an , bn be n real numbers such that x  yand ai  bi n for 1  i  n. nProve, by induction on n, that if [x, y] ⊆ i=1 (ai , bi ), then y − x  i=1 (bi − ai ).

May 2, 2018 11:28

588

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 588

Mathematical Analysis for Machine Learning and Data Mining

(31) Let I be a bounded interval of R. Prove that if K is a compact subset of R such that K ⊆ I, then mL (I) = mL (K) + mL (I − K), where mL is the Lebesgue measure. (32) Let (S, E) be a measurable space and let m : E −→ rr ˆ 0 be a measure. Prove that the dm defined by dm (U, V ) = m(U ⊕ V ) is a semi-metric on E. spaces such that (33) Let {(Si , Ei , mi ) | i ∈ I} be a collection of measure  Si S| = ∅ if i = j for i, j ∈ I. Define the triplet ( i∈I Si , E, m), where  



Si , U ∩ Si ∈ E for i ∈ I , E = U U ⊆

i∈I

 ˆ 0 is given by m(U ) = i∈I mi (U ∩ Si ) for U ∈ E. and m : E −→ R  Prove that ( i∈I Si , E, m) is a measure space and that m(U ) is finite if and only if there exists a countable subset J of I such that if j ∈ J, then μj is finite and μi = 0 if i ∈ I − J. (34) Prove that if U is a subset of Rn that is Lebesgue measurable, then mL (U ) = inf{mL (W ) | U ⊆ W, W is open in Rn }.

(8.29)

Solution: The monotonicity of mL implies mL (U )  inf{mL (W ) | U ⊆ W, W is open }. If mL (U ) = ∞ the reverse inequality is obvious. Suppose, therefore, that mL (U ) < ∞. We have:  mL (U ) = inf



vol(Ij ) | j ∈ J, U ⊆



 Ij

,

j∈J

so there exists  a collection of n-dimensional open intervals {Ij | j ∈ N} such that j∈N mL (Ij ) < mL (U ) + . Thus, mL (U ) = inf{mL (W ) | U ⊆ W, W is open in Rn }. (35) Let U ⊆ Rn be a Lebesgue measurable set. For every  > 0 there exists an open set V in Rn such that mL (V − U ) < . Solution: If U is a subset of Rn that is Lebesgue measurable, then mL (U ) = inf{mL (W ) | U ⊆ W, W is open }. Therefore, for every  > 0 there exists a open subset W of Rn such that mL (U )  mL (W )  mL (U ) + , which implies mL (W − V )  mL (W ) − mL (V ) < .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 589

589

If mL (U ) = ∞, define Uk = U ∩ [−k, k] for k ∈ snn and k  1. We have mL (Uk )  mL ([−k, k]) = (2k)n < ∞ for k  1. Thus, by the first part of the proof, for every k  1 there exists an open  set Wk such that Uk ⊆  Wk and mL (Wk − Uk ) < 2k . Define W = k1 Wk . Note that U = k1  Wk ⊂ W and W− U ⊆ k1 (Wk − Uk ). Then, mL (W − U )  mL ( k1 Wk − Uk )  k1 2k = . (36) Prove that a subset U of Rn is Lebesgue measurable if and only if for every  > 0 there exist an open set L and a closed set H such that H ⊆ U ⊆ L and mL (L − H) < . Let C be a collection of sets. A function f : C −→ R is said to be additive  if for every finite collection of disjoint sets D = {D1 , . . . , Dn } ⊆ C such that D ∈ C we have n    f (Di ). f D = i=1

(37) Let f : S −→ R be an additive set function on an algebra of sets S. Prove that the following properties are equivalent: (a) the function f is countably additive;  (b) if Un ∈ S and Un+1 ⊆ Un for n  1, and ∞ n=1 Un = ∅, then limn→∞ f (Un ) = 0; ∞ (c) if Vn ∈ S and Vn ⊆ ∞Vn+1 for n  1, and n=1 Vn ∈ C, then limn→∞ f (Un ) = f i=1 Vi . (38) Let (U1 , . . . , Un , . . .) be an increasing or a decreasing sequence of sets in a semiring such that limn−→∞ Un = U ∈ S. Prove that if f : S −→ R0 is a countably additive function on S, then limn→∞ f (Un ) = f (U ) (39) Prove that the collection Sr = {∅} ∪ {(a, b] | a, b ∈ Q, 0  a < b  1} is a semiring and that the function f : Sr −→ R given by f ((a, b]) = b − a is an additive function but not a countably additive one. (40) Let (S, E, m) be a measure space and let f = (f0 , . . . , fn , . . .) be a seˆ for n ∈ N. If there exists quence of functions such that fn : S −→ R ˆ 0 such that fn  g for a non-negative integrable function g : S −→ R n ∈ N, prove that !

! lim sup fn dm  lim sup S

n→∞

n→∞

fn dm. S

This statement is known as the reverse Fatou Lemma. Hint: Apply Fatou’s Lemma to the sequence (g − f0 , . . . , g − fn , . . .).

May 2, 2018 11:28

590

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 590

Mathematical Analysis for Machine Learning and Data Mining

(41) Let (S, E, m) be a measure space and let f : S −→ R be a measurable function. Prove that inf f  ess inf f  ess sup f  sup f. (42) Let S = ([0, 1], B([0, 1]), m) and T = ([0, 1], B([0, 1]), m ) be two measure spaces, where m is the Lebesgue measure and m is the counting measure. Let D = {(x, x) | x ∈ [0, 1]}. Prove that: " (a) (m × m )(D) = S×S 1D d(m × m ) = ∞; " " (b) [0,1] ( [0,1] 1D dm)dm = 0; " " (c) [0,1] ( [0,1] 1D dm )dm = 1. (43) Let (S, E, m) be a measure space. Prove that a sequence of measurable functions (f1 , f2 , . . .) converges to a measurable function f if and only if for each  > 0 and each η > 0 there exists n0 ∈ N such that n  n0 implies m({x ∈ S | |fn (x) − f (x)|  η}) < . (44) Let Y be a random variable that takes values in [0, 1] such that E(Y ) = . m. Prove that for any a ∈ (0, 1) we have P (Y > 1 − a)  m−1+a a Hint: Apply Markov’s inequality to the random variable X = 1 − Y . (45) Let X1 , . . . , Xn be a sequence of independent and identically distributed random variables with E(Zi ) = m and var(Zi )  1 for 1  i  n. Prove that for any δ ∈ (0, 1),

#  

n

 1

1 Xi − m  P

> 1 − δ.

n δn i=1 Solution: Let Y be the random variable Y = E(Y ) = m and var(Y ) = n1 var(Xi ).

1 n

n i=1

Xi . We have

By applying Chebyshev’s Inequality (inequality (8.6)) to the variable i) Y we have P (|Y − m| > a)  var(X or P (|Y − m|  a) > 1 − na1 2 . a2 $ 1 If 1 − na1 2 = 1 − δ, then a = δn and the inequality follows immediately. (46) Let X be a non-negative random variable on the probability space (Ω, E, P ) and let G be a σ-algebra such that G ⊆ E. Prove that E(X|G)  0 a.e. 1 Solution: " < − k } ∈ G. " Let k  1 and let1 Ak = {ω ∈ Ω | E(X|G) Note that A E(X|G) dP < − k P (Ak ). On other hand, A X dP  0. k

k

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Integration

page 591

591

" " Since A X dP = A E(X|G) dP , we have P (Ak ) = 0 for k  1, which k k implies P (E(X|G) < 0) = P (bigcupk1 Ak ) = 0. Thus, E(X|G)  0 a.e. (47) Let (S, O) be a normal topological space and let (S, B(S), m) be the corresponding Borel measurable space, where m is a regular measure. Prove that if f : S −→ C is a complex-valued measurable function on (S, B(S), m) and δ > 0, then there exists a continuous complex-valued function g : S −→ C such that m({x ∈ S | f (x) = g(x)}) < δ. Moreover, prove that it is possible to choose g such that supx∈S |g(x)|  supx∈S |f (x)|. Solution: Assume initially that f is real valued and f (x) ∈ [0, 1) for x ∈ S. Define the sequence of simple, increasing non-negative functions (hn )n0 as h0 (x) = 0 and ⎧ ⎨k − 1 hn (x) = 2n ⎩n

k−1 k  f (x) < n and 1  k  n2n , 2n 2 when f (x)  n. if

for n  1. Let (fn )n1 be the sequence defined by fn (x) = hn (x) − hn−1 (x) for n  1 and x ∈ [0, 1]. Note thata function fn has only two possible values, 0 and 2−n . We have f = ∞ n=1 fn . Indeed, as shown next k−1 2n−1

2k−2 2n

k 2n−1

2k−1 2n

2k 2n

k k−1 if 2k−1 n−1  f (x) < 2n−1 we have hn−1 (x) = 2n−1 . One of the following 2k−2 2k−1 2k  f (x) < 2n . two cases may occur: either 2n  f (x) < 2n , or 2k−1 2n 2k−2 In the first case hn (x) = 2n and hn (x) = hn−1 (x); in the second case, and hn (x) − hn−1 (x) = 2−n . hn (x) = 2k−1 2n

Let An = {x ∈ S | fn (x) = 0}. Denote by Cn a closed subset of An and by Vn an open superset of An such that m(Vn − Cn ) < 2δn . Since (S, O) is normal there exists a continuous function g : S −→ [0, 1] such that g(x) = 1 on Cn and g(x) = 0 for x ∈ S − Vn .  −n Let g = ∞ gn . By Weierstrass n=1 2  M -test, g is a continuous map of S into [0, 1]. Note that the set T = n1 (Vn − Cn ) we have m(T ) < δ and if x ∈ T , then f (x) = g(x). Indeed, observe that for each n  1, x ∈ Cn , or x ∈ Vn . If x ∈ Cn ⊆ An , then 2−n gn (x) = 2−n = fn (x); if x ∈ Vn , 2−n gn (x) = 0 = fn (x) because x ∈ An . These considerations prove the existence of g when f is a real-valued function with f (x) ∈ [0, 1).

May 2, 2018 11:28

592

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 592

Mathematical Analysis for Machine Learning and Data Mining

Suppose now that f is a complex-valued measurable function. If f is unbounded we can write f = f1 + f2 , where f1 = f 1{x∈S | |f (x)| M.

Then, g1 is continuous, |g1 (x)|  M and f (x) = g(x) implies g(x)  M ; hence g1 (x) = g(x) = f (x). Therefore, m({x | f (x) = g1 (x)})  m({x | f (x) = g(x)}) < δ. (48) Let (S, O) be a normal topological space and let (S, B(S), m) be the corresponding Borel measurable space, where m is a regular measure. Prove that if f : S −→ C is a complex-valued measurable function on (S, B(S), m) there is a sequence of continuous complex-valued functions converging almost everywhere to f , with |fn (x)|  sup{f (x) | x ∈ S}. Solution: By Supplement 47 there is a continuous function fn such that |fn |  M = sup |f | and m({x ∈ S | fn (x) = f (x)}) < 2−n . If An = {x | fn (x) = f (x)}, and A = lim sup An , then m(A) = 0 by Borel-Cantelli Lemma (Theorem 7.33). The statement follows from the fact that x ∈ A implies fn (x) = f (x) for sufficiently large n. (49) Let (S, O) be a normal topological space, (S, B(S), m) be the corresponding Borel measurable space, where m is a regular measure, and let f : S −→ C be a measurable function. Prove that for  > 0 there exists a closed subset C of S, a continuous function g : S −→ C such that g(x) = f (x) for x ∈ C. Thus, the restriction of f to C is continuous. If m(A) = sup{m(K) | K is compact , K ⊆ A} for each A ∈ B(S), then C may be assumed to be compact. The statement contained by this supplement is known as Lusin’s Theorem. Solution: By Supplement 47 there exists a continuous g such that m({x ∈ S | f (x) = g(x)}) < 2 . Since m is regular, there exists a closed set C, C ⊆ {x ∈ S | f (x) = g(x)} with m(C)  m({x ∈ S | f (x) = g(x)}) − 2 , hence C has the desired properties. For the second part of the argument replace closedness of C with compactness.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Integration

9in x 6in

b3234-main

page 593

593

Bibliographical Comments Our presentation of product measure space follows [32]. Supplement 42 originates in [57]. The proof of Theorem 8.42 follows [113]. The sequence of Supplements 47-49 that culminates with Lusin’s Theorem follow [3].

PART IV

Functional Analysis and Convexity

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 597

Chapter 9

Banach Spaces

9.1

Introduction

Definition 9.1. A Banach1 space is a normed linear space (L,  · ) that is complete when regarded as a metric space through the metric induced by the norm  · . Results presented for complete metric spaces in Chapter 4 can be applied to Banach spaces, where the metric is induced by the norm defined on this space. For example, Theorem 5.72 stipulates that a normed space is a Banach space if and only if every absolutely convergent series is convergent. 9.2

Banach Spaces — Examples

A typical example of a Banach space is Rn equipped with the Euclidean metric (as we saw in Example 5.12). Example 9.1. Every finite-dimensional normed space is a Banach space, as Theorem 6.33 implies. Thus, Rn equipped with the Euclidean form is a Banach space. Example 9.2. The space C[a, b] equipped with the norm ·∞ is a Banach space, where f ∞ = sup{|f (x)| | x ∈ [a, b]} for f ∈ C[a, b]. Let (fn ) be a Cauchy sequence of functions in C[a, b]. For each x ∈ [a, b] the sequence (fn (x)) is a Cauchy sequence in R. Since R is complete, the sequence (fn (x)) converges to a real number denoted by f (x). We need to 1 This type of spaces that is fundamental in functional analysis are named after Stefan Banach (March 30, 1892–August 31, 1945), a Polish mathematician who is one of the founders of modern functional analysis

597

May 2, 2018 11:28

598

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 598

Mathematical Analysis for Machine Learning and Data Mining

show that the function f defined in this manner is continuous on [a, b] and that limn→∞ fn − f ∞ = 0. For x, z ∈ [a, b] we have: |f (x) − f (z)|  |f (x) − fn (x)| + |fn (x) − fn (z)| + |fn (z) − f (z)|.

(9.1)

Since (fn ) is a Cauchy sequence, for every > 0 there exists a number n0 such that fn − fm ∞ < 3 if m, n  n0 , which yields |fn (x) − fm (x)|  3 for each x ∈ [a, b]. This implies |fn (x) − f (x)|  3 for all x ∈ [a, b] when m → ∞. Similarly, |fn (z) − f (z)| ≤ 3 , which also shows that limn→∞ || fn − f || = 0. Since the functions fn are continuous, there exists δ > 0 such that |x−z| < δ implies |fn (x)−fn (z)| < 3 . By inequality (9.1), |f (x)−f (z)| < whenever |x − z| < δ, so f ∈ C[a, b]. Example 9.3. Let BF(X, T ) be the set of bounded functions defined on a set X and ranging over a normed space (T,  · ). The set of all bounded functions from X to T is denoted by BF(X, T ). BF(X, T ) is a linear space because sup{(f + g)(x) | x ∈ X}  sup{f (x) + g(x) | x ∈ M }  sup{f (x) | x ∈ M } + sup{g(x) | x ∈ M }. Note that sup{f (x) | x ∈ X} is a norm on BF(X, T ). If (T,  · ) is a Banach space, then BF(X, T ) is a Banach space. Indeed, let (fn ) be a sequence of functions in BF(X, T ). For every > 0 there exists n such that m, n  n imply fm − fn  < . Therefore, for each x ∈ X, m, n  n imply fm (x) − fn (x) < . Since (T,  · ) is a Banach space the sequence (fn (x)) converges to an element g(x) ∈ T , which implies fm (x) − g(x) < for m  n and x ∈ X. Thus, g  fm  + , so g is bounded, that is, g ∈ BF(X, T ). Since fm (x) − g(x) < for n  n it follows that the sequence (fn ) converges to g, so BF(X, T ) is a Banach space. Example 9.4. For a measure space (S, E, m), the space Lp (S, E, m) is complete by Riesz-Fischer Theorem (Theorem 8.95) and, therefore, it is a Banach space relative to the norm ·p . Important special cases are the spaces of random variables of the form Lp (Ω, E, P ) for p = 1 and p = 2. Note that for a random variable X ∈ L1 (Ω, E, P ) we have: 7 X1 = |X| dP = E(|X|). Ω

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Banach Spaces

page 599

599

Example 9.5. Note that the completeness of spaces p (R) follows from the completeness of spaces Lp (S, E, m). Indeed, let S = R, E = P(R) and let ˆ 0 be the measure defined by m(U ) = |U ∩ N|. Then, a m : P(R) −→ R sequence (xn ) belongs to p (R) if and only the function f : R −→ R defined by  |xi |p if t = i ∈ N, f (t) = 0 otherwise is integrable. Therefore, the linear spaces p (R) are Banach spaces. In a similar manner, the set of sequences x ∈ Seq∞ (C) such that xp is finite is a complex normed linear space relative to the norm νp (x) =  p p n |xn | . We will use the same notation, (R) to designate this space, if the type of the space is clear from context. Theorem 9.1. Let (L,  · ) be a normed C-linear space and let (T,  · ) be a complex Banach space. The set of linear operators L(L, T ) is a Banach space. Proof. We saw that L(L, T ) is a normed linear space. We need to show only its completeness. Let (hn ) be a Cauchy sequence in L(L, T ). Since hn (x) − hm (x) = (hn − hm )(x)  hn − hm  x, we have a Cauchy sequence (hn (x)) in (T,  · ). Since T is a Banach space we can define the operator h : S −→ T as h(x) = limn→∞ hn (x). The linearity of h follows immediately from the equality hn (ax + by) = ahn (x) + bhn (y) for n ∈ N and a, b ∈ C. Since (hn ) is a Cauchy sequences the norms of the operators hn are bounded, that is hn   M . Then, hn (x)  M x for all x ∈ S, which implies h(x)  M x. Since (hn ) is a Cauchy sequence, for every positive there exists n such that m, n  n implies hn − hm  < . Thus, if x = 1, we have hn (x) − hm (x) < when m, n  n . This means that hn (x) − h(x)  , so hn − h  when  n  n . Theorem 9.2. Let h : L −→ L be a bounded linear operator on the Banach space L. If h < 1, the operator 1L − h is invertible and (1L − h)−1 = ∞ m m=0 h .

May 2, 2018 11:28

600

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 600

Mathematical Analysis for Machine Learning and Data Mining

Proof. Define the sequence of linear operators (gp ) as gp = p ∈ N. Note that     p p    k  gp − gq  =  h   hk  k=q+1  k=q+1 

∞  k=q

hk =

p m=0

hm for

hq . 1 − h

Therefore, (gp ) is Cauchy sequence in B(L, L). Since B(L, L) is complete, there exists a bounded linear operator g : L −→ L such that limp→∞ gp = g ∞ m (which means that g = m=0 h ). We have (1L − h)gp = gp − hgp =  p p+1 m m p+1 . Taking p → ∞ we obtain (1L −h)g = g. m=0 h − m=1 h = 1L −h In the similar manner, we can show that g(1L −h) = 1L , hence (1L −h)−1 = g.  Theorem 9.3. If 1  p < ∞, the space p (R) is separable. In contrast, the space ∞ (R) is not separable. Proof. The separability of p (R) with 1  p < ∞ can be obtained by noting that the set of infinite sequences having all but a finite number of components equal to 0 and remaining non-zero components in Q is countable and dense in p (R). On the other hand, let S ⊆ Seq∞ (R) be the set of infinite sequences whose components equal either 0 or 1. This set is clearly uncountable. If x, y ∈ S, it is clear that d∞ (x, y) = 1 if x = y. Suppose that T is a dense subset of ∞ (R). For every x ∈ S there exists ˜ ∈ T such that x − x ˜ ∞ < 13 . Therefore, x = y implies x ˜ = y ˜ , so the x mapping f : S −→ T is injective. This implies that T is uncountable, so 

∞ (R) is not separable. Theorem 9.4. Let {B[xn , rn ] | n ∈ N} be a sequence of closed spheres in a Banach space S such that B[xn , rn ] ⊇ B[xn+1 , rn+1 ] for n ∈ N. If  limn→∞ rn = 0, then there exists z ∈ S such that n∈N B[xn , rn ] = {z}. Proof. Note that if m > n we have xm ∈ B[xn , rn ], so xm − xn   rn . Therefore, (xn ) is a Cauchy sequence and, since S is a Banach space, there exists z ∈ S such that limn→∞ xn = z. Since xm ∈ B[xn , rn ] for m  n and  B[xn , rn ] is a closed set it follows that z ∈ B[xn , rn ], so z ∈ n∈N B[xn , rn ].  Conversely, if y ∈ n∈N B[xn , rn ] we have z−y  z−xn +zn −y   2rn for n ∈ N which implies z − y = 0, hence y = z.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Banach Spaces

page 601

601

Lemma 9.1. Let h : S −→ T be a bounded linear operator where (S,  · ) and (T,  · ) are real normed linear spaces. Then, for any x ∈ S and r > 0 we have sup x ∈B[x,r]

h(x )  hr.

Proof. Observe that x ∈ B[x, r] if and only if x − x  r which is equivalent to x − x ∈ B[0S , r] and x − x ∈ B(0S , r). For z = x − x or z = x − x we have z  r and 1 (h(x + z) + h(x − z)) 2  h(z),

max{h(x + z), h(x − z)} 

by the triangle inequality. By taking the supremum over z ∈ B[0S , r] we obtain the desired inequality.  Recall that we introduced the notion of precompact set in a metric space in Definition 5.21. Theorem 9.5. Let L be a Banach space. A subset U of L is relatively compact if and only if it is precompact. Proof. Suppose that U is relatively compact, and for some positive r there is no finite r-net. For x1 ∈ U there exists x2 such that x1 − x2   r because, otherwise {x1 } would be an r-net. There is x3 ∈ U with x1 − x3   r and x2 − x3   r because, otherwise, {x1 , x2 } would be an r-net, etc. Thus, we obtain a sequence (xn ) with xp − xq   r for all distinct p, q. It is immediate that (xn ) contains no convergent subsequence, which contradicts the relative compactness of U . This implies that there exists a finite r-net for U . Conversely, suppose now that there exists a finite r-net for U for every positive r. Let {an1 , an2 , . . . , anpn } be an n1 -net. p1 B (ak , 1), at Let (um ) be an arbitrary sequence in U . Since U ⊆ k=1 least one of the spheres B (ak , 1) contains an infinite subsequence of (um ). in a sphere B (ak , 1). Let (u1m ) be an infinite subsequence  contained 1 At least one of the spheres B ak , 2 contains an infinite subsequence of (u1m ). Let (u2m ) be an infinite subsequence of (u1m ) contained in one of the spheres B ak , 12 . This process produces an infinite sequence of sequences (uqm ) where 1 q (uq+1 m ) is a subsequence of (um ) contained in a sphere B(ai , q+1 ).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 602

Mathematical Analysis for Machine Learning and Data Mining

602

1 Consider the diagonal sequence (um m ). For every positive r let hr = r ! the terms beyond the term of rank hr are located inside a sphere of radius 1 1 m l r , hence um − ul  < r . We have extracted a Cauchy sequence that must converge since S is a Banach space. 

This theorem offers an alternative proof of compactness for subsets of Banach spaces because it suffices to show the existence of an r-net for such sets. In Banach spaces the parallelogram equality (see Theorem 2.27) is a characteristic property of norms generated by the inner products, as we show next. Theorem 9.6. Let (L,  · ) be a Banach space. The norm  ·  is induced by an inner product if and only if x2 + y2 =

1 x + y2 + x − y2 2

for x, y ∈ L. Proof. The necessity of the condition was already shown in Theorem 2.27. To prove that this inequality is sufficient suppose initially that L is a C-normed linear space. We will show that the mapping (·, ·) : L −→ C as (x, y) =

1 x + y2 − x − y2 + ix + iy2 − ix − iy2 4

for x, y ∈ L is an inner product. It is immediate that (y, x) = (x, y) for x, y ∈ L, so condition (i) of Definition 2.20 is satisfied. Note that the real part of (x, y) is (x, y) =

1 x + y2 − x − y2 . 4

Therefore, i (ix + y2 + ix − y2 ) 4 1 = (x − iy2 − x + y2 ). 4

i(ix, y) =

Therefore, we have (x, y) = (x, y) − i(ix, y).

(9.2)

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Banach Spaces

page 603

603

This allows us to write (x + z, y) + (x − z, y) 1 = (x + z + y2 − x + z − y2 + x − z + y2 − x − z − y2 ) 4 1 = (x + z + y2 − x + z − y2 + x − z + y2 − x − z − y2 ) 4 = (x + z, y) + (x − z, y) = (2x, y). Replacing x by 12 (x + z) and z by 12 (x − z) yields (x, y) + (z, y) = (x + z, y). Further, by substituting ix for x and iz by z, we have (ix, y) + (iz, y) = (i(x + z), y), which, in turn yields (x, y) + (z, y) = (x + z, y) taking into account equality (9.2). It is easy to see that for each rational number q we have (qx, y) = q(x, y). Since the inner product is continuous, it follows that (ax, y) = a(x, y). Also, it is easy to verify that (ix, y) = i(x, y), so the third condition of Definition 2.20 is satisfied. A direct application of the definition implies (x, x) = x2 , so the last two condition of Definition 2.20 are also satisfied. We leave to the reader the simpler case of real Banach spaces.  9.3

Linear Operators on Banach Spaces

Theorem 9.7. (The Banach-Steinhaus or the Uniform Boundedness Theorem) Let F be a family of bounded linear operators from a Banach space S to a normed linear space T . If F is pointwise bounded (that is, suph∈F h(x) < ∞ for all x ∈ S), then F is norm-bounded, that is, suph∈F h < ∞. Proof. Suppose that suph∈F h = ∞ and let (hn ) be a sequence of operators in F such that hn   4n . Let x0 = 0S and define inductively xn using Lemma 9.1 by choosing xn ∈ B[xn−1 , 3−n ]. Then, by Lemma 9.1, hn (xn )  hn  · 3−n 

2 hn  · 3n . 3

May 2, 2018 11:28

604

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 604

Mathematical Analysis for Machine Learning and Data Mining

We claim that (xn ) is a Cauchy sequence. Indeed, since xn − xp  

n−1 

xk+1 − xk 

k=p



n−1 

3−(k+1) =

k=p

3 −(p+1) (3 − 3−(n+1) ) 2

1 it suffices to take n ≥ p > log3 2 to obtain xn − xp  < . Therefore, (xn ) is convergent to some x ∈ S. It is easy to see that x − xn   12 3−n , so  n 1 1 4 hn (x)  3−n hn   6 6 3

which implies limn→∞ hn (x) = ∞.



The notion of open function was introduced in Definition 4.42. Theorem 9.8. (Open Mapping Theorem) Let S and T be Banach spaces. If h : S −→ T is a surjective bounded linear operator, then h is an open function. Proof. Define the sequence of sets (Tn ) by Tn = nK(h(B(0, 1))) for  n ∈ N. All sets Tn are closed and the surjectivity of h implies T = n∈N Tn . There exists Tn such I(Tn ) is non-empty by Corollary 5.9. The homogeneity of linear operator h implies that K(B(0S , 1)) has a non-empty interior, that is, there exists y0 ∈ T and δ > 0 such that B(y0 , 4δ) ⊆ K(h(B(0S , 1))). By symmetry, −y0 ∈ K(h(B(0S , 1))). Note that if z ∈ B(0T , 2δ), then u = 2z + y0 belongs to B(0T , 4δ) because u − y0   2z < 4δ. Since z is the midpoint of the segment [y0 , u] and K(h(B(0S , 1))) is convex (by Theorem 6.37), it follows that B(0T , 2δ) ⊆ K(h(B(0S , 1))). We have B(0T , 2δr) ⊆ K(h(B(0S , r))) for every r > 0. To prove that for an open set U in S the set h(U ) is open in T it suffices to prove that B(0T , δ) ⊆ h(B(0S , 1)), or equivalently, that the equation h(x) = y has a solution x in B(0S , 1) when y ∈ B(0T , δ). Let y ∈ B(0T , δ). There exists x1 such that xS < 12 and h(x1 ) − yT < 2δ . Similarly, there exists x2 such that x2  < 212 and y − h(x1 ) − h(x2 ) < 2δ2 , etc. In general, there exists xn such that x < 21n and   y − nk=1h(xk )< 2δk . Therefore, the series xn is absolutely convergent    in S and  xn  < 1, hence it converges to some x with x < 1 and y − h(x) = 0. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Banach Spaces

b3234-main

page 605

605

Corollary 9.1. Let S and T be Banach spaces. If h : S −→ T is a continuous linear bijection, then h has a continuous linear inverse h−1 . Proof. The linearity of g = h−1 is immediate. By the Open Mapping Theorem, h(U ) is an open set in T and U = g(h(U )) means that g −1 (U ) = h(U ), which means that g is continuous.  If S and T be Banach spaces their product S × T is equipped with the norm defined by (x, y) = x + y. The set of continuous operators between the Banach spaces S and T , h : S −→ T is denoted by B(S, T ). Theorem 9.9. (Closed Graph Theorem) Let S and T be real Banach spaces. For a linear operator h : S −→ T we have h ∈ B(S, T ) if and only if its graph γh = {(x, y) ∈ S × T | y = h(x)} is a closed set in S × T . Proof. If h ∈ B(S, T ), it is immediate that γh is closed. Conversely, suppose that the set γh is closed. Then γh is a Banach space with the norm inherited from S × T , the mapping f : S × T −→ S defined by f (x, h(x)) = x is a continuous linear bijective operator. Therefore, by Corollary 9.1, the mapping f −1 : S −→ S ×T defined by f −1 (x) = (x, h(x)) is continuous, which implies the continuity of h because h = pf −1 , where p : S −→ T −→ T is the projection p(u, v) = v for (u, v) ∈ S × T (which is continuous by Theorem 4.99).  The next two lemmas establish preliminary facts needed in the proof of a very important result known as Lyusternik’s Theorem. We follow the outline of the proof presented in [133, 85]. Lemma 9.2. Let X and Y be real Banach spaces, f : X −→ Y be a mapping, Z = f −1 (0Y ), and x0 ∈ Z. If f is Fr´echet differentiable in a neighborhood V of x0 , (Df ) is continuous at x0 and (Df )(x0 ) is surjective, then {x ∈ S | (Df )(x0 )(x) = 0Y } ⊆ T (Z, x0 ), where T (Z, x0 ) is the contingent cone of Z at x0 . Proof. Since the linear mapping (Df )(x0 ) is continuous and surjective, by the Open Mapping Theorem (Theorem 9.8), the mapping (Df )(x0 ) is

May 2, 2018 11:28

606

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 606

Mathematical Analysis for Machine Learning and Data Mining

open. Thus, the image of the open ball B(0X , 1) is open, hence there exists some r such that B(0Y , r) ⊆ (Df )(x0 )(B(0X , 1)). The continuity of (Df )(x0 ) implies that there exists ρ0 = sup{r | r > 0, B(0Y , ρ0 ) ⊆ (Df )(x0 )(B(0X , 1))}. Let ∈ (0, ρ0 /2). Since (Df ) is continuous at x0 there is δ > 0 such x) − (Df )(x0 )  . that x ˜ ∈ B(x0 , 2δ) implies (Df )(˜ Let u, v ∈ B(x0 , 2δ). By Theorem 6.51 there exists a continuous linear functional : Y −→ R such that   = 1 and

(f (u) − f (v) − (Df )(x0 )(u − v)) = f (u) − f (v) − (Df )(x0 )(u − v). Let φ : [0, 1] −→ R be the differentiable function defined as φ(t) = ((Df )((1 − t)u + tv)(u − v)) for t ∈ [0, 1]. We have φ (t) = φ(t) and by the Mean Value Theorem there is a t ∈ (0, 1) such that φ(1) − φ(0) = φ (t). Thus, we can write: f (u) − f (v) − (Df )(x0 )(u − v = (f (u) − f (v) − (Df )(x0 )(u − v) = φ(1) − φ(0) = φ (t) = ((Df )((1 − t)v + tu))(u − v) − (Df )(x0 )(u − v))  (Df )((1 − t)v + tu)) − (Df )(x0 )u − v  u − v, which yields f (u) − f (v) − (Df )(x0 )(u − v  u − v, for v, u ∈ B(x0 , 2δ). Since ρ 0 < 12 there exists a > 1 such that a( 12 + ρ 0 ). Let x ∈ X such that (Df )(x0 )(x) = 0Y . If x0 = 0X we have immediately {x ∈ S | (Df )(0X )(x) = 0Y } ⊆ T (Z, 0X ). δ Assume that x0 = 0X and let λ ∈ (0, x ). Define the sequences (rn )n1 and (un )n1 as r1 = 0X , (Df )(x0 )(un ) = h(x0 +λx+rn ) and rn+1 = rn −un for n  1. The sequences (rn ) and (un ) are well-defined since (Df )(x0 ) is surjective, and therefore, for a given rn there exists un such that (Df )(x0 )(un ) = h(x0 + λx + rn ).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Banach Spaces

Since B(0Y , r) ⊆ (Df )(x0 )(B(0X , 1))} for r = un  

page 607

607 ρ0 a ,

it follows that

a h(x0 + λx + rn ). ρ0

Denote d(λ) = f (x0 + λx) and q =

a . ρ0

Since λx  δ we have d(λ) = f (x0 + λx) − f (x0 ) − (Df )(x0 )(λx)  λx  δ. Since a > 1 we have q  1 − a2 < 12 . We prove by induction on n  1 that rn   ρa0 d(λ) 1−q 1−q , f (x0 + λx + rn )  d(λ)q n−1 , a d(λ)q n−1 . un   rho 0 n

The base step, n = 1 is immediate because r1  = 0, f (x0 +λr+r1 ) = d(λ) and a a u1   f (x0 + λx + r1 ) = d(λ). ρ0 ρ0 For the induction step assume that the above inequalities are satisfied for n. Then, we obtain rn+1  = rn − xn   rn  + un    1 − q n−1 a + qn−1  d(λ) ρ0 1−q a 1 − qn = , ρ0 1 − q which proves the first inequality. Note that λx + rn   λx + rn  a 1 − q n−1  δ + d(λ) ρ0 1−q n−1 a δ 1 − q δ+ ρ 1−q   0 q (1 − q n−1 < 2δ,  δ 1+ 1−q

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 608

Mathematical Analysis for Machine Learning and Data Mining

608

because we have

q 1−q

< 1 and 1 − q n−1 < 1. Therefore,

λx + rn − un   λx + rn+1  1 − qn a  δ + d(λ) ρ 1−q  0  q (1 − q n ) < 2δ.  δ 1+ 1−q This allows us to conclude that f (x0 + λx + rn+1  = f (x0 + λx + rn − un ) =  − (Df )(x0 )(−un ) − f (x0 + λx + rn ) + f (x0 + λx + rn − un ) a   − un   d(λ)q n−1 = d(λ)q n . ρ0 The last inequality implies a un+1   f (x0 + λx + rn+1 )  δq n , ρ0 which completes the inductive proof of the inequalities. From these inequalities we infer a a δ n−1 q = δq n un   d(λ)q n−1  ρ0 ρ0 for n  1. Since q < 12 , it follows that limn→∞ un = 0X . This allows us to write rn+k − rn  = rn − un−k+1 − un+k−2 − · · · − un − rn   un  + un+1  + · · · + un+k−1  a  d(λ)(q n−1 + q n + · · · + q n+k−2 ) ρ0 a = d(λ)q n−1 (1 + q + · · · + q k−1 ) ρ0 ad(λ 1 − qk a < dn−1 , = d(λ)q n−1 ρ0 1−q ρ0 (1 − q) for all n, k  1, hence (rn ) is a Cauchy sequence. The completeness of X means that there exists r(λ) ∈ X such that limn→∞ rn = r(λ). Applying the limit to equality (Df )(x0 )(un ) = h(x0 + λx + rn ) yields f (x0 + λx + r(λ)) = 0. Since a r(λ) 1  d(λ) λ λρ0 1−q f (x0 + λx) − f (x0 ) − λ(Df )(x0 )(x) a , = ρ0 (1 − q) q we have limλ↓0

r(λ) λ

= 0.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Banach Spaces

page 609

609

The final phase of the proof aims to show that x belongs to the contingency cone T (Z, x0 ), where Z = f −1 (0Y ). δ ) such that Consider the sequence (λn )n1 in the interval (0, x limn→∞ λn = 0. Define the sequences (μn )n1 and (xn )1 such that 1 > 0 and xn = x0 + λn + r(λn ) for n  1. λn

μn =

Since f (x0 + λx + r(λ)) = 0, we have xn ∈ Z for n  1. Moreover, we have lim xn = lim (x0 + λn x + r(λn )

n→∞

n→∞

= x0 + lim (λn x + r(λn )) = x0 , n→∞

and we may conclude that 1 (λn x + r(λn )) λn r(λn ) = x + lim = x, n→∞ λn

μn (xn − x0 ) = lim

lim

n→∞

n→inf ty

hence x ∈ T (Z, x0 ).



Lemma 9.3. Let X and Y be real Banach spaces, f : X −→ Y be a mapping that is Fr´echet differentiable at x0 , where x0 ∈ Z = f −1 (0Y ). We have T (Z, x0 ) ⊆ {x ∈ S | (Df )(x0 )(x) = 0Y }, where T (Z, x0 ) is the contingent cone of Z at x0 . Proof. Let y ∈ T (Z, x0 )−{0X }. By the definition of the contingent cone, there is a sequence (xn )n1 of elements of Z and a sequence of positive real numbers (λn )n1 with x0 = limn→∞ xn and y = limn→∞ such that yn = λn (xn − x0 ) for n  1. By the definition of Fr´echet derivative we have (Df )(x0 )(y) = (Df )(x0 ) = lim λn n→∞

#

$ lim λn (xn − x0 ) n→∞ # $ lim (Df )(x0 ) lim (xn − x0 )

n→∞

n→∞ 

= − lim λn (h(xn ) − h(x0 ) − h (x0 )(xn − x0 )) n→∞

= − lim yn  n→∞

h(xn ) − h(x0 ) − h (x0 )(xn − x0 ) = 0Y . xn − x0 



May 2, 2018 11:28

610

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 610

Mathematical Analysis for Machine Learning and Data Mining

Theorem 9.10. (Lyusternik’S Theorem) Let X and Y be real Banach spaces, f : X −→ Y be a mapping, Z = f −1 (0Y ), and x0 ∈ Z. If f is Fr´echet differentiable in a neighborhood V of x0 , (Df ) is continuous at x0 and (Df )(x0 ) is surjective, then {x ∈ S | (Df )(x0 )(x) = 0Y } = T (Z, x0 ), where T (Z, x0 ) is the contingent cone of Z at x0 . Proof. The hypothesis of this theorem (which is identical to the hypothesis of Lemma 9.2) implies the hypothesis of Lemma 9.3. Therefore, by the previous two lemmas we have {x ∈ S | (Df )(x0 )(x) = 0Y } = T (Z, x0 ).  9.4

Compact Operators

Definition 9.2. Let (S,  · ) and (T,  · ) be two Banach spaces. A linear operator h : S −→ T is compact if for every bounded subset U of S the closure set K({h(x) | x ∈ U }) is compact. In other words, h is a compact linear operator if the image {h(x) | x ∈ U } of every bounded subset U of S under h is relatively compact. The definition of compact linear operators is equivalent to the requirement that for every bounded sequence (xn ) in S, the sequence (h(xn )) in T has a convergent subsequence. Suppose that h is a compact linear operator and let (xn ) be a sequence in the bounded set U . Then, {xn | n ∈ N} is a bounded set, hence {h(xn ) | n ∈ N} is relatively compact, which means that (h(xn )) contains a convergent subsequence. Conversely, let U be a bounded subset of S and assume that the set h(U ) is relatively compact. Let (yn ) be a sequence in K(h(U )). Then, the sequence (yn ) is also bounded and for each yn there exists xn ∈ U such that yn − h(xn )  n1 , which implies that the sequence (h(xn )) is also bounded. By the assumption made above, there exists a convergent subsequence (h(xnk )) and limk→∞ ynk = limk→∞ h(xnk ). This shows that the arbitrary sequence (yn ) contains a convergent subsequence, so h(U ) is indeed relatively compact. Theorem 9.11. Each compact operator between Banach spaces is bounded. Proof. Let h : S −→ T be a compact linear operator between the Banach spaces S and T . Suppose that h is unbounded. Then, there exists a

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Banach Spaces

page 611

611

sequence (xn ) in S such that xn = 0L and h(xn )  nxn  for n ∈ N. Note that the sequence (zn ) in S defined as zn = x1n xn is bounded because zn  = for n ∈ N. Since h(zn )  n for n ∈ N it is clear that h(zn ) contains no convergent subsequence.  Corollary 9.2. Every compact operator between Banach spaces is continuous. Proof.

This follows from Theorems 9.11 and 6.25.



The set of compact operators between the Banach spaces (S,  · ) and (T,  · ) is denoted by COMP(S, T ). Theorem 9.12. The set COMP(S, T ) is a closed subspace of the linear space L(S, T ). Proof. Let (hn ) be a sequence of compact operators such that limn→∞ = h in L(S, T ) (in norm, of course). We prove that h is a compact operator by showing that for every bounded sequence (xm ) in S, h(xm ) contains a convergent subsequence. Consider a bounded sequence (xn ), that is, xn   M for n ∈ N and M  0. Since h0 is compact there is a subsequence (x0n ) of (xn ) such that the sequence (h0 (x0n )) converges. Since h1 is compact, there is a subsequence (x1n ) of (x0n ) such that sequence (h1 (x1n )) converges. Continuing in this manner, we obtain subsequences (xjn ) of (xn ) such that (hj (xjn ) is a convergent sequence in T for each j ∈ N and that (xjn ) is a subsequence of (xin ) for 0  i  j − 1. The “diagonal subsequence” (xnn ) is such that h(xnn ) converges. Since limj→∞ hj = h, for every positive there is p ∈ N such that . h − hp   3M Note that (hp (xnn )) is a Cauchy sequence as a subsequence of the convergent sequence (hp (xpn )). Hence, there exists n0 > p such that m, n  n0 imply hp (xnn ) − hp (xmm ) < 3 . For m, n  n0 we have h(xnn ) − h(xmm ) = h(xnn ) − hp (xnn ) + hp (xnn ) − hp (xmm ) + hp (xmm ) − h(xmm )  (h − hp )(xnn ) + (hp (xnn ) − hp (xmm ) + (hp − h)(xmm )  h − hp xnn  + + hp − hxmm  3  ·M + + · M = . 3M 3 3M

May 2, 2018 11:28

612

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 612

Mathematical Analysis for Machine Learning and Data Mining

Thus, (h(xnn )) is a Cauchy sequence and therefore, it is convergent by the completeness of T . Since (xnn ) is a subsequence of the original sequence, it follows that h is a compact operator.  Theorem 9.13. Let h be a compact operator on a Banach space L and let g be a bounded operator on the same space. Then, both hg and gh are compact operators. Proof. Since h is compact for every bounded sequence (xn ) in L there exists a subsequence (xni ) such that (h(xni )) is a Cauchy sequence. Since g is bounded we have g(h(xni )) − g(h(xnj ))  g h(xni ) − h(xnj ). Therefore, (g(h(xni )) is also a Cauchy sequence, hence gh is compact. Since (xn ) is bounded and g is bounded, the sequence (g(xn )) is also bounded. The compactness of h implies the existence of a subsequence  (h(g(xni ))) that is Cauchy sequence, hence hg is also compact. 9.5

Duals of Normed Linear Spaces

Definition 9.3. Let L be a real normed linear space. The normed dual of L is the set of all bounded linear functionals defined on L. The normed dual space of L is denoted by L∗ . If φ ∈ L∗ its norm is φ = sup{|φ(x)| | x = 1}. As we observed in Corollary 6.8, a functional φ : L −→ R is continuous if and only if it is bounded. Theorem 9.14. Let L be a normed linear space and let φ : L −→ R be a linear functional. Then φ is continuous if and only if the Null(φ) is a closed subspace of S. Proof. The continuity of φ implies immediately that Null(φ) = φ−1 ({0}) is a closed subspace of L. Conversely, suppose that Null(φ) is a closed subspace of L and that φ is not continuous. By Theorem 6.25, φ is not bounded. This implies the existence of a sequence (xn ) in L such that xn   1 and limn→∞ φ(xn ) = ∞. Let z ∈ S such that z ∈ Null(φ). We have φ(z) = 0. If yn = z − φ(z) φ(xn ) xn , then φ(yn ) = 0 because φ is linear. Thus, yn ∈ Null(φ) for n ∈ N.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Banach Spaces

page 613

613

Furthermore, limn→∞ yn = z and this leads to a contradiction because  limn→∞ yn ∈ Null(φ) because Null(φ) is a closed subspace of L. Theorem 9.15. The normed dual L∗ of a real normed linear space L is a Banach space. Proof. We need to show only that L∗ is complete. Let (φn ) be a Cauchy sequence in L∗ . For x ∈ L the sequence (φn (x)) is a Cauchy sequence in R because |φn (x) − φm (x)| = |(φn − φm )x|  φn − φm  x. Therefore, the limit limn→∞ φn (x) exists and depends on x. This introduces a functional ψ : L −→ R defined by ψ(x) = limn→∞ φn (x). Note that ψ is linear because ψ(ax + by) = lim φn (ax + by) n→∞

= lim (aφn (x) + bφn (y)) n→∞

(because each φn is a linear functional) = a lim φn (x) + b lim φn (y) n→∞

n→∞

= aψ(x) + bψ(y). Since (φn ) is a Cauchy sequence, for every > 0 there exists n such that m, n  n implies φn − φm   , which, in turn, implies |φn (x) − φm (x)|  x for n, m  n and x ∈ L. Since limn→∞ φn (x) = ψ(x) it follows that |ψ(x) − φm (x)|  x

(9.3)

when m  n . This allows us to write |ψ(x)| = |ψ(x) − φm (x) + φm (x)|  |ψ(x) − φm (x)| + |φm (x)|  ( + φm )x, so ψ is bounded and, therefore, ψ ∈ L∗ . Moreover, from inequality (9.3)  we obtain ψ − φm  < when m > n , so limm→∞ φm = ψ. Let (S, E, m) be a measure space. Next, we examine the duals for the Banach spaces Lp (S, E, m). Any function f ∈ Lq (S, E, m), where p1 + 1q = 1 and 1  p  ∞, generates a linear functional φf on Lp (S, E, m) defined by 7 φf (u) = uf dm. S

May 2, 2018 11:28

614

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 614

Mathematical Analysis for Machine Learning and Data Mining

By H¨ older’s inequality we have: 7     |φf (u)| =  f u dm  f q uq . S

Therefore, φf   f q . Consequently, the mapping Ip : Lq (S, E, m) −→ Lp (S, E, m)∗ defined by Ip (f ) = φf is an injective linear operator with Ip   1. Theorem 9.16. Let (S, E, m) be a σ-finite measure space. For 1  p < ∞ the map Ip is a linear bijective isometry between Lq (S, E, m) and the dual of Lp (S, E, m). That is, for any φ6 ∈ (Lp (S, E, m))∗ there exists f ∈ Lq (S, E, m), such that φ = φf , φf (u) = S uf dm for every u ∈ Lp (S, E, m) and φ = f q . The Riesz representation theorem does not hold for p = ∞. We have the following equalities: 7 uf dm for f ∈ Lq (S, E, m), u ∈ Lp (S, E, m), (Ip (f ), u) = 7S (φ, u) = uIp−1 (φ) dm for φ ∈ (Lp (S, E, m))∗ , u ∈ Lp (S, E, m). S

Let L = (L, ·) an arbitrary F-normed space. For a fixed x ∈ L consider the map Fx : L∗ −→ F such that Fx (φ) = φ(x). Clearly Fx ∈ (L∗ )∗ . Moreover, by the dual expression of the norm Fx X ∗∗ = sup{Fx (φ) | φ∗ = 1} = sup{Fx (φ) | φ∗ = 1} = x. Hence, the mapping k : L −→ L∗∗ defined by k(x) = Fx is a linear isometry, k(x)∗∗ = x. Thus we obtain a canonical isometric embedding of L into L∗∗ . A Banach space X is called reflexive if the canonical isometry k is an onto mapping (and hence k is a linear bijective isometry between L and L∗∗ ). In general, k(L) ⊆ L∗∗ is a proper algebraic subspace of L∗∗ . Indeed, L∗∗ is complete as a dual of a normed space. Thus, if k(L) = L∗∗ , it follows that L is complete. However, the completeness is not sufficient for reflexivity. For many Banach spaces arising in the applications the inclusion k(L) ⊂ L∗∗ is strict. Theorem 9.17. For p  1, the dual space ( p (R))∗ is isometric to the space q (R), where p1 + 1q = 1.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 615

Banach Spaces

615

Proof. We present here the argument for 1  p < ∞; the arguments for p = ∞ and q = 1 are similar. inf ty Define the mapping Φ : q −→ ( p )∗ (R) as Φ(y)(x) = n=0 yn xn . It is immediate that Φ is linear. This mapping is well-defined because m  p1  m  q1 m    |xi yi |  |xi |p |yi |q , i=0

i=0

i=0

by H¨older’s inequality. Since |Φ(y)(x)|  yq xp , it follows that Φ(y)(p )∗  yq ,

(9.4)

q

∗

p

so Φ is a bounded linear operator between (R) and ( (R)) . To prove that Φ is surjective, let f ∈ ( p (R))∗ . If ei is the sequence in p

(R) whose components equal 0, with the exception of the ith component that equals 1, where i ∈ N, define y as yi = f (ei ). Let xm be the sequence given by  |yk |q−1 if k  m, m (x )k = 0 if k > m. We have

 x p = m

m 

 p1 |yk |

p(q−1)

=

i=0

and f (xm ) =

m 

m 

 p1 |yk |

q

(xm )k f (ek ) =

k=0

m 

(xm )k yk =

k=0

m 



|yk |q = f (xm )  f (p(R))∗ xm p = f (p(R))∗

k=0

which implies ( f (p (R))∗ , or

ykq .

k=0

Consequently, m 

,

i=0

m 

 p1 |yk |q

,

k=0

m

1

k=0

|yk |q ) q  f (p(R))∗ for every m. Therefore, yq  yq  Φ(y)(p (R))∗ .

(9.5)

The continuity of f implies that for every x ∈ (R) we have   ∞ ∞ m    f (x) = f lim xk ek = xm f (ek ) = xm ym = Φ(y)(x). p

m→∞

k=0

m=0

m=0

This shows that Φ is surjective. By inequalities (9.4) and (9.5) we have yq = Φ(y)(p (R))∗ , so Φ is an isometry. 

May 2, 2018 11:28

616

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 616

Mathematical Analysis for Machine Learning and Data Mining

Corollary 9.3. For p  1 the spaces p (R) are Banach spaces. Proof. This statement is an immediate consequences of Theorems 9.15 and 9.17.  In the special case p = 2, the spaces 2 (R) and ( 2 (R))∗ are isometric; thus, 2 (R) is isomorphic with its dual.

9.6

Spectra of Linear Operators on Banach Spaces

The notions of eigenvalue, eigenvector, and resolvent of a linear operators were introduced in Section 2.3. Definition 9.4. The complex numbers λ for which the resolvent Rh,λ is defined on the whole space L and it is continuous are known as regular values for h. The spectrum of h is the set of complex numbers σ(h) that consists of those complex numbers that are not regular values for h. Three subsets of σ(h) are distinguished: (i) the point spectrum of h, σp (h) that consists of those numbers λ such that Rh,λ does not exists; (ii) the continuous spectrum of h, σc (h) that consists of those numbers λ such that Rh,λ exists, Dom(Rh,λ ) ⊂ L and K(Dom(Rh,λ )) = L; (iii) the residual spectrum of h, σr (h) that consists of those numbers λ such that Rh,λ exists and K(Dom(Rh,λ )) ⊂ L. Example 9.6. Let h : C[a, b] −→ C[a, b] be the operator defined by h(φ) = ψ, where ψ(t) = tφ(t). If λ were an eigenvalue of h we would have h(φ)(t) = λφ(t) = tφ(t) for t ∈ [a, b], which is impossible. Thus, h has no eigenvalues. However, the residual spectrum consists of all numbers λ ∈ [a, b]. Indeed, note that if h(φ) − λφ = ψ, we have (t − λ)φ(t) = ψ(t) if ψ ∈ C[a, b] is a function such that ψ(t) = (t − λ)η(t) for some η ∈ C[a, b]. This shows that (Rh,λ )−1 (ψ) is defined if ψ has the form ψ(t) = (t − λ)η(t) for some η ∈ C[a, b], so K(Dom(Rh,λ )) ⊂ C[a, b]. Example 9.7. Consider the linear operator s : 2 (R) −→ 2 (R) defined by: s(x0 , x1 , x2 , . . .) = (0, x0 , x1 , x2 , . . .) for (x0 , x1 , x2 , . . .) ∈ 2 (R). Note that s(x) = x, so s is an isometry and, therefore a bounded operator. Suppose that λ is an eigenvalue of s.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Banach Spaces

page 617

617

Then, there exists x ∈ 2 (R) − {0} such that s(x) = λx, which means that 0 = λx0 , x0 = λx1 , . . . , xn = λxn+1 , . . . . Thus, either λ = 0, which yields the contradiction x = 0, or λ = 0, which implies the same conclusion. Thus, s has no eigenvalue. Theorem 9.18. Let L be a Banach space and let h be a bounded linear operator on L. If x1 , . . . , xn are n eigenvectors associated to the distinct eigenvalues λ1 , . . . , λn , respectively, then the set {x1 , . . . , xn } is linearly independent. Proof. The proof is by induction on n. If n = 1 the set {x1 } is linearly independent because x1 = 0L . For the induction step assume that {x1 , . . . , xj−1 } is linearly independent, where j  2. If xj = a1 x1 + · · · + aj−1 xj−1 , then by the linearity of h we have λj xj = a1 λ1 x1 + · · · + λj−1 aj−1 xj−1 , which implies a1 (λ1 − λj )x1 + · · · + aj−1 (λj−1 − λj )xj−1 = 0. This contradicts the linear independence of x1 , . . . , xj−1 because the eigenvalues are supposed to be distinct. Therefore, {x1 , . . . , xj−1 , xj } is linearly independent.  Theorem 9.19. Let h : L −→ L a continuous linear operator on a Banach space L and let λ be an eigenvalue of h. The subspace generated by the set of eigenvectors that correspond to λ is a closed subspace Lλ of L. Proof.

If x, y ∈ Lλ , then h(ax + by) = ah(x) + bh(y) = aλx + bλy = λ(ax + by),

so ax + by ∈ Lλ , which shows that Lλ is a subspace. If (xn ) is a sequence in Lλ such that limn→∞ xn = x, then h(x) = h( lim xn ) = lim h(xn ) n→∞

n→∞

(because h is continuous) = lim λxn = λ lim xn = λx, n→∞

which implies x ∈ Lλ .

n→∞



May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 618

Mathematical Analysis for Machine Learning and Data Mining

618

Theorem 9.20. Let h ∈ L(L, K) be a surjective linear operator between the Banach spaces L and K. If there exists an inverse bounded linear operator h−1 : K −→ L, then for any bounded linear operator g : L −→ K such that g < h−1 1 , the operator h+g has the bounded inverse (h+g)−1 ∈ L(K, L) and 1 (h + g)−1   1 . h−1 − g Proof. Since h has the inverse operator h−1 the equality (h + g)x = y implies h(x) = y − g(x), hence x = h−1 (y − g(x)). Define the operator ky (x) = h−1 (y − g(x)). We have ky (x1 ) − ky (x2 ) = h−1 (y − g(x1 )) − h−1 (y − g(x2 )) = h−1 g(x2 − x1 ), which implies ky (x1 ) − ky (x2 )  h−1 gx2 − x1   ax2 − x1 , −1

where h g < a < 1. Thus, ky is a contraction, which implies that the equation (h + g)x = y has a unique solution for every y ∈ K. Since x = h−1 (h(x)) for every x ∈ L, we have x  h−1  h(x) 1 which yields h(x)  h−1 x. Therefore, y = (h + g)(x)  h(x) − g(x) 1 x − gx  h−1    1 = − g x. h−1  Consequently, −1

(h + g)



−1

1

− g y h−1  for every y ∈ K. Thus, (h + g)−1 is bounded and 1 (h + g)−1   1 . h−1 − g y 



Theorem 9.21. Let h : L −→ L be a linear operator such that h < 1. Then, the linear operator 1L − h is invertible and ∞  −1 hk . (1L − h) = k=0

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Banach Spaces

b3234-main

page 619

619

Proof. Note that (1L − h)(1L + h + h2 + · · · + hn ) = 1L − hn+1 . We have limn→∞ hn+1  = 0 because hn+1   (h)n+1 and h < 1, hence limn→∞ (1L − hn+1 ) = 1L . Thus, (1L − h) lim (1L + h + h2 + · · · + hn ) = 1L , n→∞



which proves the equality.

Theorem 9.22. If h is a bounded linear operator in L(L), then λ ∈ σ(h) implies λ  h. Furthermore, the spectrum of h is a closed set. Proof. The operator h1 = −λ1L is surjective for any λ ∈ C. Thus, −1 1 1 h−1 1 = − a 1 and h1  = |a| . If h < |a|, by Theorem 9.20 applied to the operators h1 and h, the operator h1 + h has a bounded inverse and (h1 + h)−1  

1 , − h h−1 1

1

which amounts to (h − λ1L )−1  

1 . λ − h

In turn, this means that h(x) − λx = y has the unique solution, so a is a regular value for h. Thus, for any λ ∈ σ(h) we have λ  h. Suppose now that λ0 is a regular value for h, that is, Rh,λ0 = (h−λ1L )−1 1 is continuous on L. Let λ be a complex number such that |λ−λ0 | < Rh,λ . 0 By applying Theorem 9.20 to the operators g1 = h − λ0 1L and g2 = (λ0 − λ)1L , taking into account that g1−1 = Rh,λ0 and g2  = |λ0 − λ|, we infer that the operator g1 + g2 = h − λ1L has an inverse on L when 1 |λ − λ0 | < Rh,λ . Therefore, the set of regular values is an open set in C 0 which implies that its complement, the spectrum of h is a closed set.  Exercises and Supplements (1) Let φ be the functional defined on the Banach space (C[0, 1],  · ∞ ) defined by φ(f ) = f (0). Prove that φ is bounded and compute φ. (2) Let (S,  · 1 ) and (S,  · 2 ) be two Banach spaces defined over the same set S. If x1  x2 for all x ∈ S, prove that there exists a positive number c such that x2  cx1 . (3) Let (S,  · S ) and (T,  · T ) be two normed spaces. Prove that if f : S −→ T is a homeomorphism between these spaces, then (S,  · S ) is a Banach space if and only if (T,  · T ) is a Banach space.

May 2, 2018 11:28

620

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 620

Mathematical Analysis for Machine Learning and Data Mining

(4) Let L be a Banach space and let h be an invertible linear operator on 1 , then g L. Prove that a linear operator g on L such that h − g < h is also invertible. Solution: Observe that 1L −h−1 g = h−1 (h−g)  h−1 h−g < 1. Since g can be written as g = h(1L − (1L − h−1 g)), and the operator 1L − (1L − h−1 g) is invertible by Theorem 9.21, it follows that g is also invertible. (5) Prove that the set of invertible linear operators defined on a Banach space is open in the topological space of linear operators. (6) Let L be a normed linear space and T a Banach space. If M is a dense linear subspace of L and h : M −→ T is a bounded linear map, then ˜ : L −→ T such that h(x) ˜ there is a unique bounded linear map h = h(x) ˜ for all x ∈ M . Furthermore, h = h. Solution: Since M is a dense linear subspace of L for every x ∈ L there exists a sequence (xn ) of elements of M such that limn→∞ xn = x. Since h is a bounded operator, the sequence (h(xn )) is a Cauchy sequence, and therefore, it is convergent in T . Thus, we can define ˜ h(x) = limn→∞ h(xn ). ˜ is well-defined for, if (zn ) is another sequence of eleThe function h ments of M such that limn→∞ zn = x, we have xn − zn   xn − x + x−Zn , hence limn→∞ xn −zn  = 0. Since h(xn )−h(zn ) = h(xn − zn )  hxn − zn , it follows that limn→∞ h(xn ) = limn→∞ h(zn ). ˜ is an extension of h, because if x ∈ M , by using the The operator h ˜ h we obtain h(x) = h(x) constant sequence with xn = x for all n to define ˜ ˜ follows from the linearity of h. The for x ∈ M . The linearity of h ˜ follows from the fact that h(x) ˜ boundedness of h = limn→∞ h(xn )  ˜  h. Since h(x) ˜ = limn→∞ hxn  = hx. Also, we have h ˜ = h. h(x) for x ∈ M , it follows that h ˜ is the unique bounded linear operator from L to K The operator h that coincides with h on M . Suppose that g is another such operator, and let x ∈ L. Let (xn ) be a sequence in M that converges to x. Then, using the continuity of g, the fact that g is an extension of h, and the ˜ we obtain definition of h h(x). g(x) = lim g(xn ) = lim h(xn ) = ˜ n→∞

n→∞

(7) Let B[0, 1] be the open ball centered in (0, . . . , 0, . . .) in 1 (R). Prove that there exist infinitely many pairwise disjoint balls of a given radius r included in B[0, a] and that B[0, 1] is not compact.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Banach Spaces

page 621

621

(8) Let (S, Od ) be a metric space and let U be a subspace of S such that K(U ) = S. If h : U −→ T is a bounded linear operator into a Banach space T prove that h has a unique extension to a linear operator h : S −→ T . It is natural to extend the Riemann integral (discussed in Section 8.5) from realvalued functions to functions ranging over Banach spaces. Let (T,  · ) be a Banach space and let f : [a, b] −→ T be a real-argument function. Then, the notion of Riemann sum introduced in equality (8.14) extends immediately to this more general case, as σ(f, Δ, Ξ) =

n 

f (ξi )(ai − ai−1 ),

i=1

where Δ = {a0 , a1 , . . . , an } ∈ SUBD[a, b] and Ξ = {ξ1 , . . . , ξn } be a set of numbers such that ξi ∈ [ai−1 , ai ] for 1  i  n. Then, the Riemann integral is defined as limν(Δ)→0 σ(f, Δ, Xi) for any choice of Ξ. It can be shown that if f is a continuous on [a, b] is uniformly continuous and, therefore, integrable. (9) If X, Y are Banach spaces and h : X −→ Y , then "b h( a f (t) dt).

"b a

h(f (t)) dt =

(10) Let φ : [a, b] −→ R be a real-valued integrable function. If h : "b [a, b] −→ X is defined by h(t) = φ(t)x0 , then prove that a h(t) dt = "b ( a φ(t) dt)x0 . "b "b (11) Let h : [a, b] −→ X. Prove that || a h(t) dt ||  a f (t) dt. (12) Let X and Y be two Banach spaces and let h : X −→ Y be a surjective linear continuous function. Prove that there exists a positive number t such that tB[0Y , r] ⊆ K(h(B[0X , 1])). Solution: Since h is surjective we have:  Y = h(X) = h

∞  n=1

 nB[0X , 1]

=

∞  n=1

h(nB[0X , 1]) =

∞ 

n h(B[0X , 1]).

n=1

By Corollary 5.9 there exists a set n h(B[0X , 1]) that contains an open set. Therefore, K(h(B[0X , 1]) contains an open set y + tB[0Y , 1]. Since B[0X , 1] = −B[0X , 1] we also have −y + tB[0Y , 1] ⊆ K(h(B[0X , 1]). If z ∈ Y such that z  r, then there exist sequences (un ) and (vn ) in B[0X , 1] such that limn→∞  h(unn) = y + z and limn→∞ h(vn ) = −y + ∈ K(h(B[0X , 1])), which implies the z, hence z = limn→∞ h un +v 2 existence of t > 0 such that tB[0Y , r] ⊆ K(h(B[0X , 1])).

May 2, 2018 11:28

622

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 622

Mathematical Analysis for Machine Learning and Data Mining

(13) This result contained here is known as Graves’ Theorem [65]. Let X and Y be two Banach spaces, f : B[0X , r] −→ Y be a function such that f (0X ) = 0Y , and let g : X −→ Y be a surjective continuous linear mapping such that tB[0Y , 1] ⊆ g(B[0X , 1]) for some t > 0. Suppose that f − g is a Lipschitz function on B[0X , r] such that (f − g)(x1 ) − (f − g)(x2 )  cx1 − x2 , for all x1 , x2 ∈ B[0X , r], where 0  c < t. Prove that if y  (t − c)r, then (t − c)rB[0Y , 1] ⊆ f (rB[0x , r]) ,

(9.6)

that is the equation y = f (x) has a solution x ∈ B[0X , r]. Solution: Let a = t − c and let y ∈ B[0Y , ar]. Consider the sequence < r. (xn ), where x0 = 0X and let x1 is such that x1   1t y  ar t Suppose that we have generated x0 , x1 , . . . , xk . Then xk+1 is defined by g(xk+1 − xk ) = (g − f )(xk ) − (g − f )xk−1 for k  1. Since f − g is a Lipschitz function, we have g(xk+1 − xk )  cxk+1 − xk . Taking into account that tB[0Y , 1] ⊆ g(B[0X , 1]), xk+1 is chosen such that txk+1 − xk   cxk − xk−1 , hence xk+1 − xk   c xk − xk−1 , which implies t xn − xn−1  

 c n−1 t

x1 .

Since ct < 1, (xn ) is a Cauchy sequence. There exists x ∈ X such that limn→∞ xn = x because X is a Banach space. Note that xn  

n  k=1

xk − xk−1  

n    c k−1 k=1

t

x1  

x1  dr  = r, 1 − τc r−c

hence xk+1 ∈ B[0X , r]. Thus, we have x ∈ B[0X , r]. (14) Let T be a Banach space and let f : (a, b) −→ T be a function differentiable on (a, b). Prove that f (y) − f (x)  |y − x| · sup{(Df )((1 − t)x + ty) | t ∈ [0, 1]} for x, y ∈ I. Solution: Let μ be a number such that sup{(Df )((1 − t)x + ty) | t ∈ [0, 1]} < μ and let Kμ = {t ∈ [0, 1] | f ((1 − t)x + ty) − f (x)  μt|y − x|}.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Banach Spaces

page 623

623

The set Kμ is closed and 0 ∈ Kμ . Let s be the largest element of Kμ . Clearly, s  1. Suppose s < 1 and let t ∈ (s, 1). If t − s is sufficiently small we have f ((1 − t)x + ty) − f (x)  f ((1 − t)x + ty) − f ((1 − s)x + sy)) + f ((1 − s)x + sy)) − f (x)  (Df )((1 − s)x + sy))(t − s)(y − x) + o(|t − s| |y − x|) + μs|y − x|  μ(t − s)|y − x| + μs|y − x| = μt|y − x|, hence t ∈ Kμ , which contradicts the definition of s. Therefore, we must have s = 1, which concludes the argument. (15) Let S, T be Banach spaces, U be an open set in S, and let f : U −→ T . If x, y ∈ U such that [x, y] ⊆ U and h is a linear mapping, prove that f (y) − f (x) − h(y) − h(x)  y − x · sup (Df )((1 − t)x + ty) − h. t∈[0,1]

Solution: The function g(t) = f ((1 − t)x + ty) − th(y − x) is differentiable and (Dg)(t) = ((Df )((1 − t)x + ty) − h)(y − x). By Supplement 14 we have g(1) − g(0)  sup{(Dg)(t) | t ∈ [0, 1]}, hence f (y) − f (x) − h(y) − h(x)  sup (Df )((1 − t)x + ty))(y − x) − h(y − x) t∈[0,1]

 y − x sup{(Df )((1 − t)x + ty) − h | t ∈ [0, 1]}.

(16) Let (pn ) be a sequence of projections on a Banach space L such that limn→∞ pn (y) = y for each y ∈ L. Suppose that for each n ∈ N there exists xn such that pn (h(xn ) − a) = 0. Prove that if limn→∞ xn = x, then h(x) = a. Solution: By the continuity of the norm, limn→∞ pn (y) = y implies limn→∞ pn (y) = y, hence sup{pn (y) | n ∈ N} is finite. By the Uniform Boundedness Theorem (Theorem 9.7) sup{pn  | n ∈ N} is finite. Since h is continuous, we have limn→∞ hn (x) = h(x). Therefore, limn→∞ pn (h(xn ) − h(x)) = 0. The definition of (xn ) implies pn (hn (xn )) = pn (a), hence limn→∞ pn (a) − pn (h(x)) = 0. Therefore, a = h(x).

Bibliographical Comments References on Banach spaces include [22, 7, 40, 8, 101, 26] . The proof of the Uniform Boundedness Theorem (Theorem 9.7) was obtained in [123]. An useful source on infinite-dimensional spaces is the monograph [54].

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 625

Chapter 10

Differentiability of Functions Defined on Normed Spaces

10.1

Introduction

In this chapter we present a generalization of the idea of differentiability of classical analysis. Two variants of differentiability, the stronger Fr´echet differentiability of functions between normed spaces and the weaker Gˆ ateaux differentiability are discussed. 10.2

The Fr´ echet and Gˆ ateaux Differentiation

Definition 10.1. Let (S,  · ) and (T,  · ) be two normed spaces and let X be an open set in (S,  · ). A function f : X −→ T is Fr´echet differentiable at x0 , where x0 ∈ X, if there exists a linear operator (Dx f )(x0 ) : X −→ T such that lim

h→0

f (x0 + h) − f (x0 ) − (Dx f )(x0 )(h) = 0. h

The linear operator (Dx f )(x0 ) : X −→ T is referred to the Fr´echet derivative at x0 . Suppose that X = S in Definition 10.1. Note that: (i) (Dx f )(x0 )(h) is an element of T ; (ii) (Dx f )(x0 ) is a linear operator defined on S and ranging into T ; (iii) (Dx f ) is a mapping defined on S with values in the space of linear operators between S and T . The function δf : X × S −→ T defined by δf (x0 ; h) = (Dx f )(x0 )(h) is the Fr´echet differential of f at x0 . Note that the differential is linear in its second argument h. To emphasize the distinct roles played by x0 and h the arguments of the differential are separated by a semicolon. 625

May 2, 2018 11:28

626

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 626

Mathematical Analysis for Machine Learning and Data Mining

If (Dx f )(x0 ) is continuous on X, where X ⊆ S, we say that f is continuously differentiable on X. Example 10.1. A constant function k : S −→ T is Fr´echet differentiable at every point x0 of S and (Dx f )(x0 ) = 0. Example 10.2. If f : S −→ T is a continuous linear mapping then f (x0 + h) − f (x0 ) = f (h) for x0 ∈ S, so (Dx f )(x0 ) = f . For instance, consider the linear operator f : C[a, b] −→ C[a, b] defined as 7 b (f u)(x) = K(x, s)u(s) ds. a

Its Fr´echet derivative is (Dx f )(u) = f (u) because 7 b 7 b (f (u + h))(x) − (f (u))(x) = K(x, s)(u(s) + h(s)) ds − K(x, s)u(s) ds a

7 =

a b

K(x, s)h(s) ds. a

Example 10.3. Let f : Rn −→ R be the function f (x) = x a for x, a ∈ Rn . We have f (x0 + h) − f (x0 ) = (x0 + h) a − x0 a = h a = a h. The function is Fr´echet differentiable because f (x0 + h) − f (x0 ) − (Dx f )(x0 )(h) lim h→0 h  a h − (Dx f )(x0 )(h) = lim =0 h→0 h is satisfied by (Dx )(x0 )(h) = a h, so Dx (x0 ) is the inner product (a, ·) for every x0 . Next we introduce the Landau notations o and O (known as “small o” and “big O”, respectively). Let f, g : R −→ R be two functions. We write f = O(g) if there exists a positive real number M and a number x0 such that x  x0 implies |f (x)|  M |g(x)|. Also, we write f = o(g) if there exists a number x0 such that for every > 0 we have |f (x)|  |g(x)| if x  x0 . If x  x0 implies g(x) = 0, this (x) = 0. amounts to limx→∞ fg(x) We denote by o (with or without subscripts) a function o : S −→ R that has the property: o(h) = 0. lim h→0 h

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

page 627

627

These notations allow us to say that f is Fr´echet differentiable at x0 if and only if there exists a linear transformation (Dx f )(x0 ) in Hom(S, T ) such that f (x0 + h) − f (x0 ) − δf (x0 ; h) = o(h), or f (x0 + h) − f (x0 ) − (Dx f )(x0 )(h) = o(h). The Fr´echet derivative (Dx f )(x0 ) will be simply denoted by (Df )(x0 ) when x is clear from the context. Theorem 10.1. Let (S,  · ) and (T,  · ) be two normed spaces and let X be an open subset of S. If a function f : X −→ T has a Fr´echet differential, where X ⊆ S, then this differential is unique. Proof. Suppose that both δf (x0 ; h) and δ1 f (x0 , h) are differentials of f at x0 . We have δf (x0 ; h) − δ1 f (x0 ; h)  f (x0 + h) − f (x0 ) − δf (x0 ; h) +f (x0 + h) − f (x0 ) − δ1 f (x0 ; h) = o(h).



Since δf (x0 ; h) − δ1 f (x0 ; h) is bounded and linear in h it follows that δf (x0 ; h) = δ1 f (x0 ; h). Theorem 10.2. Let (S,  · ) and (T,  · ) be two normed spaces. If f : S −→ T is a Fr´echet differentiable function in x0 ∈ S, then f is continuous in x0 . Proof.

Since f is Fr´echet differentiable in x0 , f (x0 + h) − f (x0 ) − (Df )(x0 )(h) lim = 0. h→0 h Let 1 be a positive number. There exists a positive number δ(x0 ) such that if h < δ(x0 ) we have f (x0 + h) − f (x0 ) − (Df )(x0 )(h) < 1 h. If h < δ(x0 ) we can write f (x0 + h) − f (x0 )  f (x0 + h) − f (x0 ) − (Df )(x0 )h + (Df )(x0 )h  f (x0 + h) − f (x0 ) − (Df )(x0 )h + (Df )(x0 )h

 ( 1 + (Df )(x0 ))h. )(x0 ))h < , * that is, if If we choose h such that ( 1 + (Df  h < min ,δ , 1 + (Df )(x0 ) it follows that f (x0 + h) − f (x0 ) < , so f is continuous in x0 .



May 2, 2018 11:28

628

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 628

Mathematical Analysis for Machine Learning and Data Mining

A weaker form of differentiability is given next. Definition 10.2. Let (S,  · ) and (T,  · ) be normed F-linear spaces, X be an open set in (S,  · ) and let f : X −→ T be a function. The function f is Gˆ ateaux differentiable in x0 (where x0 ∈ X) if there exists a linear operator (Dx f )(x0 ) : S −→ T such that f (x0 + tu) − f (x0 ) (Dx f )(x0 )(u) = lim t→0 t for every u such that x0 + tu ∈ X. The linear operator (Dx f )(x0 ) is the Gˆ ateaux derivative of f in x0 . The Gˆ ateaux differential of f at x0 is the linear operator δf (x0 ; h) given by f (x0 + tu) − f (x0 ) . δf (x0 ; u) = lim t→0 t The Gˆ ateaux differential is denoted by the same δf (x0 ; h) as the Fr´echet differential. Similarly, the Gˆ ateaux derivative is denoted by (Dx f )(x0 ), as we denoted the Fr´echet derivative and the subscript x will be omitted when possible. The specific differential (or derivative) we are referring to will result from the context. The function f : S −→ T is Gˆateaux differentiable in x0 if for every > 0 there exists δ(x0 , u) > 0, which depends on x0 and u such that t < δ(x0 , u) implies    f (x0 + tu) − f (x0 )    < . − (D f )(x )(u) x 0   t The function f : S −→ T is Fr´echet differentiable in x0 if for every > 0 there exists δ(x0 ) > 0, which does not depend on u such that t < δ(x0 ) (x0 ) − (Dx f )(x0 )(u)|| < . In this sense, Fr´echet differimplies || f (x0 +tu)−f t (x0 ) entiability implies that the convergence of f (x0 +tu)−f to (Dx f )(x0 )(u) t is uniform relative to u. Indeed, if h ∈ S we can write h = huh , where uh  = 1. Thus, if h → 0, we have h → 0. Since f is Gˆateaux uniformly differentiable at x0 , for every > 0 there is δ(x0 ) such that if h < δ(x0 ) then f (x0 + h) − f (x0 ) − (Dx f )(x0 )(h) h f (x0 + h) − f (x0 ) − h(Dx f )(x0 )(uh ) = h    f (x0 + h) − f (x0 )    =  − (Dx f )(x0 )(uh ) < h which shows that f is Fr´echet differentiable at x0 .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

page 629

629

Example 10.4. Let a be a vector in Rn . Define f : Rn −→ R as f (x) = x a. We have: f (x0 + tu) − f (x0 ) t (x0 + tu) a − x0 a = lim t→0 t tu a = lim = u a. t→0 t

(Dx f )(x0 )(u) = lim

t→0

Example 10.5. Let A ∈ Rn×n be a matrix and let f : Rn −→ R be the functional f (x) = x Ax. We have (Df )(x0 ) = x0 (A + A ). By applying the definition of Gˆ ateaux differential we have f (x0 + tu) − f (x0 ) t→0 t (x0 + tu )A(x0 + tu) − x0 Ax0 = lim t→0 t tu Ax0 + tx0 Au + t2 u Au = lim t→0 t = u Ax0 + x0 Au = x0 A u + x0 Au

(Df )(x0 )(u) = lim

= x0 (A + A )u, which yields (Df )(x0 ) = x0 (A + A ). If A ∈ Rn×n is a symmetric matrix and f : Rn −→ R is the functional f (x) = x Ax, then (Df )(x0 ) = 2x0 A. Example 10.6. Let (S, ·) be a normed space. The norm · : S −→ R0 is not Gˆateaux differentiable in 0S . Indeed, suppose that  ·  were differentiable in 0S , which would mean that the limit: lim

t→0

tu |t| = lim u t→0 t t

exists for every u ∈ S, which is contradictory. However, the square of the norm,  · 2 is differentiable in 0S because tu2 = lim tu = 0. t→0 t→0 t lim

May 2, 2018 11:28

630

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 630

Mathematical Analysis for Machine Learning and Data Mining

Example 10.7. Consider the norm  · 1 on Rn given by x1 = |x1 | + · · · + |xn | for x ∈ Rn . This norm is not Gˆateaux differentiable in any point x0 located on an axis. Indeed, let x0 = aei be a point on the ith axis. The limit x0 + tu1 − x0 1 lim t→0 t aei + tu1 − aei 1 = lim t→0 t |t||u1 | + · · · + |t||ui−1 | + (|t||ui | − |a|) + |t||ui+1 | + · · · + |t||un | = lim t→0 t does not exists, so the norm  ·1 is not differentiable in any of these points. Definition 10.3. Let f : S −→ T be a function between the normed spaces (S,  · ) and (T,  · ) and let h ∈ S − {0S }. The directional derivative at x0 in the direction h is the function ∂f ∂h (x0 ) given by f (x0 + th) − f (x0 ) ∂f (x0 ) = lim . t↓0 ∂h t Thus, f is Gˆateaux differentiable at x0 if its directional derivative exists in every direction. Let f : Rn −→ R be a function differentiable at x0 ∈ Rn . If {e1 , . . . , en } is the standard basis for Rn , then (Df )(x0 )(ei ) is known as the partial ∂f (x0 ). derivative of f with respect to xi and is denoted by ∂x i Theorem 10.3. Let (S,  · ) and (T,  · ) be two normed F-linear spaces, X be an open set in (S,  · ) and let f : X −→ T be a function. ateaux differIf f is Fr´echet differentiable in x0 ∈ X, then it is also Gˆ entiable in x0 and the two differentials are the same. Proof. Let f be Fr´echet differentiable in x0 , that is, f (x0 + h) − f (x0 ) − δf (x0 ; h) = o(h), where δf (x0 , h) is the Fr´echet differential. For the Gˆateaux differential δ  (x0 ; h) of f in x0 we have δ  f (x0 ; h) − δf (x0 ; h)     f (x0 + tu) − f (x0 )   − δf (x0 ; h) =  lim t→0 t    f (x0 + tu) − f (x0 ) − δf (x0 ; th)   h,   = lim   t→0 th 0 )−δf (x0 ;th) || = 0 because limt→0 th = 0. Thus, and limt→0 || f (x0 +tu)−f (x th  δ  f (x0 ; h) = δf (x0 ; h).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

page 631

631

Theorem 10.4. Let (S,  · ) and (T,  · ) be two normed F-linear spaces, X be an open set in (S,  · ) and let f : X −→ T be a function. If f is Gˆ ateaux differentiable on X, then f (u) − f (v)  u − v sup{f  (au + (1 − a)v) | a ∈ [0, 1]}. Proof. Let w ∈ X such that w = 1 and f (u) − f (v) = (w, f (u) − f (v)). Define the real-valued function g as g(t) = (w, f (u + t(v − u))) for t ∈ [0, 1]. We have the inequality f (u) − f (v) = (w, f (v) − f (u)) = |g(1) − g(0)|  sup{|g  (t)| | t ∈ [0, 1]}. Since

  df (u + t(v − u)) w, dt   f (u + (t + r)(v − u)) − f (u + t(v − u)) = w, lim r→0 r  (v − u)), = (w, fu+t(v−u)

g  (t) =

 (v − u), hence we have |g  (t)|  fu+t(v−u)  |g  (t)|  fu+t(v−u) (v − u)  v − u.  fu+t(v−u)



Theorem 10.5. Let (S,  · ), (T,  · ) be normed F-linear spaces, X be an open set in (S,  · ) and let f : X −→ T be a function. ateaux derivative is If f is Gˆ ateaux differentiable at x0 ∈ X and the Gˆ continuous in x0 , then f is Fr´echet differentiable in x0 . Proof. For v ∈ S define the function gv : [0, 1] −→ T be the function defined as gv (t) = f (x0 + tv) − f (x0 ) − t(Df )(x0 )v; we have g(0) = 0. From the continuity of the Gˆ ateaux derivative it follows that gv (1) − gv (0) = f (x0 + v) − f (x0 ) − f  (x0 )v  v sup{(Df )(x0 + tv) − (Df )(x0 )} = o(v). 

Thus, gv = 0.

Example 10.8. A homogeneous polynomial u of degree k has the property u(tx1 , . . . , txn ) = tk u(x1 , . . . , xn ) for t, x1 , . . . , xn ∈ R.

May 2, 2018 11:28

632

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 632

Mathematical Analysis for Machine Learning and Data Mining

Let p(x1 , x2 ) and q(x1 , x2 ) be two homogeneous polynomials of degrees r and s, respectively, where r > s + 1, and let f : R2 −→ R be the function defined by ⎧ ⎨ p(x1 , x2 ) if x = 0 , 2 f (x1 , x2 ) = q(x1 , x2 ) ⎩ 0 otherwise, where x = 02 implies q(x) = 0. We claim that f is Gˆateaux differentiable but not Fr´echet differentiable. Indeed, f (tu1 , tu2 ) = lim tr−s−1 f (u1 , u2 ) = 0, lim t→0 t→0 t and the constant function 0 is linear in u. Fr´echet differentiability in 02 requires the existence of a linear operator g : Rn −→ R such that f (h) − f (x0 ) − g(h) f (h) − g(h) lim = lim = 0, h→02 h→02 h h which is impossible because f (h) grows faster than a linear function in h assuming that r > s + 1. Thus, f is not differentiable Fr´echet. Theorem 10.6. (The Chain Rule) Let S, T, U be three normed spaces, X be an open subset of S and Y an open subset of T . If f : X −→ T is a function Fr´echet differentiable at x0 ∈ X and g : Y −→ U is a Fr´echet function differentiable at y0 = f (x0 ) ∈ Y , then gf is Fr´echet differentiable at x0 and Dx (gf )(x0 ) = (Dy g)(f (x0 ))(Dx f )(x0 ). Proof. Since f (x0 + h) − f (x0 ) − (Dx f )(x0 )(h) = o1 (h) and g(y0 + k) − g(y0 ) − (Dy g)(y0 )(k) = o2 (k), we have g(f (x0 + p)) − g(f (x0 )) = g (f (x0 ) + (Dx f )(x0 )(p) + o1 (p)) − g(f (x0 )) = (Dy g)(y0 ) ((Dx f )(x0 )(p) + o1 (p)) + o2 ((Dx f )(x0 )(p) + o1 (p)) = (Dy g)(y0 )((Dx f )(x0 )(p)) + (Dy g)(y0 )(o1 (p)) + o2 ((Dx f )(x0 )(p) + o1 (p)). Observe that (Dy g)(y0 )(o1 (p)) + o2 ((Dx f )(x0 )(p) + o1 (p)) = 0, h→0 p because (Dy g)(y0 )(o1 (p))  (Dy g)(y0 )o1 (p), and lim

o2 ((Dx f )(x0 )(p) + o1 (p))  o2 ((Dx f )(x0 )(p)) + o2 (o1 (p)) which shows that Dx (gf )(x0 ) = (Dy g)(f (x0 ))(Dx f )(x0 ).



May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Differentiability of Functions Defined on Normed Spaces

b3234-main

page 633

633

Note that in the Chain Rule, (Dy g)(f (x0 ))(Dx f )(x0 ) is the composition of the operators (Dy g)(f (x0 )) and (Dx f )(x0 ), which results in the operator (Dx (gf ))(f (x0 )). Next we introduce the notation “∇f ” (read “nabla f ”). Let f : X −→ R, where X ⊆ Rn , and let z ∈ X. The gradient of f in z is the vector ⎛ ∂f ⎞ (z) ⎜ ∂x1 ⎟ ⎜ . ⎟ n ⎜ (∇f )(z) = ⎜ .. ⎟ ⎟∈R . ⎝ ∂f ⎠ (z) ∂xn Example 10.9. Let bj ∈ Rn and cj ∈ R for 1  j  n, and let f : Rn −→ R be the function n  (bj x − cj )2 . f (x) = j=1  n ∂f  We have ∂x for 1  (x) = 2b (b x − c ), where b = · · · b b ij j j 1j nj j j=1 i j  n. Thus, we⎛obtain: ⎞ n  2b1j (bj x − cj ) ⎟ ⎜ j=1 ⎟ ⎜ ⎟ ⎜ . .. (∇f )(x) = 2 ⎜ ⎟ = 2(B  x − c )B = 2B  xB − 2c B, ⎟ ⎜ n ⎠ ⎝  2bnj (bj x − cj ) j=1

where B = (b1 · · · bn ) ∈ Rn×n . Theorem 10.7. (The Mean Value Theorem) Let f : X −→ R be a real-valued function, where X is an open subset of Rn that contains [a, b]. If f is continuous on [a, b] and is Gˆ ateaux differentiable on (a, b), then there exists c ∈ (a, b) such that f (b) − f (a) = (∇f )(c) (b − a). Proof. For the real-valued function g defined by g(t) = f (a + t(b − a)) we have g(0) = f (a) and g(1) = f (b). By the Mean Value Theorem of calculus, there exists θ ∈ (0, 1) such that g(1) − g(0) = g  (θ). By the Chain Rule, g  (t) = (∇f )(a + t(b − a)) (b − a), so f (b) − f (a) = (∇f )(a + θ(b − a)) (b − a), which shows that the statement holds for c = a + θ(b − a).  A equivalent form of Mean Value Theorem can be obtained by defining h = b − a and observing that c ∈ (a, b) if and only if c = a + θh, where θ ∈ (0, 1). The Mean Value Theorem amounts to stating the existence of θ ∈ (0, 1) such that f (a + h) = f (a) + (∇f )(a + θh) h, when [a, a + h] ⊆ X.

May 2, 2018 11:28

634

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 634

Mathematical Analysis for Machine Learning and Data Mining

If we compute explicitly the derivative of the function g introduced in the proof of Theorem 10.7 we have: ∂f ∂f g  (t) = (a + t(b − a))(b1 − a1 ) + · · · + (a + t(b − a))(bn − an ) ∂x1 ∂xn ⎞ ⎛ ∂f (a + t(b − a)) ⎟ ⎜ ∂x1 ⎟ ⎜ . ⎟ (b − a). ⎜ .. =⎜ ⎟ ⎠ ⎝ ∂f (a + t(b − a)) ∂xn For functions of the form f : Rn −→ Rm the Gˆateaux differential at x0 has a simple form. Since (Df )(x0 ) is a linear transformation there exists a matrix A ∈ Rm×n such that lim

t→0

f(x0 + tu) − f(x0 ) = Au t

for every u ∈ Rn . In other words, A is the matrix of the linear transformation (Df)(x0 ) : Rn −→ Rm . If ai is the ith row of A, where 1  i  m, the previous equality can be written componentwise as fi (x0 + tu) − fi (x0 ) = ai u lim t→0 t for 1  i  m. Thus, we have ai = (∇fi )(x0 ) for 1  i  m and the matrix A introduced above is ⎞ ⎛ ∂f1 ∂f1 (x0 ) · · · ∂x (x ) 0 n ⎟ ⎜ ∂x1 ⎟ ⎜ . . ⎟ ⎜ ⎞ ⎛ . . ⎟ ⎜ . ··· . (∇f1 )(x0 ) ⎟ ⎜ ∂f ⎟ ⎟ ⎜ ⎜ i . ∂f i .. A=⎝ = (x ) · · · (x ) ⎜ ⎠ 0 ⎟ ∂xn ⎟ ⎜ ∂x1 0 ⎟ ⎜ .. .. (∇fm )(x0 ) ⎟ ⎜ . ··· . ⎟ ⎜ ⎠ ⎝ ∂f m m (x0 ) · · · ∂f (x ) 0 ∂xn ∂x1 The matrix A ∈ Rm×n is referred to as the Jacobian of f : Rn −→ Rm at x0 . As before, if x is understood from context we may omit occasionally the subscript x and write A simply as (Df)(x0 ). The rows of Jacobian of f at x0 consist of the transposed gradients of its component functions f1 , . . . , fm at x0 . Of course, if f : Rn −→ R we have (Df )(x0 )(u) = (∇f )(x0 ) u n for u ∈ R .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

page 635

635

Definition 10.4. The Jacobian determinant of f at x0 is the number (Jf)(x0 ) = det((Df)(x0 )). Example 10.10. Let f : R2 −→ R3 and g : R4 −→ R3 be the functions defined by ⎛ ⎞ ⎞ ⎛ y1 + y2 x1 + x2 ⎜y2 + y3 ⎟ ⎟ f(x) = ⎝x21 − x22 ⎠ and g(y) = ⎜ ⎝y1 + y3 ⎠ x1 − x2 y12 for x ∈ R2 and y ∈ R3 . We have ⎛

⎛

⎞

1 1 1 ⎜ 0 (Df)(x) = ⎝2x1 −2x2 ⎠ and (Dg)(y) = ⎜ ⎝ 1 1 −1 2y1

 If x0 = ab , then

⎞ 10 1 1⎟ ⎟. 0 1⎠ 00

⎞ a+b f(x0 ) = ⎝a2 − b2 ⎠. a−b ⎛

By applying the Chain Rule we can write: D(gf)(x0 ) = (Dg)(f(x0 ))(Df)(x0 ) ⎛ ⎞ ⎞ 1 10 ⎛ 1 1 ⎜ 0 ⎟ 1 1⎟ ⎝ =⎜ 2a −2b⎠ ⎝ 1 0 1⎠ 1 −1 2(a + b) 0 0 ⎛ ⎞ 1 + 2a 1 − 2b ⎜ 2a + 1 −2b − 1⎟ ⎟. =⎜ ⎝ 2 0 ⎠ 2(a + b) 2(a + b) Alternatively, the derivative of gf can be computed by composing first the functions f and g. This yields ⎛ ⎞ x1 + x2 + x21 − x22 ⎜x21 − x22 + x1 − x2 ⎟ ⎟. gf(x) = ⎜ ⎝ ⎠ 2x1 (x1 + x2 )2

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 636

Mathematical Analysis for Machine Learning and Data Mining

636

This implies

⎛

⎞ 1 − 2x2 1 + 2x1 ⎜ 2x1 + 1 −2x2 − 1 ⎟ ⎟. (Dgf)(x) = ⎜ ⎝ ⎠ 2 0 2(x1 + x2 ) 2(x1 + x2 )

Substituting a for x1 and b for x2 we retrieve the first expression. Example 10.11. Let h : R0 × [0, π] × [0, 2π) −→ R3 be the function that maps spherical coordinates into Cartesian coordinates. Its components are: h1 (r, θ, φ) = r sin θ cos φ, h2 (r, θ, φ) = r sin θ sin φ, h3 (r, θ, φ) = r cos θ. The Jacobian matrix in (r, θ, φ) is: ⎛ ⎞ sin θ cos φ r cos θ cos φ −r sin θ sin φ (Dh)(r, θ, φ) = ⎝ sin θ sin φ r cos θ sin φ r sin θ cos φ ⎠. cos θ −r sin θ 0 Therefore, J(h)(r, θ, φ) = r2 sin θ. Similarly, the function g : R2 −→ R2 that maps polar coordinates into Cartesian coordinates, given by g1 (r, φ) = r cos φ, g2 (r, φ) = r sin φ has the Jacobian matrix

  cos φ −r sin φ, (Dg)(r, φ) = . sin φ r cos φ

Thus, J(g)(r, φ) = r. Example 10.12. Let f : R3 −→ R2 be the function defined by:  2  x + x22 + x23 f(x) = x11 e + ex2 + ex3   x1 for x = x2 . The Jacobian of f at x is the matrix x3

(Jf)(x) =

  2x1 2x2 2x3 ∈ R2×3 . ex1 ex2 ex3

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

page 637

637

Lemma 10.1. Let a, b ∈ R and let f : [a, b] −→ T be a continuous function of a real argument, where (T,  · ) is a normed space. If (Df )(x) exists for x ∈ (a, b) and (Df )(x)  m, then f (b) − f (a)  m(b − a). Proof. Let (α, β) ⊆ (a, b). Since f is continuous, to prove the statement it suffices to show that f (β) − f (α)  (m + )(b − a) for every > 0. Let U be the closed subset of R: U = {x ∈ [α, β] | f (x) − f (α)  (m + )(b − a)}. Let x0 = sup U . Since U is compact, we have x0 ∈ U , so x0  β. We show that x0 = β by proving that x0 < β leads to a contradiction. Suppose that x0 < β. Since f is differentiable at x0 , there exists δ such that 0 < δ < β − x0 such that |h| < δ implies f (x0 + h) − f (x0 ) − (Df )(x0 )h < |h|. For h =

δ 2

and u = x0 +

δ 2

we have u = x0 + h and

f (u) − f (x0 ) − (Df )(x0 )(u − x0 ) < |u − x0 |. Therefore, f (u) − f (x0 ) = f (u) − f (x0 ) − (Df )(x0 )(u − x0 ) + (Df )(x0 )(u − x0 ) < |u − x0 | + (Df )(x0 )(u − x0 )  (m + )(u − x0 ). Since x0 ∈ U , we have f (x0 ) − f (α)  (m + )(x0 − α), hence f (u) − f (α)  f (u) − f (x0 ) + f (x0 ) − f (α)  (m + )(u − α), which shows that u ∈ U . This contradicts the fact that u > x0 . Thus, x0 = β ∈ U and f (β) − f (α)  (m + )(b − a). The result follows by continuity.



Theorem 10.8. (Mean Value Theorem for Functions between Normed Spaces) Let (S,  · ) and (T,  · ) be two normed spaces, X be an open subset of S, and let f : X −→ T . If [a, b] ⊆ X and (Df )(x) exists in each point x of S, then f (b) − f (a)  b − a sup{(Df )(x) | x ∈ [a, b]}.

May 2, 2018 11:28

638

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 638

Mathematical Analysis for Machine Learning and Data Mining

Proof. Define the function p : [0, 1] −→ X as p(t) = at + b(1 − t). It is clear that p (t) = a − b, so the derivative of p regarded as a function (by the remark following Definition 10.1) maps t into (a − b)t. Consider the function g : [0, 1] −→ T given by g = f p. We have g(1) = f (b) and g(0) = f (a). By the chain rule we have g  (t) = (Df )(at + b(1 − t))(a − b), which allows us to write: f (b) − f (a) = g(1) − g(0)  sup{g  (t) | t ∈ [0, 1]} (by Lemma 10.1)  b − a sup{(Df )(x) | x ∈ [a, b]}.



Corollary 10.1. Let (S,  · ) and (T,  · ) be two normed spaces, X be an open connected subset of S and let f : X −→ T be a differentiable function on X. If (Dx f )(x) = 0T for x ∈ X, then f is a constant function on X. Proof. The function f is continuous on X because is differentiable on this set. Let x0 ∈ X, and let W be the set W = {x ∈ X | f (x) = f (x0 )}. By Theorem 4.4, W is a closed set in the subspace X. Since X is an open set in S, there is an open sphere B(x, r) included in X. Let y ∈ B(x, r). Since [x, y] ∈ B(x0 , r), by Theorem 10.8, we have f (x) − f (y)  x − y sup{(Df )(z) | z ∈ [x, y]} = 0, which implies f (x) = f (y) = f (x0 ). Thus, y ∈ W , which implies B(x0 , r) ⊆ W . This means that W is also an open set. Since W ⊆ X, this leads to a contradiction, because, by Theorem 4.91, a connected set may not contain a clopen set. Thus, X = W , and f is constant on X.  Theorem 10.9. Let f : Rn −→ R be a continuously differentiable function in an open convex subset U of Rn . For any x ∈ U and direction h the  directional derivative ∂f ∂h (x) exists and equals (∇f )(x) h. Furthermore, for any x, x + h ∈ U we have 7 1 (∇f )(x + th) h dt. f (x + h) = f (x) + 0

There exists z ∈ (x, x + h) such that f (x + h) = f (x) + ((∇f )(z)) h.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

page 639

639

Proof. Let g : R −→ R be the function defined as g(t) = f (x + th). By the Chain Rule, for a ∈ [0, 1] we have: dg = ((∇f )(x + ah)) h, dt ∂g which implies ∂h = (∇f )(x) h by taking a = 0. Therefore, g(1) = g(0) + 61  61 g (t) dt, which amounts to f (x + h) = f (x) + 0 ((∇f )(x + th)) h dt. 0 By the Mean Value Theorem, there exists ξ ∈ (0, 1) such that g(1) = g(0) + g  (ξ). By the definition of g this implies

f (x + h) = f (x) + ((∇f )(x + ξh)) h, 

which proves the second part.

Corollary 10.2. Let f : Rn −→ Rm be a function that is continuously differentiable in the open convex subset U of Rn . For x, x + h ∈ U we have 7 1 f(x + h) − f(x) = (Df)(x + th)h dt. 0

Proof. of f.

This follows by an applying Theorem 10.9 to the components 

If there exists c > 0 such that d (f (x), f (y))  cd(x, y) for all x, y ∈ S, then we say that f is a Lipschitz function. Furthermore, if this inequality is satisfied for a number c < 1, then f is a contraction with the contraction constant c. Lemma 10.2. Let X be a Banach space and let B[x0 , r] be a closed sphere in X. If B[x0 , r] ⊆ V , where V is open set in X, and F : V −→ B[x0 , r] is a function that is differentiable on B[x0 , r] such that sup{(DF )(x) | x ∈ B[x0 , r]} < 1, then F has a unique fixed point in B[x0 , r]. Proof. Let c = sup{(Dx F )(x) | x ∈ B[x0 , r]} < 1. By Theorem 10.8, if a, b ∈ B[x0 , r] we have f (b) − f (a)  cb − a. By Theorem 5.48 there exists a unique fixed point in B[x0 , r].



Let X, Y be two Banach spaces. It is easy to verify that X × Y is a Banach space relative to the norm defined by (x, y) = x + y for (x, y) ∈ X × Y .

May 2, 2018 11:28

640

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 640

Mathematical Analysis for Machine Learning and Data Mining

Let X, Y, Z be three Banach spaces and let F : X × Y −→ Z be a function. Denote by Dx F (x0 , y0 ) and by Dy F (x0 , y0 ) the linear operators in Hom(X, Z) and Hom(Y, Z), respectively, defined by F (x0 + h, y0 ) − F (x0 , y0 ) − Dx F (x0 , y0 )h = o1 (h), F (x0 , y0 + k) − F (x0 , y0 ) − Dy F (x0 , y0 )k = o2 (k) if they exist. The operators Dx F (x0 , y0 ) and Dy F (x0 , y0 ) are the partial derivatives of F relative to X and Y , respectively. Theorem 10.10. (The Implicit Function Theorem) Let X, Y, Z be three Banach spaces, W be an open subset in X × Y , and let F : W −→ Z. Suppose that (x0 , y0 ) ∈ W and the following conditions are satisfied: (i) F is continuous at (x0 , y0 ), (ii) F (x0 , y0 ) = 0, (iii) Dy F exists in W , (iv) Dy F is continuous at (x0 , y0 ), and (v) Dy F (x0 , y0 ) is invertible. There exists a neighborhood V of x0 and a unique function f : V −→ Y such that F (x, f (x)) = 0, f (x0 ) = y0 , f is continuous at x0 . Proof. The conditions imposed in the function F are satisfied in (x0 , y0 ), if and only if the same conditions are satisfied in (0, 0) by the function H : W0 −→ Z defined by H(x, y) = F (x − x0 , y − y0 ). Therefore, without loss of generality we may assume that (x0 , y0 ) = (0, 0). There exists a positive number δ0 such that {(x, y) | x  δ0 , y  δ0 } ⊆ W. Observe that Dy F (0, 0) ∈ Hom(Y, Z) and (Dy F )−1 (0, 0) ∈ Hom(Z, Y ) by (v). For x, y such that x  δ0 and y  δ0 define Gx (y) = y − (Dy F )−1 (0, 0)F (x, y). For the derivative of Gx (y) relative to y we have Dy (Gx (y)) = I − (Dy F )−1 (0, 0) Dy F (x, y) = (Dy F )−1 (0, 0)((Dy F )(0, 0) − Dy F (x, y)). Since Dy F is continuous at (0, 0), by (iv) there exists a positive δ1 such that x  δ1 and y  δ1 implies Gx (y)  12 . We have Dy (Gx )(0) = −(Dy F )−1 (0, 0)F (x, 0) = −(Dy F )−1 (0, 0)(F (x, 0) − F (0, 0)).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

page 641

641

If δ = min{δ0 , δ1 } and is a positive number such that < δ, by the continuity of F at (0, 0), there exists δ < δ such that x  δ implies Gx (0) < 2 . If x  δ and y  , then Gx (y)  Gx (0) + Gx (y) − Gx (0)  + sup{Dy (Gx )(λy)y | λ ∈ [0, 1]} 2  + = . 2 2 Thus, for x  δ Gx is a transformation of the closed sphere B[0, ] and Gx (y) < 1. By Lemma 10.2, Gx has a unique fixed point y in B[0, ]. Since y depends on x, we can define the function f : B[0, ] −→ Y as y = f (x) such that F (x, f (x)) = 0. Since F (0, 0) = 0, it follows that G0 (0) = 0. We have 0 = f (0) by the uniqueness of y. For each ∈ (0, δ) there exists a δ such that x  δ implies Gx (0)  2 . Thus, x  δ implies f (x)  , which shows that f is continuous. Suppose that f˜ is another function defined on a neighborhood of 0 such that f˜ is continuous at 0, f˜(0) = 0 and F (x, f˜(x)) = 0. If 0 < < δ, there exists a positive number θ such that θ < δ , and x  θ implies f˜(x)  , so f˜(x) ∈ B[0, ]. From the uniqueness of the fixed point for  Gx it follows that f (x) = f˜(x) when x  θ. Another variant of the Implicit Function Theorem presented in [41] is obtained by replacing condition (i) of Theorem 10.10 with the continuous differentiability. Theorem 10.11. Let X, Y, Z be three Banach spaces, W be an open subset in X ×Y , and let F : W −→ Z. Suppose that (x0 , y0 ) ∈ W and the following conditions are satisfied: (i) F is continuously differentiable in W , (ii) F (x0 , y0 ) = 0, (iii) Dy F exists in W , (iv) Dy F (x0 , y0 ) is invertible. Then, there exists a neighborhood V of x0 such that f is unique and (Df )(x) = −(Dy F (x, f (x)))−1 Dx F (x, f (x)). Proof. The existence of a neighborhood V of x0 and a unique function f : V −→ Y such that F (x, f (x)) = 0, f (x0 ) = y0 , f is continuous at x0

May 2, 2018 11:28

642

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 642

Mathematical Analysis for Machine Learning and Data Mining

follows from Theorem 10.10. Differentiating the equality F (x, f (x)) = 0 and applying the chain rule yields (Dx F )(x, f (x))(Dx f )(x) + (Dy F )(x, f (x)) = 0, which implies (Dx f )(x) = −(Dy F (x, f (x)))−1 (Dx F )(x, f (x)).



Example 10.13. Let us consider a special case of the implicit function theorem that is important for many applications. Suppose that X = Rn and Y = Z = R in Theorem 10.11, W is an open subset in Rn × R, and F : W −→ R is a function that satisfies the conditions of this theorem in (x0 , y0 ) ∈ Rn × R. In other words, F is continuously differentiable in W , F (x0 , y0 ) = 0, and ∂F ∂y (x0 , y0 ) = 0. By Theorem 10.11, there exists a neighborhood V of x0 and a unique function f : V −→ Y such that F (x, f (x)) = 0, f (x0 ) = y0 , f is continuous at x0 and 1 (Dx F )(x, g(x)). ∂z (x, g(x))

(Df )(x0 ) = − ∂F

Theorem 10.12. Let X, Y be two Banach spaces, U be an open set in X, and let f : U −→ Y be a continuously differentiable function on U . If x0 ∈ U and > 0, then there exists δ > 0 such that if x1 , x2 ∈ B(x0 , δ), then f (x2 ) − f (x1 ) − f  (x0 )(x2 − x1 )  x2 − x1 . Proof. Since the mapping φ : X −→ Hom(X, Y ) given by φ(x) = (Df )(x) is continuous, for > 0 there exists δ1 > 0 such that x − x0  < δ1 implies (Df )(x) − (Df )(x0 ) < . Since U is an open set and x0 ∈ U , there exists δ2 > 0 such that B(x0 , δ2 ) ⊆ U . Let δ = min δ1 , δ2 . Let g : U −→ Y be the function given by g(x) = f (x) − (Df )(x0 )x for x ∈ U . Since (Dg)(x) = (Df )(x) − (Df )(x0 ), by Theorem 10.8 we have: g(x2 ) − g(x1 )  x2 − x1  sup{(Df )(x) − (Df )(x0 ) | x ∈ [x1 , x2 ]}, that is f (x2 ) − f (x1 ) − (Df )(x0 )(x2 − x1 )  x2 − x1 .



Theorem 10.13. Let X, Y be two Banach spaces, U be an open set in X, and let f : U −→ Y be a continuously differentiable function on U . If x0 ∈ U and (Df )(x0 ) has a right inverse in Hom(X, Y ), then f (U ) is a neighborhood of y0 = f (x0 ).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

page 643

643

Proof. Since (Df )(x0 ) has a right inverse, there exists ∈ Hom(Y, X) such that (Df )(x0 ) = 1Y . Let c =  . 1 there exists δ > 0 such that B(x0 , δ) ⊆ By Theorem 10.12, taking = 2c U and u, v ∈ B(x0 , δ) imply 1 u − v. 2c We prove that f (U ) is a neighborhood of y0 = f (x0 ) by showing that  δ , f (U ) includes the open sphere B y 0 2c , that is, by showing that for y ∈  δ B y0 , 2c there exists x ∈ U such that y = f (x). Define inductively a sequence (xn ) as follows. The initial member is x0 ∈ U , x1 = x0 + (y − y0 ) and f (u) − f (v) − (Df )(x0 )(u − v) 

xn+1 = xn − (f (xn ) − f (xn−1 ) − (Df )(x0 )(xn − xn−1 )) for n  1. We prove by induction on n that xn − xn−1   for n  1. For the base case, n = 1, we have:

δ 2n

x1 − x0  =  (y − y0 )  cy − y0   c

and xn ∈ B(x0 , δ)

δ δ = . 2c 2

Suppose that the above claims hold for n. We have: xn+1 − xn  =  (f (xn ) − f (xn−1 ) − (Df )(x0 )(xn − xn−1 ))   cf (xn ) − f (xn−1 ) − (Df )(x0 )(xn − xn−1 ) δ 1  c · xn − xn−1   n+1 2c 2 (by the inductive hypothesis). Also, we have: xn+1 − x0   xn+1 − xn  + xn − xn−1  + · · · + x0 − x1    1 1 1 δ + n + ···+  δ, 2n+1 2 2 so xn+1 ∈ B(x0 , δ). Moreover, (xn ) is a Cauchy sequence because for m, n ∈ N xn − xm   xn − xn−1  + · · · + xm+1 − xm  δ δ δ  n+1 + n + · · ·  n . 2 2 2 Since X is complete, there exists x = limn→∞ xn . Since xn ∈ B(x0 , δ), it follows that x − x0   δ and x ∈ U .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 644

Mathematical Analysis for Machine Learning and Data Mining

644

By the definition of xn+1 , xn+1 = xn − (f (xn ) − f (xn−1 ) − (Df )(x0 )(xn − xn−1 )) we have (Df )(x0 )(xn+1 − xn ) = −(Df )(x0 ) (f (xn ) − f (xn−1 ) − (Df )(x0 )(xn − xn−1 ) = (Df )(x0 )(xn − xn−1 ) − (f (xn ) − f (xn−1 ). By a repeated application of this equality for n, n − 1, . . . , 1 we have: (Df )(x0 )(xn+1 − xn ) = (Df )(x0 )(xn − xn−1 ) − (f (xn ) − f (xn−1 ) (Df )(x0 )(xn − xn−1 ) = (Df )(x0 )(xn−1 − xn−2 ) − (f (xn−1 ) − f (xn−2 ) .. . (Df )(x0 )(x2 − x1 ) = (Df )(x0 )(x1 − x0 ) − (f (x1 ) − f (x0 ), hence (Df )(x0 )(xn+1 − xn ) = (Df )(x0 )(x1 − x0 ) − (f (xn ) − f (x0 )) = (Df )(x0 )(x1 − x0 ) − (f (xn ) − y). Recall that f (x0 ) = y0 and x1 = x0 + (y − y0 ). Thus, we have (Df )(x0 )(xn+1 − xn ) = (Df )(x0 ) (y − y0 ) − (f (xn ) − y0 ) = y − f (xn ). Taking n → ∞ in the above equality we have y = f (x) ∈ f (U ).



Theorem 10.14. Let X be a Banach space, U be an open set in X, and let f : U −→ Rn be a vector-valued continuously differentiable function on U . If x ∈ U and (Df)(x) is a surjective mapping in Hom(X, Rn ), then then f(U ) is a neighborhood of y0 = f(x0 ). Proof. Since (Df )(x) is a surjective mapping there exist u1 , . . . , un ∈ U such that (Df)(x)ui = ei for 1  i  n, where {e1 , . . . , en } is the standard basis of Rn . Define the linear mapping : Rn −→ X by (ei ) = ui for  i  n. We have (Df)(x) (ei ) = ei for 1  i  n, so (Df)(x) = 1Y , so (Df)(x) has a right inverse. The statement then follows from Theorem 10.13.  Corollary 10.3. Let X be a Banach space, U be an open set in X, and let f : U −→ Y , be a vector-valued continuously differentiable function on U , where Y is a finitely dimensional Banach space. If x ∈ U and (Df )(x) is a surjective mapping in Hom(X, Y ), then f (U ) is a neighborhood of y0 = f (x0 ).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

page 645

645

Proof. This statement is a direct consequence of Theorem 10.14 because  a finitely dimensional Banach space is homeomorphic to a space Rn . We discuss now differentiability of functions whose sets of values are subsets of sets of the form Rm . Such functions will be denoted by bold letters. If f : X −→ Rm , where X ⊆ Rn , the components of f will be denoted by f1 , . . . , fm . Definition 10.5. Let f : X −→ Rm , where X ⊆ Rn is an open set and x0 ∈ X. If the limit lim t↓0

f(x0 + tu) − f(x0 ) t

exists, then we refer to it as the directional derivative of f in x0 with respect ∂f to u and we denote this limit as ∂u (x0 ). If f is Gˆateaux differentiable in x0 , then the directional derivative of f exists with respect to every direction u and we have ∂f (x0 ) = (Df)(x0 )(u). ∂u The partial derivative of f : X −→ Rm relative to xi is (Df)(x0 )(ei ). Unlike the case of single-argument functions, where the existence of a derivative at a point x0 implies the continuity in x0 , the existence of partial derivatives of a multi-argument function does not imply its continuity. Example 10.14. Let f : R2 −→ R be the function defined by ⎧ 2 2 ⎪ ⎨cos π x1 − x2 if x = 0 , 2 2 x21 + x22 f (x) = ⎪ ⎩0 if x = 02 . 1 1 Note that limn→∞ f n , n = 1, while f (02 ) = 0, so f is not continuous in 02 . However, partial derivatives do exist in 02 because ∂f f (x1 , 0) f (0, x2 ) ∂f (02 ) = lim = 0 and (02 ) = lim = 0. x1 →0 x2 →0 ∂x1 x1 ∂x2 x2 ∂f ∂f If ∂u exists relative to every direction u, then ∂x exists for every i, i 1  i  n. The converse is not true, as the next example shows.

Example 10.15. Consider the function f : Rn −→ R defined by  a if x = aei for some i, f (x) = 1 otherwise.

May 2, 2018 11:28

646

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 646

Mathematical Analysis for Machine Learning and Data Mining

We have

f (tei ) − f (0) ∂f =1 (0) = lim t→0 ∂xi t for 1  i  n. However, if u ∈ {e1 , . . . , en }, then f (tu) − f (0) 1 = t t ∂f 1 (0) does not exists if and limt→0 t does not exist, which shows that ∂u u ∈ {e1 , . . . , en }. Theorem 10.15. Let f : Rn −→ R be a functional that has continuous partial derivatives in x0 with respect to each variable xi . Then f is Fr´echet differentiable in x0 and its Fr´echet differential is given by  n   ∂f δf (x0 ; h) = (x0 )hi . ∂xi i=1

Proof. Since the partial derivatives of f are continuous, there exists a sphere B(x0 , ) such that    ∂f  ∂f    ∂xi (x0 ) − ∂xi (y) < n n for all y ∈ B(x0 , ). Let h = i=1 hi ei and let r0 , r1 , . . . , rn be a sequence k of vectors defined by r0 = 0n , and rk = i=1 hi ei . Clearly, rk   h for 0  k  n, and x0 + rk = x0 + rk−1 + hk ek . Observe that by the mean value theorem for functions of one argument we have ∂f (x0 + rk−1 + αk ek )hk f (x0 + rk ) − f (x0 + rk−1 ) = ∂xk for some αk ∈ [0, hk ]. We have x0 + rk−1 + αk ek ∈ S(x0 , δ) if h < δ. This implies     f (x0 + rk ) − f (x0 + rk−1 ) − ∂f (x0 )hk  < h.  n  ∂xk Therefore,   n    ∂f   (x0 )hi  f (x0 + h) − f (x0 ) −   ∂x i i=1  n     ∂f   (x0 )hk  f (x0 + rk ) − f (x0 + rk−1 ) − =   ∂xk k=1   n    f (x0 + rk ) − f (x0 + rk−1 ) − ∂f (x0 )hk  < h,    ∂xk k=1

which concludes the argument.



May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

page 647

647

Starting with a vector h ∈ Rn and the symbolic vector ∇ we consider a symbolic “scalar product” h ∇ defined by h ∇ = h 1

∂ ∂ + · · · + hn , ∂x1 ∂xn

which is a differential operator that can be applied to a functional f : Rn −→ R. The formula in Theorem 10.15 can now be written in a condensed form as δf (x0 ; h) = ((h ∇)f )(x0 ). Let f : Rn −→ Rm be a vector-valued function that is differentiable at x0 and let rx0 : Rn −→ Rm be the function defined by: rx0 (u) = f(x0 + u) − f(x0 ) − (Df)(x0 )u. Then, we have lim

u→0n

rx0 (u) = 0. u

We have seen that for a function f : Rn −→ Rm that is Gˆ ateaux differentiable at x0 , the vector (Dx f)(x0 )(u) is the product of the Jacobian matrix of f computed at x0 and u. We will consider first two simple examples. Example 10.16. Let h : Rn −→ Rm be the linear operator defined by h(x) = Ax, where A ∈ Rm×n . By the definition of the Gˆateaux derivative of h we can write h(x0 + tu) − h(x0 ) (Dh)(x0 )(u) = lim t→0 t A(x0 + tu) − Ax0 = lim t→0 t tAu = Au, = lim t→0 t hence (Dh)(x0 ) = A for every x0 ∈ Rn . Suppose now that, as before, h(x) = Ax, and x = g(z), where g : n R −→ Rp is a Gˆateaux differentiable function. An application of the chain rule yields the formula (Dz h)(z0 ) = A(Dz g)(z0 ). Example 10.17. Let f : Rm × Rn −→ R be the functional defined by f (x, y) = y Ax, where y ∈ Rm , A ∈ Rm×n , and x ∈ Rn . We have: (Dx f )(x0 , y0 ) = y0 Aand(Dy f )(x0 , y0 ) = x0 A .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 648

Mathematical Analysis for Machine Learning and Data Mining

648

By the definition of the Gˆateaux differential, (Dx f )(x0 , y0 ) is given by f (x0 + tu, y0 ) − f (x0 , y0 ) (Dx f )(x0 , y0 ) = lim t→0 t y0 A(x0 + tu) − y0 Ax0 = lim t→0 t ty0 Au = lim t→0 t = y0 Au for u ∈ Rn . Therefore, since the limit of the fraction when t → 0 is 0, we have (Dx f )(x0 ) = y0 A. Similarly, the fraction that enters in the definition of (Dx f )(x0 , y0 ) is f (x0 , y0 + tv) − f (x0 , y0 ) (Dy f )(x0 , y0 )(v) = lim t→0 t (y0 + tv )Ax0 − y0 Ax0 = lim t→0 t tv Ax0 = lim = v Ax0 . t→0 t Since v Ax0 is a scalar, it is equal to its own transpose, that is v Ax0 =   x0 A v and we have: (Dy f )(x0 , y0 ) = x0 A . Theorem 10.16. Let h : Rn −→ Rm and g : Rn −→ Rm be two differentiable operators in z0 ∈ Rn . We have Dz (g(z) h(z))(z0 ) = h(z0 ) (Dz g)(z0 ) + g(z0 ) (Dz h)(z0 ). Proof.

Let k : Rn −→ R be defined as k(z) = g (z)f(z). We have m  gi (z)fi (z) k(z) = i=1

for z ∈ Rn . Then, m m   ∂k ∂gi ∂fi (z0 ) = (z0 )fi (z0 ) + gi (z0 ) (z0 ) ∂zj ∂zj ∂zj i=1 i=1

for 1  j  n. This is equivalent to the desired equality.



Corollary 10.4. For h : Rn −→ Rm be a differentiable operator and let f : Rn −→ R be given by f (z) = h(z) h(z). We have (Df )(z0 ) = 2((Dh)(z0 )) ((Dh)(z0 )). Proof.

This follows immediately from Theorem 10.16.



May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

page 649

649

Theorem 10.17. Let A ∈ Rm×p and let h : Rn −→ Rm and g : Rn −→ Rp be two functions that are differentiable at z0 ∈ Rn . Define the functional f : Rn −→ R as f (z) = h(z) Ag(z). We have: (Df )(z0 ) = g(z0 ) A (Dh)(z0 ) + h(z0 ) A(Dg)(z0 ). Proof. Let l : Rn −→ Rp be defined by l(z) = A h(z). We have f (z) = l(z) g(z). By Theorem 10.16 we have: Dz (l(z) g(z))(z0 ) = h(z0 ) (Dz l)(z0 ) + l(z0 ) (Dz g)(z0 ). By Theorem 10.16 we have: (Dz l)(z0 ) = A (Dz h)(z0 ), which implies Dz (l(z) g(z))(z0 ) = h(z0 ) A (Dz h)(z0 ) + A h(z)(Dz g)(z0 ).

10.3



Taylor’s Formula

Let X be an open subset of Rn . The function f : X −→ R belongs to the class C k (X) if it has continuous partial derivatives of order up to and including k. Under certain conditions of differentiability of a function f : X −→ R, Taylor’s formula shows the existence of a polynomial that provides an approximation of f . If X is an interval in R, x0 ∈ X, and f : X −→ R belongs to C k (X) define the Taylor polynomial of degree k Tk (x) as x − x0  (x − x0 )k (n) f (x0 ) + · · · + f (x0 ). 1! k! The function Rk : X −→ R defined as Rk (x) = f (x) − Tk (x) is the remainder of order k for f . The derivatives of Tk (x) are given by Tk (x) = f (x0 ) +

x − x0  (x − x0 )k−1 (k) f (x0 ) + · · · + f (x0 ) 1! (k − 1)! x − x0  (x − x0 )k−1 (k) Tk (x) = f  (x0 ) + f (x0 ) + · · · + f (x0 ) 1! (k − 1)! .. . x − x0 (k) k−1 f (x0 ) Tk (x) = f (k−1) (x0 ) + 1! Tkk (x) = f (k) (x0 ). Tk (x) = f  (x0 ) +

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 650

Mathematical Analysis for Machine Learning and Data Mining

650

Therefore, (k)

(k+1)

Tk (x0 ) = f (x0 ), Tk (x0 ) = f  (x0 ), . . . , Tk (x0 ) = f (k−1) (x0 ), Tk

(x0 ) = 0.

The definition of Rk implies that this function also belongs to C k (X) and (k−1)

Rk (x0 ) = Rk (x0 ) = · · · = Rk

(x0 ) = Rkk (x0 ) = 0.

Since Rk is continuous on X we have limx→x0 Rk (x) = Rk (x0 ) = 0. Moreover, the following statement holds. The next lemma establishes Taylor’s formula in equality (10.3). Lemma 10.3. Let X be an open interval of Rn and let f : X −→ R be a function in C k (X). We have x − x0  f (x0 ) f (x) = f (x0 ) + 1! (x − x0 )k (k) (x − x0 )2  f (x0 ) + · · · + f (x0 ) + Rk (x) (10.1) + 2! k! for every x ∈ X and limx→x0 Proof.

Rk (x) (x−x0 )k

= 0.

Let g : X −→ R be defined as g(x) = (x − x0 )k . Note that g(x0 ) = g  (x0 ) = · · · = g (k−1) (x0 ) = 0 and g (k) (x0 ) = k!.

For x ∈ X, an application of Cauchy’s Theorem from elementary calculus to the functions Rn and g yields the existence of c1 between x0 and x such that Rk (x) R (c1 ) Rk (x) − Rk (x0 ) = = k . g(x) − g(x0 ) g(x) g (c1 ) Since Rk (x0 ) = g  (x0 ) = 0, applying again Cauchy’s Theorem we obtain R (c ) R (c ) the existence of c2 between x0 and c1 such that gk(c11) = gk(c22) . Thus, R (c2 ) Rk (x) − Rk (x0 ) = k . g(x) − g(x0 ) g (c2 ) Continuing this process, we obtain the existence of a point ξ in between x0 and x, but distinct from both such that (k−1)

(k−1)

(k−1)

Rk R (ξ) (ξ) − Rk (x0 ) Rk (x) − Rk (x0 ) = (k−1) = k(k−1) g(x) − g(x0 ) g (ξ) g (ξ) − g (k−1) (x0 ) (k−1)

=

(k−1)

(ξ)−Rk (x0 ) ξ−x0 g(k−1) (ξ)−g(k−1) (x0 ) ξ−x0

Rk

.

Let (xm )m1 be an arbitrary sequence in X such that limm→∞ xm = x0 . For each xm in this sequence denote by ξm the point located between x0 and

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

page 651

651

xm constructed as previously shown. Clearly, ξm = x0 and limm→∞ ξm = x0 . Therefore, we have (k−1)

(k−1)

(ξm ) − Rk (x0 ) (k) = Rk (x0 ) = 0, m→∞ ξm − x0 g (k−1) (ξm ) − g (k−1) (x0 ) = g (k) (x0 ) = k!, lim m→∞ ξm − x0 lim

Rk

which implies limx→x0

Rk (x) (x−x0 )k

= 0.



Theorem 10.18. If X is an interval in R, x0 ∈ X, and f : X −→ R belongs to C k (X), there exists a function ρ : X −→ R such that x − x0  (x − x0 )2  f (x0 ) + f (x0 ) 1! 2! (x − x0 )k (k) (x − x0 )k f (x0 ) + ρ(x0 ) +···+ k! k! for every x ∈ X and limx→x0 ρ(x) = 0. f (x) = f (x0 ) +

Proof.

By applying Lemma 10.3 we can define ⎧ ⎨ Rk (x) k! if x = x , 0 ρ(x) = (x − x0 )k ⎩ 0 if x = x0 .

Thus, by replacing Rk (x) in equality (10.1) to obtain the desired result.  If we assume that f ∈ C (k+1) (X) we can give other forms of the remainder in Taylor’s formula. Let φ : X −→ R be a function given by φ(t) = f (t) +

(x − t)k (k) x−t  f (t) + · · · + f (t) + (x − t)p Q, 1! k!

where Q ∈ R. Since f ∈ C (k+1) (X) the function φ has a derivative in each  0 point of X. We have φ(x) = f (x) and φ(x0 ) = f (x0 ) + x−x 1! f (x0 ) + · · · + (x−x0 )k (k) f (x0 ) k!

+ (x − x0 )p Q If we choose Q be such that φ(x0 ) = φ(x), then we can apply Rolle’s theorem to φ on the interval determined by x0 and x. There exists c such that φ (ξ) = 0, which amounts to (x − ξ)k (k+1) f (ξ) − p(x − ξ)p+1 Q = 0, k! hence K =

(x−ξ)k−p+1 (k+1) f (ξ). pk!

Rk (x) =

This allows to write

(x − x0 )p (x − ξ)k−p+1 (k+1) f (ξ). pk!

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 652

Mathematical Analysis for Machine Learning and Data Mining

652

For p = 1 we have the Cauchy remainder, Rk (x) =

(x − x0 )(x − ξ)k (k+1) f (ξ), pk!

while for p = k + 1 we obtain the Lagrange1 remainder: Rk (x) =

(x − x0 )k+1 (k+1) f (ξ). (k + 1)!

Corollary 10.5. Let X be an interval of R and let f : X −→ R be a function such that f ∈ C (k+1) (X). For the Lagrange remainder of the Taylor formula x − x0  (x − x0 )2  (x − x0 )k (k) f (x0 ) + f (x0 ) + · · · + f (x0 ) 1! 2! k! (x − x0 )k+1 (k+1) f + (ξ) (k + 1)!

f (x) = f (x0 ) +

we have Rk (x) = O((x − xk )k+1 ). Proof.

This statement follows immediately from the previous discussion. 

To extend Taylor’s formula to functions defined on Rn we need to use symbolic powers of the operator h ∇ that act as operators on real-valued functions of n variables. These powers are defined inductively by (h ∇)0 f = f , and (h ∇)k+1 f = (h ∇)((h ∇)k f ) for k ∈ N. Example 10.18. For

 ∇=

# $ and h =

h1 h2

∂ ∂x1 ∂ ∂x2



we have h ∇ = h 1

∂ ∂ + h2 ∂x1 ∂x2

1 Joseph Louis Lagrange, a French-Italian mathematician was born at Turin on January 25, 1736, and died at Paris on April 10, 1813. He made major contributions to mathematics and is the creator of analytical mechanics and of the calculus of variations. ´ In 1795 Lagrange was appointed to a mathematical chair at the Ecole normale and was ´ a professor at the Ecole polytechnique in starting in 1797.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

page 653

653

∂f ∂f and (h ∇)f = h1 ∂x + h2 ∂x . Note that 1 2   ∂f ∂f + h2 (h ∇)2 f = (h ∇) h1 ∂x1 ∂x2     ∂ ∂f ∂f ∂ ∂f ∂f = h1 + h2 + h2 h1 + h2 h1 ∂x1 ∂x1 ∂x2 ∂x2 ∂x1 ∂x2 2 2 2 ∂ f ∂ f ∂ f = h21 + 2h1 h2 + h22 . ∂x1 2 ∂x1 ∂x2 ∂x2 2

The mth iteration of the operator h ∇ can be computed using the multinomial formula    m ∂m (h ∇)m = hp11 hp22 · · · hpkk p1 p2 p1 p2 · · · pk ∂x1 x2 ∂ · · · xpkk   k    pi = m . p1 , p2 , . . . , pk ∈ N,  i=1

Theorem 10.19. (Taylor’s Formula) Let f : B(x, r) −→ R be a function that belongs to the class C n (B(x, r)), where B(x, r) ⊆ Rk . If h ∈ Rk is such that h < r then there exists θ ∈ (0, 1) such that: f (x + h) =

n−1 

1 1 ((h ∇)m f )(x) + ((h ∇)n f )(x + θh). m! n! m=0

(10.2)

Proof. Define the function g : (0, 1) −→ R by g(a) = f (x + ah). Since f ∈ C n (B(x, r)) it follows that g belongs to the class C n ([0, 1]). Thus, by the standard Taylor formula for g there exists θ ∈ (0, 1) such that g(1) =

n−1 

1 (m) 1 g (0) + g (n) (θ). m! n! m=0

(10.3)

Note that differentiating the function g(a) = f (x + ah) with respect to a is the same thing as applying the differential operator h ∇ to f . Indeed, we have g  (a) = (f (x1 + ah1 , . . . , xn + ahn )) ∂f ∂f = h1 (x + ah) + · · · + hn (x + ah) ∂x1 ∂xn = ((h ∇)f )(x + ah). It follows immediately that g (m) (a) = ((h ∇)m f )(x + ah)

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 654

Mathematical Analysis for Machine Learning and Data Mining

654

for m ∈ N, so by equality (10.3) there exists θ ∈ (0, 1) such that n−1 

1 1 ((h ∇)m f )(x) + ((h ∇)n f )(x + θh). m! n! m=0 The equality proven in Theorem 10.19 can be written as: f (x + h) =

f (x + h) =



n  1 ((h ∇)m f )(x) m! m=0

1 (((h ∇)n f )(x + θh) − ((h ∇)n f )(x)) n! n  1 ((h ∇)m f )(x) + Rn (x, h), = m! m=0 +

where Rn (x, h), the remainder of order n is defined by Rn (x, h) =

1 (((h ∇)n f )(x + θh) − ((h ∇)n f )(x)). n!

Let f ∈ C n (B(x, r)). We show that the remainder Rn can be written as Rn (x, h) =

1 hn ω(x, h), n!

where limh→0k ω(x, h) = 0. The coefficient of hpi11 · · · hpikk in (h ∇)n is ∂n n! . p1 p1 ! · · · pk ! ∂x1 · · · ∂xpkk Since the partial derivatives are continuous, it follows that ∂n ∂n f (x + θh) = f (x). p p h→0 ∂x11 · · · ∂xkk ∂xp11 · · · ∂xpkk lim

Define ωp1 ···pk (x, h)   ∂n ∂n n! f (x + θh) − f (x) , = p1 ! · · · pk ! ∂xp11 · · · ∂xpkk ∂xp11 · · · ∂xpkk where limh→0 ωp1 ···pk f (x, h) = 0. Also, let ω(x, h) =

1 hn

 (p1 ,...,pk )

ωp1 ....,pk (x, h)hp11 · · · hpkk .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

page 655

655

These notations allow us to write: 1 Rn (x, h) = (((h ∇)n f )(x + θh) − ((h ∇)n f )(x)) n! 1  hp11 · · · hpkk ωp1 ···pk (x, h) = n! (p1 ,...,pk )

=

hn ω(x, h). n!

Example 10.19. For the real-valued function f ∈ Theorem 10.19 implies the existence of θ ∈ (0, 1) such that f (x + h) = f (x) + ((h ∇)f )(x) + ((h ∇)2 f )(x) +

C 2 (B(x, r)),

h2 ω(x, h), 2

where limh→0k ω(x, h) = 0. The matrix-valued function Hf : Rk −→ Rk×k defined by   ∂2f Hf (x) = ∂xi1 ∂xi2 is the Hessian2 matrix of f . Example 10.20. Let f : R2 −→ R be the function defined by 1 . f (x) = 2 x1 + x22 − 1 The Taylor formula for n = 2 and x = 02 is 1 f (h) = f (02 ) + ((h ∇)f )(02 ) + ((h ∇)2 f )(θh). 2 The partial derivatives of f are: ∂f 2x1 ∂f 2x2 =− 2 and =− 2 . 2 2 ∂x1 (x1 + x2 − 1) ∂x2 (x1 + x22 − 1)2 The second order derivatives are ∂2f 2(3x21 − x22 + 1 = ∂x1 2 (x21 + x22 − 1)3 2 ∂ f ∂2f 8x1 x2 = = 2 ∂x1 ∂x2 ∂x2 ∂x1 (x1 + x22 − 1)3 ∂2f 2(3x22 − x21 + 1 = . ∂x2 2 (x21 + x22 − 1)3 2 Ludwig Hesse was born on April 22nd 1811 in K¨ onigsberg, Prussia (today Kaliningrad), and died on August 4th 1876 in Munich. He worked mainly on algebraic invariants, and geometry, and taught in several unversities in K¨ onigsberg, Halle, Heidelberg and Munich. The Hessian matrix is named after him.

May 2, 2018 11:28

656

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 656

Mathematical Analysis for Machine Learning and Data Mining

Taking into account that f (02 ) = −1, the Taylor formula can be written:    2 ∂2f ∂2f θ1 h 1 2 ∂ f f (h) = −1 + h21 + 2h h + h 1 2 2 θ2 h 1 ∂x1 2 ∂x1 ∂x2 ∂x2 2 2(3θ2 h21 − θ2 h22 + 1 (θ2 h21 + θ2 h22 − 1)3 2 2 2 2 8θ2 h1 h2 2 2(3θ h2 − θ h1 + 1 +2h1 h2 2 2 + h . 2 (θ h1 + θh22 − 1)3 (θ2 h21 + θ2 h22 − 1)3

= −1 + h21

Definition 10.6. Let f : U −→ R, where U ⊆ Rk . A point x0 ∈ U is a local minimum for f if there exists V ∈ neighx0 (O) such that f (x)  f (x0 ) for every x ∈ V . A point x0 ∈ U is a local maximum for f if there exists V ∈ neighx0 (O) such that f (x)  f (x0 ) for every x ∈ V . A point is a local extremum for f if it either a local minimum or a local maximum for f . Theorem 10.20. Let f : U −→ R, where U ⊆ Rk is an open set and f ∈ C 1 (U ). If x0 ∈ U is a local extremum, then (Df )(x0 ) = 0k . Proof. Suppose that x0 is a local maximum for f . Then, for any h ∈ Rk , the function g : R −→ R defined by g(t) = f (x0 + th) has a local minimum at t = 0. Thus, g  (0) = 0. By the Chain Rule, g  (0) = (Df )(x0 ) h for every h, hence (Df )(x0 ) = 0k . The argument for the case is a local minimum is similar.  In other words, if x0 is a local extremum of f , we have (∇f )(x0 ) = 0k . Definition 10.7. Let f : B(x, r) −→ R be a function that belongs to the class C n (B(x, r)), where B(x, r) ⊆ Rk and n  1. A point x0 is a critical point, or a stationary point if (Dx f )(x0 ) = 0k . The converse of Theorem 10.20 is false, that is, we may have (Df )(x0 ) = 0k without x0 being an extreme point. If x0 is a critical point of f but not a local extremum we say that x0 is a saddle point for f . Example 10.21. Consider the function f : R2 −→ R defined by f (x) = x21 − 2x1 x2 + x22 . We have     2(x1 − x2 ) 1 (∇f )(x) = = 2(x1 − x2 ) . 2(x2 − x1 ) −1 The critical points are located on the first bisecting line x1 − x2 = 0 and they are clearly local minima for f .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Differentiability of Functions Defined on Normed Spaces

b3234-main

page 657

657

Example 10.22. Let f : R2 −→ R be defined by f (x) = x21 − x22 + 2x1 x2 . We have   2(x1 + x2 ) . (∇f )(x) = 2(x1 − x2 ) This function has a unique critical point, namely 02 , which is a saddle point because f (02 ) = 0 and the function takes both positive and negative values in a neighborhood of 02 . Next we discuss a sufficient condition for the existence of a local extremum for a function using the Hessian matrix. Theorem 10.21. Let U be an open subset of Rk , f : U −→ R be a function in C 2 (U ). Then f is convex on U if and only if the Hessian matrix Hf (x) is positive on U ; if Hf (x) is negative on U , then f is concave on U . Proof. By Theorem 12.19, f is convex if and only if the function φx,h : R −→ R given by φx,h (t) = f (x + th) is convex, which happens only if φx,h (t)  0. Note that by the Chain Rule we have n  ∂f (x + th)hi , ∂x i i=1  n   ∂f  (x + th)hi φx,h (t) = ∂xi i=1

φx,h (t) =

=

n  n  j=1 i=1

=

n  n  j=1 i=1

∂2f (x + th)hj hi ∂xi ∂xj hj

∂2f (x + th)hi ∂xi ∂xj

= h Hf (x + th)h. If Hf (x + th) is positive, then every φx,h (t)  0, hence every φx,h (t) is convex, which means that f is convex. Conversely, suppose that f is convex on U . Let x ∈ U and let h be an arbitrary vector. Since U is open, there exists t0 > 0 such that x + t0 h ∈ U . Since f is convex, φx,t0 h is convex and φx,t0 h (0)  0. Therefore, 0  φx,t0 h (0) = t20 h Hf (x)h, hence h Hf (x)h  0. Therefore, Hf (x) is positive. The argument for concave functions is similar.



May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 658

Mathematical Analysis for Machine Learning and Data Mining

658

Theorem 10.22. Let U be an open subset of Rk , f : U −→ R be a function in C 2 (U ), and let x0 be a critical point of f . If the Hessian Hf (x0 ) is positive definite, then x0 is a local minimum of f . If Hf (x0 ) is negative definite, then x0 is a local maximum of f . Proof.

10.4

This follows immediately from Theorem 10.21.



The Inverse Function Theorem in Rn

The inverse function theorem for functions of the form f : U −→ Rn , where U is an open subset of Rn , has a local character; in other words, it states that under certain conditions, a function can be inverted in the neighborhood of a point x0 ∈ U . Example 10.23. Let f : R2 −→ R2 be the function defined by   x1 e cos x2 . f(x) = ex1 sin x2 Since f is a periodic function in its second argument, it is clear that f is not injective. However, as we shall see we can invert f in the neighborhood of a point x0 . Lemma 10.4. Let f : Rn −→ Rn be a continuously differentiable function on B(x0 , r) such that (Df)(x) is invertible when x ∈ B(x0 , r). There exists s such that 0 < s  r such that f is injective in B(x0 , s). Proof.

Let γy : B(x0 , r) −→ Rn be the function defined by γy (x) = x + ((Df)(x0 ))−1 (y − f(x)).

Note that gy (x) = x is equivalent to y = f(x). Also, (Dγy )(x) = In −((Df)(x0 ))−1 (Df)(x) = ((Df)(x0 ))−1 ((Df)(x0 )−(Df)(x)). Since (Df)(x) is continuous at x0 , there exists a positive number s such that if x ∈ B(x0 , s), (Df)(x) is invertible and 1 (Dγy )(x)  ((Df)(x0 ))−1  · (Df)(x0 ) − (Df)(x)  . 2 Therefore, for x1 , x2 ∈ B(x0 , s) we have γy (x1 ) − γy (x2 )  x1 − x2 ,

(10.4)

for any y ∈ R . This implies that the equation γy (x) = x has at most one  solution in B(x0 , s), so f is injective in B(x0 , s). n

The next lemma incorporates the results and notations of Lemma 10.4.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

page 659

659

Lemma 10.5. Let f : Rn −→ Rn be a continuously differentiable function on B(x0 , r) such that (Df)(x) is invertible when x ∈ B(x0 , r). The set f(B(x0 , s)) is open, where B(x0 , s) is an open sphere centered in x0 such that f is injective in B(x0 , s). Proof. Let V = f(B(x0 , s)) and let y1 ∈ V . There exists x1 ∈ B(x0 , s) such that y1 = f(x1 ). Choose t such that the spherical surface S(x1 , t) is contained in B(x0 , s) and let : S(x1 , t) −→ R given by (x) = f(x) − y1 . The injectivity of f established in Lemma 10.4 implies that (x) is never 0 on S(x1 , t). Since is continuous, the minimum value m on S(x1 , t) is positive. Then, for y − y1  < m 3 , we have 2 f(x) − y  f(x) − y1  − y − y1   m. 3 Define the function h : B[x1 , t] −→ R as h(x) = f(x) − y2 . Since h(x1 ) = 2 y − y1 2 < m9 and h(x)  49 m2 on the boundary S(x1 , t) of the closed sphere B[x1 , t]. Therefore, h must assume its minimum value on the closed sphere at an interior point z, where ∇h = 0. This implies n  ∂h ∂fk 0= (z) = 2 (fk (z) − yk ) ∂xj ∂xj k=1

for 1  j  n, hence (Df)(z) (f(z) − y) = 0. If f(z) = y, this implies that (Df)(z) is not invertible, so (Df)(z) is not invertible. Therefore, f(z) = y.  Thus, B(y1 , m/3) ⊆ V , which means that V is open. Theorem 10.23. (The Inverse Function Theorem) Let f : Rn −→ Rn be a continuously differentiable function on B(x0 , r) such that (Df)(x) is invertible when x ∈ B(x0 , r). If V is the open set whose existence is established by Lemma 10.5, taking into account that f is injective on B(x0 , s), and that V = f(B(x0 , s)), there exists a local inverse g : V −→ Rn at y0 = f(x0 ) such that g is continuously differentiable at y, where g(y) ∈ B(x0 , s), and f(g(y)) = y. Then, g is continuously differentiable at y0 . Proof. We begin by proving that g is differentiable at any point in y ∈ V . Let y ∈ V such that y + k ∈ V , where k is sufficiently small. There is a unique x and a unique h such that x ∈ B(x0 , s), x+h ∈ B(x0 , s), f(x) = y, and f(x + h) = y + k. Let H = ((Df)(g(y)))−1 . This inverse exists because g(y) = x ∈ B(x0 , s). We have g(y + k) − g(y) − Hk = x + h − x − H(f(x + h) − f(x)) = −H(f(x + h) − f(x) − (Df)(x)h).

May 2, 2018 11:28

660

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 660

Mathematical Analysis for Machine Learning and Data Mining

Since f is differentiable at x, we have g(y + k) − g(y) − Hk  o(h)h. By the definition of γy we have γy (x + h) − γy (x) = h − ((Df)(x0 ))−1 k. Taking into account equality (10.4), we have h − ((Df)(x0 ))−1 k  By the triangular inequality,

1 2 h.

1 k  ((Df)(x0 ))−1 k, 2 hence h  2((Df)(x0 ))−1  · k. Thus, h  ck, for some constant c, which shows that g is differentiable at all point in V and, therefore, is continuous in V . Moreover, (Dg)(y) = ((Df)(g(y)))−1 , hence g is continuously differentiable.  Note that the Jacobian matrix of g at y0 = f(x0 ) is (Dg)(y0 ) = ((Df)(f−1 (y0 ))−1 . Example 10.24. The Jacobian determinant of the function f introduced in Example 10.23 is   x e 1 cos x2 −ex1 sin x2   = e2x1 = 0, det((Df)(x)) =  x1 e sin x2 ex1 cos x2  so f has an inverse in a neighborhood of any point of R2 . Lemma 10.6. Let U be an open set in Rn and let h be a continuously  ∂fi differentiable function. If K is a closed interval included in U ,  ∂x (u) − j  ∂fi (v)  a for u, v ∈ K and 1  i, j  n, then ∂xj

h(u) − h(v) − (Dh)(u − v)  an2 u − v. Proof. Define a sequence of vectors z(0) , . . . , z(k) in Rk such that z(j) coincides with u on its first j components and with v on the last k − j components. Thus, z(0) = v and z(k) = u, and z(j) − z(j−1)  = |vj − uj |. By the Mean Value Theorem, there exists w(ij) in [z(j−1) , z(j) ] such that fi (z(j) ) − fi (z(j−1) ) =

∂fi (ij) (j) (w )(z − z(j−1) )j . ∂xj

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

Since (Dh)(z(j) − z(j−1) ))i = it follows that



∂fi (j)  ∂x (z

− z(j−1) ) =

∂fi (j) ∂xj (z

page 661

661

− z(j−1) )j ,

h(u) − h(v) − (Dh)(u − v)  |fi (z(j) ) − fi (z(j−1) ) − ((Dh)(z(j) − z(j−1) ))i | = ij

    ∂fi  ∂fi (ij)   · |(z(j) − z(j−1) )j | = (w ) − (x)  ∂xj  ∂x j ij  a|uj − vj |  an2 u − v.  

ij

Lemma 10.7. Let G = (a1 , b1 ] × (an , bn ] be an open-closed interval in Rn , h : G −→ Rn be an injective, continuously differentiable Borel measurable function and let V = h(G). We have: 7 7 dmL = |J(h)| dmL . mL (h(G)) = h(G)

G

Proof. Partition G into finitely many open-closed sub-intervals Gi such that diam(Gi ) < δ. For > 0 choose δ such that for all u, v ∈ K(G), u − v < implies J(h)(u) − J(h)(v) < . Since diam(Gi ) < δ, for xi ∈ Gi we have: 7  |J(h)(xi )|mL (Gi )  |J(h)(x)| dm + mL (G). G

i

× ···× define # # / / G i = ai1 − , bi1 + × · · · × ain − , bin + . 2 2 2 2 For x ∈ G define the affine transformation φx as

If Gi =

(ai1 , bi1 ]

(ain , bin ],

φx (z) = (Dh)(z − x) + f(x), which approximates f(z) when z is close to x. The set K(G) × {z ∈ Rn | z = 1} is a compact subset of Rn × Rn . Therefore, for some c we have (Dh)−1 z  cz for x ∈ G and z ∈ Rn . ∂h are uniformly continuous on K(G), we Since the partial derivatives ∂x i  ∂f  ∂fi i can choose δ such that z − x < δ implies  ∂x (z) − ∂x (x)  an 2 . By j j   Lemma 10.6,  ∂fi (z) − ∂fi (x)  δ . Since diam(Gi ) < δ, it follows that ∂xj

for δ < 1 we have

∂xj

a

h(z) − φxi (z)
0:   mL (h(Gi ))  mL (φxi (G i ) mL (h(G)) = i

=



i

|J(h)(xi )|mL (G i ) = (1 + )n

i

This implies mL (h(G)) 

6



|J(h)(xi )|mL (Gi ).

i G |J(h)(x)|

dm



Theorem 10.24. Let U be an open set in Rn , h : U −→ Rn be an injective, continuously differentiable Borel measurable function and let V = h(U ). If J(h) = det((Dh)(x)) = 0 for every x ∈ U , then for a non-negative measurable function f : Rn −→ Rn we have: 7 7 f dmL = fhJ(h) dmL . h(U)

U

Proof. By Lemma 10.7, the equality of the theorem is valid for openclosed intervals, f = 1G , and injective, continuously differentiable Borel measurable functions h, where V = h(G). Let EU = U ∩ B(Rn ) and let S be the semi-ring of open-closed intervals included in U . Then, Theorem 7.9 implies that Kσ-alg (S) = EU . Therefore, 6 U is a countable union of open-closed intervals and that the measure G |J(h) dm 6 6 L is finite for G ∈ S (because supG |J| < ∞). Thus, |J(h)| dm  L U h(U) dmL . This implies, by linearity and monotone convergence: 7 7 f dmL  fhJ(h) dmL . (10.5) h(U)

U

By the inverse function theorem, the mapping h is locally invertible and and a similar equality can be inferred for the mapping h−1 , that is, for V = h(U ) we have: 7 7 g dmL  gh−1 J(h−1 ) dmL , h−1 (V )

V

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

page 663

663

for any measurable non-negative measurable function g : Rn −→ Rn . By substituting f(h(x))|J(h)(x)| for g in the last equality we have 7 7 fh|J(h)| dmL  fhh−1 J(h)J(h−1 ) dmL , U

h(U)

we obtain:

7

7 fh|J(h)| dmL  U

f dmL .

(10.6)

h(U)

Inequalities (10.5) and (10.6) yield the equality of the theorem.



Example 10.25. Let U = (0, ∞) × (0, π) × (0, 2π) be an open set and let h : U −→ R3 be the function defined by h1 (r, θ, φ) = r sin θ cos φ, h2 (r, θ, φ) = r sin θ sin φ, h3 (r, θ, φ) = r cos θ. As we saw in Example 10.11 J(h)(r, θ, φ) = r2 sin θ. Therefore, the formula for integrating in spheric coordinates is 7 7 f(r sin θ cos φ, r sin θ sin φ, r cos θ)r2 sin θ dmL = f(x1 , x2 , x3 ) dmL . R3

U

Similarly, the formula for integration in polar coordinates is: 7 7 f(r cos φ, r sin φ)r dmL = f(x1 , x2 ) dmL , U

R2

where U = (0, ∞) × (0, 2π). open sphere of radius a. Let now V6 = B(02 , a) be the 6 a 6two-dimensional 2π To compute B(02 ,r dmL = 0 0 r dr dθ. we apply Fubini’s Theorem and we have: 7 2π 7 a 1 a2 · 2π = πa2 . r dr dθ = 2 2 0 0 10.5

Normal and Tangent Subspaces for Surfaces in Rn

Let X be an open subset of Rn and let h : X −→ R be a function that is differentiable on X. The set Sh = {x ∈ Rn | h(x) = 0} is a smooth surface in Rn . The curve in Rn determined by a function g : [a, b] −→ Rn is set Cg = {x ∈ Rn | x = g(t) for t ∈ [a, b]}. The curve Cg is smooth is g

May 2, 2018 11:28

664

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 664

Mathematical Analysis for Machine Learning and Data Mining

is differentiable on [a, b]. Cg is located on the surface Sh if h(g(t)) = 0 for t ∈ [a, b]. Let Cg be a curve located on the surface Sh . Both the curve and the surface are assumed smooth. Differentiating the equality h(g(t)) = 0 using the Chain Rule we obtain dg (Dh)(g(t)) (t) = 0. dt If x0 = g(t0 ), where t0 ∈ [a, b], then dg dt (t0 ) is the tangent vector to the curve Cg in x0 = g(t0 ). Thus, dg (∇h)(x0 ) (t0 ) = 0, dt which shows that any vector tangent in x0 to a curve Cg passing through x0 is orthogonal on (∇h)(x0 ). Thus, it is natural to define the tangent hyperplane to Sh in x0 by the equation (∇h)(x0 ) (x − x0 ) = 0. The vector (∇h)(x0 ) is the normal to Sh in x0 . This example can be generalized by considering surfaces in Rn determined by differentiable functions of the form h : Rn −→ Rm . The surface determined by h is the set Sh = {x ∈ Rn | h(x) = 0m }; again, we refer to S as the smooth surface defined by h. Note that Sh is the intersection m h i=1 Shi . Definition 10.8. If h : X −→ Rm , h ∈ C 1 (X) and m  n, then x0 is a regular point of h if the Jacobian matrix (Dh)(x0 ) ∈ Rm×n has rank m. In other words, x0 is a regular point of h if the vectors (∇h1 )(x0 ), . . . , (∇hm )(x0 ) are linear independent. A critical point of h is a point x0 ∈ X such that (Dh)(x0 ) is the zero operator, that is, δf (x0 ; h) = 0 for every h ∈ X. A curve Cg is located on the surface Sh if and only if it is located on every surface Shi . Therefore, a tangent dg dt (t0 ) to the curve Cg in a regular point x0 ∈ Sh must be orthogonal on every vector (∇hi )(x0 ). The subspace Nx0 generated by the vectors (∇h1 )(x0 ), . . . , (∇hm )(x0 ) is the normal subspace to Sb in x0 ; the orthogonal subspace Tx0 = (Nx0 )⊥ is the tangent subspace to Sh in x0 . Equivalently, the tangent subspace of Sh at x0 is the null space T (x0 ) of the linear transformation (Dh)(x0 ). Definition 10.9. The tangent r-plane of M at x0 is {x0 +t | t ∈ T (x0 )} = tx0 (T (x0 )).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Differentiability of Functions Defined on Normed Spaces

b3234-main

page 665

665

Example 10.26. The spherical surface S(0n , 1) in Rn is defined by a funcn tion h : Rn −→ R given by h(x) = i=1 x2i − 1. We have: (Dh)(x0 )(k) = 2x0 k, so rank((Df )(x0 )) = 1 for every x0 ∈ S(0n , 1). A normal vector to S(0m , 1) in x0 is 2x0 . Example 10.27. Let M be the subset of R3 : M = S(03 , 1) ∩ {x ∈ R3 | x1 + x2 + x3 = 0}. Define the function h : R3 −→ R2 as h1 (x) = x21 + x22 + x23 − 1, h2 (x) = x1 + x2 + x3 for x ∈ R3 . Its Jacobian matrix is

  2x1 2x2 2x3 (Dh)(x) = . 1 1 1

For any x ∈ M we have rank((Dh)(x)) = 2 and M = {x ∈ R3 | h(x) = 02 }. This time the normal subspace Nx0 is generated by the vectors 2x0 and 13 . Example 10.28. Let M = {x ∈ R2 | x1 x2 = 1, or x1 = 0 and x2 = 0, or x2 = 0 and x1 = 0}. There are three cases to consider. x2

x1

Fig. 10.1

May 2, 2018 11:28

666

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 666

Mathematical Analysis for Machine Learning and Data Mining

(i) If x1 x2 = 1, define h(x) = x1 x2 − 1; we have (Dh)(x) = (x2 , x1 ) so in this case, rank((Dh)(x) = 1. (ii) If x1 = 0 and x2 = 0, let h1 (x) = x1 , which implies (Dh1 )(x) = (1, 0), hence rank((Dh1 )(x)) = 1. (iii) Finally, if x2 = 0 and x1 = 0, let h2 (x) = x2 , so (Dh2 )(x) = (1, 0), hence rank((Dh)(x)) = 1. Thus, in each case the normal subspace is unidimensional is an unidimensional manifold.

Exercises and Supplements (1) Let f : Rn −→ R be a functions such that |f (x)|  x1+a , where a > 0. Prove that f is Fr´echet differentiable in 0n .   (2) Let f : R −→ R2 be the function given by f (x) = xx2 . Is f Fr´echet differentiable? (3) Let f : R2 −→ R be defined by f (x) = x1 + x2 for x ∈ R2 . Prove that the Gˆ ateaux derivative of f is (Df )(x0 )(h) = f (h) for h ∈ R2 . (4) Let f : R2 −→ R be defined by  f (x) =

1 0

if x1 + x2 = 0 and x = 02 , otherwise.

∂f ∂f (02 ) and ∂x (02 ) exists. Find a Prove that the partial derivatives ∂x 1 2 direction h such that the directional derivative relative to u does not exist in 02 .

Solution: We have

f (02 +th)−f (02 ) t

 f (th) =

1 0

=

f (th) . t

The definition of f implies

if t(h1 + h2 ) = 0 and th = 0, otherwise.

If h is a direction such that h1 + h2 = 0, the directional derivative with ∂f ∂f (02 ) = ∂x (02 ) respect to h in 02 does not exist. This also imply that ∂e 1 1 ∂f ∂f and ∂e1 (02 ) = ∂x1 (02 ) exist.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Differentiability of Functions Defined on Normed Spaces

b3234-main

page 667

667

(5) Let f : R2 −→ R be defined by  f (x) =

x1 0

if x1 = x2 , otherwise.

Prove that this function has directional derivatives in all directions in ateaux differentiable at 02 . 02 but is not Gˆ Solution: We have  f (th) = t

h1 0

if h1 = h2 , otherwise.

exists for ever direction h, so this function It is clear that limt↓0 f (th) t has directional derivatives in all directions. However, (Df )(0)2 (h) fails ateaux differentiable in 02 . to be a linear operator on R2 , so f is not Gˆ (6) Let f : R2 −→ R be defined by f (x) = x1 x2 for x ∈ R2 . Prove that the Gˆ ateaux derivative of f is (Df )(x0 )(h) = h1 x02 + h2 x01 for h ∈ R2 . Solution: We have: f (x0 + th) − f (x0 ) t (x01 + th1 )(x02 + th2 ) − x01 − x02 = lim t→0 t = h1 x02 + h2 x01 .

lim

t→0

(7) Let f : R2 −→ R be given by  f (x) =

x1 0

if x2 = x21 , otherwise.

Prove that f is Gˆ ateaux differentiable at 02 . (8) Let (S, ·) be a normed F-linear space, T be an F-linear space, X be an open set in (S,  · ) and let f : X −→ T be a function that is Gˆ ateaux differentiable in x0 . Prove that (Dx f )(x0 )(−u) = −(Dx f )(x0 )(u) for every u ∈ S.

May 2, 2018 11:28

668

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 668

Mathematical Analysis for Machine Learning and Data Mining

Solution: Since f is differentiable at x0 we have f (x0 − tu) − f (x0 ) t f (x0 + (−t)u) − f (x0 ) = − lim t→0 −t = −(Dx f )(x0 )(u)

(Dx f )(x0 )(−u) = lim

t→0

for every u ∈ S. (9) Prove that the function f : R2 −→ R given by ⎧ ⎨ f (x) =

x21

⎩ 0

x21 + x22

if x = 02 if x = 02

for x ∈ R2 is differentiable everywhere except in 02 . (10) Let f : Rn −→ Rm be a Gˆ ateaux differentiable function and let (Df)(x) ∈ Rm×n be its Jacobian matrix. It is clear that rank((Df)(x))  min{m, n}. Prove that the set T = {x ∈ Rn | rank((Df)(x)) = min{m, n}} is an open subset of Rm . Solution: We will show that the set Rm − T is closed. Suppose that m < n. We have z ∈ Rn − T if and only rank((Df)(z)) < m, that is, if every determinant of order m extracted from (Df)(x) is 0. Since determinants are continuous functions of their entries, if follows that for a sequence (xn ) in Rn − T such that limn→∞ xn = x, every determinant of order m extracted from a matrix (Df)(xn ) is 0, which means that every such determinant from (Df)(x) is 0. Therefore, rank((Df)(x)) < m, hence x ∈ Rm − T . Thus, Rm − T is closed, so T is open.  n n (11) Let R = n be a i=1 [ai , bi ] be a rectangle in R and let f : R −→ R ∂fi continuously differentiable function on R such that | ∂xj (x)|  K for all x ∈ I(R). Prove that f(u) − f(v)  x − yn2 K. Solution: By the Mean Value Theorem (Theorem 10.8) we have: f(u) − f(v)  u − v sup{(Df)(x) | x ∈ [u, v]}. Note that (Df)(x)  n2 K, which gives the desired inequality. (12) Let D be an open interval of R and let f : D −→ R be a differentiable function such that f  is a Lipschitz function with constant c on D. Prove that for any x, y ∈ D we have: (a) |f (y) − f (x) − f  (x)(y − x)|  (x) (b) | f (x+h)−f − f  (x)|  ch . h 2

c(y−x)2 ; 2

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

Solution: Since f (y) − f (x) = f (y) − f (x) − f  (x)(y − x) =

!

"y x

y

x ! 1

=

page 669

669

f  (z) dz, we have:

(f  (z) − f  (x)) dz [f  (x + t(y − x)) − f  (x)](y − x) dt

0

(using the change of variables z = x + t(y − x) ! 1 [f  (x + t(y − x)) − f  (x)] dt. = (y − x) 0

Therefore,

! 1



|f (y) − f (x) − f  (x)(y − x)|  |y − x|

[f  (x + t(y − x)) − f  (x)] dt

0

!

 c(y − x)

1

2

t dt = 0

c(y − x)2 . 2

The second inequality is an immediate consequence of the first. (13) Let f : Rn −→ Rm be a continuously differentiable function on an open subset D of Rn , x ∈ D, and let (Df)(x) be Lipschitz function with constant c. Then, for x, y ∈ S we have f(y) − f(x) − (Df)(x)(y − x) 

cy − x2 . 2

Solution: By Corollary 10.2 we have f(x + h) − f(x) − (Df)(x)h

! 1 (Df)(x + th)h dt − (Df)(x)h = 0 1

!

((Df)(x + th) − (Df)(x))h dt.

= 0

Since (Df)(x) be Lipschitz function with constant c, we obtain !

1

f(x + h) − f(x) − (Df)(x)h  !

(Df)(x + th) − (Df)(x)h dt

0 1



cthh dt = 0

c h2 . 2

(14) Let f : Rn −→ R be a differentiable function in C 1 (R) and let x0 ∈ Rn such that f (x0 ) = 0. Prove that the function g defined as g(x) = f (x) |f (x)| is differentiable in x0 .

May 2, 2018 11:28

670

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 670

Mathematical Analysis for Machine Learning and Data Mining

Solution: Since f ∈ C 1 (R) we have f (x0 + th) = f (x0 ) + t(h ∇)(f )(x0 + θh)) = t(h ∇)(f )(x0 + θh)). Therefore, g(x0 + th) − g(x0 ) f (x0 + th) |f (x0 + th)| = lim t→0 t t f (x0 + th) |f (x0 + th)| = lim t→0 t = lim |t|(h ∇)(f )(x0 + θh))(h ∇)(f )(x0 + θh)) = 0,

(Dg)(x0 )(h) = lim

t→0

t→0

which shows that g is differentiable in x0 . (15) Let c : Rn −→ R be a function in C 1 (R), c+ (x) = max{0, c(x)} and let  : Rn −→ R be the function defined by (x) = (c+ (x))2 for x ∈ Rn . Prove that (a) the partial derivatives of  are given by ∂c ∂ = 2 max{0, c(x)} (x); ∂xj ∂xj (b)  is Gˆ ateaux differentiable in x0 , where c(x0 ) = 0. Solution: Note that c+ (x) = 12 (c(x) + |c(x)|) for x ∈ Rn . We have:

2 c(x) + |c(x)| 2 1 2 1 = c (x) + c(x)|c(x)|). 2 2

(x) =

The first term is clearly differentiable; the second is also differentiable by Supplement 14. (16) The result contained by this supplement is known as Danskin’s Theorem [39]. Let (S, O) be a topological space and let f : X × Y −→ R be a function, where X is an open set in Rn and Y is a compact subset of S. Define the function φ : X −→ R as φ(x) = max{f (x, y) | y ∈ Y } and let Y (x) = {y ∈ Y | φ(x) = f (x, y)} be the set of maximizers. Prove that the function φ : X −→ R is continuous and has directional derivatives in every direction h such that ∂φ = max{((Dx f )(x, y), h) | y ∈ Y (x)}. ∂h

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

page 671

671

Solution: Let (xk ) be a sequence in Rn such that limk→∞ xk = x0 . Let yk ∈ Y such that φ(xk ) = f (xk , yk ). Since Y is compact, one can assume that limk→∞ yk = y0 . The definition of φ implies φ(xk ) = f (xk , yk )  f (xk , y). Therefore, for any y ∈ Y we have: lim φ(xk ) = lim f (xk , yk ) = f (x0 , y0 )  lim f (xk , y) = f (x0 , y)

k→∞

k→∞

k→∞

for all y ∈ Y . The inequality f (x0 , y0 )  f (x0 , y) implies φ(x0 ) = f (x0 , y0 ) = limk→∞ φ(xk ), hence φ is continuous. Let h be a direction and let (xk ) be a sequence defined as xk = x0 +tk h such that tk > 0 and limk→∞ xk = x0 . If y ∈ Y (x0 ) and yk ∈ Y (xk ) for k  1 we can write f (xk , yk ) − f (x0 , y) φ(xk ) − φ(x0 ) = tk tk f (xk , yk ) − f (xk , y) f (xk , y) − f (x0 , y) = + tk tk f (xk , y) − f (x0 , y)  tk (because f (xk , yk )  f (xk , y)) = ((∇f )(x0 + θk h, y), h) for some θk ∈ (0, 1) (by the Mean Value Theorem). Therefore, lim inf

k→∞

φ(xk ) − φ(x0 )  ((∇x f )(x0 , y), h) tk

for all y ∈ Y (x0 ), which implies lim inf

k→∞

φ(xk ) − φ(x0 )  max{((∇x f )(x0 , y), h) | y ∈ Y (x0 )}. tk

If φ(xk ) = f (xk , yk ) and limk→∞ yk = y0 we have f (xk , yk ) − f (x0 , y0 ) φ(xk ) − φ(x0 ) = tk tk f (xk , yk ) − f (x0 , yk ) f (x0 , yk ) − f (x0 , y0 ) = + tk tk f (xk , yk ) − f (x0 , yk )  tk (because f (x0 , y0 )  f (x0 , yk )) = ((∇f )(x0 + θk h, yk ), h) for some θk ∈ (0, 1) (by the Mean Value Theorem).

May 2, 2018 11:28

672

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 672

Mathematical Analysis for Machine Learning and Data Mining

Consequently, φ(xk ) − φ(x0 )  max{((∇x f )(x0 , y), h) | y ∈ Y (x0 )}, tk k→∞

lim sup

which allows to conclude that ∂φ = max{((Dx f )(x, y), h) | y ∈ Y (x)}. ∂h (17) Let (S, Od ) be a metric space. For α > 0 define “ α ” on S × R as (x1 , a1 ) α (x2 , a2 ) if (a2 − a1 ) + αd(x1 , x2 )  0. The relation α is a partial order on S × R. Solution: This relation is clearly reflexive. If (x1 , a1 ) α (x2 , a2 ) and (x2 , a2 ) α (x1 , a1 ), then (a2 − a1 ) + αd(x1 , x2 )  0 and (a1 − a2 ) + αd(x1 , x2 )  0, so αd(x1 , x2 )  a1 − a2 and αd(x1 , x2 )  a2 − a1 . Suppose that a1 = a2 . Then at least of the numbers a1 − a2 , a2 − a1 is negative and this leads to a contradiction because αd(x1 , x2 ) is nonnegative. Thus, a1 = a2 , which implies d(x1 , x2 )  0, so x1 = x2 , so α is antisymmetric. If (x1 , a1 ) α (x2 , a2 ) and (x2 , a2 ) α (x3 , a3 ), then (a2 − a1 ) + αd(x1 , x2 )  0 and (a3 − a2 ) + αd(x2 , x3 )  0, which implies (a3 − a1 ) + α(d(x1 , x2 ) + d(x2 , x3 ))  0. By applying the triangular inequality we have (a3 − a1 ) + αd(x1 , x3 )  0, so α is a partial order. (18) Let V be a Banach space and let f : V −→ R ∪ {∞}. Let S f (v) be the set of continuous linear functionals that -support f at v. Prove that (a) S f (v) = ∅ if v ∈ Dom(f ); (b) S f (v) is a convex subset of V ∗ ; (c) S f (v) + Sθ f (v) ⊆ S+θ (f + g)(v); (d) if θ  ,then S f (v) ⊆ θ f (v);  S  (e) if v ∗ ∈ >0 S f (v) ∩ − >0 S (−f )(v) , then f is Fr´echet differentiable and (Df )(v) = v ∗ ; (f) if S f (v) = ∅, then f is lower semicontinuous at v. Solution: Suppose that S f (v) = ∅ and let v ∗ ∈ V be a continuous linear functional such that there exists η > 0 such that w − v  η implies f (w)  f (v) + v ∗ (w − v) − w − v.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

page 673

673

We have {x ∈ S | f (x) > v}. Let X, Y be two Banach spaces, [x0 , x0 + d] be an interval in X, and let h be a mapping defined on [x0 , x0 + k] and ranging over the set of operators defined on X with values in Y . " x +k Define x00 h(x) dx as !

!

x0 +k x0

n−1 

1

h(x) dx =

h(x0 + tk)(k) dt = 0

lim ν(Δ)→0

h(x0 + ξp k)(k)(tp+1 − tp ),

p=0

where Δ = {t0 , . . . , tn } ∈ SUBD[0, 1] and ξp ∈ [tp , tp+1 ] for 0  p  n − 1. Note that this definition makes sense as we noted in the Supplements of Chapter 9, because h(x0 +tk)(k) is a function defined on [0, 1] ranging in the Banach space Y . (19) Let X, Y be two Banach spaces, [x0 , x0 + k] be an interval in X, and let f : X −→ Y be a function such that (Df )(x) is continuous in [x0 , x0 +k]. " x +k Prove that x00 (Df ) dx exists and !

x0 +k

(Df )(x) dx = f (x0 + k) − f (x0 ).

x0

Solution: By the definition of Riemann integral we have !

x0 +k

(Df )(x) dx =

lim ν(Δ)→0

x0

=

lim ν(Δ)→0

n−1 

f (x0 + ξp k)(k)(tp+1 − tp )

p=0 n−1 

f  (xp )(tp+1 − tp )k.

p=0

Note that f (x0 +k)−f (x0 ) =

n−1 

n−1 

p=0

p=0

(f (x0 +tp+1 k)−f (x0 +tp k)) =

(f (xp+1 )−f (xp )).

By the Mean Value Theorem (Theorem 10.8) we have

n−1









(f (xp+1 ) − f (xp ) − (Df )(xp )(tp+1 − tp )k







p=0

 k

n−1  p=0

(tk+1 − tk ) sup f  (xp + θ(tp+1 − tp )k) − f  (xp ). θ∈(0,1)

Since f  is uniformly continuous on [x0 , x0 +k], the desired result follows.

May 2, 2018 11:28

674

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 674

Mathematical Analysis for Machine Learning and Data Mining

Let f : [a, b] −→ T be a function of a real argument whose values belong to a Banach space T . The notions of subdivision of [a, b], Darboux sums, and Riemann integral, etc. can be immediately transferred to functions of this form. The "b "b function f : [a, b] −→ R is Riemann integrable if a f dx = a f dx. When f "b Riemann integrable on [a, b], its integral is denoted by a f (t) dt as usual. (20) Let T, U be Banach spaces. Prove that if f : [a, b] −→ T is a Riemann integrable function and h : T −→ U is a linear operator between  " b Banach  "b spaces, then the function hf is integrable and a uf dt = u a f dt . (21) Let φ : [a, b] −→ R be real-valued integrable function, T be a Banach space, and let x0 ∈ T . Prove that the function f : [a, b] −→ T defined "b "b by f (t) = φ(t)x0 is integrable and a f dt = ( a φ(t); dt)x0 . (22) Let f : [a, b] −→ T be a Riemann integrable function. Prove that the function ψ : [a, b] −→ R defined by ψ(t) = f (t) is integrable and "b "b  a f (t) dt  a ψ(t) dt. (23) Recall that the set of continuous operators between the Banach spaces S, T is denoted by B(S, T ). Let [u, v] be an interval in S and let Φ : [u, v] −→ B(S, T ). Define !

!

v

1

Φ(x) dx = u

Φ((1 − t)u + tu)(v − v) dt.

(10.7)

0

Since Φ is continuous, the integral exists and is an element of T . Let X ⊆ S, and let P : X −→ T" be an operator such that Φ(x) = v (DP )(x) for x ∈ [u, v]. Prove that u Φ(x) dx = P (v) − P (u). Solution: By the defining equality (10.7) we have: !

!

v

Φ(x) dx = u

1

Φ((1 − t)u + tu)(v − u) dt.

0

Let Δ = {t0 , t1 , . . . , tn } be a subdivision of the interval [0, 1] such that 0 = t0 < t1 < · · · < tn−1 < tn = 1. The interval [u, v] in S is subdivided by the points x0 , x1 , . . . , Xn , where xk = (1−tk )u+tk v. If tk ∈ (tk , tk+1 ) the point olxk = (1 − tk )xk + tk xk+1 belongs to [xk , xk+1 ]. Using notations introduced in Definition 8.8, we have !

1

Φ((1 − t)u + tu)(v − u) dt 0

=

lim ν(Δ)→0

=

lim ν(Δ)→0

n−1 

Φ((1 − tk )u + tk v)(tk+1 − tk )(v − u)

k=0 n−1  k=0

Φ(xk )(tk+1 − tk )(v − u).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Differentiability of Functions Defined on Normed Spaces

Since P (v) − P (u) =

n−1

k=0 (P (xk+1 )

page 675

675

− P (xk )) we have

n−1









(P (xk+1 ) − P (xk ) − Φ(xk )(tk+1 − tk )(v − u)







k=0

 v − u

n−1 

supθ∈(0,1) Φ(xk + θ(xk+1 − xk ) − Φ(xk ),

k=0

by the Mean Value Theorem. Since Φ is uniformly continuous on [u, v] the result follows.

Bibliographical Comments The monograph [94] is fully dedicated to a study of the history, proofs, and applications of the implicit function theorem. An up-to-date lucid reference on optimization is [71]. The presentation of the Implicit Function Theorem follows [29]. The extension of Riemann integrals to functions of a real argument that range in a Banach space follows [86] and is contained in Supplements 20–23. Supplement 18 was obtained in [50].

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 677

Chapter 11

Hilbert Spaces

11.1

Introduction

Hilbert spaces are inner product spaces that are complete with respect to the topology induced by the metrics generated by inner products. In other words, Hilbert spaces are special Banach spaces whose norm is generated by an inner product. This concept originated in the work on integral equations and Fourier series of the German mathematician David Hilbert.1 The term Hilbert space was introduced by John von Neumann2 in 1929 in order to describe these spaces in an axiomatic way. 11.2

Hilbert Spaces — Examples

Definition 11.1. A Hilbert space is a linear space H equipped with an inner product such that H is complete relative to the metric defined by the inner product. As usual, we denote the inner product of x, y ∈ L by (x, y).

1 The German mathematician David Hilbert (Jan. 23rd 1862–Feb. 14th 1943) was one of the most influential mathematicians of the last two centuries. His contributions span the foundations of mathematics, axiomatization of geometry and functional analysis. Hilbert taught at the University of K¨ onigsberg until 1895 and later at the University of G¨ ottingen. 2 John von Neumann (born on December 28th , 1903, Budapest, Hungary, died February 8th , 1957, in Washington, D.C., U.S.) was a Hungarian-born American mathematician. Von Neumann was one of worlds foremost mathematicians and made important contributions in set theory, quantum theory, automata theory, economics, and game theory. Von Neumann was one of the inventors of the stored-program digital computer. He was a professor at the Institute of Advanced Studies in Princeton, NJ.

677

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 678

Mathematical Analysis for Machine Learning and Data Mining

678

Example 11.1. The real linear space Rn equipped with the usual inner product defined by n  xi yi (x, y) =  x1  for x =

. . . xn

 y1  and y =

. . . yn

i=1

is a Hilbert space.

The complex linear space Cn is a Hilbert space for the inner product given by n  (x, y) = xi yi  x1  for x =

. . . xn

 y1  and y =

. . . yn

i=1

.

Example 11.2. The real Banach space 2 (R) introduced in Example 9.5 is a Hilbert space relative to the inner product  xi yi (x, y) = i∈N 2

for x, y ∈ (R). Its counterpart on the set of complex numbers, 2 (C) is equipped with the inner product  xi yi . (x, y) = i∈N

Example 11.3. As a consequence of Cauchy-Schwarz inequality, L2 (S, E, 6m) is a Hilbert space relative to the inner product defined by (f, g) = S f g dm. For the special case of random variables X, Y defined on a" probability 6 space (Ω, E, P ), we have (X, Y ) = Ω XY dP . Also, X2 = (X, X) = " E(X 2 ). Note that for X ∈ L2 (Ω, E, P ) we have " " " E(X) = (|X|, 1)  (X, X) (1, 1) = E(X 2 ), hence X1  X2 . Therefore, we have L2 (Ω, E, P ) ⊆ L1 (Ω, E, P ). Thus, if X, Y ∈ L2 (Ω, E, P ) the variance var(X) = E((X − E(X))2 ) and the covariance of X and Y defined by cov(X, Y ) = E((X − E(X))(Y − E(Y ))) = (X, Y ) − E(X)E(Y ), exist and are finite.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Hilbert Spaces

11.3

page 679

679

Classes of Linear Operators in Hilbert Spaces

Theorem 6.25 shows that a linear operator between two normed spaces is continuous if and only if it is bounded. For Hilbert spaces one can prove an inequality that links the norm of a linear operator to the inner product. Theorem 11.1. A linear operator on a Hilbert space h : H −→ H is continuous if and only if there exists a positive number M such that |(h(x), y)|  M xy for x, y ∈ H. Moreover, h = inf{M > 0 | (h(x), y)  M xy for x, y ∈ H}. Proof. Let h : H −→ H be a continuous linear operator. By Theorem 6.25 h is bounded and we have h(x)  hx. Therefore, by applying Cauchy-Schwarz inequality we have: |(h(x), y)|  h(x)y  hxy.

(11.1)

Conversely, suppose that the inequality of the theorem holds. Taking y = h(x) we obtain h(x)2  hxh(x), hence h(x)  hx when h(x) = 0. If h(x) = 0, the inequality obviously holds. Observe that if (h(x), y)  M xy then (h(x), y)  M y when x = 1, so |h(x)|  M (by taking y = h(x)) for x = 1 which implies h  M . This, in turn yields h = inf{M > 0 | (h(x), y)  M xy for x, y ∈ H}.



Theorem 11.2. Let h : H −→ H be a continuous linear operator defined on a Hilbert space H. There exists an operator h∗ such that (i) (h(x), y) = (x, h∗ (y)) for every x, y ∈ H; (ii) h∗ is linear and continuous; (iii) h∗  = h. Proof. Let fy : H −→ C be linear continuous functional given by fy (x) = (h(x), y). By Riesz’ Theorem there exists a unique y ∗ ∈ H such that fy (x) = (x, y ∗ ) by . Define the operator h∗ : H −→ H as h∗ (y) = y ∗ . We have (h(x), y) = (x, h∗ (y)) for x, y ∈ H.

May 2, 2018 11:28

680

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 680

Mathematical Analysis for Machine Learning and Data Mining

We claim that the operator h∗ is linear and continuous. Indeed, for every x, y, z we have (x, h∗ (ay + bz)) = (h(x), ay + bz) = a(h(x), y) + b(h(x), z) = a(x, h∗ (y)) + b(x, h∗ (z)) = (x, ah∗ (y)) + (x, bh∗ (z)) = (x, ah∗ (x) + bh∗ (y), which implies h∗ (ay + bz) = ah∗ (y) + bh∗ (z). In other words, h∗ is a linear operator. Furthermore, since |(h(x), y)|2  M xy it follows that |x, h∗ (y)|2   M xy for x, y ∈ H, so h∗ is continuous and h∗  = h. Definition 11.2. If h : H −→ H is a linear continuous operator on the Hilbert space H, the h∗ is the adjoint operator of h. Example 11.4. Let h : L2 ([a, b]) −→ L2 ([a, b]) be the operator defined by 7 b K(x, y)f (y) dy, h(f )(x) = a

where K is a continuous function. Note that 7 b (h(f ), g) = h(f )(x) g(x) dx a  7 7 b

b

=

K(x, y)f (y) dy g(x) dx a

7

a

b

7

=



b

K(y, x)g(x) dx f (y) dy a

a

(by exchanging the order of integration)  7 b 7 b K(x, y)g(y) dy f (x) dx = a

a

(by renaming the variables). Therefore, the equality (h(f ), g) = (f, h∗ (g)) that defines the adjoint operator is satisfied by the operator h∗ if we define 7 b 7 b ∗ h (g)(x) = K(x, y)g(y) dy = K(y, x)g(y) dy a

a

for x ∈ [a, b]. By the uniqueness of the adjoint operator we have: 7 b 7 b h∗ (f )(x) = K(x, y)f (y) dy = K(y, x)f (y) dy, a

a

(11.2)

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Hilbert Spaces

b3234-main

page 681

681

where we replaced g by f . Thus, when K(y, x) = K(x, y), h∗ coincides with h. We examine several classes of continuous linear operators that act on Hilbert spaces. Definition 11.3. A continuous linear operator h ∈ L(H) on a complex Hilbert space H is (i) self-adjoint if h∗ = h; (ii) normal if hh∗ = h∗ h; (iii) unitary if it is surjective and (h(x), h(y)) = (x, y) for every x, y ∈ H; (iv) positive if (h(x), x)  0 for every x ∈ H; (v) positive definite if it is positive and invertible; (vi) idempotent if hh = h; (vii) projection if it is both self-adjoint and idempotent. 11.3.1

Self-Adjoint Operators

Let SA(H) be the set of self-adjoint operators defined on the Hilbert space H. If h ∈ SA(H), where H is a complex Hilbert space then (h(x), x) is a real number for every x ∈ H because (h(x), x) = (x, h(x)) = (h(x), x). Theorem 11.3. If h is a self-adjoint linear operator on a Hilbert space H, then Null(h) and Img(h) are orthogonal closed subspaces of H. Proof. Let u ∈ Null(h) and let v ∈ Img(h). Then h(u) = 0H and v = h(x) for some x ∈ H. Since h ∈ SA(H), we have (u, v) = (u, h(x)) = (h(u), v) = (0H , v) = 0. so Null(h) ⊥ Img(h). By Theorem 6.19 Null(h) and Img(h) are orthogonal closed subspaces of H.  Theorem 11.4. If H is a real Hilbert space, SA(H) is a closed linear subspace of L(H). If H is a complex Hilbert space, then SA(H) is a closed subset of L(H) that is closed with respect to addition and with multiplication with real numbers.

May 2, 2018 11:28

682

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 682

Mathematical Analysis for Machine Learning and Data Mining

Proof. It is immediate that SA(H) is a subspace of L(H). Let (hn ) be a sequence in SA(H) that converges uniformly to h ∈ L(H). For every x, y ∈ H we have (h(x), y) = lim (hn (x), y) = lim (x, hn (y)) = (x, h(y)), n→∞ n→∞ which shows that h is self-adjoint. The last part of the theorem is immediate.  Lemma 11.1. Let H be a complex Hilbert space. For every y, z ∈ H there exists a such that (eia y, z) is a real non-negative number. Proof. The inner product (y, z) which is a complex number can be written in polar form as (y, z) = |(y, z)|eiα . Thus, if suffices to take a = −α to write (eia y, z) = eia (y, z) = e−iα |(y, z)| eiα = |(y, z)|, which is a real non-negative number.  Theorem 11.5. Let H be a complex Hilbert space and let h ∈ SA(H). We have h = sup{|(h(x), x)| | x = 1}. Proof. Let c = sup{(h(x), x) | x = 1}. By equality (11.1) we have c  h. We will show now that h  c. By the Polarization Identity (see Theorem 2.26) we have: 1 (y, h(x)) = ((x + y, h(x + y)) − (x − y, h(x − y)) 4 −i(y + ih(x), h(x + iy)) + i(x − iy, h(x − iy))) . Since h is a self-adjoint operator the first two terms in the right member are real and the last two terms are imaginary. By Lemma 11.1 there exists a ∈ R such that (eia y, h(x)) = |(y, h(x))| is a real non-negative number. Replacing y by eia y in the Polarization Identity mentioned above leads to the disappearance of the imaginary terms and we have 1 (y, h(x)) = ((x + y, h(x + y)) − (x − y, h(x − y))) . 4 Thus, 2 1   |(y, h(x))|2 = (x + y, h(x + y)) − (x − y, h(x − y)) 16 c2  x + y2 + x − y2  16 (by the definition of c) c2  x2 + y2 = 8 (by the Parallelogram Equality). 2 For x = y = 1 we have |(y, h(x))|2  c4 , so h  2c < c. 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Hilbert Spaces

page 683

683

Example 11.5. If h is a linear operator on the Hilbert space H, then h∗ h and hh∗ are positive operators. Indeed, by the definition of the adjoint operator we have (h∗ h(x), x) = (h(x), h(x)) = h(x)2  0. The argument for hh∗ is similar. 11.3.2

Normal and Unitary Operators

Theorem 11.6. The set of normal operators is closed in L(H). Proof. If (hn ) is a sequence of normal operators that converge in norm to an operator h, then (h∗n ) converges to h∗ , and, by the continuity of composition we have hh∗ = lim hn h∗n = lim h∗n hn = h∗ h. n→∞

n→∞



Unitary operators were defined as surjective continuous linear mappings that preserve the inner product. Lemma 11.2. Let h : H −→ H be a surjective function defined on a Hilbert space H = {0H } that preserves the inner product, i.e., (h(x), h(y)) = (x, y) for every x, y ∈ H. Then, h is a continuous linear mapping. Proof.

Let u, v, x ∈ H and let a, b two scalars. We have (h(au + bv) − ah(u) − bh(v), h(x)) = (h(au + bv), h(x)) − a(h(u), h(x)) − b(h(v), h(x)) = (au + bv, x) − a(u, x) − b(u, x) = 0 (due to the linearity of the inner product).

Since every z ∈ H can be written as z = h(x) (because h is surjective), it follows that h(au + bv) − ah(u) − bh(v) ∈ H ⊥ , which implies that h(au + bv) − ah(u) − bh(v) = 0 for every u, v ∈ H. Thus, h is indeed a linear mapping. Choosing x = y = 0H in the equality (h(x), h(y)) = (x, y) implies h(x) = x for x ∈ H − {0H }, which, in turn, implies that h is a continuous mapping.  Theorem 11.7. A surjective mapping h : H −→ H on a Hilbert space H = 0H that preserves the inner product is a unitary operator on H. Proof.

This follows from the previous lemma and its proof.



May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 684

Mathematical Analysis for Machine Learning and Data Mining

684

Corollary 11.1. For a unitary operator h on a Hilbert space H = 0H we have h = 1. Proof. This follows immediately from the fact that h(x) = x for every x ∈ H.  11.3.3

Projection Operators

We shall prove that each projection operator on a Hilbert space corresponds to a closed subspace S. Theorem 11.8. Let p be a projection operator on a Hilbert space H. There exists a closed subspace S of H such that p = pS . Proof. Let Sp = {x ∈ H | p(x) = x}. It is clear that Sp is a subspace of H. If (xn ) is a sequence of elements in Sp such that limn→∞ xn = x, then p(x) = p( lim xn ) = lim p(xn ) = lim xn = x, n→∞

n→∞

n→∞

so x ∈ Sp . Thus, Sp is a closed subspace of H. If x ∈ H we can write x = p(x) + (x − p(x)), where p(x) ∈ Sp because p(p(x)) = p(x). We claim that x − p(x) ∈ Sp⊥ . Indeed, if y ∈ Sp we have (x − p(x), y) = (x − p(x), p(y)) = (p(x − p(x)), y) = (0H , y) = 0, hence x − p(x) ∈ Sp⊥ . We conclude that p is indeed the projection on the closed subspace Sp . In other words, p = pSp for every projection operator p of H.  Theorem 11.9. Let S and T be two closed subspaces of a Hilbert space H. We have pS pT = pT pS if and only if pS pT is a projection. Proof.

Suppose that pS pT = pT pS . This allows us to write (pS pT )∗ = p∗T p∗S = pT pS = pS pT ,

which proves that pS pT is self-adjoint. Furthermore, (pS pT )2 = (pS pT )(pS pT ) = pS (pT pS )pT = (pS pS )(pT pT ) = pS pT , which implies that pS pT is a projection. Conversely, suppose that pS pT is a projection. Then, pS pT = p∗S p∗T = (pT pS )∗ = pT pS .



May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Hilbert Spaces

page 685

685

Theorem 11.10. Let S and T be two closed subspaces of a Hilbert space H. If pS pT is a projection, then pS pT = pS∩T . Proof. By Theorem 11.9, we have pS pT = pT pS . Therefore, pS pT (H) = pS (pT (H)) ⊆ pS (H) ⊆ S and pS pT (H) = pT pS (H) = pT (pS (H)) ⊆ pT (H) ⊆ T , which implies pS pT (H) ⊆ S ∩ T . Conversely, if x ∈ S ∩ T we have x = pT (x) = pS pT (x), hence S ∩ T ⊆  pS pT (H). Thus, pS pT (H) = S ∩ T , which concludes the argument. Theorem 11.11. Let S and T be two closed subspaces of a Hilbert space H. We have S ⊥ T if and only if pS + pT is a projection. Proof. Suppose that pS + pT is a projection. If x ∈ S we have x2  (pS + pT )(x)2 = ((pS + pT )(x), (pS + pT )(x)) = ((pS + pT )2 (x), x) (because pS = pT is self-adjoint) = ((pS + pT )(x), x) (because pS = pT is idempotent) = (pS (x), x) + (pT (x), x) = (x, x) + (pT (x), x) = x2 + (pT (x), x). This implies (pT (x), x) = 0. For y ∈ T we can write (x, y) = (x, pT (y)) = (pT (x), y) = 0, which shows that S ⊥ T . Conversely, suppose that S ⊥ T . Since T ⊆ S ⊥ , it follows that pS (T ) = {0H }, hence pS (pT (x)) = 0H for every x ∈ H. Similarly, pT (pS (x)) = 0H for x ∈ H. This allows us to write (pS + pT )2 = p2S + pS pT + pT pS + p2T = pS + pT , (pS + pT )∗ = p∗S + p∗T = pS + pT , hence pS +pT is a self-adjoint and idempotent operator, that is, a projection.  The proof of Theorem 11.11 shows that if pS + pT is a projection then pS pT = 0. This suggests the following definition. Definition 11.4. The projection operators p, q in the Hilbert space H are orthogonal if pq = qp = 0. Theorem 11.12. Let S and T be two closed subspaces of a Hilbert space H. The following statements are equivalent: (i) S ⊆ T ; (ii) pT pS = pS ;

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 686

Mathematical Analysis for Machine Learning and Data Mining

686

(iii) (iv) (v) (vi)

pS pT = pS ; pT − pS is a projection operator; ((pT − pS )(x), x)  0 for every x ∈ H; pS (x)  pT (x) for every x ∈ H.

Proof. (i) implies (ii): Since pS (x) ∈ S ⊆ T , it follows that pT (pS (x)) = pS (x) for all x ∈ H, hence pT pS = pS . (ii) implies (iii): We have: pS pT = p∗S p∗T = (pT pS )∗ = p∗S = pS . (iii) implies (iv): We need to verify that the operator pT − pS is both self-adjoint and idempotent. Assuming (iii) we can write (pT − pS )∗ = p∗T − p∗S = pT − pS , (pT − pS )2 = p2T − pT pS − pS pT + p2S = pT − (pS pT )∗ − pS + pS = pT − pS . (iv) implies (v): Suppose that pT − pS is a projection. We have ((pT − pS )(x), x) = ((pT − pS )(x), (pT − pS )(x)) = (pT − pS )(x)  0. (v) implies (vi): It (v) holds we can write pT (x)2 − pS (x)2 = (pT (x), x) − (pS (x), x) = ((pT − pS )(x), x)  0, which yields (vi). (vi) implies (i): Suppose that pS (x)  pT (x) for every x ∈ H. Then, for x ∈ S we have x = pS (x)  pT (x)  x, hence x ∈ T . 11.4



Orthonormal Sets in Hilbert Spaces

Definition 11.5. A subset S of a Hilbert space H is orthogonal if x, y ∈ S and x = y imply (x, y) = 0. S is an orthonormal set if it is orthogonal and x = 1 for every x ∈ S. An orthonormal set S is called a complete orthonormal set if no orthonormal subset of H contains S as a proper subset.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Hilbert Spaces

page 687

687

Theorem 11.13. A non-trivial Hilbert space H (that is, a Hilbert space such that H = {0H }) contains a complete orthonormal set. If S is any orthonormal set in H then there exists a complete orthonormal set that includes S. Proof. Every non-trivial Hilbert space contains an orthonormal set. In1 x} is orthonormal. Consider the deed, if x ∈ H − {0H }, then the set { x collection of orthonormal subsets S that contain S as a subset. S is partially ordered by the inclusion relation. If S is a totally ordered subcollection of  S, then S is an orthonormal set and an upper bound of S . By Zorn’s Lemma, there exists a maximal element S0 of S, which is an orthonormal set that contains S. Since S0 is maximal it must be a complete orthonormal set.  Example 11.6. The set 1 1 1 1 1 √ , √ sin t, √ cos t, √ sin 2t, √ cos 2t, . . . π π π π 2π is an orthonormal set in the Hilbert space L2 ([−π, π]). This fact is a consequence of the elementary equalities: ⎧ ⎪ if m = n, ⎪ 7 π ⎨0 cos mx cos nx = π if m = n  1, ⎪ −pi ⎪ ⎩2π if m = n = 0,  7 π 0 if m = n, sin mx sin nx = π if m = n, −pi 7 π cos mx sin nx = 0 for m, n ∈ N. −pi

Example 11.7. For H = 2 (C) define en as the sequence en = (0, 0, . . . , 0, 1, 0, . . .), where 1 is in the nth position. The set S = {e0 , e1 , . . . , en , . . .} is orthonormal because en  = 1 and (en , em ) = 0, when n = m. We saw that the inner product is continuous in the topology induced by the norm induced by the inner product (Theorem 2.53). Theorem 11.14. If T is a non-empty set in a Hilbert space then x ∈ K(T ) and x ⊥ T implies x = 0H .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 688

Mathematical Analysis for Machine Learning and Data Mining

688

Proof. Since x ∈ K(T ) there exists a sequence (xn ) in T such that limn→∞ xn = x. By the continuity of the inner product, since (x, xn ) = 0,  it follows that (x, x) = 0, so x = 0H . Corollary 11.2. If T is a dense subset in the Hilbert space H and x ⊥ T , then x = 0H . Proof. Since K(T ) = H, this statement follows immediately from Theorem 11.14.  Let X = {x1 , . . . , xn } be a finite orthonormal set in a Hilbert space H and let x ∈ H. To determine the best approximation of x by linear  n   combinations of elements of X we need to evaluate x − i=1 ai xi : n  2    ai xi  x −

⎛

i=1

= ⎝x −

n 

ai xi , x −

i=1

= (x, x) − = (x, x) −

n  j=1 n 

n 

⎞ aj xj ⎠

j=1

aj (x, xj ) −

n 

ai (xi , x) +

i=1

n  n 

ai aj (xi , xj )

i=1 j=1

|(x, xi )|2

i=1

+

n 

|(x, xi )|2 −

i=1

= (x, x) −

n 

aj (x, xj ) −

j=1 n  i=1

n 

ai (xi , x) +

i=1

|(x, xi )|2 +

n 

n  n 

|ai |2 aj (xi , xj )

i=1 j=1

|ai − (x, xi )|2 .

i=1

These equalities show that the best approximation of x be linear combinations of a finite orthonormal set X = {x1 , . . . , xn } is obtained when the coefficients ai are chosen as ai = (x, xi ) for 1  i  n. The minimal distance  between x and the finite-dimensional subspace  X is x2 − ni=1 |(x, xi )|2 . Lemma 11.3. (Finite Bessel Inequality) Let X be a finite orthonormal subset of a Hilbert space H, X = {x1 , . . . , xn }. We have: n  i=1

for every x ∈ H.

|(x, xi )|2  x2

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Hilbert Spaces

Proof.

page 689

689

If ai = (x, xi ) for 1  i  n the previous equalities imply 2  n n       ai xi  = (x, x) − |(x, xi )|2  0, x −   i=1

i=1



which yields the inequality of the lemma.

Theorem 11.15. If S is an orthonormal subset of a Hilbert space H and x ∈ H, then the set Sx = {s ∈ S | (x, s) = 0} is at most countable.  Proof. Let Sxn = {s ∈ Sx | |(s, x)|  n1 }. It is clear that Sx = n1 Sxn . We claim that each set Sxn is finite. Indeed, if s1 , . . . , sm are m distinct elements of Sxn , by Lemma 11.3 we have x2 

m  i=1

|(x, sj )|2  m ·

1 , n2

which implies m  n2 x2 . Thus, each set Sxn is finite. This implies that  Sx itself is at most countable. Corollary 11.3. (Bessel’s Inequality) If S is an orthonormal subset of a Hilbert space H and x ∈ H, then  |(x, s)|2  x2 s∈S

for every x ∈ H. Proof. This more general form of Bessel inequality follows by listing the  elements of Sx and applying the finite form of Bessel inequality. Definition 11.6. Let H be a Hilbert space. An orthonormal basis is a maximal orthonormal subset of H. An orthonormal subset B of H is said to be total if x ⊥ B implies x = 0H . If {x1 , . . . , xn } is a finite total sequence in H, then H is said to have dimension n. H is infinite dimensional if there exists an infinite total orthonormal set in H. Theorem 11.16. Let H be a Hilbert space. A subset B of H is an orthonormal basis if and only if it is a total set.

May 2, 2018 11:28

690

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 690

Mathematical Analysis for Machine Learning and Data Mining

Proof. Let B be an orthonormal basis and suppose that B is not total. Then, there exists x0 ∈ H such that x0 = 0 and (x0 , y) = 0 for every y ∈ B. This implies that the set B ∪ { x10 x0 } is orthonormal and strictly includes B, which violates the maximality of B. Conversely, let B be a total subset and suppose that B is not an orthonormal basis. This may happen only if B is not maximal, which implies the existence of an orthonormal basis B1 with B ⊂ B1 . Thus, for x ∈ B1 − B we would have x ⊥ B and x = 1 (and therefore, x = 0H ), which contradicts the totality of B.  Theorem 11.17. Let H be a Hilbert space. A subset X = {x1 , x2 , . . .} of H is total if and only if for every x ∈ H we have the equality x =  {(x, xi )xi | xi ∈ X}.  Proof. Suppose that X is a total set. Let y = {(x, xi )xi | xi ∈ X}. For xj ∈ X we have: (x − y, xj ) = (x, xj ) − (y, xj )  {(x, xi )(xi , xj ) | xi ∈ X} = (x, xj ) − (by the continuity of the inner product) = (x, xj ) − (x, xj ) = 0, for every xj ∈ X. Thus, x − y ⊥ X, hence x = y.  Conversely, suppose that x = {(x, xi )xi | xi ∈ X} for every x ∈ H.  If x ⊥ X, (x, xj ) = 0 for every xj ∈ X, hence x = 0H . Definition 11.7. Let H be a Hilbert space and let X = {x1 , x2 , . . .} be a total set in H. The Fourier coefficients of an element x of H relative to X  are the numbers ai = (x, xi ). The series i1 ai xi is the Fourier series of x with respect to X. Theorem 11.18. Let H be a Hilbert space, X = {x1 , x2 , . . .} be a total set  in H, and let i1 ai xi be the Fourier series of x with respect to X. The sequence (sn ) of partial sums of the Fourier series converges to x n in H, where sn = i=1 ai xi . Proof. Observe that if i  n, then (x − sn , xi ) = 0 because (x, xi ) = (sn , xi ) = ci for i  n. Therefore, (x − sn , sn ) = 0, that is, x − sn ⊥ sn . Therefore, x2 = x − sn 2 + sn 2  sn 2 .    Since X is total we have sn 2 = ( ni=1 ai xi , ni=1 ai xi ) = ni=1 |ai |2 ,  n 2 hence i=1 |ai |2  f 2 , which shows that the series ∞ i=1 |ai | converges.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Hilbert Spaces

9in x 6in

b3234-main

page 691

691

n 2 Since sn − sm 2 = i=m+1 |ai | , it follows that (sn ) is a Cauchy sequence. The completeness of H means that there exists y ∈ H such that limn→∞ sn = y. Since (y, xi ) = limn→∞ (sn , xi ) = ai for i  1, it follows  that (x − y, xi ) = 0, so y = x because X is a total set. Theorem 11.19. Let H be a Hilbert space and let X be an orthonormal  subset in H. For x ∈ H we have x = {(x, xi )xi | xi ∈ X} if and only if  x2 = {|(x, xi )|2 | xi ∈ X}.  Proof. Suppose that x = {(x, xi )xi | xi ∈ X}. Then, 2         x2 = lim  (x, xi ) = lim |(x, xi )|2 = |(x, xi )|2 . n→∞  n→∞  xi ∈X in in  {|(x, xi )|2 | xi ∈ X} for every Conversely, suppose that x2 = x ∈ H. It is easy to see that 2    n     2   x − (x, x )x = x − |(x, xi )|2 . i i     i=1 in  By taking the limit when n → ∞ we obtain x − {(x, xi )xi | xi ∈ X} =  0H . Corollary 11.4. Let H be a Hilbert space and let X be an orthonormal subset in H. The following statements are equivalent: (i) X is an orthonormal basis; (ii) X is a total subset;  (iii) x = {(x, xi )xi | xi ∈ X} for every x ∈ H;  (iv) x2 = {|(x, xi )|2 xi ∈ X}; (v) K( X ) = H, that is, the subspace generated by X is dense in H. Proof. The equivalence of (i) and (ii) was shown in Theorem 11.16. In Theorem 11.17 we proved that (ii) is equivalent to (iii). Theorem 11.19 states the equivalence of (iii) and (iv). To complete the argument we prove two more implications.  (iii) implies (v): Suppose that x = {(x, xi )xi | xi ∈ X} for every  x ∈ H. Then x = limk→∞ ik (x, xi )xi , which proves that x ∈ K(X ). Thus, K(X ) = H. (v) implies (ii) Suppose that K( X) = H. If x ⊥ X, then x is orthogonal on a set that is dense in H (by hypothesis), so x = 0H by Corollary 11.2. 

May 2, 2018 11:28

692

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 692

Mathematical Analysis for Machine Learning and Data Mining

Example 11.8. Consider the set of complex-valued functions {fk | k ∈ Z, fk : [−π, π] −→ C}, where fk (x) = √12π eikx for k ∈ Z and x ∈ [−π, π]. It is immediate that 7 π 7 π 1 |fk |2 dx = dx = 1, fk 2 = 2π −π −π for k ∈ Z. Furthermore, for k = h we have: 7 π 1 (fk , fh ) = fk (x)fh (x) dx 2π −π 7 π 1 ei(k−h)x dx = 0, = 2π −π hence {fk | k ∈ Z} is an orthonormal set in L2 ([−π, π]). The subalgebra U of the algebra of complex-valued continuous function on [−π, π] generated by the functions {fk √| k ∈ Z, fk : [−π, π] −→ C} contains the constant functions because 1 = 2πf0 (x) for every x ∈ [−π, π]. The conjugate of the function fk is the function f−k , so U is closed with respect to conjugation. Furthermore, U separates points. Indeed, suppose that x, y ∈ [−π, π] and that fk (x) = fk (y) for every k ∈ Z. This means that eikx = eiky , or eik(x−y) = 1 for every k ∈ Z, or that cos k(x − y) = 1 and sin k(x − y) = 0 for every k ∈ Z, which implies x = y. Since this is not the case, U separates points. By the Complex Stone-Weierstrass Theorem (Theorem 5.40), it follows that the subspace of L1 ([−π, π]) generated by the functions {fk | k ∈ Z} is dense in L1 ([−π, π]), which by Corollary 11.4 implies that the set { √12π eikx | k ∈ Z} is an orthonormal basis in C([−π, π]). Since fk (x) = √12π (cos kx + i sin kx) for k ∈ Z and x ∈ [−π, π], we have: : : 1 1 1 1 √ cos kz = (fk (x) + f−k (x)) and √ sin kz = (fk (x) − f−k (x)). π 2 π 2i Therefore, the set 1 1 1 1 1 √ , √ sin t, √ cos t, √ sin 2t, √ cos 2t, . . . π π π π 2π is also an orthonormal basis in L2 ([−π, π]). Theorem 11.20. Let H be a Hilbert space and let (xn ) be an orthogonal  sequence in H. The series n xn converges if and only if the numerical    series n xn 2 converges. In this case, || n xn ||2 = n xn 2 and the  sum n xn is independent of the order of the terms.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Hilbert Spaces

page 693

693

Proof. Since the sequence (xn ) consists of pairwise orthogonal elements it is immediate that xn+1 + xn+2 + · · · + xm 2  xn+1 2 + xn+2 2 + · · · + xm 2 for m  n + 1. This shows that the sequences of partial sums of the series   xn 2 are simultaneously Cauchy sequences. n xn and  Since xn 2 has non-negative terms its sum does not depend on the order of the terms.  If the series n xn is convergent its sum is independent of the order of the terms.   xn be perSuppose that the series yn is obtained from the series  muting the order of the terms. Let y = yn . Then, x−y =



(xn − yn ) = lim

k→∞

 (xk − yk ), k

which shows that x − y belongs to the closure of the subspace generated by the set {xn | n ∈ N}. For xp in this set, we have  (x − y, xp ) = lim

n−→∞

n 

 (xk − yk ), xp

k=1

= lim

n→∞

n 

((xk , xp ) − (yk , xp )) = 0,

k=1

so x − y is orthogonal on the set {xn | n ∈ N}. Thus, x = y.



Theorem 11.21. Let H be a Hilbert space and let S = {x1 , x2 , . . .} be an orthonormal basis in H. For u, v ∈ H we have (u, v) =



(u, xn )(xn , v)

n1

and u2 =



|(u, xn )|2 .

n1

Proof. We have shown (see Theorem 11.15) that if S is an orthonormal set in H then the sets Su = {s ∈ S | (u, s) = 0} and Sv = {s ∈ S |  (v, s) = 0} are at most countable, u = limn→∞ nj=1 (u, xj )xj and v = n limn→∞ k=1 (u, xk )xk . Since the inner product is continuous, we may

May 2, 2018 11:28

694

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 694

Mathematical Analysis for Machine Learning and Data Mining

further write

⎛ ⎞ n n   (x, y) = lim ⎝ (u, xj )xj , (v, xk )xk ⎠ n→∞

= lim

n→∞

= lim

n→∞

= lim

n→∞

j=1 n n  

k=1

(u, xj )(v, xk )(xj , xk )

j=1 k=1 n 

(u, xj )(v, xj ))

j=1 n  j=1

(u, xj )(xj .v)) =

∞ 

(u, xj )(xj , v)).

j=1

Taking u = v we obtain the second part of the theorem.



Theorem 11.22. Let H be a Hilbert space. The following statements are equivalent: (i) H is a separable space; (ii) there exists an orthonormal basis for H that is countable. Proof. (i) implies (ii): Suppose that H is separable, that is, it contains a countable dense subset. Let X be an orthonormal basis. The separability of H implies the separability of X and since X is the unique dense subset of X, it follows that X must be countable. (ii) implies (i): Suppose that X = {x1 , x2 , . . .} is a countable orthonormal basis for H. We deal initially with a real Hilbert space H.  Let Dn = {x ∈ H | x = nj=1 qj xj , qj ∈ Q for 1  j  n} for n  1.  We have K(Dn ) =  x1 , . . . , xn . Therefore, for D = n1 Dn we have K(D) = H, so H is separable. If H is a complex Hilbert space, define Dn as Dn = {x ∈ H | x = n j=1 (qj + irj )xj , qj ∈ Q for 1  j  n} for n  1 and proceed in a similar manner.  Theorem 11.23. Every infinitely dimensional, separable Hilbert space is isometric to 2 . Proof. Since H is infinitely dimensional and separable, it has a countably infinite orthonormal basis S = {x1 , . . . , xn , . . .} by Theorem 11.22. Define the linear mapping f : H −→ 2 as f (x) = (x, xn )n1

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Hilbert Spaces

page 695

695

for x ∈ H. We saw that (x, y) = (f (x), f (y)) and x = h(x)2 . Since Null(f ) = {0H }, f is injective. Let a = (an ) ∈ 2 . Since an xn and am xm are orthogonal and ∞ ∞   an xn 2 = |an |2 , n=1

n=1

∞

it follows that the series n=1 an xn is convergent. If we define x = ∞ a x , then a = f (x), so f is surjective. Since f preserves the norm, n n n=1  it follows that H is isometric with 2 . Let h be a bounded operator on a Hilbert space H and let {ei | i ∈ I} be an orthonormal basis in H. Recall that we proved in Theorem 11.21   that if x = i∈I (x, ei )ei and y = j∈I (y, ej )ej , it follows that  (x, ei )(ei , y). (x, y) = i∈I

Theorem 11.24. Let {ei | i ∈ I} and {fj | j ∈ J} be two orthonormal bases in a Hilbert space H and let h be a bounded linear operator on H. We have    h(ei )2 = h(fj )2 = |(h(ei ), fj )|2 . i

Proof.

j

i

j

We have:   |(h(ei ), fj )|2 = (h(ei ), fj )(h(ei ), fj ) i

j

i

=



j

⎛ ⎝

i

=





⎞ (h(ei ), fj )(fj , h(ei ))⎠

j

(h(ei ), h(ei ) =

i



h(ei )2 .

i

In a similar manner it is possible to prove the other equality.   Theorem 11.24 shows that the value of i h(ei )2 depends " only on the 2 operator h and not the orthonormal basis. The number i h(ei ) is denoted by νSC (h). Note that   h(ei )2 = |(ei , h∗ (fj ))|2 νSC (h)2 = i

=

 j

i ∗

2

j

h (fj ) = νSC (h∗ )2 .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 696

Mathematical Analysis for Machine Learning and Data Mining

696

Theorem 11.25. For a linear operator h we have h  νSC (h). Moreover νSC is a norm on the linear space of linear operators on H. Proof.

Let be an arbitrary positive number. Since  νSC (h) = h(ei )2 , i

if the basis {ei | i ∈ I} is chosen such that h(e1 )  h − (which  2 is possible by the definition of h) we have νSC (h)2 = i h(ei )  2 (h − ) . Since this inequality holds for every we have νSC (h)  h. Let h, g be two linear operators on H and let {ei | i ∈ I} be an orthonormal basis of H. We have:   (h + g)(ei )2 = (h(ei ) + g(ei )2 νSC (h + g)2 = i

=



h(ei )2 +

i



i

g(ei )2 + 2



i

h(ei )g(ei )

i

 νSC (h)2 + νSC (g)2 + 2νSC (h)νSC (g) = (νSC (h) + νSC (g))2 , because 

h(ei )g(ei ) 

;

i

h(ei )2

;

i

g(ei )2 .

i

Thus, νSC (h + g)  νSC (h) + νSC (g). The equality νSC (ah) = |a|νSC (h)  for a ∈ C is immediate, so νSC is indeed a norm. We refer to νSC as the Schmidt norm. Theorem 11.26. Let h be a linear operator on a Hilbert space H. If νSC (h) is finite, then the operator h is compact. Proof. Let (en ) be an orthonormal basis in H. For n ∈ N define the operator gn with a finite-dimensional range as gn (x) =

n  i=0

for x ∈ H.

(h(x), ei )ei ,

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Hilbert Spaces

page 697

697

Each of the operators gn is compact because it has a finite-dimensional ∞ range. Since h(x) = i=0 (h(x), ei )ei , it follows that 2  ∞      2 h(x) − gn (x) =  (h(x), ei )ei    =

=

i=n+1 ∞ 

|(h(x), ei )|2

i=n+1 ∞ 

|(x, h∗ (ei ))|2

i=n+1

 x2

∞ 

h∗ (ei )2 .

i=n+1

If νSC (h) is finite, then for any > 0 and sufficiently large n we have h(x) − gn (x)  x, so h − hn  < , so, by Theorem 9.12 h is a compact operator.



Definition 11.8. A Hilbert-Schmidt operator is a linear operator on a Hilbert space H such that νSC (h) is finite. The set of Hilbert-Schmidt operators on the Hilbert space is denoted by HS(H). Example 11.9. Let L2 (S, E, m) be a separable space and let K : S ×S −→ C be a Hermitian function 7 7 such that |K(x, y)|2 dx m dy m < ∞.

S

S

Consider the linear operator h : L2 (S, E, m) −→ L2 (S, E, m) defined by 7 (h(f ))(x) = K(x, y)f (y) dy m. S

We show that h is a Hilbert-Schmidt operator and that every such operator on L2 (S, E, m) can be obtained in this manner. Since L2 (S, E, m) is separable, by Theorem 11.22 there exists a countable orthonormal basis {en | n ∈ N} in this space. By Fubini-Lebesgue Theorem (Theorem 8.68), for almost every x ∈ S, Kx (y) = K(x, y) defines a function in L2 (S, E, m). Hence, for almost every x ∈ S we have 7 7 y h(en )(x) = K(x, y)en (y) d m = Kx (y)en (y) dy m S S 7  = |(Kx , en )|2 dy m. S

n∈N

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 698

Mathematical Analysis for Machine Learning and Data Mining

698

By the Dominated Convergence Theorem (Theorem 8.37), 7   7 h(en )2 = |(Kx , en )|2 dy m = |(Kx , en )|2 dy m. n∈N

S

n∈N

X

n∈N

Since {en | n ∈ N} is an orthonormal basis in H, by Supplement 4, the set {en | n ∈ N} is also an orthonormal basis in H. Thus,  Kx = (Kx , en )en n∈N

and Kx 2 =



|(Kx , en )|2 .

n∈N

We conclude that 7 7  h(en )2 = Kx 2 dy m = (K(x, y) dy m) dx m < ∞ S

n∈N

S

because K ∈ L2 (S×T, E×E, m×m). This shows that h is a Hilbert-Schmidt operator. Conversely, any Hilbert-Schmidt operator on L2 (S, E, m) is an operator defined by a Hermitian function K ∈ L2 (S × S, E × E, m × m). Indeed, suppose that h is a Hilbert-Schmidt operator on L2 (S, E, m) and let {en | n ∈ N} be an orthonormal basis of L2 (X, E, m). If f ∈ L2 (S, E, m) we have ⎞ ⎛   7  (f, en )en ⎠ (x) = f (y)en (y) dy m (h(en ))(x). h(f )(x) = h ⎝ n∈N

n∈N

S

  Since n∈N h(en )2 < ∞, the series n∈N h(en ) converges in L2 (S, E, m). By Fubini’s Theorem, the function K : S × S −→ C defined by  h(en )(x)en (y) K(x, y) = n∈N 2

belongs to L (S, E, m). By the dominated convergence theorem   7 y f (y)en (x) d m (h(en ))(x) (h(f ))(x) = n∈N

7

⎛

f (y) ⎝

= 7

S

S



⎞ (h(en ))(x)en (y)⎠ dy m

n∈N

K(x, y)f (y) dy m.

= S

We shall refer to the function K as an integral kernel.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Hilbert Spaces

b3234-main

page 699

699

Example 11.10. Let S = [0, ∞) and let K be defined by  1 if 0  y  x, K(x, y) = 0 if x < y. Define the integral operator v as

7

(vf )(x) =

1

K(x, y)f (y) dy. 0

The operator v is compact because it is a Hilbert-Schmidt operator. We have 7 x 7 1 K(x, y)f (y) dy = f (y) dy. 0

0

Thus, for the function g defined by g(x) = (vf )(x) for x ∈ [0, ∞) we have g(0) = 0 and g  (x) = f (x). Suppose that λ = 0 is an eigenvalue of v and (vf )(x) = λf (x). Then x f (x) = λf  (x) and this implies that f (x) = ke λ for some k ∈ R. However, we have x

x

λke λ − λk = λke λ , which implies λk = 0. Since λ = 0, this implies k = 0, so f is the function that is constant 0 on [0, ∞), which contradicts the assumption that λ is an eigenvalue. For λ = 0 there are no eigenvectors because (vf )(x) = 0 implies f (x) = 0 for x ∈ [0, ∞]. The operator v is known as the Volterra operator. The completeness of Hilbert spaces allows us to prove a stronger form of Theorem 2.53. Theorem 11.27. Let H be a Hilbert space. If S is a closed subspace of H and x ∈ H, then there exists a unique m0 ∈ S such that x − m0   x − y for every y ∈ S. Furthermore, m0 is a unique vector in S that minimizes x − y if and only if x − m0 is orthogonal on S. Proof. In view of Theorem 2.53 we need to establish only the existence of m0 . If x belongs to S, then m0 = x and the argument is completed. Suppose that x ∈ S (see Figure 11.1). Let d = inf{x − y | y ∈ S} and let z0 , . . . , zn , . . . be a sequence of members of S such that limn→∞ x − zi  = d.

May 2, 2018 11:28

700

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 700

Mathematical Analysis for Machine Learning and Data Mining

y

x−y

x

m0

0H

• S

Fig. 11.1

Configuration of elements involved in Theorem 11.27.

By an application of the parallelogram equality (Theorem 2.27) to x−zi and zj − x we have: 2(x − zi 2 + zj − x2 ) = zj − zi 2 + 2x − zi − zj 2 , which implies: zj − zi 2 = 2(x − zi 2 + zj − x2 ) − 2x − zi − zj 2 2   zi + zj  2 2   = 2(x − zi  + zj − x ) − 4 x − . 2  Since S is a subspace of H, we have implies

zi +zj 2

∈ S, so ||x −

zi +zj 2 ||

 d, which

zj − zi 2  2(x − zi 2 + zj − x2 ) − 4d2 . By the definition of the sequence (zi ), we conclude that this is a Cauchy sequence and, since S is a closed subspace, there exists m0 ∈ S such that zi → m0 . By the continuity of the norm (see Corollary 6.6) it follows that x − m0  = d.  Corollary 11.5. Let H be a Hilbert space, S be a closed subspace of H, and let x ∈ H. If V = tx (S), there exists a unique vector x0 in V of minimum norm and x0 ∈ S ⊥ . Proof. Let m0 ∈ S be the element of S whose existence and uniqueness was established in Theorem 11.27 (see Figure 11.2). For x0 = x − m0 we have x0   x − y for every y ∈ S and x0 − m0 is orthogonal on S, hence  x0 ∈ S ⊥ .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Hilbert Spaces

page 701

701

x S⊥ m0 x0

0H •

V =x+S S Fig. 11.2

Configuration of elements involved in Corollary 11.5.

Theorem 11.27 shows that x0 = x − m0 ∈ S ⊥ is of minimal norm among the vectors of the form x − y, where y ∈ S. It allows us to introduce an operator pS on the Hilbert space H defined by pS (x) = x0 , where x0 ∈ S is such that x − y is minimal and x − pS (x) is orthogonal on S. Definition 11.9. Let S a closed subspace of a Hilbert space. The projection of H on S is the operator pS . Recall that for any subset T of a topological inner product space we have shown that T ⊥ is a closed subspace (see Theorem 6.19). This holds, of course, in a Hilbert space. Theorem 11.28. If T is a subset of a Hilbert space, then (T ⊥ )⊥ is the smallest closed subspace of H that contains T . Proof. We already know that (T ⊥ )⊥ is a closed subspace of H that contains T . Suppose that W is a subspace of H that contains T . Then,  (T ⊥ )⊥ ⊆ (W ⊥ )⊥ = W , which concludes the proof. Theorem 11.29. Let S be a closed subspace of a Hilbert space H. Then, H = S ⊕ S⊥. Proof.

This follows immediately from the equality x = pS (x) + (x − pS (x))

for x ∈ H because pS (x) ∈ S and x − pS (x) ∈ S ⊥ .

(11.3) 

May 2, 2018 11:28

702

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 702

Mathematical Analysis for Machine Learning and Data Mining

Corollary 11.6. If T is a closed linear subspace of a Hilbert space, then T = (T ⊥ )⊥ . Proof. By Theorem 11.28, it suffices to show that (T ⊥ )⊥ ⊆ T . Let x ∈ (T ⊥ )⊥ . By Theorem 11.29, there exist y ∈ T and z ∈ T ⊥ such that x = y + z. Since x ∈ (T ⊥ )⊥ and y ∈ (T ⊥ )⊥ , it follows that x − y ∈ (T ⊥ )⊥ , so z ∈ (T ⊥ )⊥ . The fact that z ∈ T ⊥ implies that z ⊥ z, so z = 0H . Thus,  x = y ∈ T , hence (T ⊥ )⊥ ⊆ T , which concludes the argument. Theorem 11.30. For a closed subspace S of a Hilbert space pS is a bounded linear operator. Proof. Note that for a closed subspace S of H equality (11.3) is equivalent to x = pS (x) + pS ⊥ (x). Therefore, for x, y ∈ H we have x = pS (x) + pS ⊥ (x) and y = pS (y) + pS ⊥ (y), which implies x + y = pS (x) + pS ⊥ (x) + pS (y) + pS ⊥ (y) = pS (x) + pS (y) + pS ⊥ (x) + pS ⊥ (y). From the uniqueness of the decomposition, it follows that pS (x + y) = pS (x) + pS (y) and pS ⊥ (x + y) = pS ⊥ (x) + pS ⊥ (y). Similarly, ax = apS (x)+apS ⊥ (x), where apS (x) ∈ S and apS ⊥ (x) ∈ S ⊥ , which implies pS (ax) = apS (x), showing that pS is a linear operator. Since x2 = pS (x)2 + pS ⊥ (x)2 , it follows that x  pS (x), so pS   1. On the other hand, if x ∈ S we have pS (x) = x, hence  pS  = 1. Theorem 11.31. For a closed subspace S of a Hilbert space H we have pS ⊥ = 1H − pS and S ⊥ = {x ∈ S | pS (x) = 0H }. Proof. Since every x ∈ H can be written as x = pS (x) + pS ⊥ (x), it follows that pS ⊥ (x) = x − pS (x) for every x ∈ H, that is, pS ⊥ = 1H − pS . Thus, S ⊥ = {x ∈ H | x = pS ⊥ (x)} = {x ∈ H | x − pS (x) = x} = {x ∈ H | pS (x) = 0H }.  The next statement is a stronger form of a previous theorem for inner product spaces (Theorem 2.44). Theorem 11.32. Let H be a Hilbert space, B = {b1 , . . . , bk } be a linearly independent set in H and let S =  B.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Hilbert Spaces

page 703

703

 c1  For c =

. . . ck

∈ Rk let Uc = {x ∈ H | (x, bi ) = ci for 1  i  k}.

The vector x0 of minimal norm in Uc belongs to the subspace S. Fur d1  k thermore, if x0 = i=1 di bi , then GY d = c, where d = ... and GB is dk

the Gram matrix of the set B. Proof. We have shown in Theorem 2.44 that Uc is the translation of S ⊥ (which is a closed subspace), where S =  b1 , . . . , bk . Since S ⊥ is closed, by Corollary 11.5, there exists a unique x0 ∈ (S ⊥ )⊥ = S such that x0  is k minimal. Since S =  Y  , x0 = i=1 di yi . The definition of Uc implies the  equality GB d = c. 11.5

The Dual Space of a Hilbert Space

Theorem 11.33. (The Riesz’3 Representation Theorem) Let H be a Hilbert space and let f : H −→ F be a continuous linear functional. There is a unique a ∈ H such that f (x) = (x, a) for every x ∈ H. Moreover, f  = a. Proof. If f is the zero functional the statement is immediate by taking a = 0. So, suppose that f is a non-trivial functional and, therefore, a surjective function. Since f is continuous, Null(f ) is a closed subspace of H by Theorem 9.14. We have Null(f ) ⊂ H, so (Null(f ))⊥ = {0}. There exists b ∈ (Null(f ))⊥ such that f (b) = 1. For x ∈ H we have f (x − f (x)b) = f (x) − f (x)f (b) = 0, so x − f (x)b ∈ Null(f ). Taking into account that b ∈ (Null(f ))⊥ , it follows that 0 = (x − f (x)b, b) = (x, b) − f (x)(b, b) = (x, b) − f (x)b2 , which yields f (x) = tation of f .

1 b 2 (x, b).

Taking a =

b b 2

gives the desired represen-

3 Frigyes Riesz (Jan. 22, 1880–Feb. 28, 1956) was a Hungarian mathematician and pioneer of functional analysis. Riesz taught mathematics at the University of Cluj from 1911 and later at the University of Budapest.

May 2, 2018 11:28

704

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 704

Mathematical Analysis for Machine Learning and Data Mining

To prove the uniqueness of a suppose that (x, a) = (x, b) for x ∈ H. Then, (x, a − b) = 0, and, by taking x = a − b we have a − b = 0, which yields a − b = 0. By Cauchy-Schwarz Inequality (Theorem 2.23) we have |f (x)| = |(x, a)|  xa. Thus, f is bounded and f   a. Moreover, we have     a a = , a = a, f a a so f  = a.



Corollary 11.7. Each real Hilbert space is isomorphic with its dual H ∗ . Proof.

This is a direct consequence of Theorem 11.33.



If T is an orthonormal set in a Hilbert space H, then there exists a basis B for H that contains T .

11.6

Weak Convergence

Convergence of sequences in a Hilbert in the sense of the norm is also known as strong convergence. We consider in this section an alternative, weaker form of convergence. Definition 11.10. A sequence x in a Hilbert space converges weakly if for every y ∈ H the sequence of scalars ((xn , y)) is convergent. An element x of H is the weak limit of a sequence (xn ) if limn→∞ (xn , y) = (x, y) for every y ∈ H. This is denoted by xn →w x. Example 11.11. Note that if limn→∞ xn = x, them xn →w x due to the continuity of the inner product in Hilbert spaces. The converse implication is false. Indeed, consider an infinite orthonormal basis (en ) in a Hilbert   2 space H and let y = n an en . The series n |(y, en )| is convergent, hence limn→∞ (y, en ) = 0 for every y ∈ H, so (en ) is a weakly convergent sequence. However the sequence (en ) does not converge because en  = 1 for every n ∈ N. Lemma 11.4. Let (xn ) be a weakly convergent sequence in a Hilbert space H. There exists y˜ ∈ H and two positive numbers M and r such that for all y , r). n ∈ N we have |(xn , y)| < M for all y ∈ B(˜

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Hilbert Spaces

9in x 6in

b3234-main

page 705

705

Proof. The lemma stipulates that there exists an open sphere B(˜ y , r) such that |(xn , y)| is bounded when y ranges in this sphere. Suppose that the statement is false, that is, |(xn , y)| is unbounded in every sphere. Thus, n1 , y1 exist such that |(xn1 , y1 )| > 1. Since the inner product is continuous we have (xn1 , y) > 1 for every y ∈ B(y1 , r1 ) for some r1 > 0. Further, suppose that exists n2 , y2 such that y2 ∈ B(y1 , r1 ) and |(xn2 , y2 )| > 2. As before, the continuity of the inner product we have |(xn2 , y)| > 2 for y ∈ B(y2 , r2 ) such that B(y2 , r2 ) ⊆ B(y1 , r1 ) and r2 < r21 . By continuing this process we construct the sequences (np ), (yp ), and (rp ) such that |(xnp , y)| > p for y ∈ B(yp , rp ), yp − yp+1  < rp , and  p−1 1 1 rp < rp−1 < · · · < r1 . 2 2 This implies yp − yp+q   yp − yp+1  + · · · + yp+q−1 − yp+q    q−1  1 1 < rp 1 + + · · · + 2 2    p−1 1 1 = 2rp 1 − q < 2 r1 . 2 2 Thus, limp→∞ yp − yp+q  = 0, so (yp ) is a Cauchy sequence, which implies the existence of y = limn→∞ yn . Since each sphere B(yp , rp ) contains almost all members of the sequence (yn ) it must contain y, so |(xnp , y)| > p for all p. Since (xnp ) converges weakly, the sequence ((xnp , y)) is bounded, which contradicts the previous inequality.  Theorem 11.34. Every weakly convergent sequence in a Hilbert space is bounded. Proof. Let x be a weakly convergent sequence. Since the sequence ((xn , y)) is convergent, it is also bounded. In principle, the bound depends on y. However, we will show that an upper bound can be chosen independent of y. By Lemma 11.4, since (xn ) be a weakly convergent sequence, there exists y˜ ∈ H and two numbers M and r such that for all n ∈ N we have |(xn , y)| < M for all y ∈ B(˜ y , r). For any z with z  1, we have |(xn , y˜ + rz)| < M , hence |(xn , y˜) + r(xn , z)| < M . This implies r|(xn , y˜)|  |r(xn , z) + (xn , y˜)| + | − r(xn , y˜)| < 2M. Taking z = 2 x1n xn we obtain xn  < 4M  r .

May 2, 2018 11:28

706

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 706

Mathematical Analysis for Machine Learning and Data Mining

Theorem 11.35. A sequence (xn ) in a Hilbert space may have at most one week limit. Proof. Suppose that (xn ) is a sequence in H and we have both xn →w u and xn →w v. Then, limn→∞ (xn , y) = (u, y) and limn→∞ (xn , y) = (v, y) for each y ∈ H. This implies (u − v, y) = 0 for every y ∈ H, so u − v = 0H , which yields u = v.  Lemma 11.5. Let (xn ) be a sequence in a Hilbert space such that every subsequence (xni )i∈N contains a subsequence (xnij )j∈N that converges strongly to an element x of H. Then limn→∞ xn = x, that is, (xn ) converges strongly at x. Proof. Suppose that the condition of the lemma is satisfied but (xn ) does not converge strongly at x. Then, there exists a positive number such that for any n0 there exists n1 > n0 such that xn1 − x  . Similarly, there exists n2 > n1 such that xn2 − x  , etc. Thus, we have a subsequence (xni )i∈N with xni − x  , which cannot converge to x. Therefore, no subsequence of this subsequence may converge to x, which contradicts the  hypothesis. Thus, limn→∞ xn = x. Theorem 11.36. Let C be a compact subset of a Hilbert space H and let (xn ) be a sequence such that {xn | n ∈ N} ⊆ C. If xn →w x, then limn→∞ xn = x. Proof. By the compactness of C, every subsequence (xni )i∈N contains a subsequence (xnij )j∈N that converges strongly to some u ∈ H. By Lemma 11.5, we have xnij →w v for some v ∈ H. However, we also have xnij →w u as a subsequence of (xn ). By Theorem 11.35 we have u = v, which gives the desired conclusion.  Theorem 11.37. Let (xn ) be a sequence in a Hilbert space H. If xn →w x and h is a compact linear operator on H, then limn→∞ h(xn ) = h(x). Proof. Since xn →w x, the sequence (xn ) is bounded by Theorem 11.34. Therefore, the sequence (h(xn )) is mapped by h into a subset of a compact set (by Definition 9.2). Note that (h(xn ), y) = (xn , h∗ (y)). Since xn →w x we also have limn→∞ (xn , h ∗ (y)) = (x, h∗ (y)) = (h(x), y), so limn→∞ (h(xn ), y) = (h(x), y) for every y. By Theorem 11.36 we have the strong convergence  limn→∞ h(xn ) = h(x).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Hilbert Spaces

11.7

page 707

707

Spectra of Linear Operators on Hilbert Spaces

In this section we focus on spectral properties of Hilbert space operators. Example 11.12. Take K(x, y) = cos(x − y) in the operator h : L2 ([0, 2π]) −→ L2 ([0, 2π]) defined in Example 11.4. Seeking an eigenvalue λ and an eigenvector u (that is, a function from L2 ([0, 2π]) for this operator amounts to 7 b h(f )(x) = cos(x − y)f (y) dy = λf (x) a

for x ∈ [0, 2π]. Note that the last equality can be written as 7 b 7 b cos x cos yf (y) dy + sin x sin y dy = λf (x), a

(11.4)

a

which means that every eigenvector (in this case we use the term eigenfunction) has the form f (x) = a cos x + b sin x for some a, b ∈ R. Substituting f in equality (11.4) and taking into account that 7 2π 7 2π y cos dy = siny dy = π, 6 2π

0

0

and 0 cos y sin ydy = 0, it follows that λ = π. Thus, h has a unique eigenvalue, π, and every eigenfunction has the form a cos x + b sin x. The set of eigenfunctions for λ = π has dimension 2. The invariant subspace that corresponds to λ = 0 consists of all functions f such that 6 2π 6 2π cos yf (y) dy = sin yf (y) dy = 0 and this is an infinitely dimensional 0 0 2 subspace of L ([0, 2π]). Theorem 11.38. Let h : H −→ H be a self-adjoint linear operator on a Hilbert space H. If K is an invariant subspace for h, then so is K ⊥ . Proof. Let x ∈ K ⊥ and y ∈ K. We have h(y) ∈ K because K is an invariant subspace. This implies (h(x), y) = (x, h(y)) = 0, so h(x) ∈ K ⊥ . Since x is an arbitrary element of K, K ⊥ is also an invariant subspace for h.  Let h : H −→ H be a self-adjoint operator on the Hilbert space H. Define the numbers mh and Mh as mh = inf{(h(x), x) | x ∈ B[0H , 1]} and Mh = sup{(h(x), x) | x ∈ B[0H , 1]}. It follows immediately that mh x2  (h(x), x)  Mh x2 for x ∈ H.

May 2, 2018 11:28

708

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 708

Mathematical Analysis for Machine Learning and Data Mining

Theorem 11.39. If λ is an eigenvalue of a self-adjoint operator h on H, then λ is a real number and mh  λ  Mh . Proof. Since h is self-adjoint we have (h(x), x) = (x, h(x)) for an eigenvector x that corresponds to λ, which amounts to λ(x, x) = λ(x, x). We have λ = λ because x = 0H , which means that λ is a real number. 1 If y is an eigenvector corresponding to λ, then x = y y is an eigenvector corresponding to the same λ and x = 1.  Since (h(x), x) = (λx, x) = λ(x, x) = λ we obtain mh  λ  Mh . Corollary 11.8. If λ is an eigenvalue of a self-adjoint linear operator h on H then |λ|  h. Proof.

By Theorem 11.39, we have |λ|  max{|mh |, |Mh |} = h.



Theorem 11.40. If λ and μ are two distinct eigenvalues of a self-adjoint linear operator h on the Hilbert space H, then Hλ ⊥ Hμ . Proof. Let x ∈ Hλ and y ∈ Hμ . We have h(x) = λx and h(y) = μy. This implies λ(x, y) = (λx, y) = (h(x), h) = (x, h(y)) (because h is self-adjoint) = (x, μy) = μ(x, y), which implies (x, y) = 0. Thus, Hλ ⊥ Hμ .



Theorem 11.41. Any compact self-adjoint operator has at least one eigenvalue. Proof. Let h be a compact self-adjoint operator on the Hilbert space H. Since h = sup{(h(x), x) | x = 1}, there exists a sequence (xn ) in H such that xn  = 1 for n ∈ N, limn→∞ |(h(xn ), xn )| = 1, and all numbers (h(xn ), xn ) are real. Therefore, we may have the following three case: (i) the sequence ((h(xn ), xn )) converges to h, (ii) the sequence ((h(xn ), xn )) converges to −h, (iii) the sequence ((h(xn ), xn )) has two limit points, h and −h. The sequence ((h(xn ), xn ) may always be assumed to be convergent. This is obvious in the first two cases; in the last case, we can select the subsequence

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Hilbert Spaces

page 709

709

that converges to h or the subsequence that converges to −h and we reach the same conclusion. Let λ = limn→∞ (h(xn ), xn ). Since h is compact and the sequence (xn ) is bounded, there exists a subsequence (h(xnk )) that is convergent. Let z = limk→∞ h(xnk ). Since h is self-adjoint, λ is a real number. Define the sequence (ynk ) as ynk = h(xnk ) − λxnk . We have: ynk 2 = h(xnk ) − λxnk 2 = h(xnk )2 − 2λ(h(xnk ), xnk ) + λ2 . By the definition of λ we have: lim h(xnk ) − λxnk 2 = z2 − λ2 .

(11.5)

z = lim h(xnj )  h = |λ|.

(11.6)

k→∞

We have k→∞

Equality (11.5) and inequality (11.6) imply z = λ and limk→∞ (h(xnk ) − λxnk ) = 0. It follows that the sequence (xnk ) is convergent and x = lim xnk = lim k→∞

k→∞

z h(xnk ) − ynk = . λ λ

Since limk→∞ xnk = x, it follows that limk→∞ h(xnk ) = h(x). Thus, we have limk→∞ h(xnk ) = z = λx and, also, limk→∞ h(xnk ) = h(x). Consequently, h(x) = λx, which shows that λ is an eigenvalue of h  having x = limk→∞ xnk as an eigenvector with x = 1. Theorem 11.42. Let h be compact self-adjoint linear operator on a Hilbert space H. We have max{|λ| | λ ∈ σ(h)} = h. Proof. We have shown in Corollary 11.8 that |λ|  h for every eigenvalue of a self-adjoint operator h on H. The eigenvalue λ whose existence we have shown in Theorem 11.41 satisfies this statement.  Now we can formulate a much stronger version of Theorem 11.41. Theorem 11.43. The set of eigenvalues of a compact self-adjoint linear operator h on a Hilbert space H is a set of real numbers that is at most countable. If this set is not finite, than the set consists of members of a sequence that converges to 0.

May 2, 2018 11:28

710

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 710

Mathematical Analysis for Machine Learning and Data Mining

Proof. Suppose that the set of eigenvalues of h is infinite and does not consist of the members of a sequence that converges to 0. Then, there exists a ∈ R such that the set {λ ∈ σ(h) | |λ| > a} is infinite. Let (λn ) be a sequence of distinct eigenvalues that belong to this set and let un be an eigenvector of λn with un  = 1. The set {un | n ∈ N} is orthonormal (by Theorem 11.40) and we have h(un ) − h(um )2 = λn un − λm um 2 = λ2n + λ2m  2a2 , which implies that (h(un )) has no Cauchy subsequence, and therefore, no convergent subsequence. This contradicts the compactness of h because the  sequence (un ) is bounded. Lemma 11.6. Let h be a compact, self-adjoint operator on a Hilbert space H. There exists an orthonormal set of eigenvectors {x1 , x2 , . . .} with the corresponding eigenvalues λ1 , λ2 , . . . such that for any x ∈ H we have  x= cn xn + y (11.7) n

for some scalars cn where y ∈ Null(h). If H is infinite-dimensional, then limn→∞ λn = 0. Proof. Let λ1 be the eigenvalue of h such that |λ1 | = h and let G1 be the subspace of H generated by an eigenvector x1 that corresponds to λ1 such that x1  = 1. Define H1 = G1 and H2 = H1⊥ . Note that if y ∈ H2 , then h(y) ∈ H2 because for x ∈ H1 we have (h(y), x) = (y, h(x)) = 0 and, therefore, h(x) ∈ H1⊥ = H2 . Moreover, the operator h2 obtained by restricting h to H2 is self-adjoint, so an application of Theorem 11.41 to h2 yields the existence of an eigenvalue λ2 with |λ2 | = sup{|(h(x), x)| | x = 1, h(x) = λx and (x, x1 ) = 0}  |λ1 |, and an unit eigenvector x2 . The process continues an follows. Suppose that Gn−1 is the subspace generated by the first n − 1 orthonormal eigenvectors x1 , . . . , xn−1 . The restriction of h to G⊥ n−1 is self-adjoint and we have the eigenvalue λn λn = sup{(h(x), x) | x = 1 and (x, x1 ) = · · · = (x, xn−1 ) = 0} and a corresponding eigenvector xn with xn  = 1. Note that |λn | = |λn |xn  = λn xn  = h(xn ).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Hilbert Spaces

page 711

711

The process ends after n steps if H is n-dimensional, we have a set n orthogonal vectors that form a basis, hence equality (11.7) holds with y = 0H . If H is infinite-dimensional, the sequence (xn ) converges weakly to 0H , as we saw in Example 11.11. Since h is compact, by Theorem 11.37 we have limn→∞ h(xn ) = 0. Consequently, limn→∞ |λn | = limn→∞ h(xn ) = 0. Let T be the subspace of H that consists of all infinite linear combina tions of {x1 , x2 , . . .}. Every x ∈ H can be written as x = n cn xn + y,  where n cn xn ∈ T and y ∈ T ⊥ . The subspace U = T ⊥ is contained in every subspace Gn because every element of U is orthogonal to all xn . 1 If y ∈ U let y1 = y y. We have y1  = 1 and (y, h(y)) = y2 (y1 , h(y1 )). Since y1 ∈ U ⊆ Gn , we have (y, h(y))  λn y2. Since limn→∞ λn = 0 it follows that (y, h(y)) = 0 for all y ∈ U . This implies h(y) = 0, so y ∈ Null(h).  Theorem 11.44. (The Spectral Theorem for Compact Self-adjoint Operators) Let h be a compact self-adjoint operator on a Hilbert space H. There exists an orthonormal basis e1 , . . . , en , . . . in H that consists of eigenvectors of h that correspond to the eigenvalues λ1 , . . . , λn , . . ., respectively. If H is infinite-dimensional, then limn→∞ λn = 0. Furthermore, if   x = n cn en , then h(x) = n cn λn en . Proof. By Lemma 11.6, there exists an orthonormal set of eigenvectors {x1 , x2 , . . .} with the corresponding eigenvalues λ1 , λ2 , . . . such that for any  x ∈ H we have x = n cn xn + y for some scalars cn where y ∈ Null(h). Let y1 , y2 , . . . be an orthonormal basis for Null(h) that consists of eigenvectors that correspond to the eigenvalue 0 of h. The set {x1 , x2 , . . . , y1 , y2 , . . .} is an orthonormal basis for H. With the 0 eigenvalue included we still have limn→∞ λn = 0. Since h is continuous we have  h

∞ 

 cn xn

 =h

n=1

lim

m→∞

= lim

m→∞

=

∞  n=1

m 

m  n=1



 cn xn

= lim h m→∞

cn h(xn ) = lim

n=1

cn λn xn .

m→∞

m 

m 

 cn xn

n=1

cn λn xn

n=1



May 2, 2018 11:28

712

11.8

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 712

Mathematical Analysis for Machine Learning and Data Mining

Functions of Positive and Negative Type

Definition 11.11. Let S be a non-empty set. A function K : S × S −→ C is of positive type if for every n  1 we have: n  n  ai K(xi , xj )aj  0 i=1 j=1

for every ai ∈ C and xi ∈ S, where 1  i  n. If K : S × S −→ C is of positive type, then specializing the inequality of Definition 11.11 for n = 1 we have aK(x, x)a = K(x, x)|a|2  0 for every a ∈ C and x ∈ S. This implies K(x, x)  0 for x ∈ S. Note that K : S × S −→ C is of positive type if for every n  1 and for every x1 , . . . , xs the matrix An,K (x1 , . . . , xn ) = (K(xi , xj )) is positive. Example 11.13. The function K : R × R −→ R given by K(x, y) = cos(x − y) is of positive type because n  n  n n   ai K(xi , xj )aj = ai cos(xi − xj )aj i=1 j=1

i=1 j=1

=

n  n 

ai (cos xi cos xj + sin xi sin xj )aj

i=1 j=1

2  n 2  n         = ai cos xi  +  ai sin xi  .     i=1

i=1

for every ai ∈ C and xi ∈ S, where 1  i  n. Definition 11.12. Let S be a non-empty set. A function K : S × S −→ C is Hermitian if K(x, y) = K(y, x) for every x, y ∈ S. The notion of Hermitian function is useful in defining functions of negative type. Definition 11.13. Let S be a non-empty set. A complex-valued function K : S × S −→ C is of negative type if and only if the following conditions are satisfied: (i) K is Hermitian, and  (ii) for every ai ∈ C such that ni=1 ai = 0, and xi ∈ S, where 1  i  n we have; n  n  ai K(xi , xj )aj  0. i=1 j=1

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Hilbert Spaces

page 713

713

Theorem 11.45. Let S be a non-empty set and let S1 be a non-empty subset of S. If K : S × S −→ C is a complex-valued function of positive type (of negative type), then the restriction K1 = K S1 ×S1 is of positive type (of negative type). Proof.

This is an immediate consequence of Definitions 11.11 and 11.13. 

Theorem 11.46. Let H be a Hilbert space, S be a non-empty set and let f : S −→ H be a function. The function K : S × S −→ C defined by K(s, t) = (f (s), f (t)) is of positive type. Proof.

We can write n  n  n n   ai aj K(ti , tj ) = ai aj (f (ti ), f (tj )) i=1 j=1

i=1 j=1

 2 n     =  ai f (ai )  0,   i=1 which means that K is of positive type.



Example 11.14. Let S = Rk and let K : Rk × Rk −→ R be given by K(x, y) = (x, y)d , where d ∈ N and d  1. To show that K is of positive type we shall prove the existence of a funcsuch that K(x, y) = (φ(x), φ(y)). tion φ : Rk −→ Rm , where m = k+d−1 d Note that if ⎛ ⎞ ⎛ ⎞ x1 y1 ⎜ .. ⎟ ⎜ .. ⎟ x = ⎝ . ⎠ and y = ⎝ . ⎠ , xk

yk

then K(x, y) = (x1 y1 + · · · + xk yk )d . The expansion of (x1 y1 + · · · + xk yk )d results into a sum of m monomials of the form   d xn1 y n1 · · · xnk k yknk , n1 · · · nk 1 1 where n1 + · · · + nk = d and nj  0. The number m of these monomials equals the number of non-negative solutions of the equation

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 714

Mathematical Analysis for Machine Learning and Data Mining

714

n1 + · · · + nk = d. To evaluate this number we start from a sequence of d + k − 1 binary digits that contains d = n1 + · · · + nk ones and k − 1 zeroes: · · · 15 0 · · · 0 211 34 · · · 15 . 11 · · · 15 0 11 2 34 2 34 n1

n2

nk

A solution of n1 + · · · + nk = d and nj  0 is given by the lengths of the sequences of ones determined by the positions of the k − 1 zeroes. Note that the total length of the sequence is d + k − 1. Since the positioning of the zeroes is defined by a subset containing k − 1 elements of the set of d + k − 1 positions it follows that the number of solutions is     d+k−1 d+k−1 = . k−1 d  monomials: Thus, φ is a sum of d+k−1 d ;    d n1 nk φ(x) = . . . , x · · · xk , . . . , n1 · · · nk 1 k and K(x, y) = (φ(x), φ(y)) for x,y3 ∈ R . d+k−1 = 2 = 3 and we can write: For d = 2 we have d

K(x, y) = (x1 y1 + x2 y2 )2

= x21 y12 + 2x1 y1 x2 y2 + x22 y22 ⎞ ⎛ ⎞⎞ ⎛⎛ 2 2 √ x1 √ y1 = ⎝⎝ 2x1 x2 ⎠ , ⎝ 2y1 y2 ⎠⎠, x22 y22 so

⎛

⎞ 2 √ x1 φ(x) = ⎝ 2x1 x2 ⎠ x22

for x ∈ R2 . Example 11.15. Let K : Rk × Rk −→ C be the function defined by K(x, y) = ((x, y) + a)d , where d ∈ N, d  1 and a > 0. We prove that K is of positive type. We have K(x, y) = ((x, y) + a)d

√ √ = (x1 y1 + · · · + xn yn + a a)d    nk+1 nk+1 d = xn1 1 y1n1 · · · xnk k yknk a 2 a 2 n1 · · · nk nk+1 n1 +···+nk +nk+1 =d

= (φ(x), φ(y)),

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Hilbert Spaces

page 715

715

where, this time we have: ; 

  nk+1 d nk n1 φ(x) = . . . , x · · · xk a 2 , . . . . n1 · · · nk+1 1  For each monomial that is a component of φ(x) we have k+1 j=1 nj = d and  nj  0 for 1  j  k + 1. Note that the function φ(x) is a sum of d+k k monomials whose global degree is less or equal to d. For instance, in the case of k = 2 and d = 2 we have: K(x, y) = (x1 y1 + x2 y2 + a)2 = x21 y12 + x22 y22 + a2 + 2x1 y1 x2 y2 + 2ax1 y1 + 2ax2 y2

= (φ(x), φ(y)), where φ : R2 −→ R6 is defined as ⎛

⎞ x21 ⎜ x2 ⎟ ⎜√ 2 ⎟ ⎜ ⎟ ⎜ 2x x ⎟ φ(x) = ⎜ √ 1 2 ⎟ . ⎜ √2ax1 ⎟ ⎜ ⎟ ⎝ 2ax2 ⎠ a

Example 11.16. The radial basis function is the function K : Rk × Rk −→ x−y2

C defined by K(x, y) = e− 2σ2 . We shall prove that K is of positive type by showing that K(x, y) = (φ(x), φ(y)), where φ : Rk −→ 2 (R). Note that for this example φ ranges over an infinite-dimensional space. We have x−y2 K(x, y) = e− 2σ2 = e−

x2 +y2 −2(x,y) 2σ2 x2

= e− 2σ2 · e− Taking into account that e we can write e

(x,y) σ2

· e−

x2 2σ2

· e−

y2 2σ2

=

(x,y) σ2

y2 2σ2

·e

(x,y) σ2

.

∞  1 (x, y)j = , j! σ 2j j=0

∞  (x, y)j

e−

x2 2σ2

· e−

y2 2σ2

j!σ 2j ⎛ x2 ⎞j 2 ∞ − 2jσ2 − y  2jσ2 e e ⎝ ⎠ = (φ(x), φ(y)), = √ 1j √ 1j (x, y) σ j! σ j! j=0 j=0

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 716

Mathematical Analysis for Machine Learning and Data Mining

716

where

⎛ φ(x) = ⎝. . . ,

2

e

− x 2jσ2

σj



√ 1j j!

j n1 , . . . , nk

 12

⎞ xn1 1 · · · xnk k , . . .⎠ .

In this formula j varies in N and n1 + · · · + nk = j. In the next statement we prove that a complex-valued function of positive type is necessarily Hermitian. Note that for functions of negative type this property was included in their definition. Theorem 11.47. Let S be a set and let F : S×S −→ C be a complex-valued of positive type function. The following statements hold: (i) F (x, y) = F (y, x) for every x, y ∈ S, that is, F is Hermitian; (ii) F is a positive type function; (iii) |F (x, y)|2  F (x, x)F (y, y). Proof.

Take n = 2 in the definition of positive type functions. We have

a1 a1 F (x1 , x1 )+a1 a2 F (x1 , x2 )+a2 a1 F (x2 , x1 )+a2 a2 F (x2 , x2 )  0, (11.8) which amounts to |a1 |2 F (x1 , x1 ) + a1 a2 F (x1 , x2 ) + a2 a1 F (x2 , x1 ) + |a2 |2 F (x2 , x2 )  0. By taking a1 = a2 = 1 we obtain p = F (x1 , x1 ) + F (x1 , x2 ) + F (x2 , x1 ) + F (x2 , x2 )  0, where p is a positive real number. Similarly, by taking a1 = i and a2 = 1 we have q = −F (x1 , x1 ) + iF (x1 , x2 ) − iF (x2 , x1 ) + F (x2 , x2 )  0, where q is a positive real number. Thus, we have F (x1 , x2 ) + F (x2 , x1 ) = p − F (x1 , x1 ) − F (x2 , x2 ), iF (x1 , x2 ) − iF (x2 , x1 ) = q + F (x1 , x1 ) − F (x2 , x2 ). These equalities imply 2F (x1 , x2 ) = P − iQ 2F (x2 , x1 ) = P + iQ, where P = p − F (x1 , x1 ) − F (x2 , x2 ) and Q = q + F (x1 , x1 ) − F (x2 , x2 ), which shows the first statement holds.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Hilbert Spaces

page 717

717

The second part of the theorem follows by applying the conjugation in the equality of Definition 11.11. For the final part, note that if F (x1 , x2 ) = 0 the desired inequality holds immediately. Therefore, assume that F (x1 , x2 ) = 0 and apply equality (11.8) to a1 = a ∈ R and to a2 = F (x1 , x2 ). We have a2 F (x1 , x1 ) + aF (x1 , x2 )F (x1 , x2 ) +F (x1 , x2 )aF (x2 , x1 ) + F (x1 , x2 )F (x1 , x2 )F (x2 , x2 )  0, which amounts to a2 F (x1 , x1 ) + 2a|F (x1 , x2 )| + |F (x1 , x2 )|2 F (x2 , x2 )  0. If F (x1 , x1 ) this trinomial in a must be non-negative for every a, which implies |F (x1 , x2 )|4 − |F (x1 , x2 )|2 F (x1 , x1 )F (x2 , x2 )  0. Since F (x1 , x2 ) = 0, the desired inequality follows.



Corollary 11.9. Let S be a set and let F : S × S −→ C be a function. Then F is of positive type if and only if −F is of negative type. Proof. If F is of positive type, F is Hermitian by Theorem 11.47, which implies that −F is Hermitian. The second condition of Definition 11.13 for −F follows immediately, so −F is indeed of negative type. The reverse implication is also immediate.  Theorem 11.48. A real-valued function G : S × S −→ R is a positive type function if it is symmetric and n  n  ai aj G(xi , xj )  0 (11.9) i=1 i=1

for a1 , . . . , an ∈ R and x1 , . . . , xn ∈ S. Proof. The necessity of the conditions of the theorem is clear. To prove that they are sufficient let G a function of positive type such that n  n  ai aj G(xi , xj )  0 (11.10) i=1 j=1

for every ai ∈ C and xi ∈ S, where 1  i  n. If ai = ci + idi we have n  n  (ci + idi )(cj − idj )G(xi , xj )  0. i=1 j=1

May 2, 2018 11:28

718

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 718

Mathematical Analysis for Machine Learning and Data Mining

Thus, we have: n  n 

ai aj G(xi , Gj ) i=1 j=1 n  n 

ci cj G(xi , xj ) +

=

n  n 

i=1 j=1

di dj G(xi , xj )

i=1 j=1

⎛ ⎞ n n n  n    +i ⎝ di cj G(xi , xj ) − ci dj G(xi , xj )⎠ . i=1 j=1

i=1 j=1

Observe that n  n 

di cj G(xi , xj ) −

i=1 j=1

n  n 

ci dj G(xi , xj ) = 0

i=1 j=1

n n because G is symmetric. Thus, i=1 j=1 ai aj G(xi , Gj ) is a non-negative real number, so G is of positive type.  Theorem 11.49. Let S be a non-empty set. If Ki : S ×S −→ C for i = 1, 2 are functions of positive type, then their pointwise product K1 K2 defined by (K1 K2 )(x, y) = K1 (x, y)K2 (x, y) is of positive type. Proof.

Since Ki is a function of positive type, the matrix An,Ki (x1 , . . . , xn ) = (Ki (xj , xh ))

is positive, where i = 1, 2. Therefore, these matrices can be factored as An,K1 (x1 , . . . , xn ) = P H P and An,K2 (x1 , . . . , xn ) = RH R for i = 1, 2. Therefore, we have: n  n 

ai K1 (xi , xj )K2 (xi , xj )aj

i=1 j=1

=

n  n 

 ai K(xi , xj ) ·

i=1 j=1

=

n 

 n 

m=1

i=1

 ai rmi

n  m=1

 rmi rmj ⎛

K(xi , xj ) ⎝

n 

aj ⎞

rjm aj ⎠  0,

j=1

which shows that (K1 K2 )(x, y) is a function of positive type.



May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Hilbert Spaces

b3234-main

page 719

719

Theorem 11.50. Let S, T be two non-empty sets and let K1 : S × S −→ C and K2 : T × T −→ C be two positive type functions. The function K1 ⊗ K2 : (S × T ) × (S × T ) −→ C defined by (K1 ⊗ K2 )((s, t), (s , t )) = K1 (s, s )K2 (t, t ) for (s, t), (s , t ) ∈ S × T is of positive type. Proof. by

˜ and K ˆ defined on (S × T ) × (S × T ) Note that the functions K ˜ K((s, t), (s , t )) = K1 (s, s ), ˆ K((s, t), (s , t )) = K2 (t, t )

for (s, t), (s , t ) ∈ S × T are of positive type. The statement follows im˜K ˆ and applying Theorem 11.49. mediately by observing that K1 ⊗ K2 = K  The function K1 ⊗ K2 introduced in Theorem 11.50 is known as the tensor product of K1 and K2 . For a non-empty set S the set of functions CS×S is a linear space if the addition is defined by (F + G)(s1 , s2 ) = F (s1 , s2 ) + G(s1 , s2 ) and the multiplication with scalars is (aF )(s1 , s2 ) = aF (s1 , s2 ) for s1 , s2 ∈ S. Theorem 11.51. Let S be a non-empty set. The set of functions of positive (negative) type is closed with respect to multiplication with non-negative scalars and with respect to addition; in other words, these sets are convex cones. Furthermore, these sets are closed with respect to pointwise convergence. Proof. We discuss only the case of functions of positive type. It follows immediately from Definition 11.11 that the set of functions of positive type in CS×S is closed with respect to multiplication with non-negative scalars and with respect to addition, so this set is indeed a convex cone. Suppose that (Fp ) is a sequence of functions of positive type in CS×S that converges pointwise to F that is, limn→∞ Fp (s, t) = F (s, t) for every

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 720

Mathematical Analysis for Machine Learning and Data Mining

720

(s, t) ∈ S × S. Then, n  n 

ai F (xi , xj )aj i=1 j=1 n  n  =

ai lim Fp (xi , xj )aj p→∞ i=1 j=1 n  n 

ai Fp (xi , xj )aj  0,

= lim

p→∞

i=1 j=1



which shows that F is a function of positive type.

Theorem 11.52. Let S be a non-empty set and let F : S × S −→ C be a Hermitian function. For x0 ∈ S define the functions Gx0 : S × S −→ C and gx0 : S × S −→ C as Gx0 (x, y) = F (x, x0 ) + F (y, x0 ) − F (x, y) − F (x0 , x0 ), gx0 (x, y) = F (x, x0 ) + F (y, x0 ) − F (x, y), for (x, y) ∈ S × S. We have: (i) Gx0 is of positive type if and only if F is of negative type; (ii) if F (x0 , x0 )  0 then gx0 is of positive type if and only if F is of negative type.  Proof. Let ai ∈ C where 1  i  n such that ni=1 ai = 0, and let xi ∈ S, where 1  i  n. We have n  n  ai Gx0 (xi , xj )aj i=1 j=1 n  n 

=

ai (F (xi , x0 ) + F (xj , x0 ) − F (xi , xj ) − F (x0 , x0 ))aj .

i=1 j=1

Note that n  n 

 ai F (xi , x0 )aj =

i=1 j=1 n  n  i=1 j=1 n  n  i=1 j=1

n 

⎞ ⎛ n  ai F (xi , x0 ) ⎝ aj ⎠ = 0,

i=1

 ai F (xj , x0 )aj =

j=1

⎞ ⎛ n n   ai ⎝ F (xj , x0 )aj ⎠ = 0, i=1

ai F (x0 , x0 )aj = F (x0 , y0 )

j=1

 n  i=1

⎞ ⎛ n  ai ⎝ aj ⎠ = 0. j=1

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Hilbert Spaces

page 721

721

Therefore, n  n 

ai Gx0 (xi , xj )aj = −

i=1 j=1

n  n 

ai F (xi , xj )aj .

i=1 j=1

The last equality shows that if Gx0 is of positive type then F is of negative type. Let now F be of negative type and ai ∈ C and xi ∈ S, where 1  i  n. n n Define a0 = − i=1 ai . Since F is of negative type and i=0 ci = 0, we can write: n  n 

ai F (xi , xj )aj i=0 j=0 n  n  =

ai F (xi , xj )aj +

i=1 j=1 n 

+ =

n 

a0 F (x0 , xj )aj

j=1

ai F (xi , xj )a0 + a0 F (x0 , x0 )a0

i=1 n  n 

ai (F (xi , xj ) − F (xi , x0 ) − F (x0 , xj ) + F (x0 , x0 ))aj

i=1 j=1

n (by substituting − i=1 ai for a0 ) n  n  Gx0 (xi , xj )  0. =− i=1 j=1

Consequently, Gx0 is of positive type. Since gx0 = Gx0 + F (x0 , x0 ), it  follows that gx0 is also of positive type. A special case of kernels on Rn is obtained by defining K : Rn ×Rn −→ R as Kf (x, y) = f (x − y), where f : Rn −→ C is a continuous function on Rn . K is translation invariant and is designated as a stationary kernel. If Kf is of positive type we say that f is of positive type. In Example 11.13 we have shown that cos is a function of positive type. Theorem 11.53. If f : Rn −→ R is of positive type the following properties hold: (i) f (0)  0; (ii) f (−x) = f (x); (iii) |f (x)|  f (0) for all x ∈ Rn ; (iv) f (0) = 0 if and only if f is the constant function 0.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 722

Mathematical Analysis for Machine Learning and Data Mining

722

Proof.

Since f is of positive type, we have n  n  ai f (xi − xj )aj  0

(11.11)

i=1 j=1

for every ai ∈ C and xi ∈ S, where 1  i  n. Part (i): Since f (0) = Kf (0, 0)  0, the desired inequality is immediate. Part (ii): Note that f (−x) = Kf (0, x) = Kf (x, 0) = f (x). Part (iii): Take n = 2, a1 = |f (x)|, a2 = −f (x), x1 = 0 and x2 = x in inequality (11.11). We have: a1 f (x1 − x1 )a1 + a1 f (x1 − x2 )a2 + a2 f (x2 − x1 )a1 + a2 f (x2 − x2 )a2 = |f (x)|2 f (0) − |f (x)|f (x)f (−x) − f (x)f (x)|f (x)| + f (x)f (0)f (x) = |f (x)|2 f (0) − |f (x)|f (x)f (x) − f (x)f (x)|f (x)| + f (x)f (0)f (x) (by part (ii)) = 2|f (x)|2 f (0) − 2|f (x)|3  0, which amounts to |f (x)|  f (0) for all x ∈ Rn . Part (iv): If f (0) = 0, from part (iii) it follows that f (x) = 0. The converse implication is immediate.  11.9

Reproducing Kernel Hilbert Spaces

Let S be a non-empty set and let K : S × S −→ C be a function. If we fix the second argument of K, the resulting function K(·, t) is defined on S and ranges on C. If H is a Hilbert space whose elements are complex-valued functions defined on S and K(·, t) ∈ H, it makes sense to consider inner products of the form (f, K(·, t)). Definition 11.14. Let H be a Hilbert space of complex-valued functions defined on a set S. A reproducing kernel is a function K : S ×S −→ C such that K(·, t) ∈ H for every t ∈ S and (f, K(·, t)) = f (t) for every t ∈ S. A Hilbert space of complex-valued functions defined on a set S that has a reproducing kernel is called a reproducing kernel Hilbert space (RKHS). Theorem 11.54. A reproducing kernel K on a Hilbert space of functions of the form f : S"−→ C is a function of positive type. Furthermore, we have K(·, t) = K(t, t) for t ∈ S.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Hilbert Spaces

Proof. Let f = K(·, s). (f, K(·, t)) = f (t), that is

page 723

723

Since K is a reproducing kernel we have

(K(·, s), K(·, t)) = K(t, s)

(11.12)

for every s, t ∈ S. By Theorem 11.46, K is a function of positive type. Choosing s = t in equality (11.12) we obtain (K(·, t), K(·, t)) = K(t, t), which amounts to K(·, t)2 = (K(·, t), K(·, t)) = K(t, t).



Theorem 11.55. If a Hilbert space of functions of the form f : S −→ C has a reproducing kernel, then this kernel is unique. Proof. Suppose that K1 and K2 are kernels for H. We have (f, K1 (·, t)) = (f, K2 (·, t) = f (t) for every t ∈ S. Therefore, (f, K1 (·, t) − K2 (·, t)) = 0. Taking f (t) = K1 (·, t) − K2 (·, t) implies K1 (·, t) − K2 (·, t)2 = 0, hence K1 (·, t) = K2 (·, t) for every t ∈ S, that is,  K1 = K2 . Example 11.17. We have shown in Example 2.3 that for a finite set S and a linear space L, LS is a finite-dimensional space of functions. Namely, if S = {x1 , . . . , xn }, a basis of this space consists of functions of the form ei ∈ LS defined as  1 if x = xi , ei (x) = 0 otherwise, for 1  i  n. Any function f ∈ LS can be written as f = f (x1 )e1 + · · · + f (xn )en . If instead of a general linear space, we consider the set CS , the set CS is a finite-dimensional complex Hilbert space, where n  f (xi )g(xi ). (f, g) = i=1

Let K : S × S −→ C be the function defined by K(xi , xj ) = δij , where δij is Kronecker’s symbol given by  1 if i = j, δij = 0 otherwise for 1  i, j  n.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 724

Mathematical Analysis for Machine Learning and Data Mining

724

For xj ∈ S we have (f, K(·, xj )) =

n 

f (xi )K(xi , xj ) = f (xj ),

i=1

for every xj ∈ S which shows that K is indeed a self-reproducing kernel on CS . Definition 11.15. For a Hilbert space of complex-valued functions defined on a set S, the evaluation functional at x is the functional ex : H −→ C defined by ex (f ) = f (x) for every f ∈ H. Theorem 11.56. A Hilbert space of complex-valued functions defined on a set S has a reproducing kernel if and only if every evaluation functional et for t ∈ S is continuous. Proof.

Suppose that H has a reproducing kernel K. We have et (f ) = f (t) = (f, K(·, t)).

By Cauchy-Schwarz inequality we have

" |et (f )| = |(f, K(·, t))|  f K(·, t) = f  K(t, t) " Since et (K(·, t)) = K(t, t), we actually have et  = K(t, t), so et is bounded and, therefore, it is continuous. Conversely, if et is continuous, by Riesz’ Representation Theorem (Theorem 11.33), there exists a function ht such that et (f ) = (f, ht ) = f (t).  Thus, we obtain the reproducing kernel K given by K(s, t) = ht (s). Corollary 11.10. In a RKHS H space of complex-valued functions defined on a set S, a sequence (fn ) of functions that converges in norm is also pointwise convergent. Proof.

If (fn ) converges in norm to f then |fn (t) − f (t)| = |et (fn ) − et (f )| = |et (fn − f )|  et  · fn − f .

Thus, if limn→∞ fn − f  = 0, it follows that limn→∞ |fn (t) − f (t)| = 0 for t ∈ S.  Example 11.18. Let S be a finite set and let H be the finite-dimensional complex linear space that consists of complex-valued functions defined on S. Suppose that B = {e1 , . . . , en } is a basis in H.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Hilbert Spaces

page 725

725

The linear space H is equipped with the inner product defined as follows. n n If f, g ∈ H we can write uniquely f = j=1 aj ej and g = j=1 bj ej . Then, an inner product (f, g) can be defined as (f, g) =

n 

ai bi (ei , ej ).

i=1

 a1  If GB is the Gram matrix of B we have (f, g) = a GB b, where a =  b1  and b =

. . . bn

. . . an

. It is clear that H is a Hilbert space.

Assume now that B is an orthonormal basis. For x, y ∈ S define the kernel K as the function K : S 2 −→ C given by K(x, y) =

n 

ei (x)ei (y)

i=1

for x, y ∈ S. We claim that (f, K(·, y)) = f . Indeed, we have K(·, y) =

n 

ei (·)ei (y),

i=1

so the coefficients of K(·, y) relative to the basis B are ei (y). Therefore, (f, K(·, y)) =

n  i=1

ai ei (y) =

n 

ai ei (y) = f (y).

i=1

Example 11.19. Let S = N and let H be the 2 (C). Consider the function K : S 2 −→ C defined by  1 if i = j, K(i, j) = 0 otherwise for i, j ∈ N. Then, K(·, t) is the sequence et = (0, 0, . . . , 0, 1, 0, . . .), where the unique 1 that occurs in this sequence is located on position t. Then, for f ∈ 2 (C) we have (f, K(·, t)) = (f, et ) = f (t), which shows that K is a reproducing kernel.

May 2, 2018 11:28

726

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 726

Mathematical Analysis for Machine Learning and Data Mining

Theorem 11.57. Let H0 be a Hilbert space of functions of the form f : S −→ C having the inner product (·, ·)H0 . Suppose that the evaluation functionals ex (for x ∈ S) are continuous on H0 and any Cauchy sequence (fn ) convergent pointwise to 0H0 also converges in norm to 0H0 . If H is the set of complex functions defined on S that are pointwise limits of functions from H0 , an inner product can be defined on H such that the following statements hold: (i) If (fn ) and (gn ) are two sequences in H0 that converge pointwise to f, g ∈ H, respectively, then the sequence ((fn , gn )H0 ) is convergent and its limit depends only on f and g. (ii) If f ∈ H and (fn ) is a sequence in H0 that converges pointwise to f such that limn→∞ fn H0 = 0, then f = 0H . (iii) Let f ∈ H and (fn ) be a Cauchy sequence in H0 converging pointwise to f . Then (fn ) converges to f in the norm sense in H. (iv) H0 is dense in H. (v) The evaluation functionals are continuous on H. Proof. Part (i): Since (fn ) and (gn ) are Cauchy sequences in H0 they are bounded. This allows us to write |(fn , gn )H0 − (fm , gm )H0 | = |(fn − fm , gn )H0 + (fm , gn − gm )H0 |  fn − fm H0 · gn H0 + fm H0 · gn − gm H0 (by Cauchy-Schwarz inequality). Thus, ((fn , fm )H0 ) is a Cauchy sequence of complex numbers and, therefore, gn ) are two other Cauchy sequences converging it is convergent. If (f˜n ) and (˜ pointwise to f and g, respectively, then |(fn , gn )H − (f˜n , g˜n )H | 0

0

 fn − f˜n H0 · gn H0 + f˜n  · gn − g˜n H0 . Since (fn − f˜n ) and (gn − g˜n ) are Cauchy sequences in H0 converging pointwise to 0, these sequences converge to 0 in norm. Therefore, limn→∞ (fn , gn )H0 = limn→∞ (f˜n , g˜n )H0 . Part (ii): The pointwise convergence of fn to f means that for every x ∈ S we have f (x) = limn→∞ fn (x) = limn→x ex (fn ) = 0, because of the continuity of the evaluation functionals on H0 . Therefore, f = 0H . Part (iii): Let f, g be two functions in H that are pointwise limits of the sequences (fn ) and (gn ) in H0 . The inner product on H is (f, g)H = lim (fn , gn )H0 , n→∞

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Hilbert Spaces

page 727

727

as it can be easily verified. The topology defined on H0 by the inner product coincides with the trace topology on H0 induced by the topology of H. For > 0 let n ∈ N such that m, n  n implies fn − fm H0 < . For a fixed n, n > n . The sequence (fm − fn )m∈N is a Cauchy sequence in H0 that converges pointwise to f − fn . Therefore, f − fn H = lim fm − fn   . m→∞

Thus, (fn ) converges to f in the norm sense. Part (iv): By hypothesis, for every f ∈ H there exists a Cauchy sequence (fn ) in H0 that is convergent to f . By part (iii), fn converges to f in a norm sense, so H0 is dense in H. Part (v): Since the evaluation functionals are linear it is sufficient to show that they are continuous at 0H . We start from the fact that each evaluation functional ex is continuous on H0 . For > 0 let η such that f ∈ H0 and f H0 < η implies |f (x)| < 2 . By part (iii), for and function f ∈ H with f H < η2 there exists g ∈ H0 such that |g(x) − f (x)| < 2 and g − f H < η2 . This implies gH0 = gH  g − f H + f H < η, hence |g(x)|
0 and x, y ∈ Rn . Suppose that x∗ is a minimizer of f and consider the sequence (xk ) defined by xk+1 = xk − η(∇f )(xk )

(14.3)

for k ∈ N, where x0 is given. We have limk→∞ xk = x∗ , the sequence of numbers (xk −x∗ ) is strictly decreasing and converges to 0 and f (xk ) − f (x∗ )  Proof.

2η

k−1 j=0

5 xj − x∗ −2

.

By Supplement 7 we have: f (xk+1 ) − f (xk )  ((∇f )(xk ), xk+1 − xk ) +

1 xk+1 − xk 2 2η

η = −η(∇f )(xk )2 + (∇f )(xk )2 2 η 1 = − (∇f )(xk )2 = − xk+1 − xk 2 . 2 2η Since f is convex, by Theorem 12.13, we have f (x∗ )  f (xk ) + (∇f )(xk ) (x∗ − xk ) for k ∈ N. By defining wk = f (xk ) − f (x∗ ) for k ∈ N we have 1 wk  (∇f )(xk ) (xk − x∗ )  xk − xk+1 xk − x∗ . η Therefore, 1 xk − xk+1 2 2η  2 ηwk 1 ηwk2  = 2η xk − x∗  2xk − x∗ 2 2 ηwk+1  . xk − x∗ 2

f (xk ) − f (xk+1 ) = wk − wk+1 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Iterative Algorithms

page 881

881

Note that the inequality shown above, 2 ηwk+1 xk − x∗ 2

wk − wk+1  can be written as wk−1



−1 wk+1

 1+

ηwk+1 2xk − x∗ 2

−1 .

(14.4)

1 xk − x∗ 2 . ThereBy Supplement 7 we have wk = f (xk ) − f (x∗ )  2η ηwk+1 1 2 fore, wk+1  wk  2η xk − x∗  . This implies 2 xk −x∗ 2  14 . It is immediate that t ∈ [0, 1/4] implies 4t 1 1− . 1+t 5 Thus, 2η −1 wk−1  wk+1 − . 5xk − x∗ 2 Taking into account the first k of the above inequalities we have 2η , w0−1  wk−1 − k−1 5 j=0 xj − x∗ −2

which produces the desired inequality.



Corollary 14.3. For the approximating sequence (xk ) of the minimizer x∗ defined in Theorem 14.9 we have f (xk ) − f (x∗ ) = o(1/k). Proof.

Observe that  2   xk+1 − x∗ 2 = xk − x∗ − η(∇f )(xk ) = xk − x∗ 2 + η 2 (∇f )(xk )2 −2η(xk − x∗ , (∇f )(xk ) − (∇f )(x∗ ))  xk − x∗ 2 + η 2 (∇f )(xk )2 − 2η 2 (∇f )(xk )2 < xk − x∗ 2 .

This implies that (xk − x∗ ) is a strictly decreasing sequence. Therefore, the sequence (xk ) is bounded and, consequently, it contains a subsequence 1 ˜ . Since f (xk+1 ) − f (xk )  − 2η xk − (xkn ) that converges to a point x 2 ˜ is a minimizer of f . x) = 0, so x xk+1  , it follows that (∇f )(˜ ˜ ) is decreasing, (xk ) converges to x ˜. Since the sequence (xk − x ˜ such that Define ak = xk − x∗ −2 for k ∈ N, where x∗ is the point x limk→∞ xk − x∗  = 0. The sequence (ak ) is an increasing sequence and limk→∞ ak = ∞.

May 2, 2018 11:28

882

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 882

Mathematical Analysis for Machine Learning and Data Mining k−1

aj

Define a ˜k = j=0 for k ∈ N. This definition implies k(˜ ak − a ˜k+1 ) = k a ˜k+1 − ak . Since (ak ) is an increasing sequence, it follows that (k + 1)˜ ak+1 

k 

aj  (k + 1)ak ,

j=0

hence a ˜k+1  ak . This implies a ˜k  a ˜k+1 for k ∈ N, hence (ak ) is a increasing sequence. Let M be a positive number. There exists N ∈ N such that k  N implies ak  M . We have a ˜2N =

2N −1 2N −1 1  1  M . aj  aj  2N j=0 2N 2 j=N

Thus, limk→∞ a ˜k = ∞. By Theorem 14.9 we have ⎛ k(f (xk ) − f (x∗ )) 

2η

k−1 j=0

5k xj − x∗

−2



5 ⎝1 2η k

k−1 

⎞−1 aj ⎠

.

j=0

Therefore, f (xk ) − f (x∗ ) = o(1/k).



Example 14.1. Let f : R −→ R be the function given by f (x) = x3 − 3x2 + 6. It is immediate to establish analytically that the function has a minimum in x∗ = 2 and that f (2) = 2. We have (∇f )(y) − (∇f )(x) = |f  (y) − f  (x)| = |3y 2 − 6y − 3x2 + 6x| = 3|y − x| · |x + y − 2|  9 · |y − x|, when |x + y − 2|  3. The minimal point (2, 2) is included in the stripe defined by |x + y − 2|  3 and we can take η = 1/9.

14.7

Stochastic Gradient Descent

The stochastic gradient descent (SGD) differs from the gradient descent in that the descent direction is replaced by a random vector whose expected value is a descent direction; another approach, applicable to

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Iterative Algorithms

page 883

883

a convex function that is not necessarily differentiable is to construct a sequence of approximations of a minimizer of a function using randomly chosen subgradients of the objective function at the current approximation. Algorithm 14.7.1: Stochastic Gradient Descent Algorithm Input : a convex function f : Rn −→ R, a scalar η, the number of iterations m, m  1 ˜ of the minimizer x∗ of f Output: an approximation x 1 begin 2 initialize x1 = 0n ; 3 for i ← 1 to m do 4 choose vi at random such that E(vi |xi ) ∈ subd(f )(xi ); 5 set xi+1 = xi − ηvi ; 6 end m 1 ˜=m 7 return x i=1 xi ; 8 end In the recurrence (14.5) xk+1 = xk − ηvk , vk is a random vector whose conditional expectation E(vk |xk ) is a subgradient of f at xk . The output of the algorithm is the vector m 1  ˜= x xk , m k=1 when the algorithm is run for m steps. Note that xi is a random variable whose values are determined by the values of the random vectors v1 , . . . , vi−1 . Theorem 14.10. Let f : Rn −→ R be a convex function and let x∗ a point where f has a minimum. Assume that: (i) the algorithm constructs the sequence x1 , . . . , xm ; (ii) a minimum of f is sought in the sphere B[0n , b]; (iii) we have P (xk ∈ B(0n , r)) = 1 for 1  k  m. If η = r√bm , then E(f (˜ x)) − f (x∗ )  √brm . Proof.

We have

f (˜ x) − f (x∗ ) = f



1  xk m m

k=1



 − f (x∗ ) 

 m 1  f (xk ) − f (x∗ ) m k=1

(by Jensen’s Inequality, because f is convex) m 1  = (f (xk ) − f (x∗ )). m k=1

May 2, 2018 11:28

884

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 884

Mathematical Analysis for Machine Learning and Data Mining

By taking the expectation on Seqm (Rn ) we have   m 1  Ev1:m (f (˜ x) − f (x∗ ))  Ev1:m (f (xk ) − f (x∗ ) . m k=1

Inequality (14.8) of Supplement 8 is applicable to every sequence of vectors (v1 , . . . , vm ). Therefore, by inequality (14.7) we have:  m  br Ev1:m (xi − x∗ , vi )  √ . m i=1 Next, we prove that     m m 1  1  Ev1:m (f (xi ) − f (x∗ )  Ev1:m (xi − x∗ , vi ) . m i=1 m i=1

(14.6)

The linearity of the expectation implies   m m 1  1  Ev1:m (xi − x∗ , vi ) = Ev (xi − x∗ , vi ). m i=1 m i=1 1:m By the Tower Property of Conditional Expectations (Theorem 8.109) we have E(g(X)) = E(E(g(X)|Y )) for the random variables X and Y and the measurable function g. Choosing X = v1:t , Y = v1:t−1 , and g(v1:t ) = (xt − x∗ , vi )1:t we have: Ev1:t ((xt − x∗ , vi )1:t ) = Ev1:t−1 (Ev1:t ((xt − x∗ , vi )1:t |v1:t−1 )). Since the value of xt is determined by v1:t−1 , we have Ev1:t−1 (Ev1:t ((xt − x∗ , vi )1:t |v1:t−1 )) = Ev1:t−1 (xt − x∗ , Evt (vt |v1:t−1 )). By the definition of the sequence (vt ), Evt (vt |xt ) ∈ subd(f )(xt ). Therefore, Ev1:t (xt −x∗ , E(vt |v1:t−1 ))  Ev1:t−1 (f (xt )−f (x∗ )) = Ev1:t (f (xt )−f (x∗ )), which yields inequality (14.6) after summing over t, dividing by m and using the linearity of expectation.  Exercises and Supplements (1) Let f : R −→ R be defined as f (x) = tanh(x). It is clear that f has the unique root x0 = 0. Prove that (a) the sequence constructed by Newton’s method (equality (14.1)) is xk+1 = xk − 12 sinh(2xk ) for k ∈ N; (b) the signs of the members of the sequence (xk ) alternate;

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Iterative Algorithms

page 885

885

(c) for the sequence (|xk |) we have |xk+1 | = h(|xk |), where h(x) = 1 sinh(2x) − x; 2 (d) the equation h(x) = x has the non-negative solutions x0 = 0 and x ˜0 > 1; the sequence of approximants (xk ) where x0 = 0 converges ˜0 diverges, to the solution 0; the sequence (˜ xk ) that starts with x and thus, the method fails. (2) Let f : R −→ R be defined by f (x) = arctan x. Clearly, the equation f (x) = 0 has the solution x = 0. If (xk ) is the sequence constructed by Newton’s method, xk+1 = xk − (1 + x2k ) arctan xk starting with the 2|x0 | initial value x0 , prove that if arctan |x0 |  1+x 2 , then the sequence (|xk |) 0 diverges to ∞. (3) Prove that if A ∈ Rn×n is a symmetric and positive matrix, then ℘ : Rn × Rn −→ R given by ℘(x, y) = (Ax, y) is an inner product on Rn .  (4) Let A ∈ Rn×n . The matrix is diagonally dominant if |aii | > {|aij | | 1  j  n, j = i} for every i, 1  i  n. Prove that the matrix A is invertible. Solution: We shall prove that if A is diagonally dominant then 0 is not an eigenvalue of A. Suppose that 0 is an eigenvalue of A, that is, Ax = 0n . Let p be such that |xp | = max{|xi | | 1  i  n}, hence  n j=1,j =p apj xj = −app xp . This implies |app | |xp | 

n 



|apj | |xj |  |xp |

j=1,j =p

|apj |,

j=1,j =p

which contradicts the fact that A is diagonally dominant. (5) This supplement introduces Jacobi’s method for solving the equation Ax = b, where A ∈ Rn×n and b ∈ Rn . Let D be the diagonal matrix ⎛ a11 ⎜ 0 ⎜ D=⎜ . ⎝ .. 0

0 a22 .. . ···

⎞ ··· 0 ··· 0 ⎟ ⎟ . ⎟, · · · .. ⎠ 0 ann

and let E = A − D. Suppose that aii = 0 for 1  i  n. Consider the iterative construction of the sequence (xn ) given by xi+1 = xi + D−1 (b − Axi ) for i  0, where x0 is an arbitrary initial vector. Prove that if A is a diagonally dominant matrix then limk→∞ xk = x∗ , where x∗ is the solution of Ax = b.

May 2, 2018 11:28

886

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 886

Mathematical Analysis for Machine Learning and Data Mining

Solution: Let x∗ be the solution of Ax = b. We have xi+1 − x∗ = (xi − x∗ )(In − D−1 A) for i  0. This implies xk − x∗ = (x0 − x∗ )(In − D−1 A)k . Next we use the norm x∞ = max{|x i | | 1  i  n} and the corresponding matrix norm |||P |||∞ = max1≤i≤n n j=1 |pij |. Since xk − x∗ ∞  x0 − x∗ ∞ |||In − D−1 A|||k∞ , it suffices to show that In − D−1 A∞ < 1 to obtain the convergence of the Jacobi method. This is indeed the case because   |aji |

 i  n, i =  j ξ1 produces a strictly decreasing sequence (xk ) that converges to ξ1 . Solution: Let p(x) = a0 xn + · · · + an−1 x + an . Without loss of generality we may assume that p(x0 ) > 0. Therefore, if x > ξ1 we have p(x) > 0. This implies a0 > 0. By Rolle’s Theorem the derivative p (x) has n − 1 real roots α1 , . . . , αn−1 such that ξ1  α1  ξ2  α2  · · ·  αn−1  ξn . Since p is of degree n − 1, α1 , . . . , αn−1 are all its roots and p (x) > 0 for x > α1 because a0 > 0. Applying Rolle’s theorem again we obtain p (x) > 0 and p (x)  0

(14.9)

for x  α1 . Thus, both p and p are convex functions for x  α1 . Note that xk > ξ1 implies xk+1 = xk −

p(xk ) < xk p (xk )

because p (xk ) > 0 and p(xk ) > 0. Since xk > ξ1  α1 , by Taylor’s theorem we have 0 = p(ξ1 ) = p(xk ) + (ξ1 − xk )p (xk ) + > p(xk ) + (ξ1 − xk )p (xk ).

(ξ1 − xk )2  p (δ), ξ1 < δ < xk , 2

By the definition of xk+1 we have p(xk ) = p (xk )(xk − xk+1 ). Thus, 0 > p (xk )(xk − xk+1 + ξ1 − xk ) = p (xk )(ξ1 − xk+1 ), hence xk+1 > ξ1 follows because p (xk ) > 0. (13) exer:jan0518e Let p = a0 xn + · · · + an−1 x + an be a polynomial with real coefficients, where a0 = 0. If p has a complex root a + ib, then p has also the root a − ib and p is divisible by (x − a)2 + b2 = x2 − rx − q, where r = 2a and q = −(a2 + b2 ). We can write p(x) = p1 (x)(x2 − rx − s) + A(r, s)x + B(r, s), where p1 (x) = b0 xn−2 +· · ·+bn−2 , and seek r, q such that A(r, s) = 0 and B(r, s) = 0 in order to determine x2 − rx − s and the pair of conjugate complex roots of this trinomial. Note that q is of degree n − 2. This process known as Bairstow method entails applying Newton’s method for solving the system A(r, s) = 0, B(r, s) = 0. Once r and

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Iterative Algorithms

page 891

891

s are determined, the pair of conjugate roots mentioned above can be determined immediately. Newton’s method applied to A(r, s) = 0,

B(r, s) = 0 leads to the ri computation of the sequence of vectors defined by si

ri+1 si+1

⎛ ∂A ri ⎜ ∂r = −⎝ si ∂B ∂r

∂A ⎞−1 A ∂s ⎟ . ⎠ B ∂B ∂s

By dividing p1 by x2 − rx − s we have p1 (x) = p2 (x)(x2 − rx − s) + A1 x + B1 . Prove that: (a) the partial derivatives that enter Newton’s scheme are given by: ∂B ∂A = A1 , = B1 , ∂s ∂s ∂B ∂A = rA1 + B1 , = sA1 ; ∂r ∂r (b) the coefficients of p1 are: b0 = a 0 , b1 = b0 r + a 1 , bi = bi−2 s + bi−1 r + ai for 2  i  n − 2, A = bn−3 s + bn−2 r + an−1 , B = bn−2 s + an . (14) Regula falsi is an interpolation method that can be used to solve equations of the form f (x) = 0 for a continuous function f . It consists of an iterative process that constructs a sequence of pairs (ai , xi ) such that f (ai )f (xi ) < 0. Thus, each interval [xi , ai ] contains a zero of f . (ai ) i )−xi f (ai ) Let p(x) = f (xi ) + (x − xi ) f (xxii)−f , and let μi = ai ff (x be −ai (xi )−f (ai ) the zero of p(x). Note that μi is well-defined because f (xi )f (ai ) < 0 implies f (xi ) = f (ai ). IF f (μi ) = 0 the algorithm stops; otherwise, define xi+1 = μi and

 ai+1 =

ai xi

if f (μi )f (xi ) > 0, if f (μi )f (xi ) < 0.

Suppose that f  (x) exists, xi < ai , f (xi ) < 0 < f (ai ), and f  (x)  0 for x ∈ [xi , ai ]. Prove that the sequence (xi ) converges linearly to a zero of f .

May 2, 2018 11:28

892

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 892

Mathematical Analysis for Machine Learning and Data Mining

Bibliographical Comments Newton’s algorithm and many of its variants is discussed in [86], and in [126] from a numerical perspective. The treatment of conjugate-direction methods follows [71]. The method of conjugate gradients was introduced in [78]. Two important and readable sources for iterative algorithms are [29, 71]. An encyclopedic source for numerical algorithms is [126].

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 893

Chapter 15

Neural Networks

15.1

Introduction

In this chapter we discuss application of mathematical analysis techniques in the study of neural networks (NN). Neural networks are aggregates of computational devices referred to as neurons that model loosely the working of biological components of nervous systems (also named neurons). The use of NNs, as a part of the more general paradigm of neural computing has broad applications in pattern recognition and in various types of data analysis. After discussing the neuron as the main component of a NN, we focus on a common architectures of NN. Properties of NN as universal approximators are treated in Section 15.4. Finally in Supplements where we present in detail the rich collection of results contained in [83, 82, 60, 62].

15.2

Neurons

The time in an NN is discrete, which means that it varies in the set N. The current time is t ∈ N, the next time is t + 1 and the preceding one is t − 1. Informally, a neural network is a directed graph whose set of nodes N consists of neurons; edges of this graph represent the information flows between neurons. As we shall see, there are several types of neural networks depending on the structure of the underlying graph. A neuron is defined by a threshold value θ, an activation function σ : R −→ R and a vector of weights w ∈ Rn . When an input vector x ∈ Rn is fed into the n inputs of the neuron, the processing unit computes y = σ(w x + θ). The value of y is taken as the unit output. 893

May 2, 2018 11:28

894

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 894

Mathematical Analysis for Machine Learning and Data Mining

Definition 15.1. A sigmoidal function or a squashing function is a nondecreasing function σ : R −→ R such that limx→∞ σ(x) = 1 and limx→−∞ σ(x) = 0. In general, activation functions are sigmoidal functions. By Theorem 4.62, the set of their discontinuity points of the first kind is at most countable, and therefore, they are measurable functions by Corollary 7.7. The monotonicity may be dropped if measurability is required explicitly. Example 15.1. Several of the most common sigmoidal activation functions are listed next: (i) The Heaviside function is the function f defined by:  1 if x  0, f (x) = 0 if x < 0. This function is discontinuous at x = 0. (ii) The Fermi logistic function is defined as 1 fT (x) = x , 1 − e− T where T > 0 is a parameter, is continuous everywhere and ranges in the open interval (0, 1). ex (iii) The logistic function is given by L(x) = 1+e x. (iv) The ramp function is given by ⎧ ⎪ ⎪ ⎨0 if x < 0, f (x) = x if 0  x  1, ⎪ ⎪ ⎩1 if x > 1. (v) The cosine squasher of Gallant and White [62] is the function csq given by: ⎧ # π/ ⎪ 0 if x ∈ −∞, − , ⎪ ⎪ ⎪ 2   ⎪ ⎪ 3π ⎨ +1 cos x + # $ csq(x) = 2 π π ⎪ if x ∈ − , , ⎪ ⎪ 2 . π 2 2$ ⎪ ⎪ ⎪ ⎩1 ,∞ if x ∈ ⎧ # /2 π ⎪ 0 if x ∈ −∞, − , ⎪ ⎪ ⎨ # $2 = 1 + sin x if x ∈ − π , π , ⎪ 2 ⎪ 8 2 2 ⎪ ⎩1 if x ∈ π2 , ∞ .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Neural Networks

15.3

b3234-main

page 895

895

Neural Networks

Definition 15.2. A neural network (NN) is a triplet N = (N, E, w), where G = (N, E) is a graph (as above) and w : E −→ R is a function that specifies the weight w(i, j) of an edge (i, j) between the neurons i and j. We denote by wij = w(i, j) the weight of the connection between the neurons i and j. A simple NN consists of a layer of n input units and a layer of m processing units (see Figure 15.1). xn

xi

x2

x1

-v H ..LJ HH H H . LJ  wim L J   J L  -v  H wi2 J HL J JJ LH H wJ  H .. i1J L   L

.    J L - v  J PP

PP JLJL P    -

 v Input neurons

Fig. 15.1

g

- ym

.. . g

- y2

g

- y1

Processing neurons

Simple neural network with n input neurons and m processing neurons.

A neural network can be extended to include an output unit that sums the outputs of the processing units (see Figure 15.2). The network can be trained to approximate a class of functions F of the form f : Rn −→ Rm . When the network is supplied with new examples (x, f (x)), where x ∈ Rn an algorithm is applied to modify the weights wij and the thresholds θj such that the difference between f (x) and the  y1  network output y =

. . . ym

is minimized.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

b3234-main

page 896

Mathematical Analysis for Machine Learning and Data Mining

896

xn

xi

x2

x1

-v H ..LJ HH H H . LJ  wim L J   L J  -v  H L J HH Jwi2 JJ L H wJi1 L H .. J   L

.    J L - v  J P

JL P P P JL P    -

v  Input nodes

ym g .. .

y2 g -

@ @ β@ m @ @Σ m β2 i=1

y1 g -

Processing neurons

Fig. 15.2

15.4

9in x 6in

βi g(wi x + θ)

β1

Output node

Single output neural network.

Neural Networks as Universal Approximators

Let In = [0, 1]n be the n-dimensional cube. The set of finite signed regular Borel measures on In is denoted by M (In ). In [37] it is shown that sums of   sigmoidal functions of the form N i=1 αj σ(wj x + θj ) are dense in the space C(In ) of real-valued continuous on In equipped with the metric dsup for any sigmoidal function σ. Definition 15.3. Let m be a finite signed regular Borel measure on In . A function σ ∈ C(In ) is discriminatory relative to m if 7 σ(w x + θ) dm = 0 In

for all w ∈ In and θ ∈ R imply m = 0. Theorem 15.1. Let σ be a continuous discriminatory function. Then the   set of finite sums of the form g(x) = N i=1 αj σ(wj x+ θj ) is dense in C(In ). Proof. The statement amounts to the fact that for every f ∈ C(In ) and > 0, there exists a sum g(x) of the above form such that |g(x) − f (x)| < for x ∈ In . Let S be the subset of C(In ) that consist of functions the form g(x) = N  i=1 αj σ(wj x + θj ). It is clear that S is a linear subspace of C(In ). Suppose that the topological closure R = K(S) of S is strictly included in C(In ), that is, S is not dense in C(In ). Then R is a closed proper

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Neural Networks

page 897

897

subspace of C(In ). By the Hahn-Banach theorem, there is a non-zero bounded linear functional on C(n ) such that (R) = 0 (hence (S) = 0). 6 By the Riesz-Kakutani Theorem (Theorem 8.69), is of the form (h) = h(x) dm for some measure m ∈ M (In ) for all h ∈6 C(In ). In particular, In since σ(w x + θ) ∈ R for all w and θ, we must have In σ(w x + θ) dm = 0 for all w and θ. Since σ is discriminatory, m = 0, which contradicts the fact that is  not zero. Hence R = C(In ) and therefore S must be dense in C(In ). Theorem 15.2. Any bounded and measurable sigmoidal function σ is discriminatory. 6 Proof. Let σ ∈ C(In ) be a function such that In σ(w x + θ) dm = 0 for all w ∈ In and θ ∈ R. Note that for the function σλ defined by σλ,φ (x) = σ(λ(w x + θ) + φ) 6 for x ∈ In enjoys the same property, namely In σλ,φ (w x + θ) dm = 0 for all w ∈ In and θ ∈ R. In other words, σ is discriminatory if and only if each function σλ,φ is discriminatory. If w x + θ = 0 we have σ(λ(w x + θ) + φ) = σ(φ). Furthermore,  1 if w x + θ > 0,  lim σ(λ(w x + θ) + φ) = λ→∞ 0 if w x + θ < 0, and σ(λ(w x + θ) + φ) = σ(φ) if w x + θ = 0. The family of functions {σλ,φ | λ ∈ R0 } converges pointwise to and is dominated by the integrable function γ : In −→ R defined by ⎧ ⎪ if w x + θ > 0, ⎪ ⎨1 γ(x) = 0 if w x + θ < 0, ⎪ ⎪ ⎩σ(φ) if w x + θ = 0. >0 be the corresponding Let Hw,−θ be a hyperplane and let Hw,−θ half-space. By the 6Dominated Convergence Theorem (Theorem 8.37) 6 limλ→∞ In σλ dm = In γ dm. Note that 7 7 7 7 0= σλ dm = σλ dm + σλ dm + σλ dm >0 Hw,−θ

In

7

7

Hw,−θ

1 dm +

= >0 Hw,−θ

σ(φ) dm Hw,−θ

>0 ) + σ(φ)m(Hw,−θ ). = m(Hw,−θ

0 there exists a function g as above such that g(x) − f (x) < for x ∈ In . Proof. This statement follows from the fact that continuous sigmoidal functions are discriminatory.  Let π = {P1 , . . . , Pk } be a partition of In , where Pi is mL -measurable subsets of In for 1  i  k. Define the decision function fπ : In −→ {1, . . . , k} as f (x) = j if x ∈ Pj . Theorem 15.3. Let σ be a continuous sigmoidal function and let fπ be the decision function of the finite measurable partition π of In , For any > 0 N  there is a finite sum of the form g(x) = i=1 αj σ(wj x + θj ) and a set D ⊆ In so that mL (D)  1 − and |g(x) − f (x)| < for x ∈ D.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Neural Networks

b3234-main

page 899

899

Proof. By Lusin’s Theorem (Supplement 49 of Chapter 8) there is a continuous function h and a set D with m(D)  1 − so that h(x) = f (x) for x ∈ D. Since h is continuous, by Corollary 15.1 there is a sum of the N form g(x) = i=1 αj σ(wj x + θj ) that satisfies |g(x) − h(x)| < for all  x ∈ In . Then, for x ∈ D we have |g(x) − f (x)| = |g(x) − h(x)| < .

15.5

Weight Adjustment by Back Propagation

We consider now a neural network architecture involving three layers of vertices: input nodes, inner layer neurons, and output layer neurons. The task of the network is to learn an approximation of a bounded function f : A −→ Rh , where A is a compact subset of Rn . The training set is ((x1 , t1 ), . . . , (xm , tm )) ∈ Seq(Rn × Rh ), where x1 , . . . , xm are m input vectors that are randomly selected from A, and t1 , . . . , tm are the expected output vectors, respectively, where ti = f(xi ) for 1  i  m. The back propagation process consists of two phases: in the first phase (the forward pass) vectors xj are supplied to the input units and the output yj is collected from the output units. Then the correct output tj is compared to yj and a second backward sweep of the network involving weight adjustments begins. The activation functions of the internal nodes and of the output nodes are assumed to be the logistic functions L : R −→ R having the form ex L(x) = 1+e x . Note that L (x) =

ex = L(x)(1 − L(x)) (1 + ex )2

(15.1)

for x ∈ R. The weights wij of the edges of a neural network are adjusted through a sequential process. Suppose that the training sample of a neural network is the sequence ((x1 , t1 ), . . . , (xm , tm )) ∈ Seq(Rn ×Rh ), where x1 , . . . , xm are the input vectors and t1 , . . . , tm are the expected output vectors, respectively. In general, the network will produce the outputs y1 , . . . , ym instead of t1 , . . . , tm , which will trigger an incremental readjustment process of the weights of the network wij . When the expected output is t and the network produces the answer y, the error of the network is the function R : Rn×k × Rk×h −→ R defined as R(V ; W ) = 12 t − y2 , where V = (vi ) ∈ Rn×k is the matrix of weights of edges between input units and neurons in the hidden layer, and W = (wij ),

May 2, 2018 11:28

900

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 900

Mathematical Analysis for Machine Learning and Data Mining

x1

-

y j

.. .

^

v1 x

-

y

z1 p1

7N :

pi

 vk ~ .. . xn

-

y Input Units

Fig. 15.3



N1

-

.. .

R  zi

Ni

wi1 .. .

zk Nk

*

Inner Layer

O1

- y1

.. .

wij Oj

.. R .  wih

.. . pk

-U 

q1

- yj

.. .

qh

Oh

- yh

Output Layer

Neural networks with inner and output neurons.

wij is the matrix of weights of the edges between the hidden and the output neurons (see Figure 15.3). The j th component of the input vector xi is denoted by xij . The set of indices of output neurons connected to the neuron Ni from the hidden layer is denoted by D(i).  The input of an inner layer neuron Ni is pi = n=1 x vi for 1  i  k;  the input of an output layer neuron Oj is qj = ki=1 wij zi for 1  j  h, where zi is the output of the hidden layer neuron Ni . Finding a minimum of R(V ; W ) using a gradient descent algorithm (actually a stochastic gradient descent) requires the computation of ;W ) ;W ) and ∂R(V . ∇R(V ; W ), that is, of the partial derivatives ∂R(V ∂vi ∂wij R(V ; W ) depends on vi via pi ; therefore, ∂R(V ; W ) ∂R(V ; W ) ∂pi ∂R(V ; W ) = = x . ∂vi ∂pi ∂vi ∂pi Similarly, R(V ; W ) depends on wij via qj , the input of the output neuron Oj .

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Neural Networks

page 901

901

Thus, we have: ∂R(V ; W ) ∂R(V ; W ) ∂qj ∂R(V ; W ) = = zi . ∂wij ∂qj ∂wij ∂qj ;W ) To compute the error gradient component ∂R(V for an output neuron ∂wij observe that R(V ; W ) depends on qj via yj , the output of Oj . Thus,

∂R(V ; W ) ∂R(V ; W ) ∂yj = . ∂qj ∂yj ∂qj  Since R(V ; W ) = 12 t − y2 = 12 hj=1 (tj − yj )2 , we have: ∂R(V ; W ) = −(tj − yj ). ∂yj Since yj = L(qj ), ∂yj = L(qj )(1 − L(qj )), ∂qj which allows us to write ∂R(V ; W ) = −(tj − yj )L(qj )(1 − L(qj )), ∂qj hence ∂R(V ; W ) = −(tj − yj )L(qj )(1 − L(qj ))zi . ∂wij ;W ) To compute the error gradient component ∂R(V for an inner layer ∂vi neuron note that vi influences the error through the output neurons in Di located downstream from Ni . Therefore, we have:  ∂R(V ; W ) ∂qj ∂R(V ; W ) = ∂pi ∂qj ∂pi j∈D(i)

 ∂R(V ; W ) ∂qj ∂zi ∂qj ∂zi ∂pi j∈D(i)  = −zi (1 − zi ) (tj − yj )L(qj )(1 − L(qj ))wij , =

j∈D(i) ∂q

∂zi because ∂zji = wij and ∂p = zi (1 − zi ). i Weights of the network are adjusted using the gradient descent rule ;W ) ;W ) and wij := wij − η ∂R(V , where η is a small vi := vi − η ∂R(V ∂vi ∂wij positive constant called the learning rate.

May 2, 2018 11:28

902

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 902

Mathematical Analysis for Machine Learning and Data Mining

Exercises and Supplements Several notations for classes of functions are needed to present some of the rich collection of approximation results given in [82]. • the set of affine functions h : Rr −→ R of the form h(x) = a x + b, where a ∈ Rr } is denoted by Ar ; • for a Borel measurable function g : Rr −→ R the class of function Σr (g) consists of functions of the form f : Rr −→ R given by: f (x) =

q 

βj g(hj (x)),

j=1

where βj ∈ R and hj ∈ Ar for 1  j  q; networks that implement these functions consist of a single layer of neurons having activation function g; • the class ΣΠr (g) consists of functions f : Rr −→ R of the form

f (x) =

q  j=1

j

βi

g(hjk (x)), k=1

where hj ∈ Ar for 1  j  q and 1  k  j ; • the set of continuous functions of the form f : Rr −→ R is denoted by C r ; • the set of Borel measurable functions from Rr to R is denoted by M r . Also, by Theorem 7.13, we have C r ⊆ B(Rr ). (1) For every Borel measurable function g we have: Σr (g) ⊆ B(Rr ) and ΣΠr (g) ⊆ B(Rr ). (2) Let M be a positive integer. Prove that for the cosine squasher csq we have:

+ *   3π π + 2πm − 1 2 csq −t + − 2πm + csq t − 2 2 m=−M ⎧ ⎪ if − ∞ < t < −2πM, ⎨0 = cos t − 1 if − 2πM  t  2π(M + 1), ⎪ ⎩ 0 if 2π(M + 1) < t < ∞. M 

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Neural Networks

page 903

903

Solution: The definition of csq allows us to write ⎧

⎪ ⎨0 3π + 2πm = cos2t+1 csq t − ⎪ 2 ⎩ 1

if t  π − 2πm, if π − 2πm < t < 2π − 2πm, if t  2π − 2πm,

and 

csq −t +



⎧ ⎪ ⎨0

π − 2πm = cos2t+1 ⎪ 2 ⎩ 1

if t  π − 2πm, if − 2πm < t < π − 2πm, if t  −2πm.

Substituting these expressions in the left member of the equality of the supplement yields the right member. Let K be a compact subset of Rr . Define the semi-metric dK on C r as dK (f, g) = sup |f (x) − g(x)|. x∈K

A sequence of functions (fn ) in C r converges to a function f uniformly on compact sets if for all compact subsets K of Rn we have limn→∞ dK (fn , f ) = 0. A set S of continuous functions of the form f : Rr −→ R is uniformly dense on compact sets in C r if for every  > 0, compact set K, and for every g ∈ C r , there is f ∈ S such that dK (f, g) < . (3) If g : R −→ R is a continuous non-constant function, prove that ΣΠr (g) is uniformly dense on compact sets in C r . This means that ΣΠ NNs are capable of approximating arbitrarily close any real-valued continuous function over a compact set. Solution: Let K be an arbitrary compact set in Rr . It is immediate that for any g, ΣΠr (g) is an algebra of functions defined on K. If x, y ∈ K and x = y, there exists an affine function h ∈ Ar such that g(h(x)) = g(h(y)). Indeed, since g is not a constant function there exist a, b ∈ R such that a = b and g(a) = g(b). Let h be such that h(x) = a and h(y) = b. Then, g(h(x)) = g(h(y)), so ΣΠr (g) separates points. Therefore, ΣΠr (g) is separating on K. Let b ∈ R be such that g(b) = 0 and define h as h(x) = 0r x + b. For all x ∈ K, g(h(x)) = g(b), so ΣΠr (g) contains the constant functions. The Stone-Weierstrass Theorem (Theorem 5.40) implies that ΣΠr (g) is uniformly dense on compact sets in C r . (4) Let d be the semi-metric introduced in Exercise 18 of Chapter 7 on B(Rr ), the set of Borel measurable functions of r arguments.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 904

Mathematical Analysis for Machine Learning and Data Mining

904

Prove that the following statements that concern a sequence (fn ) of functions in B(Rr ) and a function f :∈ B(Rr ) are equivalent: (a) limn→∞ d(fn , f ) = 0; (b) for every" > 0, limn→∞ mL ({x | |fn (x) − f (x)| > }) = 0; (c) limn→∞ Rr min{|fn (x) − f (x)|, 1} dmL = 0. Solution: The equivalence of the first two parts is immediate. Define the subsets An , Bn of Rr as: An = {x ∈ Rr | |fn (x) − f (x)|  1} and Bn = {x ∈ Rr | |fn (x) − f (x)| > 1}. Clearly, An and Bn partition Rr and we have: ! min{|fn (x) − f (x)|, 1} dmL ! ! = min{|fn (x) − f (x)|, 1} dmL + min{|fn (x) − f (x)|, 1} dmL A Bn ! n ! = |fn (x) − f (x)| dmL + 1 dmL . Rr

An

Bn

Suppose that the second statement holds. Then, for sufficiently large n we have !  |fn (x) − f (x)| dmL < , 2 An !

and

1 dmL < Bn

hence

 , 2

! Rr

min{|fn (x) − f (x)|, 1} dmL < 

when n is sufficiently large, and this implies the third statement. The third statement implies the second by Markov’s inequality. Recall that the semimetric d on M r was introduced in Supplement 18 of Chapter 7. It will play an important role in the current collection of supplements. (5) If (fn ) is a sequence of functions in M r that converges uniformly on compacta to a function f , prove that limn→∞ d(fn , f ) = 0. Solution: Let  be a positive number. By" Supplement 4, it suffices to find n0 ∈ N such that for all n  n0 , we have min{fn (x)−f (x), 1} dm < . Without loss of generality we can assume that m(Rr ) = 1. By Theorem 7.61 m is a regular measure. There exists a compact subset K

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Neural Networks

b3234-main

page 905

905

of Rr with m(K) > 1 − /2. Choose n0 such that for n  n0 we have supx∈K |fn (x) − f (x)| < /2. Now, ! Rr −K

! min{fn (x) − f (x), 1} dm +

min{fn (x) − f (x), 1} dm <  K

for n  n0 . (6) Prove that for every continuous non-constant function g, every r, and every measure space (Rr , B(Rr ), m) with m(Rr ) < ∞, ΣΠr (g) is d-dense in M r , where d is the semimetric introduced in Exercise 18 of Chapter 7. Solution: Since g is a continuous non-constant function, by Supplement 3, ΣΠr (g) is d-dense on compact sets in C r . On other hand, since C r is d-dense in M r , it follows that ΣΠr (g) is d-dense in M r . (7) Prove that for every continuous squashing function f , every squashing function ψ, and every  > 0 there is a function g ∈ Σ1 (ψ) such that supλ∈R |f (λ) − g (λ)| < . Solution: Let  ∈ (0, 1). We seek a finite set {β1 , . . . , βq−1 } such that



q−1







sup f (λ) − βj ψ(hj (λ)) < ,



λ∈R j=1

where βj ∈ R and hj ∈ Ar for 1  j  q − 1. Let q be such that q > 2/. For j ∈ {1, . . . , q − 1} let βj = 1q . Since ψ is a squashing function there exists M such that: ψ(−M )
1 − . 2q 2q

Since f is a continuous squashing function there exist r1 , . . . , rq defined by: 



j

rj = sup λ f (λ) = for 1  j  q − 1,

q 



1

rq = sup λ f (λ) = 1 − .

2q For any r < s let hr,s be the unique affine function that satisfies the equalities hr,s (r) = M and hr,s (s) = −M . The desired function is given by q−1  βj (hrj rj+1 (λ)). g (λ) = j=1

It is easy to verify that on each interval (−∞, r1 ], (r1 , r2 ], . . . , (rq−1 , rq ], (rq , ∞) we have: |f (λ) − g (λ)| < .

May 2, 2018 11:28

906

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 906

Mathematical Analysis for Machine Learning and Data Mining

(8) Prove that for every squashing function ψ, every r ∈ N, and every measure space (Rr , B(Rr ), m), where m is a finite measure, the set ΣΠr (ψ) is uniformly dense on compacta in C r and d-dense in the set of Borel functions M r . Solution: We begin by showing that every function of the form Πk=1 ghk can be uniformly approximated by members of ΣΠr (ψ). Let  > 0. Since multiplication is continuous and (0, 1) , there is δ > 0 such that |ak − bk | < δ for 0  ak , bk  1 and 1  k   imply











ak − k=1

k=1





bk < .

By Supplement 7, there is a function g ∈ Σ1 (ψ) such that supλ∈R |f (λ)− g | < . (9) Prove that for every squashing function ψ, every positive , and every M > 0 there is a function κM, ∈ Σ1 (ψ) such that sup

|κM, (x) − cos x| < .

x∈[−M,M ]

Solution: If ψ = csq the result follows from Supplement 2. For an arbitrary squashing function the result follows from Supplement 7.  (10) Let g : Rr −→ R be the function defined as g(x) = qj=1 βj cos(hj (x)), r where hj ∈ A . Prove that for an arbitrary squashing function ψ, an compact K ⊆ Rr , and  > 0 there exists f ∈ Σr (ψ) such that supx∈K |g(x) − f (x)| < . Solution: Since K is compact and hj are continuous, there exists M >  0 such that hj (K) ⊆ [−M, M ] for 1  j  q. Define  = q qj=1 qj=1 |βj |. By Supplement 9 for all x ∈ K we have q | qj=1 κM,/q (hj (x)) − g(x)| <  for all x ∈ K. This implies f = q r 1   j=1 κM,/q (hj ) ∈ Σ (Ψ) because κM,/q ∈ Σ (ψ). (11) Prove that for every squashing function ψ, the set Σr (ψ) is uniformly dense on compact sets in C r . Solution: By Supplement 3 the trigonometric polynomials of the form q j r j=1 βj Πk=1 cos hjk , where q  1, j ∈ N and hjk ∈ A are uniformly dense on compacta in C r . Since cos a cos b = 12 (cos(a + b) + cos(a − b)), each polynomials is a sum of the form m of the previous trigonometric r a cos h , where h ∈ A . The result follows by an application of i i i i=1 Supplement 10. Next, we show that standard NNs with a single hidden layer can approximate any continuous function uniformly on any compact set, and any measurable function in the sense of the d semimetric, regardless of ψ, the dimension r and the measure that generates d.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Neural Networks

page 907

907

(12) Prove that for every squashing function ψ, r  1, and every finite measure space (Rr , B(Rr ), m) the set σ r (ψ) is uniformly dense on compacta in C r . Solution: By Supplement 11, Σr (ψ) is uniformly dense on compact sets in C r . Therefore, by Supplement 5, Σr (ψ) is d-dense in C r . (13) Let N be a neural net that contains three layers of neurons: two input neurons, two inner layer neurons and two output neurons (as shown in Figure 15.3. The activation function is the logistic function L. Suppose that the initial values of the matrices V and W are 11 11 V = and W = 11 11 and that the network is intended to learn the function f : {0, 1}2 −→ {−1, 1} given by f(0, 1) = f(1, 0) = (1, −1), f(0, 0) = f(1, 1) = (−1, 1). Apply the back propagation algorithm to determine the appropriate weights of the network capable of approximating f. (14) Prove that (tanh(x)) = 1 − tanh2 (x); reformulate the back propagation algorithm by replacing the sigmoidal activation function of the neurons by the hyperbolic tangent.

Bibliographical Comments The cosine squasher was introduced by A. R. Galant and H. White in [62] where the equality of Supplement 2 is stated. In [83] it was shown that standard NN with one hidden layer using arbitrary squashing functions are capable of approximating and Borel measurable function with a any degree of accuracy by using an adequate number of hidden units. The approach in [83, 82, 6] is based on the application of Stone-Weierstrass theorem and provide a distinct approach to the uniform approximation results obtained in [37, 38], which make use of the HahnBanach Theorem. See also [100]. Useful sources for approximation results for neural networks are [134] and [44]. The references [76] and [77] provide broad perspectives on neural computing.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 909

Chapter 16

Regression

16.1

Introduction

The term regression covers a series of techniques that explain the relationships between a variable that represents the results of an experiment and the independent variables that represent input parameters of the experiment. These techniques have been long explored by statisticians and machine learning researchers. The ideas and algorithms they developed are useful in forecasting and classification. We use regression to illustrate two of the most popular regularization techniques (·2 and ·1 regularization) which involve modifying the objective function of certain optimization problems in order to obtain solutions with certain desirable properties. 16.2

Linear Regression

Suppose that the results of a series of m experiments are the components of a vector y ∈ Rm . For the ith experiment, the values of the n input variables x1 , . . . , xn are placed in the ith row of a matrix B ∈ Rm×n known as the design matrix, and we assume that the outcome of the ith experiment yi is a linear function of the values bi1 , . . . , bin of x1 , . . . , xn , that is yi = bi1 r1 + · · · + bin rn . The variables x1 , . . . , xn are referred to as the regressors. Note that the values assumed by the variable xj in the series of m experiments, b1j , . . . , bmj have been placed in the j th column bj of the matrix B. Linear regression assumes the existence of a linear relationship between the outcome of an experiment and values of variables that are measured during the experiment. 909

May 2, 2018 11:28

910

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 910

Mathematical Analysis for Machine Learning and Data Mining

In general there are more experiments than variables, that is, we have n < m. In matrix form we have y = Br, where B ∈ Rm×n and r ∈ Rn . The problem is to determine r, when B and y are known. Since n < m, this linear system is inconsistent, but is possible to obtain an approximative solution by determining r such that Br − y is minimal. This amounts to approximating y by a vector in the subspace Ran(B) generated by the columns of the matrix B. The columns b1 , . . . , bn of the matrix B are referred to as the regressors; the linear combination r1 b1 + · · · + rn bn is the regression of y onto the regressors b1 , . . . , bn . A variant of the previous model is to assume that y is affinely dependent on b1 , . . . , bq , that is, y = r0 + r1 b1 + · · · + rn bn , and we seek to determine the coefficients r0 , r1 , . . . , rn . The term r0 is the bias of the model. The dependency of y on b1 , . . . , bn can be homogenized by introducing a dummy vector b0 having all components equal to 1, which gives y = r0 b0 + r1 b1 + · · · + rn bn , as the defining assumption of the model. If the linear system Br = y has no solution r, the “next best thing” is to find a vector r∗ ∈ Rn such that Br∗ − y2  Bw − y2 for every w ∈ R . This approach is known as the least square method. We refer to the triple (B, r, y) as an instance of the least square problem. Note that Br ∈ Ran(B) for any r ∈ Rn . Thus, solving this problem amounts to finding a vector Br in the subspace Ran(B) such that Br is as close to y as possible. Let B ∈ Rm×n be a full-rank matrix such that m > n, so rank(B) = n. In this case, the symmetric square matrix B  B ∈ Rn×n has the same rank n as the matrix B, and B  B is an invertible matrix. Therefore, the system n

(B  B)r = B  y 

−1



(16.1) 

has a unique solution r = (B B) B y. Moreover, B B is positive definite because r B  Br = (Br) Br = Br22 > 0 for r = 0n . Theorem 16.1. Let B ∈ Rm×n be a full-rank matrix such that m > n and let y ∈ Rm . The unique solution r = (B  B)−1 B  y of the system (B  B)r = B  y equals the projection of the vector y on the subspace Ran(B).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Regression

page 911

911

Proof. The n columns of the matrix B = (b1 · · · bn ) constitute a basis of the subspace Ran(B). Therefore, we seek the projection c of y on Ran(B) as a linear combination of the columns of B, c = Br, which allows us to reduce this problem to a minimization of the function f (r) = Br − y22 = (Br − y) (Br − y) = (r B  − y )(Br − y) = r B  Br − y Br − r B  y + y y = r B  Br − 2y Br + y y.

(16.2)

The necessary condition for the minimum is (∇f )(r) = 2B  Br − 2B  y = 0, which implies B  Br = B  y. 





The linear system (B B)r = B y is known as the system of normal equations of B and y. The solution of this system, r = (B  B)−1 B  y is known as the linear estimator. Suppose now that B ∈ Rm×n has rank k, where k < min{m, n}, and U ∈ Rm×m , V ∈ Rn×n are orthonormal matrices such that B can be factored as B = U M V  , where   R Ok,n−k M= ∈ Rm×n , Om−k,k Om−k,n−k R ∈ Rk×k and rank(R) = k. # $ For y ∈ Rm define c = U  y ∈ Rm and let c = cc12 , where c1 ∈ Rk and c2 ∈ Rm−k . Since rank(R) = k, the linear system Rz = c1 has a unique solution z1 . Theorem 16.2. All vectors r that minimize Br − y2 have the form   z r=V , w for an arbitrary w. Proof. We have Br − y22 = U M V  r − U U  y22 = U (M V  r − U  y)22 = M V  r − U  y22 (because multiplication by an orthonormal matrix is norm-preserving) = M V  r − c22 = M y − c22 = Rz − c1 22 + c2 22 ,

May 2, 2018 11:28

912

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 912

Mathematical Analysis for Machine Learning and Data Mining

where z consists of the first r components of y. This shows that the minimal value of Br − y22 is achieved by the solution of the system Rz = c1 and is equal to c2 22 . Therefore, the vectors r that minimize Br − y22 have z the form for an arbitrary w ∈ Rn−r .  w Instead of the Euclidean norm we can use the  · ∞ . Note that we have t = Br − y∞ if and only if −t1 ≤ Br − y ≤ t1, so finding r that minimizes  · ∞ amounts to solving a linear programming problem: minimize t subjected to the restrictions −t1 ≤ Br − y ≤ t1. 16.3

A Statistical Model of Linear Regression

We discuss now a statistical model of the problem stated in Section 16.2. We assume that the results of the experiments are described by an mdimensional random vector y that is affected by errors. These errors are represented by an m-dimensional error random vector : y = Br + . We also assume that the random vector is centered, that is, E( ) = 0m . Its covariance matrix cov( ) ∈ Rm×m is positive definite. We seek an linear estimate of the constant vector r as the random vector ˆr = Ky, where K ∈ Rn×m . We aim to minimize the expectation E(ˆr−r2 ) of the Euclidean norm of error random vector ˆr −r by an appropriate choice of the matrix K. Note that we have: E(ˆr) = E(Ky) = E(K(Br + )) = KBr. If KB = In , then E(ˆr) = r, and the random vector ˆr is an unbiased estimator of r. Assuming that KB = In we have: E(ˆr − r2 ) = E(Ky − r2 ) = E(K(Br + ) − r2 ) = E(K 2 ), so in this case the expected norm of the error is independent of r. Note that for v ∈ Rm we have v2 = v v = trace(vv ). This allows us to write E(ˆr − r2 ) = E(trace(K  K  )) = trace(Kcov( )K  ).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Regression

page 913

913

Conversely, if we require that a random vector of the form ˆr = Ky is an unbiased estimator of r, that is, E(ˆr) = r, it follows that KB = In . Lemma 16.1. Let Q ∈ Rm×m be a positive definite matrix and let B ∈ Rm×n be a full-rank matrix such that m > n, so rank(B) = n. The optimization problem that seeks a matrix K ∈ Rn×m that minimizes trace(KQK  ) subjected to the constraint KB = In has as a solution the matrix K given by K = Q−1 B(B  Q−1 B)−1 . Proof.

Let k1 , . . . , km be the columns of matrix K. We have trace(KQK  ) =

m 

ki Q(ki ) .

j=1

Since Q is positive definite, we have ki Q(ki )  0 for 1  i  m, so the previous problem amounts to m optimization problems: minimize (ki ) Qki , subject to (ki ) bj = δij ,

where

 δij =

1

if i = j,

0

if i = j.

In terms of the inner product defined by the matrix Q an equivalent formulation is minimize ki Q , subject to (ki , Q−1 bj )Q = δij for 1  j  n,

because (ki ) bj = ki Q(Q−1 bj ) = (ki , Q−1 bj )Q . By Supplement 6 of Chapter 11, the solution of this optimization problem  is ki = Q−1 B(B  Q−1 B)−1 ei . Therefore, K = Q−1 B(B  Q−1 B)−1 . Theorem 16.3. (Gauss-Markov Theorem) Suppose that y = Br + , where y and are random vectors such that E( ) = 0m and Q = E(  ) is a positive definite matrix. The linear minimum-variance unbiased estimate of r is ˆr = (B  Q−1 B)−1 B  Q−1 y and the error covariance is E((ˆr − r)(ˆr − r) ) = (B  Q−1 B)−1 . Proof. By Lemma 16.1, the minimum-variance estimate ˆr = Ky is given by ˆr = (B  Q−1 B)−1 B  Q−1 y.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 914

Mathematical Analysis for Machine Learning and Data Mining

914

For the error covariance matrix we have: E((ˆr − r)(ˆr − r) ) = E((Ky − r)(Ky − r) ) = E(K  K  ) = KQK  = Q−1 B(B  Q−1 B)−1 QQ−1 B(B  Q−1 B)−1 = (B  Q−1 B)−1 .



Corollary 16.1. If the covariance matrix Q = E(  ) equals Im , then the minimum-variance estimate ˆr equals the least-square estimate (B  B)−1 B  y. Proof. 16.4

This fact follows directly from Theorem 16.3.



Logistic Regression

Despite its name logistic regression is essentially a classification technique. The term “regression” is justified by the use of a probabilistic approach involving the linear model defined for linear regression. The typical problem involves classifying objects into two classes, designated as C1 and C−1 . Let s be a data sample of size m, that consists of the pairs of values of a random vector X ranging over Rn and a random variable Y ranging over {−1, 1}, namely s = ((x1 , y1 ), . . . , (xm , ym )), where x1 , . . . , xm belong to Rn and yi ∈ {−1, 1} for 1  i  m. In logistic regression we assume that the logarithmic ratio =1|X=x) is an affine function r0 + r1 x1 + · · · + rn xn . If a dummy ln PP(Y(Y=−1|X=x) component x0 that is set to 1 is added, as we did for linear regression, then the above assumption can be written as ln

P (Y = 1|X = x) = r x, P (Y = −1|X = x)

(16.3)

where r, x ∈ Rn+1 . Let : (0, 1) −→ R be the logit function defined as p

(p) = ln 1−p x

e for p ∈ (0, 1) and let f : R −→ (0, 1) be the logistic function L(x) = 1+e x defined in Example 15.1. Note that L(x) + L(−x) = 1 for x ∈ R and the fact that L and are inverse functions.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Regression

page 915

915

Equality (16.3) can be written as 

P (Y = 1|X = x) =

er x = L(r x) 1 + er x

and 1 = 1 − L(r x) = L(−r x). 1 + er x Both cases are captured by the equality P (Y = −1|X = x) =

P (Y = y|X = x) = L(yr x). Equivalently, we have (P (Y = y|X = x)) = yr x. The object x is placed in the class Cy where y ∈ {−1, 1}), where y is the values that maximizes f (yr x). Since the example of s are independently generated the probability of obtaining the class yi for each of the examples xi is defined by the likelim hood function i=1 P (Y = yi |X = xi ). To simplify notations we denote m this function of yi and xi as i=1 P (yi |xi ). Maximizing this function is equivalent to minimizing m  m + 1  1 P (yi |xi ) = − ln P (yi |xi ) Λ(r) = − ln m m i=1 i=1 1  1  1  1 = ln L(yi rxi ) = ln ln(1 + e−yi rxi ), m i=1 m i=1 L(yi rxi ) m i=1 m

=−

m

m

with respect to r. Note that small values of this expression can be obtained when yi r xi is large, that is, when r xi has the same sign as yi . The decision boundary that separates the classes C1 and C−1 is given by the hyperplane x−yr = 0 in Rn+1 . Note that the distance of an example i −yi r (xi , yi ) to this hyperplane is |x (1,r) . ∂Λ = 0 for 1  j  To minimize Λ(r) we need to impose the conditions ∂r j n + 1, which amount to m  i=1

L (yi rxi )

∂(yi rxi ) = 0, ∂rj

or m 

L(yi rxi )(1 − L(yi rxi ))yj xji = 0,

i=1

by equality (15.1) of Chapter 15, for 1  j  n + 1. This is a non-linear system in r which can be solved by approximation methods.

May 2, 2018 11:28

916

16.5

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 916

Mathematical Analysis for Machine Learning and Data Mining

Ridge Regression

When the number n of input variables is large, the assumption previously made concerning the linear independence of the columns b1 , . . . , bn of the design matrix B may not hold and the rank of B may be smaller than n. In such a case, previous models are not applicable. The linear dependencies that may exist between the columns of B (reflecting linear dependencies among experiment variables) invalidate the assumptions previously made. These dependencies are known as colinearities among variables. The solution proposed in [80] is to replace B  B in the least-square estimate ˆr = (B  B)−1 B  y by B  B + λIn and to define the ridge regression estimate as r(λ) = (B  B + λIn )−1 B  y. The term ridge regression is justified by the fact that the main diagonal in the correlation matrix may be thought of as a ridge. We retrieve the ridge regression estimate as a solution of a regularized optimization problem, that is, as an optimization problem where the objective function is modified by adding a term that has an effect the shrinking of regression coefficients. Instead of minimizing the function f (r) = Br−y2 we use the objective function g(r, λ) = Br − y22 + λr22 . This approach is known as Tikhonov regularization method and g is known as the ridge loss function. A necessary condition of optimality is (∇g)r = 0n . This yields: (∇g)r = 2B  Br − 2B  y + 2λr = 2(B  Br − B  y + λr) = 2[(B  B + λIn )r − B  y] = 0n , which yields the previous estimate of r. The ridge estimator is therefore # 2 $a f , stationary point of g. The Hessian of g is the matrix Hg (x) = ∂r∂j ∂r k and it is easy to see that Hg (x) = 2(B  B + λIn ). This implies that Hg is positive definite, hence the stationary point is a minimum. Note that the ridge loss function is convex, as a sum of two convex functions. Therefore, the stationary point mentioned above is a global minimum.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Regression

16.6

b3234-main

page 917

917

Lasso Regression and Regularization

In regression problems it is often the case that the number of observations is too small, or the problem dimension too high which makes the data sparse. Without prior knowledge about data, it is difficult to estimate a model or make predictions. When a problem solution is known to be sparse, sparsity-inducing penalties improve both the quality of the prediction and its interpretability. In particular, the  · 1 is used for that purpose in the Lasso regression introduced in [131]. In this section we use the sign and its extension to Rn as introduced in Definition 12.12. Let u ∈ Rn and let J be a subset of {1, . . . , n}, J = {j1 , . . . , jm }. The projection of u on the subspace generated by the set {ej1 , . . . , ejm } is  uj  . . .

denoted by uJ =

1

.

ujm

Theorem 16.4. Let y ∈ Rm and let B = (b1 , . . . , bn ) ∈ Rm×n . Consider the following optimization problem minimize for r ∈ Rn 1 Br − y22 + λr1 , 2 where λ > 0.

A vector r∗ ∈ Rn is a solution if and only if for all j, 1  j  n we have (b ) (Br∗ − y) + λ sign((r∗ )j ) = 0 if (r∗ )j = 0, and |(bj ) (Br∗ − y)|  λ, otherwise. Let j 

J = {j | 1  j  n, |(bj ) (Br∗ − y)| = λ}. If the matrix BJ = (bj )j∈J is of full rank, the solution is unique and we have: (r∗ )J = (BJ BJ )−1 ((BJ y − ληη J ), where η = sign(B  (y − Br∗ )) ∈ {−1, 0, 1}n. Proof. Since  · 1 is not differentiable, we need to apply the subgradient optimality condition contained in Theorem 12.37, that requires   1 Br − y22 + λr1 (r∗ ). 0n ∈ subd 2

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 918

Mathematical Analysis for Machine Learning and Data Mining

918

Since 12 Br − y22 is differentiable, its subdifferential consists of its gradient B  Br−B  y. As we saw in Example 12.21, the subgradients in subd(·1 )(r) are the vectors p in Rn such that for j ∈ {1, . . . , n} we have pj = sign(rj ) if (r∗ )j = 0, and |pj |  1, otherwise. Thus, the optimality condition can be written as: 0n ∈ {B  Br − B  y + λp | p ∈ subd( · 1 )(r)}. This condition amounts to (bj ) (Br∗ − y) + λ sign((r∗ )j ) = 0

(16.4)

|(bj ) (Br∗ − y)|  λ,

(16.5)

if (r∗ )j = 0, and

if (r∗ )j = 0. Let J be the set of column indices of the matrix B that correspond to equalities in Conditions 16.4 and 16.5: J = {j | (bj ) (Br∗ − y)| = λ} = {j1 , . . . , jq }. Equalities (16.4) define a linear system that can be written as: (BJ ) (BJ r∗ − y) + λ sign((r∗ )J ) = 0q , or (BJ ) BJ r∗ = (BJ ) y − λ sign((r∗ )J ), similar to the system 16.1. If BJ ∈ Rm×q is a full rank matrix, this system has a unique solution r∗ = ((BJ ) BJ )−1 ((BJ ) y − λ sign((r∗ )J )) = ((BJ ) BJ )−1 ((BJ ) y − λ η J ), where η = sign(BJ ) (y − Br∗ ) ∈ {−1, 0, 1}p. To show the uniqueness of the solution let s be another solution and let θ ∈ (0, 1). By convexity, rθ = (1 − θ)r∗ + θs is also a solution. If j ∈ J we have (bj ) (Brθ − y)|  (1 − θ)(bj ) (Br∗ − y) + θ(bj ) (Bs − y) < λ.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Regression

9in x 6in

b3234-main

page 919

919

Let J be the complement of the set J, that is, J = {j | 1  j  n} − J. Taking into account Conditions 16.4 and 16.5, it follows that (rθ )J = (r∗ )J = 0n−q and the vector rθ is also a solution of the reduced problem minimize for ˜r ∈ R|J | 1 BJ ˜r − y22 + λ˜r1 , 2 where λ > 0.

When BJ is of full rank, the matrix (BJ ) BJ is positive definite and the reduced problem admits a unique solution rθ = (r∗ )J . Then r∗ =  rθ = s. The optimal solution of the Lasso regression problem is dependent on the parameter λ. Therefore, it is interesting to examine the geometric properties of the regularization path {r∗ (λ) | λ > 0}. The Lasso regression has the parsimony property [45]: for any give λ value only a subset of the variables have non-zero values of their respective coefficients rj (1  j  n). The set {ηη ∗ (λ) | η ∗ (λ) = sign(r∗ (θ)), λ > 0} is the set of sparsity patterns. Theorem 16.5. Assume that for any λ > 0 and solution of the Lasso regression problem, the matrix XJ is of full rank. Then, the regularization path {r∗ (λ) | λ > 0} is well-defined, unique and continuous piecewise linear. Proof. The existence and uniqueness of the regularization path was shown in Theorem 16.4. Let λ1 and λ2 be such that λ1 < λ2 but η ∗ (λ1 ) = η ∗ (λ2 ). If θ ∈ [0, 1], and λ = (1 − θ)λ1 + θλ2 , then the solution r∗ (θ) = (1 − θ)r∗ (λ1 ) + θr∗ (λ2 ) satisfies the optimality conditions of Theorem 16.4. This shows that whenever two solutions r∗ (λ1 ) and r∗ (λ2 ) have the same sign vector for λ1 = λ2 , the regularization path between λ1 and λ2 is a line segment. Note that the number of linear segments of the regularization path is no larger than 3n , the number of possible sparsity patterns. Therefore, the regularization path is piecewise linear and contains a finite number of segments. Since the function h : R>0 −→ rn defined by h(λ) = r∗ (λ) is piecewise continuous and has left and right limits for every positive λm and these limits also satisfy the optimality conditions and are equal to r∗ (λ), it follows that h is continuous. 

May 2, 2018 11:28

920

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 920

Mathematical Analysis for Machine Learning and Data Mining

Exercises and Supplements  j (1) Let {bj | 1  j  m} be a set of points in Rn such that m i=1 b = 0n .  Determine a hyperplane H defined by r x = 0 that passes through 0n such that the sum of the square distances from the points xi to H is minimal and prove that in this case r is an eigenvector of the square matrix B  B ∈ Rn×n , where B = (b1 · · · bm ) Solution: Without loss of generality assume that the hyperplane H is bj to defined by the equation r x = 0, where r = 1. The distance m from  j  j 2 H is dj = r b . Therefore, we need to minimize L(r) = j=1 (r b ) = ⎛ j⎞ b1  m n j 2 j . ⎠ ⎝ r b , where b = for 1  j  m. . i i j=1 i=1 . bjn

Thus, we have a constrained minimization problem n 2  ri bji , minimize L(r) = m j=1 n i=1 subject to i=1 ri2 − 1 = 0. The necessary extremum condition yields  (∇L)(r) = λ∇

n 

 ri2

−1 .

i=1

Since

∂L ∂rk

=2

m n j=1

i=1

2

 ri bji bkj , we have

 n m   j=1

 ri bji

bkj = λ · 2rk .

i=1

m×n If B = (b1 , . . . , bn ) ∈ R , we have bij = bji and the previous equality m n can be written as j=1 i=1 ri bji bjk = λrk for 1  j  m and 1  i  n which amounts to (B  B)r = λr, which shows that r is an eigenvector of B  B.

Data fitting is a generalization of linear regression that seeks to determine a function fa : Rn −→ R that belongs to a prescribed class of functions parametrized that for a data set by a ∈ Rn such that fa will best fit the data set. This means 2 D = {(xi , yi ) | 1  i  m} ⊆ Rn we seek a such that g(a) = m i=1 yi − fa (xi ) is minimized. (2) Let D = {(xi , yi ) | 1  i  m} be a data set in R2 . Determine a polynomial p of degree n pa (x) = a1 xn−1 + · · · + an−1 x + an that will fit D. In other words, determine the vector a ∈ Rn such that Xa − y2 is

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Regression

minimal, where

page 921

921

⎛

⎞ a1 ⎜ a2 ⎟ ⎜ ⎟ a=⎜ . ⎟ ⎝ .. ⎠ an

and ⎛ n−1 n−2 x1 x1 ⎜ n−1 xn−2 2 X = ⎝x2 .. .. . .

··· ··· .. .

⎛ ⎞ ⎞ y1 x1 1 ⎜ y2 ⎟ ⎜ ⎟ ⎟ x2 1 ⎠ , y = ⎜ .. ⎟ . ⎝.⎠ .. .. n−1 n−2 . .xm xm · · · xm 1 yn

(3) Let b ∈ Rm − {0m } and y ∈ Rm . Prove that br − y is minimal when r = (y,b) . b2 (4) Let B = (b1 , . . . , bn ) ∈ Rm×n be a matrix having orthogonal columns (in other words, i = j implies (bi , bj ) = 0) such that m > n. Prove that (a) matrix B has full rank, that is, rank(B) = n; (b) if r is the solution of the optimization problem that consists in j ) for minimizing the function f (r) = Br − y2 , then rj = (y,b bj 2 1  j  n. In other words, the components of the solution of linear regression do not influence each other. (5) By applying Gram-Schmidt orthogonalization algorithm to the set of columns b1 , . . . , bn that represent the values of the regressors and taking into account Exercise 4 formulate an algorithm for the linear regression. (6) Let B ∈ Rm×n be a matrix of full rank, where m > n, and let C = B(B  B)−1 B  ∈ Rm×m . As the regressor vector r is computed, the vector ˆ = Cy = B(B  B)−1 B  y = Br is the expectation of the output vector y conditioned on the regression model. Prove that C is a symmetric and idempotent, trace(C) = n, rank(C) = n, and cii ∈ [0, 1]; also, prove that the matrix Im − C is symmetric and idempotent. Let B = (b1 · · · bn ) ∈ Rm×n be a matrix that contains input data of m experiments. The rows of this matrix are denoted by u1 , . . . , um , where ui ∈ Rn contains the input values of the variable for the ith experiment. The average of 1 ˜ = 0n . Note that ˜ = m summ B is the vector u i=1 ui . The matrix is centered if u 1  1m B. Note that the matrix u ˜= m ˆ= B



1 Im − 1m 1m B m

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 922

Mathematical Analysis for Machine Learning and Data Mining

922

because

1 1  1  ˆ 1m B = 1m Im − 1m 1m B m m m

1 1 1m − 1m 1m 1m B = m m  1   1 − 1m B = 0. = m m 1 1m 1m ∈ Rm×m is the centering matrix. The matrix Hm = Im − m If the measurement scales of the variables x1 , . . . , xn involved in the series are very different due to different measurement units, some variables may influence inappropriately the certain processes. The standard deviation of a $ regression  m m 1 1 vector b ∈ R is s(b) = m−1 bi −˜b , where ˜b = m i=1 bi . To scale a matrix

we need to replace each column bj by

1 bj . s(bj )

(7) Prove that the centering matrix Hn is symmetric and idempotent. (8) Let B ∈ Rm×n and y ∈ Rm the data used in linear regression. Suppose   B (m+n)×n ˆ √ ∈R and that B is centered and define the matrix B = λIn   y m+n ˆ = 0n ∈ R . Prove that the ordinary regression applied to this y data amounts to ridge regression. Solution: Starting from the objective function of the ordinary linear regression (as shown in equality (16.2), the objective function of the new optimization problem is ˆ −y ˆ 2 f (r) = Br

√ B B y r − 2(y |0n ) √ r + (y |0n ) = r (B  | λIn ) √ 0n λIn λIn = r B  Br − 2y Br + y y + λr r, which is precisely the objective function of the ridge regression. (9) Prove that if the set of columns of the matrix B is orthonormal, r∗ is the optimal estimation of r for the ridge regression and ˜r is the optimal estimation of r for the least square regression, then 1 r∗ . = ˜r∗ 1+λ

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Regression

page 923

923

(10) Let s = ((x1 , y1 ), . . . , (xm , ym )) be a sample and let F : Rn −→ R be the function defined by F (r) =

m 1  1 ln m i=1 1 + e−yi rxi

that evaluates the error of the logistic regression algorithm. Prove that 1  yi x i 1  = −yi xi L(−yi r xi ), x y r m i=1 1 + e i i m i=1 m

(∇F )(r) = −

m

where L is the logistic function. (11) Develop a gradient descent algorithm to approximate a minimum for the error function F introduced in Exercise 10. (12) The logistic regression can be extended to the classification into k classes C1 , . . . , Ck by extending equality (16.3) as P (Y P (Y P (Y ln P (Y

= 1|X = x) = r1 x = k|X = x) = 2|X = x) = r2 x = k|X = x) .. . P (Y = k − 1|X = x) ln = r2 x. P (Y = k|X = x) ln

Prove that

k i=1

P (Y = i|X = x) = 1.

(13) Suppose that for the sparsity pattern η ∗ (λ1 ) we have η ∗ (λ1 ) = 0n for some λ1 > 0. Prove that for all λ2 > 0 such that λ1 = λ2 we have η ∗ (λ2 ) = −ηη ∗ (λ1 ). Solution: Suppose that for λ2 > 0 such that λ1 = λ2 we would have η ∗ (λ2 ) = −ηη ∗ (λ1 ). Let J  = {j | 1  j  n, (ηη ∗ (λ1 ))j = 0}. Note that the matrix BJ  is of full rank. Therefore, the reduced problem minimize for ˜r ∈ R|J | 1 BJ ˜r − y22 + λ˜r1 , 2 where λ  0. involves a strictly convex function, hence the solution is well-defined. Observe that r˜∗ (λ1 ) = (r∗ (λ1 ))J  and r˜∗ (λ2 ) = (r∗ (λ2 ))J  .

May 2, 2018 11:28

924

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 924

Mathematical Analysis for Machine Learning and Data Mining

Theorem 16.4 implies that ˜r∗ (0) = (BJ  BJ  )−1 BJ  y =

λ2 λ1 ˜r∗ (λ1 ) + ˜r∗ (λ2 ). λ1 + λ2 λ1 + λ2

Since the signs of ˜r∗ (λ1 ) and ˜ r∗ (λ2 ) are opposite and non-zero, we have ˜r∗ (0)1 < ˜r∗ (λ1 )1 . It is easy to see that the function that maps λ to ˜r∗ (λ) is non-increasing, we have a contradiction. (14) By using Supplement 13 prove that the number of linear segments on n the regularization path is less than 3 2+1 .

Bibliographical Comments The main sources for the lasso regression are [131, 136, 74, 75]; our presentation follows [103]. Supplement 8 originates in [75] and is inspired by [1], where the idea of adding artificial data that satisfies model constraints was developed.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 925

Chapter 17

Support Vector Machines

17.1

Introduction

Support vector machines (SVMs) were introduced in statistical learning theory by V. N. Vapnik1 and his collaborators in a series of publications [21, 34]. One of the earliest success stories of SVMs was the handwritten digit recognition. The results obtained with SVMs show superior classification performance comparable with the best classifiers developed in machine learning. Although intended initially for classifying data where classes are linearly separable, using techniques from functional analysis, SVMs manage to successfully classify data where classes are separated by nonlinear boundaries. 17.2

Linearly Separable Data Sets

In Chapter 1 we saw that the distance between a point x0 ∈ Rn and a hyperplane Hw,a defined by the equation w x = a is given by equality (2.14): d(Hw,a , x0 ) =

|w x0 − a| . w

1 V. N. Vapnik was born on December 6, 1936. He is the co-inventor of support vector machines and the creator of the Vapnik-Chervonenkis dimension theory. V. N. Vapnik received his Ph.D. in statistics at the Institute of Control Sciences, Moscow in 1964 and moved to USA at the end of 1990. He is a Professor of Computer Science at the University of London since 1995, as well as a Professor of Computer Science at Columbia University.

925

May 2, 2018 11:28

926

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 926

Mathematical Analysis for Machine Learning and Data Mining

x0 x − x0 = λw w x Hw,a Fig. 17.1

The distance between x0 and the hyperplane Hw,a .

A data sample of size m is a sequence s = ((x1 , y1 ), . . . , (xm , ym )), where x1 , . . . , xm belong to Rn and yi ∈ {−1, 1} for 1  i  m. The positive examples of s are those pairs (xi , yi ) such that yi = 1; the remaining pairs are the negative examples. The task of a linear classifier is to construct a hyperplane Hw,a starting from the sample s such that for each positive example (xi , 1) we have xi ∈ >0 0, this implies γ = 0. In turn, this implies m 

ui = 1,

(17.5)

i=1

uj =

m 

ui xij , for 1  j  n,

(17.6)

i=1

uj = C − vj .

(17.7)

Using equality (17.5) in equality (17.6) implies that a is a convex combination of x1 , . . . , xm . Using equalities (17.5)-(17.7) the dual objective function is g(u, v, γ) = r+C = r+C = r+C

m  i=1 m  i=1 m  i=1

ξi + ξi + ξi +

m  i=1 m  i=1 m 

ui (xi − a2 − r − ξi ) − ui xi − a2 − r ui xi − a2 − r

i=1

m  i=1 m 

ui − ui −

i=1

m  i=1 m  i=1 m 

vi ξi − γr ui ξi − ui ξi −

m  i=1 m 

i=1

vi ξi − γr (C − ui )ξi − γr

i=1

(because vi = C − ui ) m   = ui xi − a2 + r − r ui − rγ i=1

Taking into account the KKT conditions (which imply, as we saw γ = 0 m and i=1 ui = 1) the objective function of the dual problem is: g(u, v, γ) = inf L(r, a, ξ , u, v, γ) =

ξ r,a,ξ m 

ui xi − a2 =

i=1

= =

m  i=1 m  i=1

m 

ui (xi − a) (xi − a)

i=1

ui xi xi − 2 ui xi xi −

m 

ui xi a + a a

i=1 m  m  i=1 k=1

m  i=1

ui uk (xi , xk ).

ui

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Support Vector Machines

17.5

b3234-main

page 939

939

Perceptrons

For support vector machines the training set part of the data set to be classified is presented to the algorithm, the classification function is inferred, and then the algorithm is tested on the test set part of the data set. The perceptron constructs the classification function contemporaneously with the analysis of the training set; this exemplifies the paradigm of on-line learning. The terminology is the same as the one used for support vector machines. A training set is a sequence of pairs S = ((x1 , y1 ), . . . , (x , y )), where (xi , yi ) ∈ Rn × {−1, 1} for 1  i  n. If y = 1, x is a positive example; if y = −1, x is a negative example. The sequence S is linearly separable if there exists a hyperplane w∗ x + b∗ = 0 such that w∗ xi + b∗  0 if yi = 1 and w∗ xi + b∗ < 0 if yi = −1. Both cases are captured by the inequality γi = yi (w∗ xi + b∗ )  0. The number γi is the functional margin of (xi , yi ). If γi > 0 then (xi , yi ) is classified correctly; otherwise, it is incorrectly classified and we say that a mistake occurred. Without loss of generality we may assume that w∗  = 1; if this is not the case, the coefficients of the hyperplane w∗ x + b∗ = 0 may be rescaled. Also, we may assume that there exists γ > 0 such that yi (w∗ xi + b∗ )  γ.

(17.8)

The algorithm builds a sequence of weight vectors (wk ) and a sequence of bias values (bk ). Upon examining the first m − 1 training examples (x1 , y1 ), . . . , (xm−1 , ym−1 ) and making the predictions y1 , . . . , ym−1 (some of which may be erroneous, in which cases modification are applied to parameters maintained by the algorithm), the algorithm is presented with the input xm . Asumming that at that moment the parameters of the algorithm are wk and bk , an error is committed if yi (wk xm + bk ) < 0. In this case, a correction of the parameters of the algorithm is applied; otherwise, the algorithm continues by analyzing the next example. The processing of the sequence of pairs ((x1 , y1 ), . . . , (x , y )) is referred to as an epoch of the algorithm. Let R be the minimum radius of a closed ball centered in 0, that contains all points xi , that is R = max{xi  | 1  i  } and let η be a parameter called the learning rate.

May 2, 2018 11:28

940

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 940

Mathematical Analysis for Machine Learning and Data Mining

If a correction is applied, the new weight vector will be wk+1 = wk + ηyi xi , while the new bias value will be bk+1 = bk + ηyi R2 . In other words, the correction of the weight vector is Δwk = wk+1 − wk = ηyi xi and the correction of the bias is Δbk = ηyi R2 . Algorithm 17.5.1: Learning Algorithm for Perceptron Data: labelled training sequence S of length and learning rate η Result: weight vector w and parameter b 1 initialize w0 = 0, b0 = 0, k = 0; 2 R = max{xi  | 1  i  }; 3 repeat 4 for i = 1 to do 5 if yi (wk xi + bk ) < 0 then 6 wk+1 ← wk + ηyi xi ; 7 bk+1 ← bk + ηyi R2 ; 8 k ← k + 1; 9 end 10 end 11 until no mistakes are made in the for loop; 12 return k, w∗ = wk and b∗ = bk where k is the number of mistakes; Note that, in principle, the algorithm may go through the sequence S cyclically, until no mistakes are made. If S is not linearly separable the algorithm will cycle indefinitely. Theorem 17.2. Let S = ((x1 , y1 ), . . . , (x , y )) be a training sequence that is linearly separable by the hyperplane w∗ x+b∗ = 0, and let R = max{xi  | 1  i  }. The number of mistakes made by the algorithm is at most 2  2R . γ Proof. As we noted before, we may assume that w∗  = 1 Let k be the update counter and let wk be the weight vector when the algorithm makes error k on example # $ xi . Then, w #k+1$ = wk + ηyi#xi $and bk+1 = bk + ηyi R. wk w ˜ ∗ = b∗∗ , and x˜i = xRi . ˜ k = bk , and w Let w R

R

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Support Vector Machines

page 941

941

Note that ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ wk + ηyi xi wk wk+1 ˜i = w ˜i. ˜ k+1 = ⎝ bk+1 ⎠ = ⎝ bk + ηyi R2 ⎠ = ⎝ bk ⎠ + ηyi x ˜ k + ηyi x w R R R ˜ ∗ x ˜ i = yi (w∗ xi + b∗ )  γ, it follows that Since yi w ˜ k+1 = w ˜ k + ηyi w ˜ ∗ wk  w ˜ k + ηγ. ˜ ∗w ˜ ∗w ˜ ∗w w ˜ ∗ wk  kηγ. By repeated application of the above equality we obtain w th ˜ i . This ˜ k+1 = w ˜ k + ηyi x If the k error occurs on input xi we have w implies ˜ k+1 = (w ˜ i )(w ˜i) ˜ k+1 w ˜ k + ηyi x ˜ k + ηyi x ˜ k+1 2 = w w ˜ k x ˜ i + η 2 ˜ ˜ k 2 + 2ηyi w xi 2 = w ˜ k xi < 0 when an error occurs and yi2 = 1) (because yi w ˜ k 2 + η 2 ˜ xi 2  w ˜ k 2 + η 2 (xi 2 + R2 )  w ˜ k 2 + 2η 2 R2 .  w

√ ˜ k 2  2kη 2 R2 , hence w ˜ k   ηR 2k. By combining the This implies w equalities √ ˜ k   ηR 2k ˜ ∗ wk  kηγ and w w we obtain

√ ˜ k  kηγ, ˜ ∗ ηR 2k  w ˜ ∗  · w ˜ k  w ˜ ∗ w w

which imply

 k2

  2 R2 2R 2 ˜ ∗  w γ γ

˜ ∗ 2  w∗ 2 + 1 = 2. because b∗  R, hence w



Exercises and Supplements Linearly Separable Data Sets Let Bn = {0, 1}n and let K ⊆ Bn . The sequence of Chow parameters of K is chow(K) = (c1 , . . . , cn , cK ) ∈ Nn defined as cK = |K| and ci = |{x ∈ K | xi = 1}|. For example, for n = 4 and K = {(0, 1, 1, 1), (0, 0, 1, 1), (0, 0, 0, 1)} we have chow(K) = (0, 1, 2, 3, 3). Two subsets K, G of Bn are equipollent if they have the same Chow parameters.

May 2, 2018 11:28

942

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 942

Mathematical Analysis for Machine Learning and Data Mining

The subsets K and Bn − K are linearly separable if there exists a pair (w, t) ∈

Rn × R such that

K = {x ∈ Bn | w x  t} and Bn − K = {x ∈ Bn | w x < t}. We say that K is linearly separable if K and Bn − K are linearly separable. The vector (w, t) is the separating weight for K. (1) Let K ⊆ Bn . Prove that chow(K) = (

 x∈K

x, |K|).

such that u = 1n − v. Prove that (2) A diagonal of Bn is a pair (u, v) ∈ if K is a linearly separable subset of Bn that contains a diagonal of Bn , then it contains a point of every other diagonal of Bn . Solution: Since K is linearly separable, there exist w, t such that K = {x ∈ Bn | w x  t}. If the diagonal (u, v) is contained in K we have both w u  t and w v = w 1n − w u  t, hence u (u + v) = w 1n  2t. Let (y, z) be another diagonal and suppose that neither y nor z belong to K. Then, w y < t and w z = w (1n − y) < t, which implies w 1n < 2t, thereby contradicting the previous inequality. Therefore, at least one of y, z must belong to K. Bn2

(3) Let K = {v1 , . . . , vm } be a subset of Bn and let Bn − K = {vm+1 , . . . , v2n }. Prove that if K is linearly separable then mfor any non-negative numbers ai where 1  i  2n the equalities i=1 ai = m 2n 2n n i=m+1 ai and i=m+1 ai vi imply ai = 0 for 1  i  2 . i=1 ai vi = Solution: Suppose that K is linearly separable. There exists w and t such that w vi  t for 1  i  m and w vi < t for m + 1  i  2n , hence w

m 

ai vi =

i=1

m i=1

t>

ai =

w

m 

n

2 

a i w v i < t

i=m+1

2n i=m+1

ai , and

i=1

n

ai vi =

i=m+1

Suppose that

ai w vi  t

i=1

n

2 

w

m 

2 

ai .

i=m+1

ai > 0. Then, we have both

2n

i=m+1 ai vi 2n i=m+1 ai

=

w

m i=1 ai vi m  t, i=1 ai

2n m which is contradictory. Therefore, i=1 ai = i=m+1 ai = 0 and this implies ai = 0 for 1  i  2n because the numbers ai are non-negative. A subset K of Bn is k-summable for k  2 if there exists j such that 2  j  k such distinct) and that there exist j points u1 , . . . , uj in K (not necessarily  j points v1 , . . . , vj in Bn − K (not necessarily distinct) such that ji=1 ui = ji=1 vi . K is summable if it is k-summable for some k  2.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Support Vector Machines

page 943

943

If K is not k-summable, then K is said to be k-assumable. K is said to be summable if it is k-summable for some k  2; otherwise K is said to be assumable.

(4) Let K be a linearly separable subset of Bn by the separating weight (wt ). Suppose that chow(K) = (c1 , . . . , cn , cK ). Prove that (a) if ci < c2K , then wi < 0; (b) if ci > c2K , then wi > 0; (c) if ci = c2K , then (x1 , . . . , xi−1 , 0, xi+1 , . . . , xn ) ∈ K if and only if (x1 , . . . , xi−1 , 1, xi+1 , . . . , xn ) ∈ K; (d) if ci > cj then wi > wj ; (e) if ci = cj , then there exists a separating weight (w, t) such that wi = wj . (5) If K is a separable subset of Bn , ci < u  v, prove that u ∈ K.

cK 2

for 1  i  n, v ∈ K and

(6) Prove that if a set K, K ⊆ Bn is linearly separable, then it is assumable. Solution: Suppose that (w, t) is a separating weight for K and there in K and  u1 , . . . , uk ∈ Bn − K. Then, exist k elements, v1 , . . . , vk  w ui < t and t  w v, hence ki=1 ui < ki=1 vi which means that K is assumable. (7) Prove that if K is not linearly separable, then K is summable. Solution: Suppose that K is not linearly separable, where K = {u1 , . . . , um } and Bn − K = {um+1 , . . . , u2n }. By standard separation results the convex sets Kconv (K)∩Kconv (Bn −K) = ∅ have a non-empty intersection. Therefore, the following system m  i=1 m  i=1

n

ai ui = ai =

2 

aj u j ,

j=m+1 2n 

aj = 1,

j=m+1

ai  0, bj  0 for 1  i  m and m + 1  j  2n has a feasible solution with rational values because the system has rational coefficients. Thus, for some positive integer p all components of the vector (pa1 , . . . , pa2n ) are non-negative integers and m  i=1 m  i=1

n

(pai )ui = (pai ) =

2 

j=m+1 2n 

(paj )uj ,

(paj ) = p,

j=m+1

May 2, 2018 11:28

944

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 944

Mathematical Analysis for Machine Learning and Data Mining

hence K is summable (by considering pai copies of ui and paj copies of uj ). (8) Prove that for two equipollent subsets K, G of Bn , either both are linearly separable and K = G, or neither is linearly separable; in other words, prove that there is at most one linearly separable set for any set of Chow parameters. Solution: Let K, G be equipollent sets in Bn . If K ⊆ G or G ⊆ K, then cK = cG imply K = G. If K ⊆ G and G ⊆ K and |K| = |G|, then |K ∩ (Bn − G)| = |K| − |K ∩ G| = |G| − |G ∩ K| = |(Bn − K) ∩ G|. Also, 

x=

x∈K∩Bn −G



x−

x∈K

 x∈K∩G

= cG −



x=

x∈K∩G



=

x = cK −  x∈G



x

x∈K∩G

−



x

x∈K∩G

x.

x∈(Bn −K)∩G

Therefore, K and G are |K ∩ (Bn − G)|-summable, hence they are not linearly separable. (9) Prove that if two equipollent subsets of Bn are distinct, then neither is linearly separable. (10) The optimization problem of the separable data case that seeks to determine a separating hyperplane in Rn can be transformed into an equivalent optimization problem in Rn+1 that seeks to identify a separating subspace. Given a data set s = ((x1 , y1 ), . . . , (xm , ym )) prove that there exists r ∈ Rn such that s is separable by a hyperplane if and only if the set ˜s = ((x1 + r, y1 ), . . . , (xm + r, ym )) is separable be a subspace M of Rn . (11) Apply the Karush-Kuhn-Tucker Theorem (Theorem 13.15) to the optimization problem in the case of a separable data set. Prove that a constraint ci (w, a) = yi (w xi − a) is active if and only if xi is a support vector, and show that the optimal value of w is a linear combination of support vectors. SVM — The Non-separable Case (12) Consider the data set D in R2 shown in Figure 17.5, where C is a circle centered in (6, 4) having radius 3. Define a transformation φ : R2 −→ R2 such that φ(D) is linearly separable. (13) The hinge function h was defined in Exercise 2 of Chapter 12 as h(x) = max{0, 1 − x}. Let Lh : Rn × R −→ R be the function given

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

Support Vector Machines

b3234-main

page 945

945

C

Fig. 17.5

Non-linearly separable data.

  by Lh (w, a) = m i=1 h(yi (w xi − a)). Prove that the optimization problem for the soft SVM non-separable case is equivalent to minimizing the function 1 f (w, a) = w2 + CLh (w, a). 2 Solution: The defining condition for the slack variables in the soft SVM case is ξi  1−yi (w xi −a). Since these variables are non-negative, the smallest value of these variables are h(1 − yi (w xi − a)). Thus, the total slack is at least Lh (w, a). This equivalent form of the optimization problem for soft SVMs has the advantage to be a regularized form. Non-linear Support Vector Machines (14) Let S be a non-empty set, H1 , H2 be Hilbert spaces, and K : S ×S −→ R be a kernel. Suppose that φ1 : S −→ H1 and φ2 : S −→ H2 are both feature maps K. Prove that for all w1 ∈ H1 there exists w2 ∈ H2 with w2   w1  and (w1 , φ1 (x)) = (w2 , φ2 (x)) for all x ∈ S. Solution: Let U1 = K(φ1 (S)) and let U1⊥ be its orthogonal comple⊥ ment in H1 . If w1 ∈ H1 we have w1 = u1 + u⊥ 1 with u1 ∈ U1 and u1 ∈ ⊥ ⊥ U1 . Given x ∈ S we have (u1 , φ1 (x)) = 0 and, therefore, (w1 , φ1 (x)) = (u1 , φ1 (x)) for all x ∈ S. The definition of U 1 implies the existence mn  such that zn = of a sequence (z n ) in  φ1 (S) m=1 anm φ1 (xnm ) and ∞ mn u1 = n=1 zn . Then, for z˜n = m=1 anm φ2 (xnm ) and 2  1  1 we have

2





 2 mi 2 mn    



2





z = amn aji (φ1 (xmn ), φ1 (xji )) n





n=1

n=1 m=1 i=1 j=1

2





2



amn aji (φ2 (xmn ), φ2 (xji )) =



z˜n

. =

n=1

n=1 m=1 i=1 j=1 2 2 mi mn    

May 2, 2018 11:28

946

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 946

Mathematical Analysis for Machine Learning and Data Mining

m  Therefore, ˜n is a Cauchy sequence, hence it converges to w2 = n=1 z  ∞ ˜n ∈ H2 . We have w2  = u1   w1 ; also, (w1 , φ1 (x)) = n=1 z (w2 , φ2 (x)) for all x ∈ S. (15) Let (S, d) be a metric space, H a real Hilbert space, K : S × S −→ H and let φ : S −→ H be a feature map of K. Prove that: (a) K is continuous if and only if φ is continuous; (b) if K is continuous, the mapping dK : S × S −→ R0 defined by dK (x, y) = φ(x) − φ(y) for x, y ∈ S is a semimetric on S and the identity mapping 1S : (S, d) −→ (S, dK ) is continuous; (c) if φ is injective, then dK is a metric. Solution: Since dK (x, y)2 = (φ(x) − φ(y), φ(x) − φ(y)) = (φ(x), φ(x)) − 2(φ(x), φ(y)) + (φ(y), φ(y)) = K(x, x) − 2K(x, y) + K(y, y), it follows that dK is continuous. This implies that {y ∈ S | dK (x, y) < } is open and therefore 1S is continuous. Furthermore, φ : (S, dK ) −→ H is continuous, hence φ regarded as a mapping from (S, d) to H is also continuous. Conversely, assume that φ is continuous. Since |K(x, y) − K(x , y  )|  |(φ(x), φ(y) − φ(y  ))| + |(φ(x) − φ(x ), φ(y  ))|  φ(x) · φ(y) − φ(y  ) + φ(y  )φ(x) − φ(y), it follows that K is also continuous. Perceptrons (16) There are 16 functions of the form f : {0, 1}2 −→ {0, 1}. For each such function consider the sequence Sf = ((x1 , y1 ), . . . , (x4 , y4 )), where xi ∈ {0, 1}2 and  −1 if f (xi ) = 0, yi = 1 if f (xi ) = 1 for 1  i  4. (a) For how many of these functions is Sf linearly separable? (b) Note that for f : {0, 1}2 −→ {0, 1} defined by f (x) = min{x1 , x2 } is linearly separable. For η = 0.1 and η = 0.8 draw the sequence of weights of the perceptron during the learning process. (17) Provide a robust implementation of the perceptron algorithm that will cycle through a prescribed number of iterations (and, thus, will cope with the case when data is not linearly separable).

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Support Vector Machines

9in x 6in

b3234-main

page 947

947

Bibliographical Comments Support vector data description was introduced in [128, 129]. See also [28, 91] for related results. The perceptron algorithm is due to F. Rosenblatt [112]. There are several variants of this algorithm in the literature [108, 59, 36]. Chow parameters were introduced in [31, 30]. Exercises 1-9 contain results of from [51, 84]. See also [70]. A comprehensive reference for applications of Boolean concepts is [35]. There is a large body of reference works on support vector machines. We suggest [36, 124] as main references for the study of these algorithms, and [119] and [96] for kernelization techniques, as well as the excellent tutorial [25].

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 949

Bibliography

[1] Abu-Mostafa, Y. S. (1995). Hints, Neural Computation 7, 4, pp. 639–671. [2] Aliprantis, C. D. and Border, K. C. (2006). Infinite Dimensional Analysis, 3rd edn. (Springer, Berlin). [3] Ash, R. B. (1972). Real Analysis and Probability (Academic Press, New York). [4] Axler, S. J. (1995). Down with determinants! American Mathematical Monthly 102, pp. 139–154. [5] Axler, S. J. (2015). Linear Algebra Done Right, 3rd edn. (Springer, Heidelberg). [6] Baldi, P. and Hornik, K. (1995). Learning in linear neural networks: a survey, IEEE Transactions on Neural Networks 6, 4, pp. 837–858. [7] Banach, S. (1963). Th´eorie des op´erations lin´eaires, 2nd edn. (Chelsea Pub. Co., New York). [8] Barbu, V. and Precupanu, T. (2012). Convexity and Optimization in Banach Spaces, 4th edn. (Springer, Dordrecht). [9] Bazaraa, M. S., Sherali, H. D. and Shetty, C. M. (2006). Nonlinear Programming — Theory and Algorithms, 3rd edn. (Wiley-Interscience, Hoboken, NJ). [10] Berlinet, A. and Thomas-Agnan, C. (2004). Reproducing Kernel Hilbert Spaces in Probability and Statistics (Kluwer Academic Publishers, Boston). [11] Berry, A. C. (1931). A metric for the space of measurable functions, Proceedings of the National Academy of Sciences 17, pp. 456–459. [12] Bertsekas, D. M. (1982). Constrained Optimization and Lagrange Multiplier Methods (Academic Press, New York). [13] Bertsekas, D. M. (1999). Nonlinear Programming, 2nd edn. (Athena Scientific, Belmont, MA). [14] Bertsekas, D. M. (2015). Convex Optimization Algorithms (Athena Scientific, Belmont, MA). [15] Bertsekas, D. P., Nedi´c, A. and Ozdaglar, A. E. (2003). Convex Analysis and Optimization (Athena Scientific, Cambridge, MA). [16] Billingsley, P. (2012). Probability and Measure, anniversary edn. (Wiley, Hoboken, NJ).

949

May 2, 2018 11:28

950

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 950

Mathematical Analysis for Machine Learning and Data Mining

[17] Bohnenblust, H. F., Karlin, S. and Shapley, L. S. (1950). Solution of discrete, two-person games, in Contribution to the Theory of Games, Vol. 1, Annals of Mathematics Studies, Vol. 24, pp. 51–72. [18] Borvinok, A. (2002). A Course in Convexity (American Mathematical Society, Providence, RI). [19] Borwein, J. M. (1982). A note on the existence of subgradients, Mathematical Programming 24, pp. 225–228. [20] Borwein, J. M. and Lewis, A. S. (2006). Convex Analysis and Nonlinear Optimization, 2nd edn. (Springer, Canadian Mathematical Society). [21] Boser, B. E., Guyon, I. and Vapnik, V. (1992). A training algorithm for optimal margin classifiers, in Proceedings of the Fifth Annual ACM Conference on Computational Learning Theory, COLT 1992, Pittsburgh, PA, USA, July 27-29, 1992, pp. 144–152. [22] Bourbaki, N. (1987). Topological Vector Spaces (Springer, Berlin). [23] Boyd, S. and Vandenberghe, L. (2004). Convex Optimization (Cambridge University Press, Cambridge, UK). [24] Broyden, C. G. (1965). A class of methods for solving nonlinear simultaneous equations, Mathematics of Computation 19, pp. 577–593. [25] Burges, C. J. C. (1998). A tutorial on support vector machines for pattern recognition, Data Mining and Knowledge Discovery 2, 2, pp. 121– 167. [26] Carothers, N. (2005). A Short Course on Banach Space Theory (Cambridge University, Cambridge, UK). [27] Carver, W. B. (1922). Systems of linear inequalities, Annals of Mathematics 23, pp. 212–220. [28] Chen, G., Zhang, X., Wang, Z. J. and Li, F. (2015). Robust support vector data description for outlier detection with noise or uncertain data, Knowledge-Based Systems 90, pp. 129–137. [29] Cheney, W. (2001). Analysis for Applied Mathematics (Springer, New York). [30] Chow, C. K. (1961). Boolean functions realizable with single threshold functions, Proceedings of IRE 49, pp. 370–371. [31] Chow, C. K. (1961). On the characterization of threshold functions, in Proceedings of the Second Annual Symposium on Switching Theory and Logical Design, pp. 34–38. [32] Cohn, D. J. (1997). Measure Theory (Birkh¨ auser, Boston). [33] Cohn, P. M. (1981). Universal Algebra (D. Reidel, Dordrecht). [34] Cortes, C. and Vapnik, V. (1995). Support-vector networks, Machine Learning 20, 3, pp. 273–297. [35] Crama, Y. and Hammer, P. L. (2011). Boolean Functions — Theory, Algorithms and Applications (Cambridge University Press, Cambridge, UK). [36] Cristianini, N. and Shawe-Taylor, J. (2000). Support Vector Machines (Cambridge, Cambridge, UK). [37] Cybenko, G. (1989). Approximation by superposition of sigmoidal functions, Mathematics of Control, Signals, and Systems 2, pp. 303–314.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Bibliography

9in x 6in

b3234-main

page 951

951

[38] Cybenko, G. (1992). Correction: Approximation by superposition of sigmoidal functions, Mathematics of Control, Signals, and Systems 5, p. 445. [39] Danskin, J. M. (1966). The theory of max-min, with applications, SIAM Journal on Applied Mathematics 14, 4, pp. 641–664. [40] Diestel, J. (1975). Geometry of Banach Spaces: Selected Topics (Springer Verlag, New York). [41] Dieudonn´e, J. (1969). Foundations of Modern Analysis (Academic Press, New York). [42] Dixmier, J. (1984). General Topology (Springer-Verlag, New York). [43] Edgar, G. (1990). Measure, Topology, and Fractal Geometry (SpringerVerlag, New York). [44] Taylor, J. G. (1993). Mathematical Approaches to Neural Networks (NorthHolland, Amsterdam). [45] Efron, B., Hastie, T., Johnstone, I. and Tibshirani, R. (2004). Least angle regression, Annals of Statistics 32, pp. 407–499. [46] Eidelheit, M. (1936). Zur Theorie der konvexen Mengen in linear und normierten R¨ aumen, Studia Mathematica 6, pp. 104–111. [47] Eisenberg, M. (1974). Topology (Holt, Rinehart and Winston, Inc., New York). [48] Ekeland, I. (1974). On the variational principle, Journal of Mathematical Analysis and Applications 47, pp. 324–353. [49] Ekeland, I. (1979). Nonconvex minimization problems, Bulletin of American Mathematical Society 3, pp. 443–474. [50] Ekeland, I. and Lebourg, G. (1976). Generic fr´echet-differentiability and perturbed optimization problems in banach spaces, Transactions of the American Mathematical Society 224, pp. 193–216. [51] Elgot, C. C. (1961). Truth functions realizable by single threshold engine, in Proceedings of IEEE Symposium on Switching Theory and Logical Design, pp. 225–245. [52] Emch, A. (1922). A model for the Peano surface, The American Mathematical Monthly 29, pp. 388–391. [53] Engelking, R. and Siekluchi, K. (1992). Topology — A Geometric Approach (Heldermann Verlag, Berlin). [54] Fabian, M., Habala, P., H` ajek, P., Santalucia, V. M., Pelant, J. and Zizler, V. (2001). Functional Analysis and Infinite-Dimensional Geometry (Springer, NY). [55] Fan, K., Glicksberg, I. and Hoffman, A. J. (1957). Systems of inequalities involving convex functions, Proceedings of the American Mathematical Society 8, pp. 617–622. [56] Fejer, P. A. and Simovici, D. A. (1991). Mathematical Foundations of Computer Science, Vol. 1 (Springer-Verlag, New York). [57] Folland, G. B. (1999). Real Analysis — Modern Techniques and Their Applications, 2nd edn. (Wiley Interscience, New York). [58] Fr´echet, M. (1906). Sur quelques points du calcul fonctionnel, Rendiconti del Circolo Matematico di Palermo 22, pp. 1–47.

May 2, 2018 11:28

952

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 952

Mathematical Analysis for Machine Learning and Data Mining

[59] Freund, Y. and Shapire, R. E. (1999). Large margin classification using the perceptron algorithm, Machine Learning 37, pp. 277–296. [60] Funahashi, S. (1989). On the approximate realization of continuous mapping by neural networks, Neural Networks 2, pp. 183–192. [61] Furstenberg, H. (1955). On the infinitude of primes, American Mathematical Monthly 62, p. 353. [62] Gallant, A. R. and White, H. (1988). There exists a neural network that does not make avoidable mistakes, in Proceedings of IEEE Second International Conference on Neural Networks (University of California Press, San Diego, CA), pp. 657–664. [63] Garling, D. J. H. (2014). A Course in Mathematical Analysis — Volume III: Complex Analysis, Measure and Integration (Cambridge University, Cambridge, UK). [64] Genocchi, A. and Peano, G. (1884). Calcolo differenziale e principii di calcolo integrale (Fratelli Bocca, Roma, Italy). [65] Graves, L. M. (1950). Some mapping theorems, Duke Mathematical Journal 17, pp. 111–114. [66] Greenberg, H. J. and Pierskalla, W. P. (1971). A review of quasi-convex functions, Operations Research 19, pp. 1553–1570. [67] Greub, W. (1981). Linear Algebra, 4th edn. (Springer-Verlag, New York). [68] Griffel, D. H. (2002). Applied Functional Analysis (Dover Publications, Mineola, NY). [69] Gr¨ unbaum, B. (1967). Convex Polytopes (Wiley-Interscience, London). [70] Gruzling, N. (2006). Linear separability of the vertices of an n-dimensional hypercube, Master’s thesis, University of Northern British Columbia. [71] G¨ uler, O. (2010). Foundations of Optimization (Springer, New York). [72] Halmos, P. R. (1951). Introduction to Hilbert Spaces and the Theory of Spectral Multiplicity (Chelsea Publishing Company, New York). [73] Hardy, G. H., Littlewood, J. E. and Polya, G. (1929). Some simple inequalities satisfied by convex functions, Messenger Mathematics 58, pp. 145–152. [74] Hastie, T., Rosset, S., Tibshirani, R. and Zhu, J. (2004). The entire regularization path for the support vector machine, in Advances in Neural Information Processing Systems 17 [Neural Information Processing Systems, NIPS 2004, December 13-18, 2004, Vancouver, British Columbia, Canada], pp. 561–568. [75] Hastie, T., Tibshirani, R. and Friedman, J. H. (2003). Note on “comparison of model selection for regression” by vladimir cherkassky and yunqian ma, Neural Computation 15, 7, pp. 1477–1480. [76] Haykin, S. (2008). Neural Networks and Learning Machines, 3rd edn. (Prentice-Hall, New York). [77] Hecht-Nielsen, R. (1991). Neurocomputing (Addison-Wesley, Reading, MA). [78] Hestenes, M. E. and Stiefel, E. (1952). Methods of conjugate gradients for solving linear systems, Journal of Research of the National Bureau of Standards 49, pp. 409–436.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Bibliography

9in x 6in

b3234-main

page 953

953

[79] Hiriart-Urruty, J. B. (1983). A short proof of the variational principle for approximate solution of a minimization problem, American Mathematical Monthly 90, pp. 206–207. [80] Hoerl, A. E. and Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems, Technometrics 12, pp. 55–67. [81] Hoffman, T., Sch¨ olkopf, B. and Smola, A. J. (2008). Kernel methods in machine learning, Annals of Statistics 36, pp. 1171–1220. [82] Hornik, K. (1991). Approximation capabilities of multilayer feedforward networks, Neural Networks 4, pp. 251–257. [83] Hornik, K., Stinchcombe, M. B. and White, H. (1989). Multilayer feedforward networks are universal approximators, Neural Networks 2, 5, pp. 359–366. [84] Hu, S. T. (1965). Threshold Logic (University of California Press, Berkeley). [85] Jahn, J. (2007). Introduction to the Theory of Nonlinear Optimization, 3rd edn. (Springer, Berlin). [86] Kantorovich, L. V. and Akilov, G. P. (1982). Functional Analysis, 2nd edn. (Pergamon Press, Oxford, England). [87] Karamardian, S. (1967). Strictly quasi-convex (concave) functions and duality in mathematical programming, J. of Mathematical Analysis and Applications 20, pp. 344–358. [88] Karamata, J. (1932). Sur une in´egalit´e relative aux fonctions convexes, Publications Math´ematiques de l’Universit´e de Belgrade 1, pp. 145–148. [89] Kelley, J. L. (1955). General Topology (Van Nostrand, Princeton, NJ). [90] Kelley, J. L. and Namioka, I. (1963). Linear Topological Spaces (Van Nostrand, Princeton). [91] Kim, S., Choi, Y. and Lee, M. (2015). Deep learning with support vector data description, Neurocomputing 165, pp. 111–117. [92] Korn, F., Pagel, B.-U. and Faloutsos, C. (2001). On the “dimensionality curse” and the “self similarity blessing”, IEEE Transactions on Knowledge and Data Engineering 13, pp. 96–111. [93] Kostrikin, A. I. and Manin, Y. I. (1989). Linear Algebra and Geometry (Algebra, Logic and Applications) (Gordon and Breach Science Publishers, New York). [94] Kranz, S. G. and Parks, H. P. (2003). The Implicit Function Theorem — History, Theory, and Applications (Birkh¨ auser, Boston). [95] Kuhn, H. W. and Tucker, A. W. (1950). Nonlinear programming, in Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability (University of California Press), pp. 481–492. [96] Kung, S. Y. (2014). Kernel Methods and Machine Learning (Cambridge University Press, Cambridge, UK). [97] Lauritzen, N. (2013). Undergraduate Convexity (World Scientific, New Jersey). [98] Lax, P. D. (2007). Linear Algebra and Its Applications (WileyInternational, Hoboken, NJ). [99] Lay, S. R. (1982). Convex Sets and Their Applications (Wiley, New York).

May 2, 2018 11:28

954

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 954

Mathematical Analysis for Machine Learning and Data Mining

[100] Leshno, M., Lin, V. Y., Pinkus, A. and Schocken, S. (1993). Multilayer feedforward networks with a nonpolynomial activation function can approximate any function, Neural Networks 6, 6, pp. 861–867. [101] Lindenstrauss, J. and Tzafriri, L. (1973). Classical Banach Spaces (Springer Verlag, New York). [102] Luenberger, D. G. (1969). Optimization by Vector Space Methods (Wiley, New York). [103] Mairal, J. and Yu, B. (2012). Complexity analysis of the lasso regularization path, in Proceedings of the 29th International Conference on Machine Learning, ICML 2012, Edinburgh, Scotland, UK, June 26–July 1, 2012. [104] Mangasarian, O. L. (1965). Pseudo-convex functions, SIAM Journal on Control 3, pp. 281–290. [105] Mangasarian, O. L. (1969). Nonlinear Programming (McGraw-Hill, New York). [106] McShane, E. T. (1973). The Lagrange multiplier rule, The American Mathematical Monthly 80, pp. 922–925. [107] Niculescu, C. and Persson, L. E. (2006). Convex Functions and Their Applications — A Contemporary Approach (Springer, New York). [108] Novikoff, A. B. J. (1962). On convergence proofs on perceptrons, in Proceedings of the Symposium on Mathematical Theory of Automata (Polytechnic Institute of Brooklyn, Brooklyn, NY), pp. 615–622. [109] Pagel, B.-U., Korn, F. and Faloutsos, C. (2000). Deflating the dimensionality curse using multiple fractal dimensions, in International Conference on Data Engineering, pp. 589–598. [110] Pervin, W. J. (1964). Foundations of General Topology (Academic Press, New York). [111] Rockafellar, R. T. (1970). Convex Analysis (Princeton University Press, Princeton, NJ). [112] Rosenblatt, F. (1958). The perceptron: A probabilistic model for information storage and organization in the brain, Psychological Review 65, pp. 386–407. [113] Royden, H. L. (1988). Real Analysis, 3rd edn. (Prentice-Hall, Englewood Cliffs, NJ). [114] Rudin, W. (1986). Real and Complex Analysis, 3rd edn. (McGraw-Hill, New York). [115] Schneider, R. (1993). Convex Bodies: The Brun-Minkowski Theory (Cambridge University Press, Cambridge, UK). [116] Shafarevich, I. R. and Remizov, A. O. (2013). Linear Algebra and Geometry (Springer, Heidelberg). [117] Shalev-Shwartz, S. and Ben-David, S. (2016). Understanding Machine Learning — From Theory to Practice (Cambridge University Press, Cambridge, UK). [118] Shalev-Shwartz, S., Singer, Y. and Srebro, N. (2007). Pegasos: Primal Estimated sub-GrAdient solver for SVM, in Machine Learning, Proceedings of the Twenty-Fourth International Conference (ICML 2007), Corvallis, Oregon, USA, June 20-24, 2007, pp. 807–814.

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Bibliography

9in x 6in

b3234-main

page 955

955

[119] Shawe-Taylor, J. and Cristianini, N. (2004). Kernel Methods for Pattern Analysis (Cambridge, Cambridge, UK). [120] Simon, B. (2011). Convexity: An Analytic Viewpoint (Cambridge University Press, Cambridge, UK). [121] Simovici, D. A. (2012). Linear Algebra Tools for Data Mining (World Scientific, New Jersey). [122] Slater, M. (1950). Lagrange multipliers revisited, Cowles Foundation for Research in Economics at Yale University, Discussion Paper Mathematics 403, pp. 1–13. [123] Sokal, A. D. (2011). A really simple elementary proof of the uniform boundedness theorem, American Mathematical Monthly 118, pp. 450–452. [124] Steinwart, I. and Christman, A. (2008). Support Vector Machines (Springer, New York). [125] Stewart, J. (1976). Positive definite functions and generalizations, an historical survey, Rocky Mountain Journal of Mathematics 6, pp. 409–484. [126] Stoer, J. and Bulirsh, R. (2010). Introduction to Numerical Analysis, 3rd edn. (Springer, New York). [127] Sylvester, J. J. (1857). A question in the geometry of situation, Quarterly Journal in Pure and Applied Mathematics 1, p. 19. [128] Tax, D. M. J. and Duin, R. P. W. (1999). Support vector domain description, Pattern Recognition Letters 20, 11-13, pp. 1191–1199. [129] Tax, D. M. J. and Duin, R. P. W. (2004). Support vector data description, Machine Learning 54, 1, pp. 45–66. [130] Taylor, M. E. (2006). Measure Theory and Integration (American Mathematical Society, Providence, RI). [131] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society. Series B (Methodological) 58, 1, pp. 267–288. [132] Trefethen, L. N. and III, D. B. (1997). Numerical Linear Algebra (SIAM, Philadelphia). [133] Werner, J. (1984). Optimization — Theory and Applications (Vieweg, Braunschweig). [134] White, H., Gallant, A. R., Hornik, K., Stinchcombe, M. and Woolridge, J. (1992). Artificial Neural Networks — Approximation and Learning Theory (Blackwell, Oxford, UK). [135] Willard, S. (2004). General Topology (Dover, Mineola, New York). [136] Zhu, J., Rosset, S., Hastie, T. and Tibshirani, R. (2003). 1-Norm support vector machines, in Advances in Neural Information Processing Systems 16 [Neural Information Processing Systems, NIPS 2003, December 8-13, 2003, Vancouver and Whistler, British Columbia, Canada], pp. 49–56.

b2530   International Strategic Relations and China’s National Security: World at the Crossroads

This page intentionally left blank

b2530_FM.indd 6

01-Sep-16 11:03:06 AM

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 957

Index

A

barycentric coordinates, 144 Borel set, 388 Borel-Cantelli Lemma, 416 Fatou’s Lemma, 495

absolutely convex set, 138 absolutely summable sequence, 308, 313 absorbing set, 139, 140 accumulation point, 179 active constraint, 826, 836 additivity property of signed measures, 456 affine basis, 125 affine combination, 120 affine mapping, 120 affine subspace, 121 affinely dependent set, 123 algebra of sets, 34 almost everywhere property, 401 almost surely true statement, 469 angle between vectors, 99 antimonotonic mapping, 27 ascent direction, 820 assumable set, 942 asymptotic density of a set, 587 Axiom of Choice, 12

C canonical form of a hyperplane relative to a sample, 927 Carath´eodory outer measure, 453 carrier of a point in a simplex, 145 Cartesian product, 13 projection of a, 13 Cauchy sequence in measure, 553 Chain Rule for Radon-Nikodym Derivatives, 532 characteristic function of a subset, 10 Chow parameters, 941 closed function, 221 closed interval in Rn , 390 closed segment, 117 closed set, 165 generated by a subset, 31 closed sphere, 46 closed-open interval in Rn , 390 closed-open segment, 117 closure operator, 29 closure system on a set S, 28 cluster point, 179 cofinal set, 204 collection intersection of a, 5

B Baire set, 451 Baire space, 170 balanced set, 138 Banach space reflexive, 614 957

May 2, 2018 11:28

958

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 958

Mathematical Analysis for Machine Learning and Data Mining

union of a, 5 collection of neighborhoods, 174 collection of sets restriction of a, 39 compact linear operator, 344 compact set, 189 complementary slack condition, 833 complete simplex, 148 complex measure total variation of a, 464 variation of a, 464 complex numbers extended set of, 258 complex-valued function integrable, 505, 548 concave function, 748 subgradient of a, 781 cone, 130 connected component of an element, 223 constraint qualification, 852 contingent cone, 370 continuity from above, 415 continuity from below, 415 continuous function, 213 continuous function at a point, 210 contracting sequence of sets, 16 contraction, 296, 639 contraction constant, 296, 639 convergence in measure, 552 convergent sequence of sets, 17 convex closure of a set, 124 convex combination, 120 convex function, 748 closed, 751 subgradient of a, 781 convex hull of a set, 124 convex set, 118 face of a, 133 Minkowski functional of a, 140 proper face of a, 133 relative border of a, 355 relative interior of a, 352 support function of a, 810 supporting set of a, 373

core intrinsic of a set, 147 cosine squasher, 894 countable additivity property of measures, 398 covariance of random variables, 678 cover subcover of a, 189 critical point, 656, 664 cycle, 49 trivial, 49

D data fitting, 920 data sample, 926 decreasing mapping, 27 derived set, 179 descent direction, 820 design matrix, 909 dimension of an affine subspace, 122 directed set, 204 directional derivative of a function, 630 discontinuity point jump of a function in a, 203 of first type, 203 of second type, 203 dissimilarities definiteness of, 43 evenness of, 43 dissimilarity, 43 space, 43 dissimilarity space amplitude of a sequence in a, 43 distance, 44 Hamming, 45 distribution marginal distributions of a, 577 doubly stochastic matrix, 48 dual optimization problem, 843 duality, 113

E epigraph of a function, 237

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Index

equality constraints, 826 equicontinuity of a collection of functions, 315 essential infimum, 446 essential lower bound, 446 essential supremum, 446 essential upper bound, 446 event, 464 exposed faces, 134 extended dissimilarity on a set, 43 extended dissimilarity space, 43 extreme point, 132

F F -closed subset of a set, 33 Fδ -set, 275 face k-, 133 feasible direction, 820 feasible region, 826 feature map, 932 feature space, 932 Fenchel conjugate of a function, 808 filter, 201 basis, 201 sub-basis, 201 finite additivity property of measures, 399 finite intersection property, 190 fixed point, 298 flat, 121 Fourier expansion, 107 function, 10 a-strongly convex, 814 absolutely continuous, 571 affine, 750 bijective, 11 choice, 12 coercive, 252 continuous in a point, 265 continuity modulus of a, 320 continuously differentiable, 626 differential of a, 625 discontinuous in a point, 202

9in x 6in

b3234-main

page 959

959

discriminatory relative to a regular Borel measure, 896 domain of a, 10 Fermi logistic, 894 gradient of a, 633 Heaviside, 894 Hessian matrix of a, 655 injective, 11 integrable, 500 left inverse of a, 11 left limit of a, 262 left-continuous, 211 limit of a, 261 Lipschitz constant for a, 296 locally Lipschitz, 296 lower semicontinuous, 230 measurable complex-valued, 398 with complex values, 414 measurable on a set, 483 negative part of, 411 onto, 11 positive part of, 411 pseudo-convex, 773 ramp, 894 range of a, 10 restriction of a, 11 right inverse of a, 12 right limit of a, 262 right-continuous, 211 semicontinuous, 230 sesquilinear, 86 sigmoidal, 894 squashing, 894 strictly pseudo-convex, 773 strongly convex, 814 support of a, 215 surjective, 11 total, 11 uniformly continuous, 265 upper semicontinuous, 230 function of positive type, 721 functional, 134 functions equivalent in measure, 562

May 2, 2018 11:28

960

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 960

Mathematical Analysis for Machine Learning and Data Mining

G Gδ -set, 275 Gibbs distribution, 859 Gram-Schmidt algorithm, 100 greatest element, 19 greatest lower bound, 20

H Hausdorff metric hyperspace, 302 Hilbert space, 677 hinge function, 794 homeomorphic topological spaces, 218 homeomorphism, 218 homogeneous polynomial, 631 hyperplane, 110 vector normal to a, 112 vertical, 112 hypograph of a function, 237

I I-open subsets of a set, 34 identity complex polarization, 89 real polarization, 90 image of a set, 14 inclusion-exclusion equality, 406 increasing mapping, 27 indefinite integral, 549 independent σ-algebras, 467 independent events, 466 indicator function of a subset, 10 inequality constraints, 826 inequality Bessel, 689 Buneman, 43 Cauchy-Schwarz, 558 Chebyshev, 511 Finite Bessel, 688 H¨ older, 557 Markov, 511 infimum, 20 infimum of a function, 445

infinite arithmetic progression, 185 infinite sequence on a set, 13 inner product, 85 inner product space, 85 instance of the least square problem, 910 integrable function, 492 integral on a measurable subset, 499 interior of a set, 169 interior operator, 33 interior system, 33 intersection associativity of, 4 commutativity of, 4 idempotency of, 4 isolated point, 179 isometric metric spaces, 47 isometry, 47 iteration of a function, 297

J Jacobi’s method, 885 Jacobian determinant of a function, 635 Jacobian matrix, 634

K K-closed subsets of a set, 30 Kantorovich’s inequality, 860 Karush-Kuhn-Tucker conditions, 837 Karush-Kuhn-Tucker sufficient conditions, 842

L Lagrange multipliers, 833 Lagrangian function Fritz John, 828 Kuhn-Tucker, 828 lateral limits, 202 learning rate, 939 least element, 19 least upper bound of a set, 20 Lebesgue’s Lemma, 269

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Index

level set of a function, 238 limit of a function along a filter base, 202 limit of a sequence of sets, 17 limit point, 179 line determined by a point and a vector, 118 determined by two points, 118 linear functional, 74, 134 linear operator, 74 compact, 610 eigenvalue of a, 84 eigenvector of a, 84 invariant subspace, 84 resolvent of a, 84 linear operator associated to a matrix, 81 linear regression, 909 linear space basis of a, 69 bounded set in a, 333 endomorphism of a, 75 Hamel basis of a, 69 linear combination of a subset of a, 68 linear operator on a, 75 Minkowski sum of subsets of a, 329 partially ordered, 132 reflection in a, 77 set spanning a, 69 set that generates a, 69 subspace of a, 69 translation in a, 77 trivial subspace of a, 70 zero element of a, 66 linearly dependent set, 68 linearly independent set, 68 linearly separable, 357, 926 linearly separable set, 939 Lipschitz function, 296, 639 local basis of a point, 178 local extremum, 656, 820 local maximum, 656 local maximum of a functional, 819 local minimum, 656

9in x 6in

b3234-main

page 961

961

local minimum of a functional, 819 locally convex, 119 locally convex linear space, 338, 339 logistic function, 894 lower bound, 19 lower limit of a sequence of sets, 17

M μ-measurable set, 419 mapping bilinear, 113 matrix Hermitian conjugate, 66 transpose, 67 matrix associated to a linear mapping, 80 maximal element, 21 measurable function, 393 Borel, 394 measurable set Lebesgue, 423 measurable space, 385 measure, 398 σ-finite, 400 absolutely continuous measure with respect to another, 525 Borel, 450 complete, 401 complex, 462 counting, 399 density of a, 551 Dirac, 399 finite, 400 Fourier transform of a, 566 image, 399 inner regular, 450 Lebesgue outer measure, 418 Lebesgue-Stieltjes, 571 locally finite, 450 monotonicity of, 400 outer regular, 450 probability, 464 Radon, 450 semi-finite, 400 tight, 452

May 2, 2018 11:28

962

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Mathematical Analysis for Machine Learning and Data Mining

measure space, 398 complete, 401 completion of a, 401, 404 extension of a, 403 Hahn decomposition of a, 459 property of elements in a, 401 regular set in a, 452 measures modularity property of, 400 mutually singular, 459 mesh of a triangulation, 146 metric, 44 discrete, 45 Hausdorff, 302 Minkowski, 95 topology induced by a, 255 metric space, 44 bounded set in a, 46 complete, 276 diameter of a, 46 diameter of a subset of a, 46 distance between an element and a set in a, 266 precompact, 291 separate sets in a, 268 topological, 256 totally bounded, 291 minimal element, 21 minimum bounding sphere, 847 monotonic mapping, 27 monotonicity of measures, 400 morphism of posets, 27 multinomial coefficient, 52 multinomial formula, 53

convergent, 205 eventually bounded, 333 eventually in a set, 205 finite, 291 frequently in a set, 205 monotonic, 205 r-, 291 subnet of a, 204 net is eventually in a set, 205 net is frequently in a set, 205 neural network, 895 neuron, 893 activation function of a, 893 threshold value of a, 893 weights vector of a, 893 non-negative combination, 120 non-negative measurable function Lebesgue integral of a, 491 non-negative orthant, 118 norm Euclidean, 94 metric induced by a, 95 Minkowski, 94 supremum, 286 uniform convergence, 286 normal subspace, 664 normal to a surface, 664 normed linear space, 88 normed space dual of a, 612 notation big O, 626 Landau, 626 small o, 626 null set, 401

N O negative open half-space, 112 neighborhood of infinity, 262 basis, 178 item of a point, 174 net, 204 anti-monotonic, 205 Cauchy, 333 clusters, 205

on-line learning, 938 one-class classification, 935 open function, 221 open interval in Rn , 390 open segment, 117 open set, 162 in R size of, 240

page 962

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Index

open sphere, 46 open-closed interval in Rn , 61, 390 open-closed segment, 117 ordering cone, 132 orthogonal set of vectors, 100 orthogonal subspaces, 103 orthogonal vectors, 100 orthonormal set of vectors, 100 outer measure, 417 regular, 426 outlier, 930

P pair ordered, 6 components of an, 6 parallel affine subspaces, 122 parallelipiped, 480 partial derivative, 630 partial order strict, 18 partially ordered set, 18 partition, 8 blocks of a, 8 decision function of a, 898 measurable, 385 related to a simple function, 393 partition topology, 185 perfect matching, 15 permutation, 47 cyclic, 49 cyclic decomposition of a, 50 descent of a, 50 even, 51 inversion of a, 50 odd, 51 permutation matrices, 48 permutation parity, 51 pointed cone, 130 points in a topological space, 162 pointwise convergence, 343 pointwise convergence of a sequence of linear operators, 344 polyhedron, 142 boundary hyperplane of a, 142

9in x 6in

b3234-main

page 963

963

polytope, 142 poset, 18 chain in a, 25 greatest element of a, 20 least element of a, 20 upward closed set in a, 163 positive combination, 120 positive linear functional, 540 positive open half-space, 112 pre-image of a set, 14 premeasure, 398 primal problem, 843 probability, 464 conditional, 465 probability conditioned by an event, 465 probability space, 464 probability space induced by a random variable, 572 product measurable space of a family of measurable spaces, 386 product of the topologies, 226 product of topological spaces, 226 pseudo-metric, 44

Q quadratic optimization problem, 846 quasi-concave function, 770 quasi-convex function, 770

R Radon-Nikodym derivative, 531 random variable, 467 σ-algebra generated by a, 467 conditional expectation, 580 discrete, 572 point mass of a, 572 distribution function of a, 573 distribution of a, 572 expectation of a, 572 mean value, 572 probability density function of a, 574 variance of a, 572

May 2, 2018 11:28

964

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 964

Mathematical Analysis for Machine Learning and Data Mining

random variables correlation coefficient of, 579 independent, 468, 577 uncorrelated, 579 version of a, 468 random vector, 577 distribution of a, 577 non-degenerate n-dimensional, 579 real linear space n-dimensional, 71 dimension of a, 71 rectangle on a set product, 387 reflexion homothety in a, 77 regressor, 910 regula falsi, 891 regular elements of an operator, 616 regular point, 664 relation antisymmetric, 9 infix notation of a, 9 irreflexive, 8 on a pair of sets, 8 partial order, 9 reflexive, 8 symmetric, 8 transitive, 9 transitive closure of a, 32 transitive-reflexive closure of a, 33 relatively open set, 352 remainder of order n, 654 residual of a vector, 874 reverse Fatou Lemma, 589 Riesz’ Lemma, 347 ridge loss function, 916 ring of sets on S, 61

S σ-algebra product, 386 separable, 471 saddle point, 656, 852 Fritz John, 828 Schauder basis, 314 section of a set, 533

semi-ring, 61 semimetric, 45 seminorm, 88 separating collection of seminorms, 339 separating hyperplane, 356 sequence, 13 Cauchy, 276 convergence from left, 261 convergence of order p of a, 865 convergent, 186, 260 limit point of a, 186 linear convergence of a, 865 quadratic convergence of a, 865 subsequence of a, 28 superlinear convergence of a, 865 convergent to a point, 186 divergent to +∞, 187 generated by conjugate directions, 876 of functions pointwise convergence of a, 283 uniform convergence of a, 283 sequence of sets contracting, 16 expanding, 16 monotone, 16, 30 series, 307 absolutely convergent, 308 convergent, 307 divergent, 307 in a normed space, 312 absolutely convergent, 313 partial sums of a, 312 terms of a, 312 partial sum of a, 307 semiconvergent, 308 sum of a, 307 terms of a, 307 set bounded, 19 complement of a, 5 first category, 171 gauge on a, 45 index, 13

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Index

measurable, 385 nowhere dense, 171 product, 6 relation on a, 8 saturated by a partition, 8 second category, 171 unbounded, 19 set bounded from below, 364 set of active constraints, 826, 836 set of continuous functions point separation by a, 286 set of permutations, 47 sets Cartesian product of, 6 collection refinement of a, 7 collection of, 4 trace of, 7 inclusion between, 5 collection of hereditary, 7 disjoint, 4 equipollent, 941 symmetric difference of, 6 entropy Shannon’s, 756 β-, 798 signed measure, 456 σ-finite, 461 absolute value of a, 461 finite, 461 Jordan decomposition of a, 460 negative set for a, 457 null set for a, 457 positive set for a, 457 total variation of a, 461 signed measure space, 456 similarity, 46, 295 ratio, 295 space, 46 simple function, 392 Lebesgue integral of a, 486 simplex, 142 dimension of a, 142 subsimplex of a, 145 triangulation of a, 146

9in x 6in

b3234-main

page 965

965

slack variables, 930 soft margin, 931 space of sequences p , 99 spectrum, 616 continuous, 616 point, 616 residual, 616 standard simplex, 145 star-shaped set, 119 stationary point, 656 step function, 560 stochastic matrix, 48 strict local maximum of a functional, 819 strict local minimum of a functional, 819 strict order, 18 strictly antimonotonic mapping, 27 strictly concave function, 748 strictly convex function, 748 strictly decreasing mapping, 27 strictly increasing mapping, 27 strictly monotonic mapping, 27 strictly quasi-convex function, 770 strictly separated sets, 357 subcollection, 5 subcover of an open cover, 184 subdivision of an interval, 514 subset closed under a set of operations, 33 subset of a linear space in general position, 124 subspace affine, 110 orthogonal complement of a, 104 sum of two linear operators, 77 summable sequence of elements, 307 summable set, 942 support hyperplane, 134 support vector machines kernelization of, 933 support vectors, 927 supporting hyperplane, 357 supremum, 20 supremum of a function, 445 symmetric set, 138

May 2, 2018 11:28

966

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

page 966

Mathematical Analysis for Machine Learning and Data Mining

system π-, 41 Dynkin, 41 irreducibly inconsistent, 367 minimally inconsistent, 367 system of normal equations, 911

T tangent hyperplane, 664 tangent subspace, 664 tangent vector, 370 Taylor’s Formula, 650 the Bolzano-Weierstrass property of compact spaces, 193 the class of functions C k (X), 649 the least square method, 910 Theorem Almgren, 752 Aronszajn, 729 Arzel` a-Ascoli, 317 Baire, 278 Banach Fixed Point, 298 Banach-Steinhaus, 603 Beppo Levi, 510 Birkhoff-von Neumann, 135 Bohnenblust-Karlin-Shapley, 779 Bolzano, 305 Bolzano-Weierstrass, 191, 193 Borwein, 786 Carath´eodory, 131, 137 Carath´eodory Extension, 425 Carath´eodory Outer Measure, 420 Chain Rule, 632 Closed Graph, 605 Completeness of Convergence in Measure, 555 Complex Hahn-Banach, 83 Complex Stone-Weierstrass, 290 Cram´er-Wold, 568 Dedekind, 23 Dini, 284 Dominated Convergence, 508 Fan-Glicksburg-Hoffman, 775 Farkas, 366

Fritz John’s Necessary Conditions, 832 Fubini, 538 Hall’s Perfect Matching, 15 Heine, 295 Gauss-Markov, 913 Geometric Version of Hahn-Banach, 357 Gordan, 366 Heine-Borel, 293 Helly, 154 Implicit Function, 640 Intermediate Value, 305 Inverse Function, 659 Inversion, 566 Jensen, 753 Jordan’s Decomposition, 460 Kantorovich, 871 Karush-Kuhn-Tucker, 837, 842 Krein-Milman, 374 Lax-Milgram, 739 Lebesgue Decomposition, 533 Leibniz, 308 Lindel¨ of, 184 Lyusternik, 610 Measure Continuity, 415 Mean Value, 499, 633 Mercer, 734 Monotone Collection, 39 Monotone Convergence, 494 Moreau-Rockafellar, 784 Motzkin, 368 Open Mapping, 604 Partition of Unity, 236 Prolongation, 354 Pythagora, 104 Radon-Nikodym, 531 Real Hahn-Banach, 82 Regularity Theorem for Measures on Rn , 447 Riemann, 310 Riesz, 553, 703 Riesz-Fischer, 563 Riesz-Markov-Kakutani, 546 Schoenberg, 731 Separation, 358

May 2, 2018 11:28

Mathematical Analysis for Machine Learning

Index

Shapley-Folkman, 138 Spectral Theorem for Compact Self-adjoint Operators, 711 Stone, 127 Stone-Weierstrass, 289 Strong Duality, 851 Tietze, 282 Tonelli, 537 Tychonoff, 229 Uniqueness of Measures, 414 Vitali, 448 Weak Duality, 844 topological F-linear space, 329 topological linear space complete, 333 quasi-complete, 333 sequentially complete, 333 topological property, 220 topological set compact, 189 topological space, 162 T0 , 194 T1 , 194 T2 , 194 T3 , 194 T4 , 194 arcwise connected, 250 basis of a, 180 boundary of a set in a, 172 clopen set in a, 184 closed cover in a, 184 completely regular, 218 connected, 222 connected subset of a, 222 continuous path in a, 250 cover in a, 184 dense set in a, 168 disconnected, 222 empty, 162 first axiom of countability for a, 183 Hausdorff, 194 locally compact, 197 compactification of a, 201 metrizable, 256 normal, 196

9in x 6in

b3234-main

page 967

967

open cover in a, 184 regular, 196 second axiom of countability for a, 183 separable, 168 separated sets in a, 242 subspace of a, 165 totally disconnected, 225 topologically equivalent metrics, 259 topology, 162 Alexandrov, 163 coarser, 165 cofinite, 163 discrete, 162 Euclidean, 256 finer, 165 indiscrete, 162 metrizable, 259 pull-back, 239 push-forward, 240 stronger, 165 sub-basis of a, 182 usual, 162 weak, 220 weaker, 165 total order, 25 totally bounded set, 291 trace of a matrix, 86 translation, 330 transposition, 49 standard, 50 tree metric, 44 triangular inequality, 43

U ultrametric, 44 ultrametric inequality, 43 uniform convergence, 343 uniform dense set on compact set, 903 uniform distribution, 575 uniform equicontinuity, 316 union associativity of, 4 commutativity of, 4 idempotency of, 4

May 2, 2018 11:28

968

Mathematical Analysis for Machine Learning

9in x 6in

b3234-main

Mathematical Analysis for Machine Learning and Data Mining

upper bound, 19 upper limit of a sequence of sets, 17 usual topology of the extended set of reals, 181 Uryson’s Lemma for normal spaces, 216 for locally compact spaces, 235

V variance, 511 variety, 121

Vitali cover, 448 volume, 427 homogeneity, 435

W Weierstrass M -test, 315

page 968