Linear algebra with applications

Table of contents :
0. Systems of Equations and Matrices --
1. Vector Spaces --
2. Linear Operators --
3. Inner Product Spaces --
4. Diagonalizable Linear Operators --
5. The Structure of Normal Operators --
6. Bilinear and Quadratic Forms --
7. Small Oscillations --
8. Factorizations and Canonical Forms.

Citation preview

International Séries in Pure and Applied Mathematics

/ •

J. T. Scheick




* .

Linear Algebra . ‫י‬ with Applications



Linear Algebra with Applications

International Series in Pure and A pplied M athem atics Ahlfors: Complex Analysis Bender and Orszag: Advanced Mathematical Methods for Scientists and Engineers Bilodeau and Thie: An Introduction to Analysis Boas: Invitation to Complex Analysis Brown and Churchill: Complex Variables and Applications Brown and Churchill: Fourier Series and Boundary-Value Problems Buchanan and Turner: Numerical Methods and Analysis Buck: Advanced Calculus Burton: Elementary’ Number Theory Burton: The History of Mathematics: An Introduction Chartrand and Oellermann: Applied and Algorithmic Graph Theory Colton: Partial Differential Equations Conte and de Boor: Elementary Numerical Analysis: An Algorithmic Approach Edelstein-Keshet: Mathematical Models in Biology Farlow: An Introduction to Differential Equations and Their Applications Goldberg: Matrix Theory with Applications Gulick: Encounters with Chaos Hill: Experiments in Computational Matrix Algebra Keisler and Robbin: Mathematical Logic and Computability Kurtz: Foundations of Abstract Mathematics Lewin and Lewin: An Introduction to Mathematical Analysis Malik, Mordeson, and Sen: Fundamentals of Abstract Algebra Morash: Bridge to Abstract Mathematics: Mathematical Proof and Structures Parzynski and Zipse: Introduction to Mathematical Analysis Pinsky: Partial Differential Equations and Boundary-Value Problems Pinter: A Book of Abstract Algebra Ralston and Rabinowitz: A First Course in Numerical Analysis Ritger and Rose: Differential Equations with Applications Robertson: Engineering Mathematics with Maple Robertson: Engineering Mathematics with Mathematica Rudin: Functional Analysis Rudin: Principles of Mathematical Analysis Rudin: Real and Complex Analysis Scheick: Linear Algebra with Applications Simmons: Differential Equations with Applications and Historical Notes Small and Hosack: Calculus: An Integrated Approach Small and Hosack: Explorations in Calculus with a Computer Algebra System Vanden Eynden: Elementary Number Theory Walker: Introduction to Abstract Algebra

A lso available from M cG raw -H ill Schaum ’s O utline Series in M athem atics and Statistics Most outlines include basic theory, definitions, hundreds of example problems solved in step-by-step detail, and supplementary problems with answers. Related titles on the current list include: Advanced Calculus Advanced Mathematics for Engineers & Scientists Analytic Geometry Basic Mathematics for Electricity & Electronics Basic Mathematics with Applications to Science & Technology Beginning Calculus Boolean Algebra & Switching Circuits Calculus Calculus for Business, Economics, & the Social Sciences College Algebra College Mathematics Combinatorics Complex Variables Descriptive Geometry Differential Equations Differential Geometry Discrete Mathematics Elementary Algebra Finite Differences & Difference Equations Finite Element Analysis Finite Mathematics Fourier Analysis General Topology Geometry Group Theory Laplace Transforms Linear Algebra Mathematical Handbook of Formulas & Tables Mathematical Methods for Business & Economics Mathematics for Nurses Matrix Operations Modem Abstract Algebra Numerical Analysis Partial Differential Equations

Probability Probability & Statistics Real Variables Review of Elementary Mathematics Set Theory & Related Topics Statistics Technical Mathematics Tensor Calculus Trigonometry Vector Analysis

Sch aum ’s Solved Problem s Series Each title in this series is a complete and expert source of solved problems with solutions worked out in step-by-step detail. Related titles on the current list include: 3000 Solved Problems 2500 Solved Problems 2000 Solved Problems 3000 Solved Problems 2000 Solved Problems 3000 Solved Problems

in Calculus in Differential Equations in Discrete Mathematics in Linear Algebra in Numerical Analysis in Precalculus

Bob M iller’s Math Helpers Bob Miller’s Calc I Helper Bob Miller’s Calc 11 Helper Bob Miller’s Precalc Helper McGraw-Hill Paperbacks Arithmetic and Algebra . . . Again How to Solve Word Problems in Algebra Mind Over Math Available at most college bookstores, or, for a complete list of titles and prices, write to: Schaum Division The McGraw-Hill Companies, Inc. 11 West 19th Street New York, NY 10011

Linear Algebra with Applications

John T. Scheick Duke University

THE McGRAW-HILL COMPANIES, INC. New York St. Louis San Francisco Auckland Bogotá Caracas London Madrid Mexico City Milan Montreal New Delhi San Juan Singapore Sydney Tokyo Toronto


This book was set in Times Roman by Publication Services, Inc. The Editors were Jack Shira, Karen M. Minette, and John M. Morriss; the production supervisor was Kathryn Porzio. Project supervision was done by Publication Services, Inc. R. R. Donnelley & Sons Company was printer and binder.

McGraw-Hill A Division of The M cGraw-Hill Companies

LINEAR ALGEBRA WITH APPLICATIONS Copyright ©1997 by The McGraw-Hill Companies, Inc. All rights reserved. Printed in the United States of America. Except as permitted under the United States Copyright Act of 1976, no part of this publication may be reproduced or distributed in any form or by any means, or stored in a database or retrieval system, without the prior written permission of the publisher. This book is printed on acid-free paper. 1 2 3 4 5 6 7 8 9 0

DOC DOC 9 0 9 8 7 6

ISBN 0-07055184-7‫־‬ Library of Congress Cataloging-in-Publication Data Scheick, John T. Linear algebra with applications / John T. Scheick. p. cm.—(International series in pure and applied mathematics) Includes index. ISBN 0-07-055184-7 1. Algebras, Linear. I. Title. II. Series. QA184.S34 1997 512' .5—dc20 96-27488


J. T. SC H EIC K received his Ph.D. in mathematics, in approximation theory, from Syracuse University in 1966. His interests turned to applied mathematics and numerical analysis in the mid 1970s. He has done consulting for General Motors for seven years and has worked on research projects with the Electroscience Laboratory at Ohio State University for several years. Professor Scheick has a strong interest in teaching engineering students at the undergraduate and graduate levels and has served on hundreds of Ph.D. examination and dissertation committees in the College of Engineering at Ohio State University. He is also a member of the Society for Industrial and Applied Mathematics (SIAM). He recently retired from Ohio State University and now teaches at Duke University.






Systems of Equations and Matrices



Systems of Linear Equations and Matrices



Solution of Homogeneous Systems Gauss elimination / Row-reduced echelon form / Existence of nontrivial solutions and description of the solution of Ax = 0



Solution of Inhomogeneous Systems Gauss elimination revisited / Existence and uniqueness of solutions



Matrix Algebra Matrix multiplication, transposition, and conjugation / Identities and rules



Systems of Equations and Matrix Inverses Conditions for the existence of an inverse / Computation of the inverse / Elementary matrices





Vector Spaces



Vector Spaces Axioms / Principal examples



Subspaces Subspaces / Linear combinations and span / Connection to matrices



Linear Independence Examples / Connection to matrices



Basis and Dimension Basic theory / Examples / Applications of the theory to some fundamental vector spaces, matrices, and systems of equations



Coordinates with Respect to a Basis Change of basis and change of coordinates


Linear Operators




Preliminary Topics

Functions / Linear operators / Null space and range / Examples IX

Table of Contents and Outline




The Rank and Nullity Theorem Rank and nullity theorem / Operator inverses / Application to matrix theory / Computation of the range and null space o f a matrix



Linear Operators and Matrices Matrix of an operator / Operator algebra / Change of basis and similar matrices / Applications


Inner Product Spaces



Preliminaries Definitions and examples / Norms; angle between vectors



Orthogonal Sets Computational advantages of orthogonal sets / Fourier coefficients and Parseval’s identity / Gram-Schmidt process / QR factorization



Approximation and Orthogonal Projection Equivalence of the problems / Computations using orthogonal and nonorthogonal sets / Normal equations / Projection operator / M 1 /Examples in many settings



Applications of Projection Theory Projections in Fm / Weighted linear regression / Data fitting with polynomials, weighted and unweighted



Orthogonal Complements 151 Decomposition of the vector space /Applications to approximation and matrix theory / Fredholm alternative


The Gram Matrix and Orthogonal Change of Basis Matrix representation of inner products / Orthogonal change of basis / Rank of a Gram matrix


Diagonalizable Linear Operators



Eigenvalues and Eigenvectors Definitions / Spectrum and eigenspaces of an operator / Theoretical computations using determinants / Properties of the characteristic polynomial / Geometric and algebraic multiplicities / Remarks on numerical calculations



Linear Operators with an Eigenbasis Diagonalizable operators and their computational advantages / Similarity to a diagonal matrix



Functions of Diagonalizable Operators Two competing definitions / Functions of matrices / General properties of functions of diagonalizable operators / Minimal polynomial


Table of Contents and Outline


First-Order Matrix Differential Equations Decoupling the differential equations: Two viewpoints for diagonalizable matrices / eAl



Estimates of Eigenvalues: Gershgorin’s Theorems



Application to Finite Difference Equations Biological models, Markov chain examples, and finite difference equations





Review of determinants


The Structure of Normal Operators



Adjoints and Classification of Operators Definitions / Normal, Hermitian, and unitary operators / Matrix characterization



The Spectral Theorem Spectral theorem and resolution / Functions of normal operators / Simultaneous diagonalization of normal operators



Applications to Matrix Theory Functions of normal matrices / Generalized eigenvalue problem



Extremum Principles for Hermitian Operators The Rayleigh quotient and its extremal properties / CourantFischer theorem / Interlacing theorem for bordered matrices



The Power Method Estimating the dominant eigenvalue and eigenvector / Sharpening estimates / Approximation of secondary eigenvalues and eigenvectors / Inverse power method / Subspace methods



The Rayleigh-Ritz Method Approximation of a finite number of eigenvalues and eigenvectors of a Hermitian operator defined on an infinite dimensional space


6 Bilinear and Quadratic Forms



Preliminaries Definitions and examples / Elementary properties / Polar identities / The matrix of a bilinear form and change of basis; congruent matrices



Classification of Hermitian Quadratic Forms Classification of quadratic forms, operators, and matrices / Diagonalization and the law of inertia / Minimization problems



Table of Contents and Outline




Orthogonal Diagonalization Orthogonal diagonalization and the Principal Axis theorem / Applications: level sets and the strain ellipsoid



Other Methods of Diagonalization Completion of squares, simultaneous row and column operations, a method of Jacobi



Simultaneous Diagonalization of Quadratic Forms Simultaneous diagonalization of two and three quadratic forms and the connection to the generalized eigenvalue problem


Small Oscillations



Differential Equations of Small Oscillations Derivation of the differential equations by Newton’s laws, Kirchoff’s laws, Lagrange’s equations of motion, and the Ritz method for continuous systems



Undamped Small Oscillations Solution of the differential equations using two viewpoints / The language of small oscillations / Application of the power method / Examples, including the Ritz method / Response to harmonic excitation



Damped Small Oscillations Conditions in order that the differential equations can be decoupled / Solution of the differential equations / Response to harmonic excitation



Galerkin’s Method for Partial Differential Equations Illustration of the method for the wave equation


Factorizations and Canonical Forms



The Singular Value and Polar Decompositions The SVD and its interpretations / Polar decomposition



Applications of the SVD Solutions of Ax = y and the pseudoinverse / Applications in numerical analysis / Applications in pattern recognition



Schur’s Theorem Schur s theorem / Matrix norms and the sequence A" / Iterative solution of equations



Jordan Canonical Form Applications to differential equations / eAt


Answers to Selected Problems







Linear algebra is a tool used extensively in a wide variety of mathematical, science, and engineering fields. Most textbooks focus on an elementary introduction to the topic, an abstract approach with emphasis on the algebraic aspects of the field, or on technical aspects of the subject. The purpose of this book is to present the core topics of linear algebra that are essential to its use in contemporary mathematics, applied mathematics, engineering, and the sciences. Also, in an effort to illustrate the applicability of this core material, many applications are woven into the text in a wide variety of settings. The book is designed for strong upper-level undergraduates or beginning graduate students. It can be used in several ways: • A s a one-semester course for advanced undergraduates • A s a topics course for undergraduates • As a one-semester course for beginning mathematics graduate students, possibly followed by a more algebraically oriented course • As a one-semester course for graduate engineers with emphasis on application of the theory and computations A wide variety of exercises allows ample selection of problems for each category of students. There are sections and chapters o f the book that focus on applications. These may be used as special topics in lecture or as projects for students. Often students will read them out o f curiosity. Many good engineering students will find material in the book that is a valuable resource for their work. The book is self-contained and rigorous, but not designed to be covered one section per lecture. It is thematically arranged around the important concepts and language of abstract vector spaces and operator theory (e.g., the concepts o f basis, dimension, inner product, orthogonal expansion, projection, eigenvectors, Hermitian operators, and unitary operators). Many of these ideas will carry over to subsequent courses, and references are cited for follow-up reading. An essential premise is that students need to know about function spaces as well as «-tuple spaces (matrix theory), since many applications involve function spaces. The theory is given in an abstract setting, but is enhanced by many examples taken from vector spaces indispensable to the sciences and mathematics. This shows the value of an abstract theory, as well as illustrating the ideas behind the theory. Real and complex scalars, «-tuple spaces, and function spaces are stressed from the beginning. The use of several examples to illustrate one abstract idea will help the student to see the general principles and to learn to think abstractly. The theory for some serious contemporary applications is developed. For example, the linear algebra needed by a finite elements specialist is included. The interplay between matrix theory and vector spaces is constantly stressed, and the dual viewpoints are frequently used to show complementary methods of problem solving and thinking. Students should be adept at the use o f matrices and matrix methods as well as the use of vector space and operator methods after studying this book. Xlll



The text is flexible enough to allow the instructor to choose which chapters to cover and in which order. For example, there are sections and chapters that can be omitted, if the instructor so chooses, without interfering with the main ideas presented later in the book. For example, Chapter 7 can be used without Chapter 6, or both Chapters 6 and 7 can be omitted. The following sections could be considered optional: 3.4, 3.5, 3 .6 ,4 .4 ,4 .5 ,4 .6 , 5.5, 5.6, 6.4, 6.5, 7.4, and 8.3. If students have had an elementary course, then Chapter 0 could be skipped or quickly reviewed, or its essential facts could be mentioned by the instructor as Chapter 1 is traversed. Perhaps some or all of Section 4.1 could be assumed to be known as well. If Chapter 0 is skipped, the main results that will be used later in the book are Theorem 3 of Section 0.2, Theorem 2 of Section 0.3, the matrix identities (2a)-(2f) and (3) in Section 0.4, and Theorem 4 and Corollary 5 of Section 0.5. Some sample courses are given in the following table. Recommended sections are listed in each column under the student category. Chapter


0 1 2 3 4 5 6 7 8

1-5 all all 1-3 1-3

Math graduate all all 1-3, 5-6 1-3,5 1-4 1-3,5 all

Engineering graduate review as needed all all 1-5 1-4 1-5 1-3 (4, 5 optional) 1-3 1-2

The sections that may be considered primarily applications are 3 .4 ,4 .4 ,4 .6 ,5 .6 , and Chapter 7. These sections and some sections not typically found in undergraduate courses could serve as topics in a reading course for a second course or as a topics course for advanced undergraduates.

ACKNOWLEDGMENTS I would like to thank the following reviewers for their perceptive and valuable comments: Michael Ecker Hans Engler Larry Grove Murli Gupta Vyron M. Klassen Glen Ledder Justin Peters Stewart M. Robinson Alexander Soifer Horace Wiser

Pennsylvania State University Georgetown University University of Arizona George Washington University California State University University of Nebraska Iowa State University Cleveland State University University of Colorado Washington State University



I would also like to thank Jack Shira at McGraw-Hill for his encouragement and unflagging efforts to improve the book during review process. Finally, I would like to thank the many energetic and thoughtful engineering students at Ohio State University who have read previous drafts of the book and who offered many insightful and valuable suggestions for its improvement, both while taking the course and later, during the various stages of their research. Their enthusiasm contributed immensely to the pleasure of writing the book and to developing the contents of the book.

Linear Algebra with Applications


Systems of Equations and Matrices

A s a prelude to the study of vector spaces in general, we briefly consider systems of equations, procedures for solving them, the basic rules o f matrix algebra, and matrix inverses. Most of our examples involve real or complex numbers, but our considerations are valid for any field. The last section of this chapter contains a brief overview of fields. In this chapter, only the essentials o f systems o f equations and matrices will be developed. In later chapters, especially Chapters 1 and 2, the theory o f systems of equations and matrices will be extended to more sophisticated results as a byproduct o f the study o f vector spaces and linear operators.


SYSTEMS OF LINEAR EQUATIONS AND MATRICES A system of linear equations is a set of m equations in n unknowns:

ã\\X\ + Ü\2X2 + *• • + a \nxn = b x #21 X\ + Ü22%2 + *‫ • י‬+ ainXn = b2


am\X\ + am2X2 + * ‫־‬1‫ ־‬QmnXn = b!n. tion, this is more compactly written as n

^ L aijXj = bj

i - 1 , 2 , . . . , m.


y=1 A simple example of a system of two equations in three unknowns is

x\ + 3x2 5 ‫־־‬x3 = 1 (*) 2X| — X2 + 4x3 = 7•



0: Systems of Equations and Matrices

System (1) is said to be homogeneous iff all b !• = 0 and inhomogeneous, or nonhomogeneous, otherwise. In discussing procedures to solve a system o f linear equations, it is convenient to introduce the concepts of a matrix, a column, and the multiplication of a matrix and a column. The matrix involved consists of the coefficients of the unknowns x!, and the columns consist of the unknowns and the bt. This results in an even more compact notation for system (1) and assists in the discussion of solving the system. In addition, procedures for solving a system of equations often are given solely in terms of the matrix of coefficients and the numbers bj. A column of length n is an n X 1 array *1

X = (x 1,X2, . . . , X n)T =


-X‫״‬. where the superscript T means transpose, i.e., convert the row (*!, x 2, . . . , xn) to a column. An m X n matrix is an array of objects consisting of m rows and n columns

A = [au ] =

‫־‬a 11 a2\

a \2 a 22

- 0 0 3.

1 1 1

1 0 0

R2 *—/?2

1 -1 2

-r -1 2.

2/? ‫ן‬

/?3 + R\


Next add row 2 to row 1 and twice row 2 to row 3. Tidy up the result by multiplying row 2 by —1. Schematically,

‫־‬1 0 0

-1 0 0

1 -1 2

- 1‫־‬ -1 2

‫־‬1 0 0

-1 0 0

‫־‬1 - 2‫־‬ -1 -> 0 0. 0

0 -1 0

/?! ־‬ (e) (AB)‫ = ' ־‬B‫' ־‬A ‫' ־‬

B are invertible, then so are A-1 , AT, A, AH, and AB. Fur-



0: Systems of Equations and Matrices

Proof. To prove (a ), note that from the symmetry of the requirements AB = / and BA = /, both A = B~l and B = A 1‫ ־‬hold. Thus /J-1 = (,4-1 )-1 . The remainder of the assertions involve verifying the requirements of the definition. To prove (b), one has to show that B = (A_1)T satisfies BAJ = Ar B = /. But taking the transposes of AA-1 = A_1A = l and using the properties of the transpose yields BAT = AJB = I. Statement (c) is proved similarly, and (d) follows from (b) and (c). To prove (e), just verify that (B1‫ ־‬A-1 )AB = / = AB{B~XA~]). The next theorem discusses the issue of uniqueness. LA = I and AR = /, then L = R = /4-1 . Hence the inverse of A is unique, when it exists.

t h e o r e m 2. If

Proof One has L = LI = L(AR) = (LA)R = Rl = R.

Matrix Inverses and A x



Next, we turn to the connection between matrix inverses and systems of equations. 3. If A is invertible, then the system of equations Ax = y has the unique solution x = A~]y.

p r o p o s it io n

Proof Suppose .4x = y.T\\cnA~xAx = l x = x = A 1‫ ־‬y. This shows thatx is unique. On the other hand, A(A~*y) = l y = y, which shows that A~' y is a solution to Ax = y.

The next theorem is very important in matrix theory. It asserts the logical equivalency of a number of statements. This means that, if any one statement is true, then all are true, or equivalently, if any one statement is false, then all are false. The proof also contains a method for the construction of A~]. 4. L e t/lb e a square matrix, and let r denote the row rank of A. The following statements are logically equivalent:


(n) A is row-reducible to /. (b) The only solution of ^4x = 0 is x = 0. (c) r = n. (d) The system Ax = y is consistent for every y. (e) There is a matrix B such that AB = /. ( / ) A has an inverse.

Proof The scheme of the proof is to show that (a) => (b) => (c) => (d) => (e) => ( / ) . (0)

(b ): Assume (a) is true. Since A row-reduces to /, Ax = 0 and lx — 0 have the same solution, namely, x = 0. (b) (c): Let R be the row-reduced echelon form of A. If r < n, then there would be free variables in the equations Rx = 0, and therefore a nonzero solution to Ax = 0. Therefore r = n must hold. (c) (d): By Theorem 2 of Section 0.3, Ax = y is solvable for each y since r =

m = n. (d) (