[1st Edition] 9781483194660

Table of contents :
Content:
Contributors to this VolumePage ii
Front MatterPage iii
Copyright pagePage iv
ContributorsPage v
PrefacePages vii-viiiC.T. LEONDES
Contents of Previous VolumesPages xi-xii
Adaptive Optimal Steady State Control of Nonlinear SystemsPages 1-50ALLAN E. PEARSON
An Initial Value Method for Trajectory Optimization Problems1Pages 51-132D.K. SCHARMACK
Determining Reachable Regions and Optimal Controls1Pages 133-196DONALD R. SNOW
Optimal Nonlinear FilteringPages 197-300J.R. FISHER
Optimal Control of Nuclear Reactor SystemsPages 301-388D.M. WIBERG
On Optimal Control with Bounded State VariablesPages 389-419JOHN McINTYRE, BERNARD PAIEWONSKY
Author IndexPages 421-424
Subject IndexPages 425-426

Citation preview

C O N T R I B U T O R S T O THIS V O L U M E J. R. FISHER JOHN McINTYRE BERNARD PAIEWONSKY ALLAN E. PEARSON D. K. SCHARMACK DONALD R. SNOW D. M. WIBERG

ADVANCES

IN

C O N T R O L SYSTEMS THEORY

AND

APPLICATIONS

Edited by C.

T.

L E O N D E S

DEPARTMENT OF ENGINEERING U N I V E R S I T Y OF CALIFORNIA L o s A N G E L E S , CALIFORNIA

V O L U

ACADEMIC

Μ Ε

5

PRESS

I

967

New York and London

C O P Y R I G H T © 1 9 6 7 , B Y ACADEMIC PRESS I N C . ALL RIGHTS RESERVED. NO PART OF THIS BOOK MAY B E REPRODUCED IN ANY FORM, B Y PHOTOSTAT, MICROFILM, OR ANY OTHER MEANS, WITHOUT WRITTEN PERMISSION FROM THE PUBLISHERS.

ACADEMIC PRESS INC. I l l Fifth Avenue, New York, New York 10003

United Kingdom Edition published by ACADEMIC PRESS INC. (LONDON) Berkeley Square House, London W . l

L T D .

L I B R A R Y OF CONGRESS CATALOG CARD N U M B E R : 6 4 - 8 0 2 7

PRINTED IN THE UNITED STATES OF AMERICA

Contributors Numbers in parentheses indicate the pages on which the authors' contributions begin.

J . R. F I S H E R , * Aerospace Group, Hughes Aircraft Company, Culver City, California (197) J O H N M c I N T Y R E , North American Aviation, Inc., Downey, California (389) B E R N A R D P A I E W O N S K Y , Institute for Defense Analyses, Arlington, Virginia (389) A L L A N E . P E A R S O N , Division of Engineering and Center for Dynamical Systems, Brown University, Providence, Rhode Island (1) D. K . S C H A R M A C K , Honeywell Systems and Research Division, Honeywell Incorporated, Minneapolis, Minnesota (51) D O N A L D R. S N O W , Department of Mathematics, University of Colorado, Boulder, Colorado (133) D. M . W I B E R G , Department of Engineering, University of California, Los Angeles, California (301)

^Present address: Texas Instruments Inc., Apparatus Division, Advanced Radar Systems Branch, Dallas, Texas.

ν

Preface T h e fifth volume of Advances in Control Systems continues in the purpose of this serial publication to bring together diverse information on important progress in the field of control and systems theory and applications, as achieved and discussed by leading contributors. T h e problem of the optimal control of a system for which accurate knowledge of its dynamic characteristics exists has received a great deal of attention over the past few years. An aspect of the control problem which has received far less attention is that of optimizing system performance in the absence of any a priori knowledge of the system or plant dynamic characteristics. I f the dynamics of the system are unknown and vary from time to time in an unpredictable manner, it is evident that some type of identification scheme must be incorporated into the system operation in order to achieve and maintain optimal performance in any meaningful sense. T h e techniques which have been proposed to cope with this problem, often referred to as an adaptive or adaptive optimal control problem, usually require some a priori information of the plant dynamic characteristics (e.g., the order and form of the differential equations may be known though some of the coefficients maybe unknown). Such techniques are useful when the order and form of the system's differential equations are known. On the other hand, there exist many practical situations in which the dynamic characteristics of the system are too complex to permit a representation in any reasonably simple form. A method for optimizing in some sense the system performance which does not require the complete identification of the system dynamics and which does not presume a knowledge of the order or form of the system differential equations is clearly desirable in such cases. T h e first contribution in this volume by A. E . Pearson deals with this problem and presents some rather basic techniques for it, many of which are original with Pearson. Fundamental necessary and sufficient conditions in the calculus of variations, basic to the optimal control problem, have been under investigation by mathematicians for many decades, in fact, for hundreds of years. Algorithms for the solution of the optimization problem have been under investigation for many years, but it is only in the last five or ten years that this extremely important area has received intensive effort. There are several fundamental approaches to algorithms for the solution of the optimization problem or what we may also refer to as the trajectory optimization or reference control input problem, and several of these have been treated in earlier volumes of this series. T h e contribution by D. K . Scharmack presents one of the most important efforts to date on the initial value vii

viii

PREFACE

iteration algorithmic approach to the solution of the optimization problem. A further notable feature of this contribution is the application of this approach to several substantive illustrative problems. One of the basic problems in control and systems theory in general is the determination of the set of states that can be reached at time Τ given a prescribed class of admissible control functions and an initial state at a specified initial time for a nonlinear system. This problem is treated in the contribution by D. R. Snow, and it is referred to there as the problem of determining the Γ-reachable region. A related problem, also treated by Snow, is the determination of the Γ-controllable region described there. In developing the results presented in this contribution Snow has extended many of the classical results of the calculus of variations and HamiltonJacobi theory to optimal control problems through the use of Carathéodory 's unifying approach. Although some of these extensions have been discussed in the literature during the past three or four years, they are presented here in a unified form for the first time. T h e contribution by J . R. Fisher presents a number of rather fundamental results in optimal nonlinear filtering. T h e differential equations of a system which, when driven by a noise-corrupted measurement vector, will generate either the conditional probability density or the conditional characteristic function of the state vector of a nonlinear system continuously in time are derived. It is shown that certain general classes of identification problems are results of this theory. It is also shown that the optimal nonlinear, non-Gaussian prediction problem is simply a two-stage application of the theory developed in this contribution. A rather comprehensive survey of earlier work and contributions is presented here also. An important application area of control and systems theory is the optimal control of nuclear reactors. T h e contribution by D. M. Wiberg presents a rather comprehensive treatment of some of the fundamental techniques possible here. A mathematical model of the system to be controlled is developed. Methods of estimating the effects of spatial variation on system stability are presented. Analytical design techniques for spatial feedback control systems are presented. Questions of controllability for distributed parameter systems are examined here also. This volume closes with a contribution by J . Mclntyre and B. Paiewonsky on optimal control with bounds on the state variables. There are many practical instances of where such problems occur. There have been numerous published results in the literature exploring various aspects of this important problem. This contribution reviews many of these results and presents an over-all view of the status of the techniques in this field. July, 1967

C . T . LEONDES

Contents of Previous Volumes

Volume 1

On Optimal and Suboptimal Policies in Control Systems Masanao Aoki T h e Pontryagin Maximum Principle and Some of Its Applications James J. Meditch Control of Distributed Parameter Systems P. K. C. Wang Optimal Control for Systems Described by Difference Equations Hubert Halkin An Optimal Control Problem with State Vector Measurement Errors Peter R. Schultz On Line Computer Control Techniques and Their Application to Reentry Aerospace Vehicle Control Francis H. Kishi AUTHOR I N D E X — S U B J E C T INDEX Volume 2

T h e Generation of Liapunov Functions D. G. Schultz T h e Application of Dynamic Programming to Satellite Intercept and Rendezvous Problems F. T. Smith Synthesis of Adaptive Control Systems by Function Space Methods H. C. Hsieh Singular Solutions in Problems of Optimal Control C. D. Johnson Several Applications of the Direct Method of Liapunov Richard Allison Nesbit AUTHOR I N D E X — S U B J E C T INDEX xi

Xii

CONTENTS OF PREVIOUS VOLUMES

Volume 3

Guidance and Control of Reentry and Aerospace Vehicles Thomas L. Gunckel, II Two-Point Boundary-Value-Problem Techniques P. Kenneth and R. McGill The Existence Theory of Optimal Control Systems W. W. Schmaedeke Application of the Theory of Minimum-Normed Operators to Optimum-Control-System Problems James M. Swiger Kaiman Filtering Techniques H. W. Sorenson Application of State-Space Methods to Navigation Problems Stanley F. Schmidt AUTHOR I N D E X — S U B J E C T INDEX Volume 4

Algorithms for Sequential Optimization of Control Systems David Isaacs Stability of Stochastic Dynamical Systems Harold J. Kushner Trajectory Optimization Techniques Richard E. Kopp and H. Gardner Moyer Optimum Control of Multidimensional and Multilevel Systems R. Kulikowski Optimal Control of Linear Stochastic Systems with Complexity Constraints Donald E. Johansen Convergence Properties of the Method of Gradients Donald E. Johansen AUTHOR I N D E X — S U B J E C T INDEX

Adaptive Optimal Steady State Control of Nonlinear Systems ALLAN E. PEARSON Division of Engineering and Center for Dynamical Systems Brown University Providence, Rhode Island

I« II.

III.

IV.

V.

Introduction

1

Formulation A. Statement of the Problem B. Evaluation of System Performance

4 4 6

Iterative Optimization A. Derivation of the Performance Gradient B. Recurrence Relations

.

.

.

.

17 19 25

Identification of the Performance Gradient A. Relationships between Adjoint-Derivative and Derivative Operators B. Measurement of Differentials

28

Multivariable Systems A. Formulation B. Iterative Optimization and Identification

41 41 46

References

.

.

.

.

30 38

49

I. Introduction Optimization problems in the theory of control systems have received considerable attention in recent years. T h e mathematical techniques which have been applied to optimal control problems depend upon an accurate knowledge of the plant dynamic characteristics. Given the differential equations of the plant to be controlled, a performance functional to be minimized over a certain set of control functions, and various constraints pertinent to the problem at hand, the mathematical techniques facilitate the determination of the necessary conditions for optimal system performance (1-3). An aspect of the control problem which has received far less attention is that of optimizing the system performance in the absence of any a priori knowledge of the plant dynamic characteristics. I f the plant dynamics are unknown and vary from time to time in an unpredictable manner, it is 1

2

ALLAN Ε. PEARSON

evident that some type of identification scheme must be incorporated into the system operation in order to achieve and maintain optimal performance in any meaningful sense. The techniques which have been proposed to cope with this problem, often referred to as an adaptive or adaptive-optimal control problem, usually require some a priori information of the plant dynamic characteristics ; e.g., the order and form of the differential equations may be known though some of the coefficients may be unknown [see (4) for a review of adaptive control techniques]. Such techniques are useful when the order and form of the plant differential equations are known. On the other hand, there exist many practical situations in which the plant dynamic characteristics are too complex to permit a representation in any reasonably simple form. A method for optimizing (in some sense) the system performance which does not require the complete identification of the plant dynamics and which does not presume a knowledge of the order or form of the plant differential equations is clearly desirable in such cases. An adaptive optimal control scheme which does not rely upon a priori knowledge of the order or form of the plant differential equations is the Draper and Li extremum or peak holding controller (5, 6). This approach depends upon the existence of a plant output variable, or collection of variables, possessing an unknown maximum (or minimum) which may be slowly varying in time. The performance criterion in this case is to maintain the plant output as close to the extremum value as possible. T h e peak holding controller applies a slowly changing input signal (slow compared to the longest time constant of the plant) in a fixed direction until it is observed that the pertinent plant output variable has passed through its extremal value, whereupon the controller switches the direction of the input signal to force the plant variable back through its extremum. Assuming that the transient properties of the plant (the plant dynamic characteristics aside from the unknown extremal value) remain reasonably fixed in time, the peak holding controller continually adapts its operation to follow the slowly varying extremum of the output variable. In 1961, Kulikowski (7) introduced another approach to the particular adaptive optimal control problem considered by Draper and Li. In order to remove the assumption that the transient properties of the plant remain essentially fixed, Kulikowski proposed alternating periods of identification with periods of optimization in determining the form of the input signal. This is in contrast with the peak holding controller in which the sawtoothed nature of the input signal is specified in advance. Kulikowski introduced a greater degree of mathematical formalism into the problem by focusing attention upon minimizing the performance functional T

P(u) = xf*u>(t)dt-f Qy(t)dt

(1)

STEADY STATE CONTROL OF NONLINEAR SYSTEMS

3

where y(t) is the output variable which possesses an unknown maximum value and u(t) is the input to the plant. (The notation ζ is used to indicate the time function segment z = {(tyz(t))\0< t< T}.) T h e motivation for specifying the functional (1) stems from the desire to maximize the entire transient behavior of the plant over a fixed interval (0, T) and, at the same time, to minimize the cost for control, 2

j*u (t)dt as weighted by the constant λ > 0. In brief, Kulikowski's approach involves carrying out a certain type of identification at each step in the construction of a sequence of input functions { u w } , η = 1 , 2 , . . . , which under appropriate conditions will converge to an optimum input function u* satisfying P ( u * ) ^ P ( u ) . T h e relationship between the plant input and output functions is denoted symbolically by an operator A, y(t) = A(u)> 0 < t < T> which maps elements u from a space of input functions °ll into elements y belonging to a space of response functions ^ . T h e amount and type of identification which is to be performed with respect to each element u n is based upon the information needed to compute u n + 1 in an iterative minimization of (1). Kulikowski showed that the identification at each step involves the measurement of output elements Α{μη + ev) corresponding to various known input elements of the form u„ + ev where e is a small parameter. Although the assumption was made in (7) that the plant operator A possess certain symmetry properties which are rarely upheld in practice, this assumption was removed in subsequent work (8-10). One of the most important results of Kulikowski's original paper concerns the identification requirement that output elements y = A(un + ev) be measured as time function segments rather than requiring explicit or detailed knowledge of the operator A. Thus, in practice, the dynamic characteristics of the plant may be very complex since there is no need to assume linearity or any specific form of plant differential equations. T h e only assumptions needed concerning the plant dynamics are that they vary slowly in time relative to the time spent in constructing the sequence {un} and that the plant operator A possess a sufficient degree of smoothness to guarantee the existence of the gradient of the functional P(u). In subsequent papers (8-10) Kulikowski expanded the above approach and paved the way for further investigation (11-14). Pearson and Sarachik (15) later showed that the memory of the plant influenced the mathematical formulation of Kulikowski's approach. It was shown that this influence could be taken into account by requiring that the plant be in the proper steady state operation before measuring output elements A(un + ev) during the identification procedure. A somewhat different approach to basically

4

ALLAN Ε. PEARSON

the same class of adaptive optimal control problems has been reported by Zaborszky and Humphrey (16). This chapter extends the work which has been reported on Kulikowski's approach to adaptive optimal control problems. The formulation has been modified to include optimizing the total amount of input accumulation,

in addition to optimizing the form of the periodic input signal, in the case of plants possessing infinite memory, i.e., plants possessing pure integrators within their structure. The class of adaptive problems has been extended to include the optimization of system performance when the desired response of the system is a periodic function of time. T h e formulation of the adaptive optimal control problem is presented in Section I I in cognizance of the practical considerations for identification, Section IV, and the computational aspects of achieving optimum system operation, Section I I I . Sections I I - I V are concerned only with single input-single output plants ; however, the extension to the general case of multivariable nonlinear plants is indicated in the final section.

II. Formulation In general terms the problem of concern here is the optimization of the steady state performance of a plant whose dynamic characteristics are unknown and slowly varying with time. No precise meaning will be attached to the phrase *'slowly varying with time" although it wiH be clear that the plant dynamic characteristics must remain essentially fixed during the time required to carry out the identification and computations for the adaptive optimizing procedure. Emphasis will be placed upon determining a periodic input signal such that the plant output is forced into a steady state periodic signal and the resulting over-all system performance is optimum. T h e meaning of optimum steady state system performance is to be interpreted as the minimization of a performance index which evaluates the system performance over one period of steady state operation. A.

Statement of the Problem

It is assumed that there is a performance or cost functional of the general form (2) one period

STEADY STATE CONTROL OF NONLINEAR SYSTEMS

5

which serves to evaluate the output behavior of the plant and to assess the cost of operating the plant over one period of steady state operation. T h e plant input and output variables are denoted by u(t) and y(t), respectively, and yd(t) is a desired output. T h e function G{u,y,yd) is assumed to be twice differentiable in each of its arguments. T h e desired output of the plant, yd(t), is assumed to be either a constant or a periodic function of time with period Td. T h e object is to find a control function u* = {[t,u*(i)]\0 < t < T} in a space of admissible control functions where Γ is a submultiple of Td (Td = aT, α = an integer), such that a periodic input signal constructed from suitable repetitions of the element u* forces the plant into its optimum steady state operation. T h e choice of the integer α and the value of Τ = Ι/αΤ^ depends upon the energy storage properties of the plant as will be discussed in Part Β of this section. In addition to finding the best control element u* e °U to be used in forming a periodic input signal, it is necessary to determine the optimum level of input accumulation,

which is present at the start of each period of steady state operation in the case of plants possessing pure integrators within their structure. T h e problem of optimizing the steady state performance of a plant with unknown dynamic characteristics will be approached utilizing a step-bystep optimization and identification procedure. Mathematically, the problem will be viewed in terms of constructing a sequence of elements { u n } , n= 1 , 2 , i n the case of finite memory plants, or a sequence of ordered pairs {u,,, sn} in the case of plants possessing infinite memory, such that steady state optimal performance is achieved in the limit as η ->· oo. T h e space of functions °U from which the elements u„ are drawn in forming a periodic input signal is assumed to be an unbounded space of square integrable functions defined on the interval (0, Γ ) . Physically this assumption means that there is sufficient fuel, energy, or power to accomplish the control objectives. Amplitude constraints, such as would be caused by a valve or saturating amplifier, can be included in this formulation if the device responsible for the saturation can be approximated by a twice -1 differentiable function, e.g., u = t a n ku'. T h e saturating device can be included as part of the nonlinear plant operator with the input to the device a member of an unbounded space. T h e identification of certain essential aspects of the plant dynamic characteristics is to be carried out with respect to each element u„, or pair (u,,, sn), in order to compute the succeeding element u „ + 1 , or pair ( u ^ , sn+1)9 in an iterative minimization of the functional (2). T h e dynamic characteristics of the plant in steady state operation will be represented symbolically

6

ALLAN Ε. PEARSON

by an operator A which in the case of finite memory plants maps elements u from the space °ll into corresponding output elements y in a space of output functions . In the case of plants possessing infinite memory, the steady state operator A maps ordered pairs of elements (u, s) from the product space °ll χ 0t (where 3% denotes the set of real numbers) into a function space of response time function segments y. Although it is not necessary to assume any specific knowlege about the plant dynamic characteristics, it is assumed that the plant possesses a finite settling time and that it can be forced into a steady state operation with respect to an arbitrary input element u e ^ which comprises a suitable periodic input signal.

B. Evaluation of System Performance

The first question to be considered in formulating the adaptive optimal steady state control problem is the manner in which an arbitrary input element u G ^ should be used in forming a periodic input signal such that the plant is forced into the proper steady state operation. This question is equivalent to establishing a basis for evaluating the system performance in steady state operation which guarantees that the periodic output of the plant is due only to the input element u in question. It is clear that a distinction must be made between plants which possess a finite memory, referred to here as type-zero plants, and plants possessing pure integrators within their structure which are capable of storing energy indefinitely. In either case it is necessary to assume that the plant possesses a finite settling time and that the output of the plant can be forced into a periodic signal of the same fundamental period as the input, for an arbitrary element u e °U to be used in forming the periodic input signal.

1. T Y P E - Z E R O PLANTS

I f the plant does not possess any pure integrators within its structure which are capable of storing energy indefinitely, it is clear that simple repetitions of an input element u over successive time intervals of length Τ will eventually force the output into a periodic signal of the same fundamental period. After such steady state operation has been achieved, the output y measured over one period in phase with the input can be considered as the image element under a map A of the input element u, y = A(u)

(3)

where A is the steady state plant operator (see Fig. 1). T h e steady state

7

STEADY STATE CONTROL OF NONLINEAR SYSTEMS

performance of the system is appropriately evaluated with respect to an arbitrary input element u G °U via the functional T

(4)

P(u) = j oG(u,y,yd)dt where Τ is chosen such that T=Td

and y = A(u) is measured after steady state operation has been achieved. INPUT ι

U

U

U

U

F I G . 1. Procedure for establishing steady state operation of type-zero plants.

If the desired output yd is a constant (Td = oo), it will be assumed that Τ is chosen independently of the optimization procedure. It is natural in this case that Τ should be chosen larger than the anticipated plant settling time if the latter is known. There are other ways of constructing a periodic input signal from an arbitrary element u which would suffice to ensure that the ensuing periodic output of a type-zero plant depends only on the element u. Consider, for example, choosing the length Τ of the time function segment u according to (5)

T=\Td

and forming a periodic input signal of period Td such that the sequence fu u(t) = \ (0

for

0 ^

(h, g e J T )

(37)

where P ' ( h ) g is the first differential of the functional P ( h ) in the Fréchet sense. In general terms, the Fréchet differential F'(h) g of an operator F whose

STEADY STATE CONTROL OF NONLINEAR SYSTEMS

domain is a normed linear space from either F

F ' m ^

19

can be obtained at the point h e « f

O

l

+

y

t

à

-

m

(38)

y

y-»0

or F'(h)g = | - F ( h + rg ) d

7

y=0

where g, called the variation in h, is an element belonging to the same space Jf? as h, and y is a (real) scalar parameter. In particular, the spaces of interest here are the Hilbert spaces T h e operators of interest 0 and are the functionals (4), (8), (16), and (24) and the various steady state plant operators which were introduced to represent the mapping from the spaces Ji^o and J f ^ into the response segment space W during steady state operation [cf. Eqs. (3), (7), (17), and (23)]. T h e response elements y = {y(t)\0< t< T) will be assumed square integrable on the interval (0, T) such that the various steady state plant operators can be viewed as operators with the range space equivalent to J f 0 = ^· All operators in question are assumed to be twice differentiable in the Fréchet sense. T h e second differential F " ( h ) f g of an operator F can be obtained by repeated application of Eq. (38), thus 2

F»(h)fg="

a iXh +

y i

f+y2g)I '

(39)

It should be emphasized that Eqs. (38) and (39) do not define the Fréchet differentials of an operator F but serve only as practical means by which they can be determined for a given operator F when such differentials exist [see (18) or (19) for the definition of Fréchet differentials]. I f F is a linear operator with a domain Hilbert space and a range Hilbert space ^ , then a linear operator F> called the adjoint operator of F, can be defined by the inner product relation (19) ^=jr

(40)

where h and ζ are arbitrary elements in £F and ^ , respectively, and < , and < , >jr denote the inner products in and $f. It is to be noted that the domain space of F becomes the range space of F, while the range space of F is the domain space of F.

A.

Derivation of the Performance Gradient

T o derive the gradients of the various forms of the performance functional (2), it is necessary to use Eq. (38) in conjunction with definition (37)

20

ALLAN Ε. PEARSON

relative to the functional P. Consider as an example the performance functional specified by Eq. (4) for the evaluation of steady state system performance of type-zero systems. Using Eq. (38) the first differential is obtained as P ' ( u ) v = | - P ( u + yv)

- / . Ί *

τ

Ι * Η

+

Λ

( 4 l)

where the notation dG du

= {(*, ^Ht),y(t),y (t)])\0

0

(43b)

where h and g are arbitrary elements in J f 0 . Using Eqs. (43a) and (43b), Eq. (41) can be expressed in terms of the linear operator ^4'(u):

'

From definition (37) with functional (3) is seen to be VP(u) = ^

—pr—r dG\

dG

=^

, the gradient of the performance

0

+^'(u)^

(44)

(0^t^T=Td)

For the alternative formulation presented for type-zero systems, Eq. (8), the gradient is given by

ν ») = (|ί) Λ

ι+

^ ( § )

(0o=

(78)

Substituting Eq. (77) into Eq. (43) which defines the adjoint operator A'(u), the coefficients α,· are seen to be related to the plant differentials according to «,·= < v , - 4 ' ( u ) w f > 0

= J i

BJhjv

= R{BJh)

R{v}}

(92)

Combining Eqs. ( 8 0 ) and ( 9 2 ) , the pair [AJh) v, ÂJhjv] G ^ χ 01 for a type-one linear plant is related to measurements of the plant differential operator via ÄJh) ν = [R{AJh)

R{v}}, 0 ]

for an arbitrary v G W and h = (u, s) G ^ χ i. Nonlinear Systems. The above analysis which was applied to linear systems can be extended to certain nonlinear systems in establishing a direct relationship between A'(h) and ^l'(h). As examples of this class, attention will be given to the nonlinear plants structured as shown in Fig. 7. The operators Ν χ, N2, and TV are assumed to be zero-memory nonlinear functions, and L is a linear operator which is characterized by an unknown kernel function k(t) with a settling time less than mTd for some integer m. T h e

35

STEADY STATE CONTROL OF NONLINEAR SYSTEMS

forms of the zero-memory functions are assumed to be given in any particular application. However, it is possible that the values of various parameters associated with a given nonlinearity may be determined experimentally. au For example, the nonlinear function Ni(u) = 1 — e~ > defined for u ^ 0, represents a saturation device characterized by a parameter a. The intermediate variables in Fig. 7(a), z(t) and w(t), are not assumed to be accessible to direct measurement.

F I G . 7. Nonlinear plant structures.

Consider the nonlinear plant structure of Fig. 7(a) in which L is a typezero linear operator with settling time less than mTd = mT. T h e steady state plant operator A(u) appearing in the performance functional (4) can be derived analogous to the derivation of Eq. (81): i4(u) = JV 2 [L 0 [tfi(u)]]

(93)

where the linear operator L0 is defined by Eq. (81). T h e derivative operator A'(u) and its adjoint A\u) are found to be dN2,

„,/ χ A

xr

(dNx,

-^< >M-* v

W

x

( u ) ν

\

)·

dNX(

-im-^ (

A

u

)

v

-

x r

du ( ) u

/dN2, L o

(^

\

x ( w ) v

)

(94) where w = L0[Ni(u)], and L0 is given by Eq. (83). Applying the reverse operator R to ^4'(u) ν in Eq. (94) and utilizing the properties of R in Eq. (85), Ä

« ) V }

W

=

ä

{ ^ ( w ) } / : O ( Ä { ^ ( « ) V } )

(95)

#

Let a particular variation element ν = v be defined as

such that Eq. (95) becomes R{A'(n) v*} = R ^

(w)} L0

(w) v )

(97)

36

ALLAN Ε. PEARSON

Combining Eqs. (94) and (97), it follows that

1 dw where the particular variation element v* e is defined by Eq. (96). Equation (98) establishes the relationship between the adjoint operator ^l'(u) and the derivative operator ^4'(u) for a type-zero nonlinear plant structured according to Fig. 7(a). It is necessary to compute the time function segments d

N

U

\

d

a

2

N f

χ

in addition to the plant differential A\\x)\* in order to compute ^4'(u)v. This can be accomplished in an indirect manner by measurements of the plant input and output signals during steady state operation provided the nonlinear functions are invertible. T o illustrate this point, consider the particular nonlinear function forms 1

Nx(u) = tan" aw,

N2(w) = vfi

(99)

which are characterized by the parameters α and β. The value of β is assumed to be restricted to those values for which the function N2 is invertible. Equation (98) in this case becomes 2

1

ν = [1 + ( a u ) ] " R{yWß-»A'(u)v*}

(100)

where 2

ν* = [1 + (au) ]

RivyV-W}

and y = A(u) is the output signal observed over one period of steady state operation. Thus the relationship between A'(u) and A'(u) is given solely in terms of the input-output data and the parameters α and β which characterize the nonlinear forms of Eq. (99). One method for experimentally determining the parameters α and β would be to apply constant level inputs u(t) = C, = const, i = 1,2,3, each over a sufficiently long time interval such that the output has settled out to a constant value A(C{) in each case. Referring to Eqs. (93) and (81) with u(t) = Cif A(C{) = N2 (N^d)

m

\ J k(r) dr)

Eliminating the unknown constant

(i = 1,2,3)

(101)

37

STEADY STATE CONTROL OF NONLINEAR SYSTEMS

in Eq. (101) in the case of the nonlinear functions specified by Eq. (99), it is seen that α and β can be computed from the pair of transcendental equations ι A(CX) _ rtan^aCyiP A(C2) = \\&τΓ α€Λ? 1 1 A(C^) ~ [tan" aC 3 J A(C2) " [tan" aC 2 J ' Consider now the plant structure of Fig. 7(a) in which L is a type-one system and the function Nx(u) is linear, i.e., Nx(u) = u. Under these circumstances it is possible to establish a direct relationship between ^4u(h) and Au(h) for an arbitrary pair h = (u, s) e °1ί χ St. T h e performance functional (16) is the appropriate form in this case for evaluating the steady state system performance. Assuming that the linear operator L possesses a settling time less than mTd= 2mT, the operators A and Β of Eq. (17) take the form B(h) = N2 (*(«>)* + A(oo) f* udr - L t (u))

A(h) = N2[k(oo)s + Lt(u)],

where Lx is defined by Eq. (88). By a similar analysis, using the properties of the reverse operator R as has been demonstrated above, the relationship between Au(h) and Au(h) can be shown to be

hâr J

Ur

(Wi)

(W2)

J

where v,* =

*{f(

W l

)4

*(f( )v }

v 2* =

W 2

and

2

ru

VÎI = k(oo)s + Lj(u),

W = k(cc)s + k(cc) I udr — Li(u) 2

The nonlinear plant structures of Figs. 7(b) and 7(c) also admit to establishing a direct relationship between A'(h) and A'(h). In the case of the plant structure of Fig. 7(b), in which L is a type-zero operator and the performance functional (4) is used to evaluate the steady state system performance, the relationship is given by ν = ν [f

(u) - R {f

(„)}]

+ R{A'iu) * { v } }

If the operator L in Fig. 7(b) is type one, the relationship is given by

^ ^

v

=

v

v

=

-

[^ v

(u)

- {^ )]

( _ u )

(u)

R

+

R { A u { )h

- {^ - )] R

(

u)

+

R

R { B

-

{

{ h )

v

]}

R

{

v

]}

38

ALLAN Ε. PEARSON

For the nonlinear plant structure of Fig. 7(c), the analogous relations are

wk -

N{u)R

^

v

=

[ f

v

+

R

{

W

m

(u)

\M

.

where y = A(u)> and

M,(h)P{v7V(u)} + * ( — m —

(Bu(h)R{vN(-u)}\ N(-u) where y ί = A(u, s), y 2 = B(u, s). Although the above examples do not exhaust the class of nonlinear plants for which it is possible to establish a direct relationship between the adjoint operator A'(h) and the plant derivative operator A'(h), the relationships tend to become more complex when nonlinear plants of a more complex structure are considered. It is necessary in each case to assume a particular form of the various zero-memory nonlinear functions and to be able to measure either directly or indirectly the various corresponding elements 7V(u), dN(u)lduy etc. In addition, any unknown parameters associated with the nonlinear forms, e.g., α and β of Eq. (94), must be determined from separate experiments and computations. Additional examples involving the latter can be found in Pearson (11).

B. Measurement of Differentials

The previous section has demonstrated that it is possible to express the identification of the performance gradient V P ( h w ) corresponding to any particular element hu = h in terms of measurements of the plant differential operators A'(u)v> Au(h)v, Bu(h)v, etc. As was mentioned previously, the image elements of the various plant differentials are to be obtained through the use of finite difference approximations. For example, in the case of

39

STEADY STATE CONTROL OF NONLINEAR SYSTEMS

type-zero plants the first-order approximation to A'(u) ν corresponding to a given u G °M and v G is

A(\x + yv)- A(u)

At λ ΑA γΑ(ύ)ν

= —

—

7

~

In the case of type-one plants the first-order approximations to AJ^i) ν and ^ ( h ) 1 are given by h ) v

=

A(n

v,s)-A(u,s)

+

Y

t

= 1

A(u,s

)-A(u,s) Y

+

y

y

corresponding to the given element h = (u, s) e

v e t andl6£

χ 8% and variations

The question arises as to whether it is possible to obtain a better approximation to the image element A'(h) g than a first-order one, and if an estimate of the resulting error can be made. Kulikowski (8) suggested that higherorder approximations can be obtained by measuring higher-order differf ences J y M ( h ) g and combining them according to the formula m

i—1 J

S y ^(h)g = 2

J

h

V r' ( )g''

(

1 0 2

)

i=l

Equation (102) is an approximation to the exact formula 1

^'(h)g = 2 1

( - l ) ' ^ Ç J y «^(h)g--

(103)

=1

called the Gregory formula (21), which is valid provided A(h) possesses Fréchet differentials of all orders. T h e higher-order differences, each of which is in itself a first-order approximation to the corresponding higher-order Fréchet differential, can be generated as follows : A^A(h)g

= A

v

ASA{h.)g> =

A (

h

)

g

^

h

+

^ -

A

^

â [n A{h)g] v

v

_ 1 p4(h + 2yg) - ^ ( h + yg) y|_

A(h + yg) - ^(h)1

y

y

J

_ A(h + 2yg) - 2A(h + yg) - ^ ( h ) y

2

or, in general, Δγ'Α{\χ) g'' = i . J

( - l ) ' " ' ( \ ) A(h + j y g )

(104)

40

ALLAN Ε. PEARSON

where (*) represents the binomial coefficient. Combining Eqs. (102) and (104),

1 8YA(h)g

= —S

m

1

/'\

1

>

42

( " 1 ) · U(h

(105)

+jYg)

represents an mth order approximation to the image element A'(h)g in terms of measurements of image elements A(h +./yg), j = 0 , 1 , . . . , m. It should be noted that the proper steady state plant operation must be established with respect to each of the input elements h+7'yg before measuring the corresponding image element ^4(h +7yg). T o investigate the error between S y A(h) g and A'(h) g, let A(h + ßg) be represented by Taylor's formula [cf. Eq. (74)]

A(h + ßg) = J] S^( )g* + ^ Y - , ^ h

m + 1

( )g z

m + 1

(106)

where ζ = h + aßg for some a e [0,1]. Substituting Eq. (106) with ß=jy into Eq. (105) and interchanging orders of summation, the resulting equation can be simplified yielding

8 A(h)g Y

= A'(h)g-

t^A^\z)g^

(107)

where ζ = h + ayg for some a e [0,1]. As a measure of the error involved, the norm of the difference between 6 y^ ( h ) g a n d ^ ' ( h ) g i s

\\A'(h)g-

8 A{h)g\\ y

= i2t||^+'(2)g-M||

(108)

For m = 1, Eq. (108) represents the error due to approximating ^4'(h)g by the first-order approximation J y ^ 4 ( h ) g . In the case of nonlinear plants, the structures of which consist of linear operators interconnected with zero-memory nonlinear functions, the n n higher-order differentials A (h) g are zero for η > m if the zero-memory functions can be sufficiently well approximated by polynomials of degree ^m. In these cases it is of interest to note that the measurement of differentials through the use of Eq. (105) can be made without error. In a similar manner the error in approximating the wth-order Fréchet n n n n differential A (h)g by the wth-order finite difference Ay A(h)g can be obtained through the use of Eqs. (104) and (106): +1

| | ^ ( h ) g " - J yM ( h ) g " | | = £ | y | M" (z)g»+>|| ζ = h + ayg,

a e [0,1]

(109)

41

STEADY STATE CONTROL OF NONLINEAR SYSTEMS 2

The second-order differential A"(h)g is of particular interest in the application of the Altman-Newton iterations, Eq. (69), and the recurrence relation based on steepest descent, Eqs. (35) and (72). T h e first-order 2 approximation to A "(h) g is given by Eq. (104) with i = 2, and the resulting 2 error by Eq. (109) with η = 2. Higher-order approximations to A"(h)g can be obtained by squaring the Gregory formula, Eq. (103), and truncating the resulting infinite series, m

t =

l

m

( 1 V + / i+j—2

j =\

J

Using a method analogous to the derivation of Eq. (107), the difference 2 2 2 between A"(h) g and δγ A(h) g for m = 2 can be shown to be 5

A"(h)g* - S y M ( h ) g 2 = ? y M < ( h ) g < + ^ ( z ) g

5

where z = h + ayg for some a G [0, 1], T h e details associated with the above derivations as well as additional error relations can be found in (12). Of course, the attempt to improve the accuracy in measuring plant differentials by means of higher-order differences results in a longer identification period since the proper steady state plant operation must be established anew for each output measurement A(h + jyg). Evidently there is a point of diminishing returns involved, although it is not clear how to specify this point.

V. Multivariable Systems With the basic ideas of the adaptive optimal control procedure for single-input-single-output systems in mind, the extension to plants consisting of a multiple of inputs and outputs can be made with relative ease. T h e major problems are to establish a suitable notation to handle multiple inputs and outputs, and to discuss the identification problem for the general case.

A.

Formulation

The general structure of the plant under consideration is shown in Fig. 8. T h e operators Hl and H2 are vector-valued operators on vector-valued function spaces and are generally nonlinear with memory. T h e plant is assumed to consist of m inputs and η outputs denoted by the column vectors

42

ALLAN Ε. PEARSON (1)

(1)

m

(n)

u(t) = Col[w (*),..., é \t)] and y(i) = C o l [ j ( i ) , . . .,y (t)l respectively. In this section lower-case letters such as v(t), d, s will denote column vectors in Euclidean spaces of appropriate dimension. The bold-faced type such as v, w will denote vector-valued time function segments defined on an appropriate time interval. (a) In general it will be the case that certain inputs, say u^\t),..., w (i), will (a+1) (m) affect the outputs in a finite memory manner while others, w ( i ) , . . . , u (t), will feed directly into pure integrators thus affecting the outputs in the

F I G . 8 . A general multi variable plant structure.

manner of a type-one system. Furthermore, it may be the case that certain inputs of the latter group are such that the integrators they feed are preceded only by zero-memory odd functions. Accordingly, it is convenient to partition the input vector u(t) as follows :

u(t) =

MO x(t)

where the α-vector v(t) includes all inputs which affect the outputs in a finite memory manner. The components of w(t), w^\t)y i= 1 , 2 , . . . , j3, feed directly into pure integrators which are preceded by only zero-memory odd il functions, and the components of x(t), x \t)y i = l , 2 , . . . , y , feed pure integrators which are preceded by nonlinear operators possessing properties analogous to Eq. (21) but suitably generalized for vector-valued operators. The desired output of the plant, yd(t)y is assumed to be a vector of periodic functions with period Td. A scalar-valued performance functional of the form jl'G[u(t),y(t),yd(t)]dt

(111)

is assumed to be given which evaluates the output behavior of the plant and assesses the cost associated with operating the plant over one period of steady state operation. I f all the components of the desired vector yd(t) are constants, it is assumed that Td=Tis chosen independent of the optimization procedure. Otherwise, it is assumed that the value of Td is commensurable with each periodic component of yd(i) in order that a scalar-valued performance functional of the form (111) might be specified.

STEADY STATE CONTROL OF NONLINEAR SYSTEMS

43

On the basis of the formulations which have been presented for single input plants, Section I I , it is to be anticipated that the steady state system performance will depend upon a quintuple of elements of the form h = (v,w,r,x,j)

(112)

where the following notation applies : v={[t,v(t)]\0 - *™ Ψ) [

+
{y) = >Hy) +

y

(129)

The point y0 is assumed to be close to the solution point y, so the left-hand side of Eq. (129) is taken to be zero. A little matrix algebra then gives the Newton-Raphson correction for the next iteration as Δ

γ

=

_ ψ ψ γ

φ

{

ν

ο )

o)

( 1 3

TRAJECTORY OPTIMIZATION PROBLEMS

73

The modified Newton-Raphson method results when this correction is multiplied by a factor C, where 0 < C ^ 1. Thus, dy = CAy = - c [ ^ ]

_

1

-A(jo),

0o) is the value of either Eqs. (91)—(93) or Eqs. (97) and (98). T h e vector ψ(γ0) is nonzero unless y0 is the solution to the problem. The scalar C is the only experience factor required for the method. Usually, the initial guess of y0 is so far from the minimizing point that a small value of C is required (to prevent divergence). As the solution approaches the critical value, it is normally found that acceptable values become larger, until a full Newton-Raphson step may be taken. A scheme for the proper selection of C can be automated on the computer using, for example, simple halving and doubling logic.

B. T h e O p t i m a l N e w t o n - R a p h s o n Method

This scheme is the same as the modified Newton-Raphson method, except that the scalar C of Eq. (131) is chosen to minimize a function of y at each step. T h e function is obtained by multiplying the system (128) by a diagonal weighting matrix W> so that

*Cy)=wtfO0

(132)

and then forming the scalar product to obtain f(y)

= 4>'(y)f(yi)> ^ ^g(y\) f(y) in the direction s may be approximated by the cubic equation 2

f(yo + λί) = a0 + αχλ + a2X

3

+ a2X

(138)

Values for the coefficients are found to be ia =

Suppose that m solutions of Eq. (146), are known to the system (145) for equally spaced values of the parameter b (call the solutions yx,..., j> m), and that the corresponding derivatives (call them yx\ ...,ym') from Eq. (147) have also been found. Then the problem is that of predicting the next member of the family, j m + 1 . Open-type integration formulas are well suited for this task, and many are given in Chapter 6 of (10). In particular, a formula truncated after third differences (the Adams-Moulton predictor equation) is ym+i

=ym + & ( 5 5 V - 5 9 j ; _ t + 37y'm_2 - 9y'm_3)

(148)

where h is the spacing between parameter values. Notice that the present point and derivative and three previous derivatives are required for this equation. Other formulas using less back information are easily derived from the results given in (10). T h e simplest such equation, using only the present point and derivative, is ym+x=ym + hy„;

(149)

As an example, suppose that the parameter is Χλ, which appears in the first of Eq. (94) or Eq. (97). It then appears in the first of Eq. (145), and the vector ^φ|^b is found to be ( ^ ) ' = [-1.0,...,0]

(150)

Since 3i/r/3y is the Newton-Raphson matrix, Eq. (150) and Eq. (147) show that the derivatives dy\dXx are contained in the first column of the inverse Newton-Raphson matrix. I f the parameter b is contained in Eqs. (2) and (3), then ^φ|^b is deter-

{

TRAJECTORY OPTIMIZATION PROBLEMS

77

mined from integration of the nonhomogeneous accessory equations [analogous to Eqs. (99) and (100)], 2

d/dxXd !^ + dt[db) ~ dpdb dpdx[db)

2

+

d H/dp\ 2 'dp W F

dt\db) ~

dxdb

2

dx \db)

(151) dxdp\db)

with appropriate initial conditions. In practice, Eq. (148) is used to predict initial conditions for the next member of the family of optimal trajectories. A predicted trajectory is then generated by integrating the reduced differential equations of the extremals with predicted initial conditions. I f the spacing h is sufficiently small, the predicted trajectory is usually an optimal trajectory, and the next member of the family is sought. I f the predicted trajectory does not satisfy Eq. (145), a Newton-Raphson correction is usually sufficient to produce the optimal trajectory. Notice that more elaborate corrector formulas from (10) may be used for this purpose. T h e method is illustrated in more detail in Section V I I I , C .

VI. Sufficiency Conditions A.

Sufficiency Conditions for a Strong Relative M i n i m u m

T h e sufficiency theorems presented are based upon those of Bliss (1) and Valentine (2). Their forms are considered by the author to be the most convenient from a computational standpoint. T h e problem with corners is not considered, since it does add complications. However, Reid (4) gives sufficiency conditions for problems with corners, and the methods developed here can be extended to this case. [Reid considers a problem of Mayer, which is equivalent to the problem of Bolza through the transformation in ( J ) , pp. 189-190.] T h e statement of the sufficiency theorem follows a preliminary discussion. There is a certain vagueness in the sufficiency theorem, and this is removed in the remainder of the subsection. 1. CONJUGATE SYSTEMS OF SOLUTIONS

Consider the accessory equations (99) and (100), for an unconstrained extremal path with no corners. Let (η, ζ) be a solution to (99) and (100), where η corresponds to the vector dx/da, and ζ corresponds to dp/da (except

78

D. Κ. SCHARMACK

possibly for the initial conditions). Let another solution be denoted by (77, ζ). Then the solutions have the property that

(152)

η'ζ-ζ'η-C

which may be verified by differentiation and insertion of Eqs. (99) and (100) to obtain zero. I f C is zero, then the solutions (77, ζ) and (ή, ζ) are said to be conjugate. There is a maximum of n solutions which are conjugate to each other, and any such system is called a conjugate system of solutions. Many conjugate systems of solutions may be constructed. In particular, if the 2n χ 2n system of solutions to Eqs. (99) and (100) is denoted by ",2(01

kn(0)

π 1 2( 0 ) - | Ι 7

.|π (ί) »22(i)J*

ki(0)

^ 2 2(0)J

\*u(t)

0]

Lo

i\ then two conjugate systems of solutions are (πη, π1λ) and (ττ 1 2, 7 r 2 )2, for 21

^ll

77

77

77

2\ — 2\ 11 = 0>

77 77

'\2 22

77

77

~ 22 \2

= 0

(154)

as verified by using the initial conditions to evaluate the constant matrix. A relationship between the two families of Eq. (154) is found to be 7Τ22πη-π'ηπ2ι=

(155)

I

Any conjugate system of solutions (£/, V) may be expressed in the form [V(t)\

[n21(t)

π22(ή\[ν0\

^

so long as the constant matrices (U0i V0) are chosen to be conjugate, for from Eqs. (154) and (155) it is found that U'V—V'U=

U0' V0-

V0' U0

(157)

[For the systems of equations (154), U0 and V0 are chosen as (7,0) and (0,1), respectively.] Now consider a path which consists of two subarcs, with no corners, with the first subarc unconstrained, and with Gx = 0 over the second subarc. The accessory equations for the second subarc are given by Eqs. (114) and (115). It can be shown that any two solutions (77, ζ) and (77, ζ) satisfy Eq. (152), and that conjugate systems of solutions can be constructed by proper choices of initial conditions at the junction point t = t2. Assume that a conjugate system of solutions (£/, V) has been established for the first subarc from integration of Eqs. (99) and (100) with proper initial conditions. I f the initial conditions for Eqs. (114) and (115) are chosen as the first subarc terminal conditions [U(t2), V(t2)], then a continuous conjugate system of solutions ( [ / , V) for the entire path is established. This result is readily

TRAJECTORY OPTIMIZATION PROBLEMS

79

extended to more complex paths. In the following, the conjugate system of solutions (Uy V) is supposed to be constructed in this manner.

2. A SUFFICIENCY T H E O R E M FOR A STRONG RELATIVE M I N I M U M

Let Condition I be satisfaction of the multiplier rule of Section ΙΙΙ,Α, and let Condition llN' be satisfaction of the Weierstrass condition of Section Ι Ι Ι , Β , in the strengthened form H(ty xypy u) < H(ty xypy U)

(158)

in a neighborhood Ν of the path to be tested for a minimum. Similarly, let Ι Ι Γ be satisfaction of the Clebsch condition, Section I I I , C , in the strengthened form * ' ^ > 0

(159)

Then a sufficiency theorem, based upon (1) and (2), may be stated. A SUFFICIENCY T H E O R E M . Let Ε be an arc without corners satisfying an Eqs. (2)-(4). If Ε satisfies Conditions I, \\N' y d Ι Ι Γ , it is a nonsingular extremal. If Ε has a conjugate system of solutions U(t)y V(t)y with determinant JJ(t) Φ 0,/or which

2Y[U(T)ay£]

- a' U'(T) V(T)a > 0

(160)

for all (£, α) Φ 0 satisfying

then Ε is a strong relative minimum in the sense of Bliss (1). Proof. Condition I establishes Ε as an extremal arc which satisfies the transversality conditions. I f Ι Ι Γ is satisfied, the determinants Rx and R2 of Section I I I , F are nonsingular, so Ε is a nonsingular extremal. Equations (99) and (100) and (114) and (115) are then well defined so that conjugate solutions U(t)y V(t) may be determined. I f the determinant of U(t) is nonzero, and IIN' is satisfied, a field of extremals can be constructed [see (7), Lemma 84.2, and (2) for extension to inequality constraints]. Equation (160) is a second derivative test in this field which, if positive for nonzero (£ya)y establishes £ as a minimum, through Bliss' Theorem 85.1. (Of course, if the problem under consideration has fixed endpoints, test (160) does not apply, since then {ξ, a) = (0,0). T h e theorem then holds with the last condition omitted.)

80

D. K. SCHARMACK

T h e function 2γ in Eq. (160) is determined from the second variation [see (7), p. 227]. T o evaluate, let Q[T,x(T)]=g[T,x(T)]

+

e^[T,x(T)]

(161)

where g and φ are defined in Eqs. (1) and (4), and e is the constant vector in Eq. (22). Differentiating twice gives 8T

2

d Q = [dT,dx']

2

(

[dTdx

dTdx

dT (162)

2

dQ 2 'dx

V

dx

In terms of variations, dx = x(T)dT

(163)

+ 8x(T)

and in the field the terms of Eqs. (162) and (163) are evaluated as dT = t

(164)

8x(T)=U(T)a

Then 2y[U(T)ay

ξ] = -lp\T)x(T)

2

ξ + 2p\T)

U(T)αξ] (165)

+ diQ[t,U(T)a\

where the last term of Eq. (165) is Eq. (162) evaluated with Eqs. (163) and (164). T h e sign of the last term in the first of Eq. (160) is the opposite of the corresponding term in the general formula given by Bliss [see (7), Lemma 85.1]. T h e difference comes from use of multipliers with opposite signs, as mentioned in Section Ι Ι Ι , Β , in the derivation of the expression. Test (160) simplifies considerably for the simple problem of Section I I , C . T h e function g in Eq. (161) is absent, and the functions φ are one of the linear equations (14) and (15). Then the matrix of Eq. (162) is zero, so the last term of Eq. (165) is zero. T h e last of Eq. (160) becomes, for terminal Eq. (14), = 0, i=\,...,r (166) xi(T)$+Ui(T)a with Ui(T) taken as the ith row of matrix U(T). Equation (165) is zero for the terminal conditions (15), for then the last of Eq. (160) reduces to ξ = 0,

t/,(7> = 0

(»=l,...,r-l)

(167)

From a computational standpoint, each path determined by the methods of Sections I I - V is an extremal which satisfies Conditions 1 1 ^ ' and Ι Ι Γ (excluding corners). Condition I is satisfied when the iterations have converged to a solution. It then remains to show that there is a conjugate

81

TRAJECTORY OPTIMIZATION PROBLEMS

system of solutions U(t)y V(t), with determinant U(t) nonzero, which satisfies test (160). This is done by choosing a set of initial conditions U0y V0 for the accessory equations, and examining the determinant of the solution U(t) at a suitable number of points along the path. I f det U Φ 0 at each point (i.e., does not change signs) then the rest of the test, if necessary, is easily performed. T h e choice of U0y V0 is somewhat arbitrary, since there are many ways of imbedding an extremal in a field. However, for some choices it is conceivable that the determinant of U will pass through zero at some point along the path. Moreover, it may be impossible to imbed the extremal in a field, in which case there is no set U0y V0 for which det U φ 0. So the e question is, can a conjugate system of initial conditions U00y ^oo t> determined such that, if det U(t3) = 0 at some point t0 ^ t3 ^ Ty the path cannot be a minimizing path ? T h e answer is contained in the examination of the second variation. 3.

T H E ACCESSORY M I N I M U M PROBLEM AND DETERMINATION OF

U00y

V00

T h e second variation may be written in the form η) = 2γ[ξ2, η(Τ)] + Jl 2ω(τ, η,Vu) dr

(168)

where 2y is defined by Eq. (165) and Eq. (162), except that the variations, analogous to Eqs. (163) and (164), are taken to be

j mti r

$2 = dT(b)ldb,

+ -0{T)

=dx(t,b)ldb, v

(169) =Vu du(t,b)ldb

(170)

[see Bliss (1), pp. 226-234, for derivation of these expressions]. T h e quadratic form 2ω may be written 2

^

,d H, = 1^Τν

,VHX + 2η -^η»

,d*Hr + ηα -^Vu

(171)

T h e function Hx of Eq. (41) is used here, since constrained subarcs are to be included in the analysis [see (2) for extension to this case]. As a necessary condition, the second variation must be nonnegative [see (7), Theorem 80.1], so one is led quite naturally to the problem of finding the minimum value of Eq. (168). Constraints for this problem include linearized versions of the differential equations

î" ï"=° +

172

82

D. Κ. SCHARMACK

and on constrained subarcs, say one for which Gx = 0,

The end conditions are = 0,

(tv 0)

= 0

(174)

for the initial point, where iji(t0) = dt0(b)ldb, and dT

dx

2 T

3Λ: Τ

V(T) = 0

(175)

at the terminal point. The problem of minimizing Eq. (168) subject to the conditions in Eqs. (172)—(175) is the accessory minimum problem. With slight modifications [see (1), Lemma 81.3] it is a problem to which the multiplier rule of Section ΙΠ,Α may be applied. It is found that the reduced differential equations of the extremals are Eqs. (99) and (100) for unconstrained subarcs, and Eqs. (114) and (115) for constrained subarcs. [They are therefore called the accessory equations. The variables of Eqs. (99) and (100) and (114) and (115) are taken to be (77, ζ), rather than dx/day dp/da in the present discussion.] The transversality conditions are found to be ,

€ *(*o) + e 0 = 0,

£(*) = - €

(176)

for the initial point, and

dx

(177)

at the terminal point. Any solution ( r ^ , £j) of the accessory equations which satisfies Eq. (174) and Eq. (176) is necessarily a linear combination of the solutions whose initial conditions are ( 0 , — / ) . Let these solutions be denoted by r(t),s(t), so that = r(t)a, ii(t) = s(t)a (178) Vl(t) Notice that — r(t)y —s(t) are the conjugate solutions πί2(ί)> π12(ί) in Eq. (153), and are also identified as dx/dp0, dp/dp0 in Section I V , B . It is, in general, impossible to satisfy the endpoint conditions, Eqs. (175) and (177), with the solutions of Eq. (178). However, the (n + r + 1) equations of (175) and (177) in the (2n + r + 1) variables [η(Τ), ζ(Τ), ν, ξ2] may be solved (under appropriate conditions) to yield n linearly independent solutions. These determine terminal conditions v(T), w(T) for a conjugate system of solutions [refer to (7), pp. 243-249]. Any solution (η2, ζ2) of the

83

TRAJECTORY OPTIMIZATION PROBLEMS

accessory equations satisfying Eq. (175) and Eq. (177) may be expressed in the form = v(t)b, l2(t) = w(t)b (179) V2(t) These solutions do not, in general, satisfy the initial conditions, Eq. (174) and Eq. (176). Thus two sets of solutions for the accessory minimum problem have been conditionally established which satisfy end and transversality conditions at one end, but not the other. J2 may be evaluated with these solutions at an arbitrarily selected point £ 3 , ^ 0 ^ h ^ T, for a path defined by Φ) = ηι{ή = τ(ήα, (ή = η2(ή

η

t 0^ t ^ t 3

= v(t) b,

= r(t3) a =

t3^t^T

(180)

v(t3)b

T h e resultant expression is J2 = a' [r\t3) w(t3) - s'(t3) v(t3)] b

(181)

which must be nonnegative for all (a, b) satisfying the last of Eq. (180). Now it was noted after Eq. (178) that r(t) = - 7 7 1 2 ( 0 ,

s(t) = -π22(ή

(182)

If a constant symmetric matrix Κ can be found such that Φ) = τη iW + ^12(0 K,

w(t) = n2i(t) + π22{ί)Κ

(183)

then the matrix of Eq. (181) is the identity matrix, as seen from Eqs. (154) and (155), so Eq. (181) reduces to J2 = a'b. T h e symmetry of Κ is required so that v, w form a conjugate system. Then the choice U(t) = r(t) + v{t\

V(t) = s(t) + w(t)

(184)

produces a conjugate system, with initial conditions t/oo = / ,

V00 = K-I

(185)

which has the properties desired for the sufficiency theorem. For as shown by Bliss, if det U = 0 at some point t3y it follows that there is a nonzero vector a for which r(t3)a = -v(t3)a (186) The choice b = —a in the last of Eq. (180) gives J2 the negative value —a'a, so the path cannot be a minimizing path. T h e problem then reduces to the determination of matrix K. For simplicity, the problem of Section I I , C with terminal conditions (14) is assumed in the following. (This restriction is removed in a later section.)

84

D. Κ. SCHARMACK

In their most general form, the end and transversality equations (91)—(93) may be written Xi(Tfx0,p0)-

Xi = 0,

Pj(T,Χο,Ρο)

= 0,

i=

l,...,r

; = r + 1,...,η

(187)

#i(*o>/>o) = 0 A solution (Typ0) for given x0 is assumed to have been found. If the NewtonRaphson matrix, made up of the partials of Eq. (187) with respect to (Typ0)y is nonsingular, then by the implicit function theorem there is a neighborhood of the point x0 in which Eq. (187) has the solution (188) In particular, the linearized version of Eq. (187) reads

(189)

This may be expressed as the matrix equation (190) in which Β is the Newton-Raphson matrix and A is the ( n + l ) x n matrix of partials with respect to x0. It follows that (191) and Κ is identified as the lower η χ η matrix dp0/dx0. T o show that this is so, and that the system (183) satisfies the accessory end and transversality conditions, consider the variations

(192)

85

TRAJECTORY OPTIMIZATION PROBLEMS

Then, since V(0) = dx,

S(0)

dx0

dx0 = dp0

(193)

Eq. (189) may be rewritten it(T)i

+ Vi{T)

Ρ;(Τ)ξ+ζ;(Τ) -ρ'(0)η(0)

+

j=

χ'(0)ζ(0)

-Ρ'(Τ)η(Τ)

r+\,...,n + χ'(Τ)ζ(Τ)

(194) = 0

The last expression follows, since it is constant in time. This may be verified by differentiation and evaluation in terms of Eqs. (42) and (43) and Eqs. (114) and (115). On the other hand, the accessory and transversality conditions, from Eqs. (175), (177), and (165), are found to be (T)t

Xi

+ Vi(T)

ΜΤ)ξ+ζ;{Τ) -ρ'(Τ)χ(Τ)ξ-ρ'{Τ)η{Τ)

+ viXi(T)

= 0

= 0

0' = r + l , . . . , « )

= 0

(i summed)

(195)

T h e first and third of these are the same as the first two of Eq. (194), and the second of Eq. (195) can always be satisfied. T h e last of Eq. (194) results when the second and third of Eq. (195) are multiplied by X{(T) and Xj(T), respectively, summed, and added to the last. Thus, the variations (192) satisfy the accessory end and transversality conditions. Then v(t), w(t) is a conjugate system of solutions, and it follows that K, determined from Eq. (191), is a symmetric matrix. These arguments, together with others presented below, justify the following : T H E O R E M . Let Ε be a path, without corners, satisfying Conditions I and Ι Ι Γ . If the Newton-Raphson matrix is nonsingular, Ε is normal, and there exists a symmetric constant matrix K, determined by end and transversality conditions and the fundamental solution matrix (153) ofthe accessory minimum equations, such that U(t) and V(t) in the sufficiency theorem may be chosen as

U(t) = nn(t)

+

nn(t)[K-I],

V(t) = n2l(t) +

n22(t)[K-I],

If det U(t3) = 0 at some point t3, t0^t3^ path.

K-I

(196)

T, then Ε cannot be a minimizing

86

D. Κ. SCHARMACK

According to (7), pp. 230-231, the accessory minimum problem has order of abnormality q if there exist q linearly independent sets of constants and solutions of the accessory equations of the form η = 0, ζ(*), e, ν which satisfy Eq. (176) and

m - v ' | * < r ) = 0,

,'[! g(7W)]=0

(197)

+

If 9 = 0, the accessory problem is normal. Furthermore, the accessory problem and the original problem have the same order of abnormality [refer to (7), Lemma 81.1]. For simplicity, again consider terminal equation (14). Then Eq. (197) simplifies to i = l,...,r ί,·(Γ) = ο, ^*ί(Γ)-0,

> = r + Ι,.,.,η

(198)

(summed over i)

Any solution for which η = 0 is necessarily a linear combination of the solutions 77-12(1), π 2 2( ι ) of the form iiW = - w i 2 ( 0 « .

(199)

£(i) = - * 2 2 ( 0 «

It follows that if 7 τ 1 2 ( Τ ) is nonsingular the problem is normal. For then η{Τ) = 0 requires e = 0, so from Eqs. (198) and (199) it follows that ζ(Τ) = 0 and ν = 0. Hence, there are no nonzero solutions of the form required for abnormality, so the problem is normal. On the other hand, Eqs. (194) and (199), with ξ = 0, give [*i(T)] 0

• 0 •

Κ2/Γ)] [ - ^ Γ ) π 1 2( Γ ) + * ' ( 7 > 2 2( Γ ) ] .

0

(200)

—€_

where πί2ί(Τ) is the ith row of πί2(Τ) and the brackets indicate the matrix made up of the rows of rr12(T) for i= 1 , . . . , r. A similar interpretation holds for the vector x(T), and p{T), π22(Τ) for j = r + 1 , . . . , n. Equation (200) implies satisfaction of Eq. (198) when η(Τ) = 0. However, the last row of the matrix may be evaluated at t = 0 through the last of Eq. (194), and in this form it satisfies Eq. (176) with € 0 = 0. Equation (200) may then be written '[*i(T)] [πί2ι{Τ)Υ • 0 ' 0 (201) [PAT)] [ π 2 2, ( Γ ) ] e 0 x'(0) T h e matrix of Eq. (201) is the Newton-Raphson matrix, which is nonsingular by hypothesis. Hence « = 0, and the problem is normal.

TRAJECTORY OPTIMIZATION PROBLEMS

87

B. A n " A b s o l u t e M i n i m u m " Test

The test given here does not necessarily establish global sufficiency. It does, however, allow a large region of solution space to be examined for other solutions to the optimization problem. T h e method is not applicable to fixed endpoint problems. T h e idea behind the test is simple: Replace one of the transversality conditions by a new terminal equation in which a parameter is included. Obtain a set of solutions to this problem, as functions of the parameter, and examine the set for satisfaction of the omitted transversality condition. T o illustrate, again consider terminal surface equation (14), with r < n. A solution satisfying the associated equations (91)—(93) is assumed to have been found. T h e first of Eq. (92) is replaced by the new equation xr+1(T,po)-Xr+1

(202)

=0

in which Xr+i is initially the solution value of xr+l(T,p0), system of equations is xi(T,p0)-Xi pj(T,p0)

= 0,

i=1

= 0,

j = r + 2,...,n

and the new

r +1 (203)

H(Po) = 0 If the Newton-Raphson matrix associated with this system is nonsingular, Eq. (203) has the solution T=T(Xr+l),

Po=Po(Xr+i)

(204)

in a neighborhood of the solution values. T h e methods of Section V may then be applied to obtain solutions as functions of the parameter Xr+X, as long as the Newton-Raphson matrix remains nonsingular. These solutions may then be examined for satisfaction of pr+x(Xr+x) = 0. T h e method is illustrated in more detail in Section V I I I , C .

VII. Extensions of the Method A.

Extension to the General Problem

Thus far, the initial value method has been confined to the simple problem with terminal conditions described by one of Eqs. (14) and (15). T h e objective here is to extend the method to the problem of Section ΙΙ,Α. As shown in Section I I I , F , the solutions depend functionally upon (t,p0)

88

D. Κ. SCHARMACK

and, at the terminal point, are functions of the η + 1 variables (T,p0). The end and transversality conditions, Eq. (4) and Eqs. (25) and (26), may then be written

)] = ο (

(205)

0

0

in which the omitted arguments of the partials of g and φ are [Γ, x(Typ0)]. This is a system of (n + r + 1) equations in the (n + r + 1) unknowns (Typ0ye). In the simple problem it was possible to eliminate at least r of these equations. This is generally impossible for the problem considered here, although it may be possible to reduce the dimension of Eq. (205) to some extent. In any event, if the initial value problem is redefined as that of finding a solution (Typ0y e) of Eq. (205), then it is found that the theory goes through essentially as before. T h e partials of x(Typ0)y p(Typ0)y and H(p0) are determined as in Section I V , B , and the Newton-Raphson matrix, now defined as the partial derivative matrix of Eq. (205) with respect to (Typ0ye)y has the form 9φ5χ(Τ) dx dp0 dTdx 2

|_ [dT

c dTdx

2

d Q dx(T)~ 2 dx dp0

W )

x ( O)

0

+ dTdx

dp0 _

dx

(206)

dT

with Q defined by Eq. (161). I f ν of Section V is redefined as the vector (Typ0ye)y and φ is taken as Eq. (205), the results of that section hold for the expanded problem. Furthermore, it can be shown that the theorem of Section VI,A,3 is true, by suitably extending the arguments presented there.

B. The Bounded State Coordinate Problem

T h e objective here is to add inequality constraints of the form G(x) > 0

(207)

to the problem stated in Section I I , C , and to develop a method for numerically solving the resulting problem. Only necessary conditions are considered.

TRAJECTORY OPTIMIZATION PROBLEMS

89

1. NECESSARY CONDITIONS

A set of necessary conditions for this problem has been known for some time [see (11)-(14)]. They are stated here with one control function and one inequality constraint [Eq. (207)] assumed for simplicity. Generalization to more control functions and more inequality constraints is readily accomplished. It is also assumed that the time derivative

0 =

dG dx

,_dG dx

(208)

contains the control function explicitly. T h e case where higher derivatives are required to involve the control function is treated in (11), (14). Now a constrained subarc is one over which inequality (207) is an equation. A necessary and sufficient condition for G to be zero over such a subarc is that ύ be identically zero over that arc [refer to (13)]. This condition is included in a new Hamiltonian, defined by Ηι = Η + μΟ

(209)

where the new multiplier μ is identically zero over unconstrained subarcs. Thus, ô replaces G in Eq. (41), and the equations of the extremals are Eqs. (42)-(44). These are the same as Eqs. (17) and (18) and (21) in the multiplier rule, and the remainder of the rule holds (with ( ? ) , except that the multipliers may be discontinuous at junction points between constrained and unconstrained subarcs. I f there are only two subarcs, the condition G = 0 may be treated as either an initial condition or a terminal condition, depending on the ordering of constrained and unconstrained subarcs, and the multipliers will be continuous over the path. I f the ordering is constrained-unconstrained-constrained for a three-subarc path, the multipliers will be continuous for the same reason. All other cases with three or more subarcs will produce discontinuous multipliers. It is well known (12) that the discontinuities take place at one end of the constrained subarc and that the multipliers are continuous at the other end. It does not matter which end has the discontinuities, so the initial point t2 is chosen here. T h e necessary conditions at t2 (the analog of the Weierstrass-Erdman corner conditions) then read +

p (h) H

=p-(t2)+

= H-

G-(t2) = 0

(210) (211) (212)

90

D. Κ. SCHARMACK

where superscripts plus and minus indicate limits from the right and left, respectively. [If G contains t explicitly, then the Hamiltonian is discontinuous by the amount ν dG(t2)/dt.] Notice that when Eq. (210) is substituted into the left-hand side of + Eq. (211), the coefficient of ν becomes G , Eq. (208), which is zero by definition. Equation (211) is thus independent of vy and contains only p~ values of the multipliers. This equation can usually be reduced to an equivalent necessary condition (see Section V I I I , D ) . T h e necessary conditions of Weierstrass and Clebsch must also hold (Sections Ι Ι Ι , Β and I I I , C ) , with G in place of G.

2. BASIS FOR COMPUTATIONAL SCHEME

In the optimization problem treated in previous subsections, the extremal solutions were functions of the independent variable (call it t) and the multiplier vector p0. The constant ν in Eq. (210) cannot be determined from the necessary conditions and, hence, becomes an additional parameter for the solutions. The extremal solutions thus have the functional forms x = x(t,p0yv),

P=p(t,po,v)

(213)

Each time the multipliers are discontinuous another constant ν is introduced. Unless one has a prior knowledge of the number of constrained subarcs, the problem could have a variable number of variables. This gives no theoretical difficulty, but the practical bookkeeping problems in a digital computer program could become unmanageable. In what follows, then, it is assumed that the optimal path consists of three subarcs, ordered unconstrained-constrained-unconstrained. T h e necessary conditions at the terminal point give (n + 1) equations in the (w + 2) variables (Τ,ρ0,ι>). The conditions (211) (or equivalent) and (212) determine the point t2 and give an additional equation in (Typ0,v). T h e problem is thus one of determining the solution of (n + 2) equations in (n + 2) unknowns and the methods of Section V may be used to find the solution. Partial derivatives of the solutions with respect to p0 are obtained as before, by integrating the accessory differential equations. An additional column in the solution matrix is reserved for partials with respect to v. Initial conditions for this solution are obtained by differentiating Eq. (210) and noting that x(t2) is independent of v. T h e method of solving the problem is illustrated in Section V I I I , D .

TRAJECTORY OPTIMIZATION PROBLEMS

91

VIII. Examples A.

A Simple Analytical Example

The following problem was given in (15). It is: minimize J

(214)

0

subject to differential equations and initial conditions Xl = X2y

x2 = -xi +

*l(0) = #io,

u,

x2(0) = x2o

(215)

and terminal conditions T-k

= Oy

(216)

(T) = 0

Xl

The problem falls within the framework of Section I I , C , with terminal conditions of the form (15). T h e Hamiltonian, from Eq. (23), is 2

H = u +p1x2+

p2(-xx

+ u)

(217)

so the Euler-Lagrange equations, from Eqs. (43)-(44), are P\=Pi>

p2 + 2u = 0

p 2 = ~Pu

(218)

The last of Eq. (218) is Eq. (47) for this problem, and so the form (48) is (219)

u = -p2j2

This control satisfies both the Weierstrass and Clebsch conditions in their strengthened forms (equality excluded). T h e reduced differential equations of the extremals [Eqs. (50) and (51)] are x\

=

x2

x2y

=

(220)

—X\ — Pi\2

and the first two of Eq. (218). They have the solutions cos t -sin t l-sint

sin / cos l t\[p20\

X\(t)~l_[ cost x2(t)\ [—sin t

sin£~|pc10] cos t\ L^2oJ

Ip2(t)\

[

in(* c o s * - s i n * ) 4.

(221)

tsmt

"|Γ/>ιο"|

^22\

(sin* + *cosJ)JL/>2oJ

-tûnt

T h e terminal and transversality conditions, Eqs. (97) and (98), are (k cos k — sin k) 4 —sinÄ

kûnk"] P\o 4

COSA

L/>20J

Xxocosk + #20 sin Λ 0

(223)

92

D. Κ. SCHARMACK

which have the solution p10 = Acosk>

p2o =

Asmk (224)

4(JC 10 cos k + #20 sin k)

A

(k — sin k cos k)

Substitution of Eq. (224) into Eq. (222) gives the optimal path in terms of (ty k,Χ\ο>Χιο). Furthermore, the value of Eq. (214) is found to be 2

2

/ = è[/>?o(* - sinkcosk) - 2pi0p20sm k

+ p 20(k + sinkcosk)]

(225)

The accessory Eqs. (99) and (100), from Eq. (220) and the first two of Eq. (218), are dx.

0

1

0

0

-1

0

ο

-i

Hâ dx2 d_ la dt 3pi dp2 _3a_

0

0

0

0

0 - 1

~dx, la dx2 lä

1

dp, da

0

dpj da

(226)

These may be solved with appropriate initial conditions; however, the partials are obtained more directly by differentiating Eqs. (221) and (222). Thus, dx(t) dx0

dx(t)

1 W

Γ cosi L-sinr

• π (ί) =

sinH cos t]

(t cos t — sin t) 4

12

isinf

isini (sin t + t cos t)

(227)

= w (i) = 21

dxdp 0 - = 7T22\t) = dp o [—sin t

cos t

The matrix of Eq. (223) is now identified as the Newton-Raphson matrix, which is nonsingular for all k > 0.

93

TRAJECTORY OPTIMIZATION PROBLEMS

The matrix Κ of Section VI,A,3, from differentiation of Eq. (224), is 2

Κ

dpo:

4 cos Λ k — s'mk cos k

4 sin k cos k k — sin k cos k

4 sin k cos k k — sin k cos k

4 sin k k — sin k cos k

2

(228)

Κ might be used to construct a conjugate system of solutions, as in Eq. (196). However, this leads to complex expressions, and it is easier to prove sufficiency assuming Κ = 0. Thus, consider υ(ή =

πη(ή-πί2(ή cos t +

(t cos t — sin t) 4

-sin t

tsmt sin t + •

tsint

(sin t + t cos *)

cos* + —sin —cos

(229)

(230)

T h e determinant of Eq. (229) is 2

2

, T T, . . ί ί — sin i det £/( sin t for all t > 0, it is always positive. Test (160) reduces to > 0

[a,b]U'(k)V(k)

(232)

for nonzero (a, b) satisfying U(k)

a A

"0"

(233)

The first of Eq. (233) is the linearized constraint equation (216), whereas the last is an equation added for convenience. Certainly α can always be chosen to satisfy the equation. Equation (233) may be inverted and substituted into Eq. (232), with the result [0,«][-F(*)]

U~\k)

> 0

(234)

94

D. Κ. SCHARMACK

Thus, the lower right-hand element of the product matrix must be positive to complete the sufficiency proof. One easily verifies that £ _|_ (sin k + k cos k)

/ . ,

COSA +

U(k)

^

sin A +

kûnk

s

i

n

A sin/Λ Ä

+

__j

, v(k cos k — sin k); cos * + 4 (235)

so that the required matrix element is 1

[1 + l(k — sin k cos k)] (236) det u(ky which is positive for all k > 0. Thus the solution is a strong relative minimum according to the sufficiency theorem of Section VI,A. Since Eq. (223) has only one solution, the path is also the absolute minimizing solution. a22 =

B. A Problem with a Constrained Subarc

Consider the problem of minimizing J =

j \ d

(237)

o

subject to the differential equation χ = —χ + w,

*(0) = x0

(0 < x0 < J )

(238)

inequality constraint 1 - u2 > 0

(239)

and terminal condition x(T)-X=0

(± terminal surface equations are

with the constants XX = 1650 ft/sec, X2 = 75,530 ft, and X3 = 979 statute miles. Note that the final flight path angle and terminal time are left unspecified.

100

D. K. SCHARMACK

The Hamiltonian may be written

«ι = 4 + P'f+ M " i - « ) 2

(287)

2

in which p is the four-dimensional multiplier vector and / represents the right-hand side of the system (282). T h e Euler-Lagrange equations, where zero terms have been omitted, then read •

3?

tfi

3/i

3/4

3/3

sfi 3f 3A a/ -P2=pi^+P24^+P3^+P4^ 3y 3y ογ &y 4

2

3 Î

+

^ 3 F

+

^ W

+

/ooox (288)

4

^ 3 Î

(p4=p4o)

^ 4 = 0,

1. T H E UNCONSTRAINED SUBARC

The multiplier /x is zero here, so Eq. (289) is used to determine the control function. After the substitutions have been made, the resulting formula is t

a

Z^2£_2

n/ i =

)

CDLpxV

;v

and u is centered about zero by the constraint (285) : -ux^u^

(291)

ux

The minimum-principle equation is —pi vCDL cos u+p2CLO

sin u ^ —pi vCDL cos U + p2CLO

sin U (292)

in which U is any admissible value in the range (291). T h e left-hand side of Eq. (292) may be considered as a dot product, and the choice of a unit vector (cosw, sinw) which has minimum dot product with the vector (-p\vCDL,p2CLO) is sin u = -CLOp2[(CLOp2) cos u = CdlPI

2

(C Pi

v) ]' '

+ (CdlPI

v) ]-"

+ 2

v[(CLOp2)

DL

2

2

1 2

2

(

2

9

TRAJECTORY OPTIMIZATION PROBLEMS

101

This is parallel but in the opposite direction. Then, from the signs of px and/> 2, assuming CLO negative, it follows that: If

p2 = 0

and

px > 0,

then

u= 0

p2 > 0,

px > 0,

0 0,

px
0,

—πβ < u < 0

Pi < 0,

/>! = 0,

u = -π β

p2 < 0,

^ < 0,

—π < u < —πβ

The strengthened Weierstrass and Clebsch conditions hold, so long as px and/> 2 are never simultaneously zero. T h e subarc ends either when Eq. (285) becomes zero, or when the stopping condition, the first of (286), is satisfied.

2. T H E CONSTRAINED SUBARC

Let φ be the angle defined by Eq. (293) and the sign conventions given by Eq. (294). Then substitution into the minimum-principle equation (292) gives cos (φ-u)^

cos (φ - U)

(295)

which is satisfied if u and φ < ±π have the same sign. T h e condition φ = π indicates a bang. Furthermore, substitution into Eq. (289) and some rearrangement gives

"- - f£

K 296

Since u and sin (φ — ύ) have the same sign, μ < 0, as required. The strengthened Weierstrass and Clebsch conditions hold, and the control function is continuous at the junction between constrained and unconstrained subarcs. Thus μ must start and end with value zero, since at such points ιι = φ. Then the terminal surface is either μ = 0, provided μ Φ 0, or the stopping condition.

102

D. K. SCHARMACK

3. ACCESSORY EQUATIONS AND CORNER CONDITIONS

T h e accessory equations may be written in the form

0 0 0 0 0 0 0 0 0 0 0 0 dh 0 0 0 0 0 di ii/ 0 0 dy 0 de 0 de 0 0 0 0 0 0 0 0 where η · and ζ , i,j = 1,..., 4, are the elements of the dx/dp Vu

Vu

' Vi dv

Vi dy

dfi dv

dfi dy

dh dv

dh dy

V*

dv2 dH 2

dv

2

dH

dU dy 2

dH dv dy 2

dH dy

dv2 dH dvd£

dh dPx

dh dpi

Vu

dh

dh dpi

dh dpi

•ην

να

dh

2

£2/

dh dî

2

dH

dvdi

2

dydi

dfi dv

dh dv

dh dv

dh dy

dh dy

dh dy

dh dy

dh

dfi dS

(297)

2

dH

dydi

dH

dh dv

2

dH

dh d$

hi

and dp/dp0 ί} ϋ 0 solution matrices, respectively, if Eq. (109) is used for initial conditions. For the unconstrained subarc the partials required in Eq. (297) are those of Eq. (282) and Eq. (288) with control given by Eq. (290). These are in general quite complex, and, except for the zero elements shown, are not reproduced here. Over the constrained subarc the control is a constant, and several of the partials simplify. In particular, the partials of fx and f2 with a re respect to px a n d p 2 zero. Corner points occur on constrained subarcs at points tx when the angle φ becomes ± 7 7 , or, equivalently, wherep 2 = 0 with/)! < 0. At these points the control switches signs, which means that the second of Eq. (282) is discontinuous. All components of Eq. (288) are continuous, since p2(tx) = 0. Then, according to Eqs. (123) and (124) and Eq. (127), only the second row of dx/dpo is discontinuous at t = tx, with 1,...,4

(298)

and superscripts minus and plus signify limiting values from the left and from the right, respectively. Notice that O0(tx)y from Eq. (40), is ßo(*i) = - 2 y - ( ' i ) / > 2 ( * i ) This must be nonzero.

(299)

103

TRAJECTORY OPTIMIZATION PROBLEMS

4. O P T I M A L TRAJECTORY COMPUTATIONS

T h e terminal equation (286) is of the form (14), so the corresponding equations (91)—(93) are v{T,po)-Xl

= 0,

£{T,pu)-X2lR p2(T,p0) = 0,

= 0,

ζ(Τ,ρο)-Χ3

= 0 (300)

H(po) = 0

T h e modified Newton-Raphson equation (131) for this system is ~dT~ dpi. dpi. dp3o

Vu(T) =-C

-1

Vu{T)

~v{T)

V32(T)

V33(T)

ζ(Τ)

ζ{Τ)-Χ3

P2(T) 0

0

/i(0)

Cl2(T)

l2l{T)

W)

W)

0=ig^ » >

fl

2+c

2

1/2

302

The unconstrained maximum value is 20.5 g's. T h e flight time increases (Fig. 3) which is a consequence of the lengthening skip. This is apparent in the velocity curves (Fig. 4) which tend to level out over the skipping

105

TRAJECTORY OPTIMIZATION PROBLEMS

240

100

80

60

U

I

I

I

I

I

I

I

2.4

2.8

3.2

3.6

4.0

4.4

4.8

5.2

6

RANGEC ( F T x l O " )

F I G . 1. Altitude versus range for several optimal trajectories, parameter u\.

portions of the trajectories. T h e flight-path angles (Fig. 5) also show the deeper dive and higher skip. All these curves apparently pass through a common point, corresponding roughly with the bottom of the first dip (see Fig. 1). T h e convective and radiative heating rates are displayed in Fig. 6. They peak higher, and fall off faster, as the constraint is relaxed. T h e total heating rates of Fig. 7 have the same characteristics, and show that even though the peaks are higher the enclosed area becomes smaller.

106

D. K. SCHARMACK

γ

u,« \eor-~ 55·

45"

35·

25

e

>i

Λ2

5·

y-35

e

55*

\^\

180

24

2.Θ

3.2

3.6 RANGE ζ

4.0

6

4.4

4.8

52

(FT χ Ι Ο " )

F I G . 2. Sensed acceleration versus range for several optimal trajectories, parameter u\.

The optimal control functions are displayed in Fig. 8. When the 16-deg trajectory (which seems to be in a category of its own) is excluded, the control curves tend nicely to the unconstrained trajectory curve. They all have a "bang" which goes towards the endpoint as the constraint is relaxed and, in the limit, produces the — 180-deg value of the control function (the angle φ goes to —180 deg at the bang). T h e first portions of these curves show that the trajectories are forced into the atmosphere, since positive control corresponds to negative lift. Small values of the control also correspond to maximum drag, so maximum energy is dissipated. Before the

107 FIG. 3. Range versus time for several optimal trajectories, parameter u\.

108

D. K. SCHARMACK

FIG. 4. Velocity versus range for several optimal trajectories, parameter u\.

bottom of the dive, the control functions all pass through zero and then on to the maximum lift condition (—90 deg for the 180-deg optimal). T h e positive lift is required for ranging purposes and is maintained for the remainder of the re-entry process. T h e terminal conditions following Eq. (286) were then rounded out to X2 = 75,000 ft and X3 = 1000 miles with a few additional Newton-Raphson iterations. T h e original values (75,530 feet and 979 miles, respectively) were made necessary by the initial conditions and the almost ballistic constraint = 16 deg. T h e resulting trajectory is displayed in Figs. 9-13 as the first

TRAJECTORY OPTIMIZATION PROBLEMS

109

FIG. 5 . Flight-path angle versus range for several optimal trajectories, parameter u\.

member of a family of optimal trajectories for which the terminal range is the parameter. 5.

RANGE EXTENSION

T o illustrate the use of the predictor scheme of Section V , D , consider the problem of extending the terminal range of the unconstrained optimal trajectory. T h e differential equations of the extremals are Eqs. (282) and (288), with the control given by Eqs. (290) and (294), and the accessory

110

D. K. SCHARMACK

FIG. 6 . Convective and radiative heating rates versus range for several optimal trajectories, parameter u\.

equations are Eqs. (297). T h e boundary conditions are Eqs. (286), where X3 is the parameter. T h e stopping condition for the integrations is now the attainment of a desired terminal time Τ (updated after each iteration), so the zero element of the last vector in Eq. (301) is replaced by the term [v(T) — Xx]. T h e change in the stopping condition was made since it is somewhat easier and faster to stop at a given value of Τ than it is to interpolate for v(T) = Xx. T h e stopping condition for iterations is that the maximum ratio of variable change to variable be less than a specified 5 constant (usually 10~ ). Starting trajectory initial conditions are v0 = 35,000 ft/sec, y 0 = —5.75 deg, h0 = 400,000 ft, and ζ 0 = 0» and terminal conditions are Xx = 1650 ft/sec, X2 = 75,000 ft, and X3 = 1000 miles. T h e next three trajectories were

TRAJECTORY OPTIMIZATION PROBLEMS

111

RANGE ζ ( F T χ 10**)

F I G . 7. Total heating rate versus range for several optimal trajectories, parameter U\.

obtained using the optimal Newton-Raphson method of Section V,B, with terminal range increments of 10 miles. Succeeding members of the family were obtained using the Adams-Moulton predictor equation (148). T h e variable y of Eq. (148) is one of (T,p0)f h is the range spacing (10 miles), and the derivatives of (T,p0) with respect to terminal range are contained

112

-Ι80 2^

D. K. SCHARMACK

34

36

3.8

40

4 2

RANGE C (FT χ 10"*)

F I G . 8 . Control function u versus range for several optimal trajectories, parameter u\.

in the third column of the inverse matrix of Eq. (301). T h e values of (Typ0) and the derivatives were stored for the first four trajectories, and used to predict (T,p0) for the fifth optimal path. When this path (corrected, if necessary, by Newton-Raphson changes) was obtained, the sixth path optimal values (Typ0) were predicted on the basis of the data from paths 2 - 5 . The range was extended to 2020 miles by this process. Most of the intermediate predicted values of (Typ0) produced optimal trajectories, which shows the power of the predictor scheme. As the upper limit on range was approached, prediction gradually worsened, and it is doubtful that range could be extended much farther for the vehicle and initial conditions considered. Five of the family of trajectories are plotted in Figs. 9-15. Figure 9 shows that the first dive into the atmosphere becomes shallower, as range is

FIG. 9. Altitude versus range for unconstrained optimal trajectories, parameter ζ(Τ).

TRAJECTORY OPTIMIZATION PROBLEMS

113

FIG. 10. Control function u versus range for unconstrained optimal trajectories, parameter ζ(Τ).

114 D. K. SCHARMACK

( 930) 'n N0I13NÎ1J 108.

FIG. 11. Velocity versus range for unconstrained optimal trajectories, parameter ζ(Τ).

TRAJECTORY OPTIMIZATION PROBLEMS

(SdJ) Α1Ι30Ί3Λ

115

FIG. 12. Sensed acceleration versus range for unconstrained optimal trajectories, parameter ζ(Τ).

16 D. K. SCHARMACK

FIG. 13. Flight-path angle versus range for unconstrained optimal trajectories, parameter ζ(Τ).

TRAJECTORY OPTIMIZATION PROBLEMS

(030) VWWVO

117

118

D. K. SCHARMACK

£5 28,000^-

1000

1250

1500

1750

2C

TERMINAL RANGE (MILES)

F I G . 1 4 . Total heat versus terminal range for unconstrained optimal trajectories. 650

350

I 1000

1 1250

I 1500

I 1750

I 2000

TERMINAL RANGE (MILES)

F I G . 1 5 . Terminal time versus terminal range for unconstrained optimal trajectories.

extended, and that the skip which follows becomes higher and longer. This behavior is caused by the control function (Fig. 10) which leaves the negative lift region (u > 0) and goes to the maximum lift condition (u = —90 deg) earlier in the flight as range is extended. Less energy is lost on the first dive, as can be seen in the velocity curves of Fig. 11, and the

119

TRAJECTORY OPTIMIZATION PROBLEMS

decrease in the first acceleration peak of Fig. 12. T h e flight-path angle excursions, Fig. 13, also become smaller. Toward the end of the skip, the control approaches the negative lift condition, and, for the longer ranges, produces negative lift to more quickly terminate the skip. The secondary acceleration peak rises with increasing range in order to dissipate the increased remaining energy. There is a minor sashay in the paths near the endpoints, due to the control passing through maximum lift and (L/D) on its way to —180 deg. Figure 14 shows that total heat increases with terminal range, as might be expected. However, it is somewhat surprising that the total flight time of Fig. 15 first increases with range, and then decreases for longer terminal ranges. 6. " A B S O L U T E M I N I M U M " T E S T

Since the terminal flight-path angle is unconstrained for all the paths considered to this point, it is natural to replace the fourth of Eq. (300) by (303)

γ(Τ,ρ0)-Γ=0

in which Γ is a parameter, to obtain a set of equations similar to Eq. (203). T h e Newton-Raphson equations for the new system are ~dT~ dp\o dpzo dp30 _dp4o_

~v(T)

Vn(T)

m = — i(T) ζ{Τ) 0

-1

Vi3(T)

-v(T) - χ ,

V23(T)

/.(0)

-X2/R

^2{T)

m

V42(T)

ζ(Τ) - χ .

M0)

/s(0)

-

-Γ

Λ(0) _

Η (304)

T h e derivatives of the initial conditions with respect to the parameter Γ are in the second column of the inverse matrix. These are used with the predictor equation (148) to establish the family of solutions. An unconstrained optimal trajectory with terminal range of 1450 miles was used for the test. During computer runs it was found that an increment ΔΓ =\ deg initially produced a member of the family of solutions at each step. Prediction gradually worsened, however, as the extremes of Γ were approached, and it is unlikely that further extension of Γ can be accomplished. T h e total heat J for the family of trajectories is plotted in Fig. 16. There is only one minimum for the range of Γ given, and it is noted that the minimum is quite insensitive to large variations in terminal flight-path angle. From Hamilton-Jacobi theory, it is known that —p2(T) is the slope of Fig. 16. The slope is zero for the optimal trajectory, and becomes very

120

D. K. SCHARMACK 28,900

28£50

1

1

-60

1

-50

1

-40

—» -30

1

1 -20

1 -10

1 0

r(DEG) F I G . 1 6 . Absolute minimum test results—total heat / versus terminal flight-path angle Γ.

large (in absolute value) at the extremes of the curve, so —p2(T) is never again zero over the range of Γ. It is concluded that the original trajectory is the only solution to Eq. (300) over the obtainable range of the parameter Γ. 7.

SUFFICIENCY T E S T FOR A RELATIVE M I N I M U M

T h e 1500-mile trajectory of Figs. 9-13 was tested for sufficiency in a modified coordinate system. T h e transformations V = v\

h = R£y

z = i/c3y

(^3 = 5280)

(305)

121

TRAJECTORY OPTIMIZATION PROBLEMS

together with a change of independent variable from t to ζ (permissible since ζ is an increasing function of t)y produce a Hamiltonian system, (306)

H2 = g0 + P'g related to the old through the relations

P =p2,

Λ=*ι/2»,

P =P IR

2

P, = -H=0,

3

H2 =

3

-c3p,

The differential equations in the new system are 2

312

_dq _ ax{R + h)PV V ^° dz cosy

a2(R + h) ( P_\ ( 8 c o s y \p0) \ 1 0 /

^î-^'^^mrr) dh g3 = —=a5(R

/ kr

>

(308

. + h)Ί tan y

dt _ a5(R + h) 112 ~dz~ V cos y

g4

with constants defined by 4

il2

a i

a2 = 7.5 χ 10" Λ^α 5,

= cN- a5y

aA = 2aly

as = c3fRy

at = -a3\2y

a3 =

-c3S/mR

a7 =

-c3g0R

Since P 4 = 0, the last of Eq. (308) may be neglected, and the EulerLagrange equations may be written _dPi dz

dPj dz

=

Sgo,p dv^

3go + dh

Equation (311) has the solution

dgi dv^

1

1

fa + dh

ρ Sgl 2 dv

p 2

dgj. + dh

ρ Sgi 3

dh

122

D. K. SCHARMACK

and u satisfies the sign conventions (294). The strengthened Clebsch and Weierstrass conditions are satisfied as long as Px and P2 are not simultaneously zero. T h e terminal conditions corresponding to Eq. (286) are V(zT)

2

- Xx

= 0,

h(zT) -X2

= 0,

zT-X3\c3

= 0

(313)

which have the form (15). The accessory equations are

Vi

d_ dz

Sgl dv

Sgl dy

Sgl dh

Sgl SPi

Sgl dP2

Sg2 w

hi dy

Sgl 8h

Sgl SPi

Sgl dP2

Vi

0

3?3

Sgl 8h

0

0

Vi

Sgl w

Sgl dV

_Sgi dy

_Sgi dy

Sgl dy

Sgl dh

Sgl dh

Sgl ' dh

dy

2

£i

2

d H2 2 dV 2

d H2 2 dy

2

d H2 dh dy

il

d H2 dVdy

Î3

d H2 dvdh

2

d H2 dVdy

d H2 dVdh

2

d H2 dh dy

2

2

d H2 2 dh

2

0

(314) 0

ii

Let the fundamental system of solutions be n

L"2l(*)

7Τ(0)

= /

V(z) = π2ί(ζ)

-

W22(*). a-

(315)

and consider the conjugate system of solutions U(z) = ττη(ζ)

- ιτη{ζ),

π22(ζ)

(316)

which corresponds to Eq. (196) with matrix K=0. In the computer program the determinant of U was computed at points spaced 15 miles apart. It started at unity (# = 0), and rose to very large values 29 as ζ increased (about 1 0 maximum). Because of the smoothness of the solution, it was concluded that the determinant did not vanish or become negative between output points. In view of Eq. (313), the test of Eq. (160) may be written -a'U\zT)

V{zT)a>

0

(317)

for all a satisfying U (zT) a =

0 Δ 0

(318)

TRAJECTORY OPTIMIZATION PROBLEMS

123

with Δ arbitrary. When Eq. (318) is inserted into (317) there results -[0,Δ,Ο\ν{ζτ)

> 0

U-\zT)

(319)

Thus, the center element of the product matrix must be negative. It was, so the 1500-mile trajectory of Figs. 9-13 is indeed a strong relative minimizing path.

D.

The Bounded Brachistochrone Problem

The bounded brachistochrone problem was chosen to test the theory. It is simple enough to have an analytical solution, yet nonlinear in nature. The equations of motion, as given in {11), are

χ = ν cos y,

y = ν sin y,

ν = —g sin y

(320)

The problem is to minimize the time it takes to go from a given initial point to χ = Xf while satisfying the path constraint G = y-ax-b>

0

(321)

T h e state vector has the components (x,y,v); γ is the control function to be determined. Figure 17 shows the path constraint, terminal condition, and a possible path.

F I G . 1 7 . Bounded brachistochrone problem geometry.

The time derivative of (321) is Ô = ν (sin y — a cos y)

(322)

124

D. K. SCHARMACK

so the Hamiltonian is written as Hx = 1 + px ν cos y + p2 ν sin y — p3g sin y + /xt;(sin y — a cos y) (323) The Euler-Lagrange equations become p2 = 0,

Pi = 0,

0 = —px ν sin y -f ^>2

v

c os

~[pi cos y + / > 2 sin y]

/>3 =

Ύ ~ p3g

c os

7+ /^(

c os

(324)

y + # sin y) (325)

Over unconstrained subarcs the multiplier μ is zero, so Eq. (325) may be solved for y{vypXyp2yp3) as t a n

P2*-P3g

Y

or sin y = -(P2v-p3g)[(pi cos y =

)

pxv 2

2

v) + {p2v

2

-p3g) }2

*•[(/>! ζ;) + {p2v

-p^g) }'

}K

112

112

The minus signs in Eq. (327) are chosen to minimize the Hamiltonian (323). On constrained subarcs, solution of ό = 0 [Eq. (322)] gives tan y = a

(328)

since by assumption ν Φ 0, and Eq. (325) gives va-(p2v-p3g) 2 v(\+a )

Pi r

;v

Now let φ be the angle defined by Eq. (327). This is a convenient definition, since at the boundary point t2 (see Fig. 17) everything is continuous, so φ becomes y at that point. T h e constrained subarc Hamiltonian may then be written H=

1 - [{Pivf

+ (ρ2ν-ρ,β)ψ

2

c o s ( y - φ)

(330)

The minimizing value of y makes the cosine positive, so it must satisfy φ-πβ 0's at that point. The first of Eq. (349) is used as the stopping condition for the integrations, which accounts for the zero in the left-hand vector of Eq. (350). T h e accessory differential equations are

3 / 33 y ^

^

z

y

d

v

£i = 0,

^

+

d

=

^

W

i

¥

ι

3y

¥

+

1

Ί

ζ 3

ti = 0

Pi 3y

r ζ3

9y

dv^

ap [ 3y äy" [dp, 3

+

Y

ζι +

3y

3y Î2

dp~i

¥ 3

]

Ί

ζ

(351)

TRAJECTORY OPTIMIZATION PROBLEMS

127

and the initial conditions are

~dx{0)~

0

0

0

0

0

0

0

0

0

0

0

0

(352)

Γ "Ö"Ö 0 _

S

Po_

0

1 0

0

0

0 1 0

T h e last column of Eq. (352) will, of course, give a zero vector solution of Eq. (351). It is included for the three-subarc case in which case it will contain the partials of χ and p with respect to v. Case II: Two or Three Subarcs. T h e first and third (if present) subarcs satisfy the differential equations for Case I. Over the second subarc the reduced differential equations of the extremals are Eq. (320) and Eq. (324) with y determined from Eq. (328) and Eq. (331). Since y is a constant, the accessory equations simplify to ήχ = η3 £ι =

ο,

ή2 = η3 sin y,

COS y ,

ζ2 =

ο,

ζ3 = -[ζί

ή3 = 0

c o s y + £ 2 sin y]

(353)

T h e initial conditions for Eq. (324) are given by Eqs. (341)—(343), with constant v to be determined, whereas initial conditions for Eq. (353) are ••Vij + [*i ~ ^ 4

= 0,

x

i

+

]

Wo,'

=

ζ+ _ i

1,2,3

i= 1,2,3

; = 1,2,3 b

+1

(P3

(354) (355)

Hi = tlj

S i / £Tj •

i,j=

(356)

D IT

(357)

In these equations j designates the column number in the matrices dx/dp0, dp/dp0y and / corresponds to the numbering in Eq. (353). Intermediate extremals computed during iterations will not in general satisfy the necessary conditions at the point tx. In particular, the control

128

D. K. SCHARMACK

function may be discontinuous, which implies Ö φ 0. In this case the partials of tx in Eq. (354) and Eq. (356) are computed from G(tx) = 0, i.e., 3*i _

aVij~V2j

dp0j

-ax

( > = 1,2,3)

+y

(358)

(6-*0)

I f Û~ happens to be zero, the coefficients of DTXLDP0J disappear in Eq. (354) and Eq. (356). These partial derivative solutions are then continuous at need not be computed. All variables are continuous at T = TXY and DTI/DPQJ the junction point T2 between the second and third subarcs (if there happens to be a third subarc). Two additional necessary conditions are required to set up the modified Newton-Raphson equations. These are

o~(t )

= y(ti ,po) - ax(tx ,p0) -b = 0

(359)

= j T ( * i ,po) - ax-(U

(360)

,PO) = 0

If and Eq. (360) is the addix Φ 0, then Eq. (359) determines tX(P0), tional necessary condition to be satisfied. T h e modified Newton-Raphson equations are then -

0

-χ(Τ)

~

0 -C

P2(T) pm

Î2ÂT)

Î32(T)

P*(T)

H(Po)

0

Ο-(ί,)

0

Ï2*(T)

àpu

UT) 0

dpi. (361) dpi*

0

dv

/ 3( 0 )

/.(Ο)

MO)

3Ö-(tt)

dÛ~{tx)

fa. fa.

faa

where °Pi0

~dT~

-ίΐ3(Γ)

••Vu —

AYH+Giti)

dt POi

x

d

1,2,3

2

2

(362) (363)

G(tx) = ag(s'm y~ — cos y~)

and DTX/DP0I is defined by Eq. (358). I f Eq. (360) is satisfied, then it determines TX, and Eq. (359) becomes the additional equation which must be satisfied. In this case the last equation of Eq. (361) is replaced by -CG(tx)

dG(tx)

.

dG(tx)

fa

fa

.

dG(tx)

.

(364)

fao

where dG(tt) fa,

V2i~

ar

)u>

1=1,2,3

(365)

129

TRAJECTORY OPTIMIZATION PROBLEMS 6

5

4

y

j

!

y + 0.5) :-5 =

0 ^

1

i 1

3 x ι( t 2 )

x(t,) 2

4

5

6

F I G . 18. Bounded brachistochrone optimal solution.

The First Constrained Path. It is possible to choose the initial values t r i at t r ie a t ri str (p\0,p20ip30) such P i k e s the boundary (359) before the stopping condition, the first of Eq. (349), is satisfied. There will then be a second subarc, and the problem is that of determining a value of ν to go with it. Since the optimal path is to consist of three subarcs, the scheme chosen is based upon leaving the constrained subarc as soon as possible. The multiplier discontinuities may be computed at either end of a constrained subarc. Assuming they are determined at the point t = t2, the last of Eq. (324) takes the form

p3 = -(ΡΓ

cos y + p2~ sin y ) +

+

(366)

Since p{~ =p\Q, pï~ =Ρι0> d 7 is found from Eq. (328) and Eq. (331), + + p3(t) is well determined. T h e conditions p (t2) = 0, G (t2) > 0, and γ a n

+

130

D. K. SCHARMACK

continuous must be satisfied if t2 is the endpoint of the second subarc. T h e first condition is solved for ν giving ν •-

(Pi v-p3

g)-px

v(l

+

va

(367)

2

a)

whereas the second requires that +

G (t2)

2

2

+

=g(l - a ) cos γ > 0

(368)

Inequality Eq. (368) is always satisfied for the value of a considered. T h e determining factor is the continuity of y which requires (from the minimum principle) s n g (/>i~ + va) = sgn(cos y+) s

n

g [(/>2~

-v)v-

+

p3g] = -sgn(sin y )

Satisfaction of Eq. (369) then determines the endpoint of the second subarc. I f the third subarc comes back to the boundary, the second subarc T A B L E III COMPARISON OF C O M P U T E R AND E X A C T O P T I M A L PATHS

Quantity yo (deg)

U

Ö-(fi)

h y(t2) τ x(T) y(T) H(0)

Hit,) Pi(T) P3(T) a

Computed value

0

-85.2577 0.385117 1.60687 4.19657 1 χ 10-5 - 0 . 7 χ 10-3 0.533530 3.18470 3.40766 0.742246 6.00001 2.75571 10 0.1 χ 1 0 0.6 χ ΙΟ"» 0.6 χ 1 0 - ! 3 0.7 χ 1 0 - n

Exact value -85.2578 0.385141 1.60710 4.19645 0.25 χ 10 0 0.533526 3.18465 3.40767 0.742245 6 2.75571 0 0 0 0

T h e differences between computed and exact values resulted from rather loose interpolation procedures in the computer program. Although these could have been corrected to obtain more accurate results, it was not felt necessary since the purpose of solving the problem was to prove the method.

TRAJECTORY OPTIMIZATION PROBLEMS

131

is extended to this point, and testing for the end of the second subarc is resumed. On the other hand, if the second subarc extends to the endpoint, 1 , then τ ? 1 4( 7 ) = ζ 3 4( 7 ) = 0, and ζ24(Τ) = 1. Solution of Eq. (361) (or its equivalent if ô~ = 0) gives iterative corrections which tend to satisfy the last three necessary conditions. T h e computed dv is ignored, since the necessary condition for p2(T) on a two-subarc optimal is p2(T) = v, rather than zero. T h e optimal path is to have three subarcs, so the corrections are used iteratively with the first constrained path logic. A three-subarc path is eventually obtained. Numerical Results. The constants for the problem solved are the same as those used by Dreyfus (11). They are x0 = 0, y0 = 6, v0=l, g = 32.172, a = —0.5, i = 5, and ay = 6. A set of initial multipliers were found which produced a three-subarc path having a terminal time of 1.25429 sec. T h e optimal path of Fig. 18 resulted after 33 iterations, and a comparison of this path with the exact solution is given in Table I I I .

References 1. G. A . BLISS, "Lectures on the Calculus of Variations." Univ. of Chicago Press, Chicago, Illinois, 1 9 4 6 . 2. F . A. VALENTINE, T h e problem of Lagrange with differential inequalities as added side conditions, in "Contributions to the Theory of Calculus of Variations ( 1 9 3 3 - 1 9 3 7 ) , " pp. 4 0 3 - 4 4 7 . Univ. of Chicago Press, Chicago, 1 9 3 7 . 3. L . D. BERKOVITZ, Variational methods in problems of control and programming, /. Math. Anal. Appl. 3 , No. 1 , 1 4 5 - 1 6 9 ( 1 9 6 1 ) . 4. W . T . R E I D , Discontinuous solutions in the non-parametric problem of Mayer in the calculus of variations. Am. J. Math. 57, 6 9 - 9 3 ( 1 9 3 5 ) . 5 . R . FLETCHER and M . J . D. P O W E L L , A rapidly convergent descent method for minimization, Computer J. 6 , No. 2 , 1 6 3 - 1 6 8 ( 1 9 6 3 ) . 6. W . C. DAVIDON, Variable metric method for minimization, AEC ANL-5990 (Rev.) (1959).

7. R . A. VOLZ, T h e minimization of a function by weighted gradients, Proc. IEEE

53,

No. 6 ( 1 9 6 5 ) .

8. D. W . MARQUARDT, An algorithm for least-squares estimation of nonlinear parameters, /. Soc. Ind. Appl. Math. 2 , No. 2 ( 1 9 6 3 ) . 9. H . A. SPANG, I I I , A review of minimization techniques for nonlinear functions, /.

SIAM4,

No. 4 , 3 4 3 - 3 6 5

(1962).

10.

F . Β . HILDEBRAND, "Introduction to Numerical Analysis." McGraw-Hill, New York,

11.

S . DREYFUS, T h e numerical solution of variational problems, / . Math. Anal. Appl. 5,

12.

R . V. GAMKRELIDZE, Optimal processes with bounded phase coordinates, Izv. Akad. Nauk. SSSRt Ser. Mat. 2 4 , 3 1 5 - 3 5 6 ( 1 9 6 0 ) . V. A. TROITSKII, Variational problems on the optimization of control processes in systems with bounded coordinates, Prikl. Mat. Mekhan. 2 6 , No. 3 , 4 3 1 - 4 4 3 ( 1 9 6 2 ) . Available as Minneapolis-Honeywell translation No. 3 8 5 .

1946. No. 1 ( 1 9 6 2 ) .

13.

132

D. K. SCHARMACK

14.

A . E . BRYSON and W . F . DENHAM, Optimal programming problems with inequality

15.

R . E . KALMAN, T h e theory of optimal control and the calculus of variations, in "Mathematical Optimization Techniques" ( R . Bellman, ed.), pp. 3 0 9 - 3 3 1 . Univ. of Calif. Press, Berkeley and L o s Angeles, 1 9 6 3 .

constraints, AI A A J. 1 , No. 1 1 , 2 5 4 4 - 2 5 5 0

(1963).

Determining Reachable Regions and Optimal Controls Conti DONALD R. SNOW

1

2

Department of Mathematics University of Colorado Boulder y Colorado

I.

II.

III.

1

Introduction to Optimal Control Problems and Reachable Regions 135 A. Control Theory and Optimal Control Theory . . . 135 B. T h e Control Set 136 C. Open Loop and Closed Loop Controls . . . . 137 D. T-Reachable and T-Controllable Regions . . . . 1 3 7 E . Relationship of the Controllable and Reachable Regions for Time-Reversed Systems 139 F . General Description of Present Paper 141 Carathéodory-Hamilton-Jacobi Approach in Optimal Control A. Introduction and Statement of the Problem . . . B. Equivalent Problems C. Hamilton-Jacobi Partial Differential Equation . . D. Hamilton-Jacobi Equation by Pontryagin Maximum Principle E . Sufficient Conditions and the Weierstrass Excess Function F . Solution of the Hamilton-Jacobi Partial Differential Equation by the Method of Characteristics . . . G. Reachable Region for a Related Control Problem . . A Heuristic Method of Determining the Reachable Region A. Problem Considered B. Functional to Introduce as an Aid in Determining the Reachable Region C. T-Reachable Region for Modified Optimal Control Problem D. Heuristic Limiting Argument E . Solution of Limiting Hamilton-Jacobi Equation . . F . Summary of the Heuristic Method

142 142 144 145 147 151 154 157 159 159 162 163 166 171 172

This paper constitutes a revision of the author's Ph.D. dissertation, written under the direction of Professor M . M . Schiffer, Department of Mathematics, Stanford University, Stanford, California. It was supported in part by Lockheed Missiles and Space Company Research Labs., Palo Alto, California. 2 Formerly at Department of Mathematics and Control Sciences Center, University of Minnesota, Minneapolis, Minnesota. 133

134

DONALD R. SNOW

IV.

V.

VI.

Rigorous Method of Determining the Reachable Region for a Particular Class of Problems A. Introduction B. Condition of Singularity C. A L e m m a D. Characterization of the T-Controllable Region E . Optimal Controls for Singular Initial States . F . Optimal Controls for Regular Initial States

173 173 174 176 177 181 183

Illustration and Comparison of Rigorous and Heuristic Methods A. Solution by Rigorous Method B. Solution by Heuristic Method

186 186 190

Concluding Remarks

194

References

195

This paper presents two methods for the determination of the reachable region for control problems. One of these methods is proven rigorously and applies to a certain class of optimal control problems which are singular in the sense that the Pontryagin maximum principle does not characterize the optimal controls completely. This method consists of a geometrical argument based on areas and "generalized centroids" of the areas under the desired control function curves. It determines the reachable regions as well as the optimal controls. In developing this method we have introduced a concept of regular and singular terminal states (or initial states), the singular states being those for which the Pontryagin maximum principle is not useful. T h e regular and singular terminal state regions are described completely. T h e other method of determining the reachable region applies to a large class of control problems, but has not been proven rigorously. It is presented in the form of a heuristic or plausible argument only, and the steps remaining to be proved are indicated. This method consists in removing the constraints on the control set and introducing a special penalty functional. It is then observed that a limiting process with this functional leads back to the original problem and to a Hamilton-Jacobi partial differential equation which describes the reachable region. This heuristic method is applied to specific examples in the class of problems for which the rigorous method holds and the results are shown to be the same. In developing the heuristic method we have extended many of the classical results of the calculus of variations and Hamilton-Jacobi theory to optimal control problems through the use of Carathéodory's unifying approach. Though some of these extensions have been discussed already in the literature during the past 3 or 4 years, they are presented here in a unified form which has not been done before. Section II presents Carathéodory's approach used in optimal control. Section I I I presents the heuristic method of determining the reachable regions and depends on the material in Section I I . Section IV discusses the rigorous method and is independent of the preceding sections. Section V illustrates both methods and compares the results for a particular class of problems.

REACHABLE REGIONS AND OPTIMAL CONTROLS

135

I. Introduction to Optimal Control Problems and Reachable Regions A.

C o n t r o l T h e o r y and O p t i m a l C o n t r o l T h e o r y

The problem of automatic control has been discussed for many years and numerous techniques to handle control problems have been developed. Optimal methods (as rated by some performance criterion) of accomplishing control have been of theoretical interest for a long time. One of the earliest optimal control problems was presented in 1929 by Zermelo ( ί ) , and concerned the problem of navigating an airplane between two given points in the least time, when the wind velocities are known. Within the last 10 or 15 years such problems have become of much greater "practical" interest. This is due to the development of new controlling devices (computers for example) which make the implementation of optimal control schemes feasible now, and new applications for which optimal or near-optimal control is highly desirable. As a result, the last few years have seen the development of a new field of mathematics called optimal control theory. This field includes new mathematical techniques as well as the adaptation of older methods to the newly formulated problems. One of the basic problems in control theory in general is the following: suppose a system (physical, electrical, chemical process, business organization, etc.) is described by a set of ordinary differential equations: Xi =fi(t,

x u . . . , x n , u u . . . , um),

ί = 1,2,..η

(1)

In these equations, t is the independent variable (time, in most problems), the vector x(t) = [x\(t),..., xn(t)] is the state of the system at time t, and the vector u(t) = \ux(t),..., um(t)] is the control vector. Suppose a particular pair of states x0 and xTi are given. We are then to find a control vector [i.e., a set of control functions, ua(t)] so that if the system starts at state x0 at time t0, it will reach state xT at time Γ ; i.e., x(t0)

= x0

and

x(T)

= xT

(2)

The above-described control problem becomes an optimal control problem if we add the requirement that of all the control vectors which drive the system in the prescribed manner, we are to determine those which also optimize some performance criterion. This criterion is usually a functional depending on time, the state of the system, and/or the control vector itself. It can frequently be expressed in the form of an integral to be minimized : J[u] = Jlfo(t,x,u)dt

(3)

136

DONALD R. SNOW

where / 0 is a scalar function. In many problems the functional is proportional to a physically significant quantity. Examples are : the time required e t r ie ( -g->/o = 1)> the fuel or energy expended ( e . g . , / 0 = Σα|^α|)> distance traveled in the xx direction (e.g., f0 = xx), the mean-square error (e.g.,

/ο = Σ.· Xi )> etc. 2

B. The

C o n t r o l Set

The control vector is to be selected from a prescribed class of functions U depending on the problem. T o define the class U, two types of conditions are needed. These two types are : (1) U consists of vector functions u(t) whose range lies in some specified m m subset S c E for all t in some interval, where E is the m-dimensional Euclidean space ; (2) the components of each vector function in U satisfy certain continuity or differentiability requirements with respect to t. In Condition (1), the set S may change with time, though in most applications this is not the case. For most problems of interest, S is a closed and m uu...yum bounded set; for example, all points in E whose components satisfy \ua\ < 1, α = 1,2,...,m By considering additional components on the control vector, more general types of constraints can be expressed. For example, using u = (ul,...,

um, ùx,...,

2m

ùm) e Sx cz E

we can consider bounds on the derivatives of the control functions also. For Condition (2), the requirement is frequently taken to be piecewise continuity. However, since the limit of a sequence of piecewise continuous functions may not be piecewise continuous, this set of functions is not closed. Sometimes the class of functions is enlarged to include all measurable functions so that limiting arguments can be used to guarantee the existence of optimal controls in the set. It is usually found that the optimal control is piecewise continuous anyway, in spite of the inclusion of measurable functions in the class. Being given the set U, we can then define : D E F I N I T I O N . A control vector in the set U will be called admissible if it drives the system in the prescribed manner; i.e., such that X

x(t0) = x0 and (T) = XR T h e problem of selecting a function to minimize a functional belongs to the field of mathematics called the calculus of variations. However, since

REACHABLE REGIONS AND OPTIMAL CONTROLS

137

the most interesting control problems require the control set U to consist of functions whose range lies in a closed set S, the classical techniques of the calculus of variations cannot be used without modification. This is because arbitrary variations are not allowed when the control functions assume the extreme values permitted.

C . O p e n Loop and Closed Loop Controls

The problem of determining the optimal control as a function of time only, u(t)y is called the open loop problem in the engineering literature. In contrast, the closed loop problem consists in determining the optimal control as a function of the state of the system and possibly the time as well, u(t, x). This is the ''feedback control" and may be thought of as the open loop control for initial state χ at time t. T h e open loop control, u(i), may be obtained from the closed loop u(t> x) by integrating the differential equation system (1) with u(t, x) and using the solution in u[t, x(t)]. T h e closed loop control can be obtained to any desired degree of accuracy (or perhaps even exactly, depending on the problem) by solving the open loop control problem for different initial states and combining the results. This is done by defining the closed loop control v(t> x) as x

*>(*o > o) = u(t0)

and

ν [τ, χ(τ)] = U(T)

for t0^ τ ^ Γ , where u(t) is the open loop control corresponding to initial state x0 at time t0. This definition gives the desired result since u(t) for τ must transfer the system from the state x(r) on the trajectory to the state x(T) = xT in an optimal manner (otherwise the entire trajectory from x0 to xT would not be optimal). If it is desired to describe a general control law that will always drive the system optimally to a given final state regardless of the initial state used, the closed loop control is the most convenient formulation. It is also the most convenient form if there are constraints on the control set which depend on the state variables; e.g., if the control is restricted by \u(t, x)\ < 1 for all λ: in a given domain in state space for any t, and u is unrestricted elsewhere. Furthermore, from a practical standpoint, it may be easier to construct a mechanical controller device based on the closed loop optimal control rather than the open loop control. D.

T-Reachable and T-Controllable Regions

The above-described general control problem leads to the following problem : for the given class of functions U and initial state x(t0) = x0) what is the set of states that can be reached at time Τ by use of controls from the

138

DONALD R. SNOW

set U ? This set of final states will be called the T-reachable region relative to x0. Depending on the differential equations and the control set £/, the Γ-reachable region may be the entire state space or only a subset of it. This problem is independent of any performance criterion but underlies questions of optimal controls since it must first be shown that admissible controls exist before we can ask for the admissible controls that are optimal in some sense. A problem related to determining the Γ-reachable region is the following : for a given class U and final state x(T) = xTy what is the set of initial states for which there is a control in U that drives the system to the given final state at time Τ ? This set of initial states will be called the Τ-controllable region relative to xT. This region is the T-reachable region when considering the system with time reversed. For a discussion of the relationship between these regions and the proof that it does not hold directly when considering a single wth-order differential equation instead of a system of η first-order differential equations, see the next subsection. The problem of optimal control, once it is known that there is at least one admissible control, can now be stated as follows: given the system (1), initial state x(t0) — XQ , an d final state x(T) = xT in the T-reachable region from x0y find the admissible control (or controls) which minimize the functional (3). A concept called complete controllability was recently introduced for linear systems (2-5) and has been extended to some classes of nonlinear systems ( 6 ) . A system is said to be completely controllable if given any two states in the entire state space there is a control u(t) that will drive the system from one state to the other in finite time. The concept of complete controllability just means that the T-reachable region for any initial state x0 must change with Γ so that any given point in the state space is eventually contained in the T-reachable region for some T. The minimum time optimal control problem (5, 7, 8) may be stated as follows : given the initial state x(t0) = x0 and the desired terminal state xf, find the control which drives the system from x0 to xf in the shortest time. Knowing the T-reachable region for all Τ gives us an immediate solution to this problem since we need only find the smallest time Τ for which the terminal state xf is contained in the T-reachable region and the corresponding control, which will then be optimal. Under the assumptions that U is compact and satisfies a convexity condition, and that the differential equation system satisfies some fairly general conditions, Roxin (9) has shown that the T-reachable set is closed in the state space. Hence, for such problems, there is no possibility of having an "infimum time" only, instead of a minimum time. Other recent papers dealing with the subject of reachable regions are

REACHABLE REGIONS AND OPTIMAL CONTROLS

139

(10-13). These papers present some general properties of the reachable regions.

E. Relationship of the Controllable and Reachable Regions for Time-Reversed Systems

This subsection shows the relationship between the T-controllable region for a given system and the T-reachable region for the system with time reversed. THEOREM

1. Suppose we have the system *i=Mt,x,u),

i=l,2,...,n

(4)

the state x0, time t0, and the control set U which satisfies the following condition : for time t0y if u e U where u = u(t) on t0^t^T> then v e U where ν = u(T — t) on 0 ^ t ^ Τ — t0. Then the T-reachable set of states relative to x(t0) = x0 is the same as the (T — t0)-controllable region relative to x0for the system with time reversed; i.e., the system where τ = Τ — t is the independent variable and τ goes from 0 to Τ — t0. Proof. Since t=T—r, let j > T( T ) = x{(t) = χ^Τ—τ). Thendy^dr = —dx^dt. For any given u eU, let v((r) = u{(t) = u{(T — τ). Then the time-reversed system is ^

= -fi[T - r,y{r), v(r)l = gi[r,y{r),v{r)]

» = 1,2,..., η (5)

Now let x(T) = xT be the end point of a trajectory of the original system (4) which starts at x(t0) = x0 and uses control u(t). Consider the trajectories of system (5) which start at xT\ i.e., for whichy(0) = xT. By the hypothesis on the control set U, there is a v e U such that v(r) = u(t). Using this v(r) in (5), we arrive at state y(T — t0) =yj. But changing the variable back to t, we get yf = x(t0) = x0. Thus the T-reachable state xT for the forward-time system is (T — i 0)-controllable for the reversed-time system, both relative to x0. We have shown that the (T — i 0)-controllable region for the reversed system contains the T-reachable region for the original system. Since the argument can be reversed, this proves the theorem. Q.E.D. The above theorem does not hold if we consider a single wth-order differential equation instead of a first-order system of differential equations.

140

DONALD R. SNOW

T o see this, suppose we have a single /zth-order differential equation N X (n) _ p(t9xtx,..., X^ ~ \ u) and consider the corresponding reversed-time equation. Then the phase space associated with the original equation is not the same as that associated with the reversed-time equation. T h e two phase spaces are related, however, by the following: the directions in the phase spaces corresponding to each of the even-order derivatives in both differential equations are the same, but the directions corresponding to the oddorder derivatives have their signs reversed. T o see why this occurs, consider the first-order differential equation system corresponding to the original wth-order equation, which is obtained by defining XX = XY X2 = X Y . . ·, XN = Then the corresponding firstorder system is

x

X

I

XN =

=

X

I+L F(ty

>

î = 1,2,..., w — 1

X \ , . . . , XNY

u)

The reversed-time system is T

y A?) = -yi+i( ) > ί = ι, 2 , . . η - 1 yn{r) = -F[T-ryy,(r)y...yyn{r)y ν(τ)] In the phase space for the original equation, the (z + l)th component is the rate of change of the ith component. But in the reversed-time phase space, the (z + l)th component is minus the rate of change of the ith component. Thus for the even orders, i = 0 , 2 , 4 , . . . , the component directions of the two spaces are the same, but for the odd orders, i= 1 , 3 , 5 , . . . , they differ in sign. T o see what this means, suppose we reverse the time in an /zth-order ordinary differential equation and integrate backwards from a given state ( n - 1 ) - 1 ) (xy xy..., # ) . I f the state reached is (x0y x 0 y. . . , λ : ^ ) then this must be n _1 X) changed to [# 0 , —X0Y X 0 Y . . . , (— l ) X§~ ] in order to be an initial state that ( n _ 1 ) will lead to (XY X Y . . . , x ) when integrating forward. In some cases the reversed-time differential equation appears in exactly the same form as the original differential equation. An example of this is any nth-order linear differential equation with constant coefficients where there are no odd-order derivatives present and for which the control set does not change with time. In such cases we might be tempted to say that the T-reachable region for the forward-time equation is precisely the same as the (T — z^-controllable region for the reversed-time system. Such is not the case, however, since reversing the time reverses the sign of the odd-order derivatives even though they do not appear in the equation. A simple example will illustrate this. Consider the "double integrator'' 2 or " I / * " plant: X = u

REACHABLE REGIONS AND OPTIMAL CONTROLS

141

where 0 ^ t ^ 1 and the control set consists of all piecewise continuous functions that satisfy \u(t)\ < l o n O < f ^ l . It may easily be shown using the rigorous method developed in Section I V of this paper that the reachable and controllable regions relative to the origin are as shown in Fig. 1. X

(a)

X

(b)

F I G . 1 . Regions for χ = u, \ u\ ^ 1 : (a) reachable region ; (b) controllable region.

It should be emphasized that the reason for this phenomenon is that the usual w-dimensional space associated with an nth-order differential equation has a particular interpretation as a phase space for the equation; i.e., the ith direction represents the rate of change of the (/— l)th direction. For a system of η first-order differential equations, however, the w-dimensional space does not have this implied interpretation. In most numerical integration programs for digital computers, this will not be a problem since they work with systems of equations, and, upon integrating backwards, the sign of the differences automatically leads to the correct result. However, in theoretical computations, the results of this section should be kept in mind.

F.

General Description of Present Paper

T h e present paper presents methods of determining the T-reachable region. As stated earlier this region is independent of any functional to be optimized. However, one method consists of introducing a suitable functional (a sort of ''penalty" function) as an aid in solving the problem. We use Carathéodory's approach to the calculus of variations, modified for optimal control problems, to obtain a partial differential equation that describes the reachable surface of a related problem. This partial differential equation is the Hamilton-Jacobi equation. It is shown by the method of characteristics how to characterize the particular solution of this partial differential equation which describes the boundary of the reachable region of this related problem. Then by a limiting process that is justified only heuristically in this paper a Hamilton-Jacobi equation for the original system without any functional is obtained. Using what we call data curves

142

DONALD R. SNOW

and the method of characteristics the particular solution of the HamiltonJacobi partial differential equation that describes the reachable region as a function of time is obtained. Since the limiting process has not been justified rigorously, several examples that have been solved rigorously by another method developed for a special class of problems are used to show that the results are the same. Section I I presents the Carathéodory-Hamilton-Jacobi approach as we have modified it for optimal control problems. T h e relationship between this method and the Pontryagin maximum principle is shown and the determination of the Γ-reachable region boundary of a modified problem is obtained by the method of characteristics. Section I I I shows the functional to introduce for a particular problem in order to obtain a related problem. Then the heuristic limiting process is shown which leads to the Hamilton-Jacobi partial differential equation which the desired T-reachable surface must satisfy, independently of any functional. T h e data curves that are used with the method of characteristics in the solution of this partial differential equation are then introduced and it is shown how the solution is obtained. Section I V presents a rigorous and independent derivation of the Treachable region for a special class of control problems. This class of problems was originally studied since they are singular optimal control problems in the sense that the Pontryagin maximum principle cannot be used in their solution. The rigorous method developed for the solution of this special class of problems gives the optimal controls as well as the T-reachable regions. In Section V, an example class of problems is solved first by the rigorous method of Section I V and then by the heuristic method of Section I I I . T h e results are shown to be the same. Section V I discusses the possibility of making the heuristic argument rigorous and extending it, as well as the specialized method, to additional classes of problems.

II. Carathéodory-Hamilton-Jacobi Approach in Optimal Control A.

Introduction and Statement of the Problem

In articles published in 1925 (14) and 1926 (15), and culminating in his book on the calculus of variations in 1935 (16) [see also (17)], Carathéodory introduced a unifying approach to the solution of problems in the calculus of variations, namely, the concept of equivalent problems. Recently this

143

REACHABLE REGIONS AND OPTIMAL CONTROLS

approach has been applied to optimal control problems (2, 18-20). It leads to results analogous to those of the classical Hamilton-Jacobi theory and will be presented and amplified in this section. A more complete discussion of this approach was published by the author in (21). T h e next section will show the application of this theory to the problem of determining the reachable region in the general case. Suppose we are given the system of η first-order ordinary differential equations : (6) *i =fi(t> *!,...,*„,!*!,..., um) where the control vector u = (ux,..., um) is to be taken from any given set of vectors, E/, which satisfies the following two conditions : (1) each u e U is a continuously differentiable function of t\ (2) for each u e £/, and at each time t, there are small variations Su e U such that u + 8u are in U for Su's of either sign. m

Requirement (2) here means that the range set, S of the differential equations on t0 ^ ί ^ Τ for any u e U, where t0 and Τ are given numbers. Assumptions of differentiability of the / / s will be made later. For each control u(t) and corresponding trajectory x(t), suppose a functional is defined by J[u]=

(7)

F L(t,x,u)dt J TO

where L(t, x, u) is a given function which satisfies differentiability assumptions to be stated later. Suppose, further, that we are given two states : «^0

=

x

( 10J

x

20J

Χ

· · ·> ΗΘ)

l

a n (

T

= XX

( LT>

x

2Ty

Χ

· · ·> ΗΤ)

(8)

D E F I N I T I O N . A control vector which is in U and which drives the system from the initial state x(t0) = x0 to the final state x(T) = xT will be called an admissible control.

The optimal control problem we consider in this section is then : find an admissible control vector which minimizes the functional (7).

144

DONALD R. SNOW

B. Equivalent Problems

Suppose S(t, x) is any given function that is continuously differentiable. We then define BS v L*(t, x, u) = L(t, x,u)--^ (f, x) - ^ x)fi(t, xy u) (9) and J*[u] = j * L*(t,x,u)dt

(10)

Using any control u and integrating along the corresponding trajectory,

since along the trajectory / t ( i , x9 u) = dx^dt. Then

J*[u]=J[u]-S(t,x)\Z

(11)

For specified end states x0 and the last term on the right-hand side of (11) is a known constant for a given S(t, x) and does not depend on the trajectory between x0 and xT since dSjdt is a total derivative. Hence, any admissible control u which minimizes (10) also minimizes (7) and vice versa. Thus, minimizing (10) subject to (6) is equivalent to minimizing (7) subject to (6). By the above, any continuously differentiable S(t, x) leads to an equivalent problem. However, certain choices of the function S(t, x) lead to equivalent problems for which the minimization of J* [u] is easily accomplished. This will be the case if S(t, x) is such that, for each (t, x), L*(ty x, u) satisfies the following two conditions : (i) (ii)

L * ( i , x , u ) ^ 0 for all u e U;

(12)

there is an admissible ü = ü(t, χ) such that 5) = 0.

(13)

Using such an S(t, x), the equivalent problem has

/·[«] > ο and, for u = ü,

7·[δ] = 0 Thus u would be an optimal control which minimizes (10), and hence also (7).

REACHABLE REGIONS AND OPTIMAL CONTROLS

C.

145

Hamilton-Jacobi Partial Differential Equation

Assuming S(ty x) is a function which makes L* satisfy (12) and (13), we will determine the conditions it must satisfy. Conditions (12) and (13) require that L * have a minimum at ü as a function of u and that the value of L* at this minimum be zero. Since dSjdt does not depend on uy this may be stated as (14) Now suppose L and the /,· s are continuously differentiable with respect to the control functions ua. Then a minimizing control ü must satisfy

or

(15)

since the range set of the control vector functions is open. Assumption (2) on the control set U in Section Ι Ι , Α was made so that, with differentiability of L and the / / s , the minimizing condition would reduce to this relation. We will write this as (16) in which we consider the p/s as independent variables. We then make the assumption that we can solve these m relations among the t/ a's to find (17) where ρ = (piy.. .ypn) and u e U. There may be several such solutions, W(t,x,p). Having found a function (17), we set dS/dx^pi and use it in (14) to obtain 35 dt

+2^4'^

(18)

or with the Hamiltonian defined by H{t, x,p) = J Pif,{t, x, ®(t, x,p)] - L[t, x, ®(t, x,p)]

(19)

i

ds dt

+ Η

Χ

{*> >ψχ)

=°

W

146

DONALD R. SNOW

This is the Hamilton-Jacobi partial differential equation. Thus, for each function W(tyxyp) in (17) which satisfies the minimizing conditions (16) there is a corresponding Hamilton-Jacobi equation. We see that if S(ty x) is any function which makes L* satisfy conditions (12) and (13), then it must satisfy one of these partial differential equations. It should be emphasized here that obtaining these partial differential equations was only a matter of inverting systems of finite equations, not integrating differential equations. If L and the/ t 's are linear in the controls, conditions (16) are independent of the wa's and this method cannot be used directly. In applications of the theory presented in this section and used in the next section, we will still be able to handle problems with differential equation systems which are linear in the controls since the functionals which will be introduced in the next section as aids in solving the problem of the reachable region will be nonlinear in the controls. For linear (singular) problems, see (22). The Hamilton-Jacobi partial differential equation is first order, but is usually nonlinear. By our differentiability requirements on L,w, and the/ t 's, H(tyxyp) is continuously differentiable in its arguments. Then, by characteristic theory for first-order partial differential equations, the HamiltonJacobi equation has infinitely many solutions. The differentiability assumption on the ^ a ' s will be dropped in the next section where continuously differentiable controls will be used only to approximate the piecewise continuous controls. Now suppose S(ty x) is a particular solution of a Hamilton-Jacobi equation which corresponds to one of the solutions (17) of the minimizing conditions (16). Using this S(ty x) in the corresponding (17), we obtain a feedback control : (21) In order to have ü continuously differentiable we will have to require that S(ty x) be twice continuously differentiable. For each initial state used, this control will drive the system to a final state at time Τ that depends on the initial state. Thus each solution of a Hamilton-Jacobi equation leads to a set of infinitely many trajectories for t0 ^ t ^ T, each one connecting different pairs of end states. Along any one of these trajectories, the control ü given by (21) makes L* satisfy condition (13) ; that is, L*(ty xy u) = 0 since this is just the Hamilton-Jacobi equation (20). Then, since / * [ « ] = 0, (11) gives

s{t,x)\z=m

(22)

i.e., the value of the functional (7) taken along the path given by u between a pair of these corresponding end states is given by the difference of the

REACHABLE REGIONS AND OPTIMAL CONTROLS

147

values of S(ty x) at the end states. Berkovitz (23) and others have shown 4 previously that the Value function" in (22) when taken along optimal trajectories satisfies the Hamilton-Jacobi equation. However, we now see that this is true for any control, optimal or not, which arises in the above manner. These trajectories may not be optimal, since we do not know whether S(t, x) makes L* satisfy condition (12) ; i.e., whether L*(ty x, u) ^ 0 for all u e U. I f we can show that (12) is satisfied, then these trajectories are optimal trajectories between the end states. A discussion of sufficient conditions will be deferred until Section I I , E . Since the initial and final states for our problem are given, (8), we will only be interested in those solutions of the respective Hamilton-Jacobi equations which lead to admissible controls when used in the corresponding function (17) ; i.e., such that the trajectory starting at x0 ends at xT. For each of these trajectories (optimal or not) between x0 and xT, there will be infinitely many solutions of the corresponding Hamilton-Jacobi equation. T h e reason for this is that the trajectories (optimal or not) given by the feedback control (21) are (shown in Section I I , F to be) characteristic curves of the particular Hamilton-Jacobi equation which corresponds to the function (17), and infinitely many solution surfaces of a partial differential equation intersect along a characteristic of it. T o solve our problem, we must find those admissible controls which are optimal. As noted above, each solution S(ty x) of a Hamilton-Jacobi equation leads to a feedback control which usually leads to only one trajectory passing through any given initial state. However, it will be shown in Section I I , F that each Hamilton-Jacobi equation has one solution for which the given initial state x0 is a singular point and which leads to many trajectories passing through this initial state. Each of these trajectories will have the feedback control °ll\ty xy dS(t, x)jdx] which corresponds to the given HamiltonJacobi equation. This particular solution of the Hamilton-Jacobi equation can lead to more than one trajectory when used in °U\t, xy 8S(ty x)jdx] since the partial derivatives of this solution function S(ty x) are not uniquely defined at the singular point x0. Hence, different choices of initial values for these partial derivatives lead to different trajectories.

D.

Hamilton-Jacobi Equation by Pontryagin Maximum Principle

We have shown by the Carathéodory approach that the value of the functional, when using a feedback control as above, is given by a function (22) of the end states which satisfies the Hamilton-Jacobi equation (20). We will now show this is true [cf. (23)], under the same differentiability assumptions, for any optimal control by means of the necessary condition

148

DONALD R. SNOW

called the Pontryagin maximum principle (24, 25', 10). This will introduce the adjoint system and variables and allow us to show in the next subsection the relationship between the maximum principle and the Carathéodory approach. We first define a new Hamiltonian function : (23) In this function the variables ua appear explicitly and are to be considered as independent variables. T h e p/s here are the adjoint variables and are required to satisfy the adjoint differential equation system :

(24) The Pontryagin maximum principle states that if ü(t) is an optimal control which gives x(t) as the optimal trajectory, then there must be a continuous vector /> = (/>i,..., Αι) which satisfies the adjoint system (24) with x(t) and as a function of u, is ü(t) and for which the Hamiltonian, H0(t,x,p,u)y maximized by w. Then, since we have assumed the minimum occurs in the interior of the range of the control set (the range is an open set), we can set the partial derivatives of H0 with respect to the wa's equal to zero to obtain relations that the optimal controls must satisfy :

or (25) We note that these conditions are precisely conditions (16), which we obtained before in the Carathéodory approach. Solving them gives (26) We now assume that there is a region containing x0 in state space which is covered by a field; i.e., through each state X in the region, other than XQ, and for each time Γ , t0^T ^Tly where Tx is given, there is exactly one optimal trajectory which has initial state x0 and final state x(T) = X. Let us consider the functional J[u;T,X]

=

j*L[t,x(t),u(t)]dt

(27)

REACHABLE REGIONS AND OPTIMAL CONTROLS

149

where u(t) is any control in U which drives the system to x(T) = X f r o m x0. Then for each state X in the field this functional is minimized by the ü(t) that gives the optimal trajectory x(t). Define S(T,X)

(28)

= J[u;T,X]

By our assumptions of a field, S(T,X) is a single-valued function defined everywhere in the region. Now suppose we consider the first variation of functional (27) about the optimal trajectory. T o do this, we change the control to ü + Su, which determines a new trajectory χ -f Sx (not necessarily optimal) near the optimal trajectory x(t), the new trajectory terminating at X + SX at time Τ + ST. The new trajectory will be near the old one for small changes in the control because of the theorem of continuous dependence of the solution of a differential equation system on the parameters of the system. T h e first variation, δ J> is the first-order terms of the difference between J evaluated along this new trajectory and along the old one which was an optimal trajectory. T h e total difference is JJ

= J[ü + 8u;T + ΓΤ+δΤ

=

8T,X+8X]-J[u;T,X]

_

ΓΤ

L(t, x-\-Sx,ü

+ Su) dt —

J TO

= ]^

L(t, xy ü) dt J

+δΤ

to

L(t, x + Sxy ü + Su) dt

+ Î [L(ty X + SXy Ü + SU) — L(ty XY Ü)] dt T

J

tQ

Expanding L(ty χ + 8xy ü + Su) in a Taylor series about χ and w, the firstorder effects are

SJ = [L(ty XY ü)] ST t=T

= [L(ty XY U)]t =T8T

+ jy |j

Xi

+ J* |2

+ ^Pj(t)^(t9x9ü)8ua(t) Α,/

x, ü) 8 (t)

^{ty

XY

υ)

hX^t)

dt A

by the maximizing conditions (25). T h e new trajectory satisfies (xj + Sxj)' = fj(ty χ + Sx, ü + Su),

j = 1,2,..., η

150

DONALD R . SNOW

Expanding in a Taylor series and neglecting terms higher than first order, we obtain

since x(t) satisfies system (6) with u(t). Using this, δ J becomes

The term in square brackets is just p{(t) by the adjoint system (24) with ü(t) and x(t). Thus

Since the initial time t0 and initial state x0 are considered fixed, all the are zero. At the terminal end of the trajectory we have

SXAÏQYS

Using this we obtain (29) We have thus shown that the variation of / about an optimal trajectory (i.e., the first-order changes in / due to the varied trajectory) can be expressed in terms of the changes of the final time and state only, and we do not need to consider the changes of the trajectory along the trajectory. The above result, (29), holds for any trajectory (optimal or not) close to the optimal trajectory considered. Now consider the value of the function S(T9X), defined by (28), at the point (T+8T,X). This will be the value of J when computed along the optimal trajectory to point (Τ + δΓ, X). Since (29) holds for all neighboring trajectories, we have

Then, dividing by 8T and letting 8T -> 0, we see that (30)

151

REACHABLE REGIONS AND OPTIMAL CONTROLS

By similar reasoning with S(T, X + 8X) — S(Ty X) it can be shown that

(31)

W =Pi(T)

dJ

D

Now recall that x(T) = X and, by ( 2 6 ) , û(T) = W[T,x(T),p(T)]. Using these and ( 3 1 ) and the definition of H0 given by ( 2 3 ) , in ( 3 0 ) , we obtain ,

D

S

^

X

)

,W(T,X,

^y^))

= 0

(32)

Comparing definitions ( 2 3 ) for H0 and ( 1 9 ) for Hy we see that the partial differential equation ( 3 2 ) is the Hamilton-Jacobi equation ( 2 0 ) with (Τ, X) in place of (tyx). By ( 3 1 ) , the adjoint variables are really the coefficients that express the sensitivity of S(Ty X) to changes in the end conditions 8X{. E. Sufficient Conditions and the Weierstrass Excess Function

T h e Carathéodory method leads quite easily to a set of sufficient conditions. Basically, these are just hypotheses that guarantee that L*(ty xy u) satisfies condition ( 1 2 ) . We will show that they are the optimal control analogs of the classical sufficient conditions (26y pp. 1 4 6 - 1 4 9 ; 2 7 , pp. 8 3 - 8 7 )

which are based on the Weierstrass excess function. T o facilitate the statement of the theorems, we define: HYPOTHESIS A . Let L and the f/s of functional ( 7 ) and system ( 6 ) be continuously differentiable in all their arguments. Let S(tyx) (used in L * ) be a twice continuously differentiable function which is a solution of the HamiltonJacobi equation ( 2 0 ) where the function °ll(ty x,p) is continuously differentiable and satisfies the minimizing condition ( 1 6 ) . Let x(t) be the trajectory starting at x(t0) = x0 and ending at x(T) = xT which results from using

as the control. THEOREM 1.

Assume Hypothesis A holds. Then, if for each point (t, x) and

for all u e U, L*(t,x,u)^0

(33)

the trajectory x(t) is optimal and ü[t, x(t)] is an optimal control in U for the end states given. Proof. On the basis of these assumptions, the equivalent problem (see Section I I , B ) has J*[u] ^ 0 for all u Ε U> and J*[u] = 0 since the HamiltonJacobi equation is satisfied. Therefore, ü(ty χ) minimizes the functional and hence is optimal.

Q.E.D.

152

DONALD R. SNOW

By analogy to the classical definition of the Weierstrass excess function we define the following function : E(t, x, v, u) = L*(ty x, u) - L*(t, *» «0 - 2 Α

= L(ty x, u) - L(ty xyv)-^ dL ,

~ *

>

^

-^(t, x) [fi(t, x, u) -/,·(*, xy v)] 1 i dS, v df: Κ - Ο (35)

^

x

V

3ΪΓ ^' *' ^ Α

Using the 5 ( ί , χ) of Hypothesis A and replacing ν by w, Eq. (34) becomes E(tyxyüyu)

(36)

= L*(tyxyu)

since L*(ty xy u) = 0 and the last term vanishes by the minimizing condition (16). Also, for S(ty x) and w, Eq. (35) becomes E(tyxyüyu)

= H0^tyxy^yü^

— H0^tyxy^yuj

(37)

where H0 is the (Pontryagin) Hamiltonian defined in (23). Hence, for S(ty x) and ü(ty x)y L*(tyxyu)

^ , üj - H0^tyxy^,uj

= H0^tyxy

(38)

We now see the relationship between the Carathéodory approach, the Weierstrass excess function, and the Pontryagin maximum principle. T h e analog of the Weierstrass necessary condition would be that E(ty xy w, u) ^ 0 along an optimal trajectory. By Eqs. (36)—(38), this corresponds to the Hamiltonian being maximized along an optimal trajectory for a certain set of adjoint variables, i.e., to the Pontryagin maximum principle. And this condition in terms of Carathéodory approach says that L*(ty xyu)^0 along an optional trajectory. By (34), we see that if L * is twice continuously differentiable in the w a's, the excess function is the difference between the value of the function L*(ty xy u) (as a function of the w a's) at the point u and the first two terms of its Taylor series expansion about the point v. Thus, E(ty xy vy u) can also be expressed as the remainder of the Taylor series: i.e., I — = ^2^

2

3 L* x

ü

(*> > ) Κ - Ο ("β - *>β) OL, β * ß where ü = ν + θ(η — v)y 0 ^ θ ^ 1. Or, by (36), with an appropriate S(ty x) and ü(ty x)y E(t, xyvyu)

L*(ty xyu) = 2 2, Α,

du

du

x

du

β

d uß

ü

δ

u

(*> > ) Κ - « ) ( ß - "β)

(39)

REACHABLE REGIONS AND OPTIMAL CONTROLS

153

where ü = u + 6(u — u), O ^ 0 ^ 1 . Let us consider the matrix of this quadratic form :

Note that this matrix is also given by Γ

2

d H0

I

dS,

.

\1

THEOREM 2. Assume Hypothesis A holds. Furthermore, assume L*(t, x, u) is twice continuously differentiable in the ua's and that the range set S of the w +1 control set U is convex. Then, if for each (t, x) in some open region of Z? containing (t, x) the matrix (40) is positive semidefinite for all controls in U, the trajectory x(t) is an optimal trajectory in the open region and u[t, x(t)] is an optimal control in U for this open region.

Proof L*(t, x, u) convexity whenever

Positive semidefiniteness of matrix (40) guarantees that ^ 0 by (39) for all (t, x) in the open region and all u eU. The of the control range is necessary to insure that ü in (39) is in U u is. Applying Theorem 1 completes the proof. Q.E.D.

This theorem guarantees that x(t) is a local optimum only unless the n+1 n+1 open region in E is all of E . Also, we cannot conclude that x(t) is unique even in the open region since positive semidefiniteness leaves open the possibility that there are other admissible controls which also make L*(t, x, u) vanish. However, under the following conditions we can conclude that the optimal trajectory and control are unique. THEOREM 3. Assume the hypothesis of Theorem 2 holds but that matrix (40) is positive definite on the open region. Then x(t) is the unique optimal trajectory in the open region and ü [t, x(t)] is the unique optimal control in U for this n+1 region. If the open region is all of E , then the optimal trajectory and control are globally unique.

Proof. Since matrix (40) is positive definite, its determinant is nonvanishing. Then the implicit function theorem guarantees that the minimizing conditions (16) have a unique solution °ll{t, x,p) and this solution is continuously differentiable. Also, positive definiteness of (40) in conjunction with Eq. (39) shows that L*(t, x,u) > 0 for all u Φ ü, at each point n+X (t,x) of the open region in E . As in Theorem 2, the convexity of the

154

DONALD R. SNOW

control range is necessary to insure that ü is in U whenever u is. Hence,

(

t, X,

dS

\

(ty X) /

is the only control which makes L * [t, x, ü(t, χ)] = 0 in the open region. Then x(t) and ü[ty x(t)] are the unique optimal trajectory and control, respectively.

Q.E.D.

In applying the sufficient conditions given in the theorems of this subsection, we must find, or at least show the existence of, a suitable solution S(t, x) of the Hamilton-Jacobi equation. This is equivalent to showing that the optimal trajectory can be embedded in a field.

F. Solution of the Hamilton-Jacobi Partial Differential Equation by the Method of Characteristics

The characteristic strips of a partial differential equation consist of curves on solution surfaces of the equation together with a tangent plane at each point of the curve which tangent plane coincides with the tangent plane of the solution surface at that point (28, see pp. 9 7 - 1 0 3 ) . It is equivalent to consider normal vectors at each point of the curve instead of tangent planes. We will now show that the ordinary differential equation system which describes the characteristic strips of the Hamilton-Jacobi equation, Eq. ( 2 0 ) , is the original system ( 6 ) plus the adjoint system ( 2 4 ) with the function ( 1 7 ) used as the control function. The Hamilton-Jacobi equation is

where H(t, x,p) = Jpifi[t, xy W(t, x,p)] - L[t, x, W(t, x,p)]

(19)

Ι

Suppose ζ = S(t, x) is a solution surface of the partial differential equation on the considered in t,χ,ζ space, and let P0 be the point z0 = S(t0,x0) surface. We will find conditions so that a strip (curve and normal vectors) which coincides with the surface and normal at P0 is a characteristic strip. T o describe a characteristic strip we will determine the characteristic curve x(t) and the normal or gradient p(t) = dS[t, x(t)/dx] of the surface along the curve. Suppose x(t) satisfies

155

REACHABLE REGIONS AND OPTIMAL CONTROLS

We integrate this system using x0 as the initial condition to obtain x(t) and use this in S (t> x) to obtain z(t) = S[t, x(t)] Since dS

MO = ^ M O I we can differentiate to obtain P

M

~ dtdxi

+

Xk

Ζ

dx^Xi 2

2

dS ? 3 S dH/ V tyXi dt dx,^ZdxkdXidpk[

8S\ )

dx)

Since the Hamilton-Jacobi equation is an identity in the x/s, we can differentiate it with respect to the x/s to obtain 2

dS

dH \-

2

scdHdpk μ y

dS

Λ

n r

= 0

or

ν h >

dx{ dt dx{ ^ dpk dx{ Hence (42) becomes

dx{ dt

^

92

5 dH

dH

— =

dx{ dxk dpk

A=- | f

dx{

( ) 43

We can then write (41) without reference to 35/dx by using/): (44)

*i = ^(t,x9p)

Thus, if x(t) and p(t) satisfy (44) and (43), they describe a characteristic strip. Let us now compute dH/dpj and dH/dx^ T o do this, we consider H and .,/>„. °U to be functions of the 2n + 1 independent variables t, xt,.. .,xn,pi,.. Then keeping in mind the minimizing conditions (16) we obtain f . = / * * , * > + ^

=

2

fi(t,X,€)

Α

|
x, üyp) = 0, where Aa(ty x, ü,p) is given by (66), solve this system for ü = W°(ty x>p). (4) Using the functions and write the three Hamilton-Jacobi partial differential equations, (67). + (5) In each region of (t, x) space where ^ is continuously differentiable, find a time-optimal trajectory to use as a data strip there. Each data strip must be noncharacteristic for the corresponding Hamilton-Jacobi equation in that region. Do the same for °U~ and (6) Solve the Hamilton-Jacobi partial differential equations in each of these regions by the method of characteristics, using the corresponding data strip there as the given boundary condition. (7) T h e reachable region is the region bounded by the surfaces S(t, x) = S(t0ix0) where S(t, x) represents the solutions of the HamiltonJacobi equations. In specific problems, the Hamilton-Jacobi equation corresponding to ^ ° may not be needed, since this may not determine any surface.

173

REACHABLE REGIONS AND OPTIMAL CONTROLS

IV. Rigorous Method of Determining the Reachable Region for a Particular Class of Problems A.

3

Introduction

In this section we will examine a class of optimal control problems. It will first be shown that this class of problems is singular, in the sense that there are initial states for which the Pontryagin maximum principle cannot be used to determine the optimal controls. Then, a rigorous method of solution for these problems will be presented which determines the controllable (or reachable) regions, as well as the optimal controls. It will be presented from the standpoint of determining the T-controllable region. T h e conversion to the T-reachable region may be made as indicated in Section I , E . T h e rigorous method will be applied to a particular subclass of these problems in the next section. Also in Section V, the reachable region for this subclass will be determined by the heuristic method of Section I I I and it will be shown that the results of the two methods are the same. We will consider the following scalar differential equation : χ + a(t) χ + b(t) χ + c(t) = u(t)

(77)

where a(t) and b(t)

are given continuous functions,

c(t)

is a given piecewise continuous function,

u(t)

is the control function, assumed to be in the class U,

U

is the set of all piecewise continuous functions u(t)y \u\ ^ 1, on 0 ^ t < Γ ,

Τ

is a given (fixed) number.

T h e initial state and desired terminal state on x(t) are WO)^(0)] = (^o^o)

(78)

Μ Γ ) , * ( Γ ) ] = (0,0)

(79)

and T h e functional or payoff function to be minimized is taken to be

J[u] = ]l\u{t)\dt 3

T h e material in Sections IV, B - V , A also appears in (30).

(80)

174

DONALD R. SNOW

The general problem may then be formulated as : find a function u(t) e U such that the solution of the differential equation (77) has initial state (78), terminal state (79), and makes (80) as small as possible. This is a minimum fuel or effort problem. Aspects of this problem or solutions to specific examples have been discussed by various authors (31-34). A general discussion of minimum effort control problems is given in (35). As is proved in Appendix 1 of (36), any completely controllable linear system with two state variables can be described by Eq. (77) by using a suitable nonsingular linear transformation.

B. C o n d i t i o n of Singularity

Since 1958 the standard method of solution of problems of the type posed in the previous section has been the application of the Pontryagin maximum principle [see Section I I , D above; (24, 10, 25)] or some modification of it. It will be shown in this subsection that for those systems where

b(t) = a(t)

(81)

this method breaks down for large regions of initial states because of insufficient characterization of the optimal control. Note that condition (81) is precisely the condition under which the differential expression in Eq. (77) is "exact" [see Eq. (87)]. T h e maximum principle, which is a necessary but not sufficient condition for optimality, is not useful here since an infinity of control functions are described by it, not all of which are optimal. Problems for which (81) holds therefore belong to the class of singular optimal control problems. We will present a rigorous analytic method of solution of these singular problems. A recent paper on singular control problems is (22). We will now show that when condition (81) holds the problem is singular. Let X\ — X and x2 — x be the state variables and px and p2 be the adjoint variables of the Pontryagin method. Using these variables, the equivalent [to Eq. (77)] first-order system, the Hamiltonian, and the adjoint system are : (i) Equivalent system : x2 = —a(t) x2 — b(t) xx — c(t) + u

xx = x2,

(82)

(ii) Hamiltonian:

H = px l

1

Λ-piXi — \u\

= P\ 2-P2 ί

dH

p2 = —^ = p2 1 \p2(t)\ d integrate the system of Eqs. ( 8 2 ) and ( 8 3 ) , choosing u(t) according to requirement ( 8 4 ) . I f this trajectory does not have the desired terminal state ( 0 , 0 ) , we guess new values of />i(0),/> 2 (0) and try again. D E F I N I T I O N . A T-controllable (see Section I , D ) initial state (XQ,XQ) for which the Pontryagin maximum principle characterizes at least one optimal control will be called a regular initial state ; i.e., there is at least one set of adjoint variable initial conditions such that ( 8 4 ) gives an optimal control. All other T-controllable initial states will be called singular initial states. THEOREM 1. There are singular initial states for this problem if the coefficients of the differential equation ( 7 7 ) satisfy

b(t) = a(t)

(81)

Remark. I f this condition holds only on subintervals of 0 ^ t ^ T, the problem can be treated separately on these ''intervals of singularity/' Note that condition ( 8 1 ) does not involve c(t). Proof. We first show that condition ( 8 1 ) is equivalent to p2(t) = ± 1 being a permissible solution of the adjoint system ( 8 3 ) . When ( 8 1 ) holds, the adjoint system leads to the equation p2 = a(t)p2 which has the solutions p2(t)=±l. Conversely, substituting pi(t)=±^ into ( 8 3 ) , we get p\(t) =±a(t) and/>!(*) = ±b(t). Hence ( 8 1 ) follows. Now note that, when p2(t) is given, requirement ( 8 4 ) determines the optimal control uniquely except at the values of t for which p2(t) = ± 1 . I f these exceptional values of time are isolated, the value of the optimal control at these times is immaterial. But when p2(t) = + 1 , we have H = u— \u\, and then any u ^ 0 will maximize H giving the maximum value H = 0 . When p2 = —l, any w ^ O will maximize H, again giving the maximum value H = 0 . Thus, when p2(t) = ± 1 , the maximum principle does not characterize the optimal control except to indicate that it must not change sign.

176

DONALD R. SNOW

It will be shown in Section I V , E that the two choices of adjoint variable initial conditions [pi(0),p2(0)] = ±[a(0)y 1] which give p2(t) = ± 1 correspond to two regions of Γ-controllable initial states. It will also be shown in Section I V , E that there are no other adjoint variable initial conditions that lead to optimal controls for initial states in these regions. Hence these are regions of singular initial states. Q.E.D.

C.

A

Lemma

We now state a lemma without proof. LEMMA 1. Given any K(t) ^ 0, continuous and strictly monotone increasing 1 A on [0, 7 ], and two real numbers A and M with 0 ^ A < MT. Let UM be the class of all piecewise continuous functions with 0 ^ u(t) ^ M on [0, T] which have

j\(t)dt A

For each u e UM ,

=A

let T

jo

(85)

K(t)u(t)dt

Let uL{t) -

0 < t^ A/M AjM ^t^T

M, 0,

and uR{t)

JO, \M,

O^t^T-A/M T-AjM^t^T A

(see Fig. 6). Then, for all u e UM , CT

,.s

,.

we have .

~

.

K(t) uL(t) dt < Β
^Ο)· Region R and its subregions are independent of the functional to be minimized, but it will be shown in Section I V , E that the regions P° and iV° are the regions of singular initial states when the particular functional (80) is used. THEOREM

3. An initial state (x0y x0) is in region R if and only if (92)

-T^A^T and

jy

+T)/2

2

J K(t) T

K{t) dtx0) R- Then there is a u e U satisfying (90) and (91), and hence a v(t) satisfying (94) and (95). Since 0 < ν ^ 2, (94) shows that Use of Lemma 1 with M = 2 and the left-hand sides of (94) and (95) as the constants gives

or

s

Now suppose that the initial state (x0 > *o) * such that A and Β satisfy (92) and (93). By Lemma 1, there is at least one v{t), 0 ^ ν ^ 2, that satisfies (94) and (95). This v(t) corresponds to a u g U that is admissible. Hence (x0,x0)eR. Q.E.D. THEOREM

4.

initial state (x0, x0) is in region Ρ if and only if (96)

O^A^T and A

T

j oK(t)dt^B^j T_AK(t)dt

(97)

where A, B, and K(t) are defined by (88) and (89). The boundaries of the region are given by considering equality to hold on the left and right, respectively, in inequality (97). Proof. Suppose (x0, x0) G P. Then there i s a w G Î 7 , O ^ w ^ l , satisfying (90) and (91). Condition (90) with 0 ^ u ^ 1 gives 0^ Α ζ Τ Use of Lemma 1 with M =1 gives T

j o K(t) uL(t) dt^Bti

T

j Q K(t) uR(t) dt

By the meaning of uL and uR in Lemma 1 this is just (97). Now suppose initial state (x0, x0) is such that A and Β satisfy (96) and (97). By Lemma 1 with M = 1 there is at least one u e U with 0 ^ u ^ 1 that satisfies (90) and (91). By Theorem 2 this u is admissible and, since u ^ 0, we have (x0,

x0)

e P.

Q.E.D.

180

DONALD R. SNOW

By analogous reasoning with — u(f) we can prove the following : THEOREM

5. An initial state (x0, x0) is in region Ν if and only if (98)

-T^A^O and -\

T

K(t) dt^B^-

t +a

j~

A

(99)

K{t) dt

where A, B, and K(t) are defined by (88) and (89). The boundaries of the region are given by considering equality to hold on the left and right, respectively, in inequality (99). By (96), (98), (97), and (99) the regions Ρ and Ν are disjoint except for the initial state corresponding to A = Β = 0. This unique point in R is the initial state for which the control u(t) = 0 is admissible, and is the origin if and only if c(t) = 0. When c(t) = 0, the Γ-controllable region is symmetric with respect to the origin since then — u in place of u in Eq. (87) leads to the solution —x instead of x. For this case region Ν is the image of region Ρ under reflection in the origin. Region R transformed into AB space is always symmetric with respect to the origin [regardless of c(t)] since conditions (90) and (91) are linear and homogeneous in u(t) and the class U is symmetric. We will now subdivide the set of initial states R ~ (P° Ό N°) into four mutually disjoint sets Rif i= 1 , 2 , 3 , 4 . It will be shown in Section I V , F that the set R ~ (P° Ό N°) is the set of regular initial states and that the optimal controls for initial states in each of the regions R{ have a specific form. T o define these regions, let

2Jl

A+Tm

BL = B

^

2

\

T (

T

K(t)dt-j oK(t)dt K

T - ^

^

d

t

K

- l l

^

d

(100)

t

and, if 0 < A < T, let BLP

A

= j

T

K(t) dt,

Β «ρ = j T _ A K{t) dt

(101)

or, i f - T s : A < 0, let

™ SI a

B

=

K{t)

>

dt

+

-\T

Brn=

K{t) dt

(102)

Then, if we know the value of A, the second inequality in the description of region R, inequality (93), can be written as BL

^B ^ BR

REACHABLE REGIONS AND OPTIMAL CONTROLS

181

If A ^ 0, region P, which requires 0 ^ A ^ T, is described by [see (97)] BLP

< Β

< P

Ä P

Since P x2 = -ax2 + u (131)

REACHABLE REGIONS AND OPTIMAL CONTROLS

191

We will now determine the T-reachable region relative to X\(0) = x2(0) = 0 by the heuristic method developed in Section I I I and show that it is the same as the region described by Eqs. (126)—(128). I f a = 0, we must evaluate all quantities involving a by taking the limit as a -> 0. For this problem, the function φ(μ) which describes the control set in Section Ι Π , Α is φ(υ) = u. Hence, the functional we introduce [see (53)] is

(132)

J [u] = j u™dt T

k

o

The minimizing condition (16) is then 2k l

2kü - =p2 or 2k

(133)

ü = (p2ßk)^ -^ In the limit, |w| = 1, as shown by Eq. (74), or else p2 = 0. The limiting Hamilton-Jacobi equation, (67), is dS

3S

35

._

x

ä **äi 0 for 0 ^ t ^ Τ we get a time-optimal control u* = +1 Similarly, if we choose adjoint variable initial conditions so that p2(t) < 0 for 0 ^ t ^ Ty we get another time-optimal control u* = - \ If we use either of these controls in system (131) and determine the trajectory passing through #i(0) = #2(0) = 0 , we obtain

. =

-a

„ = ,[!^