Representations of Solvable Lie Groups: Basic Theory and Examples 1108428096, 9781108428095

Table of contents :
Contents
Preface
1 Basic theory of solvable Lie algebras and Lie groups
1.1 Solvable Lie algebras
1.2 Representations of a Lie algebra and weights
1.3 The Lie theorem and its first consequences
1.4 Adjoint and coadjoint representations
1.5 Ado theorem for a solvable Lie algebra
1.6 Lie groups
1.7 Nilpotent and solvable Lie groups
1.8 Exponential groups
1.9 Finite-dimensional group representations
2 Stratification of an orbit space
2.1 Matrix normal form
2.2 Layers for a representation
2.3 Orbit structure for a completely solvable action
2.4 Construction of rational supplementary elements
2.5 Orbit structure for an action of exponential type
2.6 Structure of the generic layer
3 Unitary representations
3.1 Unitary representations
3.2 Decomposition of representations
3.3 Induced representations
3.4 Elements of Mackey theory
4 Coadjoint orbits and polarizations
4.1 Coadjoint orbits
4.2 Polarizations
4.3 Admissibility and the Pukanszky condition
4.4 Isotropic subspaces associated to a flag
4.5 Fine layering and Vergne polarizations
4.6 Positivity and properties of polarizations
4.7 Integral orbits
5 Irreducible unitary representations
5.1 Holomorphic induction
5.2 Construction of irreducible representations
5.3 Orbit method for solvable groups
6 Plancherel formula and related topics
6.1 Invariant measure on a coadjoint orbit
6.2 Character formula
6.3 Semicharacters and the Plancherel formula
List of notations
Bibliography
Index

Citation preview

Representations of Solvable Lie Groups The theory of unitary group representations began with finite groups, and blossomed in the twentieth century both as a natural abstraction of classical harmonic analysis, and as a tool for understanding various physical phenomena. Combining basic theory and new results, this monograph is a fresh and self-contained exposition of group representations and harmonic analysis on solvable Lie groups. Covering a range of topics, from stratification methods for linear solvable actions in a finite-dimensional vector space, to complete proofs of essential elements of Mackey theory and a unified development of the main features of the orbit method for solvable Lie groups, the authors provide both well-known and new examples with a focus on those relevant to contemporary applications. Clear explanations of the basic theory make this an invaluable reference guide for graduate students as well as researchers. D i d i e r A r n a l is Emeritus Professor at the University of Burgundy and previously was Director of the Burgundy Mathematics Institute. He instituted and has worked over the past 15 years on a cooperation project between France and Tunisia. He has authored papers on a diverse range of topics including deformation quantization, harmonic analysis, and algebraic structures. B r a d l e y C u r r e y III is Professor at Saint Louis University (SLU), Missouri. Formerly the Director of Graduate Studies in Mathematics at SLU, he has also served as a co-organizer in the Mathematics Research Communities program of the American Mathematical Society. Much of his recent research has explored the interplay of the theory of solvable Lie groups and applied harmonic analysis.

N E W M AT H E M AT I C A L M O N O G R A P H S Editorial Board Béla Bollobás, William Fulton, Frances Kirwan, Peter Sarnak, Barry Simon, Burt Totaro All the titles listed below can be obtained from good booksellers or from Cambridge University Press. For a complete series listing visit www.cambridge.org/mathematics. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38.

M. Cabanes and M. Enguehard Representation Theory of Finite Reductive Groups J. B. Garnett and D. E. Marshall Harmonic Measure P. Cohn Free Ideal Rings and Localization in General Rings E. Bombieri and W. Gubler Heights in Diophantine Geometry Y. J. Ionin and M. S. Shrikhande Combinatorics of Symmetric Designs S. Berhanu, P. D. Cordaro and J. Hounie An Introduction to Involutive Structures A. Shlapentokh Hilbert’s Tenth Problem G. Michler Theory of Finite Simple Groups I A. Baker and G. Wüstholz Logarithmic Forms and Diophantine Geometry P. Kronheimer and T. Mrowka Monopoles and Three-Manifolds B. Bekka, P. de la Harpe and A. Valette Kazhdan’s Property (T) J. Neisendorfer Algebraic Methods in Unstable Homotopy Theory M. Grandis Directed Algebraic Topology G. Michler Theory of Finite Simple Groups II R. Schertz Complex Multiplication S. Bloch Lectures on Algebraic Cycles (2nd Edition) B. Conrad, O. Gabber and G. Prasad Pseudo-reductive Groups T. Downarowicz Entropy in Dynamical Systems C. Simpson Homotopy Theory of Higher Categories E. Fricain and J. Mashreghi The Theory of H(b) Spaces I E. Fricain and J. Mashreghi The Theory of H(b) Spaces II J. Goubault-Larrecq Non-Hausdorff Topology and Domain Theory ´ J. Sniatycki Differential Geometry of Singular Spaces and Reduction of Symmetry E. Riehl Categorical Homotopy Theory B. A. Munson and I. Voli´c Cubical Homotopy Theory B. Conrad, O. Gabber and G. Prasad Pseudo-reductive Groups (2nd Edition) J. Heinonen, P. Koskela, N. Shanmugalingam and J. T. Tyson Sobolev Spaces on Metric Measure Spaces Y.-G. Oh Symplectic Topology and Floer Homology I Y.-G. Oh Symplectic Topology and Floer Homology II A. Bobrowski Convergence of One-Parameter Operator Semigroups K. Costello and O. Gwilliam Factorization Algebras in Quantum Field Theory I J.-H. Evertse and K. Györy Discriminant Equations in Diophantine Number Theory G. Friedman Singular Intersection Homology S. Schwede Global Homotopy Theory M. Dickmann, N. Schwartz and M. Tressl Spectral Spaces A. Baernstein II Symmetrization in Analysis A. Defant, D. García, M. Maestre and P. Sevilla-Peris Dirichlet Series and Holomorphic Functions in High Dimensions N. Th. Varopoulos Potential Theory and Geometry on Lie Groups

Representations of Solvable Lie Groups Basic Theory and Examples

D I D I E R A R NA L University of Burgundy BRADLEY CURREY Saint Louis University

University Printing House, Cambridge CB2 8BS, United Kingdom One Liberty Plaza, 20th Floor, New York, NY 10006, USA 477 Williamstown Road, Port Melbourne, VIC 3207, Australia 314–321, 3rd Floor, Plot 3, Splendor Forum, Jasola District Centre, New Delhi – 110025, India 79 Anson Road, #06–04/06, Singapore 079906 Cambridge University Press is part of the University of Cambridge. It furthers the University’s mission by disseminating knowledge in the pursuit of education, learning, and research at the highest international levels of excellence. www.cambridge.org Information on this title: www.cambridge.org/9781108428095 DOI: 10.1017/9781108552288 © Didier Arnal and Bradley Currey 2020 This publication is in copyright. Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press. First published 2020 Printed in the United Kingdom by TJ International Ltd, Padstow Cornwall A catalogue record for this publication is available from the British Library. Library of Congress Cataloging-in-Publication Data Names: Arnal, Didier, 1946- author. | Currey, Bradley, III 1955- author. Title: Representations of solvable Lie groups : basic theory and examples / Didier Arnal, Bradley Currey III. Description: Cambridge, United Kingdom ; New York, NY : Cambridge University Press, 2020. | Series: New mathematical monographs | Includes bibliographical references and index. Identifiers: LCCN 2019038659 (print) | LCCN 2019038660 (ebook) | ISBN 9781108428095 (hardback) | ISBN 9781108552288 (epub) Subjects: LCSH: Representations of Lie groups. | Solvable groups. Classification: LCC QA387 .A76 2020 (print) | LCC QA387 (ebook) | DDC 512/.482–dc23 LC record available at https://lccn.loc.gov/2019038659 LC ebook record available at https://lccn.loc.gov/2019038660 ISBN 978-1-108-42809-5 Hardback Cambridge University Press has no responsibility for the persistence or accuracy of URLs for external or third-party internet websites referred to in this publication and does not guarantee that any content on such websites is, or will remain, accurate or appropriate.

Dedicated to Françoise and Julie for their support and patience.

Contents

Preface

page ix

1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9

Basic theory of solvable Lie algebras and Lie groups Solvable Lie algebras Representations of a Lie algebra and weights The Lie theorem and its first consequences Adjoint and coadjoint representations Ado theorem for a solvable Lie algebra Lie groups Nilpotent and solvable Lie groups Exponential groups Finite-dimensional group representations

2 2.1 2.2 2.3 2.4 2.5 2.6

Stratification of an orbit space Matrix normal form Layers for a representation Orbit structure for a completely solvable action Construction of rational supplementary elements Orbit structure for an action of exponential type Structure of the generic layer

68 69 84 104 120 130 154

3 3.1 3.2 3.3 3.4

Unitary representations Unitary representations Decomposition of representations Induced representations Elements of Mackey theory

166 166 190 218 248

4 4.1 4.2

Coadjoint orbits and polarizations Coadjoint orbits Polarizations

269 269 278

vii

1 1 5 10 19 23 28 36 52 56

viii

Contents

4.3 4.4 4.5 4.6 4.7

Admissibility and the Pukanszky condition Isotropic subspaces associated to a flag Fine layering and Vergne polarizations Positivity and properties of polarizations Integral orbits

283 289 297 310 319

5 5.1 5.2 5.3

Irreducible unitary representations Holomorphic induction Construction of irreducible representations Orbit method for solvable groups

333 334 351 369

6 6.1 6.2 6.3

Plancherel formula and related topics Invariant measure on a coadjoint orbit Character formula Semicharacters and the Plancherel formula

402 403 409 418

List of notations Bibliography Index

433 440 445

Preface

The theory of unitary group representations began with finite groups, and blossomed in the twentieth century both as a natural abstraction of classical harmonic analysis and as a tool for understanding various physical phenomena. An important early step for abstract harmonic analysis was the theorem of Stone–von Neumann, whereby all irreducible unitary representations of the Heisenberg group are classified up to isomorphism. Subsequent work of J. Dixmier and A. A. Kirillov may be regarded as a generalization of this theorem to all simply connected nilpotent Lie groups, and ushered in the era of the method of coadjoint orbits for the more general solvable Lie groups. The list of pioneers and important results in this subject is long and distinguished; three central achievements are the Auslander–Kostant classification of irreducible unitary representations of type 1 simply connected solvable groups, the Duflo– Moore Plancherel theory, and the extension by L. Pukanszky of the Auslander– Kostant theory to non-type 1 solvable groups. A milestone in this development is the broadly influential monograph [11] by Bernat et al., in which many related results are described. During this time equally prodigious activity was devoted to the classification of irreducible unitary representations of semisimple Lie groups. The later part of the twentieth century saw an expanded and deepened understanding of the unitary dual via the orbit method, as well as the application of the orbit method in analysis on homogeneous spaces. After the essentially exhaustive construction of factor representations, the focus shifted to properties of these representations, and in the type 1 case, to the orbital parametrization for irreducible unitary representations, as well as to the orbital decomposition of naturally occurring reducible unitary representations. Harmonic analysis also witnessed the rise of the theory of non-orthogonal expansions. Just as time-frequency analysis was found to be deeply related

ix

x

Preface

to harmonic analysis on the Heisenberg group, discrete wavelet analysis was found to be closely related to harmonic analysis on the affine group. In the context of Duflo–Moore theory, general methods such as co-orbit theory for discretization of unitary representations provided precise descriptions of this connection. This book arises in the context of the continuing exploration of the role of abstract harmonic analysis in contemporary applications. The goal of this book is to develop abstract harmonic analysis in the context of a very concrete presentation of solvable Lie algebras and Lie groups and their representations. A real, connected, simply connected Lie group is completely determined by its Lie algebra, and every Lie algebra has Levi decomposition as a direct sum of a solvable ideal and a semisimple subalgebra. The theory of semisimple Lie algebras is very well known: there is a complete classification with a very precise and useful presentation, using Cartan subalgebras and roots. On the other hand, solvable Lie algebras are very far from being classified, even though the topology of the corresponding simply connected Lie group is homeomorphic with Rn . Nevertheless, the structure of solvable Lie algebras allows for explicit embeddings into matrix algebras which, though not canonical, are useful for many purposes. Moreover, solvability makes possible a broadly successful method for explicitly describing large classes of unitary representations, including irreducible unitary representations, up to unitary equivalence. This method is based upon information about the orbits of the coadjoint representation – the canonical linear action of the group on the linear dual of its Lie algebra – and is closely related to the construction due to G. Mackey of an irreducible unitary representation – an induced representation – via the action of the group on the unitary dual of a normal subgroup. Indeed, a precise understanding of this relationship in the case where the normal subgroup is a vector group is motivated by contemporary applications of continuous and discrete wavelets. When the group is exponential, i.e., when the exponential map from the Lie group to the Lie algebra is a global diffeomorphism, then the theory of unitary representations is understood via a canonical and explicit mapping K associating to each coadjoint orbit O a unitary irreducible representation K(O). By virtue of the work of J. Ludwig (see the book of Leptin and Ludwig [61]) this map is understood to be a homeomorphism between the quotient space of coadjoint orbits and the unitary dual of the Lie group with the Fell topology. For connected, simply connected, non-exponential solvable groups, the exponential mapping is neither injective nor surjective (in fact not even everywhere regular) and irreducible unitary representations are no longer described

Preface

xi

simply by coadjoint orbits. For many such groups however, the coadjoint orbits together with additional data arising from the topology of the orbits, is sufficient to explicitly construct all irreducible unitary representations. This is the main result of [9], where Auslander and Kostant build and describe all the irreducible unitary representations of type 1 solvable groups. However, if the group is not type 1, so that irreducible decompositions of unitary representations may not be unique or even well-defined, coadjoint orbits must be replaced by so-called generalized orbits, in order to construct factor representations. Still, for the class of type 1 solvable groups (which includes the exponential groups) the theory of irreducible unitary representations is, in some sense, simpler than for semisimple Lie groups, in that there is essentially a single construction (albeit a complicated one) that always works. In Chapter 1, elementary results essential for solvable Lie algebras and Lie groups are presented in detail. Beginning with Lie algebras, we present fundamental definitions and results for representations of solvable Lie algebras in detail, furnishing examples along the way. When Lie groups are taken up, we briefly summarize elementary theory of Lie groups without proof, citing numerous references. Nevertheless, a number of theoretical facts specific to solvable Lie groups are proved for completeness. In particular, we recall the characterization of exponential Lie groups due to Dixmier, even though this has been presented in numerous prior sources. Important facts about finitedimensional representations of solvable Lie groups are also proved. Examples are provided at nearly every step. In this sense the book is almost selfcontained: many elements of Lie theory and representation theory are discussed in detail, and simplifications that result when restricting consideration to matrix Lie groups will suffice to account for much of the theory for which proofs are omitted. It is our hope that the book is broadly accessible and a useful tool even for beginners in Lie theory. In Chapter 2, a stratification procedure for classifying orbits of a linear action of a solvable Lie group G on a finite-dimensional real vector space V is described in detail. This entails a suitable choice of basis for the complexification of V and a decomposition of V into finitely many G-invariant subsets called “layers,” in each of which the G-orbits are homeomorphic. If the action in a given layer is regular (that is, produces locally compact orbits) then the quotient space of orbits in the layer is Hausdorff, and for broad classes of groups can be explicitly parametrized. Since each solvable Lie group can be realized as a matrix group, application of stratification methods to the adjoint action gives a presentation of any solvable Lie group, allowing to “localize” in some sense the singularities of the action of such a Lie group. Of course

xii

Preface

such a presentation does not provide a classification, but it gives a powerful way to produce a wide variety of examples of solvable Lie algebras and Lie groups in a very explicit and useful form. Therefore, stratification methods are used throughout the book for describing examples of solvable Lie groups and their actions. In particular, when the action in question is the coadjoint action, we use our presentation to build recursively “orthogonal” elements in the Lie algebra, rationally depending upon point in a given layer. For the class of exponential groups, we present a parametrization of the quotient space of orbits in a given layer by way of explicit cross-sections, and we indicate through examples how this can be done for metabelian groups as well. Basic facts and results about unitary representations are presented in Chapter 3. The inducing construction is developed in detail, as it is a crucial tool for subsequent results. The approach to unitary representations of solvable groups via coadjoint orbits is based upon the fundamental notion of a polarization, and the theory of polarizations is developed in detail in Chapter 4. In particular, the notion of polarization naturally leads to a generalization of the inducing construction, the resulting generalization being known as holomorphic induction; this topic is developed in detail in Chapter 5. Application of stratification results from Chapter 2 to the coadjoint representation shows that the usual construction of a polarization is in a certain sense regular in each layer, and within a given layer the associated unitary representations produced by holomorphic induction share useful properties that are used later. Chapter 5 culminates in the fundamental construction of irreducible unitary representations associated to orbit data. In Chapter 6, results of Chapter 2 are applied first to the coadjoint action. An explicit Plancherel formula is proved for exponential groups, in a form which is analogous to the well-known formula of Pukanszky for nilpotent groups (see [79]) which also appears in the book by Corwin and Greenleaf [16]. Much of the material in this book is a distillation of a massive body of research dating from the middle twentieth century. We are especially indebted to the work of prior books, [11], [40], and [16]. For nearly a half-century the monograph [11] of Bernat et al. has served as a seminal source for the theory of unitary representations of solvable Lie groups. The present volume does not intend to supplant this monograph, but rather to augment its presentation of the theory with a development that is illuminated by explicit stratification procedures and numerous examples, and to present a few of the more recent developments and applications of the theory that have appeared after its publication. The present book borrows from the efficient presentation of Mackey theory in the book [40] by G. Folland, upon which we built a self-

Preface

xiii

contained exposition of necessary machinery for solvable groups. The book by L. Corwin and F. Greenleaf [16] serves in some respects as a model for the present work: the inclusion of examples at every step, the application of stratification and cross-sections for finite-dimensional representations, as well as the assumptions regarding the reader’s familiarity with Lie theory, are somewhat imitated in the present text. Just as with the monograph [11], the present volume is envisioned as a supplement or companion to [16]. It is our belief that as such, this book provides a useful path to learning the basics of Lie theory and harmonic analysis in the setting of solvable Lie groups.

1 Basic theory of solvable Lie algebras and Lie groups

Let A(t) be a one-parameter subgroup of n × n invertible matrices with complex entries: t → A(t) is continuous and A(s + t) = A(s)A(t) for all s,t ∈ R. Then A(t) is generated by a single matrix X: there is a unique n × n matrix X such that A(t) = etX , and properties of the matrix X provide a simple way to describe the action of A(t). Consideration of multi-parameter matrix groups leads to the notion of a Lie group, and a connected Lie group is generated in essentially the same sense by its Lie algebra. Just as in the oneparameter case, most fundamental properties of Lie groups and their linear actions are best understood in terms of properties of their corresponding Lie algebras. It is therefore natural to begin the study of solvable Lie groups and their linear actions with a study of solvable Lie algebras.

1.1 Solvable Lie algebras We begin with elementary definitions, facts, and examples. Let g be a vector space over a field k. A skew-symmetric bilinear form [ , ] : g × g → g that satisfies the Jacobi identity [X,[Y,Z]] + [Y,[Z,X]] + [Z,[X,Y ]] = 0, for all X,Y,Z ∈ g, is called a Lie bracket on g. A Lie algebra over k is a vector space g over k that is equipped with a Lie bracket. A Lie algebra is said to be commutative if its Lie bracket is identically zero. Though many of the results presented below will hold for any field k of characteristic zero, we will generally be concerned only with the cases where k is R or C, and a Lie algebra over C will implicitly be regarded as a Lie algebra over R also. Given Lie algebras g and h over a field k, a homomorphism of g into h is a k-linear map β : g → h for which

1

2

Basic theory of solvable Lie algebras and Lie groups

β([X,Y ]) = [β(X),β(Y )] holds for all X, Y ∈ g. An isomorphism of Lie algebras is a bijective homomorphism. Let g be a Lie algebra over k, and let h and k be subsets of g. Define   [h,k] = span [X,Y ] : X ∈ h,Y ∈ k . A subalgebra of g is a k-subspace h for which [h,h] ⊂ h, and a subalgebra h of g is an ideal in g if [h,g] ⊆ h. It is clear that a subalgebra of a Lie algebra g is itself a Lie algebra with Lie bracket inherited from g. If h is an ideal in g, then the quotient space g/h is a Lie algebra when it is equipped with the canonical Lie bracket defined by [X + h,Y + h] = [X,Y ] + h. The center of g is defined by cent(g) = {Z ∈ g : [X,g] = {0}}; it is clear that cent(g) is an ideal in g. Lemma 1.1.1 Let h and k be ideals in g. Then both h + k and [h,k] are ideals. Proof: Since Lie brackets are bilinear, it is evident that h + k is an ideal. To verify that [h,k] is an ideal, let X ∈ h and Y ∈ k. Then for any Z ∈ g, [Z,X] ∈ h and [Z,Y ] ∈ k. The Jacobi identity therefore gives [Z,[X,Y ]] = [[Z,X],Y ] + [X,[Z,Y ]] ∈ [h,k]. Put g(0) = g and define g(k) , k = 1,2,3, . . . recursively by g(k) = [g(k−1), g(k−1) ]. The sequence g = g(0) ⊇ g(1) ⊇ g(2) ⊇ · · · is called the derived series of g. The descending central series is defined similarly, with g(0) = g and g(k) = [g(k−1),g]. Definition 1.1.2 A Lie algebra g is said to be solvable if there is some positive integer n such that g(n) = {0}, and nilpotent if g(n) = {0} for some n. A few observations are in order. By induction one has g(k) ⊆ g(k) for all k, and therefore every nilpotent Lie algebra is solvable. It is evident that any subalgebra of a solvable (nilpotent). Lie algebra is solvable (nilpotent). If g = {0} is solvable then [g,g] = g, and hence g contains an ideal of codimension one, namely any codimension one subspace containing [g,g]. Conversely if g is a Lie algebra containing an ideal h of codimension one such that the Lie algebra h is solvable, then g itself is solvable. Indeed we have g(1) ⊂ h, and by induction, for any k, g(k) ⊂ h(k−1) . If g is nilpotent and n = min{k : g(k) = {0}}, then g(n−1) is contained in the center cent(g) of g; thus if g is nilpotent, then cent(g) = {0}.

1.1 Solvable Lie algebras

3

Example 1.1.3 Given a vector space V over k, the set gl(V ) of endomorphisms of V is a Lie algebra with Lie bracket [A,B] = AB − BA, A,B ∈ gl(V ). It is evident that gl(V ) is finite dimensional if and only if V is finite dimensional. When V = kn , then we write gl(V ) = gl(n,k); gl(n,k) will be automatically identified with the Lie algebra of n×n matrices via the canonical basis, but in some cases other bases may be specified. A subalgebra of gl(m,k) is called a matrix Lie algebra. For 1 ≤ m < n, gl(n,k) contains a subalgebra isomorphic with gl(m,k), namely the range of the homomorphism   A 0 gl(m,k) → gl(n,k), A → .  0 0 Example 1.1.4 Let A be an associative algebra. A map d : A → A is a derivation of A if d(xy) = (dx)y + x(dy) for all x,y ∈ A. The space Der(A) of derivations is a Lie subalgebra of gl(A). Similarly, a derivation of a Lie algebra g is a linear map d : g → g such that d([X,Y ]) = [dX,Y ] + [X,dY ],X,Y ∈ g, and the space Der(g) of derivations of g is a Lie subalgebra of gl(g). Given X ∈ g, the Jacobi identity shows that the map ad(X) : g → g, defined by ad(X)Y = [X,Y ], belongs to Der(g). In addition, one checks that the map ad : g → Der(g) is a Lie algebra homomorphism.  Example 1.1.5 (Heisenberg algebra) Given a positive integer d, the Heisenberg Lie algebra of dimension 2d + 1 is a vector space over R with Lie bracket [·,·], having a basis (Z,Y1, . . . ,Yd ,X1, . . . ,Xd ) such that [Xi ,Yi ] = Z, 1 ≤ i ≤ d, and for all i = j , [Xi ,Yj ] = [Xi ,Xj ] = [Yi ,Yj ] = 0. Its center is then RZ, and since all Lie brackets lie in the center, the Heisenberg Lie algebra is nilpotent. The Heisenberg algebra of dimension three is most simply presented in matrix form as ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 0 1 0 0 0 0 0 0 1 X = ⎣0 0 0⎦ , Y = ⎣0 0 1⎦ , Z = ⎣0 0 0⎦ .  0 0 0 0 0 0 0 0 0 Example 1.1.6 Let aff(R) be the Lie subalgebra of gl(2,R) with basis given by     1 0 0 1 A= , Y = . 0 0 0 0

4

Basic theory of solvable Lie algebras and Lie groups

Then [A,Y ] = Y , aff(R)(1) = RY and aff(R)(2) = {0}. Thus aff(R) is solvable, but for all k = 1,2,3, . . ., aff(R)(k) = RY and hence aff(R) is not nilpotent.  Example 1.1.7 Fix λ ∈ C \ {0}, and let gλ ⊂ gl(2,C) be the span of       λ 0 0 1 0 i A= , X= , Y = . 0 0 0 0 0 0 Writing λ = a + ib, a, b ∈ R, we have [A,X] = aX + bY and [A,Y ] = −bX + aY . If λ = 1 + ib, with b = 0, we shall call the Lie algebra gλ , the Grélaud Lie algebra. Observations similar to those made in Example 1.1.6  show that the Lie algebra gλ is solvable but not nilpotent. Example 1.1.8 Let g ⊂ gl(n,k) be the set of all upper-triangular matrices g = {X = [xi,j ] : xi,j = 0 if i > j }. Observe that g(k) = {[xi,j ] ∈ g : xi,j = 0 if i < j + k} and hence g(n) = {0}. Thus g is solvable. Now g is not nilpotent, since it contains a Lie subalgebra isomorphic with aff(R). We denote g by t(n,k).  The vector space R3 with the usual cross product as the Lie bracket is a Lie algebra that is not solvable since the cross product of R3 with itself is all of R3 . This Lie algebra is easily seen to be isomorphic with so(3): the real Lie algebra of all 3 × 3 matrices X satisfying t X = −X. The following three-dimensional example is also not solvable. Example 1.1.9 The Lie algebra g = sl(2,R) of real 2×2 matrices with trace 0 is spanned by       1 0 0 1 0 1 D= , X= , Y = . 0 −1 1 0 −1 0 One finds that [g,g] = g so that g is not solvable. This Lie algebra is not isomorphic with so(3): the span of D and X + Y is a Lie subalgebra, while so(3) has no two-dimensional subalgebra.  It follows immediately that gl(2,R) is not solvable, and hence for all n ≥ 2, gl(n,R) is not solvable. The Lie algebras so(3) and sl(2,R) are simple Lie algebras in that they have no proper nontrivial ideals. It is known that every three-dimensional simple Lie algebra over R is isomorphic to so(3) or to sl(2,R). By contrast, there are several infinite families of nonisomorphic three-dimensional solvable Lie algebras. A complete classification of three-dimensional solvable Lie algebras up to isomorphism is provided at the end of Section 1.7.2.

1.2 Representations of a Lie algebra and weights

5

In Section 1.5 it is shown that every solvable Lie algebra over R or C is isomorphic with a matrix Lie algebra, in fact, a subalgebra of t(n,C) for some n. Nevertheless it will often be more convenient to present Lie algebra in the same way as Example 1.1.5: by specifying nonvanishing Lie brackets between basis elements (usually written as capital roman letters). This is the way that most of the examples in this book are presented.

1.2 Representations of a Lie algebra and weights Let g be a Lie algebra over k and V a vector space over k. A representation of a Lie algebra g in V is a Lie algebra homomorphism from g into the Lie algebra gl(V ). If dimk (V ) < ∞ then we say that the representation is finite dimensional, and most of the results of Chapters 1 and 2 are for finitedimensional representations. If g is a Lie subalgebra of gl(n,k), then the identity map is called the standard representation of g. More generally, an injective representation is sometimes also called a faithful representation, or simply a realization. Representations β1 : g → gl(V ) and β2 : g → gl(W ) are equivalent if there is a vector space isomorphism T : V → W such that T β1 (X) = β2 (X)T holds for all X ∈ g. When a representation β of g in a vector space V is given, then we shall frequently write β(X)v = Xv, X ∈ g, v ∈ V , and say that V is a g-module. Given a representation β on V , the map β ∗ : g → gl(V ∗ ), defined by β ∗ (X)() = − ◦ β(X),  ∈ V ∗ is a representation of g on the dual space V ∗ of V , called the dual or contragredient representation. If V = kn and V ∗ is identified with kn as usual (via the basis dual to the canonical basis), then β ∗ (X) = −t β(X). In particular the dual of the standard representation of a subalgebra of gl(n,k) is called the transpose representation. It was noted in Example 1.1.4 that the map ad : g → Der(g) is a representation of g; this map is called the adjoint representation. The coadjoint representation of g is the dual ad∗ of the adjoint representation, and will play an important role in the construction of unitary representations. Given representations β1 : g → gl(V ) and β2 : g → gl(W ), the representation β1 ⊕ β2 is defined on V ⊕ W by (β1 ⊕ β2 )(X)(v ⊕ w) = β1 (X)(v) ⊕ β2 (X)(w)

6

Basic theory of solvable Lie algebras and Lie groups

and the representation β1 ⊗ β2 is defined on V ⊗ W by (β1 ⊗ β2 )(X)(v ⊗ w) = β1 (X)(v) ⊗ w + v ⊗ β2 (X)(w). The kernel ker(β) of a representation β is an ideal of g and β defines a faithful representation of the Lie algebra g/ ker(β). Suppose that I is a g-invariant subspace of V . Then there is a natural quotient representation βI of g in V /I for which βI (X)(v + I ) = β(X)v + I . We also say that I is a submodule of V and V /I is a quotient module of V . If the only invariant subspaces in V are {0} and V , then we say that β is irreducible, or that the g-module V is irreducible. Suppose that V is a vector space over R. The complexification of V is the real vector space Vc = V ⊗R C. Vc naturally has the structure of a complex vector space with a(v ⊗b) = v ⊗ab, v ∈ V , a,b ∈ C. Complex conjugation on C defines a natural involution σ on Vc given by σ (v⊗b) = v⊗b and sometimes called the standard conjugation on Vc . The usual practice, which we adopt here, is to write bv instead of v ⊗ b ∈ V ⊗ C, and w = σ (w), w ∈ Vc . For each w ∈ Vc , there are unique elements u, v ∈ V , denoted by Re(w) and Im(w) respectively, such that w = u + iv, and then w = u − iv. Thus V is regarded as an R-subspace of Vc . Note that V  CV = {bv : b ∈ C, v ∈ V }  Vc . An element w ∈ Vc will be called almost real if w belongs to CV . Given a C-subspace J of Vc , it is important to know when J is the complexification of a R-subspace I of V ; in this case we will say that J is real. Lemma 1.2.1 Let J be a C-subspace of Vc . Then the following are equivalent. (i) (ii) (iii) (iv)

J = J. J is spanned over C by a subset of V . J = Ic for some subspace I of V . dimR J ∩ V = dimC J .

Proof: Suppose J = J . Then, for any w ∈ J , u = Re(w), v = Im(w) belong to I = J ∩ V , thus J is spanned over C by I ⊂ V . If J = spanC (X), with X ⊂ V , put I = spanR (X). Then clearly Ic = I + iI = J . If there is a real vector subspace I in V such that J = Ic = I ⊕ iI , then any (real) basis for I is a complex basis for J , dimR J ∩ V = dimR I = dimC J . Finally, suppose dimR J ∩ V = dimC J = dimC J , pick any real basis for J ∩ V = J ∩ V , it is a system of vectors in J and J , which is free on C, thus it is a basis of J and J, J = J. Now let W be a vector space over R and let f : V → W be an R-linear map. Then f extends in the natural way to a C-linear map from Vc to Wc , which we also denote by f , defined by f (iv) = if (v) (v ∈ V ). Observe

1.2 Representations of a Lie algebra and weights

7

that in this case f satisfies f (w) = f (w) for all w ∈ Vc . Conversely, a C-linear map f : Vc → Wc satisfying f (w) = f (w) for all w ∈ Vc is called a real linear map and every such map is the extension of its restriction to V . Thus if β is a representation of the real Lie algebra g on V , then for each X, β(X) is a real linear map on Vc and X → β(X) is a representation of g on Vc , that is Vc is a g-module. Also, V ∗ is regarded as the R-subspace of real linear functionals in (Vc )∗ . Finally, observe that (Vc )∗ is the complexification of V ∗ , with standard conjugation defined by g(Z) = g(Z), g ∈ (Vc )∗ , Z ∈ Vc . (Clearly a similar statement holds for the linear maps from Vc to Wc but we do not need that here.) Another important instance of the preceding is that of a quotient map: let J be a subspace of Vc that is real, with J = Ic . Let π be the canonical quotient map from Vc to Vc /J defined by π(w) = w + J . Then π is a real linear map and is the extension of the canonical quotient map from V to V /I . Observe that the complexification gc of a real Lie algebra g is a Lie algebra over C when the bracket operation is extended in the natural way: [X1 + iY1,X2 + iY2 ] = [X1,X2 ] − [Y1,Y2 ] + i([X1,Y2 ] + [Y1,X2 ]), for Xi ,Yi ∈ g. A subalgebra of gc that is a real subspace of gc will be called a real subalgebra of gc , and is the complexification of a subalgebra of g. It is evident that for any subspaces h and k of g, [hc,kc ] = [h,k]c . Hence gc is solvable (nilpotent), if and only if g is solvable (nilpotent). For representations of solvable Lie algebras a notion of a “common eigenvector” is useful. Given a representation β of g in a complex vector space U (we say that U is a complex g-module), let λ : g → C be a linear form, and put Eλ = {v ∈ U : Xv = λ(X)v for all X ∈ g}. If Eλ = {0}, then we say that λ is a weight for the module U and that Eλ is its weight space. By extension, if V is a g-module over R, then a weight for Vc will also be called a weight for V . Lemma 1.2.2 Suppose that λ is a weight for a complex g-module U . Then [g,g] ⊆ ker λ. Proof: Let X,Y ∈ g, and let v ∈ Eλ,v = 0. Then λ([X,Y ])v = [X,Y ]v = (XY − Y X)v = (λ(X)λ(Y ) − λ(Y )λ(X))v = 0 so λ([X,Y ]) = 0. Example 1.2.3 Fix real numbers a and b, with b = 0, and let g = RX where   a b X= . −b a

8

Basic theory of solvable Lie algebras and Lie groups

Regard X as an element  gl(2,R) in the usual way via the canonical basis

of e1 = 10 and e2 = 01 , and let β be the standard representation of g in V = R2 ; then β is regarded as a representation in U = Vc = C2 . The linear form λ defined by λ(tX) = t (a + ib) is a weight for the identity representation with weight space C(e1 + ie2 ). Its conjugate λ is also a weight. In this case C2 is the sum of the weight spaces for λ and λ, meaning that the elements of β(g) are simultaneously diagonalizable: with the basis {e1 + ie2, e1 − ie2 } of C2 , we can write  β(tX) =

 λ(tX) 0 , t ∈ R. 0 λ(tX)



Given a real n × n matrix X, the above example evidently generalizes to the standard representation of g = RX in gl(n,R). An application of the Jordan normal form shows that there is a weight for each Jordan block. More precisely, let V be a vector space over R and g = RX, where X ∈ gl(V ). Then for each eigenvalue μ of X, the generalized eigenspace    = v ∈ Vc : ∃s ∈ N, (X − μ1)s v = 0 Eμ,X is X-invariant, and Vc is the direct sum of these subspaces. In each nonzero  , every element of g acts by a scalar plus nilpotent endosubspace Eμ,X morphism. This result generalizes in a useful way to any finite-dimensional representation of a nilpotent Lie algebra over R. Proposition 1.2.4 Let g be a nilpotent Lie algebra over R and let β be a representation of g in a finite-dimensional space V over R. For a linear map λ : g → C, define the generalized weight space Eλ =



 Eλ(X),X .

X∈g

Then each Eλ is g-invariant, and Vc is the direct sum of the nonzero Eλ . Proof: The following relation is easy to prove by induction: for any A, B ∈ gl(Vc ),   k

t k (−1) Ak−t BAt . (ad(A)) B = t k

t=0

(1.2.1)

1.2 Representations of a Lie algebra and weights

9

  . To see this, ⊂ E0,A Now observe: if (ad(A))k B = 0 for some k, then, BE0,A suppose that (ad(A))k B = 0 and use (1.2.1) to write

Ak B =

k

ct Ak−t BAt

t=1

with ct ∈ R. It follows by induction that for any positive integer s, there are dt such that Ak+s−1 B =

k+s−1

dt Ak+s−1−t BAt .

t=s

Indeed, if this holds for s; then it holds for s + 1: A

k+s

B = ds A BA + k

s

k+s−1

dt Ak+s−t BAt

t=s+1

= ds

k

ct Ak−t BAt+s +

t=1

k+s−1

dt Ak+s−t BAt .

t=s+1

 , there is s such that As v = 0, therefore Ak+s−1 Bv = 0, Now if v ∈ E0,A   BE0,A ⊂ E0,A , verifying the observation. Fix a basis (X1, . . . ,Xr ) for g. Since g is nilpotent, there is k such that (adX1 )k X2 = 0. Let μ1 be an eigenvalue of β(X1 ): applying the observation to A = β(X1 )−μ1 1, and B = β(X2 ), we get β(X2 )Eμ 1,X1 ⊂ Eμ 1,X1 . Repeating this proof for the restriction of β(X2 ) to Eμ 1,X1 , with μ2 an eigenvalue of this restriction, we get that Eμ 1,X1 ∩ Eμ 2,X2 is invariant under the action of both β(X1 ) and β(X2 ). Continuing in this way, we obtain that the subspace

Eμ 1,...,μr :=

r

Eμ i ,Xi

i=1

is a β(g)-invariant subspace of Vc . Since the generalized eigenspaces for a single endomorphism constitute a direct sum, it follows that at each step,  the nonzero subspaces ti=1 Eμ i ,Xi give a direct sum decomposition of Vc . Finally, let M denote the set of all r-tuples (μ1, . . . ,μr ) where μi is a eigenvalue of β(Xi ), 1 ≤ i ≤ r, and where Eμ 1,...,μr = {0}; then  Vc = Eμ 1,...,μr . (1.2.2) (μ1,...,μr )∈M

Since Eμ 1,...,μr is invariant, there is a basis for this subspace for which the restriction of β has upper-triangular form. By definition, for each i, the

10

Basic theory of solvable Lie algebras and Lie groups

diagonal entries of the matrix of β(Xi ) are thus all equal to μi . For each nonzero space Eμ 1,...,μr , define the form λ on g by λ(Xi ) = μi for 1 ≤ i ≤ r, and let  be the set of forms so defined. Then (1.2.2) is rewritten as   Vc = Eλ(X . 1 ),...,λ(Xr ) λ∈

Now, since, for any X ∈ g, the matrix of the restriction of β(X) to  is upper triangular, with λ(X) on the diagonal, then Eλ ⊂ Eλ(X 1 ),...,λ(Xr )  Eλ(X1 ),...,λ(Xr ) . The reverse inclusion is clear. Thus, for any representation β of a nilpotent Lie algebra g over R, there is a basis for Vc for which all β(X), X ∈ g, simultaneously have an uppertriangular “block” form. Moreover, each block of the matrix for β(X) has a scalar plus nilpotent form, and the scalar is given by λ(X), where λ is a weight for β. This is not true for even the simplest of solvable Lie algebras. Example 1.2.5 Let g = aff(R) have basis A,Y with [A,Y ] = Y (Example 1.1.6). A faithful representation of g in gl(3,R) is given by ⎡ ⎤ ⎡ ⎤ 0 1 0 3 0 0 A = ⎣0 2 0⎦ , Y = ⎣0 0 0⎦ . 0 0 0 0 0 1 Denote by (e1,e2,e3 ) the canonical basis of R3 , and let λk be the linear form on g defined by λk (A) = k and λk (Y ) = 0 (k = 1,2,3). Observe that λ1 and λ3 are weights for this realization, and Eλ 1 = Eλ1 = Ce3 , Eλ 3 = Eλ3 = Ce1 . Since there is no eigenspace for the matrix Y which is a supplementary space to Ce1 + Ce3 , then C3 cannot be a direct sum of generalized eigenspaces.  Proposition 1.2.4 can also be proved as a consequence of the following well-known theorem; we state it here without proof. Theorem 1.2.6 (Engel Theorem) ([16], theorem 1.1.9, corollary 1.1.10) Let g be a Lie algebra, and let β be a representation of g in a finite-dimensional vector space V over R such that β(X) is nilpotent for all X ∈ g. Then there is a basis of Vc for which each β(X) has strictly upper-triangular form. In particular, if ad(X) is nilpotent for all X ∈ g, then g is a nilpotent Lie algebra.

1.3 The Lie theorem and its first consequences It has already been observed that there is no classification of solvable Lie algebras in general, and that even in low dimensions, there is a rich variety

1.3 The Lie theorem and its first consequences

11

of isomorphism classes of solvable Lie algebras. One is therefore motivated to determine structural principles for (finite-dimensional) representations of solvable Lie algebras, not only in order to simplify the presentation of examples, but to be applied in studying representations of special importance. We begin with a lemma. Lemma 1.3.1 Let β be a representation of a Lie algebra g in a finitedimensional vector space V over R or C, and let h be an ideal in g. Suppose that λ is a weight for the restricted representation β|h . Then the corresponding weight space E is g-invariant. Proof: Regarding V as a g-module, we write Xv for β(X)v. Denote U = Vc if V is a module over R, U = V if V is a module over C. Choose any v ∈ E, v = 0, and X ∈ g. Let W be the subspace of U spanned by the elements Xj v, j = 0,1,2, . . .. For any Y ∈ h, [Y,X] belongs to h and Y (Xv) = X(Y v) + [Y,X]v = λ(Y )Xv + λ([Y,X])v.

(1.3.1)

More generally, for each j , there are Yj,1 , Yj,2, . . . ,Yj,j in [h,g] ⊂ h such that Y (Xj v) = λ(Y )Xj v + λ(Yj,1 )Xj −1 v + λ(Yj,2 )Xj −2 v + · · · + λ(Yj,j )v. (1.3.2)   In fact, one finds by induction Yj,i = ji [. . . [Y,X],X], . . . ,X] (X appears here i times). Thus W is h + RX-invariant, and it is easy to check that there is an integer k ≥ 0 such that {v,Xv, . . . ,X k v} is a basis for W . Then (1.3.2) shows that for any Y ∈ h, the trace of β(Y )|W is (k + 1)λ(Y ). Now the action of [Y,X] on any h-invariant subspace must have trace 0. Hence 0 = trace(β([Y,X])|W ) = (k + 1)λ([Y,X]) so λ([Y,X]) = 0. The relation (1.3.1) now shows that Xv ∈ E. Recall that a solvable Lie algebra always contains an ideal of codimension one. This simple fact combined with Lemma 1.3.1 results in the following. Theorem 1.3.2 (Lie Theorem) Let g be a solvable Lie algebra over R, and let β be a representation of g in a finite-dimensional vector space U over C. Then the g-module U has a weight. Moreover, if U = Vc is the complexification of a module over R and λ is a weight, then λ is real-valued if and only if its weight space is real.

12

Basic theory of solvable Lie algebras and Lie groups

Proof: We proceed by induction on the dimension of g. Suppose that dim g = 1,with g = RX. Let γ be an eigenvalue of X, and define the weight λ in the obvious way by λ(tX) = tγ . Thus Eλ = {w ∈ U : Xw = γ w} is the corresponding eigenspace for the eigenvalue γ . If λ is a weight with λ(X) = γ , then Eλ is the eigenspace for γ so E λ = Eλ and λ is real-valued if and only if γ ∈ R, if and only if E λ = Eλ , that is, Eλ is real. Suppose now that dim g > 1 and the theorem holds for all solvable Lie algebras of dimension smaller than dim g. Let h be an ideal of codimension one in g. Then h is solvable and the induction hypothesis applies to the hmodule U . Hence we have a weight λ0 for this module with weight space F , and so that if λ0 is real, then F = F . Let X ∈ g such that g = RX + h. By Lemma 1.3.1, β(X)F ⊂ F . Let γ be an eigenvalue for β(X)|F and let E = {w ∈ F : β(X)w = γ w}. Define λ : g → C by λ(Y + tX) = λ0 (Y ) + tγ , for Y ∈ h, t ∈ R. Then for any w ∈ E and Y + tX ∈ g, we have (Y + tX)w = Y w + tXw = λ0 (Y )w + tγ w = λ(Y + tX)w. Thus, E ⊂ Eλ . Conversely, for w ∈ Eλ , (Y + tX)w = λ(Y + tX)w. Taking first t = 0 then Y = 0 in this relation shows that w ∈ E. So E = Eλ . Now, if U = Vc , then λ is real-valued if and only if λ0 is real-valued and γ is real. By induction, λ0 is real-valued if and only if F is real, and so λ0 is real-valued and γ is real if and only if Eλ is real. Let us say that an ascending flag for a vector space U of dimension n is a sequence of subspaces {0} = U0 ⊂ U1 ⊂ · · · ⊂ Un = U where dim Ui = i, 1 ≤ i ≤ n. Let U be a finite-dimensional g-module, and (Ui ) an ascending flag of subspaces, we say that (Ui ) is an ascending flag of modules if each Ui is a submodule of U . The following is almost immediate. Corollary 1.3.3 Let g be a real solvable Lie algebra and let U be a finite-dimensional complex g-module. Then U admits an ascending flag of g-modules. Proof: We proceed by induction on n = dim U , the result being clear if n = 1. Suppose the result holds for any representation in a complex vector space of dimension less than n. By Lie’s theorem, U has a weight λ. Fix a nonzero eigenvector v ∈ Eλ and put U1 = Cv, U0 = 0. Put U˜ = U/U1 and let π : U → U˜ be the canonical quotient map. By induction, U˜ has a sequence of

1.3 The Lie theorem and its first consequences

13

submodules U˜ = U˜ n−1 ⊃ U˜ n−2 ⊃ · · · ⊃ U˜ 0 = 0, with dim(U˜ i /U˜ i−1 ) = 1, 1 ≤ i ≤ n − 1. Put U0 = {0} and Ui = π −1 (U˜ i−1 ), 1 ≤ i ≤ n; then (Ui ) satisfies the required conditions. Given a representation β of g in a complex vector space U , an ascending flag of g-modules will also be called an ascending flag for β. Let (Ui ) be an ascending flag of g-modules. For 1 ≤ i ≤ n the quotient module U/Ui−1 contains Ui /Ui−1 as a submodule, and the action of g on Ui /Ui−1 is given by a complex linear form λi : given f ∈ Ui \ Ui−1 and a ∈ C, X(af + Ui−1 ) = λi (X)af + Ui−1 . Note that λi is a weight for the quotient module U/Ui−1 . The sequence (λi )ni=1 of forms so obtained is called the weight sequence for the flag (Ui ). An ordered basis (f1,f2, . . . ,fn ) is said to be adapted to the flag (Ui ) if fi ∈ Ui \ Ui−1 , 1 ≤ i ≤ n. Note that the matrix of each β(X) with respect to an adapted basis (f1,f2, . . . ,fn ) is upper triangular with diagonal entries given by the values λi (X). When β is the standard representation of a solvable Lie subalgebra of gl(n,R) we have the following. Corollary 1.3.4 Let g be a solvable Lie subalgebra of gl(n,R) or gl(n,C). Then there is s ∈ GL(n,C) such that sgs −1 ⊂ t(n,C). Let β be a representation of g in a vector space U , and consider the dual representation β ∗ in U ∗ . For each 1 ≤ i ≤ n, the subspace Ui⊥ = { ∈ U ∗ : Ui ⊂ ker } is β ∗ (g)-invariant and U ∗ = U0⊥ ⊃ U1⊥ ⊃ · · · ⊃ Un⊥ = {0}. In general, a sequence of subspaces U = U0 ⊃ U1 ⊃ · · · ⊃ Un = {0} where dim Ui = n − i, 1 ≤ i ≤ n, will be called a descending flag. As the dual representation is of particular interest in harmonic analysis, descending flags of submodules will often arise. For the remainder of this section we assume that g is a solvable Lie algebra over R, and β is a representation of g acting in a vector space V over R (and hence in U = Vc also). A complex-valued R-linear form λ is called a generalized weight for β if there is an invariant subspace W of U = Vc such that λ is a weight for the quotient module U/W . Evidently, the set of generalized weights depends only upon the equivalence class of β, and each form in the weight sequence of a flag is a generalized weight. The next result shows that more is true.

14

Basic theory of solvable Lie algebras and Lie groups

Proposition 1.3.5 Let (Ui )0≤i≤n be an ascending flag of g-modules in U = Vc with weight sequence (λi )1≤i≤n . Then {λ1, . . . ,λn } is exactly the set of all generalized weights for β. Proof: Let λ be any generalized weight and W a g-invariant subspace of U = Vc such that λ is a weight for the quotient module U/W . Then we have v ∈ U \ W such that for any X ∈ g, Xv = λ(X)v + wX

(1.3.3)

where wX ∈ W . Let i be the smallest index for which v ∈ Ui + W , and choose fi ∈ Ui \ Ui−1 , so that v = fi + ui + w with ui ∈ Ui−1 , w ∈ W . Substituting into (1.3.3) we get Xv = λ(X)fi + λ(X)ui + λ(X)w + wX . Now Xfi = λi (X)fi + ui,X

(1.3.4)

where ui,X ∈ Ui−1 . Computing Xv using (1.3.4), we get Xv = Xfi + Xui + Xw = λi (X)fi + ui,X + Xui + Xw. / Ui−1 + W , so in the quotient U/(Ui−1 + W ), Since v ∈ / Ui−1 + W , fi ∈ fi + Ui−1 + W = Ui−1 + W , while (λ(X) − λi (X))(fi + Ui−1 + W ) = Ui−1 + W for every X ∈ g. Thus λ = λi . Next we turn to the description of an irreducible finite-dimensional representation of a solvable Lie algebra. Proposition 1.3.6 Let g be a real solvable Lie algebra and V a finitedimensional irreducible g-module. Then one of the following holds. (1) V = Re for some nonzero element e and β has a single real-valued weight, so that Xe = λ(X)e holds for all X ∈ g. (2) V = spanR {e1,e2 } is two dimensional and Vc has the two distinct weights λ and λ, where X(e1 + ie2 ) = λ(X)(e1 + ie2 ), X ∈ g.

1.3 The Lie theorem and its first consequences

15

Proof: By Theorem 1.3.2 Vc has a weight λ and Eλ ∩V is g-invariant. Suppose that λ is real. Then Eλ is real and hence Eλ ∩ V = {0}. Since Eλ ∩ V is g-invariant, then by irreducibility of V we have Eλ ∩ V = V and dimR V = 1. Thus (i) holds. Suppose that λ is not real-valued. Let f be a nonzero element of Eλ . Since V is g-invariant and λ is not real, then f is not almost real. Indeed, there is X ∈ g such that λ(X) ∈ / R. If f were in CV , there would be a real eigenvector e = zf for X, but then: Xe = Xe = Xe = λ(X) e = λ(X) e = λ(X)e. So write f = e1 + ie2 where e1 , e2 ∈ V are linearly independent. Then spanR {e1,e2 } is a nontrivial invariant subspace of V , hence equal to V . In this case (2) holds. If V = spanR {e1,e2 } is an irreducible g-module of dimension two then U1 = C(e1 + ie2 ) is an irreducible submodule of U = Vc which is not real, that is, U 1 = U1 . If we then put U2 = U = Vc and U0 = {0}, then (Ui ) is a flag of modules satisfying U2 = U1 + U 1 . The basis (f1,f2 ) for U , where f1 = e1 + ie2 and f2 = e1 − ie2 , is adapted to the flag. Note also that V is isomorphic with C via the map z → Re(z(e1 + ie2 )). Our next task is to make a similar construction for any real g-module V . In particular, it will be useful to have an ordered basis (fi ) for U = Vc where each fi is either an element of V , or f i is an adjacent basis element. Proposition 1.3.7 Let g be a real solvable Lie algebra and V a n-dimensional g-module over R, with U = Vc . Then there is an ascending flag (Ui )0≤i≤n of modules such that if U i = Ui , then Ui+1 = Ui + U i and λi+1 = λi . Moreover, there is an ordered basis (f1, . . . ,fn ) for U , adapted to the flag (Ui ), such that if Ui−1 and Ui are real then fi ∈ V , and if U i = Ui , then fi+1 = f i . Proof: The proof is by induction on the dimension of V ; by Proposition 1.3.6, the base case is proved. By Theorem 1.3.2 (Lie Theorem), U admits a weight λ with weight space Eλ ⊂ U . Choose f ∈ Eλ , f = 0. Case (1): suppose that f is almost real. Then we may choose f ∈ V . Put λ1 = λ, U0 = 0, U1 = Cf . Then U 1 = U1 and U1 is the complexification of I = Rf , so U/U1 is the complexification of V /I . We can apply induction to the quotient representation in V /I to obtain an ascending flag (Wi )0≤i≤n−1 for U/U1 , with associated generalized weights λi , 1 ≤ i ≤ n − 1 satisfying the conditions of the proposition. Now define Uj = π −1 (Wj −1 ), λj = λj −1 , 2 ≤ j ≤ n.

16

Basic theory of solvable Lie algebras and Lie groups

Fix 1 < j ≤ n, suppose that Uj −1 is real, and suppose that a basis for Uj is chosen satisfying the conditions of the proposition. If Uj is real, then Uj ∩ V = Uj −1 ∩ V and we choose fj ∈ Uj ∩ V \ Uj −1 . If Uj is not real, then Wj −1 is not real, and hence by induction Wj = Wj −1 + W j −1 , hence Uj = / Uj . Put fj +1 = f j , and we Uj −1 + U j −1 . Pick fj ∈ Uj \ Uj −1 ; then f j ∈ have fj +1 ∈ Uj +1 \ Uj . This proves the proposition in Case (1). Case (2): suppose that f is not almost real: f and f are linearly independent. Write f = e +ie and put U1 = Cf and λ1 = λ, and put J = U2 = Cf +Cf , λ2 = λ, and f1 = f and f2 = f . Then J = J and J is the complexification of I = Re + Re , so U/J is the complexification of V /I . We can apply induction to the quotient representation in V /I to obtain an ascending flag (Wi )0≤i≤n−2 for U/J , with associated generalized weights λi , 1 ≤ i ≤ n − 2 satisfying the conditions of the proposition. Now define Uj = π −1 (Wj −2 ), λj = λj −2 , 3 ≤ j ≤ n. In a similar way as Case (1) we choose the elements f3,f4, . . . ,fn , and (Ui )0≤i≤n , with associated generalized weights (λi )1≤i≤n , and basis (fi )1≤i≤n satisfy the conditions of the proposition. Definition 1.3.8 Let g be a real solvable Lie algebra and V a finite-dimensional g-module over R. An ascending flag (Ui ) of submodules of Vc satisfying the conditions of Proposition 1.3.7 will be called a good ascending flag for Vc . Similarly, an ordered basis (f1, . . . ,fn ) for Vc satisfying the conditions (a) Ui = span{f1, . . . ,fi } is a submodule of Vc , 1 ≤ i ≤ n,  U i implies fi+1 = f i , and (b) Ui = (c) Ui = U i and Ui−1 = U i−1 implies fi ∈ V , will be called a good ascending basis for Vc . Observe that if (f1, . . . ,fn ) is a good ascending basis for Vc , then Ui = span{f1, . . . ,fi },1 ≤ i ≤ n defines good ascending flag for Vc . Observe that if the weight λ in the proof of Proposition 1.3.7 is real-valued, then we could choose f ∈ Eλ so that f ∈ V . The following is an immediate consequence of this observation. Corollary 1.3.9 Assume that every generalized weight is real-valued. Then there is a good ascending basis (fi ) for U = Vc such that fi ∈ V for all i. Hence V admits a flag of submodules {0} = V0 ⊂ V1 ⊂ · · · ⊂ Vn = V with dim Vi = i,1 ≤ i ≤ n. Note that if all generalized weights are real, then one can still choose a complex basis.

1.3 The Lie theorem and its first consequences

17

Example 1.3.10 Let V = R2 , and let g = RA where   1 0 A= 0 1 2 so that g has only one generalized weight defined by λ(A) = 1. Then V c =  C 1 and we could the good ascending basis (f1,f2 ) with f1 = i and 

1choose .  f2 = f 1 = −i

Given a submodule of V it may be useful to choose a good ascending flag which includes its complexification. Corollary 1.3.11 Let β be a representation of g in a vector space V over R, and let W be a proper, nonzero submodule. Then there is a good ascending flag (Ui )0≤i≤n for Vc for which Ud = Wc for some 1 ≤ d < n. Proof: Let d = dim W and π : U → U/Wc the quotient map. Apply Proposition 1.3.7 to the action of g in W to get a flag Wc = Ud ⊃ · · · ⊃  U0 = {0}, and also to the quotient action in V /W to get U/Wc = Un−d   −1 ⊃ · · · ⊃ U0 = {0}. Define Uj = π (Uj −d ), d < j ≤ n. Then (Ui )0≤i≤n satisfies the desired conditions. Now given a representation β of g, fix a good ascending basis (fi ) for the module Vc with Ui = span{f1, . . . ,fi },1 ≤ i ≤ n, and with weight sequence (λi ). As observed in the discussion following Corollary 1.3.4, the dual representation of g in V ∗ has the descending dual flag U ∗ = U0⊥ ⊃ U1⊥ ⊃ · · · ⊃ Un⊥ = {0}, with Ui⊥ = Ui⊥ if and only if U i = Ui , and Ui⊥ = Ui⊥ implies Ui⊥ + Ui⊥ = ⊥ . Observe that the weight sequence for (U ⊥ ) is (−λ ). Ui−1 i i It is easy to check that the dual flag and dual basis (gi ) defined by gi = fi∗ satisfy the following definition. Definition 1.3.12 Let β be a representation of g in a finite-dimensional vector space V over R, and let Vc = U0 ⊃ U1 ⊃ · · · ⊃ Un = {0}, be a descending flag of submodules. If (Ui ) satisfies U i = Ui implies Ui + U i = Ui−1 , 0 ≤ i ≤ n − 1, then (Ui ) is called a good descending flag for Vc . An ordered basis (gi ) for Vc satisfying (a) Ui = span{gi+1, . . . ,gn } is a submodule of Vc , 0 ≤ i ≤ n − 1,  U i implies gi = g i+1 , and (b) Ui =

18

Basic theory of solvable Lie algebras and Lie groups

(c) Ui−1 = U i−1 and Ui = U i implies gi ∈ V , will be called a good descending basis for Vc . The Lie algebra introduced below is an example that will be useful in subsequent chapters. Example 1.3.13 Let g = spanR {A,X,Y,Z} where [A,X] = Y , [A,Y ] = −X, and [X,Y ] = Z; g is an example of what is often called a diamond Lie algebra, and we adopt that name here. Note that the subalgebra n = spanR {X,Y,Z} is a Heisenberg Lie algebra (Example 1.1.5); observe however that g itself is not nilpotent since [g,n] = n. A faithful representation of g in gl(6,R) is given by ⎡ ⎤ ⎡ ⎤ 0 1 0 0 0 0 0 0 0 −1 0 0 ⎢−1 0 0 0 0 0⎥ ⎢0 0 1 0 0 0⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ 0 1⎥ ⎢ 0 0 0 0 0 0⎥ ⎢0 0 0 0 A=⎢ ⎥, X = ⎢ ⎥, ⎢ 0 0 0 0 0 0⎥ ⎢0 0 0 0 −1 0⎥ ⎢ ⎥ ⎢ ⎥ ⎣ 0 0 0 0 0 1⎦ ⎣0 0 0 0 0 0⎦ 0 0 0 0 −1 0 0 0 0 0 0 0 ⎡ ⎤ ⎡ ⎤ 0 0 1 0 0 0 0 0 0 0 0 −2 ⎢0 0 0 1 0 0⎥ ⎢0 0 0 0 2 0 ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢0 0 0 0 1 0⎥ ⎢0 0 0 0 0 0 ⎥ Y =⎢ Z=⎢ ⎥, ⎥. ⎢0 0 0 0 0 1⎥ ⎢0 0 0 0 0 0 ⎥ ⎢ ⎥ ⎢ ⎥ ⎣0 0 0 0 0 0⎦ ⎣0 0 0 0 0 0 ⎦ 0 0 0 0 0 0 0 0 0 0 0 0 Let (e1, . . . ,e6 ) be the canonical ordered basis for V = R6 , and define an ordered basis (f1, . . . ,f6 ) for Vc = C6 by f1 = e1 + ie2, f2 = f 1, f3 = e3, f4 = e4, f5 = e5 + ie6, f6 = f 5 . With Ui = span{fj : 1 ≤ j ≤ i} for i = 1,2, . . . ,6, and U0 = {0}, it is easy to check that (Ui )0≤i≤6 is a good ascending flag for Vc and hence (fi ) is a good ascending basis. For the associated sequence (λi )1≤i≤6 of generalized weights, we have λ1 = λ5 = iA∗ , λ3 = λ4 = 0, so the set of generalized weights for V is {λ1, λ1, 0}. Each generalized weight λ corresponds to a generalized weight space Eλ = {w ∈ Vc : for all B ∈ g,(B − λ(B)1)s w = 0 for some s}. In this example the generalized weight spaces are Eλ 1 = span{f1,f5 }, Eλ = span{f2,f6 }, E0 = span{f3,f4 }. 1

1.4 Adjoint and coadjoint representations

19

Since Yf5 = e3 + ie4 , Yf6 = e3 − ie4 , and Y e3 = e1 , then none of these are g-invariant. Thus, this gives another example of a solvable Lie algebra module for which the action cannot be presented in block diagonal form (as in Proposition 1.2.4 for nilpotent Lie algebras).  We say that a complex number z is purely imaginary if z ∈ iR \ {0}. Definition 1.3.14 Let g be a real Lie algebra and let β be a representation of g in a vector space V over R. Then β is said to be of of nilpotent type if β(X) is nilpotent for every X in g. We say that β is of completely solvable type if for every X in g, all eigenvalues of β(X) are real, and that β is of exponential type if for every X ∈ g, no eigenvalue of β(X) is purely imaginary. We may also say that the g-module V is of nilpotent type (resp. completely solvable type, exponential type). From Proposition 1.3.5 it is clear that if g is solvable, then the set of eigenvalues of all elements β(X), X ∈ g is precisely the set of values of the generalized weights for β. Thus β is of nilpotent type if and only if its only generalized weight is the zero form. Similarly, β is of completely solvable type if and only if each of its generalized weights is real-valued. Observe that a representation is of nilpotent (resp. completely solvable, exponential) type if and only if its dual representation is of nilpotent (resp. completely solvable, exponential) type. Recall the the proof of Proposition 1.3.7 consists of two cases, and note that if V is of completely solvable type, then only Case 1 occurs. This means that it is possible to choose a good flag (Ui ) so that each Ui is real, and a good basis (fi ) so that each fi belongs to V . It follows that the matrix of each β(X), X ∈ g with respect to the basis (fi ) is upper triangular with real entries. Finally, β is of exponential type if and only if the imaginary part of λi is a multiple of its real part, or equivalently, there is ai ∈ R such that λi = Re(λi )(1 + iai ). The faithful representation of the diamond Lie algebra given in Example 1.3.13 is not of exponential type.

1.4 Adjoint and coadjoint representations We now apply the results of Section 1.3 to the important cases of the adjoint and coadjoint representations. Since a submodule for the adjoint representation is just an ideal, the following is immediate. Corollary 1.4.1 Let g be a real solvable Lie algebra, and put s = gc . Then there is a sequence of ideals

20

Basic theory of solvable Lie algebras and Lie groups {0} = s0 ⊂ s1 ⊂ s2 ⊂ · · · ⊂ sn = s

with the property that if sj = sj then sj +1 = sj +sj , 1 ≤ j ≤ n−1. Moreover, there is a basis (Z1,Z2, . . . ,Zn ) for s with Zj ∈ sj for all 1 ≤ j ≤ n, and such that if sj = sj , then Zj +1 = Z j , and if sj = sj and sj −1 = sj −1 hold, then Zj ∈ g. A sequence of ideals in s satisfying the conditions of Corollary 1.4.1 will be called a good Jordan–Hölder sequence, and a basis as above will be called a good Jordan–Hölder basis for s. Given a good Jordan–Hölder sequence (sj ) for gc , the sequence (s⊥ j ) is a good descending flag for the coadjoint action ∗ ∗ on s = (g )c , with good descending basis (Zj∗ ). Observe that for each j , the ∗ canonical quotient map πj : s∗ → s∗ /s⊥ j  (sj ) is just the restriction map  → |sj . A generalized weight for the adjoint representation will be called a root of the Lie algebra g, and given a good Jordan–Hölder sequence, the associated sequence (λj )1≤j ≤n of generalized weights for the adjoint representation is also called the associated root sequence. Definition 1.4.2 Let g be a Lie algebra over R. Then g is called an exponential Lie algebra if its adjoint representation is of exponential type, and a completely solvable Lie algebra if its adjoint representation is of completely solvable type. Thus g is exponential if and only if for every X in g, ad(X) has no purely imaginary eigenvalue, it is completely solvable if and only if the eigenvalues of every ad(X) are real. In a later section (Corollary 1.8.2), we shall see that if g is exponential, then it is solvable. Therefore a real Lie algebra is exponential (resp. completely solvable) if and only if it is solvable and its roots are never assuming a purely imaginary value (resp. are realvalued). Moreover any subalgebra, any quotient algebra of an exponential (resp. a completely solvable) Lie algebra is an exponential (resp. a completely solvable) Lie algebra. Example 1.4.3 Let g be the Heisenberg Lie algebra (see Example 1.1.5), with basis {X,Y,Z} and nonvanishing bracket [X,Y ] = Z. Then the basis Z1 = Z, Z2 = Y , Z3 = X is a good Jordan–Hölder basis, and the only root is the zero form. Likewise, a good descending basis for the coadjoint representation on g∗ is e1 = Z ∗ , e2 = Y ∗ , e3 = X∗ . Now let g be the diamond Lie algebra (see Example 1.3.13), with basis {A,X,Y,Z} and nonvanishing brackets [A,X] = Y , [A,Y ] = −X, [X,Y ] = Z. The basis of s = gc given by Z1 = Z, Z2 = X + iY , Z3 = X − iY , Z4 = A is a good Jordan–Hölder basis for s. The corresponding root sequence

1.4 Adjoint and coadjoint representations

21

(λj )1≤j ≤4 is given by λ1 = λ4 = 0 and λ3 = λ2 , where λ2 is the linear form for which λ2 (A) = −i, λ2 (X) = λ2 (Y ) = λ2 (Z) = 0. A good descending basis for the coadjoint representation on s∗ is f1 = Z ∗,f2 = (X + iY )∗, f3 = (X − iY )∗,f4 = A∗ . Observe that with e2 = Y ∗ , e3 = X∗ , one has  f2 = (e2 − ie3 )/2, f3 = (e2 + ie3 )/2. An important structural consequence of the above is that a real solvable Lie algebra has a unique maximal nilpotent ideal, called the nilradical of g. Corollary 1.4.4 Let g be a solvable Lie algebra over R and let R be its set of roots. Then nil(g) = ∩λ∈R ker λ is a nilpotent ideal in g and every nilpotent ideal in g is included in nil(g). Proof: Choose a good Jordan–Hölder basis (Zj )1≤j ≤n for s = gc . With respect to this basis, the matrix for each ad(X), X ∈ g is upper triangular with diagonal entries λj (X), and if X ∈ nil(g), then the matrix of ad(X) is strictly upper triangular. Now for each positive integer k, (nil(g))(k) is the span of all elements of the form [X1,[X2, . . . [Xk−1,Xk ] . . .] = ad(X1 ) ◦ ad(X2 ) ◦ · · · ◦ ad(Xk−1 )Xk , with Xj ∈ nil(g) for any j . Thus (nil(g))(n+1) = 0, and nil(g) is nilpotent. It follows from Lemma 1.2.2 that [g,g] ⊂ nil(g), and so nil(g) is an ideal. Now let n be an ideal in g and suppose that n is a nilpotent Lie algebra. Given X ∈ n, ad(X) maps g into n and the definition of nilpotent Lie algebra implies that ad(X)|n is nilpotent. Hence ad(X) is nilpotent and its only eigenvalue is 0. It follows that n ⊆ nil(g). Since a subalgebra of a nilpotent Lie algebra is a nilpotent Lie algebra, the following is immediate. Corollary 1.4.5 Let g be a real solvable Lie algebra. Then [g,g] is a nilpotent Lie algebra. Conversely, observe that, if g is a Lie algebra such that [g,g] is nilpotent, then g(1) = [g,g] = (g (1) )(0) , and by induction g(k) ⊂ (g(1) )(k−1) , for any k, thus g is solvable. Example 1.4.6 Recall the Lie algebra t(n,R) of upper-triangular n × n real matrices. Since [t(n,R),t(n,R)] is included in the nilpotent Lie algebra of strictly upper-triangular matrices, then t(n,R) is a solvable Lie algebra. The generalized weights of the standard representation of t(n,R) are realvalued: they are the maps: λi : X → xii ,

22

Basic theory of solvable Lie algebras and Lie groups

associating to the matrix X its i th entry on the main diagonal, 1 ≤ i ≤ n. Now consider the subspace:   tk = Y = [yij ] : yij = 0 if j − i < k . If Y ∈ tk , and X ∈ t(n,R), the matrix Z = [X,Y ] has entries zij =

j −k j

(xir yrj − yir xrj ) = xir yrj − yir xrj . r

r=i

r=i+k

We see that zij = 0 if j − i < k and if j − i = k, then zij = xii yij − yij xjj = (λi (X) − λj (X))yij . This proves that the tk are ideals of t(n,R) and for each matrix Ei,i+k , we have [X,Ei,i+k ] = (λi − λi+k )(X)Ei,i+k modulo tk+1 . This shows that the adjoint action in tk /tk+1 is diagonal, with weights λi − λi+k . In particular the roots of the Lie algebra t(n,R) are real: they are the λi − λj , for j ≥ i. Thus the Lie algebra t(n,R) is completely solvable.  The following immediate consequence of Corollary 1.3.11 is frequently useful when applied to the ideals nil(g) or [g,g]. Corollary 1.4.7 Let g be a real solvable Lie algebra and let n be an ideal in g. (1) There is a good Jordan–Hölder sequence (sj )0≤j ≤n such that sp = nc for some p. (2) Suppose that n contains [g,g]. Then there is a good Jordan–Hölder sequence (sj )0≤j ≤n with root sequence (λj )nj=1 , such that (a) sp = nc for some p, (b) sj is real for all j > p, and (c) λj ≡ 0 for all j > p. Returning to familiar examples: (1) If g is a commutative Lie algebra, then nil(g) = g while [g,g] = {0}. (2) Let g be the Heisenberg Lie algebra of Example 1.1.5. Then nil(g) = g while [g,g] = RZ. (3) Let g be the diamond Lie algebra of Example 1.3.13. Then nil(g) = [g,g] = spanR {X,Y,Z} is the Heisenberg Lie algebra. Lemma 1.4.8 Let g be a solvable Lie algebra over R and let (sj )0≤j ≤n be a good Jordan–Hölder sequence of ideals in gc . If sj = sj , then [sj +1,sj +1 ] ⊂ sj −1 .

1.5 Ado theorem for a solvable Lie algebra

23

Proof: Fix a corresponding good Jordan–Hölder basis (Zj )j for s and suppose / g. Since sj is an ideal we have [Zj ,Z j ] = that sj is not real, so that Zj ∈ aZj + W where a ∈ C and W ∈ sj −1 . Denote X = Re(Zj ), Y = Im(Zj ), and observe that (X,Y ) is a basis of the two-dimensional vector space sj +1 /sj −1 . Now if a = x + iy, −2i[X,Y ] = [Zj ,Z j ] = aZj + W = xX − yY + i(yX + xY ) + W ∈ ig. This implies xX − yY ∈ sj −1 , thus x = y = 0, and [Zj ,Z j ] ∈ sj −1 .

1.5 Ado theorem for a solvable Lie algebra We frequently present examples of solvable Lie algebras g by a realization of g. The Ado theorem shows that any solvable Lie algebra can be presented by one of its realization. For completeness, we present here a proof of the Ado theorem for solvable Lie algebras. More precisely, we show: Theorem 1.5.1 (Ado Theorem for the real solvable case) Let g be a solvable Lie algebra over R, with nilradical r. Then there exists a faithful finitedimensional representation of g such that every element of r acts by a nilpotent linear map. The method of proof of Theorem 1.5.1 is by induction on the dimension of g. The proof is presented with the illustration of a simple example along the way. Proof: If g = RZ, then the map  zZ →

0 0

 z 0

gives a realization of g by nilpotent matrices. Suppose that g is a solvable Lie algebra over R with dimension d > 1. If g is nilpotent so that r = g, then choose any d − 1-dimensional ideal a in g. If g is not nilpotent, it contains a d − 1-dimensional ideal a containing r. In either case, r ∩ a is a nilpotent ideal, which by Corollary 1.4.4 is included in the nilradical ra of a. Now a is solvable, so by induction we have a faithful representation ρ : a → gl(W ) such that ρ(Z) is nilpotent for all Z ∈ r ∩ a. Example 1.5.2 Suppose g has a basis (A,X,Y,Z), with nonvanishing bracket: [X,Y ] = Z,

[A,Y ] = Y + Z,

and [A,Z] = Z.

24

Basic theory of solvable Lie algebras and Lie groups

Choose then a = r = span(X,Y,Z), and the faithful representation ρ of r is given by Example 1.1.5: ⎡ ⎤ 0 x z ρ(xX + yY + zZ) = ⎣0 0 y ⎦ , x,y,z ∈ R, 0 0 0 acting on W = R3 or C3 .



Let A be an element in g\a, so that g = a+RA, and define a representation β of g acting in the (infinite-dimensional) vector space C ∞ (R,W ) of all smooth W -valued functions on R by (β(A)F ) (t) = F  (t) and (β(Z)F ) (t) = ρ(etad(A) Z)F (t), Z ∈ a. To see that β is a representation, let Z, Z  ∈ a. We claim that for each t ∈ R, etad(A) [Z,Z  ] = [etad(A) Z,etad(A) Z  ]. Define f : R → a by f (t) = e−tad(A) [etad(A) Z,etad(A) Z  ], t ∈ R. By Jacobi identity, the derivative of f is vanishing:   f  (t) = e−tad(A) [etad(A) Z,etad(A) Z  ],A     + [A,etad(A) Z],etad(A) Z  + [etad(A) Z ,A],etad(A) Z = 0. Therefore f is constant and f (t) = f (0) = [Z,Z  ] holds for all t, and   etad(A) Z,etad(A) Z  = etad(A) [Z,Z  ] proving the claim. This means that for F ∈ C ∞ (R,W ),     β([Z,Z  ])F (t) = ρ(etad(A) [Z,Z  ])F (t) = ρ etad(A) Z,etad(A) Z  F (t)   

  = ρ(etad(A) Z),ρ(etad(A) Z  ) F (t) = β(Z),β(Z  ) F (t). It remains to show that [β(A),β(Z)] = β([A,Z]); but this is easy:  d  tad(A) Z)F (t) − ρ(etad(A) Z)F  (t) ρ(e ([β(A),β(Z)]F ) (t) = dt = ρ(etad(A) [A,Z])F (t) = (β([A,Z])F ) (t) Thus β is a representation of g.

1.5 Ado theorem for a solvable Lie algebra

25

Example 1.5.3 Returning to Example 1.5.2, we find that etad(A)  X = X, w1 (t) etad(A) Y = et Y + tet Z, and etad(A) Z = et Z. Writing F (t) = w2 (t) , then w3 (t)    (β(A)F )(t) =

w1 (t) w2 (t) w3 (t)

and:

⎤ ⎤⎡ w1 (t) 0 x et (z + ty) (β(xX + yY + zZ)F ) (t) = ⎣0 0 et y ⎦ ⎣w2 (t)⎦ 0 0 0 w3 (t) ⎡ ⎤ xw2 (t) + et (z + ty)w3 (t) ⎦. =⎣ et yw3 (t) 0 ⎡

 Observe that the subspace U of functions F (t) =

w1 (t) w2 (t) w3 (t)

 , with w3 a constant

w1 (t) = + c) (a, b, c real numbers) is a finitefunction, w2 (t) = ∞  dimensional invariant subspace of C (R,W ). aet ,

et (bt

Returning to the proof, we now apply Theorem 1.3.7 to choose a good ascending basis (f1, . . . ,fn ) for the a-module W , with generalized weight sequence (λk ). With respect to the basis (fi ), write ⎡

⎤ w1 (t) ⎢ ⎥ F (t) = ⎣ ... ⎦

and

ρ(Z) = [ak,l (Z)]1≤k,l≤n .

wn (t) Since (fi ) is a good ascending basis, ak,l (Z) = 0 for k > l. Moreover, ak,k (Z) = λk (Z) = 0 for Z ∈ (ra )c . Next put ⎡

a1,1 (etad(A) Z) ⎢ .. (β(Z)F )(t) = ⎣ .

···

an,1 (etad(A) Z)

···

⎤⎡ ⎤ ⎡ ⎤ a1,1 (etad(A) Z) u1 (Z,t) w1 (t) ⎥⎢ .. ⎥ ⎢ .. ⎥ .. ⎦⎣ . ⎦ = ⎣ . ⎦. . an,n (etad(A) Z)

wn (t)

un (Z,t)

Similarly, use Proposition 1.2.4 (or the Jordan form of ad(A)) to decompose   the representation t → ad(tA) of R in ac , writing ac = μ∈M aμ , with ad(A)|aμ = μ1aμ + nμ , with nμ a nilpotent map. Put also M \ {0} = {μ1, . . . ,μm }. Especially if μ = 0, aμ = ad(A)aμ ⊂ rc ∩ ac ⊂ (ra )c,

26

Basic theory of solvable Lie algebras and Lie groups

thus ak,k (Z) = ak,k (etad(A) Z) = 0 for each Z ∈ aμ . Therefore, if Z ∈ aμ ,

uk (Z,t) = ak,l (etad(A) Z)wl (t) = etμ ak,l (etnμ Z)wl (t). l≥k

l>k

Recall that t → etnμ Z is a polynomial, with degree at most d − 2. While if μ = 0, then ad(A)a0 = n0 (a0 ) ⊂ (ra )c , thus, if Z ∈ a0 ,

uk (Z,t) = ak,l (etad(A) Z)wl (t) = ak,k (Z)wk (t) + ak,l (etn0 Z)wl (t). k≤l

kk degs Ql (s,t) ≤ n − k, and degt Rk (Z,s,t) ≤ max{(n − l)(d − 1) + d − 2} ≤ (n − k)(d − 1). l>k

Thus U is a finite-dimensional invariant subspace of C ∞ (R,W ). Example 1.5.5 Coming back to our example we get in this case ⎧ ⎡ ⎤⎫ a1 (t)e2t + b1 (t)et + c1 (t) ⎬ ⎨ ⎦ , U = F (t) = ⎣ b2 (t)et + c2 (t) ⎩ ⎭ c3 with a1 (t), b1 (t), c1 (t) polynomials, of degree at most 6, a2 (t), b2 (t) polynomial with degree 3, c3 constant. The space U is invariant since: (β(aA + xX + yY + zZ)F )(t) = aF  (t) ⎡ t ⎤ e (xb2 (t) + ytc3 + zc3 ) + xd2 (t) ⎦. +⎣ yet c3 0  To see that β is faithful on U , suppose that β(aA + Z) = 0 for some a ∈ R, Z ∈ a, then, for any F ∈ U , any t, aF  (t) + (ρ(etad(A) Z)F )(t) = 0. Taking F (t) = w constant, this gives ρ(Z)w = 0 for each w, thus Z = 0. Note now that U contains nonconstant functions, since d > 1, so β(A) = 0. To see that the restriction of β(r) to U is nilpotent, recall that by choice of the codimension one ideal a, either r ⊂ a, or g itself is nilpotent. Let Z ∈ r and let λ be an eigenvalue of β(Z) with eigenvector F . Suppose that Z ∈ a and write the analytic function F (t) as F (t) = t a G(t) where G(0) = 0. Then, t a λG(t) = λF (t) = (β(Z)F )(t) = ρ(etad(A) Z)F (t) = t a ρ(etad(A) Z)G(t) = t a (β(Z)G)(t) so G(t) is an eigenvector with eigenvalue λ; but then λG(0) = (β(Z)G)(0) = ρ(Z)G(0) shows that λ is an eigenvalue for ρ(Z). Since Z ∈ r ∩ a ⊆ ra , then ρ(Z) is nilpotent, and λ = 0. Suppose now g nilpotent and Z = A. In this case the subspace U consists only of polynomials, hence β(A) is nilpotent. Observe that, by Corollary 1.3.4 we can choose a basis of U such that each matrix of β(X) is upper triangular, and if X ∈ r(g), β(X) is strictly upper triangular.

28

Basic theory of solvable Lie algebras and Lie groups

1.6 Lie groups As with Lie algebras in Section 1.1, we begin by collecting a few definitions, facts, and examples. We shall provide proofs only for some selected points. A Lie group is a second countable topological group G with the structure of a real analytic manifold for which the product mapping (x,y) → xy is analytic on the product manifold G × G, and the inverse mapping x → x −1 is analytic on G. Lie groups G1 and G2 are isomorphic if there is a bianalytic group isomorphism φ : G1 → G2 , and a Lie group is said to be a realization of its isomorphism class. The Lie algebra g = Lie(G) of a Lie group G is the real vector space of left invariant vector fields on G, equipped with the usual bracket of vector fields, and is naturally identified with the tangent space to G at the identity element e ∈ G simply by evaluating the vector field at the identity.

1.6.1 Examples of Lie groups We describe a few examples of Lie groups and their Lie algebras that will be useful in this setting. As usual k = R or C. Example 1.6.1 The group Rn with vector addition is a Lie group whose Lie algebra is the space of constant vector fields, canonically identified with Rn , with vanishing Lie bracket. Similarly, the torus Rn /Zn is also a Lie group whose Lie algebra can be identified with Rn . Thus nonisomorphic connected Lie groups can have isomorphic Lie algebras.  Example 1.6.2 The group GL(n,R) of n × n invertible matrices with real entries is a Lie group, where the underlying n2 -dimensional manifold structure is obtained by identifying GL(n,R) with an open subset of the vector space gl(n,R) of n × n matrices with real entries. Each X ∈ gl(n,R) defines the unique left-invariant vector field on G, given at f ∈ C ∞ (G) by # d ## Xf (s) = f (setX ), s ∈ G. (1.6.1) dt #t=0 The Lie bracket of two such vector fields is given by the usual Lie bracket in gl(n,R). Thus the Lie algebra of the group GL(n,R) is naturally identified with gl(n,R). Similarly, the group GL(n,C) has a 2n2 -dimensional real manifold structure by which it is a Lie group with Lie algebra gl(n,C).  Example 1.6.3 The group of invertible upper-triangular matrices T (n,k) = {g ∈ GL(n,k) : gi,j = 0 (i > j )}

1.6 Lie groups

29

is an open subset of the vector space t(n,k), and is therefore a Lie group. Note that t(n,k) is closed in gl(n,k), so T (n,k) is a closed subgroup of GL(n,k).  A closed subgroup of GL(n,k) (k = R or C) is called a linear group or matrix group. It is possible to prove that a matrix group has a unique manifold structure whose underlying topology is the relative topology, and by which it is a Lie group (see Result 1.6.9 in Section 1.6.2). If G ⊂ GL(n,k) is a matrix group, then the Lie algebra g of G is identified via (1.6.1) with the set {X ∈ gl(n,k) : exp RX ⊂ G}. Thus the Lie algebra of T (n,k) is t(n,k). Given a Lie group G, a representation of G is a continuous homomorphism of G into GL(V ), where V is a topological vector space over R or C. Here GL(V ) has the strong topology: Tn → T means Tn v → T v holds for all v ∈ V . Just as with Lie algebras, an injective representation is called faithful, or a realization of G. As with Lie algebras, our immediate interest will be in finite-dimensional representations: representations of G where dim V < ∞. It turns out that finite-dimensional representations are an indispensable tool in the study of unitary representations in Hilbert spaces that we take up in Chapter 3, even though unitary representations are generally not finite dimensional. Example 1.6.4 Fix d ∈ N. The set G = Rd × Rd × R, with group product (x,y,z)(x ,y ,z ) = (x + x ,y + y ,z + z + x1 y1 + x2 y2 + · · · + xd yd ); is called the Heisenberg group. The Lie algebra of vector fields on G that are left-invariant for this group product is spanned by X1, . . . ,Xd ,Y1, . . . ,Yd , and Z, where Xi =

∂ ∂ ∂ ∂ , Yi = + xi , Z = . ∂xi ∂yi ∂z ∂z

Observe that [Xi ,Yi ] = Z, 1 ≤ i ≤ d, [Xi ,Z] = [Yi ,Z] = 0, and [Xi ,Yj ] = 0 if i = j . Thus, we recover the Lie algebra of Example 1.1.5. The simplest realization of the Heisenberg group is given by (the missing entries are zero) ⎤ ⎡ z 1 x1 x2 . . . xd ⎢ 1 0 ... 0 y1 ⎥ ⎥ ⎢ ⎢ .. .. ⎥ .. ⎥ ⎢ . . . ⎥ (x,y,z) → ⎢ ⎥ ⎢ ⎢ 1 0 yd−1 ⎥ ⎥ ⎢ ⎣ 1 yd ⎦ 1

30

Basic theory of solvable Lie algebras and Lie groups

The image of this map is closed in GL(d + 2,R), so we shall see this implies it is a realization of G as a matrix group. Next, consider the group product on Cd × R defined by (w1,w2, . . . ,wd ,z)(w1 ,w2 , . . . ,wd ,z ) $ =

w1 + w1 , . . . ,wd

+ wd ,z

%

1 +z + Im(w i wi ) 2 d



i=1

((w,z),(w,z ) ∈ Cd × R.) The map $

1 (x,y,z) → x1 + iy1, . . . ,xd + iyd ,z − xi yi 2 d

%

i=1

is a Lie group isomorphism from G to Cd × R. The Lie algebra of Cd × R is spanned by the following vector fields: Xj =

yj ∂ xj ∂ ∂ ∂ ∂ − + , Yj = , Z= . ∂xj 2 ∂z ∂yj 2 ∂z ∂z

In terms of the complex variables wj , since     1 ∂ 1 ∂ ∂ ∂ ∂ ∂ = −i = +i , , ∂wj 2 ∂xj ∂yj ∂wj 2 ∂xj ∂yj then these fields can be written wj − w j ∂ ∂ ∂ + − , ∂wj ∂w j 4i ∂z   wj + w j ∂ ∂ ∂ ∂ − Yj = i , Z= . + ∂wj ∂w j 4 ∂z ∂z

Xj =



Example 1.6.5 Let R be the group of real numbers under addition, and R× + the group of positive real numbers under multiplication. The group Aff(R), defined as R × R× + with the product (b,a)(b,a  ) = (b + ab,aa  ), is a Lie group. We say that Aff(R) is the semidirect product R  R+ of R by R+ . The Lie algebra aff(R) of Aff(R) is the real span of the operators ∂ ∂ , a . ∂a ∂b It is simple to check that Aff(R) is isomorphic with the group of affine maps x → ax + b on R under composition.

1.6 Lie groups

31

A faithful representation is given by sending (b,a) to the matrix   a b , a > 0, b ∈ R, 0 1 whose Lie algebra is that of Example 1.1.6, the span of     1 0 0 1 A= , Y = . 0 0 0 0 The linear map defined by ∂ → A, ∂a

a

∂ → Y ∂b 

is a realization of aff(R).

Example 1.6.6 The group E2 of Euclidean displacements of the plane is similarly a semidirect product C  T, where T = {τ ∈ C : |τ | = 1}, and (also similarly) has a realization given by &  ' τ z τ ∈ T, z ∈ C . E2 = 0 1 More generally, fix λ ∈ C \ R, and set & λt  ' e z Gλ = : t ∈ R, z ∈ C . 0 1 The Lie algebra gλ is that of Example 1.1.7: it consists of all matrices in gl(2,C) of the form &   ' tλ z tA + Re(z)X + Im(z)Y = : t ∈ R, z ∈ C 0 0 where A, X, and Y are as in Example 1.1.7. It is easy to check that in the complexification of gλ , tA+Re(z)X+Im(z)Y = tA+ 12 (z(X−iY )+z(X+iY )) and [A,X − iY ] = λ(X − iY ). If λ = i, then gi is the Lie algebra e2 of E2 , if λ = 1 + ia, with a = 0, G1+ia is the Grélaud group.  Each of the groups: Heisenberg groups, Aff(R), E2 , and Gλ are (realized as) matrix groups. A direct computation shows that in each case, the Lie algebra g of the group G realized by the definition (1.6.1), satisfies g = {X ∈ gl(n,k) : exp RX ⊂ G} where exp is the matrix exponential.

32

Basic theory of solvable Lie algebras and Lie groups

1.6.2 Basic Lie group theory results We now itemize some basic results from the theory of Lie groups, which we will freely use in the sequel without further comment. A detailed exposition of Lie theory can be found in numerous texts, for example, in the books by Helgason [48] or Varadarajan [88]. 1.6.7 Let G be a Lie group with Lie algebra g. There is a unique analytic map exp : g → G with the property that for each X ∈ g, γX : t → exp tX is the integral curve γ tangent to the vector field X such that γ (0) is the identity element e of G. The map γX is the only analytic morphism γ of the additive group R into G such that γ  (0) = X. There is an open neighborhood U of 0 in g such that the restriction of exp to U is a homeomorphism of U onto an open subset V of G. If [X,Y ] = 0 in g then exp(X + Y ) = exp X exp Y . Every element of the connected component G0 of the identity in G is a finite product of elements in the image of exp. Observe that, for the group GL(n,k), the exponential mapping is simply the usual exponential of matrices. By uniqueness of the exponential map, the same holds for any linear group. If two Lie groups have the same Lie algebra, then they are locally isomorphic. The Lie algebra structure, which is an algebraic setting on a finitedimensional vector space encodes to a large extent the Lie group structure. 1.6.8 A Lie subgroup of G is a subgroup for which the inclusion mapping is an immersion. A Lie subgroup H of G has the structure of a Lie group with Lie algebra h isomorphic with the Lie subalgebra {X ∈ g : exp RX ⊂ H } of g. We remark that a Lie subgroup need not be a closed subgroup. Let H be a Lie subgroup G, then H is a subgroup of G, but the underlying topology for the manifold structure of H is not necessarily the relative topology. The usual example of this situation is the rope with irrational slope on the torus: consider the two-dimensional torus G = {(τ1,τ2 ) ∈ C2 : |τi | = 1}. If a is irrational (a ∈ R \ Q), then the subset H = {(eit ,eiat ) : t ∈ R} is a Lie subgroup of G even though it is clearly not closed and its manifold topology (the topology of R) is strictly finer The Lie algebra h of H is the  than the relative topology. 2 1 linear subset R a of the Lie algebra g = R of G. 1.6.9 Every closed subgroup of G is a Lie subgroup, and a Lie subgroup is closed in G if and only if its topology as a Lie group coincides with its relative topology as a subset of G. If H is a closed subgroup of G, then

1.6 Lie groups

33

G/H has a unique differentiable structure with underlying topology the quotient topology, and such that the canonical map from G onto G/H is analytic. The tangent space at eH is canonically identified with g/h. In fact, the structure of smooth manifold on G/H can be described by a choice of local charts around eH ∈ G/H . Let m be a supplementary space for h in g, such local charts are given by maps X → exp XH ∈ G/H , where X belongs to sufficiently small open subsets U in m. Especially, if H = G0 is the connected component containing e in G, it is an open and closed subgroup of G, and the quotient G/G0 is a discrete Lie group. 1.6.10 Given any subalgebra h of g, there is a unique connected Lie subgroup H of G whose Lie algebra is isomorphic with h. If H is a normal subgroup in G then its Lie algebra h is an ideal in g, and if G is connected and h an ideal in g, the corresponding connected subgroup H of G is normal. Thus there is a canonical bijection between connected Lie subgroups H of G and Lie subalgebras h of g. If h is a subalgebra of g, then, as a set, the connected Lie subgroup of G, associated to h is the subgroup generated by the subset exp h of G. If H is a Lie subgroup which is not connected, the connected component H 0 of e in H is a Lie subgroup of G, in fact the connected Lie subgroup of G associated to the Lie algebra h of H viewed as a subalgebra of g. If h is an ideal of g, the corresponding connected Lie subgroup H may not be normal. Here is a nilpotent example. Consider the following nonconnected linear Lie group:

G=

⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩

⎡

1

⎢ (x,y,z,n) = ⎢ ⎣

−y 1

x − ny n 1

⎫ ⎤ 2z ⎪ ⎪ ⎬ x⎥ 3 ⎥ : (x,y,z) ∈ R , n ∈ Z . ⎦ ⎪ y ⎪ ⎭ 1

The connected component of the identity G0 of G is the set of the elements (x,y,z,0); it is a realization of the Heisenberg Lie group (see Example 1.6.4). Thus the Lie algebra g of G is the Heisenberg Lie algebra, with basis (X,Y,Z), where ⎡

⎤ 0 0 1 0 ⎢ 0 0 1⎥ ⎥, X=⎢ ⎣ 0 0⎦ 0

⎡

0 −1 0 ⎢ 0 0 Y =⎢ ⎣ 0

⎤ 0 0⎥ ⎥, 1⎦ 0

⎡

⎤ 0 0 0 2 ⎢ 0 0 0⎥ ⎥. and Z = ⎢ ⎣ 0 0⎦ 0

34

Basic theory of solvable Lie algebras and Lie groups

Now h = RY + RZ is an ideal in g, but the corresponding connected subgroup H is not normal. Indeed, ⎫ ⎧ ⎡ ⎤ 1 −y 0 2z ⎪ ⎪ ⎪ ⎪ ⎬ ⎨ ⎢ ⎥ 1 0 0 2 ⎥ : (y,z) ∈ R i H = (0,y,z,0) = ⎢ ⎣ ⎦ ⎪ ⎪ 1 y ⎪ ⎪ ⎭ ⎩ 1 and for n ∈ Z \ {0}, (0,0,0,n)(0,y,0,0)(0,0,0,n)−1 = (ny,y,0,0) ∈ / H. 1.6.11 Let g be a real Lie algebra. Then there is a unique (up to isomorphism) connected and simply connected Lie group G with Lie algebra g. Every connected Lie group G1 with Lie algebra g is the quotient of G by a discrete normal subgroup. From now on, every simply connected group is assumed to be connected. Especially, if G is a simply connected Lie group, and α : G → GL(V ) is a continuous homomorphism, if α(G) is closed in GL(V ), then α(G), equipped with the relative topology, is a connected Lie group, α is analytic and by this result it is a covering map. If it is one-to-one, α(G) is a Lie group isomorphic to G, that means α(G) is a realization of G. Observe that the group E2 of Example 1.6.6 is homeomorphic to C × T where T = {τ ∈ C : |τ | = 1}, so E2 is connected but not simply connected. The simply connected group with Lie algebra e2 can be realized as ⎫ ⎧ ⎤ ⎡ t 0 0 e ⎬ ⎨ E˜ 2 = (z,t) = ⎣ eit z ⎦ : t ∈ R, z ∈ C . ⎭ ⎩ 1 The kernel of the covering map is {(0,t) : t ∈ 2π Z}. Result 1.6.11 says that a simply connected Lie group is completely determined by its Lie algebra. We saw in the preceding section that each solvable Lie algebra has an upper-triangular realization. We shall prove that the corresponding simply connected Lie group has also an upper-triangular realization. 1.6.12 Let G and H be two Lie groups, with Lie algebras g and h. Let φ : G → H be a continuous homomorphism of Lie groups. Then φ is infinitely differentiable, in fact analytic, and the differential dφ : g → h at the identity is a Lie algebra homomorphism satisfying φ(exp X) = exp dφ(X) for all X ∈ g. The Lie algebra of ker φ is ker dφ. The map φ is a covering if and only if φ is surjective and ker φ discrete.

1.6 Lie groups

35

For instance, if α : G → GL(V ) is a representation of the Lie group G on the finite-dimensional k vector space V , then α is smooth and its differential is the representation β = dα of the Lie algebra defined by: # d ## α(exp tX)v. β(X)v = dt #t=0 The stabilizer of v is the closed subgroup G(v) = {s ∈ G : α(s)v = v}; its Lie algebra is the annihilator of v, denoted by g(v) = {X ∈ g : β(X)v = 0}. We shall study this situation in more detail in Section 1.9. Suppose φ : G → H is a surjective continuous homomorphism. Then ker φ is a closed, normal subgroup, and G/ ker φ has a natural differential structure (see 1.6.9). With this structure G/ ker φ is a Lie group, isomorphic to H , and the Lie algebra h is (via dφ) isomorphic to g/ ker dφ. Let G be a Lie group. For any y ∈ G, the inner automorphism σy : x → yxy −1 has a differential, denoted Ad(y), which is an automorphism of the Lie algebra g of G. Especially, exp(Ad(y)X) = y exp Xy −1 . If G is a linear group, then, for any X in g, Ad(y)X = yXy −1 . Let G be a Lie group with Lie algebra g. Then the group Aut(g) of g is a Lie group, the map Ad : G → Aut(g), y → Ad(y) is a continuous homomorphism, its differential is the Lie algebra homomorphism ad : g → Der(g) (Der(g) is the Lie algebra of the derivations of g). Especially, Ad(exp Y )X = ead(Y ) X. 1.6.13 Let g and h be Lie algebras, and π : g → h a Lie algebra homomorphism. Let G and H be the corresponding simply connected Lie groups with Lie algebras g, h, respectively. Then π is the differential of a unique Lie group homomorphism  : G → H . This Result 1.6.13 says that there is a one-to-one natural correspondence between simply connected Lie groups homomorphisms and corresponding Lie algebras homomorphisms. Of course this does not hold for all connected Lie groups: for instance R and R/Z have the same Lie algebra R, but there is no continuous homomorphism from R/Z into R whose differential is the identity. Lemma 1.6.14 Let G be a simply connected Lie group with Lie algebra g. Let h be an ideal in g, and H the corresponding connected Lie subgroup in G. Then H is closed and normal. Moreover G/H is the simply connected Lie group with Lie algebra g/h. Proof: Denote by G1 the simply connected Lie group with Lie algebra g/h. The homomorphism π : g → g/h gives rise to a unique Lie group homomorphism  : G → G1 . Since the rank of d = π is the dimension

36

Basic theory of solvable Lie algebras and Lie groups

of G1 , then the group (G) is open in G1 . Every open subgroup is closed, and since G1 is connected, we have (G) = G1 . Now consider the normal subgroup K = ker ; since K is closed, it is a Lie subgroup of G, and since the Lie algebra homomorphism corresponding to  is π , the Lie algebra of K is h. But H is a connected Lie subgroup with Lie algebra h, so H is the connected component of K containing e, and hence H is closed. Consider the canonical map p : G/H → G/K = G1 ; its kernel is K/H , which is discrete, and hence p is a covering. Since G1 is simply connected and G/H connected, it is the trivial covering, and G/H  G1 is simply connected with Lie algebra g/h. Lemma 1.6.14 may fail if G is not simply connected. Example 1.6.15 Let G be the Lie group defined by G = {(x,y,τ,σ ) : x,y ∈ R, τ,σ ∈ C, |τ | = |σ | = 1} with the group product 

(x,y,τ,σ )(x ,y ,τ ,σ  ) = (x + x ,y + y ,τ τ  eixy ,σ σ  ei

√

2xy 

).

It is clear that G is connected but not simply connected. A basis for its Lie algebra g is (X,Y,T ,S), such that exp xX = (x,0,1,1), exp yY = (0,y,1,1), exp tT = (0,0,eit ,1), exp sS = (0,0,1,eis ). The only nonvanishing bracket among elements of this √ basis is [X,Y ] = T + √ 2S. Hence the ideal h = [g,g] is just [g,g] = R(T + 2S), and the connected subgroup corresponding to [g,g] is {(0,0,eit ,ei

√

2t

) : t ∈ R}, 

which is not closed.

1.7 Nilpotent and solvable Lie groups For any group G, the commutator [G,G] of G is the normal subgroup generated by the commutators [x,y] := xyx −1 y −1, x,y ∈ G. It is easy to show that [G,G] is the smallest normal subgroup H in G such that G/H is abelian.

1.7 Nilpotent and solvable Lie groups

37

The derived series and lower central series for any group G are defined inductively, as in the case of Lie algebras. Put G(0) = G(0) = G and for k ≥ 1, G(k) = [G(k−1),G(k−1) ] and G(k) = [G,G(k−1) ]. A group G is solvable if G(k) = {e} for some k and nilpotent if G(k) = {e} for some k. Clearly any nilpotent group is solvable, any subgroup of a solvable (resp. nilpotent) group is solvable (resp. nilpotent). A similar statement holds for a quotient G/H of a solvable or nilpotent group G by a normal subgroup. Proposition 1.7.1 Let G be a connected Lie group with Lie algebra g. Then [G,G] is the connected Lie subgroup of G with Lie algebra [g,g]. ˜ the simply connected Lie group with Lie algebra g, and Proof: Denote by G ˜ with Lie algebra [g,g]. Then L is a by L the connected Lie subgroup of G ˜ ˜ closed normal subgroup of G, and by Lemma 1.6.14, G/L is simply connected ˜ with Lie algebra g/[g,g]. Since g/[g,g] is abelian, then G/L is a vector group, ˜ G] ˜ ⊂ L. Denote by π the covering map π : G ˜ → G; then and hence [G, π is a homomorphism, H := π(L) is a Lie subgroup of G, and [G,G] ⊂ ˜ G]) ˜ ⊂ H . The kernel of π is a normal discrete subgroup of G, ˜ and so π([G, the differential of π is the identity, and it follows that the Lie algebra of H is dπ([g,g]) = [g,g]. Since H is connected, it is the connected Lie subgroup of G with Lie algebra [g,g]. It remains to show that H ⊂ [G,G]. For this we appeal to the notion of maximal integral manifold in G associated to the involutive distribution x → dlx ([g,g]) (here x ∈ G and lx is the left translation lx (y) = xy). Recall that H is the only such integral manifold in G, containing e (see [88]). Since [G,G] is a subgroup of H , to prove that H ⊂ [G,G], it is therefore enough to build, for any [X,Y ] ∈ [g,g], a C 1 map γ : [0,1[→ G such that γ (0) = e, γ  (0) = [X,Y ], and γ (t) ∈ [G,G] for any t. Indeed, for any x ∈ [G,G], the curve lx ◦ γ would be tangent at x to dlx ([X,Y ]), therefore the integral manifold H associated to [g,g], and passing through e would be a subset of [G,G]. Consider thus X and Y in g; define the analytic function φ : R → G, with values in [G,G] by: φ(t) = exp tX exp tY exp −tX exp −tY . Following S. Helgason, [48, chapter II, lemma 1.8], for instance, this function can be expressed as   φ(t) = exp t 2 [X,Y ] + O(t 3 ) . √ The desired map γ is now obtained by putting γ (t) = φ( t), t ∈ [0,1].

38

Basic theory of solvable Lie algebras and Lie groups

By Lemma 1.6.14, if G is simply connected, then [G,G] is closed. However, it is easy to check that the Lie algebra of Example 1.6.15 is solvable, so the group defined in this example is both connected and solvable, but not simply connected. Thus if G is a connected (but not simply connected) and solvable Lie group, then the Lie subgroup [G,G] may not be closed. The following is almost immediate from Proposition 1.7.1. Corollary 1.7.2 Let G be a connected Lie group with Lie algebra g. Then for each k, G(k) is the connected Lie subgroup of G with Lie algebra g(k) , and G(k) is the connected Lie subgroup of G with Lie algebra g(k) . Therefore G is a solvable (resp. nilpotent) Lie group if and only if g is solvable (resp. nilpotent). Proof: We only prove the result for G(k) . By repeated application of Proposition 1.7.1 we see that G(k) is the connected Lie group with Lie algebra g(k) . Solvability means that g(n) = {0}, and this holds if and only if the corresponding connected Lie subgroup G(n) is {e}. Observe that if G is a solvable (resp. nilpotent) Lie group, then its connected component of the identity G0 is solvable (resp. nilpotent) and connected, thus the Lie algebra g of G is solvable (resp. nilpotent). The vector groups and the Heisenberg group are nilpotent. The Lie algebra of the group Aff(R) is solvable but not nilpotent (Example 1.1.6), hence the same can be said of the group Aff(R). Similarly, the Lie group Gλ of Example 1.6.6 is solvable but not nilpotent, since [gλ,gλ ] = RX + RY is abelian, but for all k, g(k) = RX + RY . Finally, GL(n,R) (n > 1) is not a solvable Lie group since gl(n,R) is not solvable (Example 1.1.9).

1.7.1 Semidirect product Let N be a simply connected Lie group with Lie algebra n. The group Aut(n) consisting of all automorphisms of n is a closed subgroup of GL(n), and hence a Lie group. Let K be a Lie group and  : K → Aut(n) a continuous homomorphism. Since N is simply connected, each (k) defines a unique analytic automorphism φ(k) of N , and φ : K → Aut(N ) is a homomorphism. Define a group operation on the product manifold N × K by (x,k)(x ,k  ) = (x φ(k)(x  ),kk  ). It is easy to check that the product manifold becomes a Lie group, denoted by N φ K. The Lie algebra of N φ K is canonically isomorphic with the product n × k with bracket

1.7 Nilpotent and solvable Lie groups

39

[X + Y,X + Y  ] = [X,X ] + d(Y )(X ) − d(Y  )(X) + [Y,Y  ], X,X  ∈ n,Y,Y  ∈ k.

(1.7.1)

We say that N φ K is the right semidirect product of N by K. We shall frequently omit the subscript φ, when there is no possible confusion. Since N is simply connected, N φ K is simply connected if and only if K is simply connected. If we identify N with the normal subgroup N × {eK } of N φ K and K with {eN } × K in N φ K, then N φ K = N K, (k) becomes Ad(k)|n and φ(k)x = kxk −1,x ∈ N . Conversely, suppose that G = N K is a simply connected Lie group, with N and K closed simply connected Lie subgroups of G with Lie algebras n,k respectively, such that N is normal in G and N ∩ K = {e}. Then g = n ⊕ k, and G is canonically isomorphic with the semidirect product N φ H , where φ(k)(h) = khk −1 . Remark 1.7.3 The definition of semidirect product can be made equally well with the conormal subgroup K on the left. The bracket operation in the Lie algebra does not change, while the group product is (k,h)(k ,h ) = (kk ,φ(k  )−1 (h)h ). The resulting group, denoted by K φ H , is called the left semidirect product of N by K. The map α : H φ K → K φ H defined by α(h,k) = (k,φ(k)−1 (h)) is a Lie group isomorphism. With G,H,K as above, we may identify G = N K with N  K, G = KN with K  N. We have the following application of Lemma 1.6.14. Lemma 1.7.4 Let G be a simply connected Lie group with Lie algebra g, and let n and k be subalgebras of g, with corresponding connected subgroup N and K, such that (1) n is an ideal in g, (2) g = n ⊕ k, and (3) K is simply connected, Then N ∩ K = {e}, N is simply connected and G = N K = KN. Proof: Since n is an ideal then by Lemma 1.6.14, the corresponding connected Lie subgroup is normal in G. Then N K is a connected Lie subgroup whose Lie algebra is g, and it follows that G = N K = KN. Again, by Lemma 1.6.14, G/N = NK/N  K/N ∩ K is simply connected. But the Lie algebra of N ∩ K is n ∩ k = {0}, so N ∩ K is discrete. Since K is simply connected, then

40

Basic theory of solvable Lie algebras and Lie groups

N ∩ K = {e}. Since G is homeomorphic to N × K and simply connected, N is simply connected. Using the semidirect product we have the following sharpening of the Ado Theorem. Proposition 1.7.5 Let G be a simply connected solvable Lie group. Then G has a homeomorphic realization as a linear group of upper-triangular matrices. If G is nilpotent, then each s in G is realized as an upper-triangular unipotent matrix. Proof: Thanks to the Ado Theorem (Theorem 1.5.1), the Lie algebra g of G has a realization β in a finite-dimensional vector space U over C. By Proposition 1.3.7, there is a basis F1, . . . ,Fq of U for which β(g) ⊂ t(q,C). By Lie Group Result 1.6.13, β is the differential of a unique homomorphism denoted α : G → T (q,C). Moreover, if G is nilpotent, α(s) = 1 + n(s), with n(s) strictly upper triangular. The only point is to prove that there is such a realization β  for which the group α  (G) is closed and α  is a homeomorphism. To do this, we proceed by induction on the dimension of G. If G = R is one dimensional, then G has the simple homeomorphic realization  t →

1 0

 t 1

which is clearly closed. Suppose dim g > 1, let a be a codimension 1 ideal in g and H the corresponding connected subgroup in G. H is normal and simply connected. Suppose there is a finite-dimensional realization ρ of H on a complex vector space W such that ρ is an homeomorphism on the closed set ρ(H ). In Theorem 1.5.1, we built a finite-dimensional realization β acting in a subspace U of C ∞ (R,W ). Fix a vector A in g which is not in a. Since a is an ideal, then G = R  H , and the following is a realization α  of G on C2 ⊕ U : ⎡

1

α  (exp tA h) = ⎣

t 1

⎤ 0 ⎦ 0 α(exp tA h)

(t ∈ R, h ∈ H ).

Clearly, if G is nilpotent, α(s) is 1 + n(s), then α  (s) = 1 + n (s) with n (s) strictly upper triangular. To see that α  (G) is closed, consider a sequence (exp tn A hn )n∈N in G such that (α  (exp tn Ahn hn ))n converges to some complex matrix. Then (tn )n converges to some t ∈ R and (α(exp tn A hn ))n converges to some complex

1.7 Nilpotent and solvable Lie groups

41

matrix in GL(U ). For each positive integer n, we have α(hn ) = α(exp −tn A) α(exp tn Ahn ), and so both sequences (α(exp −tn A))n and (α(hn ))n converge. Now recall that for each w ∈ W , if F (t) = w for any t, then F is in U and by construction: (α(hn )F )(0) = ρ(hn )w. Thus (ρ(hn )) is converging in GL(W ), (hn ) is converging to h ∈ H . This implies that (exp tn A hn ) converges to exp tA h, thus α  (G) is closed in GL(U ). By the discussion after Result 1.6.11, α  is a realization of G.

1.7.2 Coexponential bases and coordinates A concrete expression of the intimate connection between an arbitrary Lie group G and its Lie algebra g is described in Section 1.6.2, Result 1.6.1: there is an open neighborhood U of 0 in g and an open neighborhood V of e in G such that exp |U is a diffeomorphism of U onto V . For some groups G it is possible to take U = g, V = G; if this is the case then the Lie group G is said to be exponential. Every exponential Lie group is evidently simply connected, and in Section 1.8, it is shown that every exponential Lie group is solvable (Corollary 1.8.2). In the present section it is shown that, while not every simply connected solvable Lie group is exponential, its Lie algebra can still be used to provide global coordinates for the group. Moreover, given a solvable Lie algebra g over R, there is a procedure for constructing a simply connected Lie group with Lie algebra g. First, recall that a simply connected nilpotent Lie group is exponential (this can be viewed as a consequence of the Ado Theorem 1.7.5). Theorem 1.7.6 ([16], Theorem 1.2.1) Let G be a simply connected nilpotent Lie group with Lie algebra g. Then (1) The exponential map exp : g → G is a diffeomorphism from g onto G. (2) When G is identified with g via the exponential map, the inverse and group operation maps are polynomial functions. Given a nilpotent Lie algebra g over R, exp X exp Y = exp CH (X,Y ), where CH (X,Y ) is defined by the Campbell–Baker–Hausdorff formula 1 1 CH (X,Y ) = X + Y + [X,Y ] + [X,[X,Y ]] 2 12 1 − [Y,[X,Y ]] + · · · , X,Y ∈ g. 12

42

Basic theory of solvable Lie algebras and Lie groups

Since g(k) = {0} for some k, this formula defines a polynomial group operation on g, and thus g becomes a simply connected Lie group with Lie algebra g, which is a realization of the corresponding simply connected nilpotent Lie group G. For an extensive and detailed exposition of the theory of nilpotent Lie groups, the reader is referred to the book by L. Corwin and F. Greenleaf [16]. Even in the nilpotent case, the preceding may not provide the most convenient coordinate system for G. To develop an alternative, a somewhat careful choice of basis for the Lie algebra g is required. Definition 1.7.7 Let g be a Lie algebra over R. Let (X1, . . . ,Xn ) be an ordered basis of g and put gi = spanR {X1, . . . ,Xi }, 1 ≤ i ≤ n. (X1, . . . ,Xn ) is said to be a weak Malcev basis if gi−1 is an ideal in gi for all 2 ≤ i ≤ n. Every solvable Lie algebra has a weak Malcev basis. Lemma 1.7.8 Let g be a solvable Lie algebra over R. Let (Z1, . . . ,Zn ) be a good Jordan–Hölder basis for s = gc with sj = spanC {Z1, . . . ,Zj }, 1 ≤ j ≤ n. Suppose that (X1, . . . ,Xn ) is a basis for g such that (1) if sj and sj −1 are real, then Xj ∈ sj \ sj −1 , and (2) if sj is not real, then sj +1 = sj −1 + spanC (Xj ,Xj +1 ). Then (X1, . . . ,Xn ) is a weak Malcev basis. Proof: Put gj = spanR {X1, . . . ,Xj }, 1 ≤ j ≤ n. Observe that if sj is real, then (1) and (2) imply that gj = sj ∩ g, so gj is an ideal in g. Suppose that sj is not real. Then sj +1 is real, and by Lemma 1.4.8, [sj +1,sj +1 ] ⊂ sj −1 . Hence [gj +1,gj +1 ] ⊂ gj −1 ⊂ gj . Thus gj is an ideal in gj +1 . Given a solvable Lie algebra over R equipped with a good Jordan–Hölder sequence (sj ), a weak Malcev basis defined as in Lemma 1.7.8 is said to be a weak Malcev basis associated to the good Jordan–Hölder sequence (sj ). Next comes the notion of a coexponential basis. Definition 1.7.9 Let G be a simply connected solvable Lie group, with Lie algebra g of dimension n and let h be a subalgebra of g of dimension r. A coexponential basis for h in g is an ordered set (Y1,Y2, . . . ,Yn−r ) in g such that (Y1 + h, . . . ,Yn−r + h) is a basis of g/h, and such that (t1, . . . ,tn−r ,h) → exp(t1 Y1 ) . . . exp(tn−r Yn−r )h is a global diffeomorphism from Rn−r × H onto G, where H is the connected subgroup of G with Lie algebra h.

1.7 Nilpotent and solvable Lie groups

43

If h = {0} then we may say that (Y1, . . . ,Yn ) is an exponential basis of g. If (Y1, . . . ,Yn−r ) is a coexponential basis for h, then the map: (t1, . . . ,tn−r ,h) → h exp(tn−r Yn−r ) . . . exp(t1 Y1 ) is a global diffeomorphism from H × Rn−r onto G. A weak Malcev basis is used to define a coexponential basis. Proposition 1.7.10 Let G be a simply connected solvable Lie group with Lie algebra g, and let h be a subalgebra of g of dimension r. (1) There is a weak Malcev basis (X1, . . . ,Xn ) for g and an increasing subsequence (i1, . . . ,ir ) of (1,2, . . . ,n), such that (Xi1 , . . . ,Xir ) is a weak Malcev basis for h. (If h = {0}, then r = 0 and the subsequence (i1, . . . ,ir ) is empty.) (2) Let (Xi )1≤i≤n be any weak Malcev basis for g, satisfying condition (1) for h, let (u1 < · · · < un−r ) be the ordered complement of (i1, . . . ,ir ) in (1, . . . ,n). Put Y1 = Xun−r ,Y2 = Xun−r−1 , . . . ,Yn−r = Xu1 and Zt = Xit , 1 ≤ t ≤ r. Then (Y1, . . . ,Yn−r ) is a coexponential basis for h in g and (Z1, . . . ,Zr ) is an exponential basis of h. Proof: Given a weak Malcev basis (X1, . . . ,Xn ) for g with gi = span{X1, . . . ,Xi },1 ≤ i ≤ n, put Ih = {i : gi ∩ h ⊂ gi−1 } = {i1 < · · · < ir }. Put {1, . . . ,n} \ Ih = {u1 < · · · < un−r }. For i ∈ Ih we have Wi ∈ gi−1 such that Xi +Wi ∈ h; replacing Xi by Xi +Wi if necessary, we may assume that Xi ∈ h for i ∈ Ih . Then it is easy to check that (Xi1 , . . . ,Xir ) is a weak Malcev basis for h. This proves (1). Define the map ϕ : Rn−r × Rr → G by ϕ(t,s) = exp(tn−r Xun−r ) . . . exp(t1 Xu1 ) exp(s1 Xi1 ) . . . exp(sr Xir ). We use induction on n to show that (a) ϕ is a diffeomorphism, and (b) s → ϕ(0,s) is a diffeomorphism from Rr onto H . If n = 1, then either h = {0} or h = g, and G = exp g; in either case (a) and (b) are true. Suppose n > 1 and consider the simply connected subgroup Gn−1 with Lie algebra gn−1 = spanR {X1, . . . ,Xn−1 }. Let ϕn−1 be the map as above for Gn−1 and assume that (a) and (b) are true for ϕn−1 . There are two cases. (i) n ∈ / Ih . In this case, Hn−1 = H ∩ Gn−1 = H , n = un−r and the map ϕn−1 (t,s) = exp(tn−r−1 Xun−r−1 ) . . . exp(t1 Xu1 ) exp(s1 Xi1 ) . . . exp(sr Xir ) is a diffeomorphism between Rn−r−1 × Rr and Gn−1 , such that s → ϕ(0,s) is a diffeomorphism between Rr and H . Now with

44

Basic theory of solvable Lie algebras and Lie groups K = exp RXn , Lemma 1.7.4 implies that G is the left semidirect product of the normal subgroup Gn−1 by K. Hence ϕ(t,s) = exp(tn−r Xun−r )ϕn−1 (t1, . . . ,tn−r−1,s)

satisfies both (a) and (b). (ii) n ∈ Ih . In this case Hn−1 = H , n = ir and the ϕn−1 now has the form ϕn−1 (t,s) = exp(tn−r Xun−r ) . . . exp(t1 Xu1) exp(s1 Xi1 ) . . . exp(sr−1 Xir−1 ). By induction (a) and (b) hold for ϕn−1 , which means that ϕn−1 is a diffeomorphism from Rn−r × Rr−1 onto Gn−1 , and s → ϕ(0,s) is a diffeomorphism from Rr−1 onto the group Hn−1 = H ∩ Gn−1 . Now we invoke Lemma 1.7.4 again: G is the right semidirect product of Gn−1 by exp RXir = exp RXn . We have ϕ(t,s) = ϕn−1 (t,s1, . . . ,sr−1 ) exp sr Xir and it follows that (a) and (b) hold for ϕ. The following corollaries are immediate. Corollary 1.7.11 Let G be a simply connected solvable Lie group with Lie algebra g, and let (X1, . . . ,Xn ) be a weak Malcev basis for g. Then the maps (t1, . . . ,tn ) → exp(tn Xn ) . . . exp(t1 X1 ), (t1, . . . ,tn ) → exp(t1 X1 ) . . . exp(tn Xn ) are global diffeomorphisms from Rn to G. Corollary 1.7.12 Let G be a simply connected solvable Lie group with Lie algebra g and let h be a subalgebra of g. Then the connected Lie subgroup H corresponding to h is closed and simply connected. We adopt the terminology used by Corwin and Greenleaf in [16] in referring to coordinate systems. Definition 1.7.13 Let G be a simply connected solvable Lie group with Lie algebra g. (a) If G is exponential and (X1, . . . ,Xn ) is a basis for g, then the map (t1, . . . ,tn ) → exp(t1 X1 + · · · + tn Xn ) is called a system of canonical coordinates of the first kind.

1.7 Nilpotent and solvable Lie groups

45

(b) If (X1, . . . ,Xn ) is a weak Malcev basis for g, then either of the maps (t1, . . . ,tn ) → exp(tn Xn ) . . . exp(t1 X1 ), (t1, . . . ,tn ) → exp(t1 X1 ) . . . exp(tn Xn ) are called a system of canonical coordinates of the second kind. The following are simple nonnilpotent examples illustrating first and second kind coordinates. Example 1.7.14 Fix λ ∈ C \ R, and recall the group Gλ of Example 1.6.6: & λt  ' e z Gλ = : t ∈ R, z ∈ C . 1 If Reλ = 0, then the group Gλ is homeomorphic with C×R, and hence simply connected. But if Reλ = 0, then Gλ is homeomorphic with C × T. It is easy to check that the basis (X,Y,A) for gλ given in Example 1.1.7 is a weak Malcev basis for g, and that      λt  1 x 1 iy eλt 0 e x + iy exp(xX) exp(yY ) exp(tA) = = . 1 1 1 1 (1.7.2) Thus (x,y,t) → exp(xX) exp(yY ) exp(tA) is surjective. If Re(λ) = 0, then Gλ is simply connected and the map is bijective and provides canonical coordinates for the second kind according to Corollary 1.7.11. If Re(λ) = 0, (x,y,t) are only some coordinates for Gλ . On the other hand, compute the expression  λt  e F1 (λt)(x + iy) exp(xX + yY + tA) = (1.7.3) 0 1 where eθ − 1 , θ ∈ C \ {0}, F1 (0) = 1. θ If Reλ = 0, then (x,y,t) → exp(xX +yY +tA) does in fact provide canonical coordinates of the first kind. To see this, compute that       λ log |a| λ log |a| log |a| a w exp−1 A + Re w X + Im w Y. = 1 Reλ (a − 1)Reλ (a − 1)Reλ F1 (θ ) =

Thus Gλ is an exponential Lie group when Reλ = 0; in particular the Grélaud group is exponential. Suppose now that λ = i so that Gλ = E2 . Then Gλ is homeomorphic with C × T and so is not simply connected. In this case (1.7.2) shows that

46

Basic theory of solvable Lie algebras and Lie groups

the mapping (x,y,t) → exp(xX + yY + tA) is surjective but not injective; in fact (1.7.3) shows that the mapping itself is not injective since 

1 exponential exp(xX + yY + 2π A) = 1 . Passing to the universal cover E˜ 2 for E2 allows second kind coordinates, but does not really improve the situation for the exponential map itself. Realize E˜ 2 as a subgroup of GL(3,C): ⎫ ⎧⎡ t ⎤ 0 0 ⎬ ⎨ e E˜ 2 = ⎣ eit z ⎦ : t ∈ R, z ∈ C . ⎭ ⎩ 1 If we realize e2 in gl(3,C) as ⎡ ⎤ ⎡ ⎤ ⎡ 0 0 0 0 0 0 1 Y = ⎣ 0 i⎦, X = ⎣ 0 1⎦ , A = ⎣ 0 0

⎤ 0 0 0 0⎦ i

then exp : e2 → E˜ 2 is the matrix exponential. As was the case for simply connected Gλ , the system of canonical coordinates of the second kind for E˜ 2 obtained by using the weak Malcev basis (X,Y,A) is natural: ⎡ ⎤⎡ ⎤⎡ t ⎤ 1 0 0 1 0 0 e 0 0 exp(xX) exp(yY ) exp(tA) = ⎣ 1 x ⎦ ⎣ 1 iy ⎦ ⎣ eit 0⎦ 1 1 1 ⎡ t ⎤ e 0 0 it ⎣ = x + iy ⎦ . e 1 Even though E˜ 2 is simply connected, the exponential map exp : e2 → E˜ 2 is still not a bijection. One computes that ⎡ t ⎤ e 0 0 exp(xX + yY + tA) = ⎣ eit F1 (it)(x + iy)⎦ . 1 It is easy to check that exp(X + 2π A) = exp(Y + 2π A), and that the group element ⎡ 2π ⎤ e 0 0 a=⎣ 1 x + iy ⎦ 1 is not in the image of exp if x + iy = 0.



1.7 Nilpotent and solvable Lie groups

47

Example 1.7.15 In this example, the diamond group G is described as a semidirect product. Let H be the Heisenberg group C × R with the group product of Example 1.6.4. Define γ : R → Aut(h) by γ (s)(w,z) = (eis w,z), and let G = H γ R. Since γ is a real analytic mapping then it is clear that G is a Lie group. The Lie algebra g = C × R × R of G is as follows. Let X,Y, and Z be the basis of h for which exp xX = (x,0,0), exp yY = (iy,0,0), exp zZ = (0,z,0) in H . As expected [X,Y ] = Z. Let A ∈ g such that exp sA = (0,0,s) ∈ G, so that exp sA(w,z,0) exp −sA = (γ (s)(w,z),0) = (eis w,z,0). Then α(s) := exp Ad(exp sA)X = exp sA(1,0,0) exp −sA = (eis ,0,0) is a smooth curve in H and exp[A,X] = α  (0) = (i,0,0) = exp Y . Since the exponential mapping is injective on h we have [A,X] = Y . Similarly one finds that [A,Y ] = −X and [A,Z] = 0. Thus g is the diamond Lie algebra, and we say that G is the simply connected diamond group. Consider the following realization of the diamond Lie algebra g ⎡ ⎤ ⎡ ⎤ 0 0 0 0 0 0 0 0 ⎢ ⎢ ⎥ 0 0 2i ⎥ ⎥ , Y = ⎢ 0 −i 0⎥ , Z=⎢ ⎣ ⎣ 0 0⎦ 0 i⎦ 0 0 ⎤ ⎡ ⎤ 0 0 0 0 1 0 0 0 ⎢ ⎥ ⎢ 0 0 0⎥ 0 1 0 ⎥, A = ⎢ ⎥. X=⎢ ⎣ ⎣ 0 1⎦ i 0⎦ 0 0 ⎡

Apply the semidirect product construction in the discussion preceding Proposition 1.7.11: then the simply connected Lie subgroup of GL(4,C) with Lie algebra g is given by the set of all matrices of the form exp zZ exp yY exp xX exp sA ⎤ ⎡ s 0 0 e 0 ⎢ 1 eis (x − iy) 2iz − ixy + (x 2 + y 2 )/2⎥ ⎥ =⎢ ⎦ ⎣ (x + iy) eis 1 as z,y,x, and s run through R.

48

Basic theory of solvable Lie algebras and Lie groups

Just as in the case of E˜ 2 , the exponential mapping for G is not injective. To θ θ for θ ∈ C \ {0}, and see this, recall F1 (θ ) = e θ−1 and put F2 (θ ) = e −1−θ θ2 4 F1 (0) = 2F2 (0) = 1. For any (z,y,x,s) ∈ R , exp(zZ + yY + xX + sA) ⎡ s 0 e 0 ⎢ (is)(x − iy) 1 F 1 =⎢ ⎣ eis

⎤ 0 2iz + F2 (is)(x 2 + y 2 )⎥ ⎥. ⎦ F1 (is)(x + iy) 1

Since F1 (2π i) = 0, then exp(X + 2π A) = exp(Y Moreover the element ⎡ 2π e 0 ⎢ 1 a = exp X exp 2π A = ⎢ ⎣

+ 2π A). ⎤ 0 0 1 1/2⎥ ⎥ 1 1 ⎦ 1

is not in the range of exp. Therefore G is not an exponential group.



Proposition 1.7.11 has the following easy consequence. Corollary 1.7.16 Let G be a simply connected solvable Lie group with Lie algebra g. Let h, k be subalgebras of g, with h ⊂ k and H , K the corresponding connected subgroups in G. Let (X1, . . . ,Xm ) be a weak Malcev basis for g such that for increasing subsequences (i1, . . . ,ir ) of (1, . . . ,m) and (is1 , . . . ,isr  ) of (i1, . . . ,ir ), (Xi1 , . . . ,Xir ) is a weak Malcev basis for k, and (Xis1 , . . . ,Xis  ) is a weak Malcev basis for h. Let (u1,u2, . . . ,um−r ) be r the ordered complement of (i1, . . . ,ir ) in (1, . . . ,m) and (it1 , . . . ,itr−r  ) be the ordered complement of (is1 , . . . ,isr  ) in (i1, . . . ,ir ). Put Y1 = Xum−r ,Y2 = Xum−r−1 , . . . ,Ym−r = Xu1 , and Z1 = Xit

r−r 

,Z2 = Xit

r−r  −1

, . . . ,Zr−r  = Xit1 .

Then (1) (Z1, . . . ,Zr−r  ) is a coexponential basis for h in k, and (2) (Z1, . . . ,Zr−r  ,Y1, . . . ,Ym−r ) is a coexponential basis for h in g. In Examples 1.7.14 and 1.7.15, a simply connected group appears as a semidirect product of a simply connected nilpotent group with a one-parameter group exp RA, and canonical coordinates of the second kind are illustrated in each case. One observes in these examples that if the eigenvalues λ for

1.7 Nilpotent and solvable Lie groups

49

ad(A) are not purely imaginary, then the exponential map is a bijection, in fact, a diffeomorphism from the Lie algebra onto the Lie group. When some λ is purely imaginary however, the exponential map is neither injective nor surjective. In Section 1.8 we explore these properties in detail, and we will see that these examples illustrate a general result.

1.7.3 Small-dimensional solvable Lie groups We conclude this section with a classification of all simply connected solvable Lie groups of dimension no more than three (up to isomorphism), and we discuss the exponential map for each. If G is abelian, then G = Rn , n ≤ 3, so we focus on nonabelian groups. The only nonabelian simply connected group of dimension two is the group Aff(R) of Example 1.6.5: &  ' a y G= : a > 0, y ∈ R . 1 Its Lie algebra is g = spanR (Y,A), where     0 1 1 0 Y = , A= . 0 0 The exponential map is a bijection:  exp(tA + yY ) = and exp

−1



a

et



et −1 t

  y

1

   log(a) x xY, = log(a)A + 1 a−1

and thus exp is a diffeomorphism. Now let G be a simply connected, nonabelian solvable Lie group of dimension three with Lie algebra g. Let R be the set of roots for g and s = gc . Then r := dim spanC R ≤ dim nil(g)c = dim s − dim spanC R. Thus, when dim s = 3, r ≤ 1 and dim nil(g)c ≥ 2. By Corollary 1.4.7, we have a good Jordan–Hölder sequence passing through the complexification of the nilradical of g, so choose a good Jordan–Hölder basis {Z1,Z2,Z3 } with s1 = CZ1,s2 = span{Z1,Z2 } so that s2 ⊂ nil(g)c . Since the only

50

Basic theory of solvable Lie algebras and Lie groups

noncommutative Lie algebra of dimension two is not nilpotent, s2 is real and commutative, and we may write A = Z3 and g = n  RA where n = s2 ∩ g. The bracket in g is described by the map φ = adn (A) : n → n. The matrix of φ with respect to the basis chosen above is upper triangular and nonzero since g is nonabelian. Let S be the spectrum of φ. For t = 0 the basis Z1 , Z2 , tA is also a good Jordan–Hölder basis. Hence we may choose A so that φ and its spectrum S fulfill exactly one of the following: (a) (b) (c) (d) (e) (f)

S S S S S S

= {0} and φ is not diagonalizable, = {1,0}, = {1,α}, with α ∈ R, |α| ≥ 1 and φ is diagonalizable, = {1} and φ is not diagonalizable, = {1 ± ia} with a > 0, = {±i}.

(a) Here φ is nilpotent, and we may choose A so that  φ=

 0 1 . 0

Thus, g is the (three-dimensional) Heisenberg Lie algebra, and the corresponding simply connected Lie group is the Heisenberg group (Examples 1.1.5 and 1.6.4). By Theorem 1.7.6, the Lie group exponential map is a diffeomorphism; this is easily checked directly using the usual matrix realizations of g and G and the matrix exponential. (b) If S = {1,0}, then g is the direct product R × aff(R), realized as the span of ⎡

1

Z=⎣

⎡ ⎤ 0 0 0 0 0⎦ , Y = ⎣ 0

⎡ ⎤ 0 0 0 0 1⎦ , A = ⎣ 0

0 1

⎤ 0 0⎦ . 0

The corresponding simply connected group is of course R × Aff(R); just as for the group Aff(R), we see that the exponential map is a diffeomorphism. (c) Suppose that S = {1,α} with α ∈ R,|α| ≥ 1. g is realized as the span of ⎡ Y1 = ⎣

0

⎤ ⎡ 0 1 0 0 0⎦ , Y 2 = ⎣ 0

⎡ ⎤ 1 0 0 0 1⎦ , A = ⎣ 0

0 α

⎤ 0 0⎦ . 0

1.7 Nilpotent and solvable Lie groups One computes that for any y1,y2,t ∈ R, ⎡

et

exp(y1 Y1 + y2 Y2 + tA) = ⎣ Thus

and

⎧⎡ ⎨ a G= ⎣ ⎩ ⎡

a

exp−1 ⎣

0 aα

0 aα

0 eαt

51

⎤ F1 (t)y1 F1 (αt)y2 ⎦ . 1

⎫ ⎤ b ⎬ c ⎦ : a > 0, b, c ∈ R ⎭ 1

⎤     b log a α log a ⎦ Y1 + b Y2 + (log a)A. c =c a−1 aα − 1 1

(d) Suppose that S = {1} and φ is not diagonalizable. Here we can write g as the span of ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 0 0 1 0 0 0 1 1 0 Y1 = ⎣ 0 0⎦ , Y2 = ⎣ 0 1⎦ , A = ⎣ 1 0⎦ 0 0 0 so that [A,Y1 ] = Y1 , [A,Y2 ] = Y2 + Y1 . Then ⎡ t ⎤ e tet F1 (t)y1 + (et − F1 (t))y2 ⎦. exp(y1 Y1 + y2 Y2 + tA) = ⎣ et F1 (t)y2 1 Thus

and

⎧⎡ ⎨ a G= ⎣ ⎩ ⎡

a

exp−1 ⎣

a log a a

a log a a

⎫ ⎤ b ⎬ c ⎦ : a > 0, b, c ∈ R , ⎭ 1

⎤ $     % b log a log a 2 ⎦ − ac Y1 c = (b + c) a−1 a−1 1   log a +c Y2 + (log a)A. a−1

(e) Here we suppose S = {1 + ia,1 − ia} with a > 0. Put λ = 1 + ia and choose Z ∈ nc such that [A,Z] = λZ. This is the Grélaud Lie algebra gλ of Examples 1.6.6 and 1.7.14, with X = ReZ, Y = −ImZ. It is shown in

52

Basic theory of solvable Lie algebras and Lie groups

Example 1.7.14 that the exponential map is a diffeomorphism onto the simply connected group Gλ . (f) Similarly, if S = {−i,i} then g = e2 as in Examples 1.6.6 and 1.7.14. The corresponding simply connected Lie group is E˜ 2 , and it is shown in Example 1.7.14 that the exponential map is not a diffeomorphism. Thus the only simply connected Lie group having dimension 1, 2, or 3 that is not exponential is the simply connected cover E˜ 2 of E2 . Observe also that, in contrast with the semisimple case, there are uncountably many nonisomorphic simply connected solvable Lie groups of dimension three.

1.8 Exponential groups Let G be a Lie group with Lie algebra g. Identifying the tangent space Te (G) with g we have Texp X G = d(lexp X )e (g) where for any s ∈ G, ls : G → G is the left product map ls (t) = st. The differential of the exponential mapping exp : g → G maps g to d(lexp X )e (g) and is given by the formula (d exp)X = d(lexp X )e ◦

1 − e−ad(X) . ad(X)

(1.8.1)

(See for example [48, Chapter II, Theorem 1.7] in the book by S. Helgason.) We say that X is a regular point for the exponential map if (d exp)X is an isomorphism. The formula (1.8.1) shows for each X ∈ g, X is a regular point −ad(X) for the exponential map if and only if det( 1−e ad(X) ) = 0. Now λ ∈ C is an eigenvalue of ad(X) if and only if 1−e−λ λ

1−e−λ λ

is an eigenvalue of

1−e−ad(X) ad(X) .

Since

= 0 if and only if λ ∈ 2π iZ \ {0}, then the regularity of the exponential map is characterized algebraically. Lemma 1.8.1 Every X in g is regular for the map exp : g → G if and only if g is an exponential Lie algebra. Using the Levi decomposition (see the Bourbaki book [12] for instance) of g we get the following. Corollary 1.8.2 If g is an exponential Lie algebra, then g is solvable. Proof: Suppose that g is not solvable. Then g contains a nontrivial, semisimple Levi factor, therefore g contains a Lie algebra isomorphic to sl(2,R) or to

1.8 Exponential groups

53

su(2). Now sl(2,R) and su(2) contain a vector X for which ad(X) has 2iπ for −ad(X) an eigenvalue, and hence det( 1−e ad(X) ) = 0. Recall that a Lie group is said to be an exponential Lie group if its exponential mapping is a diffeomorphism. In the classification of simply connected solvable Lie groups of dimension three or less given at the end of Section 1.7.2, the only one that is nonexponential is the group E˜ 2 . Observe also that e2 is the only Lie algebra of dimension three or less which is not exponential. This observation illustrates the algebraic characterization of exponential groups due to J. Dixmier [24]. Theorem 1.8.3 ([24], Theorem 3, [61], Theorem 1) Let G be a simply connected solvable Lie group with Lie algebra g. Let exp : g → G be the exponential mapping for G. Then the following are equivalent: (i) exp is injective, (ii) exp is surjective, (iii) g is exponential. Moreover, if the above conditions hold, then G is exponential. Proof: We will prove that (i) and (iii) are equivalent, and that (ii) and (iii) are equivalent. We proceed by induction on the dimension of g. From the examples presented thus far, we observe that the theorem holds for all simply connected solvable Lie groups of dimension no more than three. Suppose then that dim G = n > 3, and that the theorem holds for any simply connected solvable Lie group with dimension less than n. Choose a good Jordan–Hölder sequence (sj ) for s = gc such that sn−1 is real. First, suppose that (iii) is false. Then there is A ∈ g such that ad(A) has eigenvalue i. Let W ∈ si be an eigenvector: [A,W ] = iW , and i is the smallest index j for which sj is containing such an eigenvector. Write W = X + iY , and put Z = [X,Y ]. Then [W,W ] = −2iZ and i i i [A[W,W ]] = [[A,W ],W ] + [W,[A,W ]] 2 2 2 i = (i[W,W ] − i[W,W ]) = 0. 2

[A,Z] =

Since i is minimal, thus si is not real, and by Lemma 1.4.8, Z ∈ [si ,si ] ⊂ si−1 . Now the same computation gives [A,[W,Z]] = i[W,Z], by the minimality of i, [W,Z] = 0 and [W,Z] = 0. Put h = spanR {A,X,Y,Z}. If Z = 0, then h = e2 , while if Z = 0 then h is the diamond Lie algebra. By Examples 1.7.14 and 1.7.15, we see that in either case exp |h is not injective, so exp is not injective. Thus (i) implies (iii).

54

Basic theory of solvable Lie algebras and Lie groups

Now suppose that (iii) holds. Let A and B be elements of g for which exp A = exp B. We first show that B − A belongs to m := sn−1 ∩ g. Since (iii) holds for m, then the induction hypothesis gives that M = exp m is an exponential Lie group. If both A and B belong to m, then A = B and we are done. Suppose that one of A or B does not belong to m, say A, and write B = tA + Y with Y ∈ m. Now G/M is isomorphic to R, and expg/m (tA + m) = expg/m (B + m) = exp B · M = exp A · M = expg/m (A + m). Since expg/m is injective, then t = 1 and B − A = Y belongs to m. To show that A = B, suppose the contrary, that Y = B − A = 0, and let j , 1 ≤ j ≤ n − 1 such that Y ∈ sj \ sj −1 . Since Y ∈ g, then sj is real. Put n = sj ∩ g and k = sj −1 ∩ g; then dim n/k = 1 or 2. Put h = RA + n and let H and K be the (simply) connected Lie subgroups of G with Lie algebras h and k, respectively. Then dim h/k = 2 or 3, and the condition (iii) holds for h/k, so H /K is an exponential Lie group. Now Y ∈ / k so A + k = B + k. But then in H /K we have exp A · K = exph/k (A + k) = exph/k (B + k) = exp B · K which implies exp A = exp B, a contradiction. Thus (iii) implies (i). We now prove that (ii) and (iii) are equivalent. Suppose first that (iii) holds and let s ∈ G. Put k = (s1 + s1 ) ∩ g. By induction, expg/k is surjective, so there is A such that sK = expg/k (A+k). Put h = RA+k and let H be the connected Lie subgroup of G with Lie algebra h. Then dim h ≤ 3 and (iii) holds for h, so H is an exponential group. Since s ∈ H then we have X ∈ h ⊂ g such that s = exph (X) = exp X. Thus (iii) implies (ii). Suppose now that (iii) does not hold. Then there is k, 1 ≤ k < n, such that the root λk assumes an imaginary value and we denote j + 1 the largest such k. Remark that sj is not real. Put n = (sj + sj ) ∩ g, k = sj −1 ∩ g and let N, K be the corresponding simply connected Lie subgroups. By Lemma 1.4.8, [n,n] ⊂ k, that is, n/k is commutative. Choose Zj ∈ sj such that sj + sj = spanC {Zj ,Zj } ⊕ sj −1 , and so that for all A ∈ g, [A,Zj ] = λj (A)Zj mod sj −1 . Choose A ∈ g such that λj (A) = i and put h = RA + n, and let H be the simply connected Lie subgroup of G with Lie algebra h. Consider the group H /K with Lie algebra h/k. Since H and K are simply connected and K is normal in H , then H /K is the simply connected group with Lie algebra h/k (Lemma 1.6.14). Since n/k is commutative with dimension two,

1.8 Exponential groups

55

)2 . and adh/k (A + k) has eigenvalues ±i, then h/k  e2 , and H /K  E We have seen (Example 1.7.14) that exph/k (ReZj ) exph/k (2π A) does not belong to the range of exph/k . Put s = exp(ReZj ) exp(2π A) ∈ H . Now {exp X K : X ∈ h} ⊂ Range(exph/k ) so sK = exph/k (ReZj ) exph/k (2π A) ∈ / exph/k (h/k) implies that s ∈ / exp(h). Suppose now that for some Y ∈ g, s = exp Y . We thus have exp(2π A) N = exp Y N, so expg/n (2π A+n) = expg/n (Y +n). Here is where we apply the choice of j : since j +1 is the largest index k for which λk assumes a purely imaginary value, then condition (iii) holds for g/n. Hence by induction, G/N is exponential, which gives 2π A + n = Y + n and hence Y belongs to h. But then s ∈ exp(h), a contradiction. This proves that (ii) implies (iii). The last statement of the theorem follows from Lemma 1.8.1. Corollary 1.8.4 Let G be a simply connected solvable Lie group with Lie algebra g. Then the following are equivalent. (i) G is an exponential Lie group. (ii) Each root λ of g is of the form λ = μ(1 + ia) where μ is a real linear form on g and a ∈ R. (iii) There are no subalgebras k ⊂ h in g where k is an ideal in h and h/k  e2 . Proof: Suppose that G is exponential. Then by Lemma 1.8.1, ad is of exponential type. By the remark following Definition 1.3.14, we obtain (ii). Now (ii) implies (i) by Theorem 1.8.3. Suppose there is a subalgebra h and an ideal k in h such that h = k ⊕ RA + RX + RY , with h/k  e2 : [A,X] = Y mod k,

[A,Y ] = −X mod k.

Taking a good Jordan–Hölder sequence for hc passing through kc , we see that i is a generalized eigenvalue for ad(A)|h , and thus a generalized eigenvalue for ad(A). On the other hand, using any good Jordan–Hölder sequence for gc , and the corresponding sequence (λj ) of roots for g, we see that the only

56

Basic theory of solvable Lie algebras and Lie groups

generalized eigenvalues for ad(A) are the λj (A). Thus (ii) does not hold, and (ii) implies (iii). Suppose that (ii) is false. Let (sj ) be a good Jordan–Hölder sequence for s = gc , and let {λj } be the corresponding root sequence. Since (ii) is false there is j , 1 ≤ j < n, and A ∈ g, such that λj (A) = i. Put n = (sj + sj ) ∩ g and k = sj −1 ∩ g; then dim n/k = 2. By Lemma 1.4.8, [n,n] ⊂ k, that is, n/k is commutative. Put h = RA + n; then h/k  e2 . Thus (iii) implies (ii). Considering the roots of a subalgebra or a quotient of a solvable algebra g, the following is immediate. Corollary 1.8.5 Let G = exp(g) be an exponential Lie group. (1) Every connected Lie subgroup of G is closed, and is an exponential Lie group. (2) For any connected normal subgroup K of G, G/K is an exponential Lie group. Now if g is a completely solvable Lie algebra, then g is an exponential Lie algebra. In this case the simply connected Lie group G with Lie algebra g is called a completely solvable Lie group.

1.9 Finite-dimensional group representations Let G be a connected Lie group with Lie algebra g, and let V be a real finite-dimensional vector space. A representation of G on V is a continuous homomorphism α of G into GL(V ). Since GL(V ) is a Lie group, then every representation is infinitely differentiable, in fact, analytic. Given a representation α, define the map dα : g → gl(V ) by # d ## α(exp tX), X ∈ g. dα(X) = dt #t=0 For each X ∈ g, since t → α(exp tX) is a one parameter subgroup of GL(V ), then Result 1.6.7, implies that α(exp tX) = exp tdα(X)

(1.9.1)

holds for all t ∈ R. The mapping dα is a representation of g, and the Lie algebra of the kernel of α in G is the kernel of dα in g (Result 1.6.12). Observe also that, given a representation α of G in V , the semidirect product S = V α G is a Lie group. If β is a representation of g in V , then it is not necessarily the case that

1.9 Finite-dimensional group representations

57

there is a representation of G whose differential is β; however if G is simply connected then there is a unique such representation (Result 1.6.13). Lemma 1.9.1 Let G be a connected Lie group with Lie algebra g, let α be a representation of G in a finite-dimensional real vector space V , and let W be a subspace of Vc . Then W is g-invariant if and only if W is G-invariant. Proof: The proof follows immediately from the fact that G is generated by elements of the form exp X, X ∈ g. The following are now immediate (for Corollary 1.9.3 confer Example 1.1.8). Corollary 1.9.2 Let G be a connected solvable Lie group with Lie algebra g, and let α be a representation of G in a finite-dimensional real vector space V . Let {Ui : 1 ≤ i ≤ n} be a flag for the corresponding representation of g in Vc . Then each subspace Ui is G-invariant. Corollary 1.9.3 Let G be a connected solvable Lie group with Lie algebra g, and let α be a representation of G in a finite-dimensional real vector space V . Then α(G) is a connected solvable subgroup of GL(V ). We make a few observations about derivatives. Let f be a smooth function on V and fix v ∈ V . We identify the tangent space Tv (V ) at v with V in the usual way. For each X ∈ gl(V ), recall the elementary relation # d ## f (exp tXv) = dfv ,Xv dt #t=0

(1.9.2)

whereby Xv is regarded as an element of the tangent space Tv (V ) at v. Let α be a representation of G in V and let v ∈ V . As mentioned in Section 1.6, the stability group G(v) = {s ∈ G : α(s)v = v} is closed. The orbit of v is Gv = {α(s)v : s ∈ G}, and we may also denote the orbit of v by O or Ov . The stabilizer is a closed subgroup of G, but not necessarily connected. As also mentioned in Section 1.6 (Result 1.6.9), the orbit Ov = G/G(v) has naturally the structure of an analytic manifold. Elements of Tv (V ) that are tangent to Ov are given by ξvX f

# d ## = f (α(exp −tX)v), X ∈ g. dt #t=0

(1.9.3)

Observe that X → −ξvX is the differential at the identity of the smooth map φv : G → V defined by φv (s) = α(s)v.

58

Basic theory of solvable Lie algebras and Lie groups

Lemma 1.9.4 For each X ∈ g and s ∈ G, one has d(φv )s (X) = −α(s)d α(X)v. In particular ξvX = −dα(X)v, and the kernel of X → ξvX is g(v). Proof: Let ψv : GL(V ) → V be defined by ψv (a) = av, so that φv = ψv ◦ α. Then (1.9.2) says that the differential of ψv at the identity is the map A → Av. More generally, for a ∈ GL(V ), A ∈ gl(V ), we compute that d(ψv )a (A) = aAv, and so by the chain rule, d(φv )s (X) = α(s)dα(X)v. In particular ξvX = −d(ψv )e ◦ dα(X) = −dα(X)v. The adjoint representation Ad : G → GL(g) is characterized by the relation exp Ad(s)X = s exp Xs −1, and one has dAd = ad, and ξYX = [X,Y ]. Given any representation α, and any s ∈ G, we have α(s)α(exp tX)α(s)−1 = α(exp tAd(s)X) holds for all t ∈ R, and hence α(s)dα(X)α(s)−1 = dα(Ad(s)X).

(1.9.4)

We retain most of the conventions adopted for representations of Lie algebras: given a representation α of G in V , V may be regarded as a Gmodule, and we may write α(s)v = sv. If V is a real vector space then we regard Vc as a G-module defined by α. We say that α is irreducible if the only G-invariant subspaces are {0} and V . Lemma 1.9.5 Let v ∈ V . Then the Lie algebra of G(v) is the annihilator g(v) of v in g. Proof: This follows immediately from the relation (1.9.1). Lemma 1.9.6 Let v ∈ V and s ∈ G. Then g(α(s)v) = Ad(s)(g(v)). Proof: Let X ∈ g(v). Then the relation (1.9.4) shows that dα(Ad(s)X)α(s)v = α(s)dα(X)v = 0 so that Ad(s)X belongs to g(α(s)v). Thus Ad(s)(g(v)) ⊂ g(α(s)v), and similarly we find that Ad(s)−1 (g(α(s)v)) ⊂ g(v). Let O be a G-orbit in V , fix v ∈ O, and consider the canonical map φ˜ v : G/G(v) → V with image O defined by φ˜ v (sG(v)) = α(s)v. By Lemma 1.9.4, φ˜ v is an immersion.

1.9 Finite-dimensional group representations

59

Lemma 1.9.7 Let O be a G-orbit in V . Then O is a submanifold of V , and the tangent space at each point v is canonically isomorphic with dα(g)v  g/g(v). Proof: Fix v ∈ O. By Lemma 1.9.4, the differential of φv at each s ∈ G is X → α(s)dα(X)v. Hence the image of d(φv )s is α(s)dα(g)v = dα(g)α(s)v and rank d(φv )s = rank d(φv )e = dim g − dim g(v). It is immediate that the rank of d(φ˜ v )sG(v) is independent of the point s G(v) in G/G(v), and hence O is a submanifold of V. Observe that dim g/ dim g(v) = dim V = n if and only if the orbit O of v  is an open subset of V. Fix a basis (e1, . . . ,en ) in V , with v = i vi ei , and a basis (X1, . . . ,Xm ) in g. If there is an open orbit O in V, then  = {v ∈ V : dim g/g(v) = dim V = n} is the set of v such that the rank of the matrix M(v) = [(Xi v)j ] is maximal, equal to n. It is easily seen that the set of such v is a Zariski open subset of V. Although orbits are submanifolds of V, they are not necessarily regularly embedded. Example 1.9.8 Let G = R and V = C2 . Fix real numbers a and b, both nonzero, and define α(t)(z1,z2 ) = (e2π ait z1,e2π bit z2 ). Observe that the kernel K of α is K = (1/a)Z ∩ (1/b)Z. If the real numbers a and b are linearly dependent over Q, then there are integral numbers p and q such that a = pq b, thus K = lcm(p,q) Z is nontrivial and the orbits of nonzero points v = (z1,z2 ) pb are homeomorphic to circles in V . Otherwise K is trivial, and if both z1 and z2 are not zero, then the orbit O of v = (z1,z2 ) is not a locally compact subset of V, hence φ˜ v is not a homeomorphism onto O. Thus O is not regularly embedded. More generally, let G = Rp act on V = Cn by coordinate-wise rotations: given real forms β1 ,. . . βn on Rp , define α by α(A)(z1, . . . ,zn ) = (eiβ1 (A) z1, . . . ,eiβn (A) zn )

(A ∈ Rp ).

Let dR denote the real dimension of the real vector space spanned by the βj ,1 ≤ j ≤ n, and let dQ be the dimension of the Q-span of the βj . Given z = (z1, . . . ,zn ) ∈ Cn such that z1 . . . zn = 0, it follows from Proposition 2.4  in [8] that the orbit of z is regularly embedded if and only if dR = dQ .

60

Basic theory of solvable Lie algebras and Lie groups

We recall a fundamental result for smooth group actions, characterizing regularly embedded orbits (see for instance Varadarajan [88]). Theorem 1.9.9 Let G be a Lie group acting smoothly on a smooth manifold X, and let O be an orbit in X. For any x ∈ O, let φ˜ x be the canonical mapping of G/G(x) into X with image O. Then the following are equivalent: (i) For some (hence any) x ∈ O, φ˜ x is a regular embedding. (ii) O is locally compact in the relative topology. (iii) For any compact neighborhood K of e in G, there is a neighborhood V of x in X such that O ∩ V ⊂ Kx. Proof: Let π : G → G/G(x) be the canonical projection map. Let m be a supplementary space for g(x) in g. Recall (Result 1.6.9) that any neighborhood U of π(e) ∈ G/G(x) contains the domain of a local chart, of the form π(exp M), where M is a sufficiently small compact neighborhood of 0 in m. Suppose first that (i) holds. Since G/G(x), with the quotient topology is locally compact, and φ˜ x is an homeomoprhism, O is also locally compact for the relative topology. Thus (ii) holds. Suppose now (ii) holds, and let K be a compact neighborhood of e in G. Then K contains a compact neighborhood K  such that K −1 = K  , and K  K  ⊂ K. Since O is locally compact, it is a Baire space. G being second countable, there is a sequence (sn ) in G such that G = ∪n sn K  . Thus O = ∪n sn K  x, and each subset sn K  x is compact in O. Thanks to the Baire Lemma, one of these sets, say sn0 K  x contains an open subset in O: there is an open subset V  in X such that V  ∩ O ⊂ sn0 K  x. Thus, if V  is the open V  , V  ∩ O ⊂ K  x. Let y = kx be a point in V  ∩ O, with k ∈ K  . subset sn−1 0 Put V = k −1 V  , then V ∩ O ⊂ K  K  x ⊂ Kx, and x = k −1 y ∈ V ∩ O. This proves (iii). Suppose finally that (iii) holds. Let U be a neighborhood of π(s) ∈ G/G(x), for the quotient topology. Then s −1 U is a neighborhood of π(e) in G/G(x). Let M be a compact neighborhood of 0 in m such that π(exp M) ⊂ s −1 U . Consider a compact neighborhood K0 of e in G(x), put K = exp MK0 . By (iii), there is an open subset V in X such that x ∈ O ∩ V ⊂ Kx ⊂ φ˜ x (s −1 U ). This means φ˜ x (s −1 U ) is a neighborhood of x = φ˜ x (π(e)), for the relative topology on O ⊂ X. This implies φ˜ x (U ) is a neighborhood of sx = φ˜ x (π(s)) in the relative topology. The map φ˜ x is a bijective bicontinuous map, φ˜ x is a regular embedding. If an orbit O of a smooth action of G on a smooth manifold X satisfies one of the properties of Theorem 1.9.9, we say that the orbit O is regular. It follows

1.9 Finite-dimensional group representations

61

from condition (iii) of Theorem 1.9.9 that an orbit is regular if and only if its manifold topology coincides with the relative topology in V . Let G be a simply connected Lie group with Lie algebra g, let α be a representation of G in a finite-dimensional real vector space V , and let Z be a subspace of V that is G-invariant, with π : V → V /Z the canonical map. Denote the quotient representation by ρ. Fix a supplementary space W for Z in V , identify V /Z with W , and identify the canonical map π as the projection onto W parallel to Z. Let p be the projection onto Z parallel to W , so that π + p is the identity map. Then ρ can be regarded as a representation in W . Now for v ∈ V , write v = w + z where w ∈ W , z ∈ Z. For each s ∈ G, we can write α(s)v in the form α(s)v = α(s)w + α(s)z = ρ(s)w + p(α(s)w) + α(s)z. Fix w ∈ W and put G(w) = {s ∈ G : ρ(s)w = w}. Define ϕw : G(w) → Aff(Z) by ϕw (s)z = α(s)v − w = p(α(s)w) + α(s)z. Since α(s1 )(w+α(s2 )v−w)−w = α(s1 s2 )v−w then ϕw is a homomorphism. Fix v = w + z ∈ V , put O = α(G)v, O0 = ρ(G)π(v), and ω = ϕw (G(π(v)))z. For simplicity of notation we write α(s)v = sv. We have the following result (see [6] for a slightly more general result). Proposition 1.9.10 If O is regular, then ω is regular. On the other hand if O0 and ω are both regular, then O is regular. Proof: Keep the preceding notation, and suppose O regular. Let g(w) be the Lie algebra of G(w) = G(π(v)), and m a supplementary space to g(w) in the Lie algebra g of G. Fix a compact neighborhood M of 0 in m, sufficiently small such that the map M → exp(M)G(w) ⊂ G/G(w) defines a local chart for the manifold G/G(w). Let Kw be any compact neighborhood of e in G(w), and put K = exp (M)Kw . Then Kv ∩ (w + Z) = Kw v = w + ϕw (Kw )z. Indeed, for any X ∈ M, k ∈ Kw , we have x = exp X kv belongs to w + Z if and only if ρ(exp X)w = w, if and only if X = 0. Similarly, O ∩ (w + Z) = G(π(v)) v = w + ω, since sv ∈ w + Z if and only if s ∈ G(π(v)). Since O is regular, there is a neighborhood V of v in X such that O ∩ V ⊂ Kv, thus w + ϕ(Kw )z = Kv ∩ (w + Z) contains O ∩ (w + Z) ∩ V = (w + ω) ∩ V. Since W = p ((w + Z) ∩ V) is a neighborhood of z in Z contained in ϕw (Kw )z, then Theorem 1.9.9 implies that ω is regular.

62

Basic theory of solvable Lie algebras and Lie groups

Now suppose that both O0 and ω are regular. Let K be any compact neighborhood of e in G; K contains a compact neighborhood of e of the form exp(M)Kw (M and Kw are as above). Since ω is regular, we have a neighborhood Z of z in Z such that Z ∩ ω ⊂ ϕw (Kw )z. Define f : m × W × Z −→ Z by f (X,w ,z ) = ϕw (exp X)z . By continuity of f , we have a compact neighborhood M  of 0 in m, a neighborhood W of w in W , and a neighborhood Z  of z in Z such that f (M  × W × Z  ) ⊂ Z. We may assume that M  ⊂ M, and moreover, −M  = M  . Since O0 is regular, we have a neighborhood W  of w in W such that W  ∩ O0 ⊂ ρ(exp M  )w. We can choose W  ⊂ W. We claim that (W  + Z  ) ∩ O ⊆ Kv. Let x = w  + z ∈ (W  + Z  ) ∩ O. Then w ∈ W  ∩ O0 , and so we have X ∈ M  such that w  = ρ(exp X)w. Now (−X,w ,z ) belongs to M  × W  × Z  , and hence exp −Xx = ρ(exp −X)w  + ϕw (exp −X)z = w + ϕw (exp −X)z = w + f (−X,w ,z ) belongs to O ∩ (w + Z). But O ∩ (w + Z) = w + (ω ∩ Z) ⊆ w + ϕw (Kw )z. Hence exp −Xx belongs to w + ϕw (Kw )z and so   x ∈ exp X w + ϕw (Kw )z = exp XKw v ⊆ Kv. Thus the claim is proved. Now by Theorem 1.9.9, O is regular. Example 1.9.11 In this example, we consider the following action of the abelian group G = R2 , identified with its Lie algebra g = spanR {A1,A2 }, acting on the R-vector space V = R4 × C × C, by the matrices:

1.9 Finite-dimensional group representations ⎡

0 ⎢0 ⎢1 ⎢ A1 = ⎢ ⎢0 ⎢ ⎣ ⎡

0 ⎢0 ⎢0 ⎢ A2 = ⎢ ⎢1 ⎢ ⎣

0 0 0 1

0 0 0 0

0 0 0 0

⎤

0 0 0 0

0 0 0 0

⎥ ⎥ ⎥ ⎥, ⎥ ⎥ ⎦

  i 1 − √1 2

0 0 1 0

63

0

⎥ ⎥ ⎥ ⎥. ⎥ ⎥ ⎦

  i 1 + √1 2

⎤

3i

Such an abelian action was studied in detail in [7, 8]. Let us write v(v1, . . . ,v4, r1 y1,r2 y2 ) with ri ≥ 0 and |yi | = 1 and study the regularity of the orbit of a vector v satisfying v1 r1 r2 = 0. Put w = (v1,v2,v3,v4 ), and identify v with (r,w,y) where r = (r1,r2 ), y = (y1,y2 ). Using the coordinates (r,w,y), the G-action is split:  et1 A1 +t2 A2 (r,w,y) = r,(v1,v2,v3 + t1 v1 + t2 v2,v4 + t1 v2 + t2 v1 ), √  × (ei(t1 +t2 )+i/ 2(t2 −t1 ) y1,ei3t2 y2 ) . We apply Proposition 1.9.10: the orbit of (r,w) is regular, and ⎧ ⎪ {1} if v12 − v22 = 0, ⎪ ⎨ G(r,w) = G(w) = eR(A1 −A2 ) if v2 = v1, ⎪ ⎪ ⎩ eR(A1 +A2 ) if v = −v . 2

1

Now consider the G(w)-orbit ω of y: if v12 − v22 = 0, then ω is trivial. If √

v1 = v2 , then ω = {(ei 2t y1,ei3t y2 ) : t ∈ R}, while if v1 = −v2 , then ω = {(ei2t y1,ei3t y2 ) : t ∈ R}. Therefore, if v1 r1 r2 = 0, the orbit of v is regular if  and only if v2 = v1 . Let G be a connected solvable Lie group and let α be a representation of G in a finite-dimensional vector space over R. Suppose that Z is an irreducible subspace of V . By Proposition 1.3.6, dim Z = 1 or 2, and the action of g on Z is given by a weight λ as follows. If dim Z = 1, then fix e = 0 in Z, identify Z with R by ze = z, and we have Xz = λ(X)z for all X ∈ g. If dim Z = 2 choose a basis e1,e2 for Z, and identify Z with C by identifying ae1 + be2 = Re (a + ib)(e1 − ie2 ) with a + ib. Then for z = a + ib, Xz = λ(X)z for all X ∈ g, and since Z is irreducible, λ(X) ∈ / R for some X. In either case

64

Basic theory of solvable Lie algebras and Lie groups sz = (s)z, s ∈ G, z ∈ Z,

where  : G → C\{0} is the multiplicative homomorphism such that d = λ. Fix v = w + z ∈ V . Then ϕw : G(w) → Aff(Z) is given by ϕw (s)z = (s)z + p(sw).

(1.9.5)

Put K = G(w) ∩ ker , and put E = ϕw (G(w)), U = ϕw (K); then U is the abelian normal subgroup of translations in E, which we identify with the additive subgroup p(Kw) of Z. The map s → p(sw) is a continuous homomorphism of K into U with kernel G(v) ∩ K = G(v) ∩ ker , and G(v) ∩ ker  = ker ϕw . The Lie algebra of K is k = g(w) ∩ ker λ. Lemma 1.9.12 Suppose that U = {0} and E = {1}. If s1 , s2 belong to G(w) \ ker , then p(s2 w) p(s1 w) = . 1 − (s1 ) 1 − (s2 ) The point z0 = z0 (w) given by the above formula is the unique fixed point in Z for E. Moreover, the E-orbit ω of a point z = z0 is ω = {(s)(z−z0 )+z0 : s ∈ G(w)}. Proof: Let s1 , s2 ∈ G(w) \ ker . Since U = {0}, then ker ϕw = G(w) ∩ ker  = K and hence ϕw (s1 s2 ) = ϕw (s2 s1 ). It follows that for any z ∈ Z, (s2 )(s1 )z + (s2 )p(s1 w) + p(s2 w) = (s1 )(s2 )z + (s1 )p(s2 w) + p(s1 w) and hence p(s2 w) p(s1 w) = . 1 − (s1 ) 1 − (s2 ) Moreover for any s ∈ G(w) \ ker  we compute that p(sw) + (s)z0 = z0 , and hence ϕw (s)z = (s)z + p(sw) = (s)(z − z0 ) + p(sw) + (s)z0 = (s)(z − z0 ) + z0 . If s ∈ ker  then ϕw (s)z = z so the proof is complete. We say that the representation α is of exponential type if its differential is of exponential type: each generalized weight λ for the representation dα of g is of the form λ = μ(1 + ia) where μ is a real linear form on V . Recall that this is equivalent with the condition that for every X ∈ g, dα(X) has no purely imaginary eigenvalues. Similarly, we say that α is of completely solvable type if each of the generalized weights of dα is real-valued.

1.9 Finite-dimensional group representations

65

Proposition 1.9.13 Let G be a connected solvable Lie group with Lie algebra g, and let α be a representation of G in a finite-dimensional vector space V over R. If α is of exponential type, then every G-orbit in V is regular, and the stabilizer G(v) is connected for every v ∈ V . Proof: We proceed by induction on the dimension of V , the case where V = {0} being trivially true. Fix a nontrivial irreducible subspace Z in V , with associated weight λ, and let π : V → V /Z be the canonical map. Since α is of exponential type, λ = (1 + ia)μ where a ∈ R and μ is a real linear form on g. Fix v ∈ V , write v = w + z as before, and keep the previous notation. By induction, the orbit O0 = Gw = π(Gv) in V /Z is regular, and G(w) is connected. Recall the group homomorphism ϕw : G(w) → Aff(Z): ϕw (s)z = (s)z + p(sw) and U := ϕw (G(w) ∩ ker ). Recall also that ker ϕw = G(v) ∩ ker . In order to show that the orbit Gv is regular, it suffices to show that the affine orbit ω = ϕw (G(w))z is regular (Proposition 1.9.10). We must also show that G(v) is connected. If G(v) = G(w) there is nothing to prove, so we assume that G(v) = G(w). We begin by showing that K = G(w) ∩ ker  is connected. Given s ∈ G(w), write s = exp X1 . . . exp Xr where X1, . . . ,Xr belong to g(w). Suppose that s ∈ K; we have 1 = (s) = (exp X1 ) . . . (exp Xr ) = e(1+ia)μ(X1 +···+Xr ) so μ(X1 + · · · + Xr ) = 0, and hence (exp tX1 ) · · · (exp tXr ) = et (1+ia)μ(X1 +···+Xr ) = 1 holds for all t ∈ R. Thus exp tX1 . . . exp tXr ∈ G(w) ∩ ker  for all t ∈ R and K is connected. Now it follows that ker ϕw = G(v) ∩ ker  is also connected. Indeed, given s ∈ K we may write s = exp X1 . . . exp Xr where now X1, . . . ,Xr belong to the Lie algebra k of K. But for X ∈ k we have X2 w = 0. It follows that p(exp Xw) = p(Xw) and hence p(sw) = p(exp X1 w) + · · · + p(exp Xr w) = p((X1 + · · · + Xr )w). Now if s ∈ G(v) ∩ ker , then p((X1 + · · · + Xr )w) = 0. Hence for all t ∈ R we have p(exp tX1 . . . exp tXr w) = tp((X1 + · · · + Xr )w) = 0 so exp tX1 · · · exp tXr ∈ ker ϕw and ker ϕw is connected. We now consider three cases, corresponding to three distinct affine orbit classes in Z.

66

Basic theory of solvable Lie algebras and Lie groups

(a) Suppose that U = {0}. Here our assumption that G(v) = G(w) implies that G(w) ⊂ ker  and we are in the situation of Lemma 1.9.12. Choose A ∈ g(w) \ ker λ, so that G(w) = K exp RA. By Lemma 1.9.12, the orbit of z is + * ω = etμ(A)(1+ia) (z − z0 ) + z0 which is clearly regular. Since G(v) = K, then G(v) is connected. (b) Suppose that U = {0} and that G(w) ⊂ ker . Here G(w) = K, and G(v) = G(v) ∩ ker  so G(v) = ker ϕw is connected by the above, and ω = z + U . But U = ϕw (K) is connected so ω is an affine subset of Z, and hence regular. (c) Suppose that U = {0} while G(w) ⊂ ker . Choose A ∈ g(w) \ ker λ so that G(w) = K exp RA. We claim that in this case, the connected normal subgroup U of translations is in fact the full group of translations in Z. Let s ∈ G(w). Given X ∈ k, we compute that ϕw (s exp Xs −1 )z = ϕw (s)ϕw (exp X)ϕw (s)−1 z   = ϕw (s) (s −1 )z + p(s −1 w) + p(exp Xw) = z + (s)p(exp Xw) + (s)p(s −1 w) + p(sw). Here, too p(exp Xw) = p(Xw), and z = ϕw (s)(ϕw (s −1 )z) = (s)((s −1 )z + p(s −1 w)) + p(sw), therefore ϕw (s exp Xs −1 )z = z + (s)p(Xw). This implies that (G(w))U ⊆ U . If dim U = 1, this means that  is real but nontrivial on G(w), hence dim Z = 1, and the claim follows. Now ω = Z is regular, and it remains to show that G(v) is connected. Let H1 = (G(v) ∩ ker ) exp RA and E1 = ϕw (H1 ). Then ϕw (H1 ∩ ker ) is trivial, so Lemma 1.9.12 applies to the affine action of H1 : there is z0 ∈ Z such that E1 z0 = {z0 }. Now using a coexponential basis for G(w) modulo H1 belonging to k (see Proposition 1.7.10) we have a subset X of K such that (x,h1 ) → xh1 is a homeomorphism of X × H1 with G(w) and such that ϕw |X is a bijection of X with U . Letting v0 = w +z0 we claim that G(v0 ) = H1 . It is clear that H1 ⊂ G(v0 ), and if s = xh1 ∈ G(v0 ) then ϕw (xh1 )z0 = z0 implies u = ϕw (x) = 0, x = e, and s ∈ H1 . Choosing x0 ∈ X such that ϕw (x0 )z0 = z, we get G(v) = x0 H1 x0−1 . Since H1 is connected, then G(v) is connected also. The following example shows that a nonexponential group may have a faithful representation of exponential type.

1.9 Finite-dimensional group representations

67

Example 1.9.14 Consider a four-dimensional real vector space V , with basis (e1, . . . ,e4 ). Put f1 = e1 − ie2 , f2 = f 1 , f3 = e3 − ie4 , and f4 = f 3 . This defines a basis for U = Vc . Recall the lie algebra e2 , with complex basis (A,Z,Z) and brackets: [A,Z] = iZ, [A,Z] = −iZ, [Z,Z] = 0. The following map α, written as matrices on the basis (f1, . . . f4 ) is a representation of e2 , of exponential type: ⎤ ⎡ a(1 − i/2) ⎥ ⎢ 0 a(1 + i/2) ⎥. α : aA + zZ + zZ → ⎢ ⎦ ⎣ z 0 a(1 + i/2) z 0 a(1 − i/2) 0 This representation gives rise to a faithful representation of exponential type of the simply connected group E˜ 2 .  However, even if G is exponential, a representation which is not of exponential type can have both regular and nonregular orbits. The simplest example is the following: 2 Example 1.9.15 Let  it G = R, V the real vector space C , and α the representation t → e i √0 2t . Let (z1,z2 ) = t [z1,z2 ]. Then 0 e

If z1 z2 = 0, the orbit is a rope with irrational slope on the torus {(w1,w2 ) : |wi | = |zi |}. It is not a regular orbit. If z2 = 0 and z1 = 0, the orbit is the circle {(w,0) : |w| = |z1 |}, and is regular.



2 Stratification of an orbit space

Linear actions of Lie groups on finite-dimensional vector spaces occur broadly in areas of both pure and applied mathematics. Two areas where solvable Lie group actions are particularly important are the orbit method for construction of irreducible unitary representations, and the theory of generalized wavelets. For both areas, the structure of the orbit space for a dual action is of interest. As usual the associated Lie algebra action provides a useful starting point and framework for constructions. In this chapter computational methods are developed in order to study finite-dimensional representations of solvable Lie algebras and Lie groups. A classification of orbits is obtained by means of a matrix-based stratification of the vector space into invariant sets called layers. The orbits in a given layer share common characteristics; nevertheless, for general linear actions of solvable groups there is still much to be learned about the orbit structure of a layer. For some classes of representations however, a great deal is known about the structure of the orbit spaces in each layer. Moreover, the nature of a stratification means that in applications, it is frequently the case that only one of the layers is needed. In particular, one of the layers is always Zariski open, and hence conull for Lebesgue measure. One class of representations for which much is known is the class of representations that are of exponential type. Given such a representation, we construct analytic functions by which, for a given layer, all orbits are simultaneously parametrized, and an explicit topological cross-section for the orbits is defined. We first work out the construction for the completely solvable case, since this case arises naturally in applications and avoids certain technical complications. For the Zariski open layer, we define a natural measure on the cross-section, as well as semi-invariant measures on the orbits, by which Lebesgue measure is then decomposed. We also use examples to illustrate

68

2.1 Matrix normal form

69

recent results for another class, wherein the group in question is only a vector group, but the action is no longer assumed to be of exponential type. The constructions in this chapter are then seen to have applications both to the coadjoint action and to generalized wavelets. Specifically, we apply these results to the theory of polarization in Chapter 4, and to Plancherel theory in Chapter 6. We also give examples in Chapter 3 illustrating the application of these results to generalized wavelets where the action is of exponential type.

2.1 Matrix normal form Let M and M  be n × m matrices with entries in k = R or C. Recall that M and M  are said to be equivalent if there are matrices S ∈ GL(n,k) and T ∈ GL(m,k) such that M  = S −1 MT . The equivalence class of M is characterized by its rank: if d = rankM, then M is equivalent with the normal form matrix   Id 0 Nd = . (2.1.1) 0 0 A more restricted notion of equivalence is of interest here. Definition 2.1.1 Let M and M  be n × m matrices; M and M  are said to be flag equivalent if there are S ∈ GL(n,k) and T ∈ GL(m,k) for which M  = S −1 MT , and where S is lower triangular, and T is upper triangular. The matrices M and M  are said to be strictly flag equivalent if they are flag equivalent, and if the triangular matrices S and T can be chosen to be unipotent. It is clear that flag equivalence and strict flag equivalence are equivalence relations. Our first task will be to describe equivalence classes for flag and strict flag equivalence. We begin with some notation. Let M be an n × m matrix with entries in k, and for subsets i ⊂ {1, . . . ,n} and j ⊂ {1, . . . ,m}, denote the submatrix consisting only of the rows indexed by elements of i by Mi , and the submatrix consisting only of the columns j indexed by elements of j by M j . Write Mi = (Mi )j = (M j )i . If R is any n × n j matrix, then (RM)i = Ri M and (RM) = RM j . Similar relations hold when multiplying M on the right by an m × m matrix. If i and j are written as i = j j ,j ,...,j {i1,i2, . . . ,ir }, j = {j1,j2, . . . ,js }, then we may also write Mi = Mi11,i22,...,ir s . In particular for any 1 ≤ i ≤ n and 1 ≤ j ≤ m, Mi is the i-th row of 1,...j M, M j is the j -th column of M, and M1,...,i is the principal i × j submatrix of M. A relation of particular importance is obtained when considering flag

70

Stratification of an orbit space

equivalence. Let S ∈ GL(n,k) be lower triangular, and T ∈ GL(m,k) upper triangular. An elementary computation shows that for any 1 ≤ i ≤ n and 1 ≤ j ≤ m, (S −1 MT )1,...,i = (S −1 )1,...,i 1,...,i M1,...,i T1,...,j . 1,...,j

1,...,j

1,...,j

(2.1.2)

Hence if matrices M and N are flag equivalent (resp. strictly flag equivalent) 1,...,j 1,...,j then M1,...,i and N1,...,i are flag equivalent (resp. strictly flag equivalent). Now define the set e = e(M) of jump indices for the matrix M: e = e(M) ⊂ {1,2, . . . ,n} is defined by e = {1 ≤ i ≤ n : rank M1,2,...,i−1 < rank M1,2,...,i }. Observe that for any lower-triangular S ∈ GL(n,k) and upper-triangular T ∈ GL(m,k), and for each 1 ≤ i ≤ n, the relation (2.1.2) shows that (S −1 MT )1,...,i = (S −1 )1,...,i 1,...,i M1,...,i T has the same row space as M1,...,i T so rank (S −1 MT )1,...,i = rank M1,...,i T = rank M1,...,i . Hence e(S −1 MT ) = e(M). Now for e ∈ e define h(e) ∈ {1,2, . . . ,m} by * + 1,...,j 1,...,j h(e) = min 1 ≤ j ≤ m : rank M1,...,e−1 < rank M1,...,e . Since rank (S −1 MT )1,...,i = rank (S −1 )1,...,i 1,...,i M1,...,i T1,...,j = rank M1,...,i 1,...,j

1,...,j

1,...,j

1,...,j

holds for any i and j , then it is clear that the function h : e → {1,2, . . . ,m} is also an invariant for the relation of flag equivalence. The function h is called the jump index function for M. It will be shown that the sets e and functions h are sufficient to characterize flag equivalence classes. Let e be any nonempty subset of {1, . . . ,n} and let h : e → {1, . . . ,m} be a one-to-one function. Let N (e,h) = [bi,j ] be defined by bi,j = 1 if i ∈ e and j = h(i), and bi,j = 0 otherwise. A matrix of the form N (e,h) for some e and h is said to have flag normal form. The matrix N (e,h) can be viewed as a sort of graph of the map h. For instance with n = m = 4, if e = {1,2,4} and h(1) = 3, h(2) = 1, h(4) = 4, then ⎡ ⎤ 0 0 1 0 ⎢1 0 0 0⎥ ⎥ N (e,h) = ⎢ (2.1.3) ⎣0 0 0 0⎦ . 0

0

0

1

Note also that the matrix Nd of (2.1.1) has flag normal form: Nd = N (e,h) where e = {1,2, . . . ,d} and h(e) = e, 1 ≤ e ≤ d.

2.1 Matrix normal form

71

Now let c : e → k \ {0} be any function and D(e,c) = [di,j ] the n × n / e. A matrix diagonal matrix defined by di,i = c(i) if i ∈ e and di,i = 1, if i ∈ N(e,h,c) of the form N (e,h,c) = D(e,c)N(e,h) is said to be in strict flag normal form. For example, if e and h are as above and c(1) = 5, c(2) = 3, c(4) = 8, then ⎡ ⎤ 0 0 5 0 ⎢3 0 0 0⎥ ⎥ N (e,h,c) = ⎢ (2.1.4) ⎣0 0 0 0⎦ . 0 0 0 8 It will be shown that every matrix is flag equivalent with a matrix in flag normal form, and strictly flag equivalent with a matrix in strict flag normal form. Before doing this, a few more words about notation, and an example. First, if e = ∅ it is natural to write the subset e as an increasing sequence: e = {e1 < e2 < · · · < ed }. Then the functions h and c are written as a sequences: h = (h1, . . . ,hd ) where hi = h(ei ), c = (c1, . . . ,cd ) where ci = c(ei ). If e = ∅ then of course h and c are undefined and M = N (e,h) = N (e,h,c) = 0. The case where e = ∅ must be considered later, whence we shall have occasion to write h = (). Second, given an n × m matrix M, we use unipotent row and column operations in order to construct the triangular matrices S and T : given a unipotent m × m upper-triangular matrix T , the matrix MT is the result of applying a sequence of elementary unipotent column operations of the form

yl,j M l . M j −→ M j + l e1

and put M(1) = S1−1 MT1 = [a(1)i,j ]. Observe that a(1)e1,h1 = ae1,h1 = 0, and that a(1)e1,h1 is the only nonzero entry in both the row M(1)e1 and the column M(1)h1 . Observe also that each of the entries a(1)i,j with (i,j ) = (e1,h1 ) satisfy the relation a(1)i,j =

ai,j ae1,h1 − ai,h1 ae1,j . ae1,h1

Now suppose that i > e1 and that Mi is a multiple of Me1 . Then Mi is a multiple of Me 1 . Since the row operations insure that a(1)i,h1 = 0 for i > e1 , then M(1)i is identically zero. Hence if d = rank M = 1, then M(1)i is identically zero for all i > e1 , and M(1) = N ({e1 }, (h1 ),c) with c(e1 ) = a(1)e1,h1 . If rank M > 1, then M(1)e2 is the first nonzero row of M(1) after M(1)e1 . Let h2 = h(e2 ) be the column index of the first nonzero entry in M(1)e2 . Step 2: Let k > 1, and suppose that hr = h(er ) is defined for 1 ≤ r < k, and that unipotent upper-triangular m × m matrices T1, . . . ,Tk−1 , and unipotent lower-triangular matrices S1, . . . ,Sk−1 , are defined, so that the matrix M(k − 1) = [a(k − 1)i,j ] defined by −1 M(k − 1) = Sk−1 . . . S1−1 MT1 . . . Tk−1

74

Stratification of an orbit space

has the following properties: (1) hr = hs for r = s, 1 ≤ r,s < k. (2) For each 1 ≤ r < k, a(k − 1)er ,hr is the only nonzero entry in both the row M(k − 1)er and the column M(k − 1)hr . (3) If 1 ≤ j ≤ m and j ∈ / {h1, . . . ,hk−1 }, then M(k − 1)j = (MT1 . . . Tk−1 )j . (4) If i ∈ / {e1 < e2 < · · · < ek−1 } and the row Mi is dependent upon the rows Mer , 1 ≤ r ≤ ek−1 , then the row M(k − 1)i is identically zero. If rank M = k − 1, then it is immediate from the above conditions that M(k − 1) = N (e,h,c) with c(r) = a(k − 1)er ,hr , 1 ≤ r ≤ k − 1. Suppose that rank M > k − 1; then M(k − 1)ek is the first nonzero row of M(k − 1) below row M(k − 1)ek−1 . Let hk be the column index of the first nonvanishing entry in this row. By conditions (2) and (4), / {h1, . . . ,hk−1 }. Now proceed exactly as in Step 1: let Tk be the hk ∈ m × m unipotent upper-triangular matrix for which multiplication on the right side of M by Tk is exactly the set of column operations M(k − 1)j → M  (k − 1)j = M(k − 1)j a(k − 1)ek ,j − M(k − 1)hk , j > hk , a(k − 1)ek ,hk and put M  (k − 1) = M(k − 1)Tk . Now the only nonzero entry of the row M  (k − 1)ek is a  (k − 1)ek ,hk = a(k − 1)ek ,hk . Then let Sk−1 be the n × n unipotent lower-triangular matrix for which multiplication on the left side of M by Sk−1 effects the set of row operations M  (k − 1)i → M(k)i = M  (k − 1)i −

a  (k − 1)i,hk  M (k − 1)ek , i > ek a  (k − 1)ek ,hk

and put M(k) = Sk−1 M(k − 1)Tk . Now it is evident that for all 1 ≤ r, s ≤ k, r = s implies hr = hs , and that a(k)ek ,hk = a(k − 1)ek ,hk is the only nonzero entry in both the row M(k)ek and the column / {h1, . . . ,hk }, then M(k)hk . Thus (1) and (2) hold for M(k). If j ∈ a  (k − 1)er ,j = 0 for all 1 ≤ r ≤ k, so M(k)j = (MT1 T2 · · · Tk )j , and (3) holds for M(k). Now suppose that i > ek and that the row Mi is dependent upon the  / {h1, . . . , hk }, rows Me1 , . . . ,Mek ; write Mi = kr=1 xi,r Mer . If j ∈ then using properties (2) and (3) for 1 ≤ r ≤ k,

2.1 Matrix normal form

a(k)i,j = a  (k − 1)i,j = Mi (T1 · · · Tk )j =

75 k

xi,r Mer (T1 · · · Tk )j

r=1

=

k

xi,r a  (k − 1)er ,j = 0.

r=1

If j = hr for some 1 ≤ r ≤ k, then a(k)i,hr = 0 by property (1). Thus the row M(k)i is zero and property (4) holds for M(k). This process continues until k = d = rank M, whence M(d) has the strict normal form N (e,h,c) where c(r) = a(d)er ,hr , 1 ≤ r ≤ d. Moreover, N (e,h) = D(e,c)−1 N (e,h,c) = (SD(e,c))−1 MT , is flag equivalent to M, and has a flag normal form. Since e and h are invariants for the relation of flag equivalence, then we have shown that every flag equivalence class contains one and only one flag normal form matrix. Finally, suppose that N (e,h,c) and N (e,h,c ) are strictly flag equivalent, so that for some unipotent lower-triangular matrix S and unipotent upper-triangular matrix T , SN (e,h,c) = N (e,h,c )T . Computing the entry at row e and column h(e) of SN (e,h,c) = N (e,h,c )T , we immediately get c(e) = c (e). Thus c = c and N (e,h,c) = N (e,h,c ). The following is almost immediate. For an n × m matrix M, put nullspace(M) = {v ∈ km : Mv = 0}, and col(M) = span{M j : 1 ≤ j ≤ m}. Lemma 2.1.4 Let M be an n × m matrix and let S and T be the triangular matrices constructed in Proposition 2.1.3, so that S −1 MT = N (e,h). For a / J } and for a subset I subset J of {1,2, . . . ,m} let EJ = {v ∈ km : vj = 0,j ∈ n of {1,2, . . . ,n} define FI ⊆ k similarly. Then we have the following. (1) nullspace(M) = span{T j : j ∈ / h(e)}, and Eh(e) is a supplementary subspace for nullspace(M) in km . Moreover, the projection of km onto nullspace(M) parallel to Eh(e) is given by the matrix Tnull obtained by replacing the columns T h , h ∈ h(e) by 0. (2) col(M) = span{S e : e ∈ e}, and Fec is a supplementary space for col(M) in F . Moreover, the projection of kn onto col(M) parallel to Fec is given / e by 0. by the matrix Scol obtained by replacing the columns S j , j ∈ Proof: We prove (1), the proof of (2) being very similar. The relation nullspace(M) = span{T j : j ∈ / h(e)}

76

Stratification of an orbit space

is immediate, since nullspace(N (e,h,c)) = nullspace(MT ) = Eh(e)c . Now the kernel of the linear transformation of km given by Tnull clearly contains Eh(e) , and maps Eh(e)c onto nullspace(M). Now observe that T = 1 + T  where T  is strictly upper triangular and the only nonzero columns of T  are the columns indexed by h(e). Thus Tnull is the matrix of a projection and Part (1) follows. Similarly MT = SN (e,h), thus col(M) = col(MT ) = col(SN (e,h)), and (2) follows. 1,...,j

The preceding can be applied to the submatrices M1,...,i . Details are left to the reader. Corollary 2.1.5 Let M be an n × m matrix and let S and T be the triangular matrices constructed in Proposition 2.1.3, with N (e,h) and N (e,h,c) the flag normal and strict flag normal forms for M. Write e = {e1 < e2 < · · · < ed }, with h = (h1, . . . ,hd ), c = (c1, . . . ,cd ). Fix i, 1 ≤ i ≤ n, and j , 1 ≤ j ≤ m. 1,...,j Consider the procedure of Proposition 2.1.3 applied to the i ×j matrix M1,...,i , and let S(i,j ) and T (i,j ) be the triangular matrices constructed thereby. We have the following. 1,...,j

1,...,j

(1) The index set e(i,j ) = e(M1,...,i ) for M1,...,i is e(i,j ) = {e ∈ e : e ≤ i,h(e) ≤ j }. Writing 1,...,j e(i,j ) = {es1 < es2 < · · · < esr }, the column map for M1,...,i is 1,...,j

(hs1 , . . . ,hsr ), and the entries of the strict normal form of M1,...,i are given by (cs1 , . . . ,csr ). 1,...,j (2) S(i,j ) = (Ss1 · · · Ssr )1,...,i 1,...,i and T (i,j ) = (Ts1 · · · Tsr )1,...,j . 1,...,j

(3) The strict flag normal form for M1,...,i is S(i,j )−1 M1,...,i T (i,j ) = N (e,h,c)1,...,i 1,...,j

1,...,j

= N ({es1 , . . . ,esr },(hs1 , . . . ,hsr ),(cs1 , . . . ,csr )). 1,...,j

(4) The flag normal form for M1,...,i is −1 ((SD(e(i,j ),c(i,j )))1,2,...,i 1,...,i ) M1,...,i T (i,j ) = N (e,h)1,...,i 1,...,j

1,...,j

= N ({es1 ,es2 , . . . ,esr },(hs1 , . . . ,hsr )). 1,...,j

1,...,j

Note that nullspace(M1,...,i ) and col(M1,...,i ) are given as in Lemma 2.1.4. Remark 2.1.6 The entries c1, . . . ,cd of the strict normal form appear already in the matrix MT . Specifically, let M be a n × m matrix and let S and T be the triangular matrices constructed in Proposition 2.1.3, so that N (e,h,c) = S −1 MT is the strict flag normal form matrix equivalent to M. Write the matrix MT = SN (e,h,c) = [bi,j ], S = [si,j ] and N (e,h,c) = [zi,j ]. Then

2.1 Matrix normal form

be,h(e) =

77

se,j zj,h(e) = ze,h(e) = c(e).

j h(e)

Observe also that for each e ∈ e, the column M1,...,e is the first column which is not in the span of the columns indexed by h(e ), e < e, that is h(e )

h(e) = min{j : M1,...,e ∈ / spank {M1,...,e : e < e}}. j

The following example shows that there is more than one way to obtain the strict flag normal form of a matrix. Example 2.1.7 In Example 2.1.2, the matrix ⎡

1 ⎢2 M=⎢ ⎣3 3

1 2 0 3

⎤ 3 5⎥ ⎥ 1⎦ 8

is shown to have e = {1,2,3}, h = (1,3,2), and c = (1, −1, −3). Instead of the operations presented there, begin with row operations and follow by column operations: ⎡

1 ⎢2 ⎢ ⎣3 3

1 2 0 3

⎤ 3 op 5⎥ ⎥ row  → − 1⎦ 8 col op

−→

⎡

⎤ 1 1 3 ⎢0 0 −1⎥ row op ⎢ ⎥ ⎣0 −3 −8⎦ −→ 0 0 −1 ⎡ ⎤ 1 0 0 ⎢0 0 −1⎥ col op ⎢ ⎥  → ⎣0 −3 −8⎦ − 0 0 0

The result is N(e,h,c) = S˜ −1 M T˜ where ⎡ ⎤⎡ 1 0 0 0 1 0 0 ⎢ ⎥ ⎢ 0 1 0 0 ⎥ ⎢−2 1 0 S˜ −1 = ⎢ ⎣0 0 1 0 ⎦ ⎣−3 0 1 0 −1 0 1 −3 0 0

⎡

1 ⎢0 ⎢ ⎣0 0 ⎡ 1 ⎢0 ⎢ ⎣0 0

⎤ ⎡ 0 1 ⎥ ⎢ 0 ⎥ ⎢−2 = 0 ⎦ ⎣−3 1 −1

⎤ 1 3 0 −1⎥ ⎥ −3 −8⎦ 0 0 ⎤ 0 0 0 −1⎥ ⎥. −3 0 ⎦ 0 0

⎤ 0 0 0 1 0 0 ⎥ ⎥ 0 1 0 ⎦ −1 0 1

and ⎡

1 ˜ ⎣ T = 0 0

−1 1 0

⎤⎡ 1 −3 ⎦ ⎣ 0 0 1 0

⎤ ⎤ ⎡ 1 −1 − 13 0 0 1 − 83 ⎦ = ⎣0 1 − 83 ⎦ . 0 1 0 0 1



78

Stratification of an orbit space

In the rest of this chapter, given an n × m matrix M, we exclusively use the triangular matrices constructed in Proposition 2.1.3, writing S = S(M) and T = T (M). We say that these matrices are the normalizing matrices for M. Fix positive integers n and m and denote by kn×m the vector space of all n×m matrices with entries in k. For e ⊂ {1,2, . . . ,n} and h : e → {1,2, . . . ,m} denote by Me,h the associated flag equivalence class of n × m matrices. Let L be the set of all pairs (e,h), with e ⊆ {1, . . . ,n} and h : e → {1, . . . ,m} injective. The partition {Me,h : (e,h) ∈ L} has a useful algebraic structure described as follows. Let V be a finite-dimensional vector space over k and let S be a finite partition of V . Suppose that there is a well-ordering ≺ on S, and for each  ∈ S, a complex polynomial function P : V → C, such that  = {v ∈ V : P (v) = 0, and P (v) = 0 if  ≺ }. Then S will be called a stratification of V . Elements  of a stratification will be called layers. Observe that if S is a stratification, then each element  of S is a semialgebraic set, and  ∪ (∪{ ∈ S :  ≺ }) is a Zariski open set in V . In particular, the minimal element of S is dense and open in V . Example 2.1.8 Let V = R2 . Define  = {v : v2 = 0},  = {v : v2 = 0, v1 = 0}, = {(0,0)}. With P (v) = v2 , P (v) = v12 + v22 and P = 1, then S = {,, } with the ordering  ≺  ≺  is a stratification  of R2 . We now show that the partition {Me,h : (e,h) ∈ L} is a stratification of kn×m . Fix (e,h) ∈ L with e = ∅, write e = {e1 < e2 < · · · < ed } and h = (h1,h2, . . . ,hd ) where ht = h(et ), 1 ≤ t ≤ d. For each 1 ≤ r ≤ d denote by ε(h1, . . . ,hr ) the sign of the permutation placing the ht in increasing order ,...,hr h1,...,hr and Peh11,...,e r (M) = ε(h1, . . . ,hr ) det Me1,...,er (that is the determinant of the h1,...,hr matrix deduced from Me1,...,er by putting first the h1 -column of Me1,...,er , then the h2 -column, and so on) and define ,...,hd Pe,h (M) = Peh11 (M)Peh11,e,h22 (M) · · · Peh11,...,e (M). d

If e = ∅, put Pe,h = 1. Define a natural total ordering on L as follows. Let (e,h) and (e,h ) be two pairs in L, e = {e1 < · · · < ed }, e = {e1 < · · · < ed  }, h = (h1, . . . ,hd ), and h = (h1, . . . ,hd  ). Write e d  , or 2. d = d  and e < e for the lexicographic ordering.

2.1 Matrix normal form

79

Then write (e,h) ≺ (e,h ) if 1. e ek , j ∈ / {h1, . . . ,hk }, ⎪ ⎪ ,...,hk ⎪ Peh1,...,e ⎪ k 1 ⎪ ⎪ ⎨ ,...,hr a(k)i,j = Peh11,...,e r ⎪ if (i,j ) = (er ,hr ) (r ≤ k), ⎪ ⎪ h1,...,hr−1 ⎪ P ⎪ e ,...,e 1 r−1 ⎪ ⎪ ⎪ ⎪ ⎩ 0 otherwise.

(2.1.7)

/ {h1, . . . ,hk }. As in the proof of Lemma 2.1.9, we Suppose first i > ek and j ∈ have:

(MT1 . . . Tk )j = M j + xhr ,j M hr . hr ek and j = hr (r ≤ k). But by construction of M(k), the corresponding entries of M(k) are zero. Now the entries of each Tk = Tk (M) are determined by the entries of the M(k): we saw that the nonzero off-diagonal entries of Tk are h ,...,h

(Tk )hk ,j = −

,j

a(k − 1)ek ,j Pe 1,...,ek k−1 = − 1 h ,...,h . a(k − 1)ek ,hk Pe11,...,ek k

(2.1.8)

The function M → T is thus a rational function, regular on Me,h . The same proof holds for M → S −1 , and M → S. Finally to get N (e,h), we multiply S −1 MT by the diagonal matrix D −1 (e,c) whose diagonal entries are 1 or: cr−1 = a(d)−1 er ,hr =

h ,...,h

r−1 Pe11,...,er−1

,...,hr Peh11,...,e r

.

The function M → D(e,c)S is a rational function, regular on Me,h .

84

Stratification of an orbit space

An examination of the proofs of Propositions 2.1.3 and 2.1.12, especially the relations (2.1.7) and (2.1.8), shows the following. Corollary 2.1.13 Let M → T (M) be the rational function on Me,h defined in Proposition 2.1.12 and write T (M) = T1 (M)T2 (M) · · · Td (M) as in the construction of Proposition 2.1.3. For each 1 ≤ r ≤ d, T (M)hr is a vector in nullspace(M1,...,er−1 ), but not in nullspace(M1,...,er ). Moreover, for r ≤ k, Tk (M)hr is the hr -canonical basis vector in km . Therefore T (M)hr = (T1 (M) . . . Tr−1 (M))hr h ,...,h

r−1 and M → T (M)hr depends only upon the entries of the matrix Me11,...,er−1 .

In the next and later sections, the preceding results will be applied to linear actions of solvable Lie groups in vector spaces. We make some preliminary observations here. Let E and F be vector spaces over k of dimension m and n, respectively. Fix an ascending flag {0} = E0 ⊂ E1 ⊂ · · · ⊂ Em = E in E and a descending flag F = F0 ⊃ F1 ⊃ · · · ⊃ Fn = {0} in F . Fix bases BE = {Zj : 1 ≤ j ≤ m} for E and BF = {fi : 1 ≤ i ≤ n} in F , adapted to the chosen flags. Given an n × m matrix M with entries in k, recall the matrices S(M) and T (M), so that T (M) is unipotent upper triangular and depends rationally on the entries of M, S is lower triangular and depends rationally on the entries of M, and S(M)−1 MT (M) = N (e,h). Then, using the basis BE , T (M) determines a linear map from E to E, and we denote this map also by T (M). Similarly, M itself determines a linear map from E to F , denote this map also by M. Moreover, let πi : F → F /Fi be the natural map for each 1 ≤ i ≤ n. Note that the matrix of πi ◦ M is M1,...,i . The preceding corollary is restated as follows. Corollary 2.1.14 For each 1 ≤ r ≤ d, T (M)Zhr belongs to (ker M1,...,er−1 ) \ (ker M1,...,er ). Moreover, Tk (M)Zhr = Zhr for r ≤ k. Hence T (M)Zhr h ,...,hr−1 depends only upon the entries of the matrix Me11,...,er−1 .

2.2 Layers for a representation Let β be a finite-dimensional representation of a solvable Lie algebra g in a vector space V over R. The methods of Section 2.1 are applied to construct

2.2 Layers for a representation

85

a stratification of V naturally associated to β. Suppose that G is a connected solvable Lie group with Lie algebra g, and β = dα where α is a representation of G in V . Then the layers of this stratification are α(G)-invariant, and thus provide a classification of the α(G)-orbits As before, let U = Vc be the complexification of V , s = gc the complexification of g, and extend β to a representation of s in U . In applications, α will actually be a dual representation, in particular the coadjoint representation, and for this reason we use descending flags of g-modules. As before we use module notation, writing Zv for dα(Z)v, Z ∈ s, v ∈ U . Fix a descending flag (Ui )0≤i≤n of G-submodules in U , and a Jordan– Hölder sequence (sj )0≤j ≤m for s, such that for some 1 ≤ p ≤ m, sp = [s,s]. Fix a good descending basis (fi )1≤i≤n for U , with corresponding weight sequence (λi ), so that for each 0 ≤ i < n Ui = span{fi+1, . . . ,fn }. Fix a Jordan–Hölder basis (Zj )1≤j ≤m for s, so that for each 1 ≤ j ≤ m, sj = span{Z1, . . . ,Zj }. With these bases fixed, identify U with Cn and s with Cm . For v = t [v1,v2, . . . ,vn ] ∈ U write pi (v) = vi , 1 ≤ i ≤ n. Given v ∈ U , define the n × m action matrix M(v) at v by 

M(v) = pi (Zj v) 1≤i≤n, 1≤j ≤m . Then M(v) is the matrix for the linear map Z → Zv from s to U : for  Z= m j =1 zj Zj ∈ s, ⎡

⎤ z1 ⎢ ⎥ Zv = M(v) ⎣ ... ⎦ . zm Denote the quotient representation acting in U/Ui by βi : βi (Z)πi (v) = πi (β(Z)v); thus U/Ui has a natural module structure and πi : U → U/Ui is a module homomorphism. As a vector space, U/Ui is identified with Ci via the ordered basis (f1 + Ui , . . . ,fi + Ui ), and similarly, it is identified with the subspace Wic = span{f1, . . . ,fi } of U . (The superscript c is just to recall that, contrary to the W defined in Section 1.9, here we consider complex vector spaces.) Note that if i  < i, then Wic is a subspace of Wic . With these identifications, πi becomes simply the map t [v1, . . . ,vn ] → t [v1, . . . ,vi ],

86

Stratification of an orbit space

and βi (Z)πi (v) = πi (Zv). Thereby we can write πi (v) = pi (v)fi + t [v1, . . . , vi−1,0] = pi (v)fi + wi−1 and, as in Relation (1.9.5), pi (Zv) = pi (βi (Z)πi (v)) = λi (Z)vi + pi (βi (Z)wi−1 ).  Observe that for Z = m j =1 zj Zj ∈ s, ⎡ ⎤ z1 ⎢ .. ⎥ βi (Z)πi (v) = πi (Zv) = M(v)1,...,i ⎣ . ⎦ .

(2.2.1)

(2.2.2)

zm Thus the matrix for the map Z → πi (Zv) is given by M(v)1,...,i . More generally, observe that the restriction of the map Z → πi (Zv) to sj is given 1,...,j by the matrix M(v)1,...,i . We write 1,...,j

M(πi (v)) = M(v)1,...,i , and M(πi (v),sj ) = M(v)1,...,i . The kernel of the map Z → βi (Z)πi (v) is the nullspace of M(v)1,...,i and is denoted by s(πi (v)): s(πi (v)) = nullspaceM(v)1,...,i = {Z ∈ s : βi (Z)πi (v) = πi (Zv) = 0}. Observe that s(πi (v)) is a subalgebra of s for each i: given Z,W ∈ s(πi (v)), π(Zv) = π(W v) = 0, so πi ([Z,W ]v) = βi (Z)πi (W v) − βi (W )πi (Zv) = 0. We have s = s(π0 (v)) ⊇ s(π1 (v)) ⊇ · · · ⊇ s(πn (v)) := s(v) and it is clear that for each i, dim s(πi−1 (v))/s(πi (v)) is zero or one. Now observe that when Ui is real, then U/Ui is the complexification of V /Vi , with Vi = Ui ∩ V . One has the following. Lemma 2.2.1 Let 1 ≤ i ≤ n and suppose that v ∈ V . If Ui is real, then s(πi (v)) is also real, and is the complexification of the Lie algebra g(πi (v)) of the stabilizer G(πi (v)) of πi (v) in G. Proof: Let Z ∈ s(πi (v)) and write Z = Z1 + iZ2 with Z1,Z2 ∈ g. Then both πi (Z1 v) and πi (Z2 v) belong to V /Vi , and the relation 0 = βi (Z)πi (v) = πi (Z1 v) + iπi (Z2 v) in U/Ui implies that πi (Z1 v) = πi (Z2 v) = 0 so Z1 and Z2 belong to g(πi (v)). Now confer Lemma 1.2.1.

2.2 Layers for a representation

87

Continue to fix the good Jordan–Hölder basis (Z1, . . . .,Zm ) and the good descending basis (f1, . . . ,fn ) for U , refer to the notations of Section 2.1. Put Cn×m (U ) = {M(v) : v ∈ U } and Cn×m (V ) = {M(v) : v ∈ V }. The set of jump indices for v is e = e(v) = e(M(v)), and the jump index function for v is the jump index function h for M(v). Perform the procedure of Proposition 2.1.3 to obtain the unipotent upper-triangular matrix T (v) = T (M(v)) and S(v) = S(M(v)), so that M(v) is flag equivalent with a flag normal form matrix S(v)−1 M(v)T (v) = N (e,h). The matrices T (v), S(v) are the normalizing matrices at the point v. Observe that e(v) = ∅ if and only if M(v) = 0, if and only if v is annihilated by g, or is a fixed point for G. Assume e = ∅ and write e = {e1 < · · · < ed }, and (h1, . . . ,hd ) as before, for each 1 ≤ i ≤ n write also e(i) = e ∩ {1,2, . . . ,i} = {e1,e2, . . . ,er }. Referring to Corollaries 2.1.13 and 2.1.14, we have s(πi (v)) = span{T (v)j : j ∈ / h(e(i))} = span{T (v)Zj : j ∈ / h(e(i))}; (2.2.3) in particular s(v) = span{T (v)Zj : j ∈ / h(e)}.

(2.2.4)

Then we have the following. Lemma 2.2.2 Let v ∈ U and let e be the jump index set for v and h the jump index function for v, so that M(v) ∈ Me,h . Then (1) e = {1 ≤ i ≤ n : s(πi−1 (v)) = s(πi (v))}, and (2) for each e ∈ e, h(e) = min{1 ≤ j ≤ m : sj ∩ s(πe−1 (v)) ⊂ s(πe (v))}.   Proof: For each 1 ≤ i ≤ n, i ∈ e means that rank M(v) 1,...,i−1    =  is equivalent with nullspace M(v)1,...,i−1 = rank M(v)   1,...,i . This nullspace M(v)1,...,i , that is s(πi−1 (v)) = s(πi (v)). This proves (1). Now for each 1 ≤ i ≤ n and 1 ≤ j ≤ m, the kernel of the map Z → πi (Zv) restricted to sj is   1,...,j sj ∩ s(πi (v)) = nullspace M(v)1,...,i . Given e ∈ e, then j = h(e) means that j is the minimal column index for which   1,...,j  1,...,j  rank M(v)1,...,e−1 = rank M(v)1,...,e .   1,...,j  1,...,j  Since this is equivalent with nullspace M(v)1,...,e−1 = nullspace M(v)1,...,e , then (2) follows.

88

Stratification of an orbit space

The flag equivalence classes Me,h naturally determine subsets U e,h of U . For each (e,h) ∈ L, put U e,h = {v ∈ U : M(v) ∈ Me,h }

and

e,h = U e,h ∩ V .

The sets U e,h depend upon the structure of U as a G-module, that is, upon n×m (U ) ∩ the given representation α; in particular U e,h = ∅ if and only if C Me,h = ∅. Put L(U ) = {(e,h) : U e,h = ∅}

and

L(V ) = {(e,h) : e,h = ∅}.

Observe that if G acts trivially on V = {0} then L = L(U ) = L(V ) = {(∅,())}. The following simple example shows that L(U ) = L(V ) is also possible. Example 2.2.3 Let V = R2 , and A a 2 × 2 real matrix, diagonalizable over C with complex eigenvalues λ, λ, and let G = exp RA. Choose f1 so that Af1 = λf1 , and f2 = f 1 . With A as the basis of g, then for v ∈ U ,   λv1 M(v) = , λv2 and v ∈ V means v2 = v 1 . Now L(V ) consists of two index pairs: ({1},(1)) and (∅,()) with {1},(1) = {v ∈ V : v = 0},

∅,() = {0}.

Therefore M{2},(1) ∩ Cn×m (V ) = ∅. But if v = t [0,v2 ], with v2 = 0, then  M(v) ∈ M{2},(1) ∩ Cn×m (U ). So in this example, L = L(U ) = L(V ). Remark that if the action is of completely solvable type, then we can choose the basis (Zj ) to be real. In that case L(U ) = L(V ). Let L(U ) and L(V ) inherit the well-ordering ≺ defined for L in Section 2.1. For each (e,h) ∈ L put Pe,h (v) = Pe,h (M(v)). It is a polynomial function, and   by definition of L(U ), if v ∈ U e,h and (e ,h ) ≺ (e,h), then Pe,h (v) = 0 and Pe,h (v) = 0. We have the following. Proposition 2.2.4 The partition {U e,h : (e,h) ∈ L(U )}, with the ordering ≺ inherited from that of L, and with the polynomials Pe,h (v), (e,h) ∈ L(U ), is a stratification of U . Moreover, each layer U e,h is G-invariant. Proof: That {U e,h : (e,h) ∈ L(U )} is a stratification as described follows immediately from Proposition 2.1.10. U For G-invariance, let v ∈ U , s ∈ G, suppose that v ∈ U e,h and sv ∈ e,h .  To see that e = e, note first that for each 1 ≤ i ≤ n and Z ∈ s, πi (Zsv) = αi (s)πi (Ad(s −1 )Zv), so πi (Zsv) = 0 if and only if πi (Ad(s −1 )Zv) = 0.

2.2 Layers for a representation

89

Thus s(πi (sv)) = Ad(s)s(πi (v)) (see also the proof of Lemma 1.9.6). Hence for i > 0, s(πi (v)) = s(πi−1 (v)) if and only if s(πi (sv)) = s(πi−1 (sv)). This means e = e. Now since sj is an ideal, then for  any 1 ≤ j ≤ m and e ∈ e, sj ∩ s(πe (sv)) = Ad(s) sj ∩ s(πe (v)) , thus sj ∩ s(πe−1 (sv)) ⊂ s(πe (sv)) if and only if sj ∩ s(πe−1 (v)) ⊂ s(πe (v)). Therefore h (e) = h(e), and hence h = h. Corollary 2.2.5 The partition {e,h : (e,h) ∈ L(V )}, with the ordering ≺ inherited from L, and with the polynomials Pe,h,(e,h) ∈ L(V ), is a stratification of V . Moreover, min L(V ) = min L(U ), so that the minimal Zariski open set U e,h has nonempty intersection with V . Proof: By definition L(U ) \ L(V ) = {(e,h) : U e,h ∩ V = ∅}, and for such , U (e,h), Pe,h vanishes on V \ (e,h )≺(e,h) e,h . It follows that for (e,h) ∈ L(V ), e,h = {v ∈ V : Pe,h (v) = 0, and Pe,h (v) = 0 for all (e,h ) ∈ L(U ),(e,h ) ≺ (e,h)} = {v ∈ V : Pe,h (v) = 0, and Pe,h (v) = 0 for all (e,h ) ∈ L(V ),(e,h ) ≺ (e,h)}. Finally, observe that a nonzero polynomial function on U must be nonvanishing on V . Hence the minimal elements of L(U ) and L(V ) coincide, and the minimal layer e,h is the intersection of a Zariski open subset of U with V . Definition 2.2.6 Let β be a representation of a real solvable Lie algebra g in a real finite-dimensional vector space V , and fix a Jordan–Hölder sequence for s = gc and a flag for U = Vc . Then the resulting partition {e,h : (e,h) ∈ L(V )} is called the fine stratification for β. Elements of the fine stratification will be called fine layers. Given (e,h) ∈ L(V ) and v ∈ e,h , Lemma 2.2.1 shows that dim Gv = dimR g/g(v) = dimC s/s(v) = |e|. Hence we have the following. Corollary 2.2.7 If (e,h) ∈ L(V ), then all G-orbits in e,h have dimension |e|. The following is an immediate consequence of the preceding and Corollary 2.1.5.

90

Stratification of an orbit space

Corollary 2.2.8 Fix 1 ≤ i ≤ n and 1 ≤ j ≤ m. Let (e,h) ∈ L(U ) and recall the indices e(i,j ) and h(i,j ). Let Me(i,j ),h(i,j ) denote the flag equivalence class in Ci×j corresponding to e(i,j ),h(i,j ) and U/Ui

(sj

)e(i,j ),h(i,j ) = {w ∈ U/Ui : M(w,sj ) ∈ Me(i,j ),h(i,j ) }. U/Ui

Then πi (e,h ) ⊆ (sj

)e(i,j ),h(i,j ) .

Now fix 1 ≤ i ≤ n. For each Z ∈ s the matrix of v → Zv with respect to the basis (f1, . . . ,fn ) is lower triangular, so pi (Zv) depends only upon v1, . . . ,vi , and so the i-th row of M(v) depends only upon v1, . . . ,vi . Suppose that v ∈ U e,h . By construction, for each k = 1, . . . ,d, the entries of Tk (v) depend only upon v1,v2, . . . ,vek , so by Corollary 2.1.14, v → T (v)Zh(ek ) = T1 (v) · · · Tk−1 (v)Zh(ek ) depends only upon v1, . . . ,vek−1 . Moreover, the proof of Proposition 2.1.12 (specifically the expressions (2.1.8)) shows the function v → Pe,h (v)T (v) is polynomial in the variables v1, . . . ,vn . Finally, the strict normal form of M(v) is N (e,h,c(v)) = S(v)−1 M(v)T (v), where c(v) = (c1 (v), . . . ,cd (v)) is the sequence of (ek ,hk )-entries in this normal form. These observations are itemized in the following lemma. Lemma 2.2.9 Let (e,h) ∈ L(U ). Assume e = ∅ and write e = (e1,e2, . . . ,ed ), h = (h1, . . . ,hd ) as above. (1) The function v → T (v) is rational and regular on U e,h , and Pe,h (v)T (v) is a polynomial function. (2) For each 1 ≤ k ≤ d, the map v → Tk (v) depends only upon v1, . . . ,vek . (3) For each 1 ≤ k ≤ d, the map v → T (v)Zhk depends only upon v1, . . . ,vek −1 . (4) For each v ∈ U e,h , T (v)Zh(e) ∈ (s(πe−1 (v)) ∩ sh(e) ) \ s(πe (v)). Now fix 1 ≤ i ≤ n, and write e(i) = {e1, . . . ,er } as above. Since U/U πi (e,h ) ⊆ (sj i )e(i,j ),h(i,j ) , then for each v ∈ U e,h , the map T (πi (v)) : s → s for the quotient action βi in U/Ui is given by T (πi (v)) = T1 (v) · · · Tr (v). Hence the i coordinate of βi (T (v)Z)πi (v) is pi ((T (v)Z)v) = pi (βi (T (v)Z)πi (v)) = pi (βi (T1 (v)T2 (v) · · · Tr (v)Z)πi (v)). (2.2.5) Moreover v → pi (βi (T (v)Z)πi (v)) depends only upon v1, . . . ,ver −1 .

2.2 Layers for a representation

91

The following application of Lemma 2.2.9 will be useful later. Proposition 2.2.10 Let G be a solvable Lie group with Lie algebra g and let α be a representation of G in a real finite-dimensional vector space V with U U = Vc . Let U = U e,h be the minimal layer built as above so that  is a Zariski open subset of U . Let 1 ≤ i ≤ n such that i ∈ / e, and recall the notation / h(e(i)) and e(i) = e ∩ {1,2, . . . ,i}. Then λi (T (v)Zj ) = 0 holds for all j ∈ / e, then s(v) ⊂ ker λi . v ∈ U . In particular, if i ∈ Proof: Put β = dα. Suppose first i < e1 . Then for all v ∈ U the row M(v)i is identically zero, which means that pi (β(Zj )v) = 0 for 1 ≤ j ≤ m. Given any 1 ≤ j ≤ m, the relation (2.2.1) now gives λi (Zj )vi + pi (βi (Zj )wi−1 ) = 0 for all v in U (recall wi−1 + vi fi = πi (v)). Since U is Zariski open, this polynomial function of v is identically zero, and in particular, λi (Zj ) = 0, 1 ≤ j ≤ m. But this is equivalent with λi (T (v)Zj ) = 0, 1 ≤ j ≤ m. / e. Observe that πi (U Suppose now that i > e1 , i ∈ e,h ) is open in U/Ui , and by Corollary 2.2.8, U/U

i πi (U e,h ) ⊂ {e1,...,er },(h1,...,hr ) .

Consider wi−1 in U/Ui as described earlier. Since i ∈ / e, then er ≤ i − 1. It U/Ui , both w and π (v) belong to {e1,...,e , follows that for v ∈ U i−1 i e,h r },(h1,...,hr ) and by Corollary 2.1.5 and Lemma 2.2.9, Tk (wi−1 ) = Tk (v) for all k ≤ r. Let j ∈ / {h(e1 ), . . . ,h(er )}. By Relation (2.2.3) T1 (v) . . . Tr (v)Zj belongs to s(πi (v)) and again by (2.2.1), 0 = pi (T (v)Zj u) = pi (T1 (v) . . . Tr (v)Zj v) = λi (T1 (v) . . . Tr (v)Zj )vi + pi (βi (T1 (v) . . . Tr (v)Zj )wi−1 ). U/U

i , Similarly, since wi−1 is in {e1,...,e r },(h(e1 ),...,h(er ))

0 = pi (βi (T1 (v) . . . Tr (v)Zj )wi−1 ). Thus λi (T1 (v) . . . Tr (v)Zj )vi = 0, for u in the open set U e,h , therefore for any j∈ / {h1, . . . ,hr }, λi (T1 (v) . . . Tr (v)Zj ) = 0.

92

Stratification of an orbit space

Observe now that Tr+1 (v) . . . Td (v)Zj = Zj + Y (v) where Y (v) is in the / {h(e1 ), . . . ,h(er )} span of Zj  where j  ∈ {h(er + 1), . . . ,h(ed )}. Thus j  ∈ and: λi (T (v)Zj ) = λi (T1 (v) . . . Tr (v)Zj ) + λi (T1 (v) . . . Tr (v)Y (v)) = 0.

Example 2.2.11 Let g the Lie algebra with basis A, B, X, and [A,X] = −X, acting on R3 via the realization: ⎡ ⎤ ⎡ ⎤ ⎡ 0 1 1 ⎦ , Z2 = A = ⎣0 1 ⎦ , Z3 = B = ⎣0 Z1 = X = ⎣0 0 1 0 0 0 0 0 0 ⎤ ⎡ 0 v1 v1 Then the matrix M(v) is ⎣ 0 v2 v2 ⎦, and the minimal fine layer v1 0 v3 {1,3},(2,1) . Observe that 2 ∈ / e, 3 ∈ / {h(e1 )} = {2}, and λ2 (T (v)Z3 ) = λ2 (Z3 − Z2 ) = 0.

bracket ⎤ ⎦ 1 0 1 is  =



The following example exhibits distinct fine layers e,h and e,h where e = e and the sets h(e) and h (e ) are the same, but h = h . Example 2.2.12 Consider the following realization of the Heisenberg Lie algebra g = spanR {Z,Y,X} of Example 1.1.5 given by ⎡ ⎤ ⎡ ⎤ 0 0 0 0 0 0 ⎢0 0 0 ⎥ ⎢1 0 0 ⎥ ⎢ ⎥ ⎢ ⎥ ⎢1 0 0 ⎥ ⎢0 0 0 ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ Z=⎢ 0 0 0 0⎥ , Y = ⎢ 0 0 0 0⎥ , ⎢ ⎥ ⎢ ⎥ ⎢ ⎢ 0 0 0 0⎥ 0 0 0 0⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎣ 0 0 0 0⎦ 1 0 0 0⎦ 0 0 0 0 0 1 0 0 ⎡ ⎤ 0 0 0 ⎢0 0 0 ⎥ ⎢ ⎥ ⎢0 1 0 ⎥ ⎢ ⎥ ⎢ ⎥ X=⎢ 0 0 0 0⎥ . ⎢ ⎥ ⎢ 0 0 0 0⎥ ⎢ ⎥ ⎣ 0 1 0 0⎦ 1 0 0 0

2.2 Layers for a representation

93

The canonical basis of V = R7 is a basis adapted to a natural good flag for Vc .  For v = i vi fi , the matrix M(v) is ⎡

0 ⎢ ⎢0 ⎢ ⎢v1 ⎢ ⎢ M(v) = ⎢ 0 ⎢ ⎢0 ⎢ ⎢ ⎣0 0

0 v1 0 0 0 v4 v5

⎤ 0 ⎥ 0⎥ ⎥ v2 ⎥ ⎥ ⎥ 0⎥. ⎥ 0⎥ ⎥ ⎥ v5 ⎦ v4

The fine layers are determined by the following conditions: v1 = 0. v5 = 0, then e = {2,3,6}, and h = (2,1,3). v5 = 0, v4 = 0, then e = {2,3,7}, and h = (2,1,3). v5 = v4 = 0, then e = {2,3}, and h = (2,1). v1 = 0, v2 = 0. v4 = 0, then e = {3,6}, and h = (3,2). v4 = 0, v5 = 0, then e = {3,7}, and h = (3,2). v4 = v5 = 0, then e = {3}, and h = (3). v1 = v2 = 0. v42 = v52 , v4 = 0, then e = {6,7}, and h = (2,3). v42 = v52 , v4 = 0, then e = {6,7}, and h = (3,2). v42 = v52 = 0, then e = {6}, and h = (2). v4 = v5 = 0, then e = ∅, and h = ( ). Observe that the two fine layers {6,7},(2,3) and {6,7},(3,2) have same sets e and h(e). The following table gives the fine layering of the real space V, following the total ordering of L.

94

Stratification of an orbit space

(e,h)

Pe,h

e,h

{2,3,6},(2,1,3)

v15 v5

v1 = 0, v5 = 0

{2,3,7},(2,1,3)

v15 v4

v1 = 0, v5 = 0, v4 = 0

{2,3},(2,1)

−v13

v1 = 0, v4 = v5 = 0

{3,6},(3,2)

−v22 v4

v1 = 0, v2 = 0, v4 = 0

{3,7},(3,2)

−v22 v5

v1 = 0, v2 = 0, v4 = 0, v5 = 0

{6,7},(2,3)

v4 (v42 − v52 )

v1 = v2 = 0, v4 = 0, v42 = v52

{6,7},(3,2)

−v53

v1 = v2 = v4 = 0, v5 = 0

{3},(3)

v2

v1 = 0, v2 = 0, v4 = v5 = 0

{6},(2)

v4

v1 = v2 = 0, v4 = 0, v42 = v52 = 0

∅,()

1

v1 = v2 = v4 = v5 = 0 

Example 2.2.13 Let g = span{A,B} be the commutative Lie algebra with realization in gl(5,R) given by ⎡ ⎤ ⎡ ⎤ 0 0 0 0 0 0 ⎢0 0 0 ⎥ ⎢0 0 0 ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ A = ⎢1 0 0 ⎥ , B = ⎢0 1 0 ⎥. ⎢ ⎥ ⎢ ⎥ ⎣ ⎣ 1 0⎦ 0 0⎦ 0 1 1 0 The canonical basis for R5 provides a good flag for its complexification C5 . Let s = gc have the ordered Jordan–Hölder basis given by Z1 = A, Z2 = B. For each v ∈ R5 , we have ⎤ ⎡ 0 0 ⎢0 0⎥ ⎥ ⎢ ⎥ ⎢ M(v) = ⎢v1 v2 ⎥ . ⎥ ⎢ ⎣v4 0 ⎦ v5 v4

2.2 Layers for a representation

95

The fine layers are listed in order:

(e,h)

Pe,h

e,h

{3,4},(1,2)

−v1 v2 v4

v1 = 0, v2 = 0, v4 = 0

{3,4},(2,1)

−v22 v4

v1 = 0, v2 = 0, v4 = 0

{3,5},(1,2)

v1 (v1 v4 − v2 v5 )

v1 = 0, v2 v4 = 0, v1 v4 − v2 v5 = 0

{3,5},(2,1)

−v22 v5

v1 = 0, v2 = 0, v4 = 0, v5 = 0

{4,5},(1,2)

v43

v1 = v2 = 0, v4 = 0

{5},1

v5

v1 = v2 = v4 = 0, v5 = 0

∅,()

1

v1 = v2 = v4 = v5 = 0

Now consider the fine layer  = {3,5},(1,2) . Observe that for all v ∈ , the matrix N(e,h,c(v)) is the matrix: ⎡

0 ⎢0 ⎢ ⎢ N (e,h,c(v)) = ⎢v1 ⎢ ⎣0 0

⎤ 0 ⎥ 0 ⎥ ⎥ 0 ⎥. ⎥ ⎦ 0 v4 − v2 v5 /v1

This example is of particular interest since the layer  contains some orbits that are closed in V , and others that are not. Let v ∈ ; since G is abelian it is clear that the orbit of v is the set of all points of the form ⎤ v1 ⎥ ⎢ v2 ⎥ ⎢ ⎥ ⎢ exp t1 A exp t2 B v = ⎢v3 + t1 v1 + t2 v2 ⎥ . ⎥ ⎢ ⎦ ⎣ et1 v4 et1 (v5 + t2 v4 ) ⎡

Recall that since v ∈  then v2 v4 = 0 while v1 v4 − v2 v5 = 0. Suppose first that v2 = 0; then v4 = 0 so

96

Stratification of an orbit space ⎡

v1 0 v3 + t1 v1

⎤

⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥. exp t1 A exp t2 B v = ⎢ ⎥ ⎢ ⎥ ⎢ t ⎦ ⎣ e 1 v4 t e 1 (v5 + t2 v4 ) Since both t1 and t2 appear as translations, then the orbits Ov are closed subsets of V . Suppose, however, that v4 = 0, while v2 = 0 and v5 = 0; then ⎤ v1 ⎥ ⎢ ⎥ ⎢ v2 ⎥ ⎢ ⎥ exp t1 A exp t2 B v = ⎢ v + t v + t v 1 1 2 2⎥ . ⎢ 3 ⎥ ⎢ ⎦ ⎣ 0 et1 v5 ⎡

 v1  In this case the orbits Ov are not closed subsets of V : O v \ Ov .

v2 0 0 0

belongs to 

The preceding example suggests that a suitable classification of G-orbits should be a refinement of the fine stratification that accounts for the role played by the characters e of G corresponding to the weights λe , for e ∈ e. Fix a fine layer e,h and let e ∈ e so that for all v ∈ , s(πe−1 (v)) = s(πe (v)). Recall the generalized weight λe defined by Zfe = λe (Z)fe mod Ue . Now we use the discussion starting after the proof of Theorem 1.9.9. Fix v ∈  and suppose that s(πe−1 (v)) is real; that is, s(πe−1 (v)) ∩ g = g(πe−1 (v)). The group G(πe−1 (v)) (which was denoted G(w) in the preceding discussion) has Lie algebra g(πe−1 (v)) and acts in the quotient module U/Ue . Following (1.9.5), we write πe (v) = w + zfe where w ∈ W c = span{f1, . . . ,fe−1 } and z = pe (v) = ve . The action of s ∈ G(πe−1 (v)) is thus πe (sv) = w + (ϕw (s)z)fe mod Ue , where ϕw (s) is given by (see (1.9.5)): ϕw (s)z = pe (sv) = e (s)z + pe (sw). Characteristics of the orbit ω = {ϕw (s)z : s ∈ G(πe−1 (v))} depend on whether or not e |G(πe−1 (v)) is trivial.

2.2 Layers for a representation

97

Fix a fine layer  = e,h with e = ∅, and let e ∈ e. Recall the corresponding weight λe and the map T (v) on s. The index h(e) is the smallest index j for which sj ∩ s(πe−1 (v)) ⊂ s(πe (v)). By part (4) of Lemma 2.2.9, T (v)Zh(e) ∈ sh(e) , and sh(e) ∩ s(πe−1 (v)) = (sh(e) ∩ s(πe (v))) ⊕ CT (v)Zh(e) . The preceding example and discussion suggest we consider the two possibilities: λe (T (v)Zh(e) ) = 0 and λe (T (v)Zh(e) ) = 0 in the layer . Lemma 2.2.14 We have s(πe (v)) = s(πe−1 (v)) ∩ ker λe if and only if λe (T (v)Zh(e) ) = 0. Proof: If s(πe (v)) = s(πe−1 (v)) ∩ ker λe , since T (v)Zh(e) belongs to s(πe−1 (v)) \ s(πe (v)), then λe (T (v)Zh(e) ) = 0. / [s,s] = sp , Suppose now that λe (T (v)Zh(e) ) = 0. Then T (v)Zh(e) ∈ but T (v)Zh(e) ∈ sh(e) so h(e) > p. Let Z ∈ s(πe (v)); then Zv ∈ Ue , so T (v)Zh(e) Zv ∈ Ue and πe (T (v)Zh(e) Zv) = 0. Since [Z,T (v)Zh(e) ] ∈ [s,s] = sp , then by definition of h(e), [Z,T (v)Zh(e) ] ∈ s(πe−1 (v)) ∩ sp ⊂ s(πe (v)). Hence πe (ZT (v)Zh(e) v) = πe ([Z,T (v)Zh(e) ]v) = 0. On the other hand, since T (v)Zh(e) ∈ s(πe−1 (v)) \ s(πe (v)), then pe (T (v) Zh(e) v) = 0 and 0 = pe (ZT (v)Zh(e) v) = λe (Z)pe (T (v)Zh(e) v). Therefore λe (Z) = 0, and s(πe (v)) ⊂ s(πe−1 (v)) ∩ ker λe . Since these spaces have same dimension, they coincide. Denote the subset of indices e such that λe (T (v)Zh(e) ) = 0 by δ(v); that is δ(v) = {e ∈ e : s(πe−1 (v)) ∩ ker λe = s(πe (v))}.

(2.2.6)

We say that δ(v) is the set of dilation indices. Thus the fine layer e,h is partitioned into finitely many subsets e,h,δ = {v ∈ e,h : δ(v) = δ} where each δ is some subset of e for which e,h,δ = ∅. The nonempty sets e,h,δ are called ultrafine layers for the representation α.

98

Stratification of an orbit space

Example 2.2.15 Returning to Example 2.2.13, recall the fine layer  = {3,5},(1,2) . For each v ∈ , ⎤ ⎡ 1 ⎥ ⎢0 1   ⎥ ⎢ 1 −v2 /v1 ⎥ ⎢ −1 T (v) = , and S(v) = ⎢0 0 1 ⎥. ⎥ ⎢ 1 ⎦ ⎣0 0 −v4 /v1 1 0 0 −v5 /v1 0 1 Let v ∈  and consider the action of G(πe2 −1 (v)) = G(π4 (v)). The subalgebra g(π4 (v)) is the real span of T (v)Zh(5) = T (v)Z2 = B − (v2 /v1 )A, and since G is exponential, G(π4 (v)) = exp RT (v)Z2 . Write v = w + v5 f5 . For each t ∈ R, the action of G(π4 (v)) on v is given by the affine action of G(π4 (v)) = G(w) on the coordinate z = v5 , according to (1.9.5). Thus p5 (exp tT (v)Z2 v) = ϕw (exp tT (v)Z2 )v5 is given by ⎧ ⎨v5 + tv4, if v2 = 0, v4 = 0, p5 (exp tT (v)Z2 v) = ⎩e−(v2 /v1 )t v , if v2 = 0, v4 = 0. 5 There is only one nonvanishing root λe = λ5 , and h(5) = 2. The map v → λ5 (T (v)Zh(5) ) is v → −v2 /v1 . Hence there are two ultrafine layers in : {3,5},(1,2),∅ = {v ∈  : v2 = 0},

and {3,5},(1,2),{5} = {v ∈  : v2 = 0}. 

The ultrafine layering is particularly useful when α is of exponential type. Suppose that this is the case so that each weight λ has the form λ = μ(1 + ia) where a is a real number (possibly zero), and μ is a real form on g (possibly zero). We then choose the good flag U = U0 ⊃ U1 ⊃ · · · ⊃ Un = {0} so that the following holds for each Ui with corresponding weight λi : If Ui = U i then λi = μi (1 + iai ) with ai = 0.

(2.2.7)

Note that this condition still permits that μi = 0. Fix a fine layer e,h with v ∈ e,h . Suppose that e ∈ e is such that Ue is not real. Observe that Ue+1 = Ue ∩ U e . Then the real subspace Y = Cπe+1 (v) + Ue−1 /Ue+1 of the module U/Ue+1 is invariant for the action of s(πe−1 (v)). Then a basis for Ue−1 /Ue+1 is {πe+1 (fe ),πe+1 (fe+1 ) = πe+1 (fe )}, and the jump indices in Ue−1 /Ue+1 are e ∩ {e,e + 1}.

2.2 Layers for a representation

99

Observe that if πe+1 (v) ∈ Ue−1 /Ue+1 (so that Y = Ue−1 /Ue+1 ), then the action of X ∈ g(πe−1 (v)) has the matrix:   λ (X) 0 β(X) = e λe (X) 0 and e necessarily belongs to δ(v). In this case s(πe+1 (v)) = s(πe (v)) = s(πe−1 (v)) ∩ ker λe . This proves that {e,e + 1} ∩ e = {e}. / Ue−1 /Ue+1 then Y has the ordered basis Suppose then that πe+1 (v) ∈ {πe+1 (fe ),πe+1 (fe+1 ),πe+1 (v)}, with respect to which the action of X ∈ g(πe−1 (v)) has the following matrix: ⎡ ⎤ λe (X) 0 γ (X) (2.2.8) β(X) = ⎣ 0 λe (X) γ (X)⎦ 0 0 0 where γ , γ are forms on g(πe−1 (v)). Lemma 2.2.16 Suppose that α is of exponential type. Let v ∈ e,h , let e ∈ e, and suppose that Ue is not real. If e + 1 ∈ e, then {e,e + 1} ∩ δ(v) = ∅. Proof: From the preceding discussion, it follows that πe+1 (v) ∈ / Ue−1 , and that the matrix of βe+1 (X), X ∈ g(πe−1 (v)), acting in Y = Cπe+1 (v) + Ue−1 /Ue+1 with the ordered basis πe+1 (fe ), πe+1 (fe+1 ), πe+1 (v), is given by (2.2.8). It follows that s(πe (v)) = ker γ ,

s(πe+1 (v)) = ker γ ∩ ker γ .

Since e and e + 1 are in e, then γ and γ are independent. Therefore γ is not almost real, and ker γ = s(πe (v)) is not real. But α being of exponential type, both ker λe and s(πe−1 (v)) ∩ ker λe are real, thus: s(πe (v)) = s(πe−1 (v)) ∩ ker λe . This means e ∈ / δ(v). Suppose now that e + 1 ∈ δ(v), observe that ker λe+1 = ker λe , thus: s(πe+1 (v)) = ker γ ∩ ker γ = s(πe (v)) ∩ ker λe+1 = ker γ ∩ ker λe . This implies that λe |s(πe−1 (v)) is in span{γ ,γ }, and the space of the matrices βe+1 (X), X ∈ g(πe−1 (v)) is a two-dimensional real Lie subalgebra of the Lie algebra: ⎧ ⎡ ⎡ ⎤ ⎤⎫ ⎤ ⎡ 0 0 1 0 0 i ⎬ λ 0 0 ⎨ a = span A = ⎣0 λ 0⎦ , X1 = ⎣0 0 1⎦ , X2 = ⎣0 0 −i ⎦ ⎭ ⎩ 0 0 0 0 0 0 0 0 0

100

Stratification of an orbit space

with λ a complex, nonreal number. The only two dimensional real subalgebra of a is span{X1,X2 }. This means that g(πe−1 (v)) ⊂ ker λe , and s(πe (v)) ∩ ker λe+1 = s(πe (v)) = s(πe+1 (v)), a contradiction. Therefore, e + 1 ∈ / δ(v). Here is an example that is not of exponential type, where the preceding lemma does not hold. Example 2.2.17 Let V be a vector space over R of dimension two and let its complexification Vc have the basis (f1,f2 ) where f2 = f 1 . Let g be the commutative Lie algebra spanned by elements A and B where Af1 = if1,Af2 = −if2,Bf1 = f1,Bf2 = f2 . Then (A,B) is a good Jordan–Hölder basis of s = gc , and (f1,f2 ) is a good flag basis for U = Vc with weight sequence (λ1,λ2 = λ1 ). Write v ∈ V as v = v1 f1 + v 1 f 1 . Here there are two fine layers: for v1 = 0, then e = {1,2} and h = (1,2), while if v1 = 0, then v = 0 and e = ∅. The group G = exp g acts by both rotations and dilations, and is not of exponential type since λ1 (A) = i. Note also that if v1 = 0, then the G-orbit of v is V \ {0}. Suppose v1 = 0. Then U1 = Cf2 is not real, and both 1 and 2 belong to e, so the hypothesis of Lemma 2.2.16 is fulfilled. However in this case s(π1 (v)) = ker λ1 = s(π0 (v)) ∩ ker λ1 = C(A − iB) and s(π2 (v)) = s(π1 (v)) ∩ ker  λ2 = {0}. Hence both e = 1 and e + 1 = 2 belong to δ. Example 2.2.18 Coming back to Example 1.9.11, it is clear that there is a good descending flag basis for Vc such that the action of g is given as the standard action of ⎡ ⎤ 0 0 0 0 ⎢0 0 0 0 ⎥ ⎢ ⎥ ⎢1 0 0 0 ⎥ ⎢ ⎥ ⎢0 1 0 0 ⎥ ⎢ ⎥, A1 = ⎢ ⎥ iλ 1 ⎢ ⎥ ⎢ ⎥ −iλ 1 ⎢ ⎥ ⎣ ⎦ 0 0 ⎡ ⎤ 0 0 0 0 ⎢0 0 0 0 ⎥ ⎢ ⎥ ⎢0 1 0 0 ⎥ ⎢ ⎥ ⎢1 0 0 0 ⎥ ⎢ ⎥. A2 = ⎢ ⎥ iλ2 ⎢ ⎥ ⎢ ⎥ −iλ2 ⎢ ⎥ ⎣ ⎦ 3i −3i

2.2 Layers for a representation

101

√ √ where λ1 = 1 − 1/ 2, λ2 = 1 + 1/ 2, and where v ∈ V is written as t [v ,v ,v ,v ,v ,v ,v ,v ]. 1 2 3 4 5 5 7 7 With the good Jordan–Hölder basis (A1,A2 ), the matrix M(v) is ⎡ ⎤ 0 0 ⎢ 0 0 ⎥ ⎢ ⎥ ⎢ v v2 ⎥ 1 ⎢ ⎥ ⎢ ⎥ v1 ⎥ ⎢ v2 M(v) = ⎢ ⎥. ⎢ iλ1 v5 iλ2 v5 ⎥ ⎢ ⎥ ⎢−iλ1 v5 −iλ2 v5 ⎥ ⎢ ⎥ ⎣ 0 3iv7 ⎦ 0 −3iv7 Thus the first layers, precisely the layers for which (v1,v2 ) = (0,0) are the following: (e,h)

δ

e,h,δ

Regularity

{3,4},(1,2)

∅

v1 = 0, v12 − v22 = 0

regular

{3,4},(2,1)

∅

v1 = 0, v2 = 0

regular

{3,5},(1,2)

{5}

v1 = 0, v12 = v22, v5 = 0

v1 = v2, not regular v1 = −v2, regular

{3,5},(1,2)

{5}

v1 = 0, v12 = v22, v5 = 0, v7 = 0

regular

{3,7},(1,2)

{7}

v1 = 0, v12 = v22, v7 = 0, v5 = 0

regular

{3},(1)

∅

v1 = 0, v12 = v22, v5 = v7 = 0

regular

Regularity of orbits of the group G was studied in Example 1.9.11. Observe that the third layer 3 = {3,5},(1,2),{5} has four connected components: * + * + C1 = v ∈ 3 : v1 = v2 > 0 , C2 = v ∈ 3 : − v1 = v2 > 0 , * + * + C3 = v ∈ 3 : v1 = v2 < 0 , C4 = v ∈ 3 : v1 = −v2 > 0 . Every orbit in the connected components C2 , C4 is regular, every orbit in C1 and C3 is not regular. Thus the layer 3 is the disjoint union of two nonempty

102

Stratification of an orbit space

− + open G-invariant subsets 3 = + 3 ∪ 3 such that all orbits in 3 are not − regular, and all orbits in 3 are regular:

+ 3 = C1 ∪ C3 ,

− 3 = C2 ∪ C4 .



Proposition 2.2.19 Let G be a connected solvable Lie group with Lie algebra g and let α be a representation of G in a real finite-dimensional vector space V . Let e,h,δ be an ultrafine layer for the corresponding representation of g, and a choice of flags. Then e,h,δ is G-invariant. Proof: Recall that e,h is G-invariant. The invariance of v → δ(v) comes from its definition (see 2.2.6). Indeed, since s(πi (sv)) = Ad(s)s(π(v)) (s ∈ G), and ker λe is an ideal, we get s(πe−1 (sv)) ∩ ker λe = Ad(s) (s(πe−1 (v)) ∩ ker λe ) , thus e ∈ δ(sv) if and only if e ∈ δ(v). The index set for ultrafine layers is the set L∗ (V ) of all triples (e,h,δ) such that e,h,δ = ∅, that is, L∗ (V ) = {(e,h,δ) : (e,h) ∈ L(V ), and e,h,δ = ∅}. Define the ordering ≺≺ on L∗ (V ) by 1. (e,h) ≺ (e,h ), or 2. (e,h) = (e,h ) and |δ  | < |δ|, or 3. (e,h) = (e,h ), |δ| = |δ  | and δ < δ  for the lexicographic ordering. For any (e,h,δ), define the rational function Qe,h,δ by Qe,h,δ (v) = 1 if δ = ∅ and Qe,h,δ (v) = e∈δ λe (T (v)Zh(e) ) if δ = ∅. Remark that, thanks to the relation (2.1.8) Pe,h (v)Qe,h,δ (v) is a polynomial function. Proposition 2.2.20 The collection {e,h,δ : (e,h,δ) ∈ L∗ } of all ultrafine layers, with ordering ≺≺ and polynomial functions Pe,h (v)2 Qe,h,δ (v) is a stratification of V . Observe that e,h,δ is the semi-algebraic subset of V defined by Pe,h (v) = 0, for any (e,h ) ≺ (e,h), Pe,h (v) = 0, and Qe,h,δ  (v) = 0, for any δ  such that (e,h,δ  ) ≺≺ (e,h,δ), Qe,h,δ (v) = 0. Recall that the minimal ultrafine layer is a Zariski open subset. Finally, we remark that the maximal element (e,h) in L(V ) (resp. (e,h,δ) in L∗ (V )) is that for which e = ∅, and the corresponding layer is ∅,() = ∅,(),∅ = {v ∈ V : Xv = 0, for all X ∈ g}. The projection of ultrafine layers is also well-behaved.

2.2 Layers for a representation

103

Corollary 2.2.21 Suppose α is of exponential type, and let (Ui ) be a good flag for U = Vc satisfying the condition (2.2.7). Let e,h,δ be an ultrafine layer. Fix 1 ≤ i ≤ n such that U i = Ui , and 1 ≤ j ≤ m such that sj = sj . Consider the ultrafine layering for the quotient module U/Ui (with respect to the good flag (Ui  /Ui )i  ≤i ) and the Lie algebra sj . Then  U/U  πi (e,h,δ ) ⊆ sj i e(i,j ),h(i,j ),δ(i,j ) where e(i,j ) and h(i,j ) are defined as above, and where δ(i,j ) = e(i,j ) ∩ δ. U/U

Proof: By Lemma 2.2.8 πi (e,h ) ⊆ (sj i )e(i,j ),h(i,j ) . But for each e ∈ e(i,j ), and v ∈ e,h , we have h(i,j )(e) = h(e), λe (T (πi (v))Zh(e) ) = λe (T (v)Zh(e) ), so e ∈ δ(i,j ) if and only if λe (T (πi (v))Zh(e) ) = 0. Example 2.2.22 Consider now Examples 2.2.13 and 2.2.15. Taking into account the ultrafine layering, we get the following table:

(e,h)

Pe,h

δ

Qe,h,δ

e,h,δ

{3,4},(1,2)

−v1 v2 v7

∅

1

v1 = 0, v2 = 0, v4 = 0

{3,4},(2,1)

−v22 v4

{4}

1

v1 = 0, v2 = 0, v4 = 0

{3,5},(1,2) v1 (v1 v4 − v2 v5 )

{5} −v2 /v1 v1 = 0, v2 = 0, v4 = 0, v5 = 0 ∅

1

v1 = 0, v2 = 0, v4 = 0

{3,5},(2,1)

−v22 v5

{5}

1

v1 = 0, v2 = 0, v5 = 0

{4,5},(1,2)

v43

∅

1

v1 = v2 = 0, v4 = 0

{5},1

v5

{5}

1

v1 = v2 = v4 = 0, v5 = 0

∅,()

1

∅

1

v1 = v2 = v4 = v5 = 0 

104

Stratification of an orbit space

2.3 Orbit structure for a completely solvable action Let G be a simply connected solvable Lie group with Lie algebra g, let V be a finite-dimensional real vector space, and let α be a representation of G in V . Given v ∈ V , let G(v) be the stabilizer of v in G; G(v) is a closed subgroup of G and the Lie algebra of G(v) is g(v) = s(v) ∩ g. Recall that G/G(v) has a natural manifold structure, and that the G-orbit O of v has the structure of a submanifold of V for which the natural map G/G(v) → O is a diffeomorphism (Lemma 1.9.7). If G(v) is connected, then by Proposition 1.7.16, there are elements X1,X2, . . . ,Xd ∈ g, such that the map (t1,t2, . . . ,td ) → exp t1 X1 exp t2 X2 · · · exp td Xd G(v) is a diffeomorphism of Rd onto G/G(v). This gives a smooth bijection (t1, . . . ,td ) → α(exp t1 X1 · · · exp td Xd ) v

(2.3.1)

that we say is an orbit parametrization of O by Rd . Even in this case O may not be locally compact in the relative topology from V ; it follows from Theorem 1.9.9 that the manifold topology on O coincides with the relative topology if and only if O is regularly embedded in V . Recall that if O is regularly embedded, then we say that O is regular. Now consider the set V /G of all orbits. Recall that the quotient topology on this set is the finest topology for which the projection map is continuous, and that the projection map is open. The quotient topology has a countable basis, so it is possible to describe it by using sequences: a sequence (On ) of orbits converges to an orbit O if and only if there is a sequence (vn ) of points in V such that vn ∈ On for each n, and (vn ) converges to some point v ∈ O. A G-orbit O in V is a closed subset of V if and only if it is a closed point in the quotient space V /G. In the following examples, and frequently in what follows, we denote the linear actions defined by α and dα multiplicatively. Example 2.3.1 Let G = R with g = RA acting on V = R by Av = v; this action is of completely solvable type, and there are two fine layers: 1 = V \ {0}, which is the union of the two open G-orbits O+ = {v : v > 0} and O− = {v : v < 0}, and 2 = {0}. In the space V /G = {O+,O−,{0}}, the closure of {O+ } is {O+ } ∪ {{0}}. Thus V /G does not have the T1 separation property.  Even if all orbits are closed, the topology of V /G may not be Hausdorff. Example 2.3.2 Let g = RA where



0 A= 1

 0 . 0

2.3 Orbit structure for a completely solvable action

105

The standard action is of nilpotent type and there are two fine  layers: 1 = {v : v1 = 0}, in which G-orbits are vertical lines, Ov1 = { vt1 : t ∈ R}, and 2 = {v : v1 = 0}, in which orbits are single points. Each orbit is closed in V , and the quotient

 space 1 /G, 2 /G in each layer is Hausdorff (clearly Ov1 → v1 and { v02 } → v2 are homeomorphisms). However, since  

 = 1/n exp v2 nA 1/n v2 , then the sequence (O1/n ) of orbits in 1 converges 0 to every orbit in 2 .  Observe that in Examples 2.3.1 and 2.3.2 all orbits are regular. Indeed, by Proposition 1.9.13, if α is of exponential type (which is the case in these examples), then all orbits are regular, and the stabilizers are connected. Though the quotient space is not Hausdorff, the quotient space in each layer is Hausdorff, and is easily described by means of an explicit cross-section for the orbits. In general, given a G-invariant subset  in V , a subset  ⊂  is a cross-section for the action of G in  if the intersection of each orbit O in  with  is a singleton set. If  is a cross-section, the cross-section map, from  to , is the map which associates to each v ∈  the point in  ∩ Gv. A cross-section  in  is topological if the natural bijection between /G and  is a homeomorphism, where  has the relative topology. A necessary and sufficient condition that a cross-section be topological is that the cross-section map is continuous. In this and the next three sections we show that if G is solvable and α is of exponential type, then, with a careful choice of coexponential basis, the map (2.3.1) together with constructions of Sections 2.1 and 2.2 yield a construction of an explicit topological cross-section in each ultrafine layer. These constructions in the case where G is nilpotent and the action is unipotent were first given by L. Pukanszky [79]; see [16] for a more recent exposition. An intermediate case occurs when all the weights of β = dα are real valued, that is, when β is of completely solvable type. Since this case occurs rather frequently in applications and exhibits much of what is new when passing beyond the unipotent case, we treat the completely solvable case separately in the present section. Suppose that α and β = dα are of completely solvable type. By the remark following Definition 1.3.14, there is a good descending basis (fi )1≤i≤n for Vc such that fi ∈ V , 1 ≤ i ≤ n. Thus we have the real flag V = V0 ⊇ · · · ⊇ Vn = {0}, where Vi = spanR (fi+1, . . . ,fn ), and β(g) is a subalgebra of the Lie algebra of real lower-triangular matrices, which is completely solvable (see Example 1.4.6), thus β(g) is completely solvable, and we can suppose without loss

106

Stratification of an orbit space

of generality that g itself is completely solvable. Accordingly, choose a real basis (Zj )1≤j ≤m for g, which is a good Jordan–Hölder basis of s = gc . We interpret the canonical map πi : V → V /Vi as the projection of V onto Wi = spanR {f1,f2, . . . ,fi } parallel to Vi . Now for v ∈ V , each s(πi (v)) is real, s(πi (v)) ∩ g = g(πi (v)). To sum up, for the present section, we make the following assumptions: G is a simply connected completely solvable Lie group, and α is a representation of G of completely solvable type acting in the vector space V of dimension n. A Jordan–Hölder basis (Zj ) for gc is fixed with Zj ∈ g for all j , and a good descending flag basis (fi )ni=1 for Vc is fixed with fi ∈ V for all i. Following Section 2.2, we carry out the layering procedure with the chosen bases, and let {e,h,δ : (e,h,δ) ∈ L∗ (V )} be the ultrafine stratification of V. Fix an ultrafine layer  = e,h,δ for which e = ∅. Recall that we write e = {e1 < e2 < · · · < ed } and h as the sequence (h1, . . . ,hd ) defined by hs = h(es ), 1 ≤ s ≤ d. For each v ∈ , recall the map T (v), given by a unipotent upper-triangular matrix; since all bases are chosen to be real, then T (v) has real entries. Observe that (T (v)Zj )1≤j ≤m is a weak Malcev basis for g. The following is a consequence of Corollary 1.7.16. Lemma 2.3.3 Fix 1 ≤ i < n. Let e(i) = e ∩ {1, . . . ,i} = {e1,e2, . . . ,er }, h(i) = (h1, . . . ,hr ) and δ(i) = δ ∩ e(i). (1) Write {1,2, . . . ,m} \ {h1, . . . ,hr } = {j1 < j2 < · · · < jm−r }. Then (T (v)Zj1 ,T (v)Zj2 , . . . ,T (v)Zjm−r ) is a weak Malcev basis of g(πi (v)). (2) Suppose that i = er ∈ e. Then (T (v)Zhr ) is a coexponential basis for g(πi (v)) in g(πi−1 (v)). (3) (T (v)Zh1 ,T (v)Zh2 , . . . ,T (v)Zhr ) is a coexponential basis for g(πi (v)) in g. Proof: The statement (1) follows from the fact that T (v) is unipotent upper triangular and the relation (2.2.3). Parts (2) and (3) are immediate consequences of Corollary 1.7.16. For each e ∈ e define the normalized acting element Xe (v) =

T (v)Zh(e) . pe (T (v)Zh(e) v)

So that Xe (v)v + Ve = fe + Ve . For each t ∈ Rd , v ∈ , and 1 ≤ k ≤ d, let gk (t,v) be the element of G defined by gk (t,v) = exp t1 Xe1 (v) exp t2 Xe2 (v) · · · exp tk Xek (v).

2.3 Orbit structure for a completely solvable action

107

By part (3) of Lemma 2.3.3, (Xe1 (v),Xe2 (v), . . . ,Xed (v)) is a coexponential basis for g(v) in g. Define the map Q : Rd ×  →  by Q(t,v) = gd (t,v)v = exp t1 Xe1 (v) exp t2 Xe2 (v) · · · exp td Xed (v) v. (2.3.2) For each 1 ≤ i ≤ n, denote the coordinate map pi ◦ Q by Qi . Fix i = e ∈ e consider the G-module V /Ve , identified with We = span{f1, . . . ,fe }. Via the real basis element fe , identify the subspace Ve−1 /Ve with R, writing pe (v) = z ∈ R, and put w = πe (v) − zfe ∈ We−1 = span{f1, . . . ,fe−1 }. Recall the affine action ϕw of G(w), described (1.9.5), and here acting on the real line by ϕw (exp tXe (v))z = pe (exp tXe (v)v) = e (exp tXe (v))z + pe (exp tXe (v)w). (2.3.3) Now e ∈ / δ if and only if e (exp tXe (v)) = 1 for all t, whence the affine action is just translation on R. Now suppose that e ∈ δ. Then by Lemma 1.9.12 there is a fixed point z0 (w) for this action, satisfying z0 (w) =

pe (exp tXe (v)w) 1 − e (exp tXe (v))

for all t, and the action can be written in a more transparent form as ϕw (exp tXe (v))z = e (exp tXe (v))(z − z0 (w)) + z0 (w).

(2.3.4)

Differentiating the identity (1 − e (exp tXe (v)))z0 (w) = pe (exp tXe (v)w) at t = 0 we get # d ## pe (exp tXe (v)w) = pe (Xe (v)w). −λe (Xe (v))z0 (w) = dt #t=0 Now by definition of Xe (v), we have 1 = pe (Xe (v)v) = pe (Xe (v)w) + λe (Xe (v))z so we may write z0 (w) = −

pe (Xe (v)w) 1 =z− . λe (Xe (v)) λe (Xe (v))

Thus if e ∈ δ, we define the nonvanishing rational function be :  → R by be (v) = z − z0 (w) =

1 . λe (Xe (v))

(2.3.5)

Lemma 2.3.4 Let v ∈  and suppose that e ∈ δ. Then for each A ∈ g(πe−1 (v)) \ g(πe (v)), we have be (v) = pe (Av)/λe (A). Moreover, the real function be is semi-invariant with multiplier e .

108

Stratification of an orbit space

Proof: Given A ∈ g(πe−1 (v))\g(πe (v)), put B = λe (Xe (v))A−λe (A)Xe (v); then B belongs to g(πe−1 (v)) ∩ ker λe . Since e ∈ δ, this implies B ∈ g(πe (v)). Thus pe (Bv) = 0, and be (v) =

pe (Xe (v)v) pe (Av) = . λe (Xe (v)) λe (A)

To prove semi-invariance, let s ∈ G. Observe that Ad(s)Xe (v) g(πe−1 (sv)) \ g(πe (sv)) and by (1.9.4), Ad(s)Xe (v)sv = sXe (v)v so be (sv) =

∈

pe (Ad(s)Xe (v)sv) pe (sXe (v)v) pe (sfe ) = = = e (s)be (v). λe (Ad(s)Xe (v)) λe (Ad(s)Xe (v)) λe (Xe (v))

The function Q(t,v) of (2.3.2) is described as follows. Proposition 2.3.5 Let  be an ultrafine layer, and define Q as in (2.3.2). Then each the following hold. (1) Q is a real analytic function on Rd × . (2) For each v ∈ , Q(·,v) is a diffeomorphism from Rd to the orbit Gv. (3) Let 1 ≤ i ≤ n. (3.1) If i ∈ / e, then Qi has the form Qi (t,v) = i (gr (t,v))vi + Yi (t1, . . . ,tr ,v1, . . . ,vi−1 ), where er = max e(i − 1) = max{e ∈ e : e < i}. (If i < e1 , this means Qi (t,v) = vi .) / δ, then Qi has the form (3.2) If i = ek ∈ Qi (t,v) = i (gk−1 (t,v))(vi + tk ) + Yi (t1, . . . ,tk−1,v1, . . . ,vi−1 ). (3.3) If i = ek ∈ δ, then Qi has the form Qi (t,v) = i (gk (t,v))bek (v) + Yi (t1, . . . ,tk−1,v1, . . . ,vi−1 ). (4) The semialgebraic set  = e,h,δ = {v ∈  : ve = 0 if e ∈ e \ δ, be (v) = ±1 if e ∈ δ} is a topological cross-section for the orbits in . Proof: Each g-valued function v → Xek (v) is a rational function, nonsingular on . Hence the function (t,v) → exp t1 Xe1 (v) exp t2 Xe2 (v) · · · exp td Xed (v) is real analytic from Rd ×  into G. By Lemma 2.3.3, (X1 (v), . . . ,Xd (v)) is a coexponential basis for g(v) in g, and the linear action of G defined by the representation α is smooth. Both (1) and (2) follow from these observations.

2.3 Orbit structure for a completely solvable action

109

By Lemma 2.2.9, for each e ∈ e, the map v → Xe (v) depends only upon v1, . . . ,ve−1 , and hence for each t ∈ R, the same is true for v → exp tXe (v). Hence for each (t,v) ∈ Rd × , (t,v) → gk (t,v) depends only on t1, . . . ,tk and v1, . . . ,vek −1 . Now fix i,1 ≤ i ≤ n. Recall that the quotient map πi is equivariant for the actions of G on V and on V /Vi = Wi , and recall the vector wi−1 = wi−1 (v) = πi (v) − pi (v)fi ∈ Wi . If i < e1 , by definition of e1 , for any X ∈ g, pi (Xv) = 0, so G(πi (v)) = G, and Qi (t,v) = pi (gd (t,v)v) = vi . Suppose that i ∈ / e but that i > e1 . Define er ∈ e by er = max e(i − 1). If k > r then πi (exp tk Xek (v)v) = exp tk Xek (v)πi (v) = πi (v), so πi (Q(t,v)) = πi (gd (t,v)v) = gd (t,v)πi (v) = gr (t,v)πi (v)   = gr (t,v) vi fi + wi−1 (v) . We have:   gr (t,v) vi fi = i (gr (t,v))vi fi (mod Vi ). Since by definition wi−1 (v) depends only upon v1, . . . ,vi−1 , then the map v → gr (t,v)πi−1 (v) depends only upon v1, . . . ,vi−1 . Thus: Qi (t,v) = pi (gr (t,v)v) = i (gr (t,v))vi + pi (gr (t,v)wi−1 (v)) = i (gr (t,v))vi + Yi (t1, . . . ,tr ,v1, . . . ,vi−1 ).

(2.3.6)

Next suppose that i = ek ∈ e \ δ, and consider the action of exp tk Xek (v). Since Xek (v) ∈ g(πek −1 (v)) \ g(πek (v)), Xek (v)v = pi (Xek (v)v)fi = fi (mod Vi ) and since λi (Xek (v)) = 0 then for m ≥ 2, Xek (v)m v = 0 (mod Vi ). Hence pi (exp tk Xek (v)v) = pi (v + tk Xek (v)v) = vi + tk . If k = 1, this gives Qi (t,v) = vi + tk . If k > 1, r = k − 1, and we calculate as in (2.3.6), except with vi replaced by vi + tk : Qi (t,v) = pi (gd (t,v)v) = pi (gk−1 (t,v) exp tk Xek (v)v)   = pi gk−1 (t,v) ((vi + tk )fi + wi−1 (v)) = i (gk−1 (t,v))(vi + tk ) + Yi (t,v) where again Yi (t,v) = pi (gk−1 (t,v)wi−1 (v)) depends only upon v1, . . . ,vi−1 .

110

Stratification of an orbit space

Suppose now i = ek ∈ δ, and consider again the action of exp tk Xek (v). Then using (2.3.3), (2.3.4) and (2.3.5), we get: Qi (t,v) = pi (gk−1 (t,v) exp tk Xek (v)v)

  = pi gk−1 (t,v) wi−1 (v) + ϕwi−1 (v) (exp tk Xek (v))vi fi = pi (gk−1 (t,v)wi−1 (v))    + i gk−1 (t,v) i (exp tk Xek (v))bek (v) + z0 (wi−1 (v)) = i (exp t1 Xe1 (v) . . . exp tk Xek (v))bek (v) + Yi (t1, . . . ,tk−1,v1, . . . ,vi−1 ), with Yi (t1, . . . ,tk−1,v1, . . . ,vi−1 ) = pi (gk−1 (t,v)wi−1 (v)) + z0 (wi−1 (v)). Thus in each case, Qi (t,v) has the form described in part (3). To prove (4), consider the system of equations in the variables t1, . . . ,td : ⎧ ⎨Qe (t,v) = 0, if ek ∈ e \ δ, k (2.3.7) ⎩|b (exp t X (v) . . . exp t X (v)v)| = 1, if e ∈ δ. ek 1 e1 k ek k This system can be solved recursively. If k = 1, the first equation is either ve1 + t1 = 0 or |et1 λe1 (Xe1 (v)) be1 (v)| = 1. This equation has a unique solution t1 (v) in either case. Suppose the same holds for the k − 1 first equations. Now the k-th equation reduces to ek (gk−1 (t1 (v), . . . ,tk−1 (v),v))(vek + tk ) + Yek (t1 (v), . . . ,tk−1 (v),v) = 0 or ek (gk−1 (t1 (v), . . . ,tk−1 (v),v))etk λek (Xek (v)) |bek (v)| = 1. There is a unique solution tk (v) of this equation. By construction, the map v → (t1 (v), . . . ,td (v)d ) is continuous. It is clear that  is the image of the continuous mapping v → Q(t (v),v) (v ∈ ), and  meets each orbit in a single point, so it is a topological cross-section. Remark 2.3.6 Suppose that dα is of nilpotent type, so that δ = ∅, each weight character i is identically 1, and α(G) is a nilpotent Lie group. For each v ∈ , t → Qi (t,v) is a polynomial map and for i ∈ e, has the form Qi (t,v) = vi + tk + Yi (t1, . . . ,tk−1,v1, . . . ,vi−1 ). Consequently, if (tn ) is a sequence on Rd such that the sequence (Q(tn,v)) converges in V , then (tn ) is convergent in Rd . Hence the orbit of v is a closed subset in V . Observe also that in this case,  is of the form T ∩  where T is a subspace of V .

2.3 Orbit structure for a completely solvable action

111

Finally, one can see that if dα is of nilpotent type, then for each t ∈ Rd , v → Q(t,v) is a nonsingular rational function on . For a proof of this last fact as well as other related results for the nilpotent case, see the books of Pukanszky [79] or of Corwin and Greenleaf [16]. Example 2.3.7 Let G = Aff(R), with g = span{A,Y }, [A,Y ] = Y , with the standard action given by the realization: 

  0 0 −1 Y = , A= 1 0 0

 0 . 0

The basis (Y,A) is a good Jordan–Hölder basis for g and the good descending basis (f1,f2 ) is the canonical basis of R2 . The open ultrafine layer is  = {v ∈ R2 : v1 = 0}, where e = {1,2}, h = (2,1), δ = {1}. We have T (v) = I , X1 (v) = −v1−1 A, X2 (v) = v1−1 Y , and b1 (v) = v1 . Thus Q(t1,t2,v) = exp t1 X1 (v) exp t2 X2 (v) v = Q1 (t1,t2,v)f1 + Q2 (t1,t2,v)f2 where Q1 (t1,t2,v) = et1 /v1 v1, Q2 (t1,t2,v) = v2 + t2 . Thus  consists of two open orbits O+ = {v : v1 > 0} and O− = {v : v1 < 0}. The cross-section for the G-action in  described in part (4) of Proposition 2.3.5 is  = {v : v1 = ±1, v2 = 0}.



Example 2.3.8 As before, let G = Aff(R), with Lie algebra g = span{A,Y }, with [A,Y ] = Y . Now realize g as the Lie algebra of 4 × 4 real matrices spanned by ⎡

0 ⎢1 Y =⎢ ⎣0 0

⎤ 0 1 0 0 0

⎥ ⎥, ⎦ 0

⎡

0 ⎢0 1 A=⎢ ⎣0 0 0 0

⎤ 2 0

⎥ ⎥ ⎦

(a ∈ R).

a

Let α be the representation defined by the standard action of G obtained by exponentiation of this realization. The basis (Z1,Z2 ) = (Y,A)) is a good Jordan–Hölder basis and since the realization is by lower-triangular matrices

112

Stratification of an orbit space

the canonical basis (f1, . . . ,f4 ) is a good descending basis of V . The resulting matrix M(v) is ⎤ ⎡ 0 0 ⎢v1 v2 ⎥ ⎥ M(v) = ⎢ ⎣v2 2v3 ⎦ . 0

av4

The minimal layer  is included in the set of all v for which the second and third rows of M(v) are independent. Thus the jump indices for  are e = {2,3}, given by the conditions v1 = 0 and 2v1 v3 − v22 = 0. Now for the minimal layer, we also have h(2) = 1 and h(3) = 2, T (v)Z1 = Y, and T (v)Z2 = A − (v2 /v1 )Y . Note that p3 (T (v)Z2 v) = 2v3 − v22 /v1 = 0. The normalized coexponential basis is Xe1 (v) = Y /v1 and Xe2 (v) =

1 (v1 A − v2 Y ). 2v1 v3 − v22

Observe also that δ = {3} and b3 (v) = v3 − v22 /2v1 , and the fixed point for the affine action by G(π2 (v)) in V /V3 is z0 (w2 (v)) = v22 /2v1 . Therefore: Q1 (t1,t2,v1,v2,v3,v4 ) = v1, Q2 (t1,t2,v1,v2,v3,v4 ) = v2 + t1, Q3 (t1,t2,v1,v2,v3,v4 ) = et2 /b3 (v) b3 (v) +

v22 t2 v2 + t1 + 1 , 2v1 v1 2v1

Q4 (t1,t2,v1,v2,v3,v4 ) = eat2 /(2b3 (v)) v4 . Then t1 (v) = −v2 and t2 (v) = −b3 (v) ln |b3 (v)| and the cross-section map is: ⎛ #−a/2 ⎞ $ % # # 2 2 v v ## # (v1,v2,v3,v4 ) → ⎝v1,0,sign v3 − 2 , #v3 − 2 # v4 ⎠ .  # 2v1 2v1 # The expressions for each Qi (t,v) are considerably simplified by a change of variables. For each 1 ≤ k ≤ d, put ⎧ ⎨Qe (t,v), if ek ∈ e \ δ, k zk = zk (t,v) = ⎩|b (Q(t,v))|, if e ∈ δ. ek k In light of Proposition 2.3.5, several observations can be made. (a) For each v ∈ , t → z(t,v) is a real analytic map from Rd onto the open subset Z of Rd defined by Z = {z ∈ Rd : zk > 0 if ek ∈ δ}.

2.3 Orbit structure for a completely solvable action

113

(b) For each v ∈ , 1 ≤ k ≤ d, zk (t,v) = zk (t1, . . . ,tk ,v1, . . . ,vek ). (c) For each v ∈ , t → z(t,v) is a bijection whose inverse is given explicitly by solving for t1 in terms of z1 , subsequently t2 in terms of z1,z2 , and so on. The inverse map z → (z,v) is analytic on Z, and for each 1 ≤ k ≤ d, k (z,v) depends only upon z1, . . . ,zk ,v1, . . . ,vek . Thus for each v ∈ , the map z → Q((z,v),v) is an analytic bijection from Z onto the G-orbit of v. Put P (z,v) = Q((z,v),v), z ∈ Z,v ∈ . We now claim that for each s in G, P (z,sv) = P (z,v). Put u = P (z,v), u = P (z,sv). Inductively, suppose that u1 = u1 , . . . , ui−1 = ui−1 , that is, πi−1 (u) = πi−1 (u ). Observe that u and u are in the same orbit by definition, and thus there is s  ∈ G(πi−1 (u)) such that u = s  u. 1. If i ∈ / e, then G(πi (u)) = G(πi−1 (u)) = G(πi (u )). In this case, πi (u ) = s  πi (u), with s  ∈ G(πi−1 (u)), thus πi (u ) = πi (u) and Pi (z,v) = Pi (z,sv). 2. If i = ek ∈ e \ δ, then ui = zk = ui or Pi (z,v) = Pi (z,sv). 3. If i = ek ∈ δ, since u and u are in the same orbit, bi (u) and bi (u ) have the same sign, thus bi (u ) = sign(bi (u ))zk = sign(bi (u))zk = bi (u). But ui − bi (u) depends only on u1, . . . ,ui−1 , therefore ui = ui , Pi (z,sw) = Pi (z,w). If i = 1 only cases 1. and 3. happen and the same argument proves that u1 = u1 . This proves our claim. We summarize the properties of P as follows: Proposition 2.3.9 Let  = e,h,δ be an ultrafine layer, and put Z = e∈e\δ R × e∈δ (0,∞). Then there is an analytic map P : Z ×  →  satisfying the following conditions. (1) P (z,·) is invariant: P (z,sv) = P (z,v) for all s ∈ G, z ∈ Z, and v ∈ . (2) P (z,v) = Q((z,v),v) where (·,v) : Z → Rd is an analytic bijection such that for each 1 ≤ k ≤ d, k (z,v) depends only on z1, . . . ,zk and v1, . . . ,vek . Thus z → P (z,v) is an analytic parametrization of the orbit Gv by Z. (3) If i ∈ / e, then Pi (z,v) = i (exp 1 (z1, . . . ,zr ,v)Xe1 (v) . . . exp r (z,v)Xer (v))vi + Yi (z1, . . . ,zr ,v1, . . . ,vi−1 ) where and er = max{e ∈ e : e < i}. (If i < e1 , then Y (z,v) = 0.) (4) If i = ek ∈ e \ δ, then Pi (z,v) = zk .

114

Stratification of an orbit space

(5) If i = ek ∈ δ, then Pi (z,v) = sign(bi (v))zk + Yi (z1, . . . ,zk−1,v1, . . . ,vi−1 ). (6) Let  be the cross-section of Proposition 2.3.5, part (4). Then P defines an analytic homeomorphism from Z ×  to . Moreover, the cross-section mapping  →  is given by v → P (z∗,v) where zk∗ = 0 if ek ∈ e \ δ and zk∗ = 1 if ek ∈ δ. Let  be an ultrafine layer with  the cross-section defined in Proposition 2.3.5. Observe that since the layer is realized as a product Z × via the map P , alternate smooth cross-sections are obtained by choosing a smooth function ϕ :  → Z. Then ϕ = {P (ϕ(σ ),σ ) : σ ∈ } is also a cross-section for the orbits in . Example 2.3.10 Let V = R2 and G = R. Consider the representation α of G defined by α(t) = exp tA where   λ 0 A= 1 . 0 λ2 Assume that λ1 λ2 = 0. The canonical basis (e1,e2 ) of eigenvectors for A is a good Jordan–Hölder basis in Vc = C2 . There are three ultrafine layers e,h,δ . (1) The minimal layer is e = {1}, h(1) = 1, δ = {1} when v1 = p1 (v) = 0. Then X1 (v) = (1/λ1 v1 )A and b1 (v) = v1 . The Q-parametrization is given by Q(t1,v) = exp t1 X1 (v)v so Q(t,v) = (et1 /v1 v1,e(λ2 /λ1 v1 )t1 v2 ). Put z1 = |b1 (Q1 (t1,v))| = e(1/v1 )t1 |v1 |, and we get Z = (0, + ∞) and P (z1,v) = (sign(v1 )z1,(z1 /|v1 |)λ2 /λ1 v2 ). The cross-section defined in part (4) of Proposition 2.3.5 is 1 = {P (1,v) : v ∈ e,h,δ } = {v ∈ R2 : v1 = ±1}. (2) The next layer in the layer ordering is given by e = {2},h(2) = 1,δ = {2}, when v1 = 0,v2 = 0. Here X1 (v) = (1/λ2 v2 )A and b2 (v) = v2 . We then have Q(t,v) = (0,et1 /v2 v2 ) and P (z,v) = (0,sign(v2 )z1 ). The crosssection for this second layer is 2 = {(0, ± 1)}. The last layer is of course ∅,(),∅ = {(0,0)}. Suppose that λ1 and λ2 have the same sign, and let  be the union of the first two layers:  = R2 \ {(0,0)}. Then /G is a compact Hausdorff space that is parametrized by 1 ∪2 , but 1 ∪2 is not a topological cross-section for /G. However,

2.3 Orbit structure for a completely solvable action

115

there is a smooth function ϕ such that the closure of the section ϕ in the minimal layer is a topological section for . Writing σ ∈ 1 as σ = (±1,σ2 ), define ϕ : 1 → (0, + ∞) by −1/2λ1  . ϕ(σ ) = 1 + σ22λ1 Then ϕ = {P (ϕ(σ ),σ ) : σ ∈ 1 } = {(v1,v2 ) : v1 = 0,v1λ2 + v2λ1 = 1}.  Example 2.3.11 We return to Example 2.3.7: the minimal layer contains two open orbits, its parameter set Z is (0, + ∞) × R, and the simplified parametrization is P (z1,z2,v) = (sign(v1 )z1,z2 ). The second layer is the last one: v1 = 0 and all orbits are singletons.



Example 2.3.12 We return to Example 2.3.8. Here z1 = v2 + t1 and 2 −1 z2 = |b3 (Q(t,v))| = et2 (v3 −v2 /2v1 ) |v3 − v22 /2v1 |, therefore the map  is: $

$

v2 (t1,t2 ) = (z1,z2,v1, . . . ,v4 ) = z1 − v2, v3 − 2 2v1

% $

z2 ln |v3 − v22 /2v1 |

%% .

We get now: P1 (z1,z2,v1,v2,v3,v4 ) = v1, P2 (z1,z2,v1,v2,v3,v4 ) = z1,

$

v2 P3 (z1,z2,v1,v2,v3,v4 ) = sign v3 − 2 2v1

% z2 +

z12 , 2v1

a/2

z2 P4 (z1,z2,v1,v2,v3,v4 ) = # v4 . # v 2 #a/2 # #v3 − 2v21 # Especially, if σ = (σ1,0,ε,σ4 ) ∈  (σ1 = 0, ε = ±1), we get: % $ z12 a/2 , z σ4 . P (z,σ ) = σ1, z1, εz2 + 2σ1 2



116

Stratification of an orbit space

Consider the minimal layer  = e,h,δ , with its cross-section  = e,h,δ . Recall that  is a Zariski open subset of V ; we now show that  is a submanifold of V with a natural global parametrization. For each ε ∈ {±1}δ , put ε = {v ∈  : be (v) = εe, e ∈ δ}. and ε = {v ∈  : sign(be (v)) = εe, e ∈ δ}. Then  is the disjoint union of the subsets ε , and ε =  ∩ ε is a cross-section for the orbits in ε . / e} and Vδ = span{fe : e ∈ δ}. Fix ε ∈ {±1}δ and Put V0 = span{fi : i ∈ recall that the cross-section is given by  = {P (z∗,v) : v ∈ } where z∗ ∈ Z is defined by 2 / δ, 0, if ek ∈ ∗ zk = 1, if ek ∈ δ. Each v ∈ ε satisfies ve = 0 for e ∈ e \ δ and if e ∈ δ, then ∗ ,v1, . . . ,vek −1 ). ve = εe + Ye (z1∗, . . . ,zk−1

Proceeding by recursion, this proves that for each e ∈ δ, there is a rational function Feε of the variables (vi )i ∈e, / i0 z1 ∈R

$ f

z2 a/2 x1,z1,εz2 + 1 ,z2 x4 2x1

% a/2

z2

dz1 dz2 .



The layerwise orbital description of Proposition 2.3.5 is the result of two procedures: (1) construction of a suitable coexponential basis for g(v) in g that depends rationally upon v ∈ , and (2) analysis of the resulting function Q that describes the orbits in . For completely solvable actions, the elements T (v)Zh(e) belong to g and provide the desired coexponential basis, and since the weights are real, the function Q assumes a relatively simple form. For a general finite-dimensional representation of a solvable Lie algebra g, the elements T (v)Zh(e) do not necessarily belong to g, and generalized weights need not be real-valued. The aim of the remainder of this chapter is to show that some of these procedures can be carried out more generally, albeit with extra complications. In the next section we will address the challenge of generalizing (1) by constructing for each layer, rationally varying elements as linear combinations of the elements T (v)Zh(e) that are almost real (i.e., belong to Cg). Then in Section 2.5, when the action is of exponential type, we modify these elements to obtain the desired coexponential basis as analytic functions defined on Zariski open subsets of the layer and then carry out the procedure (2), obtaining a generalization of Proposition 2.3.5. Finally, in Section 2.6, again in the exponential action case, we describe the minimal layer and the decomposition of the Lebesgue measure, generalizing Proposition 2.3.13.

120

Stratification of an orbit space

2.4 Construction of rational supplementary elements Let g be a solvable Lie algebra, and V a finite-dimensional g-module over R. Choose a good descending flag (Ui )0≤i≤n for U = Vc , and let I be the set of indices i such that Ui is real, that is I = {0 ≤ i ≤ n : U i = Ui }. / I then i − 1 ∈ I Thus (Ui )i being a good flag means that if i ≥ 1 and i ∈ and i + 1 ∈ I. Fix a basis (fi )1≤i≤n for U = Vc adapted to the flag for Vc ; / I, then recall that if both i and i − 1 belong to I, then fi ∈ V , while if i ∈ f i = fi+1 . Now identify Vc with Cn via the basis (fi )i . Recall pi : Vc → C is the coordinate map pi (u) = ui . For u = t [u1,u2, . . . ,un ] ∈ Vc = Cn , the  conjugate u = i ui f i of u in Vc satisfies ⎧ ⎪ if i ∈ I and i − 1 ∈ I, and ⎪ ⎨ui , (2.4.1) pi (u) = ui+1, if i ∈ / I, ⎪ ⎪ ⎩u , if i − 1 ∈ / I. i−1

Referring to (2.4.1), if both i and i − 1 belong to I then pi is a real map, while if i ∈ / I, then pi+1 (u) = pi (u). Now for any v ∈ V and Z ∈ s = gc , we have Zv = Zv, and hence if i ∈ / I, pi+1 (Zv) = pi (Zv),

Z ∈ s, v ∈ V .

(2.4.2)

The preceding relation will be used several times in this section. Having chosen the good basis (fi )ni=1 for U = Vc , recall the submodule Ui = span{fi+1, . . . ,fn }, and the projection πi : U → U/Ui . If i is in I, then πi is a real map, and the restriction of πi to V is the projection of V onto the real subspace Wi = span{f1, . . . ,fi } ∩ V , with kernel Vi = Ui ∩ V . We now define i  = max{t : t < i, t ∈ I}, and i  = min{t : t ≥ i, t ∈ I}. We shall study orbit spaces in an ultrafine layer  by considering the orbits in πi () for i ∈ I. For i ∈ I write πi (V )  Wi  × Zi ,

(2.4.3)

where dimR Zi = i − i  = 1 or 2. Identify Zi with R or C, writing πi (v) = (wi  ,zi ) accordingly, with zi = pi (v) if i − 1 ∈ I (zi ∈ R) or zi = pi−1 (v) if i−1∈ / I (zi ∈ C).

2.4 Construction of rational supplementary elements

121

Example 2.4.1 We recall Example 1.9.14 where β is an exponential type representation of e2 . In this example, I = {0,2,4}. So, if i = 2, then i  = 0, Wi  = 0 and Zi = spanR {e1,e2 }  C through the map z → Re(zf1 ) = Re(z(e1 − ie2 )), similarly if i = 4, then Wi  = spanR {e1,e2 } and Zi =  spanR {e3,e4 }  C through z → Re(zf3 ). Now fix v ∈ V , and let e = e(v). Recall that the indices e ∈ e are defined by the condition s(πe (v))  s(πe−1 (v)), and observe that this condition holds if and only if there is Z ∈ s(πe−1 (v)) for which pe (Zv) = 0. For e ∈ e, we have seen that if e ∈ I, then s(πe (v)) is real. However if e ∈ / I, then s(πe (v)) might be not real. A simple consequence of this observation is the following. Lemma 2.4.2 Let e ∈ e, and suppose that e − 1 ∈ / I. Then e − 1 ∈ e and s(πe−1 (v)) is not real. Proof: Since e ∈ e there is Z ∈ s(πe−1 (v)) \ s(πe (v)). Since e − 2 ∈ I, then s(πe−2 (v)) is real and both Z and Z belong to s(πe−2 (v)). But the relation (2.4.2) shows that pe−1 (Zv) = pe (Zv) = 0 which implies that Z ∈ / s(πe−1 (v)). Thus s(πe−1 (v)) is not real, s(πe−1 (v)) = s(πe−2 (v)), and e − 1 ∈ e. Lemma 2.4.2 says that the index set e is the disjoint union of the subsets e0 = {e ∈ e : e − 1 ∈ I and e ∈ I}, / I and e + 1 ∈ / e}, e1 = {e ∈ e : e ∈

(2.4.4)

/ I and e + 1 ∈ e}. e2 = {e,e + 1 : e ∈ e, e ∈ These distinct cases will arise several times in this section. For the moment, it is of interest to identify those e ∈ e for which s(πe (v)) is real. First consider the following example. Example 2.4.3 Consider the two-dimensional commutative Lie algebra g = span(X,Y ) with realization in gl(3,R): ⎡ ⎤ ⎡ ⎤ 1 0 0 0 1 0 Y = ⎣0 1 0⎦ . X = ⎣−1 0 0⎦ , 0 0 1 0 0 2

122

Stratification of an orbit space

Since the generalized weights assume the values i, −i and 2 on X, we need to use complex flags for U . Denote the canonical basis of R3 by (e1,e2,e3 ), and let v = e1 + e2 + e3 . Consider the index data for v with respect to distinct flags. We first put f1 = e1 −ie2 , f2 = f 1 , f3 = e3 , and define U1 = spanC {f2,f3 }, U2 = spanC {f3 }. Then (Ui )i is a good flag for Vc = C3 , and I = {0,2,3}. For this flag, s(π1 (v)) = {Z ∈ s : π1 (Zv) = 0} = C(X − iY ) and s(π2 (v)) = {0}, so that e(v) = {1,2}. Here s(π1 (v)) is not real and e(v) = e2 . Alternately, put g1 = e3 , g2 = e1 −ie2 , and g3 = g 2 , with U1 = spanC {g2,g3 }, U2 = spanC {g3 }. Again (Ui )i is a good flag, but here I = {0,1,3}, and s(π1 (v)) = C(X − 2Y ) and s(π2 (v)) = {0}. Again e = {1,2}, but in this case both s(π1 (v)) and s(v) are real, and now e = e0 ∪ e1 where e0 = {1} and e1 = {2}. Now suppose we want to use a decomposition of the real space V into V = π3 (V ) = W × Z, with Z irreducible, then in the first case, i  = 2 thus we shall write v = π3 (v) = (w2,z3 ), where w2 = e1 + e2 ∈ W2 , and z3 = 1 = p3 (v) ∈ Z3 = R. However, in the second case, i  = 1 and we write v = π3 (v) = (w1,z3 ), with w1 = e3 , z3 = 12 (1 + i) = p2 (v) ∈ Z3 = C.  The preceding example illustrates the following. Lemma 2.4.4 Let e ∈ e such that e − 1 ∈ I. Then s(πe (v)) is real if and only if e ∈ e0 ∪ e1 . Proof: If e ∈ e0 then we have observed that s(πe (v)) is real, while if e ∈ e1 , then e + 1 ∈ I and s(πe (v)) = s(πe+1 (v)) so again s(πe (v)) is real. Suppose that e ∈ e2 (and e − 1 ∈ I). Then by Lemma 2.4.2 with e in place of e − 1, s(πe (v)) is not real. Now fix a good Jordan–Hölder basis (Zj )1≤j ≤m in s = gc , with sj = span{Z1, . . . ,Zj }, s0 = {0}. Put J = {0 ≤ j ≤ m : sj = sj }, and for each 1 ≤ j ≤ m put: j  = max{t ∈ J : t < j },

j  = min{t ∈ J : t ≥ j }.

/ J , then j  = j + 1. In any case, If j ∈ J , then j  = j , while if j ∈ sj  = sj + sj .

2.4 Construction of rational supplementary elements

123

Recall (see Lemma 2.2.2) that h : e → {1,2, . . . ,m} is the map characterized by the relations sh(e) ∩ s(πe−1 (v)) ⊂ s(πe (v)),

and

sh(e)−1 ∩ s(πe−1 (v)) ⊂ s(πe (v)). (2.4.5)

First the configuration of the indices h(e), h(e + 1) is examined in the case where e ∈ / I and e, e + 1 ∈ e2 . / I. Then h(e + 1) ≥ h(e) . Lemma 2.4.5 Let e ∈ e2 such that e ∈ Proof: Suppose to the contrary that h(e + 1) < h(e) , so that h(e + 1) < h(e). Hence by the second condition (2.4.5), for h(e), sh(e+1) ∩ s(πe−1 (v)) ⊂ s(πe (v)), now by the first one, for h(e + 1), sh(e+1) ∩ s(πe (v)) ⊂ s(πe+1 (v)).   Let Z ∈ sh(e+1) ∩ s(πe (v)) \ s(πe+1 (v)). Then pe+1 (Zv) = 0, so by (2.4.2), pe (Zv) = pe+1 (Zv) = 0. But Z belongs to sh(e+1) ∩ s(πe−1 (v)), and this implies Z ∈ s(πe (v)), so pe (Zv) = 0, a contradiction. The preceding lemma means the following. Let us say that a pair (i,i + 1) with i ∈ / I, or a pair (j,j + 1) with j ∈ / J is a conjugate pair of indices. Suppose that (e,e + 1) is a conjugate pair of jump indices. Then {h(e), h(e + 1)} can define a conjugate pair of indices, and this is equivalent to h(e) = h(e + 1) . In this case both possibilities for the ordering of h(e) and h(e + 1) can appear, see for instance in the next example (Example 2.4.6) the first two layers, for e = 5. Lemma 2.4.5 says that if {h(e),h(e + 1)} does not define a conjugate pair, then h(e + 1) > h(e) ≥ h(e). Now let  = e,h,δ be an ultrafine layer. For each v ∈ , recall the normalizing m × m upper-triangular unipotent matrix T (v) defined in Section 2.2. Then v → T (v) is a rational map, regular on , and if e ∈ e, then T (v)Zh(e) belongs to s(πe−1 (v)) \ s(πe (v)). The element T (v)Zh(e) belongs to sh(e) , but generally does not belong to g. However, there is a construction that will suffice to extract an almost real element Ze (v) ∈ Cg from T (v)Zh(e) . We begin with an example.

124

Stratification of an orbit space

Example 2.4.6 Let g = spanR {X1,X2,A} where [A,X1 ] = −X1 + X2 , [A,X2 ] = −X1 − X2 and [X1,X2 ] = 0 with V = R6 and realization ⎤

⎡ ⎢ ⎢ ⎢ ⎢ X1 = ⎢ ⎢ ⎢ ⎣ 2 0 ⎡

2 ⎢ 0 ⎢ ⎢ ⎢ A=⎢ ⎢ ⎢ ⎣

0 −2

1 0

⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ , X2 = ⎢ ⎥ ⎢ ⎥ ⎢ ⎦ ⎣ 0 2

0 1

⎤

⎡

2 0

0 1 −1 0

⎥ ⎥ ⎥ ⎥ ⎥, ⎥ ⎥ ⎦

⎤

0 2 −2 2

2 2

1 1

−1 1

⎥ ⎥ ⎥ ⎥ ⎥. ⎥ ⎥ ⎦

Let e1, . . . ,e6 be the canonical basis of V ; a good descending basis for the standard action of g in Vc is f1 = e1 , f2 = e2 , f3 = e3 − ie4 , f4 = e3 + ie4 , f5 = e5 − ie6 , f6 = e5 + ie6 . A good Jordan–Hölder basis for s = gc is Z1 = X = X1 + iX2 , Z2 = X = X1 − iX2 , Z3 = A. Then A acts diagonally and Xf1 = 2f6,

Xf2 = 2if6,

Xf3 = 2f5,

Xf4 = 0,

Xf1 = 2f5,

Xf2 = −2if6,

Xf3 = 0,

Xf4 = 2f6 .

 Identify Vc with C6 so that for v ∈ V , v = 6i=1 vi fi = t [v1, . . . ,v6 ] with v1 , v2 real, v4 = v 3 , and v6 = v 5 . Put w = v1 − iv2 , then the matrix M(v) = [pi (Zj v)] is then: ⎡

0 ⎢ 0 ⎢ ⎢ ⎢ 0 M(v) = ⎢ ⎢ 0 ⎢ ⎣2v3 2w

0 0 0 0 2w 2v 3

⎤ 2v1 2v2 ⎥ ⎥ ⎥ 2(1 + i)v3 ⎥ ⎥ 2(1 − i)v 3 ⎥ ⎥ (1 + i)v5 ⎦ (1 − i)v 5

2.4 Construction of rational supplementary elements

125

and the ultrafine layers are listed in the table. (e,h)

δ

e,h,δ

{1,5,6}, (3,1,2)

{1}

v1 = 0,|v1 − iv2 |2 = |v3 |2 = 0

{1,5,6}, (3,2,1)

{1}

v1 = 0,v3 = 0

{2,5,6}, (3,1,2)

{2}

v1 = 0,v2 = 0,|v1 − iv2 |2 = |v3 |2 = 0

{2,5,6}, (3,2,1)

{2}

v1 = 0,v2 = 0,v3 = 0

{3,5,6}, (3,1,2)

{3}

v1 = v2 = 0,v3 = 0

{1,5}, (3,1)

{1}

v1 = 0,|v1 − iv2 |2 = |v3 |2 = 0

{2,5}, (3,1)

{2}

v1 = 0,v2 = 0,|v1 − iv2 |2 = |v3 |2

{5}, (3)

{5}

v1 = v2 = v3 = 0,v5 = 0

∅, ( )

∅

v=0

We focus on the layer  = {1,5},(3,1) . For the index e1 = 1 we have T (v)Zh(1) = Zh(1) = A is in g. Now consider e2 = 5, h(e2 ) = 1. Then T (v)Zh(e2 ) = X is not real. To describe the action of g corresponding to this jump index, we look for the action of its real and imaginary parts, namely, X1 and X2 , in the complex direction v5 . We get: p5 (X1 v) = w + v3,

p5 (X2 v) = i(w − v3 ).

Because |v3 | = |v1 − iv2 | = |w|, these complex numbers are R-linearly dependent. It follows that the actions of exp RX1 and exp RX2 produce the same orbit in this direction. Now suppose that we try to imitate the construction of Q(t,v) in Proposition 2.3.5. Since exp RA acts in the direction of v3 = ve1 , Xe1 (v) would be a real multiple rA of A. For the direction v5 = ve2 , we could choose Xe2 (v) = X1 , and put Q(t1,t2,v) = exp t1 rA exp t2 X1 v. The problem is that this cannot be

126

Stratification of an orbit space

done for all v in the layer  since for some v, p5 (X1 v) = 0. Observe however that the element Z5 (v) = p5 (X1 v)X1 + p5 (X2 v)X2 = (w + v3 )X1 + i(w − v3 )X2 = wX + v3 X is almost real since p5 (X1 v) and p5 (X2 v) are R-linearly dependent, and p5 (Z5 (v)v) = p5 (X1 v)2 + p5 (X2 v)2 = 4wv3 = 0 for all v in the layer. We shall see that a more convenient choice for Xe2 (v) will be a scalar multiple  of Z5 (v). Fix v ∈ , and let Z be any element of s(πe−1 (v)) \ s(πe (v)); observe that if e ∈ e0 , so that the coordinate map pe is real, then   1 (2.4.6) (pe (Zv) Z + pe (Zv) Z) = Re pe (Zv) Z 2 belongs to g(πe−1 (v)) = s(πe−1 (v)) ∩ g, and since pe is real,   pe (Re pe (Zv) Z v) = |pe (Zv)|2 = 0. So the formula (2.4.6) gives an element of g that belongs to g(πe−1 (v)) \ g(πe (v)). On the other hand, the element   1 (pe (Zv) Z − pe (Zv) Z) = Im pe (Zv) Z 2i belongs to s(πe (v)), since pe (pe (Zv) Zv − pe (Zv) Zv) = 0. Thus if e ∈ e0 , a g-valued rational function v → Ze (v) with Ze (v) ∈ s(πe−1 (v)) \ s(πe (v)) for all v ∈ , is easy to find: apply the formula (2.4.6) to the element Z = T (v)Zh(e) . It turns out that this construction is also useful when pe is not real. For 1 ≤ i ≤ n, and Z ∈ s, define  1 βi+ (Z,v) = pi (Zv) Z + pi (Zv) Z 2 and  1 βi− (Z,v) = pi (Zv) Z − pi (Zv) Z . 2i Observe that pi (βi− (Z,v)v) = 0. Write Z = X + iY where X, Y ∈ g. It is straightforward to verify that   2pi (Xv) = pi (Zv) + pi (Zv), 2pi (Y v) = i pi (Zv) − pi (Zv) , (2.4.7) and βi+ (Z,v) = pi (Xv)X + pi (Y v)Y,

βi− (Z,v) = pi (Xv)Y − pi (Y v)X.

If i ∈ / I, then an easy calculation using (2.4.2) shows that + − βi+1 (Z,v) = βi+ (Z,v), βi+1 (Z,v) = βi− (Z,v).

2.4 Construction of rational supplementary elements

127

Lemma 2.4.7 Let i ∈ / I, let Z ∈ s, and write Z = X + iY where X, Y ∈ g. Let v ∈ V and suppose that pi+1 (βi− (Z,v)v) = 0. Then the complex numbers pi (Xv) and pi (Y v) are R-linearly dependent, and hence both βi+ (Z,v) and βi− (Z,v) are almost real. Similarly, if − + − (Z,v)v) = 0, then βi+1 (Z,v) and βi+1 (Z,v) are almost real. pi (βi+1 Proof: Observe that:   − pi (βi+1 (Z,v)v) = pi βi− (Z,v)v = pi+1 (βi− (Z,v)v) and by the above relations, the second statement follows from the first one. We compute 0 = 2i pi+1 (βi− (Z,v)v) = pi (Zv)pi+1 (Zv) − pi (Zv)pi+1 (Zv). By (2.4.2), this yields pi (Zv)pi (Zv) − pi (Zv)pi (Zv) = 0,

or

|pi (Zv)|2 = |pi (Zv)|2 .

It is then immediate from the relations (2.4.7) that   Im pi (Xv)pi (Y v) = 0 which means that pi (Xv) and pi (Y v) are R-linearly dependent complex numbers. The following proposition applies the β ± -construction to the elements T (v)Zh(e) to construct almost real maps v → Ze (v), e ∈ e, as illustrated in Example 2.4.6. We shall use the fact that T (v)Zh(e) depends only on the coordinates v1, . . . ,ve−1 , thus the function v → pe (T (v)Zh(e) v) is: pe (T (v)Zh(e) v) = λe (T (v)Zh(e) )ve + pe (T (v)Zh(e) we−1 (v)),

(2.4.8)

where λe (T (v)Zh(e) ), we−1 (v) = πe (v) − pe (v)fe ∈ span{f1, . . . ,fe−1 } and pe (T (v)Zh(e) we−1 (v)) depend only on v1, . . . ,ve−1 . For ease of notation write β ± (·,v) = β ± (·). Proposition 2.4.8 There is a collection of rational mappings Ze :  → s (e ∈ e) with the following properties. (1) For each e ∈ e, Ze (v) is almost real, belongs to sh(e) and it depends only on v1, . . . ,ve . (If e = e + 1, then ve = v e , thus in what follows, we say that Ze (v) depends on v1, . . . ,ve .)

128

Stratification of an orbit space

(2) If e ∈ e0 ∪ e1 , then s(πe−1 (v)) = CZe (v) ⊕ s(πe (v)). / I, then s(πe−1 (v)) = spanC {Ze (v),Ze+1 (v)} + (3) If e, e + 1 ∈ e2 and e ∈ s(πe+1 (v)) and pe (Ze+1 (v)v) = ipe (Ze (v)v). Proof: Since e = e0 ∪ e1 ∪ e2 , it is enough to look at e ∈ e with e − 1 ∈ I. We consider the three cases e ∈ e0 , e ∈ e1 , and e ∈ e2 , separately. Case (0) Suppose that e ∈ e0 . Define Ze (v) = βe+ (T (v)Zh(e) ). Since e ∈ I, the relation (2.4.6) and Lemma 2.2.9 show that Ze (v) belongs to g(πe−1 (v))\g(πe (v)). Moreover Ze (v) depends only upon v1, . . . ,ve . Case (1) Suppose that e ∈ e1 . Since s(πe−1 (v)) is real, then Lemma 2.2.9 shows that βe± (T (v)Zh(e) ) belongs to s(πe−1 (v)), moreover we saw that pe (βe− (T (v)Zh(e) ) v) = 0, hence βe− (T (v)Zh(e) ) belongs to s(πe (v)). Since e+1 ∈ / e, then s(πe (v)) = s(πe+1 (v)), and hence pe+1 (βe− (T (v)Zh(e) )v) = 0. By Lemma 2.4.7, Ze (v) := βe+ (T (v)Zh(e) ) is almost real. Now one computes that pe (Ze (v)v) = 0. As in Case (0), Ze (v) depends on v1, . . . ,ve and is not in s(πe (v)). Case (2) Suppose that e, e + 1 ∈ e2 , e ∈ / I. Here s(πe (v)) is not real. Put     X(v) = Re T (v)Zh(e+1) , and Y (v) = Im T (v)Zh(e+1) . Recall T (v)Zh(e+1) belongs to s(πe (v)) \ s(πe+1 (v)). Since s(πe−1 (v)) is real and s(πe (v)) is not real, then T (v)Zh(e+1) ∈ s(πe−1 (v)) \ s(πe (v)), and it follows that s(πe−1 (v)) = spanC {X(v),Y (v)} + s(πe+1 (v)).

(2.4.9)

Observe that pe (Y (v)v) = ipe (X(v)v) = 0. If h(e) = h(e + 1) , then put Ze (v) = X(v), Ze+1 (v) = Y (v). Thus both / I, then ve+1 = v e , and Ze (v) Ze (v) and Ze+1 (v) belong to gh(e) . Since e ∈ depends only of v1, . . . ,ve . Suppose then that h(e) < h(e + 1) ; observe that in this case X(v) and Y (v) are not both in sh(e) . Consider the element T (v)Zh(e) belonging to sh(e) ∩ s(πe−1 (v)) \ s(πe (v)). Exactly as in Case (1), we find that pe+1 (βe− (T (v)Zh(e) ) v) = 0, then by Lemma 2.4.7, that Ze (v) := βe+ (T (v)Zh(e) )

2.4 Construction of rational supplementary elements

129

is almost real, and finally that Ze (v) ∈ sh(e) ∩ s(πe−1 (v)) \ s(πe (v)). Now we have a(v), b(v) ∈ C such that Ze (v) = a(v)X(v) + b(v)Y (v) mod s(πe+1 (v)).   Since Ze (v) is almost real, then Im a(v)b(v) = 0 holds for all v ∈ . Put Ze+1 (v) = −b(v)X(v) + a(v)Y (v). Then, these functions depend only on v1, . . . ,ve , and Ze+1 (v) ∈ s(πe−1 (v)) ∩ sh(e+1) , and Ze+1 (v) is almost real. From the Equation (2.4.9), the matrix   pe (Y (v)v) pe (X(v)v) pe+1 (X(v)v) pe+1 (Y (v)v) is invertible, and since a(v) or b(v)  and a(v) and b(v) linearly  is nonzero a(v) −b(v) dependent over R then the matrix b(v) a(v) is nonsingular. It follows that the matrix is nonsingular. It follows that the matrix   pe (Ze (v)v) pe (Ze+1 (v)v) pe+1 (Ze (v)v) pe+1 (Ze+1 (v)v)    pe (X(v)v) a(v) −b(v) pe (Y (v)v) = pe+1 (X(v)v) pe+1 (Y (v)v) b(v) a(v) is also invertible. This proves s(πe−1 (v)) = span{Ze (v),Ze+1 (v)} + s(πe+1 (v)). Finally, observe that in this case we still have pe (Ze+1 (v)v) = ipe (Ze (v)v). The procedure of Proposition 2.4.8 produces almost real elements by consideration of a relatively small number of cases. However, in very simple examples the elements Ze (v) constructed by this procedure are often multiples of the most natural choices. Example 2.4.9 We return to Example 2.2.17, defined by g = span{A,B}, Vc = U = span{f1,f2 = f 1 }, and Af1 = if1,Bf1 = f1 . The basis (f1,f2 ) is a good flag basis and the basis (A,B) is a good Jordan–Hölder basis for s = gc . For v = 0, we have e = e2 = {1,2} and h(1) = 1,h(2) = 2. Thus we are in Case (2) of the proof of Proposition 2.4.8. Put e = 1. Note that s(πe−1 (v)) = s while s(πe+1 (v)) = {0}. Here T (v)Zh(e) = T (v)A = A, but T (v)Zh(e+1) = T (v)B = B + iA; recall that T (v)B belongs to s(πe (v)) \ s(πe+1 (v)). In Case (2) we put X(v) = Re(B + iA) = B and Y (v) = Im(B + iA) = A. But since the basis for s = gc is real, we have h(e) = 1 and h(e + 1) = 2, so we are in the case where

130

Stratification of an orbit space

h(e) < h(e + 1) . Now Ze (v) = βe+ (T (v)Zh(e) ) = iv1 A, so continuing to follow the notation of the proof we have a(v) = 0,b (v) = iv1 , and hence Ze+1 (v) = −b(v)B + a(v)A = −iv1 B. Certainly, s = s(πe−1 (v)) = spanC {Ze (v),Ze+1 (v)} + s(πe+1 (v)). Observe also that pe (Ze (v)v) = −v12 while pe (Ze+1 (v)v) = −iv12 = ipe (Ze+1 (v)v).  The following example illustrates the utility of the construction of Proposition 2.4.8 when the elements T (v)Zh(e) are not real. Example 2.4.10 Consider again Example 2.4.6. Recall that v1 , v2 are real, v3 is complex, and we use the notation w = v1 − iv2 . The reader is referred to the original presentation for details of the action. First suppose that v lies in the minimal Zariski open layer {1,5,6},(3,1,2) . For the conjugate pair (e,e + 1) = (5,6), we get w T (v)Zh(e+1) = T (v)X = X − X. v3 Since h(e) = h(e + 1) , then Z5 (v) = Re(T (v)Zh(e+1) ), and Z6 (v) = Im(T (v)Zh(e+1) ), then: Z5 (v) =

1 Re(v 3 (v3 − w)X), |v3 |2

Z6 (v) = −

1 Im(v 3 (v3 + w)X). |v3 |2

For the next layer {1,5,6} , we still have the conjugate pair (e,e + 1) = (5,6), but now h(e + 1) = 1 so for v in this layer, T (v)Zh(e+1) = X, and Z5 (v) = X1,Z6 (v) = X2 . Finally suppose that v ∈ {1,5},(3,1) , characterized by the conditions v1 = 0 and |w| = |v3 | = 0, and where e = {1,5}. For e2 = 5 ∈ e1 , we get T (v)Zh(e) = X = X1 + iX2 , and Ze (v) = βe+ (X,v) = wX + v3 X = (w + v3 )X1 + i(w − v3 )X2 is almost real. Note that if w + v3 = 0, then sign(w + v3 )−1 Ze (v) is real. 

2.5 Orbit structure for an action of exponential type We retain all notation from Section 2.4: G is a simply connected solvable Lie group with Lie algebra g, α is a representation of G in a finite-dimensional vector space V over R, a good Jordan–Hölder basis for s = gc is chosen, and a good descending flag basis for U = Vc is chosen. Carry out the layering procedure of Section 2.2, and let  = e,h,δ be an ultrafine layer. The first task

2.5 Orbit structure for an action of exponential type

131

of the present section is to give a procedure for constructing, for each v ∈ , a coexponential basis for the annihilator subalgebra g(v) in g. Then, assuming that α is of exponential type, analogues of Propositions 2.3.5, 2.3.9, 2.3.13, as well as Lemma 2.3.14, are proved for exponential-type actions. The strategy for constructing a coexponential basis for g(v) is to use the almost real elements Ze (v) as constructed in the Proposition 2.4.8. Recall that Ze :  → s is a rational nonsingular map and that for each v ∈ , Ze (v) = a(v)X(v) + b(v)Y (v), where X(v), Y (v) belong to g, and where v → a(v) and v → b(v) are rational complex-valued functions on  that satisfy Im(a(v)b(v)) = 0, v ∈ . This means that Ze (v) is almost real: if fe (v) satisfies sign(fe (v)) = sign(a(v))−1 or sign(fe (v)) = sign(b(v))−1 , then fe (v)Ze (v) belongs to g. The strategy is to form the desired coexponential basis using the elements of the form Xe (v) = fe (v)Ze (v). Here is where a new complication arises: the functions a(v) and b(v) may vanish on parts of , as in Example 2.4.6. In this case v → Xe (v) is not defined everywhere on . With this caveat, the following construction gives, for each v ∈ , a coexponential basis for g(v) in g that varies smoothly in a large open set containing v. Proposition 2.5.1 There is a finite open covering F = F() of , and for each O ∈ F and e ∈ e, there are real analytic functions Xe = XeO : O → g with the following properties. (1) For all v ∈ O and e ∈ e, Xe (v) belongs to gh(e) . (2) Let e ∈ e0 ∪ e1 . Then g(πe−1 (v)) = RXe (v) ⊕ g(πe (v)). / I. Then (3) Let e,e + 1 ∈ e2 with e ∈ g(πe−1 (v)) = spanR {Xe (v),Xe+1 (v)} ⊕ g(πe+1 (v)). (4) If e ∈ e0 ∪ e1 , (Xe (v)) is a coexponential basis for g(πe (v)) in / I, then (Xe (v),Xe+1 (v)) is a g(πe−1 (v)), if e, e + 1 ∈ e2 with e ∈ coexponential basis for g(πe+1 (v)) in g(πe−1 (v)). (5) For each e ∈ e, |pe (Xe (v)v)| = 1, and if e ∈ e0 , pe (Xe (v)v) = 1. 

Moreover, if O  ∈ F, e ∈ e, and Xe = XeO , then for each e ∈ e, v ∈ O ∩ O  , there is e (v) ∈ {−1,1} such that Xe (v) = e (v)Xe (v). Proof: For each e ∈ e, we will write  = e,1 ∪ e,2 where e,t is Zariski open in , and we define, for each t = 1,2, an analytic function Xet : e,t → g. Then for each t : e → {1,2} put e,t (e) Ot = e∈e

132

Stratification of an orbit space

so that the family {Xet : e ∈ e} satisfies the conditions (1), (2), and (3) on O t . Then the collection F = {O t : t ∈ {1,2}e } fulfills the conditions of the proposition. Case (0): e ∈ e0 . Put Xe (v) = Ze (v)/pe (Ze (v)v); then it is immediate that Xe (v) belongs to gh(e) ∩ g(πe−1 (v)) \ g(πe (v)). In this case v → Xe (v) is rational on , and we define e,t = , t = 1,2, put Xet (v) = Xe (v). Case (1): e ∈ e1 . Recall that Ze (v) = βe+ (T (v)Zh(e) ) = pe (Y1 (v)v)Y1 (v) + pe (Y2 (v)v)Y2 (v), where Y1 (v) = Re T (v)Zh(e) and Y2 (v) = Im T (v)Zh(e) , and for each v ∈ , at least one of pe (Y1 (v) v) or pe (Y2 (v) v) is nonzero and they are R-linearly dependent complex numbers. Put e,t = {v ∈  : pe (Yt (v) v) = 0}. Define Xet : e,t → g by Xet (v) =

sign(pe (Yt (v) v))−1 Ze (v), t = 1,2. |pe (Ze (v)v)|

Then for v ∈ e,t , Xet (v) ∈ g ∩ sh(e) ∩ s(πe−1 (v)) = g(πe−1 (v)) ∩ gh(e) and / g(πe (v)). Xet (v) ∈ / I. If h(e) = h(e + 1) , then put Xe (v) = Case (2): e ∈ e2 , e ∈ Ze (v)/|pe (Ze (v)v)| and Xe+1 (v) = Ze+1 (v)/|pe (Ze (v)v)|, and it is immediate from Proposition 2.4.8 that Xe and Xe+1 satisfy the conditions (1) to (3). Here we put e,t = , t = 1,2. Suppose that h(e) < h(e + 1) . Then Ze (v) ∈ sh(e) and Ze+1 (v) ∈ sh(e+1) and by Proposition 2.4.8 we have real analytic functions X :  → g, Y :  → g, a :  → C, and b :  → C such that Ze (v) = a(v)X(v) + b(v)Y (v), Ze+1 (v) = −b(v)X(v) + a(v)Y (v). Define Xe (v) and Xe+1 (v) just as in Case (1): put e,1 = e+1,1 = {v ∈  : a(v) = 0}, e,2 = e+1,2 = {v ∈  : b(v) = 0} t : e+1,t → g by and define Xet : e,t → g and Xe+1

Xe1 (v) =

sign a(v) sign a(v) 1 (v) = Ze (v), Xe+1 Ze+1 (v), |pe (Ze (v)v)| |pe (Ze (v)v)|

Xe2 (v) =

sign b(v) sign b(v) 2 (v) = Ze (v), Xe+1 Ze+1 (v). |pe (Ze (v)v)| |pe (Ze (v)v)|

and

2.5 Orbit structure for an action of exponential type

133

t t (v)v) = p (X t (v)v), and |p t Note that, Xe+1 is real, pe+1 (Xe+1 e e+1 (Xe+1 e+1 (v)v)| = 1.   Let O t , O t ∈ F, and let e ∈ e. Then by construction, Xet (v) and Xet (v) are determined by the sign of two complex numbers which are R-linearly dependent. Thus they coïncide up to a real sign ±1. Our construction gives condition (5) for each e. It remains to prove condition (4) . For this we use Corollary 1.7.16. First let us fix a covering set O, and v ∈ O. We now define a weak Malcev basis (Xj )1≤j ≤m for g, containing each Xe (v).

1. Suppose j ∈ J and j − 1 ∈ J : For j ∈ / h(e), put Xj = Re(T (v)Zj ), for j ∈ h(e), and j = h(e), put Xj = Xe (v). 2. Suppose j ∈ / J. If {j,j + 1} ∩ h(e) = {h(e)}, put X = Re(T (v)Zj ), Y = Im(T (v)Zj ), and Xj = Xe (v) = a(v)X + b(v)Y, and Xj +1 = −b(v)X + a(v)Y . If {j,j + 1} ∩ h(e) = {h(e),h(e )}, with e < e , put Xj = Xe (v), Xj +1 = Xe (v). If {j,j + 1} ∩ h(e) = ∅, choose Xj = Re(T (v)Zj ), Xj +1 = Im(T (v)Zj ). Then, by Lemma 1.7.8, (X1, . . . ,Xm ) is a weak Malcev basis of g, containing all the Xe (v). Now fix e ∈ e. 1. If e ∈ e0 ∪ e1 , consider the subalgebras h = g(πe (v)) and k = g(πe−1 (v)), then dim k/h = 1. By construction, the above weak Malcev basis contains / h(e) or j = h(e ), e > e} of h and the weak Malcev basis Bh = {Xj : j ∈ Bh ∪ {Xh(e) } is a weak Malcev basis of k. Hence Corollary 1.7.16 says that (Xe (v)) is a coexponential basis for h in k. 2. Suppose {e,e + 1} ⊂ e2 and e − 1 ∈ I. Put k = g(πe−1 (v)), and h = g(πe+1 (v)), then dim k/h = 2 and the same argument proves that (Xe (v),Xe+1 (v)) is a coexponential basis for h in k. This ends the proof of the proposition. Remark 2.5.2 Let i ∈ I and π : U → U/Ui . We saw that for each layer  = e,h,δ , π() ⊂  = e,h,δ  with e = {e ∈ e : e ≤ i} = {e1 < · · · < er }, h = h|e and δ  = δ ∩ e . On the other hand, we saw that if v ∈ , then the matrix M(π(v)) is simply M(v)1,2,...,i , and T (π(v)) = T1 (v)T2 (v) . . . Tr (v). Therefore, for each e ∈ e , T (v)Zh(e) = T (π(v))Zh (e) (Corollary 2.1.5). Now, the construction of the open sets O in the covering F is given by O = ∩e,t (t = 1,2), where e,t =  except possibly in cases where e ∈ e1 \δ,

134

Stratification of an orbit space

or where e ∈ e2 and h(e) < h(e + 1) . In each of these cases, the proof shows that, for each e and t, there is a real analytic function f e,t , depending only on v1, . . . ,ve−1 such that: e,t = {v ∈  : f e,t (v) = 0}. In the other cases, put f e,t = 1, then O = {v ∈  : e∈e,t∈{1,2} f e,t (v) = 0}, e,t and π(O) ⊂ O  , with O  = {v ∈  : e∈e,t∈{1,2} f (v) = 0}.  Finally, for any e ∈ e , the definition of the map v → XeO (v) (O ∈ F) gives  that XeO (π(v)) = XeO (v). Although applied to general representations of exponential type, we exhibit Proposition 2.5.1 in nonexponential type examples also. Example 2.5.3 Return again to Example 2.2.17 and 2.4.9, where g = span {A,B}, Vc = U = span{f1,f2 = f 1 }, and Af1 = if1,Bf1 = f1 . Retaining the notation already set out, consider the minimal ultrafine layer  = e,h,δ where e = {1,2}, h = (1,2), and δ = {1,2}. As before write e = 1,e + 1 = 2. Recall that Ze (v) = βe+ (T (v)Zh(e) ) = iv1 A and Ze+1 (v) = −iv1 B. Referring to Case (2) of the proof of Proposition 2.5.1, we see that e,1 = ∅ and e,2 = . Thus for all v ∈  we have b(v) = iv1 = 0 and Xe (v) = Xe2 (v) = sign(iv1 )

Ze (v) 1 = A, |pe (Ze (v)v)| |v1 |

while 2 (v) = sign(iv1 ) Xe+1 (v) = Xe+1

Ze+1 (v) 1 =− B. |pe (Ze (v)v)| |v1 |

Observe that in this example g(v) = {0} but G(v) = exp 2π ZA = exp 2π |v1 |  ZXe (v). Our goal now is to generalize the orbit description given in Proposition 2.3.5 for an action of completely solvable type to the case where α is of exponential type. We continue to assume that we have chosen a good Jordan–Hölder basis for s = gc , and a good descending flag (Ui ) with basis (fi ), by using the construction of Proposition 1.3.7. Perform the layering procedure as always, and fix an ultrafine layer  = e,h,δ . Recall also that when α is of exponential type, then by Lemma 2.2.16, δ ⊂ e0 ∪ e1 . Now carry out the procedure of Proposition 2.5.1: fix O ∈ F, and v ∈ O, we often write simply Xe (v) := XeO (v). Now let i ∈ I. By Proposition 1.9.13, the stability group G(πi (v)) = {s ∈ G : sπi (v) = πi (v)} is connected. In the Lie algebra, s(πi (v)) is real, and hence the closed connected subgroup

2.5 Orbit structure for an action of exponential type

135

corresponding to g(πi (v)) = s(πi (v)) ∩ g is G(πi (v)). The following is an immediate consequence of Proposition 2.5.1. Corollary 2.5.4 Suppose that α is of exponential type, and assume all bases in place, let e,h,δ be an ultrafine layer, and carry out the construction of Proposition 2.5.1. Write e = {e1 < e2 < · · · < ed } as usual, and fix i ∈ I. Assume that i > e1 and put er = max{e ∈ e : e ≤ i}. Then the map: (t1,t2, . . . ,tr ) → exp t1 Xe1 (v) exp t2 Xe2 (v) · · · exp tr Xer (v)G(πi (v)) is a diffeomorphism of Rr with G/G(πi (v)). As in the completely solvable case, for 1 ≤ k ≤ d put gk (t,v) = exp t1 Xe1 (v) exp t2 Xe2 (v) · · · exp tk Xek (v). We may also write g(t,v) = gd (t,v). Example 2.5.5 Recall the group G of Examples 2.4.6 and 2.4.10. Here the representation α is of exponential type. Let  be the layer with e = {1,5}, h = (3,1), and δ = 1. Recall that T (v)Zh(e1 ) = A, and since e1 = 1 ∈ e0 , then following Case (0) of the proof of Proposition 2.5.1 we get Xe1 (v) = (2v1 )−1 A. Now put e2 = 5 ∈ e1 \ δ and we observed in Example 2.4.10 that Ze2 (v) = β5+ (X,v) = (w + v3 )X1 + i(w − v3 )X2 . Following Case (1) of the construction of Proposition 2.5.1 with e = 5, put O 1 = 5,1 = {v ∈  : w + v3 = 0}, and O 2 = 5,2 = {v ∈  : w − v3 = 0}. For v ∈ O 1 we have sign(w + v3 ) 1  2 Im(wv3 )  X51 (v) = |X + Ze2 (v) = X2 |w + v 3 1 |pe2 (Ze2 (v)v)| |w + v3 | 2|v3 |2 while for v ∈ O 2 , X52 (v) = −i

 sign(w − v3 ) 1  2 Im(wv3 ) Ze2 (v) = X1 + |w − v3 |X2 . 2 |pe2 (Ze2 (v)v)| |w − v3 | 2|v3 |

One computes that v ∈ O 1 ∩ O 2 if and only if Im(wv3 ) = 0 and then X52 (v) =  (v)X51 (v) where (v) = sign(Im(wv3 )). We continue to assume that α is of exponential type, with good Jordan– Hölder basis for s = gc and good descending basis (fi ) for U = Vc in place. Fix an ultrafine layer , fix a covering set O ∈ F(), and let Xe = XeO , e ∈ e be the analytic functions constructed on O in Proposition 2.5.1.

136

Stratification of an orbit space

Now, suppose that v ∈ O and e ∈ δ, so that λe (Xe (v)) = 0. As in the completely solvable case, write pe (v) = ve = z, and w = πe (v) − zfe , and the affine action ϕw of G(w) defined by: pe (exp tXe (v)v) = ϕw (exp tXe (v))z

(2.5.1)

satisfies: ϕw (exp tXe (v))z = e (exp tXe (v))(z − z0 (w)) + z0 (w)

(2.5.2)

e (w)w) where z0 (w) = − pλee(X (Xe (w)) is the fixed point of ϕw . The nonvanishing rational function be :  → C is now:

be (v) = ve − z0 (w) =

pe (Xe (v)v) . λe (Xe (v))

(2.5.3)

Observe that z0 (w) and be (v) are independent of the choice of the covering set O such that v ∈ O. Lemma 2.3.4 still holds, with the same proof: Lemma 2.5.6 Let e ∈ δ and let v ∈ . Suppose that A ∈ g(πe−1 (v)) \ g(πe (v)). Then be (v) = pe (Av)/λe (A). Moreover, the complex function be is semi-invariant with multiplier e . By Corollary 2.5.4, for each v ∈ O, the map: (t1,t2, . . . ,td ) → exp t1 XeO1 (v) exp t2 XeO2 (v) · · · exp td XeOd (v)G(v) is a diffeomorphism of Rd with G/G(v). Define Q = QO : Rd × O →  by QO(t1,t2, . . . ,td ,v) = exp t1 XeO1 (v) exp t2 XeO2 (v) · · · exp td XeOd (v)v = gd (t,v)v (2.5.4) and let Qi = pi ◦ Q be the i-th coordinate function for Q. Observe that, if i∈ / I, then Qi+1 (t,v) = Qi (t,v). Definition 2.5.7 Given a covering set O of  and v ∈ O, if e ∈ e, but e∈ / e2 ∩ I, we define the function ζe = ζeO by: ζe (v) = pe (Xe (v)v), if e ∈ e2 ∩ I, we put: ζe (v) = pe−1 (Xe (v)v) (= pe (Xe (v)v)). Proposition 2.5.8 Let α be a representation of exponential type of a simply connected solvable Lie group G acting in a vector space V over R with dim V = n. Choose good Jordan–Hölder and descending flag bases and let

2.5 Orbit structure for an action of exponential type

137

 = e,h,δ be an ultrafine layer, let O be a covering set as in Proposition 2.5.1, and let Q = QO be the orbit parametrization of (2.5.4). Assume that e = ∅. Then Q(t,v) satisfies the following. (1) The function Q : Rd × O → V is analytic on Rd × O. (2) For each v ∈ O, Q(·,v) is a diffeomorphism of Rd with the orbit Ov of v. (3) Let 1 ≤ i ≤ n, such that i − 1 ∈ I. The following, with the relation / I, defines all the Qi . Qi+1 = Qi if i ∈ (3.1) If i ∈ / e, then Qi has the form: Qi (t,v) = i (gr (t,v))vi + Yi (t1, . . . ,tr ,v1, . . . ,vi−1 ), where er = max{e ∈ e : e < i}. (If i < e1 , this means Qi (t,v) = vi .) (3.2) If i = ek ∈ (e0 ∪ e1 ) \ δ, then |ζek (v)| = 1, if i ∈ e0 , ζek (v) = 1, and   Qi (t,v) = i (gk−1 (t,v)) vi + tk ζek (v) + Yi (t1, . . . ,tk−1,v1, . . . ,vi−1 ). (3.3) If i = ek ∈ e2 \ I, then ζek (v) and ζek+1 (v) are both magnitude one, orthogonal complex numbers, and   Qi (t,v) = i (gk−1 (t,v)) vi + tk ζek (v) + tk+1 ζek+1 (v) + Yi (t1, . . . ,tk−1,vi , . . . ,vi−1 ). (3.4) If i = ek ∈ δ, then Qi (t,v) = i (gk (t,v))bi (v) + Yi (t1, . . . ,tk−1,vi , . . . ,vi−1 ). Proof: For all s ∈ G, p1 (sv) = 1 (s)v1 . Thus we have 1 ∈ e if and only / e then Q1 (t,v) = v1 , while if 1 ∈ e, then if 1 = 1 and v1 = 0. If 1 ∈ 1 ∈ δ, g(π1 (v)) = ker λ1 and G(π(v)) = ker 1 . It is clear that in this case b1 (v) = p1 (Zh(1) v)/λ1 (Zh(1) ) = v1 and hence Q1 (t,v) = 1 (exp t1 X1 (v))v1 = et1 λ1 (X1 (v)) v1 so the proposition holds in the case i = 1. Fix any i, 1 < i ≤ n such that i − 1 ∈ I. Assume that the Qj (t,v) adhere to the expressions above, for any j < i. Observe that, if i ∈ / e, and er = max{e ∈ e : e < i}, then Qi (t,v) = pi (exp t1 Xe1 (v) . . . exp tr Xer (v)v) and, if i = ek ∈ e, then Qi (t,v) = pi (exp t1 Xe1 (v) . . . exp tk−1 Xek−1 (v)(exp tk Xek (v)v)).

(2.5.5)

138

Stratification of an orbit space

Put W = spanC {f1,f2, . . . ,fi−1 } ∩ V and write w = w(v) = πi (v) − vi fi ∈ W . First suppose that i ∈ / e, put Yi (t,v) = pi (exp t1 Xe1 (v) . . . exp tr Xer (v)w). Since, for each e, Ze (v) depends only to v1, . . . ,ve , the same is true for Xe (v), and Yi depends only on v1, . . . ,vi−1 . Then Qi (t,v) = pi (exp t1 Xe1 (v) . . . exp tr Xer (v)v) = pi (exp t1 Xe1 (v) . . . exp tr Xer (v)(vi fi + w)) = i (exp t1 Xe1 (v) . . . exp tr Xer (v))vi + pi (exp t1 Xe1 (v) . . . exp tr Xer (v)w) = i (exp t1 Xe1 (v) . . . exp tr Xer (v))vi + Yi (t,v) and we have the required form for part (3.1). Now suppose that i = ek ∈ e and i − 1 ∈ I. Put G(w) = {s ∈ G : w(sv) = w(v)} = G(πi−1 (v)). Let ϕ = ϕw : G(w) → Aff(pi (V )) be the natural map defined by ϕ(s)vi = i (s)vi + pi (sw). By Proposition 2.5.1 we have G(w) = exp(RXek (v))G(πek (v)),

or

G(w) = exp(RXek (v)) exp(RXek+1 (v))G(πek+1 (v)), according as e ∈ e0 ∪ e1 or e ∈ e2 . As above, for any s ∈ G(w), we can write pi (exp t1 Xe1 (v) . . . exp tk−1 Xek−1 (v)sv) = i (exp t1 Xe1 (v) . . . exp tk−1 Xek−1 (v))pi (sv) + Yi0 (t,v)   = i (exp t1 Xe1 (v) . . . exp tk−1 Xek−1 (v)) ϕ(s)vi + Yi0 (t,v), (2.5.6) where Yi0 (t,v) = pi (exp t1 Xe1 (v) . . . exp tk−1 Xek−1 (v)w). It is clear that this function depends only on v1, . . . ,vi−1 and t1, . . . ,tk−1 . Case (1): Suppose that i = ek ∈ e0 ∪ e1 \ δ so that G(w) = exp(RXek (v)) G(πek (v)), and hence ϕ(exp tk Xek )vi = vi + tk pi (Xek (v)v), using Relation (2.5.5), this gives the result (3.2), with Yi = Yi0 .

2.5 Orbit structure for an action of exponential type

139

Case (2): i = ek ∈ e2 \I, then G(w) = exp RXek (v) exp RXek+1 (v)G(πek+1 (v)). ϕ(exp tk Xek (v) exp tk+1 Xek+1 (v))vi = vi +tk pi (Xek (v)v)+tk+1 pi (Xek+1 (v)v). We get similarly the result of (3.3), with Yi = Yi0 . Case (3): Suppose that i = ek ∈ δ . Here we have G(w) = exp RXek (v) G(πek (v)). We know that z0 (w) = vi − bi (v) is the unique point in pi (V ) that is fixed by ϕ(G(w)) and ϕ(exp tk Xek (v))vi = etk λi (Xek (v)) (vi − z0 (w)) + z0 (w) and so, by (2.5.5), Qi (t,v) = i (exp t1 Xe1 (v) . . . exp tk−1 Xek−1 (v))etk λi (Xek (v)) bi (v) + Yi (t,v) where Yi (t,v) = i (exp t1 Xe1 (v) . . . exp tk−1 Xek−1 (v))z0 (w)+Yi0 (t,v), it still depends only on v1, . . . ,vi−1 and t1, . . . ,tk−1 . Thus in each case Qi has the required form. We repeat this analysis for each i such that i − 1 ∈ I, and (3) obtains. Since the map (tk ,v) → exp tk Xek (v) is analytic on O (see Proposition 2.5.1), then (t,v) → Q(t,v) is analytic on Rd × O, this proves Point (1). Point (2) is a consequence of Corollary 2.5.4. This achieves the proof of the proposition. Example 2.5.9 We return again to Example 2.4.6, 2.4.10, 2.5.5: there g = spanR {X1,X2,A}, where [A,X] = −(1 + i)X, [A,X] = −(1 − i)X, and [X,X] = 0. Recall Vc = C6 has the basis (fi )i chosen there, with the basis (Z1 = X, Z2 = X, Z3 = A) for gc . (1) The open layer  with e = {1,5,6}, h = (3,1,2), and δ = {1}, is the set of v satisfying v1 = 0, |w|2 = |v3 |2 = 0. Then T (v)Zh(e1 ) = Z3 = A, and following the method of Proposition 2.5.1, we find: X1 (v) =

A . 2v1

Now T (v)Zh(e2 ) = T (v)X = X and T (v)Zh(e3 ) = T (v)X = X − (w/v3 )X. Here e2 = 5, e3 = 6 belong to e2 . Moreover, h(5) = h(6) = 2. In this case, we defined: Z5 (v) =

    1 1 Re (v3 − w)v 3 X , Z6 (v) = − Im (v3 + w)v 3 X . 2 2 |v3 | |v3 |

140

Stratification of an orbit space

Now p5 (Z5 (v)v) = v3 (1 − |w|2 /|v3 |2 ) =

v3 (|v3 |2 − |w|2 ), |v3 |2

does not vanish on , and p5 (Z6 (v)v) = ip5 (Z5 (v)v). Therefore: Z5 (v) 1 # Re [(v3 − w)v 3 X] # = |p5 (Z5 (v)v)| |v3 | #|v3 |2 − |w|2 # 1 # Im [(v3 + w)v 3 X] . # X6 (v) = − # |v3 | |v3 |2 − |w|2 #

X5 (v) =

Put Q(t1,t2,t3,v) = exp t1 Xe1 (v) exp t2 Xe2 (v) exp t3 Xe3 (v)v. Then using only the coordinates v1,v2,v3,v5 (recall v4 = v 3 , v6 = v 5 ) we get Q(t1,t2,t3,v)   = et1 /v1 v1,et1 /v1 v2,et1 (1+i)/v1 v3,et1 (1+i)/(2v1 ) [v5 + ζ5 (v)(t2 + it3 )] , with ζ5 (v) = sign(p5 (X5 (v)v)) = sign(v3 (|v3 | − |w|)). (2) The singular layer , where e = {1,5}, h = (3,1), and δ = {1}, is the set of v ∈ V such that v1 = 0, |w|2 = |v3 |2 = 0. Again we have T (v)Zh(e1 ) = A and T (v)Zh(e2 ) = X. Again we have X1 (v) = A/(2v1 ), b1 (v) = v1 . We saw in Example 2.4.10 that Ze2 (v) = β5+ (X,v) = (w + v3 )X1 + i(w − v3 )X2 . Put O 1 = {v ∈  : w + v3 = 0}, O 2 = {v ∈  : w − v3 = 0}. For v ∈ O 1 we saw in Example 2.5.5   1 2 Im(wv3 ) 1 X5 (v) = X2 |w + v3 |X1 + |w + v3 | 2|v3 |2 while for v ∈ O 2 , X52 (v) =

1 2|v3 |2



 2 Im((w)v3 ) X1 + |w − v3 |X2 . |w − v3 |

In this case, Qν (t1,t2,v) = exp t1 X1 (v) exp t2 X5ν (v)v (ν = 1,2); then with the same identification as above,   Qν (t1,t2,v) = et1 /v1 v1,et1 /v1 v2,et1 (1+i)/v1 v3,et1 (1+i)/(2v1 ) [v5 + ζ5ν (v)t2 ] ,

2.5 Orbit structure for an action of exponential type

141

with ζ5ν (v) = sign(p5 (X5ν (v)v)) 2 sign(w + v3 )sign(v3 w) = sign(w + v3 ) = −i sign(w − v3 )sign(v3 w) = sign(i(w − v3 ))

if ν = 1, if ν = 2. 

Recall that in Proposition 2.3.5, the function Q(t,v) allowed us to immediately give an explicit cross-section for the orbits in . To generalize this to the exponential type case, we need the following lemma. Lemma 2.5.10 Let v ∈  and suppose that e ∈ e1 \ δ. Define Fe :  → C by Fe (v) = pe (T (v)Zh(e) v)pe+1 (v) + pe+1 (T (v)Zh(e) v)pe (v). Let O ∈ F, with v ∈ O, and let Xe (v) = XeO (v) be the supplementary element as in Proposition 2.5.1, then Re(pe (v)pe (Xe (v)v)) = 0 if and only if Fe (v) = 0. Proof: Write T (v)Zh(e) = Y1 (v) + iY2 (v) where Yt (v) ∈ g. Since e ∈ e1 \ δ, pe (Y1 (v)v) and pe (Y2 (v)v) are R-linearly dependent. Moreover, for any choice of O, Xe (v) is a real linear combination of Y1 (v) and Y2 (v): Xe (v) = a1 (v)Y1 (v) + a2 (v)Y2 (v). Now pe (Xe (v)v) is nonvanishing, and it is a basis for the real subspace of C generated by the pe (Yt (v)v): for each t = 1,2, pe (Yt (v)v) is a real multiple pe (Xe (v)v). This implies that Re(ve pe (Xe (v)v)) is vanishing if and only if Re(ve pe (Y1 (v)v)) = Re(ve pe (Y2 (v)v)) = 0. Put z = pe (T (v)Zh(e) v) and w = pe (T (v)Zh(e) v). Then 2pe (Y1 (v)v) = w + z,

2pe (Y2 (v)v) = i(w − z).

Now we have Re(v e pe (Xe (v)v)) = 0 if and only if: ⎧ ⎪ ⎨ 2Re(v e pe (Y1 (v)v)) = Re(v e w) + Re(v e z) = 0 and ⎪ ⎩ 2Re(v e pe (Y2 (v)v)) = −Im(v e w) + Im(v e z) = 0.

(2.5.7)

Now (2.5.7) is equivalent to (v e w) + v e z = 0, and to: Fe (v) = pe (T (v)Zh(e) v)pe+1 (v) +pe+1 (T (v)Zh(e) v)pe (v) = ve w + v e z = 0.

142

Stratification of an orbit space

A topological cross-section is defined, independently of the choice of the covering sets O, as follows: Proposition 2.5.11 Let  = e,h,δ be an ultrafine layer and for e ∈ e define Fe :  → C by Fe (v) ⎧ ⎪ if e ∈ (e0 \ δ) ∪ e2, ⎪ ⎨pe (v), = pe+1 (T (v)Zh(e) v)pe (v)+pe (T (v)Zh(e) v)pe+1 (v), if e ∈ e1 \ δ, ⎪ ⎪ ⎩|b (v)| − 1, if e ∈ δ. e Then  = {v ∈  : Fe (v) = 0 for all e ∈ e} is a topological cross-section for the G-orbits in . Proof: We proceed by induction on the number of elements of I (recall 0 ∈ I, thus this number is at least 2). We may assume that the action of G is nontrivial. Suppose first that dim V = 1 so I = {0,1}: then V = Rf1 , λ1 = 0 and if v = 0, g(v) = ker λ1 . Suppose v = 0, then e(v) = δ(v) = {1}, moreover / [g,g], Zh(1) is real, and b1 (v) = p1 (Zh(1) v)/λ1 (Zh(1) ) = v1 , finally Zh(1) ∈ X1 (v) = Zh(1) /(v1 λ1 (Zh(1) )). Hence, for any t ∈ R, Q1 (t,v) = et/v1 v1 . There is a unique t (v) ∈ R such that |b1 (exp tX1 (v)v)| = 1. Now  = {v : F1 (v) = 0} = {±f1 } and there is exactly one point of the orbit of v belonging to . Similarly, if dim V = 2 and I = {0,2}, then for v = 0, e = δ = {1}, / [g,g] and Zh(1) is real, b1 (v) = v1 . But in this case, λ1 = 0 has the Zh(1) ∈ form Re(λ1 )(1 + ia), and X1 (v) = Zh(1) /|v1 λ1 (Zh(1) )|. Thus Q1 (t,v) = etλ1 (X1 (v)) v1 and since λ1 (Zh(1) ) is not purely imaginary, there is unique t (v) ∈ R such that |b1 (exp t (v)X1 (v) v)| = 1. Thus there is exactly one point in the orbit of v that belongs to  = {v ∈  : |b1 (v)| = 1}. Suppose now that |I| > 2, and let n = max{1 ≤ i < n} ∩ I. Let Z = Un ∩ V = Vn , and let π = πn : V → V /Z (recall n = dim V ). Recall also Remark 2.5.2: 1. π() is contained in the ultrafine layer  = e,h,δ  in V /Z where e = {e1 < e2 < · · · < er } = {e ∈ e : e ≤ n },

2.5 Orbit structure for an action of exponential type

143

2. for any O ∈ F, there is O  in F  such that π(O) ⊂ O  ,  3. for each e ∈ e , XeO (v) = XeO (π(v)). Then for any t = (t1, . . . ,td ), if t  = (t1, . . . tr ), gr (t ,π(v))π(v) = exp t1 Xe1 (π(v)) . . . exp tr Xer (π(v))π(v) = exp t1 Xe1 (v) . . . exp tr Xer (v)π(v) = exp t1 Xe1 (v) . . . exp td Xed (v)π(v) = π(gd (t,v)v), therefore Q(t ,π(v)) = π(Q(t,v)). By induction, the result holds for  ; thus we have   = { t [v1, . . . ,vn ] ∈  : Fe (v1, . . . ,vn ) = 0, e ∈ e } is a cross-section for the orbits in  , and Fe is defined by the above prescription. Observe that for each e, the function Fe depends only upon v1, . . . ,ve ; hence Fe = Fe for all e ∈ e , and π() ⊂   .

(2.5.8)

Let v ∈ ; let O ∈ F such that v ∈ O. Let d be the dimension of the G-orbit of v, and d  the dimension of the G-orbit of π(v). By induction we  have a unique t  (π(v)) = (t1 (π(v)), . . . ,td  (π(v))) ∈ Rd such that, for any t (v) = (t1 (v), . . . ,td (v)) such that tk (v) = tk (π(v)), (k ≤ d  ), π(Q(t (v),v)) = Q (t  (π(v)),π(v)) ∈   . Case (1): d  = d. That means t  (π(v)) = t (v), and by definition of  the equations of  and   are the same,  = π −1 (  ) ∩ , hence Q(t (v),v) ∈ . Case (2): Suppose that n = n − 1 and n = ed . Observe that d  = d − 1 and  = {v ∈  : π(v) ∈  , and Fn (v) = 0}. Exactly as in the completely solvable case (see the proof of Proposition 2.3.5), we have  = π −1 (  ) ∩ {v ∈  : Fn (v) = 0}, and we look for solution td = td (v) to Equation (2.3.7) ⎧ Qed (t1 (v), . . . ,td−1 (v),td ,v) = 0 if n = ed ∈ e \ δ, ⎪ ⎨ |bed (exp t1 (v)Xe1 (v) . . . exp td−1 (v) ⎪ ⎩ Xed−1 (v) exp td Xed (v)v)| = 1 if n = ed ∈ δ. This equation has clearly one and only one solution td (v).

144

Stratification of an orbit space

Case (3): Suppose that n = n − 2 and e ∩ {n − 1,n} = ∅. There are unique real numbers t1 (v), . . . ,td−1 (v) such that g0 π(v) ∈   , where g0 = g0 (v) = exp t1 (v)Xe1 (v) . . . exp td−1 (v)Xed−1 (v). Choose O0 ∈ F such that g0 v ∈ O0 . We separate into subcases. Case (3a): ed = n − 1 ∈ e1 \ δ. Put g(t) = g0 exp tXed (v)g0−1 , we have: pn−1 (g(t)g0 v) = pn−1 (g0 v) + tpn−1 (Ad(g0 )Xed (v)g0 v). But, Xed (v) ∈ g(π(v)) \ g(v), thus Ad(g0 )Xed (v) ∈ g(π(g0 v)) ∩ ker(λed ) \ g(g0 v), thus it is, modulo g(g0 v), a nonvanishing real multiple of Xed (g0 v) = O Xed0 (g0 v) and pn−1 (Ad(g0 )Xed (v)g0 v) = 0. From this, we have a unique td (v) ∈ R such that   Re pn−1 (g(td (v))g0 v)pn−1 (Ad(g0 )Xed (v)g0 v) = 0. Put thus v = g(td (v))g0 v: let us show that v is the only element in the orbit of v and in . First π(v ) = π(g0 v), thus, by Corollary 2.1.14, T (v)Zh(ed ) depends only on π(v), therefore by Relation (2.2.1), Zed (v) depends only on π(v). It follows that Xed (v ) = XeOd0 (v ) = Xed (g0 v), and pn−1 (Xed (v )v ) = pn−1 (Xed (g0 v)g0 v), the above condition is equivalent to   Re pn−1 (v )pn−1 (Xed (v )v ) = 0. With Lemma 2.5.10 this is equivalent with the condition Fn−1 (g(td (v)) g0 v) = 0. Thus we have the unique t (v) ∈ Rd such that Q(t (v),v) = g(td (v))g0 v ∈ . Case (3b): n − 1 ∈ δ. This is similar to Case 2: since λn−1 is never purely imaginary there is td (v) ∈ R such that |bn (g(td (v))g0 v)| = 1. Hence t (v) = (t1 (v), . . . ,td (v)) is the unique element in Rd such that Q(t (v),v) ∈ . Case (3c): n − 1 ∈ e2 : here n ∈ e also and d  = d − 2. Denote g(s,t) = g0 exp sXed−1 (v) exp tXed (v)g0−1 . Since: pn−1 (Ad(g0 )Xed−1(v)g0 v) = pn−1 (g0 Xed−1(v)v) = n−1 (g0 )pn−1 (Xed−1(v)v), and similarly for pn−1 (Ad(g0 )Xed (v)g0 v), we have: pn−1 (g(s,t)g0 v) = g0 v + (s + it)n−1 (g0 )ζed−1 (v) = g0 v + (s + it)z,

2.5 Orbit structure for an action of exponential type

145

with z = 0, there are unique td−1 (v) and td (v) ∈ R such that Fn−1 (g(td−1 (v),td (v))g0 v) = Qn−1 (t1 (v), . . . ,td−2 (v),td−1 (v),td (v),v) = 0. This defines the unique point t (v) ∈ Rd for which Q(t (v),v) ∈ . This proves that in any case  is a cross-section for all the orbits in  and the cross-section mapping v → v ∈  ∩ Gv is v → Q(t (v),v), for any O such that v ∈ O. Finally remark that, given a covering set O, the maps v → tk (v) are explicitly defined on the set O ∩ , and by construction they are continuous. Thus the cross-section mapping v → Q(t (v),v) is continuous on O ∩ . This implies that the cross-section mapping is continuous on . This completes the proof of the proposition. Example 2.5.12 We return to the example considered in Examples 2.4.6, 2.4.10, 2.5.5, and 2.5.9: there g = spanR {X1,X2,A} where [A,X1 ] = −X1 + X2 , [A,X2 ] = −X1 − X2 and [X1,X2 ] = 0 with V = R6 . Recall that v ∈ V is written as v = t [v1,v2,v3,v 3,v5,v 5 ], and the Jordan–Hölder basis for gc is (X,X,A) for gc where X = X1 +iX2 , we also put w = v1 −iv2 . (1) Consider the open layer  = {1,5,6},(3,1,2),{1} ; in Example 2.5.9 we wrote Q(t,v) in terms of the coordinates v1,v2,v3,v5 : Q(t1,t2,t3,v)   = et1 /v1 v1,et1 /v1 v2,et1 (1+i)/v1 v3,et1 (1+i)/(2v1 ) [v5 + ζ5 (v)(t2 + it3 )] , where ζ5 (v) = sign(v3 (|v3 | − |w|)). Here 1 ∈ δ, b1 (v) = v1 and 5,6 ∈ e2 , so the conditions for the cross-section are |v1 | = 1,v5 = v 5 = 0. Hence  = {v ∈  : |v1 | = 1,v5 = 0} = { t [ε,v2,v3,v 3,0,0] : ε = ±1,v2 ∈ R,v3 ∈ C \ {0},1 + v22 = |v3 |2 }. (2) For the singular layer  = {1,5},(3,1),{1} , there are two open set O 1 and O 2 in F and the corresponding functions Qν (ν = 1,2) are:   Qν (t1,t2,v) = et1 /v1 v1,et1 /v1 v2,et1 (1+i)/v1 v3,et1 (1+i)/(2v1 ) [v5 + ζ5ν (v)t2 ] . Recall that ζ51 (v) = sign(w + v3 ) and ζ52 (v) = sign(i(w − v3 )). Observe that Ad(exp t1 Xe1 (v))Xeν2 (v) = et1 (1+i)/2v1 Xeν2 (v). Then, to find t2 (v), we have to solve the equation:       0 = Re et1 (1+i)/2v1 v5 +t2 ζ5ν (v) et1 (1+i)/2v1 ζ5ν (v) = et1 /v1 Re v5 ζ5ν (v)+t2 .

146

Stratification of an orbit space

We see that  = {v ∈  : |v1 | = 1, wv5 + v3 v 5 = 0}. And, for each v ∈ , choosing ν such that v ∈ O ν , the unique t (v) ∈ R2 such that Qν (t (v),v) ∈  is   t (v) = (t1 (v),t2 (v)) = −v1 ln |v1 |, − Re(v5 ζ5ν (v) ) .  If the action is not of exponential type, there is no general description of a cross-section for an ultrafine layer  = e,h,δ . In fact, if there are nonregular orbits in , by a general result of Effros [36] and Glimm [44], there is no Borel cross-section for the action in . In some special cases, it is possible to describe an explicit cross-section for , if every orbit in  is regular. Example 2.5.13 Let us come back to Examples 1.9.11 and 2.2.18. We write, as in Example 1.9.11, v = (v1, . . . ,v4,r1 y1,r2 y2 ) = (r,w,y). For the minimal layer 1 = {3,4},(1,2),∅ , v1 r1 r2 = 0. We saw that G(r,w) = {1}, it is easy to prove that a cross-section for this layer is given by the product of a crosssection for the G action on (r,w) by the set Y = {(y1,y2 )} ∈ T × T. Thus we get: {3,4},(1,2),∅ = {σ = (σ1,σ2,0,0,r1 y1,r2 y2 ) : σ1 = 0, σ12 = σ22, ri > 0, |yi | = 1}. − + Now recall the layer 3 = {3,5},(1,2),{5} = + 3 ∪ 3 . The orbits in 3 + are not regular, so there is no Borel cross-section for 3 , but there is a Borel cross-section for − 3 ; a simple construction is the following. For the action on w, the cross-section for the action on w is here 3w = {(σ1, − σ1,0,σ4 ) : σ1 = 0, σ4 ∈ R}. Now recall the action of G(w) on Y :

et (A1 +A2 ) (y1,y2 ) = (e2it y1,e3it y2 ). This action is regular. To get a simple cross-section we change our parametrization by putting: y1 = y2 y 1 and y2 = y23 y 21 , (so that y1 = y2 (y  1 )3 and y2 = (y  1 )2 y2 ). The action of G(w) on these new variables is now: et (A1 +A2 ) (y1 ,y2 ) = (eit y1 ,y2 ). A cross-section for this action is clearly 3 = {(1,y2 ) : |y2 | = 1}. Coming back to the variables (z,w,y), it is easy to see that a cross-section for − 3 is: 3− = {(r,w,y) = (r,(σ1, − σ1,0,σ4 ),(y2,y2 ))} = {σ = (σ1, − σ1,0,σ4,r1 y2,r2 y2 ) : σ1 = 0, σ4 ∈ R, ri > 0, |y2 | = 1}.

2.5 Orbit structure for an action of exponential type

147

Let us now compute the function Q(t,σ ) for these two layers, with σ in the respective cross-sections. On 1 , we have T (v)A1 = A1 , Z3 (v) = v1 A1 T (v)A2 = A2 −

v2 v1 A1 ,

X3 (v) = and:

thus Z4 (v) = A1 , v1

v12 −v22 v1 T (v)A2 ,

X4 (v) =

v1 2 v1 − v22

thus:



A2 −

 v2 A1 , v1



 σ2 ia1 (t) ia2 (t) Q(t,σ ) = σ1,σ2,t1,t2 + t1,r1 e y1,r2 e y2 , σ1   1 t2 with a1 (t) = λ1 σ2 1 t2 2 + λ2 σt11 − σ2 2 t2 2 , and a2 (t) = 3σ 2 2 . Clearly the map σ1 −σ2

σ1 −σ2

σ1 −σ2

Q : R2 × 1 → 1 , (t,σ ) → Q(t,σ ) is a smooth parametrization of the open layer 1 . Consider now the layer − 3 , we have T (v)A1 = A1 , T (v)A2 = A2 +A1 , so Z3 (v) = v1 A1 , but now Z5 (v) = β5+ (A2 + A1 ) = (2ir1 y1 )(A2 + A1 ), because T (v)(A2 ) is real, there is only one nonempty covering set, namely 1 and X3 (v) =

A1 , v1

X5 (v) = X5 (v)1 =

sign(2iv5 ) 1 2iv5 (A2 +A1 ) = (A2 +A1 ). 2r1 |2iv5 |2

On the other hand b5 (v) is here b5 (v) = r1 y1 . And the map Q : R2 × 3+ → + 3 is now not exactly a parametrization of the orbit, but a covering map: Q(t,σ ) = (σ1, − σ1,t1, − t1,r1 eiλ1 /σ1 t1 eit2 /r1 y2,r2 e3it2 /2r1 y2 ). Observe that (t1,t2 ) → Q(t,σ ) is no longer an injective map, indeed Q(t,σ ) =  Q(t ,σ ) if and only if t1 = t1 and t2 − t2 ∈ 4r1 πZ. As in the completely solvable case, there is a somewhat simpler orbital parametrization in the exponential type case that is obtained by using a method of substitution. For the minimal layer we will apply this new parametrization to decompose Lebesgue measure on V , and therefore we introduce real  coordinates for elements of V . Identify v = vi fi ∈ V with a vector η(v) ∈ Rn by putting: ηi (v) = vi ,

if i ∈ I, i − 1 ∈ I,

ηi (v) = Re(vi ),

if i ∈ / I, i − 1 ∈ I,

ηi+1 (v) = Im(vi ),

if i ∈ / I, i − 1 ∈ I.

Let us denote by v → σ (v) the cross-section mapping defined on  by {σ (v)} = Ov ∩ , where Ov is the orbit of v. For every covering set O, let O = σ −1 ( ∩ O).

148

Stratification of an orbit space

Thus, we have now a map from Rd × ( ∩ O) into O which is (t,σ ) → Q(t,σ ) = QO (t,σ ). As in the completely solvable case, we now make a change of the variables tk → zk , writing η(Q(t,σ )) = P (z,σ ) in order to simplify the parametrization of the orbit Oσ . So we put for 1 ≤ k ≤ d: zk = Qek (t,σ )

if

ek ∈ e0 \ δ,

zk = |bek (Q(t,σ ))|

if

ek ∈ δ,

if

ek ∈ e2 \ I,

if

ek ∈ e1 \ δ,

zk + izk+1 = Qek (t,σ )   zk = Re ck (t,σ )−1 Qek (t,σ )

(2.5.9)

where, in the last case, ck (t,σ ) = sign(ek (gk (t,σ )))ζek (σ ).

(2.5.10)

Recall that since the representation is of exponential type, the generalized weights λi are λi = μi (1 + iai ), with μi ∈ g∗ and ai ∈ R. If e is not empty, / δ, and Zk = (0, + ∞) if we denote Z = ek ∈e Zk , with Zk = R if ek ∈ ek ∈ δ. Using the expression of Qek (t1, . . . ,tk ,σ ) given in Proposition 2.5.8, it is possible to prove recursively that, for each σ ∈  ∩ O, there is a unique solution tk = k (z1, . . . ,zk ,σ ) (z ∈ Z) to the system of equations (2.5.9). Therefore we get a new parametrization of the orbit Oσ with parameters z ∈ Z. Proposition 2.5.14 Let α be a representation of exponential type of a simply connected solvable Lie group G, choose good Jordan–Hölder and descending flag bases, let  = e,h,δ be an ultrafine layer with cross-section . Assume that e is nonempty, and define Z as above. Let O be a covering set as in Proposition 2.5.1. Then there is a unique smooth function  = O : Z × (O ∩ ) → Rd such that the unique t satisfying Relation (2.5.9) is t = (z,σ ). For each σ , (·,σ ) is a diffeomorphism from Z to Rd , and 4 4 |e (ge (z,σ ))|−1 |1 + iaek |zk−1 . J ((·,σ ))(z) = ± e∈δ /

ek ∈δ

The function P defined by P (z,σ ) = η(Q((z,σ ),σ )) is a diffeomorphism from Z × ( ∩ O) onto O and satisfies the following. (Write gk (z,σ ) = gk ((z,σ ),σ ) and ck (z,σ ) = ck ((z,σ ),σ ).) Let 1 ≤ i ≤ n, such that i − 1 ∈ I, put r = max{k : ek < i}. (1) Suppose i ∈ / e, if i ∈ I, then Pi (z,σ ) = i (gr (z,σ ))σi + Si (z,σ ), while if i ∈ / I,then Pi (z,σ ) + iPi+1 (z,σ ) = i (gr (z,σ ))σi + (Si + iSi+1 )(z,σ ),

2.5 Orbit structure for an action of exponential type

149

/ I, Si+1 (z,σ ), are real functions and depend where Si (z,σ ) and, if i ∈ only on z1, . . . ,zr and on σ1, . . . ,σi−1 . (2) If i = ek ∈ e0 \ δ, Pi (z,σ ) = zk , but if i = ek ∈ e1 \ δ, then Pi (z,σ ) + iPi+1 (z,σ )

 = ck (z,σ ) zk + i|i (gk−1 (z,σ ))|Im(ζ i (σ )σi ) + iSi+1 (z,σ ) . Here Si+1 is a real valued function and depends only on z1, . . . ,zr and on σ1, . . . ,σi−1 . (3) If i = ek ∈ e2 , then Pi (z,σ ) = zk , Pi+1 (z,σ ) = zk+1 . (4) If i = ek ∈ e0 ∩ δ, then Pi (z,σ ) = zk σi + Si (z,σ ), while if i = ek ∈ e1 ∩ δ, then Pi (z,σ ) + iPi+1 (z,σ ) =

i (gk−1 (z,σ )) 1+iai bi (σ ) 1+iai zk |i (gk−1 (z,σ ))| + (Si + iSi+1 )(z,σ ),

where Si (z,σ ) and, if i ∈ / I, Si+1 (z,σ ), are real functions and depend only on z1, . . . ,zr and on σ1, . . . ,σi−1 . Proof: We proceed by induction on the cardinality of I. Assume that |I| = 2. (1) If I = {0,1}, dim V = 1, then e = {1} = δ, σ1 = ±1, Q1 (t1,σ ) = 1 (exp t1 X1 (σ ))σ1 , and we put z1 = 1 (exp t1 X1 (σ )), and t1 = 1 (z1,σ1 ) = λ1 (Xe1 (σ )) ln(z1 ). Then 1 (z1,σ1 ) is the unique real number such 1 that |b1 (Q((z,σ ),σ ))| = z1 , hence P1 (z,σ ) = z1 σ1 . The map z → P (z,σ ) is a diffeomorphism from (0,∞) to Oσ . (2) If I = {0,2}, then e = {1} = δ, thus b(σ ) = σ1 is a complex number with magnitude 1, and Q(t1,σ ) = 1 (exp t1 X1 (σ ))σ1 = b1 (Q(t1,σ )). Hence z1 = |b1 (Q(t1,σ ))| = et1 Re(λ1 )(X1 (σ )) , we put: 1 (z1,σ ) = t1 = 1 Re(λ1 )(X1 (σ )) ln(z1 ). In this case P1 (z1,σ ) + iP2 (z1,σ ) = Q((z1,σ ),σ ) = z11+ia1 σ1 = z1 eia1 ln(z1 ) σ1, where λ1 = Re(λ1 )(1 + ia1 ) (recall that λ2 = λ1 ), this is relation (4). This formula proves also that (·,σ ) is a diffeomorphism from (0,∞) onto R, and P is a diffeomorphism from (0,∞) onto Oσ . Suppose now |I| > 2, and put n = dim V , n = max{i ∈ I : i < n}, Z = Vn , π : V → V /Vn and p = pn +1 , thus p(v) is real if n = n − 1, complex if n = n − 2. Now π() ⊂  with e = {e1, . . . ed  } and for each  O ∈ F, we have O  , such that π( ∩ O) ⊂   ∩ O  , thus π(O ) ⊂ O .

150

Stratification of an orbit space

Now, by construction of Q, for σ ∈  ∩ O, with σ  = π(σ ), we have exp tXe (σ )σ  = σ  if e > n and: 

Q (t ,σ  ) = QO (t ,σ  ) = π(Q(t,σ )) if t  = (t1, . . . ,td  ). -d   By induction we have Z  = k=1 Zk , and a unique function  =    O    d ( ) : Z ×  ∩ O → R , such that: 

P  (z,σ  ) = P O (z1, . . . ,zd  ,σ  ) = η(Q ( (z,σ  ),σ  )) satisfies all the conditions of the proposition. Solving Relation (2.5.9), we get first k (z,σ ) = k (z,σ  ) for ek ≤ n , where z ∈ Z, z = (z1, . . . ,zd  ). Then Pi (z,σ ) = ηi (Q((z,σ ),σ )) for each i ≤ n and by construction the relations (1) to (4) hold for i ≤ n . Now π ◦ P is defined, we only need to define p ◦ P , following the possible cases. Case (1): n = n − 1, n ∈ / e. In this case d = d  , Z = Z  , and (z,σ ) =    (z,σ ). We define Pn (z,σ ) = Qn ((z,σ ),σ ), this gives (1) with Sn (z,σ ) = Yn ((z,σ ),σ ). By the induction hypothesis, (·,σ ) is a diffeomorphism from Z to Rd , thus P (·,σ ) is a diffeomorphism from Z onto Gσ . It follows that P is a bijection from Z × (O ∩ ) to O . Case (2): n = n − 1, n = ed ∈ e0 \ δ. In this case σ = σ  , d = d  + 1, z = (z,zd ) ∈ Z = Z  × R and to define d (z,σ ), we solve the equation in td : zd = Qn (1 (z,σ ), . . . ,d−1 (z,σ ),td ,σ ) = n (gr (z,σ  ))td + Yn ( (z,σ ),σ ). Or, if Sn (z,σ ) = Yn ( (z,σ ),σ ), td = d (z,σ ) = n (gr (z,σ ))−1 (zd − Sn (z,σ )). This proves that (z,σ ) = ( (z,σ  ),d (z,σ )) is a diffeomorphism from Z onto Rd , now Pd (z,σ ) = zd , and P is a bijection from Z × (O ∩ ) onto O such that, for any σ , P (·,σ ) is a diffeomorphism from Z onto Oσ . Especially, we get: ∂td = ed (gr (z,σ ))−1 = ed (gd (z,σ ))−1 . ∂zd Case (3): n = n − 1, n ∈ δ. In this case π(σ ) = σ  and σn is such that bn (σ ) = εn = ±1. Similarly, z = (z,zd ) ∈ Z = Z  × (0,∞). We solve: # # zd = #bn (Q( (z,σ  ),td ,σ ))# = n (gd−1 (z,σ  ))etd λn (Xed (σ ))

2.5 Orbit structure for an action of exponential type

151

into td = d (z,σ ) =

1 ln(n (gd−1 (z,σ  ))−1 zd ). λn (Xed (σ ))

This implies that  is a diffeomorphism from Z onto Rd , and we get, with Sn (z,σ ) = Yn ((z,σ ),σ ), Pn (z,σ ) = Qn ((z,σ ),σ ) = zd εn + Sn (z,σ ). Moreover P is a bijection from Z × (O ∩ ) onto O and: ∂td 1 bed (σ ) 1 = =± . ∂zd zd ζed (σ ) zd Case (4): n = n − 2, n − 1 ∈ / e. Here Z = Z  , and (z,σ ) =  (z,σ  ). We define (Pn−1 + iPn )(z,σ ) = Qn ((z,σ ),σ ), this gives (1) with (Sn−1 + iSn )(z,σ ) = Yn−1 ((z,σ ),σ ). This case is very similar to Case 1. Case (5): n = n − 2, n − 1 = ed ∈ e1 \ δ. Observe that, using Lemma 2.2.14 and Relation (2.2.6), the vector Xed (σ ) depends only on σ  , and is in the kernel of λn−1 , thus ζed (σ ) = pn−1 (Xed (σ )σ ) depends only on σ  , and σ = (σ ,σn−1 ), with σn−1 ∈ iRζed (σ  ). Using the Qn−1 -expression in Proposition 2.5.8, we get:   Im cd ( (z,σ  ),σ  )−1 Qn−1 ( (z,σ  ),td ,σ ) = |n−1 (gd−1 (z,σ  ))|Im(ζ ed (σ  )σn−1 ). And the equation to solve is now:   zd = Re cd ( (z,σ  ),σ  )−1 Qn−1 ( (z,σ  ),td ,σ )   = |n−1 (gd−1 (z,σ  ))|td + Re cd (z,σ  )−1 Yn−1 ( (z,σ  ),σ  ) . So d (z,zd ,σ  ) = td is given by: d (z,zd ,σ  )

   = |n−1 (gd−1 (z,σ  ))|−1 zd − Re cd (z,σ  )−1 Yn−1 ( (z,σ  ),σ  ) .

Thus  = (,d ) is a diffeomorphism from Z onto Rd , and putting (Pn−1 + iPn )(z,σ ) = Qn−1 ((z,σ ),σ ), it has the desired form with Sn (z,σ ) = Im(cd (z,σ  )−1 Yn−1 ( (z,σ  ),σ  )). Now P (·,σ ) is a diffeomorphism from Z

152

Stratification of an orbit space

to Oσ . As in the preceding cases, P is a bijection from Z × (O ∩ ) onto O . Now: ∂td = |ed (gd−1 (z,σ  ))|−1 = |ed (gd (z,σ ))|−1 . ∂zd Case (6): n = n − 2, n − 1 = ed−1 , n = ed ∈ e2 . This case is similar to case 2.: σ = σ  , Z = Z  × R2 and we solve: zd−1 + izd = Qn−1 ( (z,σ  ),td−1,td ,σ  )

 = n−1 (gd−2 (z,σ  )) td−1 ζed−1 (σ ) + td ζed (σ ) + Yn−1 (z,σ  ). Denoting (A + iB)(z,σ  ) = n−1 (gd−2 (z,σ  ))−1 (zd−1 + izd ) − Yn−1 (z,σ  ) = td−1 ζed−1 (σ  ) + td ζed (σ  ), d−1 and d are given by:    −1   A(z,σ  ) d−1 (z,σ ) Re(ζed−1 (σ  )) Re(ζed (σ  )) = . d (z,σ ) Im(ζed−1 (σ  )) Im(ζed (σ  )) B(z,σ  ) Observe that since {Xed−1 (σ  ),Xed (σ  )} is a supplementary basis for g(σ ) in g(π(σ )) then the 2 × 2 above matrix is regular. Therefore  = (,d−1,d ) is a diffeomorphism from Z to Rd , P is a bijective map between Z × (O ∩ ) and O . Since the complex number ζed−1 (σ  ) and ζed (σ  ) are orthogonal, the determinant of the above matrix is iζed (σ )ζed−1 (σ ) = ±1, and the Jacobian of the transform (zd−1,zd ) → (td−1,td ) is: D(td−1,td ) = ±|ed−1 (gd−1 (z,σ ))|−1 |ed (gd (z,σ ))|−1 . D(zd−1,zd ) Case (7): n = n − 2, n − 1 = ed ∈ δ. Here Zd = (0,∞), and σ = (σ ,σn−1 ), where σn−1 is such that |bn−1 (σ )| = 1. To define d , we solve: # #  zd = #bn−1 (Q( (z,σ  )),td ,σ )# = |n−1 (gd−1 (z,σ  ))|etd Re(λn−1 (Xed (σ ))) . So td = d (z,σ ) =

ln(zd ) − Re(λn−1 (gd−1 (z,σ  ))) , Re(λn−1 (Xed (σ  )))

and  = (,d ) is a diffeomorphism from Z onto Rd . Especially, since 1 = |bed (σ )| =

1 , |λed (Xed (σ ))|

2.5 Orbit structure for an action of exponential type

153

then |Re(λed (Xed (σ ))| = |1 + iaed |−1 and: ∂td |1 + iaed | =± . ∂zd zd Now (π ◦ P )(z,σ ) = P  (z,σ  ) and: (Pn−1 + iPn )(z,σ ) = Qn−1 ((z,σ ),σ )  1+ian−1  bn−1 (σ ) + Yn−1 (z,σ  ) = n−1 (gd−1 (z,σ  )) etd Re(λn−1 (Xed (σ ))) = n−1 (gd−1 (z,σ  ))|n−1 (gd−1 (z,σ  ))|−1−ian−1 zd

1+ian−1



bn−1 (σ )



+ Yn−1 (z ,σ ). Assuming now that the Jacobian J ( (·,σ  )) has the desired form, then in Cases (1) and (4), J ((·,σ )) = J ( (·,σ  )), in Cases (2), (3), (5), and ∂td , in Case (6), J ((·,σ )) = J ( (·,σ  )) (7), J ((·,σ )) = J ( (·,σ  )) ∂z d D(td−1,td ) D(zd−1,zd ) .

The proof shows that in any case we get the desired form: 4 4 |e (ge (z,σ ))|−1 |1 + iaek |zk−1 . J ((·,σ ))(z) = ± e∈δ /

ek ∈δ

The last statement is a direct consequence of Relation (2.5.9), and the equation of  given by Proposition 2.5.11. This completes the proof. Example 2.5.15 We return to the example considered in Examples 2.4.6, 2.4.10, 2.5.5, 2.5.9, and 2.5.12: there g = spanR {X1,X2,A} where [A,X1 ] = −X1 + X2 , [A,X2 ] = −X1 − X2 and [X1,X2 ] = 0 with V = R6 . As in Example 2.5.12, let us write σ and the value of the function P in terms of the coordinates v1 , v2 , v3 and v5 . (1) In the open layer  = {1,5,6}(3,1,2),{1} , recall that the cross-section is  = {(ε,σ2,σ3,0) : ε = ±1, σ2 ∈ R, σ3 ∈ C, 1 + σ22 = |σ3 |2 = 0}. Now z1 = et1 /σ1 , and z2 + iz3 = Q5 (t1,t2,t3,σ ). So  P (z1,z2,z3,σ ) = εz1,z1 σ2,z11+i σ3,z2 + iz3 ). (2) In the singular layer  = {1,5},(3,1),{1} , we saw that the cross-section is:  = {(σ1 = ε,σ2,σ3,σ5 ) : wσ5 + σ3 σ 5 = 0}, where w = σ1 − iσ2 . Recall that ζ51 (σ ) = sign(w + σ3 ) and ζ52 (σ ) = sign(i(w − σ3 )), this does not depend on σ5 . On each O ν , the last condition is equivalent to Re(ζ ν (σ )σ5 ) = 0.

154

Stratification of an orbit space

Then the function P ν is:   i/2 P ν (z1,z2,σ ) = εz1,z1 σ2,z11+i σ3,z1 ζ5ν (σ )(z2 + iz1 Im(ζ ν (σ )σ5 )) . 

2.6 Structure of the generic layer Let G be a simply connected solvable Lie group with Lie algebra g, with (Zj ) a good Jordan–Hölder basis for s = gc . Let α be a representation of exponential type in a vector space V of dimension n, and fix a good descending flag basis (fi ) for U = Vc . As in the previous section we have the layering of V and given a covering set O, the parametrization P : Z ×  ∩ O → . In this section, we consider the minimal, Zariski open layer  = e,h,δ . We show that the corresponding cross-section  is a submanifold of  and exhibit a natural decomposition of the Lebesgue measure on . First observe that for each i ∈ e0 ∩ δ, if v ∈ , then bi (v) = ±1. So as in the completely solvable case,  is the disjoint union 5 = ε ε∈{±1}e0 ∩δ

where ε = {v ∈  : bi (v) = εi (i ∈ e0 ∩ δ)}. We now fix ε ∈ {±1}e0 ∩δ , and parametrize ε ∩ O. Proposition 2.6.1 Let R = i ∈e / Ri where 2 R if i − 1 ∈ / e1 ∩ δ, Ri = R/2πZ if i − 1 ∈ e1 ∩ δ, and let F = {O} be the open covering of  defined in Proposition 2.5.1. (1) For each O ∈ F, and each ε ∈ {±1}e0 ∩δ , there is an open set U = UεO ⊂ R, conull for the Lebesgue measure and an explicit bijection σ = σεO : U → ε ∩ O. (2) The family (UεO ,(σεO )−1 )O∈F , ε∈{±1}e0 ∩δ defines a structure of analytic manifold on . (3) There is a unique measure on  whose image in each UεO is the Lebesgue measure. Proof: We proceed by induction on the number of elements in I. Suppose first that I has two elements, so that F = {}. In this case, G(v) = G if and only if λ1 = 0.

2.6 Structure of the generic layer

155

/ e, then  = V , Suppose dim V = 1, then either 1 ∈ / e, or 1 ∈ e0 ∩ δ. If 1 ∈ we put U = R1 = R, σ (x) = x1 f1 , and the proposition holds in this case. If 1 ∈ δ, then  = +1  −1 where +1 = {f1 }, −1 = {−f1 }, and the proposition is trivial in this case. Suppose now dim V = 2, then either e = ∅ or e = e1 ∩ δ = {1} If e = ∅ then  = V , we put U = R1 × R2 = R2 , we put σ (x1,x2 ) = (x1 + ix2 )f1 + / e. Here  = {v = (x1 − ix2 )f2 and the result holds. If 1 ∈ e1 ∩ δ, and 2 ∈ v1 f1 + v 1 f2 : |v1 | = |b1 (v)| = 1}. We thus put U2 = R2 = R/2πZ and σ (x2 ) = eix2 f1 + e−ix2 f2 , in this case also the proposition holds. Suppose now |I| > 1, consider n = max{i ∈ I : i < n}. Put Z = Vn , and p(v) = pn +1 (v), and W = spanC {f1, . . . ,fn } ∩ V , so that V = W ⊕ Z, let π : V → V /Z, then W and V /Z are canonically isomorphic vector spaces. Accordingly to (2.4.3), we identify V with W × R if n = n − 1, or W × C if n = n − 2 writing v = (w,z) = (w(v),p(v)). Recall that π() ⊂  where  is the minimal layer in V /Z, with jump indices e = e ∩ {1, . . . ,n }, h = h|e , and δ  = δ ∩ {1, . . . ,n }. Now by (2.5.8) π() ⊂   , the cross-section for  . Fix ε ∈ {±1}e0 ∩δ , and a covering open set O ∈ F, put ε = (εj )j ∈e0 ∩δ, j ≤n , and let O  be the covering set in F  defined in Remark 2.5.2. Then π(O) ⊂ O  , hence π(ε ∩ O) ⊂ ε  ∩ O  . By induction, there is an open subset U  of full measure in R  =     i ∈e, / i≤n Ri and a bijection σ : U → ε ∩ O , satisfying (1), (2) and (3) for    . Denote σεO (x  ) (x  ∈ U  ) the element in W corresponding to σ  (x  ). We shall now build case by case an analytic function f O on W such that O = {v = (w,z) ∈ V : π(v) ∈ O , f O (w) = 0}. Then we put ⎧ ⎨  U = x = (x ,x + ) : x  ∈ U , f O (σεO (x  )) = 0, x + ∈ RZ = ⎩

(2.6.1)

4 i ∈e, / i>n

⎫ ⎬ Ri

⎭

,

and we shall define the parametrization of  ∩ O as 

σ (x) = σεO (x ,x + ) = (σεO (x  ),p(σεO (x ,x + ))).

(2.6.2)

Considering all the cases for the set e ∩ {n + 1,n}, we now describe f O and p(σεO (x ,x + )). / e. In this case, for each O, RZ = Rn = R, Case (1): n = n − 1, n ∈ f O = 1, x = (x ,xn ), ε = ε and p(σεO (x)) = xn . Since n is not in e, then  = π −1 ( ), and (2.6.1) holds. Moreover by definition of ε ∩ O, ε ∩ O = π −1 (ε ∩ O  ), this gives the parametrization (2.6.2).

156

Stratification of an orbit space

Case (2): n = n − 1, n ∈ e0 \ δ. There is no RZ , no x + , we put f O (w) = pn (T (w)Zh(e) w), ε = ε  , x = x  , p(σεO (x  )) = 0. Here, by definition of , v = (w,z) ∈  means that π(v) ∈  and f O (w) = 0, this is (2.6.1),  moreover εO = εO × {0}, that is (2.6.2). Case (3): n = n − 1, n ∈ δ. In this case, there is no RZ , no x + , we put f O (w) = λn (T (w)Zh(n) )pn (T (w)Zh(n) v), 

x = x  , ε = (ε,εn ) and p(σεO (x  )) = εn + z0 (σεO (x  )), where z0 (w) = −

pn (T (w)Zh(n) w) . λn (T (w)Zh(n) )

The condition f O (w) = 0 is equivalent to n ∈ e and n ∈ δ, thus we have  (2.6.1). Now (w,z) in is the set εO if and only if w is in εO and z = εn + z0 (w). This is (2.6.2). Case (4): n = n − 2, n − 1 ∈ / e. In this case, n ∈ / e, f O = 1, RZ = R2 ,   O  x = (x ,xn−1,xn ), ε = ε and p(σε (x ,xn−1,xn )) = xn−1 +ixn , the conclusion follows exactly as in Case (1). / e, the open set O is Case (5): n = n − 2, n − 1 = ed ∈ e1 \ δ. In this case, n ∈ defined as the set of (w,z) with π(v) ∈ O  and f e,t (w) = 0 where the function f e,t is either pn−1 (Re(T (v)Zh(n−1) )v) or pn−1 (Im(T (v)Zh(n−1) )v). Because these two numbers are dependent over R for each v ∈ , they are dependent / e, thus over R for any v, thus f e,t (w) = 0 ensures that n − 1 ∈ e and n ∈ we put f O = f e,t , and get (2.6.1). Now RZ = Rn = R, x = (x ,xn ), ε = ε and ζed (v)−1 p(v) is purely imaginary for any v ∈ O . Recall that here ζed (v) depends on w (see Case 5 in the proof of Proposition 2.5.14) denote here this function by ζed (w) and put: 

p(σεO (x ,xn )) = ixn ζed (σεO (x  )). Case (6): n = n − 2, n − 1 ∈ e2 . In this case it follows from Proposition 2.5.1 that the covering set is defined by the functions f n−1,t (v) which are either 1 or pn−1 (Re(T (v)Zh(n−1) )v) or pn−1 (Im(T (v)Zh(n−1) )v). Now v is in the layer  if and only if π(v) is in  and pn−1 (T (v)Zh(n−1) v) = 0 and pn (T (v)Zh(n) v) = 0. Now n − 1 and n are in e \ δ by Lemma 2.2.16, this implies that λn−1 (T (v)Zh(n−1) ) = 0 and λn (T (v)Zh(n) ) = 0. But since the action is of exponential type, ker λn = ker λn−1 is a real subalgebra, thus the real and imaginary parts of T (v)Zh(n−1) and T (v)Zh(n) are all in s(πn−2 (v)) ∩

2.6 Structure of the generic layer

157

ker λn−1 ∩ g = g(πn−2 (v)) ∩ ker λn−1 . Now if β is the action of g(πn−2 (v)) ∩ ker λn−1 on the space Y = spanC {fn−1,fn,v} described in (2.2.8), we have: ⎡ ⎤ 0 0 γ (X) β(X) = ⎣0 0 γ (X)⎦ ⎡

0 0

0 0 = ⎣0 0 0 0

0

⎤ pn−1 (Xw) pn (Xw) ⎦ 0

(X ∈ g(πn−2 (v)) ∩ ker λn−1 ).

Here the matrix T (v) of the column operation is a product T  (v)Td−1 (v) and T  (v) depends only on w, thus T (v)Zh(n−1) = T  (v)Zh(n−1) depends only on w and pn−1 (T (v)Zh(n−1) v) depends also only on w. Moreover: T (v)Zh(n) = T  (v)Zh(n) −

pn−1 (T  (v)Zh(n) v)  T (v)Zh(n−1) . pn−1 (T  (v)Zh(n−1) v)

Since T (v)Zh(n) and T  (v)Zh(n−1) are in ker λn−1 , thus T  (v)Zh(n) = X + iY depends only on w and belongs to ker λn−1 and its real and imaginary parts X and Y are in g(πn−2 (v)) ∩ ker λn−1 . Thus pn (T  (v)Zh(n) v) = γ (X) + iγ (Y ) = pn (Xw) + ipn (Y w) depends only on w. This implies pn (T (v)Zh(n) v) depends only on w. Finally the condition v ∈  is equivalent to say that π(v) ∈  and n − 1 and n are in e, or to π(v) ∈  and pn−1 (T (v)Zh(n−1) v)pn (T (v)Zh(n) v) = 0. This means that condition (2.6.1) holds with f O (w) = pn−1 (T (v)Zh(n−1) v)pn (T (v)Zh(n) v)f n−1,t (v). Since p(σεO (x ,x + )) = 0, we conclude as in Case (2). / e, as in Case (3), Case (7): n = n − 2, n − 1 = ed ∈ δ. In this case n ∈ we put f O (w) = λn−1 (T (w)Zh(n−1) )pn−1 (T (w)Zh(n−1) v), this gives that v is in  if and only if π(v) ∈  and f O (w) = 0 (Condition (2.6.1)). Now RZ = Rn = R/2πZ, x + = xn , ε = ε , and we put z0 (w) =

pn−1 (T (w)Zh(n−1) w) , λn−1 (T (w)Zh(n−1) )

and 

p(σεO (x ,xn )) = eixn + z0 (σεO (x  )). Thus we define in each case a parametrization σεO of ε ∩ O. Finally, suppose that each ε  has a structure of smooth manifold with local charts  defined by the σεO , then the transition functions are smooth, and we suppose

158

Stratification of an orbit space

they preserve the Lebesgue measures mt (t = 1,2) on the two open subset in R  = i ∈e,i≤n  Ri . Consider now the transition mappings for the local charts / O O (Uε ,σε ) of ε :   (σεO1 )−1 ◦ σεO2 : (σεO2 )−1 σεO1 (UεO1 ) ∩ σεO2 (UεO2 )   → (σεO1 )−1 σεO1 (UεO1 ) ∩ σεO2 (UεO2 ) . If there is no new parameter for  (Cases (2) and (6) where RZ = {0}) or if the O

O

new parameter is εn (Case (3)) then (σεO1 )−1 ◦ σεO2 reduces to (σε 1 )−1 ◦ σε 2 . In the Cases (1), (4) and (7), we get:  O O (σεO1 )−1 ◦ σεO2 = (σε 1 )−1 ◦ σε 2 ,1RZ ). O

In Case (5), the map is depending on the signs of ζed (σε t (x  )) (t = 1 or 2), thus the transition map is here:  O O (σεO1 )−1 ◦ σεO2 = (σε 1 )−1 ◦ σε 2 , ± 1RZ ). In any case, the transition function is smooth and preserves the Lebesgue measures on the open subsets of R on which they are defined. This proves that ε is a smooth manifold with local charts σεO and equipped  with a measure ν. Now suppose that U  = UεO is a conull subset of R  , then each UεO can be written as:     UεO = x  ∈ UεO : f O (σεO (x  )) = 0 × RZ , thus it is still a conull subset of R, and, for each Borel subset E in , the measure ν(E) is characterized by:

  ν(E ∩ O) = m (σεO )−1 (E ∩ O ∩ ε ) . ε∈{±1}e0 ∩δ

More precisely, since F is finite, and each O ∈ F is characterized by an inequality: O = {v ∈ V : g O (v) = 0}, (the analytic function g O is the product of the successive functions f O defined in Relation 2.6.1), then for any O1 and O2 , any Borel set E ⊂ ε , (σεO1 )−1 (E ∩ O1 ∩ O2c ) = (σεO1 )−1 (E ∩ O1 ) ∩ {x ∈ UεO1 : g O2 (σεO1 (x)) = 0} and the Lebesgue measure of this set is 0. Therefore:

ν(E ∩ O2c ) ≤ ν(E ∩ O2c ∩ O) = 0 O∈F

2.6 Structure of the generic layer

159

and for any ε ∈ {±1}e0 ∩δ , any O ∈ F, any Borel set E of ,

  m (σεO )−1 (E ∩ O ∩ ε ) . ν(E) = ε∈{±1}e0 ∩δ

For each covering set O and each ε, Proposition 2.6.1 gives a parametrization σ : U → ε ∩ O. Put (O,ε) = ∪{Oσ : σ ∈ ε ∩ O}; then (O,ε) is an open subset of the minimal layer , and by Proposition 2.5.14 each v ∈ (O,ε) can be written in a unique way as v = P (z,σ (x)) = gd (z,σ (x))σ (x). This parametrization of the (O,ε) allows us to define measures on the cross-section  of  and on each orbit Gσ = Oσ in . First, Proposition 2.6.1, says that there is a measure dν on each ε given by 3 3 f (σ ) dν (σ ) = f (σεO (x)) dx. UεO

ε

We now define a measure dωσ on the orbit Oσ using the parametrization (z,x) → P (z,σεO (x)) of Proposition 2.5.14. Given a covering set O, we have elements XeOk (σ ), and the corresponding element gdO (z,σ ) (σ ∈ ε ∩ O) such that: O O O gdO (z,σ )σ = exp(O 1 (z,σ )Xe1 (σ )) . . . exp(d (z,σ )Xed (σ ))σ

= QO (O (z,σ ),σ ). Let O and O  be covering sets, and let σ ∈ O ∩ O  . For each z ∈ Z, we have z ∈ Z, such that 





gdO (z,σ )σ = QO (O (z,σ ),σ ) = QO (O (z,σ ),σ ) = gdO (z,σ )σ . 

Since XeOk (σ ) = ±XeOk (σ ), these expressions represent identifications of G/G(σ ) with an element of Oσ both using the same coexponential basis for O  O O g(σ ) in g. This means that for each k, O k (z,σ )Xek (σ ) = k (z ,σ )Xek (σ )   and gdO (z,σ ) = gdO (z,σ ). Hence P O (z,σ ) = P O (z,σ ), with z and z related as follows: 



if ek ∈ δ, zk = |bek (QO (O (z,σ ),σ ))| = |bek (QO (O (z,σ ),σ ))| = zk ,  if ek ∈ e2 and ek − 1 ∈ I, zk + izk+1 = pek (QO (O (z,σ ),σ )) = zk + izk+1 ,

O  −1 O O if ek ∈ e1 \ δ, zk = Re ck (z,σ ) pek (Q ( (z,σ ),σ )) , but here ckO (z,σ ) = sign(ek (gdO (z,σ )))sign(pek (XeOk (σ )σ )) 





= ±sign(ek (gdO (z,σ )))sign(pek (XeOk (σ )σ )) = ±ckO (z,σ ), thus zk = ±zk .

160

Stratification of an orbit space

Thus, dz = dz , and we conclude that for f ∈ Cc (Oσ ), 3 4 f (P O (z,σ )) |i (gdO (z,σ ))| dz Z

=

i ∈e /

3 Z

f (P

O

(z,σ ))

4



|i (gdO (z,σ ))| dz .

i ∈e /

Thus we have a measure ωσ on Oσ such that for any covering set O containing σ , 3 3 4 f (v) dωσ (v) = f (P O (z,σ )) |i (gdO (z,σ ))| dz. Oσ

Z

i ∈e /

We continue to frequently write P (z,σ ) = P O (z,σ ) and gd (z,σ ) = gdO (z,σ ). We can now decompose the Lebesgue measure on . Corollary 2.6.2 Let dv be the Lebesgue measure on  associated to the exterior product of the forms dvi if i and i − 1 ∈ I and dRe(vi ) ∧ dIm(vi ) if i ∈ / I. Then dv is decomposed into the measures dωσ via the measure dν : for any positive Borel function f on , 3 3 3 f (v) dv = f (v) dωσ (v) dν (σ ). 



Oσ

Moreover, each orbital measure is semi-invariant: dωσ (sv) = | det(α(s))| dωσ (v) for any s ∈ G. Proof: Let us show that the Jacobian of the map P : Z × ε ∩ O →  is 4 |J (P )|z,σ = |i (gd (z,σ ))|. (2.6.3) i ∈e /

First, with the notation of Proposition 2.5.14, the Jacobian matrix of P is now:   0 J ac(P  )(z,σ  ) (n − 1 ∈ I), J ac(P )(z,σ ) = ∂Pn ∗ ∂yn ⎤ ⎡ 0 0 J ac(P  )(z,σ  ) ⎥ ⎢ ∂Pn−1 ∂Pn−1 ⎥ ⎢ ∗ (n − 1 ∈ / I), J ac(P )(z,σ ) = ⎢ ∂yn−1 ∂yn ⎥ ⎦ ⎣ ∂Pn ∂Pn ∗ ∂yn−1 ∂yn where, as in the completely solvable case, yek = zk and yi = xi if i ∈ / e.    (z ,σ ))| | (g By induction, suppose that the formula |J (P  )| = i ∈e,i≤n  i d / holds, and consider the remaining entries in the matrix J ac(P )(z,σ ), following the cases of Proposition 2.5.14.

2.6 Structure of the generic layer

161

/ e. In this case, by Proposition 2.5.14, Case (1): n = n − 1, n ∈ Pn (z,σ ) = n (gd (z,σ ))σn + Si (z,σ ) = n (gd (z,σ ))yn + Si (z,σ  ),

∂Pn = n (gd (z,σ )). ∂yn

Case (2): n = n − 1, n ∈ e0 \ δ, then Pn (z,σ ) = zd = yn ,

∂Pn ∂yn

= 1.

Case (3): n = n−1, n ∈ δ, then Pn (z,σ ) = zd σn +Sn (z,σ ) = εn yn +Sn (z,σ  ) n and | ∂P ∂yn | = 1. Case (4): n = n − 2, {i > n : i ∈ e} = ∅. In this case, Pn−1 (z,σ ) + iPn (z,σ ) = n−1 (gd (z,σ ))(Re(σn−1 ) + iIm(σn−1 )) + (Sn−1 + iSn )(z,σ ) = n−1 (gd (z,σ ))(yn−1 + iyn ) +(Sn−1 + iSn )(z,σ  ), thus: # # # D(Pn−1,Pn ) # 2 # # # D(y ,y ) # = |n−1 (gd (z,σ ))| = |n−1 (gd (z,σ ))n (gd (z,σ ))|. n−1

Case (5):

n

n

= n − 2, n − 1 ∈ e1 \ δ. Then

Pn−1 (z,σ ) + iPn (z,σ )

 = cd (z,σ ) zd + i|n−1 (gd−1 (z,σ ))|Im(σn−1 ) + iSn (z,σ )

 = cd (z,σ ) yn−1 + i|n−1 (gd−1 (z,σ ))|yn + iSn (z,σ  ) . Recall that cd is defined in Relation (2.5.10) and |cd (z,σ )| = 1. Thus, # # # D(Pn−1,Pn ) # # # # D(y ,y ) # = |n−1 (gd (z,σ ))| = |n (gd (z,σ ))|. n−1 n Case (6): n = n − 2, n − 1, n ∈ e2 . In this case, Pn−1 (z,σ ) + iPn (z,σ ) = zd−1 + izd = yn−1 + iyn , # # # D(Pn−1,Pn ) # # # # D(y ,y ) # = 1. n−1 n / e. In this case: Case (7): n = n − 2, n − 1 ∈ δ, n ∈ Pn−1 (z,σ ) + iPn (z,σ ) n−1 (gd−1 (z,σ )) 1+ian−1 = bn−1 (σ ) + (Sn−1 + iSn )(z,σ ) 1+ian−1 zd |n−1 (gd−1 (z,σ ))| n−1 (gd−1 (z,σ  )) 1+ian−1 iyn = e + (Sn−1 + iSn )(z,σ  ). 1+ian−1 yn−1 |n−1 (gd−1 (z,σ  ))|

162

Stratification of an orbit space

Now if u = u(yn−1,yn ) = yn−1 ei(yn +an−1 ln(yn−1 )) , the Jacobian of the transformation (yn−1,yn ) → (Re(u),Im(u)) is yn−1 , thus, # # # D(Pn−1,Pn ) # # # # D(y ,y ) # = |yn−1 | = |zd | = |ed (gd (z,σ ))| = |n (gd (z,σ ))|. n−1 n Thus the desired formula (2.6.3) is proved. For each positive Borel function f , the expression of dωσ (v) gives 3 3 3 f (v) dv = f (P (z,σ )) |J (P )|z,σ dz dν(σ ) V 3 3Z = f (v) dωσ (v) dν(σ ). 

Oσ

We turn to the semi-invariance of ωσ , as usual writing α(s)v = sv. As in the completely solvable case, by Proposition 2.2.10, for s ∈ G(σ ), i ∈e / |i (s)| = 1. Hence, for any s ∈ G such that sσ = P (z,σ ), we also get: 4 4 |i (s)| = |i (gd (z,σ ))|. i ∈e /

i ∈e /

Now, for each s ∈ G, z ∈ Z there is a unique zs ∈ Z such that sP (z,σ ) = P (zs ,σ ), and as in the completely solvable case: 3 3 4 f (sv) dωσ (v) = f (sP (z,σ )) |i (gd (z,σ ))| dz Oσ

3 = 3 =

Z

Z

Z

i ∈e /

f (P (zs ,σ )) f (P (zs ,σ ))

4

|i (s −1 gd (zs ,σ ))| dz

i ∈e /

s

Z

|i (gd (z,σ ))| dz

i ∈e /

3 =

4

f (P (z ,σ ))

4

|i (gd (zs ,σ ))|

i ∈e /

4

|i (s)|−1 dz.

i ∈e /

(2.6.4) On the other hand, if ek ∈ δ, then (zs )k = |bek (sP (z,σ ))| = |ek (s)|zk . If ek ∈ e0 \ δ, (zs )k = pek (sP (z,σ )) = ek (s)zk + S(s,z1, . . . ,zk−1 ), and if ek ∈ e2 and ek − 1 ∈ I, (zs )k + i(zs )k+1 = ek (s)(zk + izk+1 ) + (Sk (s,z1, . . . ,zk−1 ) + iSk+1 (s,z1, . . . zk−1 )).

2.6 Structure of the generic layer

163

Finally if ek ∈ e1 \ δ, then ck (zs ,σ )((zs )k + ixek +1 ) = pek (P (zs ,σ )) = pek (sP (z,σ )) = ek (s)ck (z,σ )(zk + ixek +1 ) + (Sk (s,z1, . . . ,zk−1 ) + iSk+1 (s,z1, . . . zk−1 ))   = sign(ek (s))ck (z,σ ) |ek (s)|zk + i|ek (s)|xek +1 + (Sk + iSk+1 )(z1, . . . ,zk−1 ). But recall that ck (z,σ ) = sign(ek (gd (z,σ ))) pek (Xek (σ )σ ) (see (2.5.10)). Thus, ck (zs ,σ ) = sign(ek (gd (zs ,σ ))) pek (Xek (σ )σ ) = sign(ek (sgd (z,σ )s0 )) pek (Xek (σ )σ ) = sign(ek (s)) ck (z,σ ) sign(ek (s0 )), where s0 ∈ G(σ ). Now s0 = exp X, with X ∈ g(σ ), by Proposition 2.2.10, / e, we have: since ek + 1 ∈ ek (s0 ) = ek +1 (s0 ) = e−iλek +1 (X) = 1, thus ck (zs ,σ ) = sign(ek (s))ck (z,σ ). So we get: (zs )k = |ek (s)|zk + Sk (z1, . . . ,zk−1 ). # s # # )# These formulae prove that # D(z i ∈e / |i (s)| in any case. Since D(z) # = -n det(α(s)) = i=1 |i (s)|, we get 3

3 Oσ

f (sv) dωσ (v) =

Z

f (P (zs ,σ ))

i ∈e /

3 =

s

Z

×

4

f (P (z ,σ ))

n 4

4 i ∈e /

|i (gd (zs ,σ ))|

4

|i (s)|−1 dz

i ∈e /

# # # D(zs ) # # |i (gd (z ,σ ))| ## D(z) # s

|i (s)|−1 dz

i=1

= | det(α(s))|−1 = | det(α(s))|

−1

3 3

Z

f (P (z,σ ))

Oσ

4 i ∈e /

f (v) dωσ (v).

|i (gd (z,σ ))| dz

164

Stratification of an orbit space 

Remark 2.6.3 We saw earlier that if σ ∈ O ∩ O  , then gdO (z,σ ) = gdO (z,σ ). Denote this element by s(v). Now suppose that the stability group G(σ ) of σ is a subset of ker s → det(α(s))), and put dμσ (v) = | det(α(s(v)))|−1 dωσ (v). Then μσ is an invariant measure on the orbit Oσ . Indeed, for s0 ∈ G, there is some h(s0,v) ∈ G(σ ) such that s0 s(v) = s(s0 v)h(s0,v), thus det(α(s(s0 v))) = det(α(s0 )) det(α(s(v))), and dμσ (s0 v) = | det(α(s(s0 v)))|−1 dωσ (s0 v) = | det(α(s0 ))|−1 | det(α(s(v)))|−1 dωσ (s0 v) = dμσ (v). Example 2.6.4 Let V = R4 , and the basis (f1,f2 = f1,f3,f4 = f4 ) of U = Vc and the action of the commutative two-dimensional Lie algebra with basis (A,X) given by: Af1 = (1 + i)f1, Af3 = (1 + i)f3,

Xf1 = f3,

and the conjugate of these expressions, thus ⎡

(1 + i)v1 ⎢(1 − i)v 1 M(v) = ⎢ ⎣(1 + i)v3 (1 − i)v 3

⎤ 0 0⎥ ⎥. v1 ⎦ v1

So we immediately get that the minimal layer is  = {1,3}(1,2) . More precisely, e1 = 1 ∈ e1 ∩ δ and e2 = 3 is in e1 \ δ, moreover T (v) is the identity. Following Proposition 2.6.1, the cross-section of  is parametrized by x2 ∈ R/2πZ, x4 ∈ R, and σ (x2,x4 ) = (eix2 ,e−ix2 ,ix4 eix2 , − ix4 e−ix2 ). Each orbit is parametrized by (z1,z2 ) ∈ Z = (0,∞)×R, and using coordinates for v from the complex basis (fi ), the map P is: P (z,σ ) = P (z1,z2,x2,x4 ) = (z11+i eix2 ,z11−i e−ix2 ,z1i eix2 (z2 + iz1 x4 ),z1−i e−ix2 (z2 − iz1 x4 )).

2.6 Structure of the generic layer

165

Now the measure dωσ is the measure z12 dz1 dz2 on Z, dν (σ ) is dx2 dx4 on R/2πZ × R and the decomposition of dv is here: 3 3 3 f (v) dv = f (P (z,σ )) dωσ (P (z,σ )) dν (σ )   Oσ 3 3 = f (z11+i eix2 ,z11−i e−ix2 ,z1i eix2 (z2 + iz1 x4 ),z1−i R/2π Z×R Z × e−ix2 (z2 − iz1 x4 )) z12 dz1 dz2

dx2 dx4 .

Finally, if s = etA ∈ G, then sP (z,σ ) = P (zs ,σ ), with z1s = et z1 , z2s = et z2 , therefore (z1s )2 dz1s dz2s = e4t z12 dz1 dz2 = | det(s)|z12 dz1 dz2 , and 3 3 1 f (sP (z,σ )) dωσ (P (z,σ )) = f (P (z,σ )) dωσ (P (z,σ )). | det(s)| Oσ Oσ 

3 Unitary representations

3.1 Unitary representations Harmonic analysis on solvable Lie groups is the study of unitary representations of these groups. Here we recall the basic notions on unitary representations in the general setting of a separable, locally compact group G.

3.1.1 Haar measure for simply connected solvable Lie groups Let G be a locally compact, second countable group. There is a nontrivial, nonnegative, Radon measure μG on G, and there is a positive function G on G, such that the measure μG is left-invariant, and the measure −1 G μG is right-invariant. Thus for all φ ∈ Cc (G), 3 3 φ(yx) dμG (x) = φ(x) dμG (x), G

and

G

3 φ(xy)dμG (x) = G (y) G

−1

3 φ(x)dμG (x). G

The measure μG is called left Haar measure on G, and is unique up to a scalar multiple; G is called the modular function. If the group G is such that G = 1, the left Haar measure is also right-invariant; in this case, we say that the group G is unimodular. If G is a Lie group, it is easy to build a left-invariant volume form on G, using the manifold structure of G, and this gives the existence of left Haar measure. Indeed, fix a basis (X1, . . . ,Xn ) of the Lie algebra g of G, consider the left-invariant vector fields ξsXi defined by: # d ## Xi φ(s exp tXi ). ξs φ = dt # t=0

166

3.1 Unitary representations

167

At each point s ∈ G, the tangent vectors ξsXi define a basis of the tangent space Ts (G) to G at s. By definition, if  is an alternate n-form on g, the relation ωs (ξsX1 ,ξsX2 , . . . ,ξsXn ) = (X1, . . . ,Xn ) defines a left-invariant volume form on G, thus a left-invariant measure. Conversely, each left-invariant form on G can be so written, and since two alternate n-forms on g are proportional, then for any two left-invariant volume forms ω, ω on G, there is a real constant C such that ω = Cω. In particular, we say that the Haar measure defined by choosing  = dX1 ∧ · · · ∧ dXn is the Haar measure associated to the Lebesgue measure dX defined by the choice of the basis (X1, . . . ,Xn ). In the case where G is a connected and simply connected solvable Lie group, the existence of an explicit Haar measure is also directly proved using a weak Malcev basis and the associated second kind coordinates on G (see Corollary 1.7.11). It is also easy to compute the function G in this setting. Fix a good Jordan–Hölder basis (Zj )nj=1 for gc with si = span{Z1, . . . ,Zi }, 1 ≤ i ≤ n. Recall I = {1 ≤ i ≤ n : si = si }. Let (Xj ) be a weak Malcev basis for g associated to the chosen Jordan–Hölder sequence. We have seen that g l (t1, . . . ,tn ) := exp tn Xn · · · exp t1 X1 is a global homeomorphism of Rn onto G, as is g r (t1, . . . ,tn ) := exp t1 X1 · · · exp tn Xn . Lemma 3.1.1 The left Haar measure on G associated to the Lebesgue measure dX is given by 3 3 l φ(s)dμ (s) = φ(g l (t))dt φ ∈ Cc (G). G

Rn

Similarly a right Haar measure is defined by 3 3 r φ(s)dμ (s) = φ(g r (t))dt, φ ∈ Cc (G). Rn

G

Proof: Here is the proof of the first point, the same argument works for the second point. By induction on n: true for n = 1. Assume true for n − 1. Let Gn−1 = exp span{X1, . . . ,Xn−1 }. By induction 3 φ → φ(exp tn−1 Xn−1 · · · exp t1 X1 )dt1, . . . dtn−1 Rn−1

defines a left-invariant measure on Gn−1 . Let a ∈ G; then a = exp sXn · b where b ∈ Gn−1 , s ∈ R. Now it is easily seen that a exp tn Xn = exp(s + tn )Xn b

168

Unitary representations

where b ∈ Gn−1 . Given φ ∈ Cc (G), the induction hypothesis and translation invariance of Lebesgue measure on R imply 3 φ(a exp tn Xn · · · exp t1 X1 )dt1 · · · dtn Rn  3 3 = φ(exp(s +tn )Xn b exp tn−1 Xn−1 · · · exp t1 X1 )dt1 · · · dtn−1 dtn R Rn−1  3 3 = φ(exp(s + tn )Xn exp tn−1 Xn−1 · · · exp t1 X1 )dt1 · · · dtn−1 dtn n−1 R 3 R × φ(exp tn Xn · · · exp t1 X1 )dt1 · · · dtn . Rn

The measure dμl is associated to the volume form ω on G defined as follows: for each i, define the vector field ηi by: # d ## i φ(exp tn Xn · · · exp(ti + u)Xi · · · exp t1 X1 ), ηexp tn Xn ··· exp t1 X1 φ = du #u=0 then at each point s ∈ G, the tangent vectors ηsi form a basis of the tangent space Ts G, and the value of the volume form ωs is given by ωs (η1, . . . ,ηn ) = 1. Now since, at the origin, ηei = ξeXi , thus by left invariance, ωs (ξ X1 , . . . ,ξ Xn ) = ωe (ξ X1 , . . . ,ξ Xn ) = dX(X1, . . . ,Xn ). This means that the measure dμl is the Haar measure associated to the Lebesgue measure dX. Suppose now that the group G is exponential: the exponential map is a diffeomorphism from g to G. In this case, we can use canonical coordinates of the first kind, and identify G with the vector space g. The g-basis (X1, . . . ,Xn ) being still fixed, we can express the Haar measure in these coordinates. Lemma 3.1.2 Suppose that the Lie group G is exponential. Then the Haar measure associated to the Lebesgue measure dX is: # # 3 3 # −ad(Y ) # 1 − e # # φ(s) dμl (s) = φ(exp Y ) #det # dY . # ad(Y ) # G g Proof: Let us now consider the vector fields ζ i defined by: # d ## i φ = φ(exp(Y + tXi )). ζexp Y dt #t=0 X

j i Let us fix Y and compare ζexp Y and the vectors ξexp Y , by writing:

exp(Y + tXi ) = exp Y exp(tUi + t 2 Ui + · · · ).

(3.1.1)

3.1 Unitary representations

169

Then for each smooth function φ, # d ## Ui i φ = φ(exp(tUi + t 2 Ui + · · · )) = ξexp ζexp Y Y φ. dt #t=0  Denote by fY : g → g the linear map defined by fY (Xi ) = Ui = nj=1 aj i Xj  Xj i (1 ≤ i ≤ n). Then we have ζexp j aj i ξexp Y and if ω is the left-invariant Y = form defining dμl , ωexp Y (ζ 1, . . . ,ζ n ) = det(fY )ωexp Y (ξ X1 , . . . ,ξ Xn ) = det(fY ). This can be written as: ωexp j xj Xj (ζ 1, . . . ,ζ n ) = det(fj xj Xj ) (dx1 ∧ · · · ∧ dxn )(ζ 1, . . . ,ζ n ), or, simply, ωexp Y = det(fY )dY , and implies: 3 3 φ(s) dμl (s) = φ(exp Y ) |det(fY )| dY . G

g

Now by differentiating Relation (3.1.1) at 0, we get: # d ## exp −Y exp(Y + tXi ) fY (Xi ) = Ui = d exp0 (Ui ) = dt #t=0 = dlexp −Y |exp Y ◦ d expY (Xi ) = Thus fY =

1−e−ad(Y ) ad(Y ) ,

1 − e−ad(Y ) (Xi ). ad(Y )

for any Y ∈ g. This proves the lemma.

The proof of the following corollary is left as an exercise for the reader: Corollary 3.1.3 If N is a simply connected nilpotent Lie group, a Haar measure on N is given by: 3 3 f (s)dμN (s) = f (exp Y ) dY . N

n

More generally, if G = N H , where N is a normal simply connected nilpotent subgroup, and H is locally compact, with Haar measure dμH , acting on n by automorphisms α(h) of n, then a left Haar measure on G is given by: 3 3 3 f (s) dμG (s) = f (exp Y,h) |det α(h)|−1 dY dμH (h). G

H

n

The modular function  = G : G → R∗+ is defined by the identity 3 3 φ(sx)(x) dμl (s) = φ(s) dμl (s). G

G

170

Unitary representations

This function does not depend on the choice of the left-invariant measure dμl , especially, it does not depend on the choice of the weak Malcev basis (Xj ) associated to the Jordan–Hölder sequence (sj ). Let α be an automorphism of G such that dα(si ) = si , 1 ≤ i ≤ n. Let i ∈ C be defined by dα(Zi ) = i Zi mod si−1 . The following has an elementary proof. Lemma 3.1.4 Given a Jordan–Hölder sequence as above and an automorphism α of G that preserves this sequence, let (X1, . . . ,Xn ) be any weak Malcev basis associated to (Zj ) and let μl be the left-invariant measure on G defined earlier. Then det dα = nj=1 j and 3 3 l φ(α(s))| det dα| dμ (s) = φ(s) dμl (s). G

G

Proof: The first statement is clear. Now either (a) gn−1 is dα-invariant, or (b) gn−2 is dα-invariant and (a) does not hold. We prove the lemma in case (b) only, the proof for (a) being a simplification of the proof given. / Choose Xn−1 and Xn in g such that Zn−1 = Xn + iXn−1 . We have n−1 ∈ R since we are not in case (a), and n = n−1 . Observe that gn−2 is an ideal and by the Lemma 1.4.8, [Xn,Xn−1 ] ∈ gn−2 . This means that g/gn−2 is abelian hence: exp tn Xn exp tn−1 Xn−1 = exp(tn Xn + tn−1 Xn−1 ) b for some b ∈ Gn−2 . At the same time

    α(exp(tn Xn + tn−1 Xn−1 )) = exp dα Re (tn − itn−1 )Zn−1 )    = exp Re n−1 (tn − itn−1 )Zn−1 ) α(c)

l (t  ) = exp tn−2 Xn−2 · · · exp t1 X1 . Then for some c ∈ Gn−1 . Now let gn−2 l α(g l (t)) = α(exp tn Xn exp tn−1 Xn−1 ) α(gn−2 (t  ))   l (t  )). = exp Re n−1 (tn − itn−1 )Zn−1 ) α(bc) α(gn−2

By induction, denoting dt  = dt1 . . . dtn−2 , # # #n−2 3 3 4 ## #    l  l φ α(gn−2 (t  )) ## j ## dt  = φ gn−2 (t  ) dt  . Rn−2 Rn−2 #j =1 #

3.1 Unitary representations

171

Since the determinant of the transformation (tn − itn−1 ) → n−1 (tn − itn−1 ) is |n−1 |2 = n n−1 , then using the left-invariance of the measure dμln−2 on Gn−2 , we get 3



Rn

l

φ α(g (t))

=

R2

× 3 =

Rn−2

n−2 4

|j | dt

j =1

3

3

n 4

    l φ exp Re n−1 (tn − itn−1 )Zn−1 α(bc) α(gn−2 (t  ))

|j | dt  |n−1 |2 dtn−1 dtn

j =1

3

R2

Rn−2

     l φ exp Re n−1 (tn − itn−1 )Zn−1 α(bc) gn−2 (t  )) dt 

× |n−1 |2 dtn−1 dtn  3 3   l   = φ exp Re (tn − itn−1 )Zn−1 gn−2 (t  )) dt  dtn−1 dtn R2

3 =

R2

3 =

Rn

3

Rn−2

Rn−2

 φ exp tn Xn exp tn−1 Xn−1 b

−1



l gn−2 (t  ))

dt



 dtn−1 dtn

 φ(gnl (t)) dt.

Since the inner automorphism α : s → xsx −1 such that dα = Ad(x) satisfies the hypothesis of Lemma 3.1.4, we immediately get: Corollary 3.1.5 Let G be a simply connected solvable Lie group, then its modular function is given by (x) = | det Ad(x)|−1 . In particular, a simply connected nilpotent Lie group is unimodular.

3.1.2 Generalities on unitary representations Left and right translations of a continuous, compactly supported function φ on a locally compact group G are defined by: (Lx φ)(t) = φ(x −1 t),

(Rx φ)(t) = φ(tx).

172

Unitary representations

These operators are uniquely extended, and still denoted Lx , Rx , to the Banach spaces Lp (G) = Lp (G,μG ) (1 ≤ p < ∞), and by the properties of the Haar measure, we get: Lx φ

p

= φ

p,

Rx φ

p

= (x)−1/p φ

p.

If S is any G-space for a continuous left action, and A is a vector space of functions invariant under the G-action, then we shall continue to denote Lx the operator (Lx f )(s) = f (x −1 s) defined on A. For instance as above, if S is locally compact, we shall consider A = Cc (S), or Lp (S). 1. A unitary representation of a locally compact group G is a strongly continuous homomorphism π : G → U (H) from G into the group of unitary operators acting on a separable Hilbert space H = {0}. We frequently use the pair (π,H) to denote this representation. If H is understood, we just write π, and we also say that π acts in H. Similarly, we may write Hπ to designate the space of the representation π . The strong continuity property means that, for any f ∈ H, the map G → H, s → π(s)f is continuous. 2. Let (π1,H1 ) and (π2,H2 ) be unitary representations of G. A bounded operator T : H1 → H2 is an intertwining operator if, for any s ∈ G, T π1 (s) = π2 (s)T . The space of such intertwining operators is denoted by C(π1,π2 ). We also write C(π ) = C(π,π ). 3. Representations (π1,H1 ) and (π2,H2 ) of G are equivalent if they have a unitary intertwining operator: there is a unitary operator U belonging to C(π1,π2 ). This is an equivalence relation. Each unitary representation is equivalent to a unitary representation acting in Cn , n = 1, . . . ,∞ (here C∞ is the space 2 of square summable sequences). With a slight abuse of language, the set Rep(G) of all unitary representations is defined as the set of unitary representations acting in Cn for some n. 4. Let (π,H) be a unitary representation of G; a closed subspace K of H is a invariant subspace if π(s)(K) ⊂ K for any s ∈ G. This is equivalent to say that the orthogonal projection on K belongs to C(π ). If K = 0, we can restrict π to K, by writing πK : K → K, πK (s)v = π(s)v, identified with a vector in K (v ∈ K). In this case, we say that (πK,K) is a subrepresentation of (π,H). Observe that if K is a closed invariant subspace in H, then its orthogonal complement K⊥ is also a closed invariant subspace of H. If v = 0 is a vector of H, the smallest closed

3.1 Unitary representations

173

invariant subspace containing v is the closure H(v) of the space span{π(s)v : s ∈ G}. If H(v) = H, we say that v is a cyclic vector. If the representation (π,H) does not admit any closed invariant subspace, except {0} and H, then we say that (π,H) is an irreducible representation of G. Clearly if (π ,H ) is equivalent to (π,H), then (π ,H ) is irreducible if and only if (π,H) is irreducible. Example 3.1.6 A unitary representation of G acting on H = C is called a unitary character of G, and is frequently denoted by χ . Of course every character is irreducible. If χ (s) = 1 for all s ∈ G then χ will be called the trivial representation. For instance if (χ,C) is a character of G = Rn , the continuity condition implies that there is  ∈ (Rn )∗ such that χ (s) = ei(s) . More generally, a representation (π,H) is said to be trivial if π(s) = 1H for every s ∈ G.  Example 3.1.7 Let G = Rn , H = L2 (Rn ), put: (L(s)f )(t) = (Ls f )(t) = f (t − s)

(s,t ∈ Rn ).

Clearly the operator L(s) is unitary, and by definition L(s + t) = L(s)L(t). It is easy to prove that the map s → L(s) is not continuous for the operator norm on U (L2 (Rn )), but L is strongly continuous: if f is a continuous, compactly supported function, then L(s)f − L(0)f → 0 if s → 0, and using the uniform boundedness of L(s) (s ∈ Rn ) this property holds for each f ∈ L2 (Rn ). Thus L is a unitary representation of Rn , we call it the left regular representation of Rn . Suppose now E is a Borel subset of Rn with positive measure and such that its complementary subset E c has also positive measure. Consider the space KE of L2 -functions f whose Fourier transform fˆ has support in E, then KE is a proper, closed invariant subspace of H. Thus L is not irreducible.  5. Let G be a separable locally compact group. The left regular representation L of G acts in L2 (G,μG ) by (L(s)f )(x) = f (s −1 x). Just as for G = Rn , L is strongly continuous, and is generally not irreducible. The right regular representation is the unitary representation (R,L2 (G)) defined by: (R(s)f )(x) = f (xs) (s)1/2 .

174

Unitary representations

Example 3.1.8 Let G be the Heisenberg group of dimension three, realized as in Example 1.6.4: G = R3 with group product (x,y,z)(x ,y ,z ) = (x + x ,y + y ,z + z + xy  ). Define a unitary representation of G as follows: fix λ ∈ R \ {0} and for (x,y,z) ∈ G define a unitary operator π(x,y,z) = πλ (x,y,z) on L2 (R) by (π(x,y,z)f )(t) = eiλz e−iλyt f (t − x), f ∈ Cc (R). A direct computation shows that π = πλ is a homomorphism of G into the unitary operators on L2 (R). To prove that π is continuous, let f ∈ Cc (R). Then for x, y and z sufficiently small, (π(x,y,z)f )(t) − f (t) ≤ eiλ(z−yt) f (t − x) − eiλ(z−yt) f (t) + eiλ(z−yt) f (t) − f (t) < ε.

(3.1.2)

Since π(x,y,z) is uniformly bounded, (3.1.2) holds for any element of L2 (R). Thus π is a unitary representation of the Heisenberg Lie group. We claim that π is irreducible. Suppose that K is a closed, nontrivial invariant subspace of L2 (R). Then K is a closed translation invariant subspace of L2 (R). But then there is a Borel subset E of R such that K = KE , where KE is as defined in the previous example (see for example the book by Rudin [86]). Now by invariance under the π(0,y,0), K also contains the subspace  KE−y , so K = L2 (R). Example 3.1.9 Suppose now G is the affine group Aff(R) defined in Example 1.6.5, define (τ,H) by putting H = L2 (R) and: (τ (x,a)f )(y) = f (a −1 (y − x))a −1/2

(x, y ∈ R, a > 0).

It is easy to verify that τ (x,a) is a unitary operator, τ ((x,a)(x ,a  )) = τ (x + ax ,aa  ) and by similar argument, it can be proved that as above τ is strongly continuous. This proves that τ is a unitary representation of G. Using the Fourier transform, we see that (τ,L2 (R)) is unitary equivalent to the representation (τ,L ˆ 2 (R)), where: (τˆ (x,a)fˆ)(ξ ) = eiξ x fˆ(aξ )a 1/2 . It is clear that the subspace L2 (0, + ∞) in L2 (R) is invariant under the action of τˆ (x,a), thus τ is not irreducible. In fact, this space and its orthogonal complement are the only closed nontrivial τˆ -invariant subspaces in L2 (R). Indeed, if K is an invariant closed subspace of H, then it is translation invariant, there is a Borel subset E, such that K = KE , and after Fourier transform, the space Kˆ is the space of functions fˆ with support in E. By

3.1 Unitary representations

175

invariance under the τˆ (0,a), E is invariant under dilations E → a −1 E (a > 0). If the measure of E ∩ (0,∞) is positive, then Kˆ contains L2 (0,∞), and the same holds for (−∞,0). More generally, let V be a finite-dimensional vector space over R, let H be a separable locally compact group, and α : H → GL(V ) a representation. Fix a Lebesgue measure on V, and let G = V α H be the generalized affine group. The quasi-regular representation τ of G in L2 (V ) is defined by τ (v,h)f (w) = f (α(h−1 )(w − v))| det α(h)|−1/2 .

(3.1.3)

Using the Fourier transform, (τ,L2 (V )) is unitarily equivalent to the representation (τ,L ˆ 2 (V ∗ )) given by (τˆ (v,h)g)() = χ (v)g(α ∗ (h)−1 )| det(α ∗ (h)|−1/2 where χ is the unitary character defined by χ (v) = ei(v) . Note that if H ⊂ GL(V ), and α is the standard representation, then α ∗ (h)−1  = t h, where t h  is the transpose of h, and | det α ∗ (h)|−1/2 = | det h|1/2 . The Shur Lemma describes the space C(π1,π2 ), if these representations are irreducible: Lemma 3.1.10 Let (π1,H1 ) and (π2,H2 ) be irreducible unitary representations of G. Then: (1) the space C(π1 ) is the space of scalar operators: C(π1 ) = C 1H1 , (2) if the representations (π1,H1 ) and (π2,H2 ) are equivalent, then dim(C(π1,π2 )) = 1, (3) if the representations (π1,H1 ) and (π2,H2 ) are not equivalent, then C(π1,π2 ) = {0}. Proof: We present the idea of the proof; the reader can find more details in the book by G. Folland [40]. Let T ∈ C(π1 ). Then its adjoint T ∗ is also in C(π1 ), and the operators T + T ∗ , i(T − T ∗ ) are in C(π1 ). Using the spectral theorem for T + T ∗ , we write: 3 ∗ λ dPλ . T +T = R

Since T + T ∗ commutes with the π1 (s), then each Pλ is in C(π1 ), but (π1,H1 ) is irreducible thus Pλ is 0 or 1H1 , this implies that there is λ ∈ R such that T + T ∗ = λ 1H1 , the same holds for i(T − T ∗ ), hence there is a ∈ C such that T = a 1H1 .

176

Unitary representations

If U is a unitary operator in C(π1,π2 ), then any intertwining operator in C(π1,π2 ) can be written U T with T ∈ C(π1 ), thus the second point is a direct consequence of the first one. Suppose now (π1,H1 ) and (π2,H2 ) are not equivalent, and let T be in C(π1,π2 ). Then T ∗ T ∈ C(π1 ), and the polar decomposition of T is T = V (T ∗ T )1/2 , where V is a partial isometry between H1 and H2 . By construction V ∈ C(π1,π2 ), thus the kernel of V and the image of V are invariant subspaces of H1 and H2 . By irreducibility, they are 0 or H1 (resp. H2 ). If ker V = 0 and V (H1 ) = H2 , V is a unitary operator and (π1,H1 ) and (π2,H2 ) are equivalent. Therefore V = 0 and T = 0. Corollary 3.1.11 Every irreducible unitary representation (π,H) of a locally compact abelian group G is a character. Proof: By the Schur Lemma, each operator π(s) is scalar, thus irreducibility of (π,H) imposes that dim H = 1. Example 3.1.12 Recall the irreducible unitary representation πλ of the Heisenberg group: (πλ (x,y,z)f )(t) = eiλz e−iλyt f (t − x). 

If λ = λ , and T is in C(πλ,πλ ), then T (eiλz f ) = eiλ z T (f ) (f ∈ L2 (R)), thus T = 0, and πλ and πλ are inequivalent. Using the Fourier transform: 3 (Ff )(ξ ) = f (t) eitξ dt, R

we define a representation πˆ λ of the Heisenberg group, equivalent to πλ : (πˆ λ (x,y,z)fˆ)(ξ ) = eiλz eixξ fˆ(ξ − λy). Thus F belongs to C(πλ, πˆ λ ). By the Schur Lemma, C(πλ, πˆ λ ) = C F .



6. Let H be a separable Hilbert space, and let (Hi )i∈I be a finite or infinite sequence of closed subspaces in H, which are pairwise orthogonal and such that the Hilbert sum: 2 6



2 Hi = vi : vi ∈ Hi , vi < +∞ , i∈I

i∈I

i

is H. Suppose that (π,H) is a unitary representation of G such that each Hi is invariant. Then the restriction of each operator π(s) to the subspace Hi defines a representation (πi ,Hi ) of G. In this situation, we say that

3.1 Unitary representations

177

(π,H) is the direct sum of the representations (πi ,Hi ) and write π = ⊕i πi . 7. Let H1 and H2 be separable Hilbert spaces, and define the Hilbert tensor product H1 ⊗ H2 , as follows: pick a Hilbert basis (ej ) of H2 , and put: ⎧ ⎫ ⎨ ⎬

T ej 2 < ∞ . H1 ⊗ H2 = T : H2 → H1 : T is conjugated linear and ⎩ ⎭ j

Then T is bounded and for any other Hilbert basis (ek ) of H2 , we have:

T ek 2 = |ej ,ek |2 T ej 2 = T ej 2, k

j,k

j

therefore the condition does not depend on the choice of the basis. Put  T 2 = j T ej 2 ; this defines a norm on H1 ⊗ H2 associated to the sesquilinear form:

T ,S = T ej ,Sej . j

If v ∈ H1 and w ∈ H2 , then the operator denoted by v ⊗ w and defined by: (v ⊗ w)(u) = w,uv is in H1 ⊗ H2 . Now if (ai ) is an orthonormal basis of H1 , then the set {ai ⊗ ej } is a orthonormal subset of H1 ⊗ H2 , and if T ∈ H1 ⊗ H2 and   tij = T ej ,ai , then i,j |tij |2 = j T ej 2 = T 2 . Thus the series  i,j tij ai ⊗ ej converges in the norm defined earlier, and its sum is T :

tij ai ⊗ ej (u) = tij ej ,uai = ej ,uT ej = T (u). ij

ij

j

This implies that H1 ⊗ H2 is a Hilbert space, with Hilbert basis (ai ⊗ ej ). Observe that if H1 = C, H1 ⊗ H2 is canonically isomorphic to H2 . Now suppose (π1,H1 ) and (π2,H2 ) are unitary representations of G; we define the unitary representation π1 ⊗ π2 acting on H1 ⊗ H2 by: (π1 ⊗ π2 )(s)(T ) = π1 (s) T π2 (s)−1 . It is easy to prove that (π1 ⊗ π2 )(s) T is in H1 ⊗ H2 , that (π1 ⊗ π2 )(s) is a unitary operator, and π1 ⊗ π2 is a unitary representation. Moreover a direct computation gives (π1 ⊗ π2 )(s)(v ⊗ w) = π1 (s)v ⊗ π2 (s)w. We call the representation (π1 ⊗ π2,H1 ⊗ H2 ) the tensor product of the representations (π1,H1 ) and (π2,H2 ).

178

Unitary representations

If (π1,H1 ) (resp. (π2,H2 )) is a unitary representation of G1 (resp. of G2 ), we define their direct product π1 × π2 , which is a unitary representation of G1 × G2 , acting on H1 ⊗ H2 by: (π1 × π2 )(s1,s2 )(v1 ⊗ v2 ) = π1 (s1 )v1 ⊗ π2 (s2 )v2 . Denoting by π˜ i (s1,s2 ) = πi (si ) the trivial extensions of πi to G = G1 × G2 (i = 1,2), then π1 × π2 = π˜ 1 ⊗ π˜ 2 . Example 3.1.13 Recall the simply connected group E˜ 2 defined after Result 1.6.11 in Chapter 1: ⎫ ⎧ ⎤ ⎡ t 0 0 e ⎬ ⎨ E˜ 2 = C  R = (z,t) = ⎣ eit z ⎦ : t ∈ R, z ∈ C . ⎭ ⎩ 1 Writing E˜ 2 as pairs (z,t), we have (z,t)(z,t  ) = (z + eit z,t + t  ). Let r > 0, and for each (z,t) ∈ E˜ 2 define the unitary operator in H = L2 (T) by (π(z,t)f )(eix ) = eirRe(e

−ix z)

f (ei(x−t) ).

The proof that π is a unitary representation of E˜ 2 is similar to that of Example 3.1.8. For each 0 ≤ ν < 1 the formula χν (z,t) = eiνt defines a unitary character of E˜ 2 . Thus for each ν, 0 ≤ ν < 1, we obtain a unitary representation πν = χν ⊗ π of E˜ 2 acting in L2 (T). Now the center of E˜ 2 is the discrete subgroup {0} × 2πZ, and πν (0,2π k) = e2π iνk 1L2 (T) . It follows that if ν = ν  , then the two representations πν and πν  are not  equivalent. We shall prove in Chapter 5 that each πν is irreducible.

3.1.3 The algebra L1 (G) and its representations Let G be a locally compact group and μG a left Haar measure on G. Exactly as in the case of the real line, the Banach space L1 (G) = L1 (G,μG ) is an associative algebra for the convolution product defined by: 3 (φ ∗ ψ)(s) = φ(t)ψ(t −1 s) dμG (t). G

3.1 Unitary representations

179

In particular, when φ and ψ are μG -integrable functions on G then t → φ(t)ψ(t −1 s) is integrable for almost every s ∈ G. The above then defines an element φ ∗ ψ in L1 (G) satisfying φ∗ψ

1

≤ φ

1

ψ

1.

Recalling the left translation Ls φ(t) = φ(s −1 t), one has Ls φ ∗ψ = Ls (φ ∗ψ). There is also an involution on L1 (G) defined by: φ ∗ (s) = (s)−1 φ(s −1 ). It has the following properties: 1. (φ ∗ )∗ = φ, 2. (φ ∗ ψ)∗ = ψ ∗ ∗ φ ∗ , 3. φ ∗ 1 = φ 1 . Proofs of all these points are left as an exercise; a good reference is (again) the book by Folland [40]. Let (π,H) be a unitary representation of G. First we define an operator π(φ) for φ ∈ L1 (G). Observe that the sesquilinear form: 3 (v,w) → φ(s)π(s)v,w dμG (s) G

is bounded on H × H: # #3 # # # φ(s)π(s)v,w dμG (s)# ≤ φ # #

1

v

w .

G

This defines a bounded operator π(φ) : H → H such that: 3 φ(s)π(s)v,w dμG (s) = π(φ)v,w. G

We may write π(φ) =

7

G φ(s)π(s)

dμG (s).

Lemma 3.1.14 Let (π,H) be a unitary representation of G and let φ,ψ ∈ L1 (G). Then (1) (2) (3) (4)

π(φ) ≤ φ 1 , π(φ ∗ ψ) = π(φ) π(ψ), π(φ)∗ = π(φ ∗ ), π(φ)v = 0 for all φ ∈ L1 (G) implies v = 0.

This means that the map π : L1 (G) → L(H) (φ → π(φ)) is a nondegenerate representation of the involutive Banach algebra L1 (G).

180

Unitary representations

Proof: The first point is a direct consequence of our estimate |π(φ)v,w| ≤ φ 1 v w . For the second point, by left invariance of dμG (s), we can write: 3 π(φ) π(ψ)v,w = φ(t)π(t)π(ψ)v,w dμG (t) 3G = φ(t)π(ψ)v,π(t)∗ w dμG (t) G 3 3 = φ(t) ψ(s)π(s)v,π(t)∗ w dμG (s) dμG (t) G G 3 3 = φ(t)ψ(t −1 s)π(s)v,w dμG (t)dμG (s) G G

= π(φ ∗ ψ)v,w. For the third point, for any v, w in H, 3 3 φ(s)π(s)v,w dμG (s) = φ(s)v,π(s −1 )w dμG (s) π(φ)v,w = G G 3 3 −1 = v,φ(s)π(s )w dμG (s) = v,φ ∗ (s)π(s)w dμG (s) G

G

= v,π(φ ∗ )w. Finally, let {φu } be an approximate identity in L1 (G), i.e., starting with a decreasing family of compact neighborhoods {Ku } of e in G, such that ∩u Ku = {e}, φu is a positive continuous function, has L1 -norm 1 and support in Ku . Then for each v ∈ H, 3 π(φu )v − v ≤ φu (s) π(s)v − v dμG (s) → 0. Ku

If π(L1 (G))v = {0}, then v = 0. Example 3.1.15 Let G = Rn and let π = χξ be the character χξ (t) = eiξ t . Then for φ ∈ L1 (Rn ), χξ (φ) = (F φ)(ξ ) is the value at ξ of the Fourier transform of φ.  Let L be the left regular representation of a separable locally compact group G; then 3 φ(t)(π(t)f )(s)dμG (t) (L(φ)f )(s) = G 3 = φ(t)f (t −1 s) dμG (t) = (φ ∗ f )(s). G

3.1 Unitary representations

181

Lemma 3.1.16 Let G be a separable locally compact group. Then the left regular representation is a faithful representation of L1 (G). Proof: Recall the approximate identity {φu } in L1 (G). We have φu ∗ φ − φ 1 → 0 for each L1 -function φ. Using conjugation, this implies that φ ∗ φu → φ in L1 (G). Since each φu is continuous with compact support, φu ∈ L2 (G) ∩ L1 (G), and by the preceding remark, L(φ)φu = φ ∗ φu → φ in L1 (G). Thus if L(φ) = 0, then φ = 0. Let Hi (i = 1,2) be Hilbert spaces, we denote by L(H1,H2 ) (L(H) if H1 = H2 = H) the space of bounded linear operators T : H1 → H2 . Equipped with the usual composition of operators, the adjoint operation and the operator-norm, L(H) is an involutive Banach algebra. Recall the left and the right translations Ls and Rs . Let (π,H) be a unitary representation of G. We have: π(Ls φ) = π(s) π(φ), and π(Rs φ) = (s)−1 π(φ) π(s −1 ). Observe that for any nondegenerate representation of the involutive algebra L1 (G), the set: D = {π(φ)v : φ ∈ L1 (G), v ∈ H} is dense in H. It is a simple consequence of Properties 3 and 4 in Lemma 3.1.14. The following converse of Lemma 3.1.14 holds. Lemma 3.1.17 Let π : L1 (G) → L(H) be a nondegenerate representation of the involutive Banach algebra L1 (G), then there is a unitary representation (π,H) of G such that 3 φ(s)π(s) dμG (s). π(φ) = G

Moreover (π,H) is irreducible if and only if for any closed subspace K of H, π(L1 (G))K ⊂ K implies K = {0} or K = H. Proof: First, if ρ is a unitary representation of G, for any ψ ∈ L1 (G), for any v ∈ H and any x ∈ G, ρ(x)ρ(ψ)v = ρ(Lx ψ)v.

182

Unitary representations

Thus we start with the representation π of L1 (G) and define π˜ (x) on the vector π(ψ)v ∈ D by putting: π˜ (x)π(ψ)v = π(Lx ψ)v. This is well-defined since if π(ψ)v = 0, then π(Lx ψ)v = lim π(Lx (φu ∗ ψ))v = lim π(Lx φu ∗ ψ)v = 0. u

u

The operator π˜ (x) is bounded, with operator norm no greater than 1, since: π˜ (x)π(ψ)v = lim π(Lx φu )π(ψ)v ≤ lim Lx φu u

u

1

π(ψ)v = π(ψ)v .

So, since D is dense, π˜ (x) is extended into a bounded operator defined on H in a unique way. We claim that the map x → π˜ (x) is a unitary representation of G. First, we have: π˜ (xy)π(ψ)v = π(Lxy ψ)v = π(x)π(L ˜ ˜ (x)π˜ (y)π(ψ)v, y ψ)v = π thus π˜ (xy) = π˜ (x)π˜ (y), since by definition π˜ (e) = 1H , then π˜ (x −1 ) = π˜ (x)−1 . Since these two operators have norm no greater than 1, v = π˜ (x −1 )π˜ (x)v ≤ π˜ (x)v ≤ v , thus π˜ (x)v = v , and π(x) ˜ is a unitary operator. Moreover the map x → Lx ψ is continuous, so π˜ is a strongly continuous representation of G. Finally: 3 ψ(x)π(Lx ψ  )v dμG (x) = π(ψ ∗ ψ  )v = π(ψ)π(ψ  )v, π˜ (ψ)π(ψ  )v = G

thus by density of D, π˜ (ψ) = π(ψ). Now let K be a closed subspace of H. Suppose π(L1 (G))K ⊂ K, thus if v ∈ K, for any ψ ∈ L1 (G), π(ψ)v ∈ K, π˜ (x)π(ψ)v ∈ K, and replacing ψ by the φu , π˜ (x)v = lim π˜ (x)π(φu )v ∈ K. u

Therefore π(G)K ˜ ⊂ K. Conversely if the representation π˜ leaves K invariant, then a standard argument using the definition of π(ψ) shows that each π(ψ) leaves K invariant. Thus the unitary representations of G are in one-to-one correspondence with nondegenerate representations of L1 (G), and the same holds for irreducible representations.

3.1 Unitary representations

183

3.1.4 C ∗ -algebras and the unitary dual of G An involutive Banach algebra A is called a C ∗ -algebra if the relation: φ∗φ = φ

2

holds for any φ ∈ A. Let H be a Hilbert space. A norm closed subalgebra of L(H), stable under the involution T → T ∗ is a C ∗ -algebra. Let (π,H) be a unitary representation of the locally compact group G. Then π(L1 (G)) is a subalgebra of the involutive algebra L(H). Since intersection of a family of C ∗ -algebras in L(H) is a C ∗ -algebra, there is a smallest C ∗ -subalgebra in L(H) containing π(L1 (G)). Let us denote by C ∗ (π(L1 (G))) this C ∗ -algebra. Clearly, C ∗ (π(L1 (G))) is the norm closure of π(L1 (G)) in L(H). Similarly, the operators π(s) (s ∈ G) generate an involutive algebra, namely  the set G of all finite linear combinations k ak π(sk ), thus there is the smallest C ∗ -algebra containing π(G), denoted by C ∗ (π(G)). C ∗ (π(G)) is the norm closure in L(H) of G. In fact, these two C ∗ -algebras can be completely disjoint, as the following example shows. Example 3.1.18 Let G = R, H = L2 (R) and the representation π defined by: (π(x)f )(t) = eitx f (x). Then for each ψ ∈ L1 (G), the operator π(ψ) is simply the multiplication operator Mψˆ by the Fourier transform of ψ: 3 ˆ (π(ψ)f )(t) = ψ(x) eitx f (t) dt = ψ(t)f (t). R

ˆ Recall that ψˆ is a continuous function, such that lim|t|→∞ ψ(t) = 0 (we say ˆ ψˆ ∈ C0 (R)). Recall also that the operator norm of Mψˆ is the sup norm of ψ. ∗ Therefore C0 (R), equipped with the supremum norm ∞ is a C algebra, isomorphic to MC0 (R) . Thus we can write C ∗ (π(L1 (G))) ⊂ C0 (R). (In fact, it can be proved that equality holds.)  On the other hand, the operator k ak π(sk ) is also a multiplication operator by a continuous function: % $ % $

isk t ak π(sk )f (t) = ak e f (t). k

k

184

Unitary representations

Let us now define the notion of almost periodic function ψ: a continuous function ψ is almost periodic if for each ε > 0, there is a finite sequence of real numbers b1, . . . ,bn such that for each t ∈ R, there is some bj such that: sup |(Lt ψ)(y) − (Lbj ψ)(y)| < ε.

y∈R

Since each continuous periodic function with period T is characterized by its restriction to [0,T ], and is uniformly continuous on this interval, it is easy to prove that any continuous periodic function is almost periodic. It can also be proved that the sum of two almost periodic functions is still an almost periodic function (see the book by G. Folland [40], for instance). Finally, the uniform limit φ of a sequence of almost periodic functions φn is still almost periodic: for each ε > 0, there is φn such that φ − φn ∞ < ε/3, for this φn there is b1, . . . ,bk such that for any t there is bj such that Lt φn − Lbj φn ∞ < ε/3, thus: Lt φ − Lb j φ

∞

≤ L t φ − Lt φ n

∞

+ Lt φn − Lbj φn

+ Lbj φn − Lbj φ

∞

∞

< ε.

Therefore, the space AP of all almost periodic functions is a C ∗ -algebra, isomorphic to the subalgebra MAP of L(L2 (R)). Thus, we can write C ∗ (π(G)) ⊂ AP . Here, too, it is possible to prove that equality holds. The preceding proves that the intersection of the two C ∗ -algebras ∗ C (π(G)) and C ∗ (π(L1 (G))) is reduced to 0. Indeed let Mφ ∈ C ∗ (π(G)) ∩ C ∗ (π(L1 (G))), then φ ∈ C0 (R) ∩ AP . Fix ε > 0 there is b1, . . . ,bk such that for each y ∈ R, Ly φ − Lbj φ ∞ < ε/2 for some bj . There is also some A > 0 such that |t| > A implies |φ(t)| < ε/2. Fix now t ∈ R and y such that |t + y − bj | > A for any j = 1, . . . ,k, then: |φ(t)| = |Ly φ(t + y)| ≤ |Ly φ(t + y) − Lbj φ(t + y)| + |φ(t + y − bj )| < ε/2 + ε/2 = ε. This proves that φ(t) = 0 for each t, or C ∗ (π(L1 (G))) ∩ C ∗ (π(G)) = {0}.



There is classical result about the strong or weak closure of a C ∗ -algebra. If M is a subspace of L(H), we define the commutant M  of M as the set of T ∈ L(H) such that T S = ST for each S ∈ M. The double commutant M  is then the commutant of M  . With this notion, J. von Neumann proves (see [91]):

3.1 Unitary representations

185

Theorem 3.1.19 Let M be a nondegenerate involutive subalgebra of L(H). Then the double commutant M  of M is the weak and the strong closure of M. w

s

Proof: Denote by M and M the weak (resp. strong) closure of M. It is easily s w w w seen that M ⊂ M , that M  ⊂ (M ) , thus M ⊂ M  . So we only have to s prove that M  ⊂ M . Let S be in M  . Recall that any neighborhood of S for the strong topology contains a set:   V(S;ε;v1, . . . ,vk ) = T ∈ L(H) : T vj − Svj < ε (1 ≤ j ≤ k) . Suppose first that k = 1, v = v1 = 0, we then consider the subspace Mv = {T v : T ∈ M} and its closure Mv. Let P be the orthogonal projection on Mv. We claim that P ∈ M  . Let u ∈ Mv, then, for each T ∈ M, there is a sequence (Tn ) in M such that: T u = lim T (Tn v) ∈ Mv. n

⊥

Thus P T u = T u = T P u. Let now u ∈ Mv , then for any T  ∈ M, T  v,T u = T ∗ T  v,u = 0, ⊥

thus T u ∈ Mv ⊥ = Mv , and P T u = 0 = T 0 = T P u. This proves our claim: P ∈ M  . Let u = (1 − P )v, thus Mu = M (1 − P )v = (1 − P ) Mv = 0. Since M is nondegenerate, this implies u = 0, and v ∈ Mv. Now since S is in M  , S is commuting with P , therefore Sv = S P v = P Sv, or Sv ∈ Mv and there is a sequence (Tn ) in M such that Sv = limn Tn v. For sufficiently large n, Tn ∈ M ∩ V(S;ε;v). Suppose now k > 1. Then we consider the direct orthogonal sum of k spaces isomorphic to H: H(k) = H ⊕ H ⊕ · · · ⊕ H. We then replace M by the involutive subalgebra M (k) of L(H(k) ), and S by the operator S (k) where: ⎤

⎡

⎡ S  ⎥ ⎢ (k) ⎦ : T ∈ M , and S = ⎣

T

M (k)

 ⎢ = T (k) = ⎣

..

. T

⎤ ..

⎥ ⎦.

. S

186

Unitary representations

A direct computation shows that M (k) is a involutive subalgebra of L(H(k) ) which is nondegenerate. Moreover, the commutant of M (k) is: ⎡ ⎤ T11 · · · T1k ⎢ .. ⎥ : T ∈ M  . (M (k) ) = ⎣ ... . ⎦ ij Tk1 · · · Tkk This implies that S (k) is in (M (k) ) . Put now v (k) = t [v1,v2, . . . ,vk ]. The preceding proof shows that there is T (k) ∈ M (k) such that:

S (k) v (k) − T (k) v (k) 2 = Svj − T vj 2 < ε2 . j s

This proves that T ∈ M ∩ V(S;ε;v1, . . . ,vk ). In other words, S ∈ M and this achieves the proof of the theorem. In the case of M = C ∗ (π(G)) or C ∗ (π(L1 (G))), this gives: Theorem 3.1.20 Let π be a unitary representation of the locally compact group G. Then (1) T ∈ L(H) belongs to C(π ) = (π(G)) if and only if T π(φ) = π(φ)T for every φ ∈ L1 (G). (2) The C ∗ -algebras C ∗ (π(G)) and C ∗ (π(L1 (G))) have the same closure in the strong and the weak operator topologies. Proof: Let T ∈ C(π ), then for each v and w, 3 ∗ T π(φ)v,w = π(φ)v,T w = φ(x)π(x)v,T ∗ w dμG (x) G 3 = φ(x)T π(x)v,w dμG (x) G 3 = φ(x)π(x)T v,w dμG (x) G

= π(φ)T v,w. Therefore, T π(φ) = π(φ)T for any φ ∈ L1 (G). Conversely, if this relation holds for T , and {φu } is an approximate identity in L1 (G), then: T (π(x)v) = lim T (π(Lx φu )v) = lim π(Lx φu )(T v) = π(x)(T v). u

u

This proves that T ∈ C(π ). For the second point, observe that C ∗ (π(G)) = π(G) = π(L1 (G)) = ∗ C (π(L1 (G))) . Using Theorem 3.1.19, this implies that the strong and weak closures of C ∗ (π(G)) and C ∗ (π(L1 (G))) do coincide.

3.1 Unitary representations

187

Example 3.1.21 Recall the preceding example (Example 3.1.18): G = R, H = L2 (R) and (π(x)f )(t) = eitx f (t). By Fourier transform F , π is unitarily equivalent to the left regular representation L of G: π(x) = F −1 L(x)F . Suppose now that P is an orthogonal projection commuting with all the π(x). Then F −1 P F is the projection onto a translation-invariant subspace in L2 (R), thus P is the restriction operator to a Borel subset E of R, that is P is the multiplication operator by the characteristic function of E. This allows us to determine the common weak (or strong) closure of the two C ∗ -algebras C ∗ (π(L1 (G))) and C ∗ (π(G)). Any self-adjoint operator T commuting with π(G) has spectral projectors P commuting with π(G), this implies that T is the multiplication operator by a real essentially bounded, measurable function f . This yields: π(G) ⊂ ML∞ (R) . Since the opposite inclusion clearly holds, we get the equality: π(G) = ML∞ (R) . Since ML∞ (R) is abelian, ML∞ (R) ⊂ π(G) . Conversely, if T ∈ π(G) = ML ∞ (R) , then T ∈ π(G) = ML∞ (R) , thus we have: C ∗ (π(L1 (G))) = C ∗ (π(G)) = ML∞ (R) .



Let us finally define the reduced C ∗ -algebra of a locally compact group: Definition 3.1.22 Let G be a locally compact group. The C ∗ -algebra C ∗ (L(L1 (G))) is called the reduced C ∗ -algebra of G and denoted Cr∗ (G). Since the left regular representation of L1 (G) is faithful, this means that the reduced C ∗ -algebra of G is the completion of the star algebra L1 (G) with respect to the norm: φ → L(φ) coming from the operator norm on L(L2 (G)). For any locally compact group G, the C ∗ -algebra of G is defined as follows. It can be shown that the supremum: φ =

sup

π∈Rep(C ∗ (G))

π(φ)

is a norm on L1 (G). The completion of L1 (G) with respect to this norm is a C ∗ -algebra called the C ∗ -algebra of G and denoted by C ∗ (G) (see J. Dixmier [29]).

188

Unitary representations

If G is a solvable Lie group, the C ∗ -algebra of G coincides with the reduced (see [29], for instance):

C ∗ -algebra

C ∗ (G) = Cr∗ (G). We do not prove this equality, but we consider Cr∗ (G) as a concrete realization of the C ∗ -algebra of the solvable Lie group G, nevertheless, the precise structure of C ∗ (G), even if G is the Heisenberg group, is not easy to describe (see the works of Ludwig together with Lin and Turowska [59, 63]). If π is a unitary representation of G, then the map φ → π(φ) (φ ∈ L1 (G)) is continuous for the C ∗ -norm on L1 (G), by the definition of this norm. Then π extends in a unique way to a nondegenerate representation of the C ∗ -algebra C ∗ (G). Conversely, any nondegenerate representation of C ∗ (G) gives by restriction to L1 (G) a representation of L1 (G) which is still nondegenerate, thus this representation is coming from a representation of G by Lemma 3.1.17. In other words, Rep (C ∗ (G)) = Rep (G). Also by this lemma, irreducibility of the representation of C ∗ (G) and the corresponding representation of G are equivalent. The unitary dual of G is the set of equivalence classes of irreducible unitary ˆ By the preceding it is identified representations of G. It is denoted by G. with the set of the equivalence classes of the irreducible star representations of C ∗ (G). ˆ Let us define a topology, called the Fell topology on G: ˆ and let (π,Hπ ) be an irreducible unitary represenLet [π ] be a point in G, tation of G with class [π ]. For each finite family of vectors (v1, . . . ,vk ) in Hπ , each compact subset K in G and each ε > 0, define the set U (v1, . . . ,vk ,K,ε) ˆ such that there are vectors w1, . . . ,wk in Hρ as the set of classes [ρ] in G such that: # #  sup #π(x)vi ,vj  − ρ(x)wi ,wj # : 1 ≤ i,j ≤ k, x ∈ K < ε. ˆ is a neighborhood of [π ] if there is (vi ), K and ε We say that V ⊂ G such that U(v1, . . . ,vk ,K,ε) ⊂ V. It can be proved that this topology has a countable basis of neighborhoods, so we use only sequences ([πn ])n∈N to describe this topology, instead of general nets ([πi ])i∈I . ˆ and let ([πn ]) be a sequence in G. ˆ Suppose that Lemma 3.1.23 Let [π] ∈ G, there is a Hilbert space H such that we can choose representatives (πn,Hπn ) and (π,Hπ ) such that H = Hπn = Hπ for any n. Then πn → π if and only if, for any v ∈ H, πn (·)v,v → π(·)v,v uniformly on compact sets of G.

3.1 Unitary representations

189

Proof: Suppose first that for any v, πn (·)v,v → π(·)v,v uniformly on compact sets of G. Then for any v1, . . . ,vk , and any compact set K ⊂ G, each ε > 0, there is N such that n > N implies: # # #(πn (x) − π(x))(vi + avj ),(vi + avj )# < ε. sup x∈K, i,j, a∈{±1,±i}

Put t (a) = (πn (x) − π(x))(vi + avj ),(vi + avj ), then |t (1) − t (−1) + it (i) − it (−i)| = 4|(πn (x) − π(x))vi ,vj | < 4ε. ˆ This proves that [πn ] → [π ] in G. ˆ Let v ∈ H, v = 0. For any Conversely, suppose that [πn ] → [π ] in G. compact K and any ε > 0, there is N such that n > N implies [πn ] ∈ U (v,K ∪ {e},ε), thus there is wn ∈ H such that: | wn

2

− v 2 | < ε, |πn (x)wn,wn  − π(x)v,v| < ε (x ∈ K).

For each n, the vectors wn and

wn v

v have same norm, so there are unitary

operators Un : H → H such that Un wn = we get:

wn v

v. Replacing πn by Un−1 πn Un ,

2 # # # # wn 2 # < ε , # wn πn (x)v,v − π(x)v,v# < ε (x ∈ K). # − 1 2 2 2 v v v

This implies: # # #(πn (x) − π(x))v,v# # #  # # wn 2 wn 2 # πn (x)v,v − π(x)v,v## πn (x)v,v + =# 1− 2 2 v v # # # # #πn (x)v,v# # wn 2 # # # < 2ε. + π (x)v,v − π(x)v,v ≤ε n # # 2 2 v v This ends the proof. An easy generalization of Lemma 3.1.23 is ˆ and let ([πn ]) be a sequence in G. ˆ Suppose that Corollary 3.1.24 Let [π ] ∈ G, there is a Hilbert space H such that we can choose representatives (πn,Hπn ) and (π,Hπ ) such that H = Hπn = Hπ for any n. Then the following are equivalent: (i) πn → π , (ii) for any v ∈ H there is a sequence (vn ) in H, converging to v, and such that πn (x)vn,vn  → π(x)v,v uniformly on compact sets in G,

190

Unitary representations

(iii) for any w in a dense subspace V of H, πn (x)w,w → π(x)w,w uniformly on compact sets in G. Proof: For any n, any vectors vn , v in H, we have: |πn (x)vn,vn  − πn (x)v,v| ≤ |πn (x)vn,vn  − πn (x)vn,v| + |πn (x)vn,v − πn (x)v,v|

(3.1.4)

≤ ( vn + v ) vn − v . This implies that for any sequence (vn ) with limit v, πn (x)vn,vn  is tending to π(x)v,v uniformly on compact subsets if and only if πn (x)v,v is tending to π(x)v,v uniformly on compact subsets. With Lemma 3.1.23, this proves the equivalence between (i) and (ii). Now by the same lemma, (i) implies (iii). Suppose that (iii) holds and let v ∈ H. For each ε > 0, there is w ∈ V such that v − w < ε. Thus for each compact set K and each x ∈ K, |πn (x)v,v − π(x)v,v| ≤ |πn (x)v,v − πn (x)w,w| + |πn (x)w,w − π(x)w,w| + |π(x)w,w − π(x)v,v| ≤ 2(2 v + ε)ε + |πn (x)w,w − π(x)w,w| ≤ 2(2 v + ε + 1)ε if n is sufficiently large. Thus, by Lemma 3.1.23, (i) holds. This achieves the proof.

3.2 Decomposition of representations The idea to generalize the notion of direct sum to that of a continuous direct sum, or direct integral, is due to J. von Neumann (see [92]). It has become an essential tool on harmonic analysis on locally compact groups, especially nonabelian groups. Following the notion of direct integral of Hilbert spaces, we present the notion of a direct integral of operators and of unitary representations. The question of the decomposition of a given unitary representation into irreducible representations is related to the theory of type for algebras of operators. A generalization of the notion of irreducible unitary representation is that of a factor representation, and we give several examples in this section. We present without proof the so-called Central Decomposition Theorem, which gives a canonical decomposition of any unitary representation into factor representations. We then define the notion of type 1 factor and type 1 group, giving examples of type 1 and not type 1 solvable Lie groups. The class

3.2 Decomposition of representations

191

of groups where the decomposition theory into irreducible representations is canonical is the class of type 1 groups.

3.2.1 Direct integrals of Hilbert spaces Let (A,B) be a Borel measurable space, and for each α ∈ A let Hα be a complex separable Hilbert space. A notion of measurability is required in order to construct a single Hilbert space from this data. Definition 3.2.1 A measurable field of Hilbert spaces is a family (Hα )α∈A of nonzero, separable Hilbert spaces with a sequence (ej )j ∈N of vector fields,   ej = ej (α) α∈A ∈ α∈A Hα , such that: (a) for each j and k, the map α → ej (α),ek (α)α is a measurable function on A, and (b) for each α ∈ A, the vector space generated by {ej (α) : j = 1,2, . . .} is dense in Hα . The Gram–Schmidt process can then be applied to each set {ej (α)}j . More precisely, for each α ∈ A, let Vα be the vector space consisting of all finite linear combinations of the ej (α), and α = (α1 < α2 < · · · ) the lexographically minimal sequence such that (ej )j ∈α is a basis of Vα . Put d(α) = dim Hα , and observe that d(α) = |α | = dim Vα . Lemma 3.2.2 For each strictly increasing sequence  = (1 < 2 < · · · < d ) (resp.  = (1 < 2 < · · · )), the set A = {α ∈ A : α = } is measurable. (k)

Proof: For each k, each 1,2, . . . ,k put A1,...,k = {α ∈ A : d(α) ≥ k, α1 =

1, . . . ,αk = k }. Let us now prove by induction, that A(k) 1,...,k is measurable.

(1)

For k = 1, for each α ∈ A, α1 = min{j : ej (α) = 0}. By construction A1 is measurable: (1)

A1 = {α ∈ A : e1 (α) = e2 (α) = · · · = e1 −1 (α) = 0, e1 (α) = 0}. Suppose now that the sets A(k−1) 1,...,k−1 are measurable. Then by construction: * 

(k) (k−1) A1,...,k = α ∈ A1,...,k−1 : det ei (α),ej (α)α 1≤i,j ≤k = 0, and for each  = (1, . . . ,k−1,j ) + 

with j < k , det ei (α),ej (α)α 1≤i,j ≤k = 0 .

192

Unitary representations

This proves that A(k) 1,...,k is measurable. Now A =



(k)

A1,...,k ,

k≤||

so A is also measurable. Proposition 3.2.3 We have the following: (1) For each m = 1,2, . . . ,∞, the set {α ∈ A : d(α) = m} is in B. (2) There is a sequence {uk : k = 1,2, . . .} ⊂ α Hα of vector fields such that (2.1) For each α, {uk (α)}d(α) k=1 is an orthonormal basis of Hα , and if d(α) is finite, uk (α) = 0, for k > d(α). (2.2) For each k, each j the map α → uk (α),ej (α)α is measurable. Proof: With notation from the preceding proof, A(k) = {α ∈ A : d(α) ≥ k} =

5

(k)

A1,...,k

1,...,k

is a countable union of measurable sets, hence measurable. Thus {α ∈ A : d(α) = m} = A(m) \ A(m + 1) is measurable. (1) (1) Define u1 (α) on A = A(1) = ∪1 A1 so that on each A1 , u1 (α) is the normalization of e1 (α). Suppose that u1 (α), . . . ,uk−1 (α) are defined on A so that for each α ∈ (k−1) A1,...,k−1 , the uj (α) (1 ≤ j ≤ k − 1) are obtained from e1 (α), . . . ,ek−1 (α) by the Gram–Schmidt procedure: (u1 (α), . . . ,uk−1 (α)) = G.S.(e1 (α), . . . ,ek−1 (α))

(k−1)

(α ∈ A1,...,k−1 ).

Then, uk is defined as follows for α ∈ / A(k), uk (α) = 0 and, for α ∈ A(k) 1,...,k , uk (α) is such that: (u1 (α), . . . ,uk (α)) = G.S.(u1 (α), . . . ,uk−1 (α),ek (α)) = G.S.(e1 (α), . . . ,ek (α)). By construction, all the maps α → uk (α),ej (α)α are measurable. For each α, the span of the uk (α) is dense in Hα , thus the nonzero uk (α) form an orthonormal basis of the Hilbert space Hα . Definition 3.2.4 Let (Hα ,(ej (α))) be a measurable field of Hilbert spaces on (A,B). A vector field v = (v(α))α ∈ α∈A Hα is said to be measurable if all the functions α → v(α),ej (α)α are measurable.

3.2 Decomposition of representations

193

Observe that the set of measurable vector fields is a vector space over C. Also, if v is a measurable vector field and φ is a complex-valued B-measurable function on A, then (φv)α = (φ(α)v(α))α defines a measurable vector field. Proposition 3.2.5 Let (Hα ,(ej (α))) be a measurable vector fields of Hilbert spaces on (A,B), and (u1 (α),u2 (α), . . .) the vector fields built in Proposition 3.2.3. Then a vector field v = (v(α))α is measurable if and only if the functions α → v(α),uk (α)α are measurable. Moreover, if v and w are measurable vector fields, then α → v(α),w(α)α is measurable. Proof: By Proposition 3.2.3, the vector fields uk (α) are measurable. On  (k) A1,...,k , we have uk (α) = kj =1 xj (α)ej (α), with xj (α) measurable. Thus if v is measurable, α → v(α),uk (α)α is measurable. Conversely, suppose (k) that α → v(α),uk (α)α are measurable. Since, for each α ∈ A1,...,k , and each j ≤ k ,

ej (α),uk  (α)α uk  (α), ej (α) = k  ≤k

therefore: ej (α),v(α)α =

ej (α),uk  (α)α uk  (α),v(α)α

k  ≤k

is measurable, thus v is a measurable vector field. Similarly, if v and w are measurable, then for any α ∈ A, v(α),w(α)α =

∞

v(α),uk (α)α w(α),uk (α)α . k=1

This proves the last assertion of the proposition. Let μ be a σ -finite positive measure on the Borel measurable space (A,B). Measurable vector fields v and w are equal almost everywhere if {α : v(α) = w(α)} (which belongs to B by the preceding proposition) is of measure 0, and this is an equivalence relation. As usual, we present an equivalence class simply as a vector field, by choosing a representative. Definition 3.2.6 Let μ be a σ -finite positive measure on the Borel measurable space (A,B), and (Hα ,(ej (α))) a measurable vector field of Hilbert spaces on (A,B). The direct integral of (Hα ,(ej (α))) with respect to μ is the vector space of (equivalence classes of) measurable vector fields v = (v(α)) such that 3 v 2= v(α) 2α dμ(α) < ∞, and is denoted by

7⊕ A

A

Hα dμ(α) or simply

7⊕

Hα .

194

Unitary representations

7⊕ 7 That A Hα dμ(α) is a Hilbert space with scalar product v,w = A v(α),w(α)α dμ(α) can be proved by standard methods and is left to the reader (see also Dixmier [23]). Here are some observations. Let (A,B,μ) be a measure space, and ∞ (Hα ,(e 7 ⊕j )) a measurable field of Hilbert spaces. If φ 7∈⊕L (A,μ), then for each v ∈ A Hα dμ(α), φv = (φ(α)v(α))α belongs to A Hα dμ(α) and φv ≤ φ v . If A is countable, B is all subsets of A, and 7 ⊕μ is the counting measure Hα is just the orthogonal on A, then every vector field is measurable and 8 H . If B ⊂ A belongs to B and direct sum α∈A Hα 7of the Hilbert spaces α 7⊕ ⊕ μ(A \ B) = 0, then B Hα dμ(α) = A Hα dμ(α). Finally, if (A,B,μ) is the product measure space (A1 × A2,B1 × B2,μ1 × μ2 ), and Hα1,α2 is a measurable field of Hilbert spaces then: 3

⊕

A

3 Hα1,α2 dμ(α1,α2 ) =

⊕

A1

3

⊕ A2

Hα1,α2 dμ2 (α2 ) dμ1 (α1 ).

Example 3.2.7 Let H0 be a Hilbert space with orthonormal basis (ej )j ∈N . Let (A,B,μ) be a measure space, and for each j let (ej (α))α be the constant vector field on A defined by ej (α) = ej . Put Hα = H0 for all α ∈ A. Then (Hα ,(ej (α))) is a measurable field of Hilbert spaces. If h is a B-measurable function on A then for each j the vector field (h(α)ej (α))α∈A is measurable, and if h is square-integrable with respect 7 ⊕ to μ, then (h(α)ej (α))α∈A determines an element of the direct integral A Hα dμ(α). This direct integral is independent of the initial choice of orthonormal basis of H0 and may be denoted L2 (A,H0 ). Another realization is as a tensor product. For v ∈ L2 (A,H0 ), define Tv : H0 → L2 (A) by Tv (w) = v(α),w. Then it is easy to check that Tv ∈ L2 (A) ⊗ H0 and Tv = v . The inverse of the map v → Tv is the map defined on elementary tensors φ ⊗ w by φ ⊗ w → φ(α)w.  For each m ∈ N let Cm have the usual Hilbert space structure and for simplicity write C∞ = 2 (N). Let (A,B,μ) be a measure space. It is easily seen that v ∈ L2 (A,Cm ) if and only if each coordinate map vk : α → v(α)k is  B-measurable, in L2 (A) and k vk 2 is finite. Proposition 3.2.8 Let (A,B,μ) be a measure space, and let (Hα ,(ej (α)))α∈A be a measurable field of Hilbert spaces with d(α) = dim Hα . For each m ∈ {1,2, . . . ,∞} put7 Am = {α ∈ A : d(α) = m}. Then there is a unitary ⊕ Hα and the orthogonal direct sum: isomorphism between L2 (A∞,C∞ ) ⊕



 ∞ ⊕ L2 (Am,Cm ) .

m=1

3.2 Decomposition of representations

195

7⊕ Proof: By Proposition 3.2.3, each Am belongs to B, and Am Hα dμ(α) is 7⊕ regarded as a closed subspace of A Hα dμ(α) in the obvious 7 ⊕way. Moreover, {Am : m = 1,2, . . . ,∞} is a partition of A, the subspaces Am Hα dμ(α) are pairwise orthogonal, and   3 ⊕ 3 ⊕ 3 ⊕ ∞ Hα dμ(α)  Hα dμ(α) ⊕ ⊕ Hα dμ(α) . A

m=1 Am

A∞

7⊕

It remains to show that each Am Hα dμ(α) is isomorphic with L2 (A,Cm ). Choose a sequence {uk (α)} of vector fields as in Proposition 3.2.3, 7 ⊕ so that for each α ∈ Am , (uk (α))k is an orthonormal basis of Hα . For v ∈ Am Hα dμ(α) put   (T v)(α) = v(α),uk (α) k ∈ L2 (Am,Cm ). Again by Proposition 3.2.3, the coordinate maps α → (T v)(α)k are measurable,7 and, for each α, T v(α) 2 = v(α) 2 , so we have a linear isometry ⊕ T : Am Hα dμ(α) → L2 (A,Cm ). Its inverse is given by (T −1 w)(α) =

w(α)k uk (α), α ∈ A.

k

Example 3.2.9 Let A = R, B the collection of the Borel subsets of R, and μ the Lebesgue measure. Fix an orthonormal basis (ej ) of L2 (R); consider each ej as a constant vector field ej (α) = ej . We shall show that the direct integral L2 (R,L2 (R)) as defined in Example 3.2.7 is canonically isomorphic with L2 (R2 ). Let f : R2 → C be a Borel measurable function that is square-integrable with respect to Lebesgue measure. Then for each α ∈ R, the section fα defined by fα (t) = f (α,t) is Borel measurable, and fα is square-integrable for a.e. α. For each j , α → fα ,ej (α) is Borel. Thus f determines a measurable vector field vf defined for a.e. α by vf (α) = fα , and by7 Fubini’s Theorem, ⊕ 2 L (R) such that an element f ∈ L2 (R2 ) determines an element vf ∈ f = vf . If f (α,t) = h(α)ej (α)(t), where h is a measurable and square integrable function on R, then vf = hej . Observe that the set of finite sums of elements of the form h(α)ej (α) (h ∈ L2 (R)) is dense in L2 (R,L2 (R)). Hence dense image, which means that f → vf the map f → vf is an isometry 7 ⊕ with 2 2 2 is unitary. Thus L (R ) and R L (R) = L2 (R,L2 (R)) are identified. Now let ψ be a nonmeasurable function on R such that |ψ(α)| = 1 for all α, and for each j ∈ N let bj (α) = ψ(α)ej (α),α ∈ A. With Hα = L2 (R), as before, (Hα ,(bj (α)))α∈R is still a measurable field of Hilbert spaces.

196

Unitary representations

A vector field (v(α)) ∈ (Hα ,(bj (α)))α∈R defines a function g on R2 by g(α,t) = v(α)(t) but g is not necessarily Borel measurable on R2 . Thus (Hα ,(ej (α)))α∈R = (Hα ,(bj (α)))α∈R , and the associated direct integrals are not equal. Nevertheless, the map (uα ) → (ψ(α)uα ) induces a linear isomorphism of the corresponding direct integrals.  n m r 2 n Let7 n = m + r and identify 7 ⊕ R 2 =mR × R . We similarly identify L (R ) ⊕ 2 r with Rm L (R ), and with Rr L (R ), using constant vector fields ej . More generally, if (X,A,μ) and (Y,B,ν) are measure spaces, then L2 (X × Y,μ × ν) is isomorphic to the direct integral over Y of spaces Hy = L2 (X,μ) for each y ∈ Y.

3.2.2 Direct integral of operators and representations Fix a measurable space (A,B). Definition 3.2.10 Let (Hα ,(ej )) and (Hα ,(ek )) be two measurable fields of Hilbert spaces on (A,B). A measurable field of operators is an element T = (Tα )α∈A ∈ α L(Hα ,Hα ) such that for each measurable vector field v ∈ Hα , the map α → Tα v(α) is a measurable vector field in Hα . For a measurable field T of operators, each function α → Tα ej (α),ek (α)α is measurable. Conversely if all these functions are measurable, then for each k, α → Tα∗ ek (α) is a measurable vector field in Hα , thus for each v measurable in Hα , α → Tα v(α),ek (α)α = v(α),Tα∗ ek (α)α is measurable, thus T is measurable. As usual, we identify two measurable operator fields T = (Tα ) and S = (Sα ) if Tα = Sα almost everywhere. The operator field T thus defines a linear map (v(α))α → (Tα v(α))α from (Hα ,(ej )) to (Hα ,(ek )). 7⊕ Let7μ be a σ -finite measure on (A,B) and put H = A Hα dμ(α) and ⊕ H = A Hα dμ(α). In order to associate to the operator field T an operator in L(H,H ), we require that T be essentially bounded in the sense that ess sup Tα

α

< ∞.

With 7 ⊕ this condition it is clear that the field T defines a bounded linear map T from H to H such that 7⊕ T ≤ ess sup Tα α . We shall also denote

7⊕

Tα or even

7⊕

α

Tα dμ(α) for the operator

7⊕

T.

3.2 Decomposition of representations

197

It is easy to prove that sum and composition of direct integral operators are direct integral operators: 7⊕ 7⊕ 7⊕ 7⊕ 7⊕ 7⊕ T + S= (T + S), T S= (T S). Defining T ∗ by T ∗ = (Tα∗ ), we also have 7⊕ 7⊕ ∗ ( T )∗ = T . In particular, if φ is an essentially bounded function on A, then the multiplication operator Mφ is the direct integral operator in L(H,H) associated to the measurable field of operators (φ(α)1Hα )α , and (Mφ )∗ = Mφ . Mφ is also 7⊕ T is as expected. called a diagonal operator. The norm of Proposition 3.2.11 Let T = (Tα ) be an essentially bounded measurable field of operators as above. Then the norm of the corresponding direct integral operator is 7⊕ T = ess sup Tα α . α

Proof: We need only prove the inequality 7⊕ T ≥ ess sup( Tα α

α ).

7⊕ Hα , and any bounded measurable function φ on A, we Given any v in compute: 3 7⊕ 2 7⊕ 2 T |φ(α)|2 v(α) 2α dμ(α) = T φv 2 A 7  ⊕ ≥ T φv 2 3 = |φ(α)|2 Tα v(α) 2α dμ(α). A

Since this holds for any bounded measurable φ, there is Nv ∈ B, such that / Nv , μ(Nv ) = 0 and for all α ∈ 7⊕ Tα v(α) α ≤ T v(α) α . Now fix a system of vector fields {uk } as in Proposition 3.2.3. Then μ being σ -finite, A is the union of an increasing sequence of sets Aj with finite measure. Consider the family V of vector fields v ∈ Hα that are finite sums of the form

v(α) = zk χAj (α)uk (α) k

198

Unitary representations

where χAj is the characteristic function of Aj and where zk ∈ Q + iQ. Then 7⊕ V is countable, each v in V belongs to Hα , and for each α ∈ A, the set {v(α) : v ∈ V } is dense in Hα . / N, v ∈ V , Now N = ∪v∈V Nv has measure 0 and for α ∈ Tα v(α)

α

≤

7⊕

T

v(α)

α.

Hence Tα ≤

7⊕

T

holds for α ∈ / N , and the proposition follows. The following is immediate.

7⊕ Corollary 3.2.12 Let T and S be measurable fields of operators on Hα . 7⊕ 7⊕ (1) 7 Tα = Sα if and only if Tα = Sα almost everywhere. ⊕ T is unitary if and only if Tα is unitary for a.e. α. (2) 7⊕ 7⊕ We assume that H = A Hα dμ(α) and H = A Hα dμ(α) are direct integrals on the same measure space (A,B,μ). An operator T ∈ L(H,H ) is said to be decomposable if T is the direct integral operator associated to some essentially bounded field of operators 7 ⊕ (Tα ) 7on⊕ (A,B,μ). Let T be an T = Tα the associated direct essentially bounded field of operators and integral operator in L(H,H ). Let φ be an essentially bounded function on A and let Mφ and Mφ be the diagonal operators in L(H) and L(H ), respectively. Clearly Mφ

7⊕

T =

7⊕

T Mφ .

The following is a fundamental result. 7⊕  7⊕ Hα and H = Hα be direct integrals of Theorem 3.2.13 Let H = Hilbert spaces on the measure space (A,B,μ) and let S ∈ L(H,H ). Then S is a direct integral operator if and only if Mφ S = SMφ holds for all essentially bounded measurable functions φ on A. Proof: We need only to prove that if Mφ S = SMφ for each φ, then S is 7⊕ Hα dμ(α) defined in the decomposable. Pick the countable subset V in proof of Proposition 3.2.11: V is the set of v = (v(α))α , with v(α) =     k zk χAj (α)uk (α), put v = (v (α))α = ( k zk χAj (α)(Suk )(α))α . Since Mχ A S = SMχAj , we have Sv = v  . j

3.2 Decomposition of representations

Now for any essentially bounded measurable φ: 3 Mφ Sv 2 = |φ(α)|2 v  (α) 2 dμ(α) = SMφ v 3 ≤ S 2 |φ(α)|2 v(α) 2 dμ(α).

199

2

Fix v, since this inequality holds for any φ, there is a null subset Nv in A such that, if α ∈ / Nv , v  (α) ≤ S

v(α) .

Since V is countable, there is a null set N such that this is holding for any v, and any α ∈ / N. By density of the set of v(α) in Hα , this proves the map v(α) → v  (α) can be extended into a linear bounded operator Tα : Hα → Hα such that Tα ≤ S . We put Tα = 0 if α ∈ N , by construction the field T = (Tα )α is measurable and for each v ∈ V, φ ∈ L∞ (A), 7⊕ SMφ v = Mφ Sv = φv  = (φ(α)v  (α))α = (φ(α)Tα v(α))α = T Mφ v. 7⊕ Suppose that w = (w(α)) ∈ Hα is orthogonal to every vector Mφ v (φ ∈ L∞ (A), v ∈ V ). Then for each j and k, 3 3 # # #w(α),uk (α)α # dμ(α) = w(α),uk (α)α sign(w(α),uk (α)α ) dμ(α) Aj

Aj

=0 (we define here sign(0) = 1). This implies w(α),uk (α)α = 0 almost every∞ where, 7 ⊕w = 0. This means {Mφ v : φ ∈ L (A), v ∈ V } is dense, therefore T. S= Example 3.2.14 Recall the Heisenberg group G defined in Example 1.6.4. The left regular representation is realized on R3 as   L(x,y,z)f (t,u,v) = f (t − x,u − y,v − z + xy − xu)

(f ∈ H = L2 (R3 )).

Denote by ∧u,v the partial Fourier transform of f with respect to its second and third variables; a direct computation gives     ∧u,v L(x,y,z)f (t,γ ,λ) = ∧u,v f (t − x,γ + λx,λ)eiγ y eiλz .

200

Unitary representations

Thus the regular representation L is unitarily equivalent to the representation S acting on L2 (R3 ) by   S(x,y,z)f (t,γ ,λ) = f (t − x,γ + λx,λ)eiγ y eiλz (f ∈ L2 (R3 )). Now define the unitary operator V on L2 (R3 ) by (Vf )(t,γ ,λ) = f (t,γ − tλ,λ). The representation T (x,y,z) = V S(x,y,z)V −1 satisfies   T (x,y,z)f (t,γ ,λ) = f (t − x,γ ,λ)eiy(γ −λt) eiλz (f ∈ L2 (R3 )). 7⊕ 2 L (R)γ ,λ dγ7dλ as warned after Identify L2 (R3 ) = L2 (R × R2 ) with ⊕ π(x,y,z) where Example 3.2.9. With this identification T (x,y,z) = π(x,y,z) is the field of operators (πγ ,λ (x,y,z))γ ,λ and   πγ ,λ (x,y,z)fγ ,λ (t) = fγ ,λ (t − x)eiy(γ −λt) eiλz . This means that each L(x,y,z) is unitarily equivalent to a decomposable operator: 3 ⊕ −1 −1 V ∧u,v L(x,y,z) ∧u,v V = πγ ,λ (x,y,z) dγ dλ. It is easy to check that for each (γ ,λ), (x,y,z) → πγ ,λ (x,y,z) is a unitary representation of G. In fact, π0,λ is the irreducible representation πλ of G defined in Example 3.1.8.  The preceding is an example of a direct integral of representations. Let G be a locally compact group, (A,B,μ) a σ -finite measure space, and (Hα ,ej (α))α a measurable field of Hilbert spaces. Suppose that for each α ∈ A, πα is a unitary representation of G acting in Hα . Definition 3.2.15 If for each x ∈ G, (πα (x))α is a measurable field of operators, then (πα )α∈A 7 ⊕ is called a measurable field of representations of G. πα (x) is a unitary representation of G The map π : x → 7 ⊕and is called ; π is written π = the direct integral of the representations π α A πα dμ(α) or 7⊕ πα . simply as The strong continuity of a direct integral of representations is an easy consequence of the Lebesgue convergence theorem. Example 3.2.16 Returning to Example 3.2.14, we see that the regular representation 7of the Heisenberg group is equivalent with the direct integral ⊕ representation R2 πγ ,λ . We now show that this direct integral is an irreducible

3.2 Decomposition of representations

201

decomposition of the regular representation, meaning that πγ ,λ is almost everywhere irreducible. Fix λ = 0. For each γ , define the unitary operator Vγ : L2 (R) → L2 (R) by  γ (Vγ f )(t) = f t + . λ Then:  γ  iyγ iλz (Vγ πγ ,λ (x,y,z)Vγ−1 f )(t) = (Vγ−1 f ) t − x + e e λ = (π0,λ (x,y,z)f )(t). Thus, for each γ ∈ R, πγ ,λ is equivalent to the representation π0,λ . Now, in Example 3.1.8, we saw that π0,λ , denoted πλ , is irreducible. Therefore, almost  every πγ ,λ is irreducible. Example 3.2.17 Let (ρ0,H0 ) be a unitary representation, let (A,B,μ) be a measure space, and let H = L2 (A,H0 ). Define (π,H) by (π(x)f )(α) 7⊕ = ρ0 (x)f (α), α ∈ A. With the identification of Example 3.2.7, π = A πα where πα = ρ0 for all α. With the identification of L2 (A,H0 ) with L2 (A)⊗H0 , we may write π(x) = 1⊗ρ0 (x), that is π = 1⊗ρ0 . Suppose that A is countable with m = |A| ∈ {1,2,3, . . . ,∞}, B is all subsets of A, and μ is the counting measure on A. Then π = ⊕ρ0 is also written as m · ρ0 . In this situation, π is  said to be a multiple of ρ0 . Example 3.2.18 We return to the quasi-regular representation of Example 3.1.9. Recall V is a finite-dimensional vector space over R, α is a representation of a locally compact group H in V , and G = V α H . The quasiregular representation τ of G acts in L2 (V ), and its Fourier transform acts in L2 (V ∗ ) by (τˆ (v,h)F () = χ (v) F (α ∗ (h)−1 )| det α ∗ (h)|−1/2 . Now assume that H is a simply connected solvable Lie group, and that α is of exponential type. Choose a good Jordan–Hölder sequence for the complexification hc of the Lie algebra h of H , and let (fi ) be a good flag basis for the dual V ∗ of V , with corresponding weight sequence (λi ). Perform the layering and parametrization procedure as described in Chapter 2, Sections 2.5 and 2.6, for the dual action of H , and let  = e,h,δ be the minimal ultrafine layer with cross-section . Recall that since  is Zariski open, then the complement of  has Lebesgue measure zero in V ∗ . Let ν be the measure on  and for each σ ∈ , ωσ the measure on the H -orbit Oσ , such that dωσ (α ∗ (h0 )) = | det(α ∗ (h0 ))|dωσ (). Thus we can write (see Corollary 2.6.2)

202

Unitary representations 3 3

3 V∗

F ()d = 

Oσ

F ()dωσ ()dν (σ ).

Let d be the dimension of orbits in , and let Z be the parameter set for orbits in . Given ε ∈ {±1}d , write εz = (ε1 z1, . . . ,εd zd ). Given a covering set O for , we have the parametrization map P O : Z ×  ∩ O →  (Proposition 2.5.8). If σ belongs to the covering set O, then for  ∈ Oσ , define zO () ∈ Z by P O (zO (),σ ) = . Thus, zO (P O (z,σ )) = z. Recall that if σ ∈ O ∩ O  , then there is ε ∈ {±1}d  such that zO () = εzO (). Moreover, the element s() = g O (zO (),σ ) ∈ H is independent of the covering set O. Let ec () = i ∈e / i (s()). With this notation, for σ ∈  ∩ O, s(P O (z,σ )) = g O (z,σ ) and ec (P (z,σ )) = O i ∈e / i (g (z,σ )), so 4 |i (g O (z,σ ))|dz = ec (P (z,σ ))dz. dωσ () = i ∈e /

Let {ej } be the sequence in L2 (Z,dz) defined by ej (z) = z11 z22 · · · zdd e−|z| (j = (j1, . . . ,jd ) ∈ Nd ). j

j

2

j

For each covering set O and each multi-index j , define the vector field (ejO (σ )) ∈ σ ∈ L2 (Oσ ,ωσ ) by ejO (σ )() For each σ ∈ ,

=

2 ej (zO ())|ec ()|−1/2, 0,

if σ ∈ O, otherwise.

3

3 Oσ

|ejO (σ )|2 dωσ =

Z

|ej (z)|2 dz < ∞

so ejO (σ ) ∈ L2 (Oσ ,ωσ ). Also it is clear that σ →

 ejO (σ ),ekO (σ )

27 =

Z ej (εz)ek (ε

 z)dz,

0,

if σ ∈ O ∩ O , otherwise

is Borel measurable on . We say that a vector field (v(σ ))σ ∈ σ ∈ L2 (Oσ ,ωσ ) is measurable if for each covering set O and each multi-index j , σ → v(σ ),ejO (σ )L2 (Oσ ,ωσ )

3.2 Decomposition of representations

203

is Borel measurable on . Let F ∈ Cc (V ∗ ); then for each covering set O, and each multi-index j , the Lebesgue Convergence Theorem implies that 3 4 F (P O (z,σ ))ej (z) |i (g O (z,σ ))|dz σ → Z

i ∈e /

is continuous, hence Borel measurable. (To apply the Lebesgue Theorem, observe that for each covering set O, the projection to Z of the support of F (P O (z,σ )) is a compact subset of Z.) Thus F defines naturally a measurable field (vF (σ ))σ in σ ∈ L2 (Oσ ,ωσ ). Moreover, 3 3 3 |F ()|2 d = |F ()|2 dωσ ()dν (σ ) = vF 2 V∗



7⊕

Oσ

so F ∈  L2 (Oσ ,ωσ ) dν (σ ) and the map F → vF extends to an isometry U of L2 (V ∗ ) into the direct integral. To see that U is onto, for each σ ∈  ∩ O, 2 apply Gram–Schmidt to produce an orthonormal basis uO j of L (Oσ ,ωσ ), 7 ⊕ starting with the sequence of vectors (ejO (σ ))j . Let v ∈  L2 (Oσ ,ωσ ) dν (σ ). For σ ∈  ∩ O write

ajO (σ )uO v(σ ) = j (σ ) j

where ajO (σ ) ∈ C. Then σ → ajO (σ ) is Borel measurable and determines an element of L2 (,ν). Put

ajO (σ )uO vN (σ ) = j (σ ). |j |≤N

7⊕

Then vN belongs to  L2 (Oσ dν (σ ), and for σ ∈  ∩ O, vN (σ ) − v(σ ) 2 → 0 and vN (σ ) − v(σ ) 72 ≤ v(σ ) 2 . Hence, by the Lebesgue ⊕ Convergence Theorem, vN → v in  L2 (Oσ ,ωσ ) dν (σ ). Thus, the linear O 2 span of all vector fields of the form guO j = (g(σ )uj (σ ))σ , where g ∈ L (,ν) 7⊕ 2 is dense in7  L (Oσ ,ωσ ) dν (σ ). Given g ∈ L2 (,ν), choose g0 ∈ Cc () such that  |g − g0 |2 dν <  and then the vector field (g0 (σ )uj (σ ))σ satisfies g0 uj − guj 2 <  also. Now define 2 if σ () ∈  ∩ O, g0 (σ ())uO j (σ ())(), F () = 0, otherwise. F is continuous on (O) and zero on (O)c ; hence, F is Borel measurable, and U F = g0 uj . It follows that the linear span of all g0 uj is in the range of U . Since this linear span is dense, then U is surjective.

204

Unitary representations

For each σ ∈ ,(v,h) ∈ G, define the operator τˆσ (v,h) in the Hilbert space L2 (Oσ ,ωσ ) by τˆσ (v,h)f () = χ (v)f (α ∗ (h)−1 ())| det α ∗ (h)|−1/2 . Since dωσ (α ∗ (h)) = | det α ∗ (h)|dωσ () holds for all σ , then τˆσ (v,h) is unitary, and it is easily seen that (v,h) → τˆσ (v,h) is7a unitary representation of G. ⊕ Moreover, for each (v,h) ∈ G, U τˆ (v,h)U −1 =  τˆσ (v,h) dν (σ ). Thus, 3 ⊕  τˆ  τˆσ dν (σ ). 

7⊕ Let (A,B,μ) be a measure space and let H = A Hα dμ(α) be the direct integral of a measurable field of Hilbert spaces on A. Let (π,H) be a direct 7⊕ integral π = A πα dμ(α) of unitary representations of a locally compact group G. Consider the algebra D = D(H) of all diagonal operators Mφ in L(H). Note that 7⊕

D ⊆ C(π ).

πα dμ(α) is an irreducible decomposition, that Proposition 3.2.19 If π = is, almost every πα is irreducible, then D is a maximal commutative subalgebra of C(π ). Proof: Suppose S is a bounded operator in C(π ), commuting with D, then 7 ⊕ by the von Neuman Theorem 3.2.13, S is a direct integral operator S = Tα . Pick a countable dense subset {si } in G. For each i, there is a Borel set Ni with measure 0 such that Tα πα (si ) = πα (si )Tα , / N , Tα is in C(πα ). Hence, for α ∈ / Ni . Put N = ∪Ni . By density, for each α ∈ Tα = ϕ(α) 1Hα holds for a.e. α. This means S = Mϕ ∈ D, and D is maximal among the abelian subalgebras of C(π ). Remark 3.2.20 Observe that if π is the representation of Proposition 3.2.19, the center of C(π ) is a subalgebra of D. It can be strictly included in D as the following example shows: let π = 2ρ = ρ ⊕ ρ, where ρ is an irreducible representation of G. Then D = {M(λ1,λ2 ) = λ1 1Hρ ⊕ λ2 1Hρ : (λ1,λ2 ) ∈ C2 }. However, C(π ) contains the operator S defined by S(v1 ⊕ v2 ) = v2 ⊕ v1 and M(λ1,λ2 ) S = SM(λ1,λ2 ) implies λ1 = λ2 . It follows that the center of C(ρ ⊕ ρ) is {M(λ,λ) : λ ∈ C} = C1Hρ⊕ρ .

3.2 Decomposition of representations

205

Example 3.2.21 In this example, we consider a simply connected solvable Lie group G called the Mautner group. It is a semidirect product C2  R = {(z,w,x) : z,w ∈ C, x ∈ R}, with multiplication: (z,w,x)(z,w ,x  ) = (z + eix z,w + eiαx w ,x + x  ), where α is an irrational real number. The method of Example 3.2.14 gives a decomposition of its left regular representation L. Here L is defined by   L(z,w,x)f (u,v,t) = f ((u − z)e−ix ,(v − w)e−iαx ,t − x). Let ∧u,v denote the real partial Fourier transform on L2 (G) with respect to the complex variables u and v, so that for g ∈ Cc (G), 3 ∧u,v g(ξ,η,t) = g(u,v,t)eiRe(uξ +vη) dudv. C2

Then a straightforward calculation shows that     ∧u,v L(z,w,x)f (ξ,η,t) = ∧u,v f (ξ e−ix ,ηe−iαx ,t − x)eiRe(zξ +wη) . Now define the unitary operator V on L2 (C2 × R) by Vf (ξ,η,t) = f (ξ eit ,ηeiαt ,t). Then we compute for each (z,w,x) ∈ G, −1 f )(ξ,η,t) = f (ξ,η,t − x)eiRe(ze (V ∧u,v L(z,w,x) ∧−1 u,v V

−it ξ +we−iαt η)

.

From this computation, it can be easily proved that 7 ⊕the left regular representation is unitarily equivalent to the direct integral C2 Tξ,η dξ dη where for each (ξ,η) ∈ C2 , (Tξ,η,L2 (R)) is defined by   −it −iαt Tξ,η (z,w,x)f (t) = f (t − x)eiRe(ze ξ +we η) . However, in this example, we can also decompose the left regular representation in a totally different way. Indeed, starting from ∧u,v L(z,w,x)∧−1 u,v , we now perform the Fourier transform with respect to the third (real) variable, and we get the representation:    T (z,w,x)f (ξ,η,τ ) = f (ξ e−ix ,ηe−iαx ,τ )eixτ +iRe(zξ +wη) . Writing the complex numbers ξ , η as ξ = ρy, with ρ = |ξ |, η = σ s, with σ = |η|, we can identify a function f (ξ,η,τ ) ∈ L2 (C2 × R) with the field (fρ,σ,τ (y,s) = f (ρx,σy,τ ))ρ,σ,τ , that is we can decompose L2 (C2 × R) into: 3 ⊕ 2 2 L (C × R)  Hρ,σ,τ ρdρ σ dσ dτ, with Hρ,σ,τ  L2 (T2 ). ρ>0,σ >0,τ

206

Unitary representations

7⊕  Tρ,σ,τ , In this decomposition, the representation T  is decomposed into with:    Tρ,σ,τ (z,w,x)f (y,s) = f (ye−ix ,se−iαx )eixτ +iRe(zρy+wσ s) . At the end of the chapter, we will see that all the representations Tξ,η (ξ = 0,  (ρ > 0, σ > 0) are irreducible and for each such (ξ,η), and η = 0) and Tρ,σ,τ  are inequivalent. (ρ,σ,τ ), the representations Tξ,η and Tρ,σ,τ Thus, the left regular representation L is decomposable over two disjoint sets of equivalence classes of irreducible representations of G. This is possible because the Mautner group G is not a type 1 group. 

3.2.3 Type 1 groups The class of type 1 groups contains many families of groups including compact groups, abelian groups, simply connected nilpotent Lie groups and connected semisimple Lie groups. However, not every connected solvable group is type 1. To define this category of groups, we first consider the notion of factor representation. Definition 3.2.22 Let G be locally compact group. A unitary representation π of G is called a factor if the center of the commuting algebra C(π ) is C1. Observe that every irreducible unitary representation is a factor. The argument in Remark 3.2.20 shows that if ρ is irreducible, then ρ ⊕ρ is a factor. Though ρ and ρ ⊕ ρ are not equivalent, it is easily seen that they are quasiequivalent. Indeed, a natural equivalence relation for factor representations is that of quasi-equivalence: Definition 3.2.23 Unitary representations π and ρ are said to be quasiequivalent if (a) for each subrepresentation π  of π , C(π ,ρ) = 0, and (b) for each subrepresentation ρ  of ρ, C(π,ρ  ) = 0. In this case we write π ∼ ρ. Quasi-equivalence is an equivalence relation, and ˇ of G is the set of all quasi-equivalence classes [π ]q of factor the quasi-dual G representations π of G. Observe that for unitarily representations π and ρ, if π and ρ are equivalent, then they are quasi-equivalent, and if π and ρ are both irreducible and quasiequivalent, then they are equivalent. These observations mean that there is a ˆ →G ˇ given by [π] → [π ]q . canonical injective map G

3.2 Decomposition of representations

207

Proposition 3.2.24 Every multiple of an irreducible representation is a factor representation, and every multiple of a unitary representation σ is quasiequivalent to σ . Proof: Let (σ,H0 ) be an irreducible unitary representation and let m ∈ N ∪ {∞}. The Hilbert space for m σ is H = Cm ⊗ H0 . Let Hi be the closed subspace of H defined by Hi = δi ⊗ H0 where δi is the i-th canonical basis element of Cm . The orthogonal projection Pi onto Hi belongs to C(π ). Let T belong to the center of C(π ); then T commutes with Pi , so T leaves Hi invariant. Moreover the restriction Ti of T to Hi commutes with the irreducible representation σ . Hence for each i there is λi ∈ C such that Ti = λi 1Hi . Now let Qij be the operator permuting the terms Hi and Hj in the direct sum. Clearly Qij belongs to C(π ), so T commutes with Qij . But now it follows that λi = λj for all i and j , that is, that T is scalar. Thus m σ is a factor representation. Suppose now (σ,H0 ) is any unitary representation. We keep the preceding notation. For each i, the map fi : H0 → H, fi (v) = δi ⊗ v is injective and intertwines σ and m σ . If (σ ,H0 ) is a subrepresentation of σ , then fi |H0 is an element of C(σ ,m σ ). On the other hand, let (π ,H ) be a subrepresentation of m σ . Since H = 0, there is i such that Pi (H ) = 0, observe that fi∗ : H → H0 , is such that fi∗ (v) = v0 if Pi (v) = δi ⊗ v0 , thus fi∗ is not vanishing, it is in C(π ,σ ). This proves that σ ∼ m σ . Definition 3.2.25 A factor representation is said to be type 1 if it is a multiple of an irreducible representation. Type 1 factors are often defined or presented as direct integrals. Lemma 3.2.26 Let (A,B,μ) be a measure space, 7 ⊕ let (ρα ,Hα )α∈A be a ρα . Assume that there measurable field of representations, and put ρ = is a representation (ρ0,H0 ) and a measurable field of unitary operators U = (Uα )α , such that for μ-a.e. α, Uα ∈ C(ρα ,ρ0 ). Then ρ is equivalent with a multiple of ρ0 . 7⊕ 7⊕ 7⊕ Proof: The operator U is unitary and by definition, ( U )ρ( U )−1 = 1 ⊗ ρ0 acting in L2 (A,μ) ⊗ H0 . Let m = dim L2 (A,μ); we show that ρ is equivalent with m ρ0 . 2 2 m Fix an orthonormal basis {gj }m j =1 of L (A,μ), by which L (A,μ)  C via T : f → (f ,gj )j . We then have the isomorphism T˜ : L2 (A,μ) ⊗ H0 → Cm ⊗ H0 , and thereby T˜ (1 ⊗ ρ0 )T˜ −1 = m ρ0 .

208

Unitary representations

The following is immediate. Corollary 3.2.27 7Let π and σ be unitary representations with σ irreducible. ⊕ πα and that πα  σ holds for a.e. α. Then π is a type 1 Assume that π  factor representation. We shall need to now prove a converse of Lemma 3.2.26. First, we prove a lemma that we shall use here and also in Chapter 5. Lemma 3.2.28 Let m π0 a multiple of an irreducible representation (π0,H0 ) with m ∈ {1,2, . . . ,∞}, realized in Cm ⊗ Hπ0 then C(m π0 ) = L(Cm ) ⊗ 1Hπ0 . Proof: Recall first the basis (δi ) of Cm and the orthogonal projection Pi on the space δi ⊗ H0 . Fix T ∈ C(m π0 ). For any w ∈ Cm , and any i, we define the operator Ti,w ∈ L(H0 ) by: Pi T (w ⊗ v) = δi ⊗ Ti,w v. Since Pi and T are commuting with π, then Ti,w is commuting with π0 , this means Ti,w = ti,w 1H0 . Let us put:

Sw = ti,w δi . i

Since

|tiw |2 v

2

=

i

tiw v

2

i

=

Pi T (w ⊗ v)

2

= T (w ⊗ v)

2

≤ T

2

w

2

v 2,

i

Sw is a well-defined vector in Cm and Sw ≤ T w . By definition S is linear, therefore S ∈ L(Cm ), and T = S ⊗ 1H0 . Hence C(m π0 ) ⊂ L(Cm ) ⊗ 1H0 . The reverse inclusion is evident. Proposition 3.2.29 Let (π0,H0 ) be an irreducible unitary representation of G, let (A,B,μ) be a measure space, and let ((πα ,Hα ))α∈A be a measurable field 7 ⊕ of irreducible representations of G. Let (π,H) be defined by π = A πα dμ(α), and assume that for some m ∈ {1,2, . . . ,∞}, π is equivalent with the representation m π0 . Then m is the dimension of L2 (A,μ) and, for almost every α, πα is unitarily equivalent to π0 . Proof: The strategy of the proof is to show that H  L2 (A,ν) ⊗ H0 where ν is equivalent with μ, and then to apply Theorem 3.2.13 to obtain unitary intertwining operators Uα for a.e. α ∈ A.

3.2 Decomposition of representations

209

Write m π0 = 1Cm ⊗ π0 , acting in Cm ⊗ H0 . As such, by Lemma 3.2.28, C(m π0 ) = L(Cm ) ⊗ 1H0 . Let U be a unitary operator in7C(π,m π0 ). Each ⊕ ϕ ∈ L∞ (A,μ) determines a diagonal operator Mϕ on H = A Hα dμ(α) and U Mϕ U −1 = Qϕ ⊗ 1H0 for some Qϕ ∈ L(Cm ). Observe that Qϕ + Qψ = Qϕ+ψ , Qϕ Qψ = Qϕψ and Qϕ = Mϕ = ess sup|ϕ|. Put A = {Qϕ : ϕ ∈ L∞ (A,μ)}. The desired measure ν will be of the form E → QχE v,v, where v is a suitable vector in Cm . We claim first that A is a maximal commutative subalgebra of L(Cm ). Indeed, let S be an operator commuting with A, and put T = U −1 (S ⊗ 1)U . Then T ∈ C(π ), and T commutes with the algebra D of diagonal operators on H. By Proposition 3.2.19, we have T ∈ D, so T = Mϕ for some ϕ ∈ L∞ (A,μ) and hence S = Qϕ belongs to A, proving the claim. From now on, if E is a Borel subset of A, we denote QE the operator QχE , where χE is the characteristic function of E. We show next that there is v ∈ Cm such that, for each E ∈ B, QE v = 0 implies QE = 0. Let (e1,e2, . . .) be an orthonormal basis of Cm . Put v1 = e1 , and put K1 = span{QE v1 : E ∈ B}. If K1 = Cm , then for E ∈ B, QE v1 = 0 implies QE QF v1 = QF QE v1 = 0 for all F ∈ B, and it follows from linearity and continuity of QE that QE = 0. / K1 } and put Else, let k2 = min{k ∈ N : ek ∈ v2 =

1 projK⊥ (ek2 ). 1 2

Define K2 as the closure of span{QE v2 : E ∈ B} and check that K2 ⊂ (K1 )⊥ : QE v1,QF v2  = QE QF v1,v2  = QE∩F v1,v2  = 0. If K1 + K2 = Cm , put v = v1 + v2 . If QE v = 0, then QE v1 = QE v2 = 0, and as above, for any Borel subsets Fi , QE v is orthogonal to both QF1 v1 and QF2 v2 , which then implies QE = 0. Continue in this way: we obtain either (a) for some r ∈ N, orthogonal elements v1, . . . ,vr in Cm , and pairwise orthogonal closed subspaces Ki so that Cm = ⊕ri=1 Ki and span{QE vi : E ∈ B} is dense in Ki , or (b) a sequence 1 = k1 < k2 < · · · of positive integers, and sequence v1,v2, . . . in Cm , and a sequence Ki , i ∈ N of pairwise orthogonal closed subspaces, so that vi ≤ 1/2i , span{QE vi : E ∈ B} is dense in Ki , and  for each k ∈ N, and i ∈ N, k < ki+1 implies ek ∈ ij =1 Ki .

210

Unitary representations

Observe that in either case the subspaces Ki are invariant under all operators QE ,E ∈ B. If (a) holds, then put v = v1 + · · · + vr , and argue as above that for E ∈ B, QE v = 0 implies QE vanishes on each Ki , and hence QE = 0. ∞ Suppose that (b) holds. Then Cm = ⊕∞ i=1 vi . Then i=1 Ki . Put v ∈ QE v = 0 implies QE vi = 0 for all i, and again using the commutativity of A we get QE = 0. Next we show that the element v is a cyclic element for A: span{QE v : E ∈ B} is dense in Cm . Consider the orthogonal projection P onto the closure of span{QE v : E ∈ B}. For each w ∈ H, write w = P w + (1 − P )w. Then for all E, F ∈ B, QE v,QF (1 − P )w = QE∩F v,(1 − P )w = 0 thus P QF (1 − P )w = 0, and since P w is in the space generated by the QE v and QF QE v = QF ∩E v is also in this space, P QF w = P QF P w = QF P w. Now P commutes with each QF = QχF , by linearity it commutes with all Qϕ , ϕ a simple function, by continuity, it commutes with all Qϕ , ϕ ∈ L∞ (A), and U −1 (P ⊗ 1)U commutes with D. Since P ⊗ 1 ∈ C(m π0 ), then U −1 (P ⊗ 1)U belongs to C(π ). Hence by Proposition 3.2.19, U −1 (P ⊗ 1)U belongs to D. Since U −1 (P ⊗ 1)U is a projection, then U −1 (P ⊗ 1)U = MχB for some B ∈ B. Thus P = QB . Now if μ(E ∩ B) = 0, then QB QE = 0, so 0 = QB QE v = QE v, and hence QE = 0, which means μ(E) = 0. Therefore, QB = QA = 1Cm , and we conclude that v is cyclic: Cm = span{QE v : E ∈ B}. Define the positive Borel measure ν for E ∈ B by ν(E) = QE v,v. For each w = w

n

i=1 xi QEi v

2

=

9

xi QEi v,

i

=

∈ H, we have:

: xj QEj v

j

xi xj QEj QEi v,v =

i,j

xi xj ν(Ej ∩ Ei )

i,j

#2 ; ;2 3 # # ; # ; # ; # xi χEi (α)# dν(α) = ; xi χEi ; = # ; A

i

i

L2 (A,ν)

.

3.2 Decomposition of representations n

211

= ϕ 2L2 (A,ν) . ˜ ϕ v from Extend the map ϕ → Qϕ v by continuity to an isometry V : ϕ → Q L2 (A,ν) to Cm . Since the image is dense, V is unitary. Observe that ν is equivalent to μ: if ν(E) = 0, then 0 = QE v,v = QE v 2 thus QE v = 0 and QE = 0, in other words, μ(E) = 0. Conversely if μ(E) = 0, QE = 0, and ν(E) = 0. Hence L2 (A,ν) and L2 (A,μ) are isomorphic via the multiplication M by the square root of the Radon-Nikodym derivative of μ with respect to ν, so L2 (A,ν) ⊗ H0 and L27(A,μ) ⊗ H0 are ⊕ isomorphic also. Thus we have an isomorphism W : H = A Hα dμ(α) → 7 ⊕ H0 dμ given by the composition L2 (A,μ) ⊗ H0 = So for any simple function ϕ =

U

i=1 xi χEi

we have Qϕ v

V −1 ⊗1

2

M⊗1

H −−−→ Cm ⊗ H0 −−−→ L2 (A,ν) ⊗ H0 −−−→ L2 (A,μ) ⊗ H0 . Observe that since U π U −1 = 1Cm ⊗ π0 , then W πW −1 = 1L2 (A,μ) ⊗ π0 . Let φ ∈ L∞ (A,μ) = L∞ (A,ν) and denote the various multiplication L2 (A,ν)

L2 (A,μ)

operators by φ by Mφ ,Mφ , and so on. For each simple function ϕ we have  L2 (A,ν)  V Mφ ϕ = V (φϕ) = Qφϕ v = Qφ (Qϕ v) = Qφ V (ϕ). L2 (A,ν)

L2 (A,μ)

L2 (A,ν)⊗H

L2 (A,μ)⊗H

0 0 = Mφ M and (M ⊗ 1)Mφ = Mφ Since MMφ (M ⊗ 1), then we find 7⊕ 7⊕ Hα dμ H0 dμ L2 (A,μ)⊗H0 W Mφ = W MφH = Mφ W = Mφ W. 7⊕ By Theorem 3.2.13, W is a direct integral operator A Wα dμ(α), 7 ⊕and we πα and have already observed that W intertwines the representations π = 7⊕ π0 = 1 ⊗ π0 . Using a countable dense subset {si } in G as in the proof of Proposition 3.2.19, we prove that for almost every α,

Wα πα (s) = π0 (s)Wα for each s ∈ G. Since Wα is almost everywhere unitary, that means that almost every representation πα is unitary equivalent to π0 . A type 1 factor π is equivalent to some m π0 with π0 irreducible. It follows from Proposition 3.2.29 that m is unique and π0 is unique up to unitary equivalence. Definition 3.2.30 A locally compact group G is a type 1 group if each factor representation of G is type 1. ˆ →G ˇ is a bijection. If G is type 1, then the natural injection G

212

Unitary representations

7⊕ πγ ,λ of the regular Example 3.2.31 Recall the irreducible decomposition representation L of the Heisenberg group given in Example 3.2.16. Now define 3 ⊕ ρλ := πγ ,λ dγ ; R

by Corollary 3.2.27, ρλ is a type 1 factor. It follows that: 3 ⊕ ρλ dλ. L R

We shall say that the left regular representation L of the Heisenberg group can be decomposed into factor representations.  The preceding example can be generalized to any locally compact group G. ˇ of G. In order to view G ˇ as a measurable set, we now Recall the quasi-dual G ˇ describe a natural σ -algebra of subsets of G, usually called the Mackey–Borel ˇ structure of G. Since each representation π is quasi-equivalent to ∞ π , then each quasiequivalent class [π ] of a factor representation of G has a representative π acting on C∞ . Denote by Fac∞ (G) the set of factor representations of G acting in C∞ . Now consider the smallest σ -algebra B  of subsets of Fac∞ (G) such that, all the mappings: π → π(s)v,w

(s ∈ G, v, w ∈ C∞ )

ˇ π → are Borel measurable. Since the natural mapping f : Fac∞ (G) → G, ˇ [π] is onto, then we define the σ -algebra B of subsets of G by E ∈ B if and only if f −1 (E) ∈ B  . This σ -algebra B is often called the Mackey–Borel ˇ structure of G. It is possible to use the Mackey–Borel structure to decompose any unitary representation π of G into factor representations: this is called the central decomposition of π , and its proof is originally due to J. Ernest [37]. Theorem 3.2.32 Let G be a locally compact group and let (π,H) be a ˇ a measurable unitary representation of G. Then there is a measure μ on G, field of representations field (Hλ,(ej )) of Hilbert spaces, and a measurable 7 ˇ and a unitary map U : H → ⊕ Hλ dμ(λ), such that ρλ ∈ λ (ρλ,Hλ ) over G, ˇ G for μ-a.e. λ, and 3 ⊕ U π U −1 = ρλ dμ(λ). 7⊕

ˇ G

If π  Gˇ ρλ dμ (λ), with ρλ ∈ λ for μ -a.e. λ, then μ and μ are equivalent and ρλ and ρλ are quasi-equivalent almost everywhere. Moreover the center

3.2 Decomposition of representations

213

−1 of 7 ⊕ C(π ) is U DU where D is the algebra of diagonal operators for ˇ Hλ dμ(λ). G

The proof of this theorem is beyond the scope of this book; a good exposition can be found in a book of J. Dixmier [29] §8.4. By central decomposition, the problem of decomposing a representation into irreducible representations is reduced to the decomposition of factor representations. Moreover Lemma 3.2.26 and Proposition 3.2.29 show that the decomposition of a type 1 factor representation is essentially unique. Suppose that G is type 1. It follows from Lemma 3.2.26, Proposition 3.2.29, and Theorem 3.2.32 that every unitary representation π of G has an essentially unique decomposition as a direct integral of multiples of irreducible unitary ˇ each ρλ ∈ λ is written in a unique way mλ πλ , representations: for λ ∈ G, where πλ is irreducible, and π is unitarily equivalent to 3

⊕ ˇ G

3 ρλ dμ(λ) =

⊕ ˆ G

mλ πλ dμ(λ).

ˆ is defined by the Mackey–Borel structure on G. ˇ It can The Borel structure on G be shown that it coincides with the Borel structure defined by the Fell topology ˆ on G. Example 3.2.33 Let G be a separable locally compact group. If G is abelian, then G is type 1. To see this, let π be a factor representation of G. Since G is abelian, each π(x) is an element in the center of C(π ), therefore there is a character χ of G such that, for any x ∈ G, π(x) = χ (x)1Hπ . But by Lemma 3.2.26, this means that π is type 1.



Example 3.2.34 Let G be a compact group, and let π be a factor representation ˆ is countable, each λ ∈ G ˆ has a representative ρλ which is of G. Recall that G a finite-dimensional subrepresentation of the left regular representation of G. Moreover, the function: hλ (x) = dim(ρλ )Trρλ (x) is such that π(hλ ) is an orthogonal projection Pλ commuting with π and such that: π|Pλ H = mλ ρλ .

214

Unitary representations

Moreover H=



Pλ H

ˆ λ∈G

(for the preceding, see [79, théorème p. 25]). Let now T be in C(π ), thus T is commuting with each π(hλ ) = Pλ , T leaves the spaces Pλ H invariant. So T = ⊕λ Tλ and each Tλ is in C(mλ ρλ ). The converse is immediate:  C(mλ ρλ ). C(π ) = ˆ λ∈G

 This implies that the space D of diagonal operators λ aλ 1Pλ H is included in the center of C(π ). Since π is a factor, thus dim D = 1 = |{λ : mλ = 0}| thus π = mλ ρλ is a type 1 factor representation. Thus G is type 1.  Example 3.2.35 Let G be the three-dimensional Heisenberg group as in Example 3.2.14, and let π be a factor representation of G. Then the image of the center Z = {(0,0,z)} under π is in the center of C(π ), therefore there is λ ∈ R such that: π(0,0,z) = eiλz 1Hπ . If λ = 0, then the Stone–von Neumann theorem (see [16, theorem 2.2.9]) says that π is a multiple of the irreducible representation π0,λ described in Example 3.2.14. If λ = 0, then [G,G] ⊂ ker π, and {π(x,y,z) : (x,y,z) ∈ G} is a family of commuting operators. Hence π(G) is included in the center of C(π ), which means that π(G) consists of scalar operators, and it follows that π is a multiple of a unitary character. In either case, the factor representation π is type 1, so the Heisenberg group is type 1. Recall that the unique decomposition of its regular representation was presented in Example 3.2.31.  We shall see that, more generally, every exponential solvable Lie group is type 1. We return to this point in Chapter 5. Example 3.2.36 Now recall the Mautner group G and the first decomposition of its left regular representation given in Example 3.2.21: 3 ⊕ Tξ,η dξ dη. LT = C2

3.2 Decomposition of representations

215

The Hilbert space for each Tξ,η is L2 (R) so the space for the direct integral T is L2 (C2,L2 (R)). The representations Tξ,η are given by 

 −it −iαt Tξ,η (z,w,x)f (t) = f (t − x)eiRe(ze ξ +we η) .

Observe that for all (ξ,η), the operator Tξ,η (0,0,x) is just the translation f → f (· − x) in L2 (R). Similar to the Heisenberg example, for (ξ,η) ∈ C \ {0} × C \ {0}, one finds that Tξ,η is irreducible, and we further claim that Tξ,η and Tξ ,η are equivalent if and only if (ξ ,η ) = t · (ξ,η) = (eit ξ,eiαt η) for some t ∈ R. This claim will be proved at the end of the chapter. Now the above decomposition can be written in iterated form: 3

⊕

C2

3 Tξ,η dξ dη  =

3

⊕

T2

ρ>0,σ >0 3 ⊕



⊕

Tρξ,σ η dμ(ξ,η) ρdρ σ dσ

πρ,σ ρdρ σ dσ, ρ>0,σ >0

where T2 = {(ξ,η) : |ξ | = |η| = 1}, and dμ(ξ,η) is the Haar measure with volume (2π )2 of T2 . Thus the Hilbert space for each πρ,σ is L2 (T2,L2 (R)). Next we show that π = π1,1 =

7⊕

T2

Tξ,η dμ(ξ,η),

is a factor representation. Let M ∈ cent C(π ). According to Proposition 3.2.19 2 2 2 and 7 ⊕ Remark 3.2.20, M is a diagonal operator on L (T ,L (R)): M = Mθ = θ (ξ,η) 1L2 (R) . We must show that M is a scalar operator, that is, θ is a constant function. For each u ∈ R, the translation operator Vu = Tξ,η (0,0, − u) on L2 (R) satisfies 

   Vu Tξ,η (z,w,x)Vu−1 g (t) = Tξ,η ((0,0, − u)(z,w,x)(0,0,u))g (t)   = Tξ,η (e−iu z,e−iαu w,x)g (t) = (Teiu ξ,eiαu η (z,w,x)g)(t).

Now define the operator Wu acting on L2 (T2,L2 (R)) by: 

Wu f

 ξ,η

= Vu−1 (feiu ξ,eiαu η ), f ∈ L2 (T2,L2 (R)).

216

Unitary representations

Then Wu is a unitary operator belonging to C(π ): for f = (fξ,η ) ∈ L2 (T2,L2 (R)), (π(z,w,x)Wu f )ξ,η = Tξ,η (z,w,x)Vu−1 (feiu ξ,eiαu η ) = Vu−1 Teiu ξ,eiαu η (z,w,x)feiu ξ,eiαu η    = Wu Tξ,η (z,w,x)fξ,η ξ,η   = Wu π(z,w,x)f ) . Now M ∈ C(π ) so M commutes with π(0,0, − u) and M ∈ cent C(π ) so M commutes with the operator Uu = π(0,0, − u)Wu . Let f be a nonzero constant vector field, so that for some φ ∈ L2 (R), fξ,η = φ for all (ξ,η). Since (Uu f )ξ,η = feiu ξ,eiαu η , we have Uu f = f and θ (ξ,η)φ = (Mf )ξ,η = (MUu f )ξ,η = (Uu Mf )ξ,η = (Mf )eiu ξ,eiαu η = θ (eiu ξ,eiαu η)φ.

(3.2.1)

It follows that for each u, θ (ξ,η) = θ (eiu ξ,eiαu η) holds for almost every (ξ,η) ∈ T2 . But it is easily seen that this implies that θ is constant. Indeed, for each k,l ∈ Z, 3 3 θ (ξ,η)ξ k ηl dμ(ξ,η) = θ (eiu ξ,eiαu η) ξ k ηl dμ(ξ,η) T2 T2 3 −iu(k+α) θ (ξ,η) ξ k ηl dμ(ξ,η) =e T2

7 this implies that the only nonvanishing Fourier coefficient ck,l = T2 θ (ξ,η) ξ k ηl dμ(ξ,η) is c0,0 . Thus the representation π is a factor representation. By a similar proof, each representation πρ,σ is a factor representation.  using the Fourier Coming back to the definition in Example 3.2.21 of Tρ,σ,τ transform with respect to the variable t, we get, with the notation in this example that V −1 ∧t is a unitary intertwining operator between 3 ⊕ 3 ⊕   πρ,σ = Tρξ,σ η dμ(ξ,η) and πρ,σ = Tρ,σ,τ dτ . T2

R

 πρ,σ

is a factor. Finally, recall that in Example 3.2.21, we gave also Thus a decomposition of L into completely different irreducible representations. Written in iterated form it is  3 ⊕ 3 ⊕ 3 ⊕   L Tρ,σ,τ dτ ρdρ σ dσ = πρ,σ ρdρ σ dσ . ρ>0,σ >0

τ ∈R

ρ>0,σ >0

 As expected from the uniqueness part of Theorem 3.2.32, since πρ,σ and πρ,σ are equivalent, this is the same decomposition of L into factors.

3.2 Decomposition of representations

217

 is irreducible. First for each z, w, Let us now prove that Tρ,σ,τ is a multiplication operator

 (z,w,0) Tρ,σ,τ

 (z,w,0)f (y,s) = Mφz,w f (y,s), with φz,w (y,s) = eiRe(zρy+wσ s) . Tρ,σ,τ

The involutive algebra generated by the continuous functions φz,w is separating points on the torus, thus it is dense in the space C(T2 ). If S is an operator commuting with Mφz,w , it is commuting with any multiplication operator by a continuous function f , then: Sf = S(Mf 1) = Mf S1 = MS1 f . By density of C(T2 ) in L2 (T2 ), S = MS1 . Then each orthogonal projection,  is also a multiplication operator, by the characteristic commuting with Tρ,σ,τ  (0,0,x), function χE of a Borel set E. Now this operator commutes with Tρ,σ,τ ix this implies that E is invariant under the map (y,s) → (ye ,seiαx ), now χE satisfies the relation (3.2.1) and the above computation shows that χE is a constant function. This proves the projection is 0 or 1, and Tρ,σ,τ is irreducible.  is equivalent to Tρ ,σ ,τ  . First let us prove that Suppose now that Tρ,σ,τ ρ  = ρ, σ  = σ , we restrict these representations to C2 , and observe that for any f ∈ L1 (C2 ):  |C2 (f ) = Mfˆ(ρ·,σ ·), Tρ,σ,τ

Tρ ,σ ,τ  |C2 (f ) = Mfˆ(ρ  ·,σ  ·) .

If ρ = ρ  , choosing a function f such that fˆ is vanishing on the torus {(ρ  y,σ  s) : (y,s) ∈ T2 } and fˆ −1 is vanishing on the torus {(ρy,σ s) : (y,s) ∈ T2 }, we get an impossibility. Now if U is a unitary operator intertwining   and Tρ,σ,τ Tρ,σ,τ  , as above, U is a multiplication operator Mϕ , for ϕ = U 1. Moreover:   (0,0,x) = Tρ,σ,τ U Tρ,σ,τ  (0,0,x)U

gives that almost everywhere: ϕ(yeix ,seiαx )ei(τ

 −τ )x

= ϕ(y,s).

Let m, n in Z such that ϕ(m,n) ˆ = ϕ,y m s n  is not vanishing. Then for any x: 3  ϕ(m,n) ˆ = ϕ(yeix ,seiαx )ei(τ −τ )x y m s n dν(y)dν(s) = eix(τ

 −τ −m−nα)

ϕ(m,n). ˆ

This gives τ  − τ = m + nα. Conversely, if this relation holds, then   and Tρ,σ,τ M(y m s n ) is an intertwinning operator between Tρ,σ,τ  . These two  representations are equivalent if and only if τ − τ ∈ Z + αZ.

218

Unitary representations

  is a type 1 factor. Then πρ,σ Suppose now that the factor πρ,σ  πρ,σ would be a multiple of some irreducible representation π0 . By Proposition  would be equivalent to π0 . But the preceding shows 3.2.29, almost every Tρ,σ,τ  the set {τ : Tρ,σ,τ  π0 } is countable, so this is not the case, thus the Mautner group admits a factor representation which is not type 1. Therefore the Mautner group is not a type 1 group. 

3.3 Induced representations The construction of an induced representation plays a central role in the theory of unitary group representations, especially for solvable Lie groups. Starting with a unitary representation (σ,Hσ ) of a closed subgroup H , the representation of G induced by (σ,Hσ ) is just translation by elements of G on an appropriately defined Hilbert space H of Hσ -valued functions on G. There are several (equivalent) ways to define H presented in numerous prior texts; see, for example, the monograph of Bernat et al. [11], the books by G. Folland [40], and the book by E. Kaniuth and K. F. Taylor [50]. The development given here is one that seems to suit the context of solvable Lie groups well, and extracts elements from each of these sources.

3.3.1 Results on homogeneous spaces Let G be a locally compact, second countable group. Recall the notions of Haar measure μG and modular function G such that 3 3 φ(yx)dμG (x) = φ(x)dμG (x) G

G

and 3

3

−1

)dμG (x) = G (y) φ(x)dμG (x), G 3 φ(x −1 )G (x)−1 dμG (x) = φ(x)dμG (x). φ(yxy

3

G

G

G

For simplicity we may write dx instead of dμG (x). Recall that if G is a simply connected solvable Lie group, then G (x) = | det Ad(x)|−1 ; it can be proved that this formula remains true for any connected Lie group. Especially, G (exp X) = |e−trace adX | (X ∈ g). If G is simply connected and (Zi ) is a good Jordan–Hölder basis of g with the associated roots sequence (λi )i , to

3.3 Induced representations

219

each λi corresponds a unique character i of G such that i (exp X) = eλi (X) if X ∈ g. By the above, we have: # n # n #4 # 4 # −λi (X) # |−1 G (exp X) = # e #= i (exp X)|, # # i=1

i=1

so for any x ∈ G, G (x) = ni=1 |−1 i (x)|. Let H be a closed subgroup of G and let q = qG/H : G → G/H be the canonical quotient map. The quotient space G/H is locally compact, and q is an open map. Denote the space of continuous, complex-valued functions on G having compact support by Cc (G), and the space of continuous, complexvalued functions on G/H having compact support by Cc (G/H ). We identify Cc (G/H ) with the space of continuous functions g on G with the property that g(xh) = g(x) for all h ∈ H and q(supp g) is compact. Define P : Cc (G) → Cc (G/H ) by 3 φ(xh)dh. P φ(xH ) = H

We collect a few preliminary results. Lemma 3.3.1 Let F ⊂ G/H compact. Then there is C ⊂ G compact such that q(C) = F . Proof: Choose a compact neighborhood V of e in G, then F ⊂ ∪x∈q −1 (F ) q(xV ), each q(xV ) is a neighborhood of q(x), and by compactness, there are x1, . . . xn such that 5 q(xi V ). F ⊂ 1≤i≤n

Put C = q −1 (F ) ∩ (∪i xi V ). Then C is compact and q(C) = F . Lemma 3.3.2 For F ⊂ G/H compact, there is η = ηF ∈ Cc (G) such that η ≥ 0 and P η ≡ 1 on F . Proof: Let E be a compact neighborhood of F in G/H . Let g ∈ Cc (G/H ), g ≥ 0, supp(g) ⊂ E, and g ≡ 1 on F . Let now C be a compact subset in G such that q(C) = E, and φ ∈ Cc (G), such that φ ≥ 0, φ > 0 on C, then consider: η(x) =

g(q(x)) φ(x), if x ∈ C, P φ(q(x))

and 0 elsewhere.

220

Unitary representations

Since supp g ⊂ E, then η is continuous with compact support included in C ⊂ G. Moreover h → Pg(q(xh)) φ(q(xh)) is constant, thus 3 P η(q(x)) = H

g(q(x)) g(q(x)) φ(xh) dh = P φ(q(x)) = g(q(x)). P φ(q(x)) P φ(q(x))

Lemma 3.3.3 For each g ∈ Cc (G/H ), there is φ ∈ Cc (G) such that P φ = g and q(supp φ) = supp g. If g ≥ 0 then we can take φ ≥ 0. Moreover if F is a compact subset of G/H , there is a compact subset C in G, CF > 0 such that for any g ∈ Cc (G/H ) with supp g ⊂ F , then φ can be chosen with support in C and sup |g(q(x))| ≤ CF sup |φ(x)|. Proof: Let F be a compact subset of G/H and suppose q(supp g) ⊂ F ; by Lemma 3.3.2, there is η = ηF ∈ Cc (G), η ≥ 0, P η ≡ 1 on F . Put φ(x) = η(x)g(q(x)). As in the preceding proof, P φ(q(x)) = g(q(x))P η(q(x)), which proves P φ(q(x)) = g(q(x)) if q(x) ∈ F , and P φ(q(x)) = 0 = g(q(x)) if q(x) ∈ / F . Observe that φ is positive if g is positive, and by definition supp φ ⊂ q −1 (supp g), but P φ = g implies supp g ⊂ q(supp η), thus q(supp φ) = supp g. Observe that, with our preceding notation, supp φ ⊂ C, and for each x, |φ(x)| ≤ sup η(y) |g(q(x))| = CF |g(q(x))|. y∈C

Suppose that the quotient map q : G → G/H has a continuous section q s s : G/H → G: the composition G/H −→ G −→ G/H is the identity map. Then the reader can verify that G is topologically homeomorphic to G/H ×H , by using the writing x = s(q(x))h(x) (h(x) ∈ H ) and reprove the preceding Lemmas 3.3.1, 3.3.2, and 3.3.3 directly in this case. Observe that if G is simply connected solvable and H is a connected closed subgroup with dimension r, an explicit continuous section is given by choosing a coexponential basis (Y1, . . . ,Yn−r ) to h in g, and putting: s(q(exp t1 Y1 . . . exp tn−r Yn−r )) = exp t1 Y1 . . . exp tn−r Yn−r .

(3.3.1)

Put δH,G (h) = H (h)/G (h), h ∈ H . For brevity we also write δ(h) = δH,G (h). Observe that if h = exp Y , with Y ∈ h, then δ(exp Y ) = |etrace adg/h Y |.

3.3 Induced representations

221

Lemma 3.3.4 Let g be solvable, and let (Zi ) be a good Jordan–Hölder basis for gc with root sequence (λi ), and h a subalgebra of g. Define, with our usual notation, e(h) = {1 ≤ j ≤ n : sj + hc = sj −1 + hc }. Let j , 1 ≤ j ≤ n such that j − 1 ∈ I and suppose that λj |h is not real-valued (so that j ∈ / I). Then j ∈ e(h) if and only if j + 1 ∈ e(h). Moreover, 4 |j (h)|. δH,G (h) = j ∈e(h)

Proof: Let j be an index with the hypothesis of the lemma, suppose that j ∈ / sj −1 + hc and Zj ∈ / g, so this implies that j ∈ / I and Zj +1 = e(h), then Zj ∈ Z j . If, to the contrary, j + 1 ∈ / e(h), then there is a ∈ C, U ∈ hc and W ∈ sj −1 such that Zj +1 = U + aZj + W . If a = 0, then Zj = Z j +1 ∈ sj −1 + hc . Observe that |a| = 1 since this relation implies Zj = U + a(U + aZj + W ) + W = |a|2 Zj + U + aU + W + W, or (1 − |a|2 )Zj ∈ hc + sj −1 , which means |a| = 1. Let b such that b2 = −1/a, then: bU = bZj +1 + bZj − bW ∈ hc, / R. Then thus Y : = Re(bZj ) − Re(bW ) ∈ h. Let A ∈ h such that λj (A) ∈ Y and [A,Y ] are independent modulo sj −1 which then implies that Zj ∈ sj −1 + hc , a contradiction. The same argument, but where the roles of Zj and Zj +1 are reversed implies the converse. Choose a good basis for the h-module s/hc as follows. Suppose j −1 ∈ I, if / I, let d = dim((sj +1 + hc )/(sj −1 + j ∈ I and j ∈ e(h), put Xj = Zj . If j ∈ hc )) = |{j,j + 1} ∩ e(h)|. If d = 2 put Xj = Zj , Xj +1 = Zj +1 , if d = 1, then λj |h is real valued, choose Xj (if j ∈ e(h)) or Xj +1 (if j + 1 ∈ e(h)) in (sj +1 + hc ) \ (sj −1 + hc ). By construction, {Xj + hc : j ∈ e(h)} is a flag basis for the h-module s/hc . Thus δH,G (h) = | det Adg/h (h)| =

4 j ∈e(h)

|j (h)|.

222

Unitary representations

Put Cc (G,H,δ) = {g ∈ C(G) : g(xh) = δ(h)g(x), q(supp g) is compact }. 7 Define Pδ : Cc (G) → Cc (G,H,δ) by (Pδ φ)(x) = H φ(xh)δ(h)−1 dh. Lemma 3.3.5 The map 7 Pδ is surjective, and ker Pδ is included in the kernel of the functional φ → G φ dμG . Proof: Let f ∈ Cc (G,H,δ) and choose any η ∈ Cc (G) such that P η ≡ 1 on q(supp f ), put φ = ηf , then 3 3 Pδ (φ)(x) = η(xh)f (xh)δ(h)−1 dh = f (x) η(xh) dh. H

H

Thus Pδ φ(x) = f (x) if q(x) ∈ q(supp f ) and Pδ φ(x) = 0 = f (x) if q(x) ∈ / q(supp f ). Suppose now φ ∈ Cc (G) is in ker Pδ , choose η ∈ Cc (G) such that P η ≡ 1 on q(supp φ), then:  3 3 3 φ(x) dx = η(xh)dh φ(x)dx G 3G 3 H = η(xh−1 )φ(x)H (h)−1 dhdx G H 3 3 = η(x)φ(xh)G (h)H (h)−1 dhdx G H 3  3 −1 η(x) φ(xh)δ(h) dh dx = H 3G = η(x)Pδ (φ)(x)dx = 0. G

By virtue of the Lemma 3.3.5, we can define a linear functional μG,H : Cc (G,H,δ) → C by 3 φ dμG . (3.3.2) μG,H (Pδ (φ)) = G

Lemma 3.3.3 implies that μG,H is positive in the sense that g ≥ 0 implies μG,H (g) ≥ 0. By definition and left-invariance of the Haar measure on G, μG,H is left-invariant: μG,H (Lx f ) = μG,H (f ) holds for all x ∈ G. Given g ∈ Cc (G,H,δ), we can compute μG,H (g) explicitly as follows: choose φ ∈ Cc (G) such that Pδ φ = g, and choose any η ∈ Cc (G) such that P η ≡ 1 on q(supp φ). Then for x ∈ supp φ, P η(x) = 1, so with the same computation as in the proof of Lemma 3.3.5,

3.3 Induced representations 3

3

3

μG,H (g) =

φ(x) dx =

=

 φ(xh)δ(h)−1 dh dx

η(x)

3G

G

223

H

η(x)g(x)dx. G

Although μG,H is not a measure on G/H , we will nevertheless abuse notation slightly and write 3 f (x)dμG,H (x). μG,H (f ) = G/H

Hence by definition of μG,H , we can write 3 3 3 φ(x)dx = φ(xh)δH,G (h)−1 dh dμG,H (x) G

G/H

(3.3.3)

H

for φ ∈ Cc (G). Example 3.3.6 Let N be a simply connected, nilpotent Lie group, with Lie algebra n and D a connected subgroup with Lie algebra d. Let be a unimodular group of automorphisms of N such that D is γ -stable for any γ ∈ . Put G = N  , and H = D  . Since each automorphism of N has a unique expression exp X → exp γ (X), where γ is an automorphism of n, we identify with a group of automorphisms of n. Denote by dx, dy, dγ the Haar measures for N , D, and . Choose the Haar measure on G defined in Corollary 3.1.3: for each continuous, compactly supported function φ, 3 3 3 φ dμG = φ(x,γ )|det γ |−1 dxdγ . G

N

This formula gives the modular function of G: 3 3 3 3   φ (x,γ )(x ,γ  ) |det γ |−1 dxdγ = φ(xγ (x  ),γ γ  )|det γ |−1 dxdγ N 3 3N = φ(x,γ )|det γ |−1 |det γ  | dxdγ . N

This proves that G (x ,γ  ) = |det γ  |−1 . Hence the Haar measures on G and H can be written: 3 3 3 φ dμG = φ(x,γ )G (γ ) dxdγ , G N 3 3 3 φ dμH = φ(y,γ )H (γ ) dydγ . H

D

Since (xy,γ ) = (x,1)(y,γ ), for each x ∈ N , y ∈ D, and γ ∈ , then the quotient G/H is canonically isomorphic to the quotient N/D. Hence it

224

Unitary representations

is natural to compare the measures dμG,H and dμN,D , defined both on this quotient. By Relation (3.3.3), 3 3 φ(x,γ )G (γ ) dxdγ N 3 3 3   = φ (x,γ )(y,γ  ) δH,G (y,γ )−1 H (γ ) dydγ dμG,H (x,γ ). G/H

D

On the other hand, since δD,N (y) = 1 for each y ∈ D, and δH,G (γ ) =

H (γ ) |det γ | = = |det γn/d |, G (γ ) |det γ |d |

we get: 3 3 φ(x,γ )G (γ ) dxdγ 3 3 3 = φ(xy,γ )dydμN,D (x)G (γ ) dγ N/D D 3 3 3 = φ((x,1)(y,γ ))δH,G (γ )−1 H (γ )dydγ dμN,D (x) N/D D 3 3 = φ((x,1)(y,γ ))dμH (y,γ ) δH,G (γ )−1 dμN,D (x). N

N/D

H

Therefore, dμG,H = dμN,D .



Suppose next that K is a closed subgroup of H . Observe that for each k ∈ K, δK,H (k) =

K (k) K (k) G (k) = = δK,G (k)δH,G (k)−1 . H (k) G (k) H (k)

Lemma 3.3.7 Let E ⊂ G such that qG/K (E) is compact. Then qG/H (E) is compact. Proof: By Lemma 3.3.1, there is C ⊂ G compact such that qG/K (C) = qG/K (E). Now qG/H (C) is compact, and it is easy to check that qG/H (C) = qG/H (E). Let f ∈ Cc (G,K,δK,G ). For each x ∈ G, define fx : H → C by fx (h) = −1 belongs to Cc (H,K,δK,H ): f (xh). Then the function fx δH,G −1 fx δH,G (hk) = f (xhk)δH,G (hk)−1 = f (xh)δH,G (h)−1 δH,G (k)−1 δK,G (k) −1 (h)δK,H (k). = fx δH,G

Let φ be in Cc (G) such that f = PδK,G φ.

3.3 Induced representations

225

Define g ∈ C(G) by −1 ). g(x) = gf (x) = μH,K (fx δH,G

Lemma 3.3.8 For each x ∈ G, g(x) = PδH,G φ(x). −1 −1 Proof: Define φx δH,G as the map h → φ(xh)δH,G (h). By construction, −1 φx δH,G ∈ Cc (H ), and we have: 3 −1 )(h) = φ(xhk)δH,G (hk)−1 δK,H (k)−1 dk PδK,H (φx δH,G 3K −1 φ(xhk)δK,G (k)−1 dk δH,G (h)−1 = fx δH,G (h). = K

This means: −1 )) = g(x) = μH,K (PδK,H (φx δH,G

3 H

φ(xh)δH,G (h)−1 dh = PδH,G φ(x).

The preceding lemma allows us to prove that μG,K can be computed by an iteration. Proposition 3.3.9 Let f ∈ Cc (G,K,δK,G ). Then  3 3 3 −1 f (y)dμG,K (y) = f (xh)δH,G (h) dμH,K (h) dμG,H (x). G/K

G/H

H /K

−1 ). Let Proof: Denote gf as in the preceding lemma by gf (x) = μH,K (fx δH,G φ ∈ Cc (G) such that f = PδK,G φ, we saw that: 3 φ(·h)δH,G (h)−1 dh. gf = PδH,G φ = H

Apply (3.3.3) to the compactly supported function h → φ(xh)δH,G (h)−1 on H to get 3 3 3 f (y)dμG,K (y) = φ(x) dx = PδH,G φ(x) dμG,H (x) G/K G G/H 3 = gf (x) dμG,H (x). G/H

Now let G be a simply connected solvable Lie group, and let (Z1, . . . ,Zn ) be a good Jordan–Hölder basis for gc = s, with si = span{Z1, . . . ,Zi }. Let H be a closed connected (hence simply connected) subgroup. Choose a weak

226

Unitary representations

Malcev basis (X1, . . . ,Xn ) for g, associated to (Zi ), and passing through h. Put gj = span{X1, . . . ,Xj }; each gj is a subalgebra of g, and by Lemma 3.1.1, the corresponding subgroup Gj has a left-invariant measure μGj defined by 3 φ → φ(exp uj Xj · · · exp u1 X1 )du. Rj

Recall that if i ∈ I, then gi = si ∩ g is an ideal, and that there is a subsequence Ih = (i1 < i2 < · · · < ir ) of (1,2, . . . ,n), so that (Xi1 , . . . ,Xir ) is a weak Malcev basis for h, and we have the associated left-invariant measure μH on H from Lemma 3.1.1. Writing the complementary subsequence as Ihc = (j1 < j2 < · · · < jn−r ) and putting Y1 = Xjn−r , . . . , Yn−r = Xj1 , we have (Y1, . . . ,Yn−r ) is a coexponential basis for h in g as in Proposition 1.8.5. Recall the continuous section s : G/H → G defined in Relation (3.3.1) s(exp t1 Y1 · · · exp tn−r Yn−r H ) = exp t1 Y1 · · · exp tn−r Yn−r . Lemma 3.3.10 Fix the Haar measures dμG and dμH as in Lemma 3.1.1, by using the weak Malcev basis (Xi )1≤i≤n for g and (Xu )u∈Ih for h. Then we have 3 3 3 φ dμG = φ(exp s1 Y1 · · · exp sn−r Yn−r h)δ(h)−1 dμH (h) ds. Rn−r

H

Hence the corresponding form μG,H is given by 3 f (exp s1 Y1 · · · exp sn−r Yn−r )ds1 · · · dsn−r , f ∈ Cc (G,H,δ). μG,H (f ) = Rd

Proof: By induction on the dimension of g, the lemma being easily verified if dim g = 1. Suppose that dim g = n > 1 and that the lemma holds for solvable Lie algebras of dimension less than n. / Ih , and We claim that if λjn−r |h is not real, then jn−r−1 = jn−r − 1 ∈ Z jn−r = Zjn−r −1 . Suppose that λjn−r |h is not real. Let A ∈ h such that / R. Suppose that for some a ∈ C \ {0}, X = Re(aZjn−r ) + β = λjn−r (A) ∈  . Then [A,X] = Re(aβZjn−r ) + W ∈ h. But X and W0 ∈ h, with W0 ∈ gjn−r    , which means that gjn−r ⊂ gjn−r +h. This [A,X] are independent modulo gjn−r   \ (gjn−r + h). The claim follows. is impossible since Y1 = Xjn−r ∈ gjn−r We conclude that there is j , 1 ≤ j ≤ n, such that k : = h + gj −1 is a subalgebra, and that the quotient g/(h + gj −1 ) is an irreducible h-module, hence of dimension one or two. Here either (a) dim g/(h + gj −1 ) = 1 whence λjn−r |h is real and j = jn−r , or (b) dim g/(h + gj −1 ) = 2, λjn−r |h is not real and Zjn−r−1 = Z jn−r , whence j = jn−r−1 . The proof for (a) is little more than a simplification of the proof for (b), so we give details only in case (b).

3.3 Induced representations

227

Suppose then that λjn−r |h is not real, Zjn−r−1 = Z jn−r , and k = h + gj −1 is a codimension two subalgebra of g, where j = jn−r−1 = jn−r − 1. Observe that if jn−r = n, and jn−r−1 = j = n − 1, then there is almost nothing to prove, as δH,K = δH,G in this case, and we apply induction to the subalgebra k. Assume then that jn−r < n. The weak Malcev basis can be written as (X1,X2, . . . ,Xj −1,Y2,Y1, Xj +2, . . . ,Xn ) and a weak Malcev basis for k is (X1, . . . ,Xj −1,Xj +2, . . . ,Xn ). Let k0 ⊂ k1 ⊂ · · · ⊂ kn−2 = k be the corresponding sequence of subalgebras of k, for each 1 ≤ i ≤ n − 2, and for t ∈ Ri , let bi (t) be the element in Ki corresponding to t via the weak Malcev basis for ki , and for each i, let μi be the left Haar measure on the subgroup Ki defined by the weak Malcev basis as in Lemma 3.1.1. Write Zjn−r = Y1 +iY2 : = Y , so s1 Y1 +s2 Y2 = Re(sY ) where s = s1 +is2 . For A ∈ h, we have modulo Kj −1 : exp A exp(s1 Y1 + s2 Y2 ) exp −A = exp Re(j (exp A)−1 sY )   = exp Re(j (exp A)−1 s)Y1 + Im(j (exp A)−1 s)Y2 . Note that the map     Re(j (exp A)−1 s) s1 → s2 Im(j (exp A)−1 s) has determinant |j (exp A)|−2 . In what follows we abuse notation slightly and write sY = s1 Y1 + s2 Y2 and   exp Re(j (exp A)−1 s)Y1 + Im(j (exp A)−1 s)Y2 = exp j (exp A)−1 sY . Note also that δH,G (h) = |j (h)|−2 δH,K (h) (h ∈ H ). Write exp un Xn · · · exp u1 X1 = exp tn−2 Xn · · · exp tj Xj +2 exp s1 Y1 exp s2 Y2 bj −1 (t) = exp tn−2 Xn · · · exp tj Xj +2 exp(s1 Y1 +s2 Y2 )a bj −1 (t) where a and bj −1 (t) are in Kj −1 . Using left-invariance of μj −1 , 3 3 3 φ dμ = G R2 Rn−2   φ exp tn−2 Xn · · · exp tj Xj +2 exp(s1 Y1 +s2 Y2 ) abj −1 (t) dtds1 ds2 3 3 = R2 Rn−2   φ exp tn−2 Xn · · · exp tj Xj +2 exp(s1 Y1 + s2 Y2 ) bj −1 (t) dtds1 ds2 .

228

Unitary representations

We claim that this integral can be written as: 3 3 3 φ dμ = φ(exp sY b) |j (b)|2 dμK (b) ds. R2

G

K

First write exp tj Xj +2 exp sY exp −tj Xj +2 = exp(s etj adXj +2 Y ) = exp(s j (exp tj Xj +2 )−1 Y ) aj −1 where aj −1 ∈ Kj −1 , so that exp tn−2 Xn · · · exp tj Xj +2 exp(s1 Y1 + s2 Y2 ) bj −1 (t) = exp tn−2 Xn · · · exp tj +1 Xj +3 exp(j (exp tj Xj +2 )−1 sY ) × exp tj Xj +2 aj −1 bj −1 (t). Using left-invariance of μj −1 again, and the fact that j (bj −1 (t)) = 1, and bj (t) = exp tj Xj +2 bj −1 (t), 3 φ dμ G 3 3  = φ exp tn−2 Xn · · · exp tj +1 Xj +3 exp(s j (exp tj Xj +2 )−1 Y ) R2 Rn−2  × exp tj Xj +2 bj −1 (t) dt ds 3 3   = φ exp tn−2 Xn · · · exp tj +1 Xj +3 exp(sj (bj (t))−1 Y )bj (t) dt ds. R2 Rn−2

Now repeat the same argument with exp tj +1 Xj +3 , so that 3 3 3 φ dμ = φ(exp tn−2 Xn · · · exp tj +2 Xj +4 R2

G

Rn−2

× exp(sj (bj +1 (t))−1 Y ) bj +1 (t))dt ds. Continuing in this way, 3 3 3 φ dμ = φ(exp(j (bn−2 (t))−1 sY ) bn−2 (t))dt ds G R2 Rn−2 3 3 = φ(exp sY bn−2 (t)) |j (bn−2 (t))|2 dt ds 2 n−2 3R 3R φ(exp sY b) |j (b)|2 dμK (b) ds. = R2

K

3.3 Induced representations

229

This proves the claim. Now exp sY = exp s1 Y1 exp s2 Y2 aj −1 for some aj −1 ∈ Kj −1 , so 3 3

3 φ dμ = G

R K

φ(exp s1 Y1 exp s2 Y2 b)|j (b)|2 dμK (b) ds.

Writing b ∈ K as b = exp s3 Y3 · · · exp sn−r Yn−r h with h ∈ H , observe that j (b) = j (exp s3 Y3 · · · exp sn−r Yn−r h) = j (h). So by induction 3

3

R2

3

=

K

R2

φ(exp s1 Y1 exp s2 Y2 b)|j (b)|2 dμK (b) ds 3 3 φ(exp s1 Y1 exp s2 Y2 exp s3 Y3 · · · Rn−r−2

H

× exp sn−r Yn−r h) |j (h)|2 δH,K (h)−1 dμH (h) ds 3 3 φ(exp s1 Y1 exp s2 Y2 · · · exp sn−r Yn−r h) δH,G (h)−1 dμH (h) ds. = Rn−r

H

This proves the lemma.

3.3.2 Definition of induced representations Now let Hσ be a Hilbert space and let σ be a unitary representation of H in Hσ . Let Cc (G,Hσ ) be the set of all continuous Hσ -valued functions on G with compact support. We retain the notation δ = δH,G , for simplicity. Let Cc (G,H,σ ) denote the subspace of C(G,Hσ ) consisting of all functions f ∈ C(G,Hσ ) such that q(supp f ) is compact, and f (xh) = δ(h)1/2 σ (h)−1 f (x) holds for all h ∈ H . For α ∈ Cc (G,Hσ ) define fα : G → Hσ by 3 σ (h)α(xh)δ(h)−1/2 dh. fα (x) =

(3.3.4)

H

Proposition 3.3.11 For each α ∈ Cc (G,Hσ ), fα belongs to Cc (G,H,σ ) and q(supp fα ) ⊂ q(supp α). Moreover, every function f belonging to Cc (G,H,σ ) is of the form fα for some α ∈ Cc (G,Hσ ).

230

Unitary representations

Proof: It is easy to check that fα belongs to Cc (G,H,σ ); indeed q(supp fα ) ⊂ q(supp α) is compact in G/H and for h0 ∈ H , 3 3 −1 −1/2 −1/2 σ (h)α(xh0 h)δ(h) dh = σ (h−1 dh fα (xh0 ) = 0 h)α(xh)δ(h0 h) H

H

= σ (h0 )−1 δ(h0 )1/2 fα (x). We show that fα is continuous; fix x0 ∈ G and let ε > 0. Let K = supp α, and fix a balanced neighborhood N (N −1 = N ) of e in G. Choose a balanced neighborhood U = Uε of e in N such that u ∈ U implies α(ux) − α(x) < ε/M, for all x, where 3 M= δ(h)−1/2 dh. (x0−1 N K)∩H

Then for any h ∈ H , any y ∈ U x0 , α(yh) − α(x0 h) < ε/M. Now for y ∈ U x0 , the support of the function h → α(yh) − α(x0 h) is included / in the set x0−1 NK. To see this, write y = ux0 with u ∈ U ⊂ N . Then h ∈ / N K ⊃ K and yh = ux0 h ∈ / K, therefore α(yh) − x0−1 N K implies x0 h ∈ α(x0 h) = 0. Using the fact that σ (h) is unitary, we compute that for y ∈ U x0 , ;3 ; ; ; −1/2 ; fα (y) − fα (x0 ) = ; (σ (h)α(yh) − σ (h)α(x0 h)) δ(h) dh; ; 3 H ≤ σ (h)(α(yh) − α(x0 h)) δ(h)−1/2 dh 3H α(yh) − α(x0 h) δ(h)−1/2 dh = (x0−1 N K)∩H

< ε. Finally, let f ∈ Cc (G,H,σ ). By Lemma 3.3.2, there is η ∈ Cc (G) such that P η ≡ 1 on q(supp f ). Now put α = ηf . Then σ (h)(f (xh))δ(h)−1/2 = f (x), so 3 σ (h)(η(xh)f (xh))δ(h)−1/2 dh fα (x) = 3H = η(xh)σ (h)(f (xh))δ(h)−1/2 dh H 3 η(xh)f (x)dh = H

= f (x). Proposition 3.3.11 says that the map Pσ defined by Pσ (α) = fα is a surjective map from Cc (G,Hσ ) onto Cc (G,H,σ ).

3.3 Induced representations

231

Now for f , g ∈ Cc (G,H,σ ), observe that x → f (x),g(x)σ is in Cc (G,H,δ), and define 3 f ,g = f (x),g(x)σ dμG,H (x). G/H

This defines a sesquilinear form on Cc (G,H,σ ) by sesquilinearity of ·,·σ and the linearity of μG,H . Suppose that f ∈ Cc (G,H,σ ) and that f 2 = f ,f  = 0. Let K = supp(x → f (x) σ ), and let η ≥ 0 in Cc (G) such that P η = 1 on q(K). Observe that f (xh) 2σ = δ(h) f (x) 2σ . Then x → f (x) 2σ ∈ Cc (G,H,δ), and 3 3 f (x) 2σ dμG/H (x) = f (x) 2σ η(x) dx. 0= G/H

G

Hence, f (x) σ = 0 for all x and f = 0. Thus, · is a norm on Cc (G,H,σ ). Now for f ∈ Cc (G,H,σ ), y ∈ G, define π(y)f (x) = f (y −1 x). The G-invariance of μG,H implies that π(y) is a linear isometric automorphism of Cc (G,H,σ ), and we have π(y1 y2 ) = π(y1 )π(y2 ) holds for all y1 , y2 ∈ G. Let H(G,H,σ ) be the completion of Cc (G,H,σ ); then for each y ∈ G, π(y) extends to a unitary operator on H(G,H,σ ). For each f ∈ Cc (G,H,σ ), it is easy to check that y → π(y)f is continuous for the norm. Since each π(y) is unitary, thus y → π(y)f is continuous for any f ∈ H(G,H,σ ). Thus π is a unitary representation of G on H(G,H,σ ). We denote the representation π by ind(G,H,σ ). Definition 3.3.12 The representation ind(G,H,σ ) is called the representation of G induced from the representation σ of H . Example 3.3.13 Suppose that H = G, then δG,H = 1 and the space Cc (G,H,σ ) is reduced to the functions f of the form: f (x) = σ (x)−1 v, where v = f (e) ∈ Hσ . The map Tf = f (e) is easily seen to be an intertwining unitary operator between ind (G,H,σ ) and σ . On the other hand, if H = {e}, and σ is the trivial representation of H , then δG,H (e) is 1, μG,H = μG , the space H(G,H,σ ) = L2 (G) and: (ind (G,H,σ )(y)f )(x) = f (y −1 x) = (L(y)f )(x). The induced representation is the left regular representation.



232

Unitary representations

Example 3.3.14 Referring to Examples 1.6.4 and 3.1.8, let G be the threedimensional Heisenberg group realized as R×R×R as in Example 1.6.4, with Lie algebra as in Example 1.1.5. The basis (X1 = Z,X2 = Y,X3 = X) is both a weak Malcev basis and a good Jordan–Hölder basis, and with the notation of Example 1.6.4, (x,y,z) = exp zZ exp yY exp xX. Since G is nilpotent, it is unimodular, and one computes that the measures μl and μr of Lemma 3.1.1 are both just Lebesgue measure: for φ ∈ Cc (G), 3 3 φ(exp zZ exp yY exp xX)dxdydz = φ(x,y,z)dxdydz 3 R3 3R φ(x,y,z + xy)dzdydx = R3 3 = φ(exp xX exp yY exp zZ)dzdydx. R3

Let H be the abelian subgroup H = {(0,y,z) : y,z ∈ R}; its unitary characters χγ ,λ,γ ∈ R,λ ∈ R are defined by χγ ,λ (0,y,z) = ei(λz+γ y) . With H identified with R2 in the obvious way, fix the Lebesgue measure dydz on H . Now δH,G = 1 since G is nilpotent, so by definition the form μG,H corresponding to these choices of Haar measure on G and H is given by 3 g(x,0,0)dx μG.H (g) = 3R g(exp xX)dx, g ∈ Cc (G,H,δH,G ) = Cc (G/H ). = R

The space Cc (G,H,χγ ,λ ) consists of functions f on R3 such that x → f (x,0,0) belongs to Cc (R) and f (x,y,z) = f ((x,0,0)(z − xy,y,0)) = e−iλ(z−xy)+iγ y f (x,0,0), (x,y,z) ∈ R3 . The representation ind(G,H,χγ ,λ ) is realized in the completion of the space Cc (G,H,χγ ,λ ) with respect to the norm 3 2 2 f = μG,H (|f | ) = |f (x,0,0)|2 dx. R

Define the mapping V : Cc (G,H,χγ ,λ ) → Cc (R) by (Vf )(x) = f (x,0,0).

3.3 Induced representations

233

Then V is a bijective with inverse (V −1 f )(x,y,z) = e−i(λ(z−xy)+γ y) f (x). The above shows that V is an isometry and can be extended to an unitary operator from H(G,H,χγ ,λ ) onto L2 (R). Via the unitary map V , the induced representation ind (G,H,χγ ,λ ) is equivalent to the representation πγ ,λ defined in Example 3.2.14:   πγ ,λ (x1,y1,z1 )f (x) = eiλz1 ei(γ −λx)y1 f (x − x1 ). Now suppose that λ = 0. By Example 3.2.16, πγ ,λ is equivalent with π0,λ . The point is that this example shows that the converse of part 1 of Proposition 3.3.19 does not hold: for γ1 = γ2 , χγ1,λ and χγ2,λ are not equivalent, but  ind (G,H χγ1,λ ) and ind (G,H,χγ2,λ ) are equivalent. Lemma 3.3.15 Let (σ,Hσ ) be a unitary representation of H , π = ind (G,H,σ ) and let Pσ : Cc (G,Hσ ) → Cc (G,H,σ ) be the natural surjection defined in Proposition 3.3.11. Let K be any compact subset of G and define Cc (K,Hσ ) = {α ∈ Cc (G,Hσ ) : supp α ⊂ K}. Then there is a constant CK such that Pσ (α)

π

≤ CK sup α(x)

σ

x∈G

holds for all α ∈ Cc (K,Hσ ). Hence the restriction Pσ,K of Pσ to Cc (K,Hσ ) is continuous from Cc (K,Hσ ) into H(G,H,σ ). Proof: Pick a compact neighborhood F of K in G, and pick η ∈ Cc (G) with supp η ⊂ F and P η = 1 on q(K). Then for any α ∈ Cc (K,Hσ ), we have supp fα ⊂ q(K), so 3 fα 2π = η(x) fα (x) 2σ dx. G

Observe that x ∈ F , h ∈ H , and xh ∈ F implies that h belongs to the compact set (F −1 F ) ∩ H . So, for each x ∈ F , ;3 ; ; ; 1/2 ; fα (x) σ = ; σ (h)α(xh)δH,G (h) dh; ; ≤ C0 sup α(y) σ (F −1 F )∩H

where

σ

3 C0 =

(F −1 F )∩H

δH,G (h)1/2 dh.

y∈G

234

Unitary representations

Hence 3 fα 2π

3

=

η(x) fα (x) dx = η(x) fα (x) F 3 ≤ C02 η(x)dx (sup α(y) σ )2 . 2 σ

G

F

2 σ

dx

y∈G

Thus fα π ≤ CK supy∈G α(y) lemma is verified.

σ

where CK = C0

7 F

η(x)dx

1/2

, and the

Let us say that a sequence of functions (αn ) in Cc (G,Hσ ) converges to α if there is a compact subset K of G such that α ∈ Cc (K,Hσ ), αn ∈ Cc (K,Hσ ) for each n, and αn → α uniformly on K. For example, if G is a Lie group, then a smooth approximate identity can be used to show that Cc∞ (G) is dense in Cc (G) for this convergence. An immediate consequence of Lemma 3.3.15 is the following. Corollary 3.3.16 Let C be a dense subset of Cc (G,Hσ ) for the preceding definition of convergence. Then Pσ (C) is dense in the space H(G,H,σ ). For v ∈ Hσ and φ ∈ Cc (G), define fφ,v : G → Hσ by fφ,v = Pσ (φ(·)v), or: 3

φ(xh)σ (h)v δH,G (h)−1/2 dh.

fφ,v (x) = H

The following construction illustrates Corollary 3.3.16. Lemma 3.3.17 Suppose that G is a connected Lie group. Let D be a dense subset of Hσ . Then (1) For each x0 ∈ G, {fφ,v (x0 ) : φ ∈ Cc∞ (G),v ∈ D} is dense in Hσ . (2) span{fφ,v : φ ∈ Cc∞ (G),v ∈ D} is dense in H(G,H,σ ). Proof: For each compact neighborhood U of e ∈ G, choose ψU ∈ Cc∞ (G) with support in U , so that 3 −1/2 ψU ≥ 0, and ψU (h)δH,G (h) dh = 1. H

Put φU (x) = ψU (x0−1 x),x ∈ G. Then for any v ∈ D, we have 3 fφU ,v (x0 ) − v =

H

φU (h)(σ (h)v − v)δH,G (h)−1/2 dh.

3.3 Induced representations

235

Given  > 0, choose U so that σ (h)v − v < ε holds for all h ∈ U ; then 3 ψU (h) σ (h)v − v δH,G (h)−1/2 dh < ε. fφU ,v (x0 ) − v ≤ H

This proves (1). As for (2), let f = fα ∈ Cc (G,H,σ ) and let ε > 0. Let K be a compact neighborhood of supp α. For each x ∈ supp α choose an open neighborhood Ux of x such that Ux is included in the interior of K, and such that y,z ∈ Ux implies α(y) − α(z) σ < ε/2. Let U1,U2, . . . ,Un be a minimal finite subcover of {Ux } for supp α. We can choose φj ∈ Cc∞ (G), 1 ≤ j ≤ n, such  that supp φj ⊂ Uj and φj ≥ 0 hold for each j , and such that j φj ≤ 1, and  j φj (x) = 1 for all x ∈ supp α. Pick xj ∈ Uj and vj ∈ D so that α(xj ) − vj σ < ε/2  / K then holds for 1 ≤ j ≤ n. Put β(x) = j φj vj . Then for any x ∈ G, if x ∈ α(x) = β(x) = 0 while if x ∈ K, ; ;

; ; ; α(x) − β(x) σ = ; φ (x)α(x) − φ (x)v j j j; ; j

σ

j

; ;

; ; ; =; φj (x)(α(x) − α(xj )) + φj (x)(α(xj ) − vj ); ; j

≤

j

φj (x) α(x) − α(xj )

σ

+

j

<

j : x∈Uj

j

φj (x) α(x) − α(xj )

σ

+

φj (x) α(xj ) − vj

σ

σ

φj (x) /2 ≤ ε.

j

Now, using Lemma 3.3.15, there is CK > 0 such that ; ;

; ; ;f − ; f φj ,vj ; = Pσ (α − β) ≤ CK ε, ; j

and the proof is complete. Remark 3.3.18 The reader can easily verify that if G is any locally compact, second countable group, then Lemma 3.3.17 holds when Cc∞ (G) is replaced by Cc (G). We choose to restrict ourselves to Cc∞ (G) here, just because we shall need these results in this setting in Chapter 5. Here are some general results on induced representations. Proposition 3.3.19 Let H be a closed subgroup of G, and let σ1 and σ2 be unitary representations of H .

236

Unitary representations

(1) If σ1 and σ2 are unitarily equivalent, then ind(G,H,σ1 ) and ind(G,H,σ2 ) are unitarily equivalent. (2) The representation ind(G,H,σ1 ⊕ σ2 ) is equivalent to ind(G,H,σ1 ) ⊕ ind(G,H,σ2 ). Proof: (1) Suppose σ1 and σ2 are equivalent. Let S : Hσ1 → Hσ2 be a unitary intertwining operator for σ1 and σ2 : Sσ1 (h) = σ2 (h)S, for each h ∈ H . For each f ∈ Cc (G,H,σ1 ), define Tf as (Tf )(x) = S(f (x)). It is easy to verify that Tf ∈ Cc (G,H,σ2 ), and if πi = ind(G,H,σi ), T π1 (x) = π2 (x)T . Now observe that T is surjective and preserves the norm, and thus can be extended to a unitary intertwining operator between π1 and π2 : indeed Sf (x) 2σ2 = f (x) 2σ1 , for each x ∈ G. Thus: 3 3 Tf 2 = Tf (x) 2σ2 dμG,H (x) = f (x) 2σ1 dμG,H (x) = f 2 . G/H

G/H

(2) Consider the representation σ = σ1 ⊕σ2 of H on Hσ = Hσ1 ⊕Hσ2 . Remark that each f ∈ Cc (G,H,σ ) can be written as f (x) = f1 (x) + f2 (x), with fi (x) ∈ Hσi ; thus, Cc (G,H,σ ) is isomorphic to Cc (G,H,σ1 ) ⊕ Cc (G,H,σ2 ) in a canonical way. Since for each fi ∈ Cc (G,H,σi ), f1 (x),f2 (x)σ = 0, then the sum is orthogonal, and by completion, H(G,H,σ )  H(G,H,σ1 ) ⊕ H(G,H,σ2 ). Clearly, if πi = ind(G,H,σi ), π = ind(G,H,σ ), then π(x)  π1 (x) ⊕ π2 (x) for each x ∈ G. Point (2) of the preceding proposition can be generalized to direct integral: 7⊕ σα dν(α), and πα7= ind(G,H,σα ) for each if σ is a direct integral: σ = ⊕ πα dν(α). The proof of α, then π = ind(G,H,σ ) is unitarily equivalent to this point is omitted. Proposition 3.3.20 Let Hi be a closed subgroup of Gi , and σi a unitary representations of Hi (i = 1,2). Put πi = ind(Gi ,Hi ,σi ). Then the representation π = π1 × π2 of G = G1 × G2 on Hπ = Hπ1 ⊗ Hπ2 is equivalent to ind(G1 × G2,H1 × H2,σ1 × σ2 ). Putting H = H1 ×H2 , G = G1 ×G2 , and π˜ i (x1,x2 ) = πi (xi ), σ˜ i (h1,h2 ) = σi (hi ), this means π˜ 1 ⊗ π˜ 2 is equivalent to ind(G,H, σ˜ 1 ⊗ σ˜ 2 ). Proof: Put G = G1 × G2 , H = H1 × H2 ; then G/H is canonically isomorphic to (G1 /H1 ) × (G2 /H2 ), through the projection map q(x1,x2 ) = (q1 (x1 ),q2 (x2 )). For fi ∈ Cc (Gi ,Hi ,σi ), define T (f1 ⊗f2 )(x1,x2 ) = f1 (x1 )⊗

3.3 Induced representations

237

f2 (x2 ). This is a continuous function f such that q(supp f ) ⊂ q1 (supp f1 ) × q2 (supp f2 ) is compact. Moreover f (x1 h1,x2 h2 ) = δH1,G1 (x1 )1/2 δH2,G2 (x2 )1/2 (σ1 (h1 )−1 f1 (x1 )) ⊗ (σ2 (h2 )−1 f2 (x2 )) = δH,G (x1,x2 )1/2 σ (h1,h2 )−1 (f (x1,x2 )), if σ = σ1 × σ2 . Thus T : Cc (G1,H1,σ1 ) ⊗ Cc (G2,H2,σ2 ) → Cc (G1 × G2, H1 × H2,σ ), and it is clearly an intertwining operator between the restriction of π1 × π2 and π to these spaces. Observe that with the notation used in Relation (3.3.2), for each φi ∈ Cc (Gi ), if φ(x1,x2 ) = φ1 (x1 )φ2 (x2 ), PδH1,G1 φ1 (x1 )PδH2,G2 φ2 (x2 ) = PδH,G (φ)(x1,x2 ), and

3

3

PδH1,G1 φ1 (x1 ) dμG1 /H1 (x1 ) PδH2,G2 φ2 (x2 ) dμG2 /H2 (x2 ) G2 /H2 3 = PδH,G φ(x1,x2 ) dμG/H (x1,x2 ).

G1 /H1

G/H

Therefore T preserves the norm: 3 Tf (x1,x2 ) 2σ dμG/H (x1,x2 ) Tf 2π = G/H 3 3 = f1 (x1 ) 2σ1 dμG1 /H1 (x1 ) = =

G1 /H1 f1 2π1 f2 2π2 f 2π1 ×π2 .

G1 /H1

f2 (x2 )

2 σ2

dμG2 /H2 (x2 )

Thus T can be extended into an isometric operator, still denoted by T : Hπ1 ⊗ Hπ2 → Hπ . Finally, fix orthonormal basis (f1,i )i in Hπ1 , (f2,j )j in Hπ2 such that each f1,i belongs to Cc (G1,H1,σ1 ), each f2,j belongs to Cc (G2,H2,σ2 ), then (f1i ⊗ f2j )i,j is an orthonormal basis in Hπ1 ⊗ Hπ2 , and is in the image of T . This proves that T is surjective, thus unitary. If H1 = G1 , and if π˜ 1 (x1,x2 ) = π1 (x1 ), π˜ 2 (x1,x2 ) = π2 (x2 ) are the trivial extensions of π1 , π2 to G1 ×G2 , then it is immediate from Example 3.3.13 and Proposition 3.3.20 that the representation π˜ 1 ⊗ π˜ 2 is equivalent to ind(G1 ×G2, G1 × H2, π˜ 1 |G1 ×H2 ⊗ σ˜ 2 ), via the operator: T (v ⊗ f )(x1,x2 ) = π1−1 (x1 )v ⊗ f (x2 )

(v ∈ Hπ1 , f ∈ H(G2,H2,σ2 )).

238

Unitary representations

The following more general situation arises in Chapter 5. Suppose that G is a semidirect product G1  G2 and that the representation π1 of G1 can be extended to G as π1ext , acting on Hπ1 as π1ext (x1,x2 )v = π1 (x1 )C(x2 )v, so that C(x2 )π1 (x1 )C(x2−1 ) = π1 (x2 x1 x2−1 ). Suppose that H2 is a closed subgroup of G2 and σ2 a unitary representation of H2 , extended by σ˜ 2 (x1,h2 ) = σ2 (h2 ) to G1  H2 . Lemma 3.3.21 Let π2 = ind(G2,H2,σ2 ) and π˜ 2 (x1,x2 ) = π2 (x2 ). Then the relation: T (v ⊗ f )(x1,x2 ) = π1ext (x1,x2 )−1 v ⊗ f (x2 )

(v ∈ Hπ1 , f ∈ H(G2,H2,σ2 ))

defines a unitary intertwining operator T between π1ext ⊗ π˜ 2 and ind(G,G1  H2,π1ext |G1 H2 ⊗ σ˜ 2 ). Proof: Observe first that δG1 H2,G1 G2 (y1,h2 ) = δH2,G2 (h2 ). Suppose f ∈ Cc (G2,H2,σ2 ), identifying G/G1  H2 and G2 /H2 , if q : G → G/G1  H2 , q(supp T (v ⊗ f )) = q2 (supp f ) is compact and (T (v ⊗ f ))((x1,x2 )(y1,h2 )) = π1ext ((x1,x2 )(y1,h2 ))−1 v ⊗ f (x2 h2 ) = π1ext (y1,h2 )−1 π1ext (x1,x2 )−1 v ⊗ δH2,G2 (h2 )1/2 σ˜ 2 (y1,h2 )−1 f (x2 ) = δG H2,G1 G2 (y1,h2 )1/2 (π1ext ⊗ σ˜ 2 )(y1,h2 )−1 (T (v ⊗ f ))(x1,x2 ). Thus T (v ⊗ f ) ∈ Cc (G,G1  H2,π1ext ⊗ σ˜ 2 ). Now for any (x1,x2 ), (y1,y2 ) in G, T (v ⊗ f )((y1,y2 )−1 (x1,x2 ))  −1 = π1ext (y1,y2 )−1 (x1,x2 ) v ⊗ f (y2−1 x2 )     = π1ext ((x1,x2 ))−1 π1ext (y1,y2 )v ⊗ π˜ 2 (y1,y2 )f (x2 ) = T ((π1ext ⊗ π˜ 2 )(y1,y2 )(v ⊗ f ))(x1,x2 ). Thus T is an intertwining operator. The proof of Proposition 3.3.20 applies here: it is unitary. In Example 3.3.14, we see that μG,H is a Borel measure on G/H . In general, this is the case when δH,G = 1. Even when δH,G = 1, it is possible to realize μG,H using a quasi-invariant Borel measure on G/H . Lemma 3.3.22 Let G be a locally compact, second countable group and H a closed subgroup of G. There is a continuous function ρ : G → (0, + ∞) satisfying

3.3 Induced representations

239

ρ(xh) = δH,G (h)ρ(x), x ∈ G,h ∈ H . Proof: Let g be a continuous, nonnegative, compactly supported function on G such that g(e) > 0 and g(x −1 ) = g(x), and let V = {x : g(x) > 0}. V is a balanced neighborhood of e. By Zorn’s Lemma, there is a maximal subset A in G such that for each a, b in A, a = b implies a ∈ / V bH . Using the maximality of A, it is easy to prove that V A intersects each coset xH (x ∈ G). Moreover for any compact set K ⊂ G, {a ∈ A : KH ∩ V a = ∅} is finite. To see this, suppose the contrary, there is an infinite sequence of distinct elements (a1,a2, . . .) in A, and a sequence (h1,h2, . . .) in H such that aj hj ∈ V K. Since V K is compact, the sequence (aj hj ) has a cluster point x. Pick now a balanced neighborhood W of e such that W W ⊂ V , we have infinitely many −1 j such that aj hj belongs to W x, so for two such points, v = aj hj h−1 j  aj  ∈ W W ⊂ V , and aj ∈ V aj  H , which is impossible. Thanks to this set A, we define a nonnegative continuous function f by

g(xa −1 ). f (x) = a∈A

Observe that for each compact set K, f |KH is a finite sum of translates of g:

f (x) = g(xa −1 ) = g(xa −1 ) a∈A: KH ∩V a=∅

a∈A

thus f is continuous and f |KH has compact support. Moreover each coset xH intersects the support of f . Define now the ρ function by: 3 δH,G (h)−1 f (xh) dh. ρ(x) = H

It is easy to prove that ρ satisfies the conditions of the lemma. A function satisfying the conditions of Lemma 3.3.22 will be called a rhofunction. Lemma 3.3.23 Let ρ be a rho-function and φ ∈ Cc (G). If P φ = 0, then 7 φ(x)ρ(x)dx = 0. G Proof: Using relation (3.3.3), we get: 3 3 3 −1 φ(x)ρ(x) dx = φ(xh)ρ(xh)δH,G (h) dh dμG,H G G/H H 3 3 = φ(xh) dh ρ(x) dμG,H = 0. G/H

H

240

Unitary representations

Now fix a rho-function on G, then the mapping φ → φρ is clearly a linear bijection on the space Cc (G). A simple computation shows that for φ ∈ Cc (G), Pδ (φρ) = (P φ)ρ. Now g → gρ is a bijection from Cc (G/H ) to Cc (G,H,δ). Define a positive form νρ on Cc (G/H ) by: νρ (g) = μG,H (gρ). Fix a compact subset F of G/H ; by Lemma 3.3.3, there is a compact subset C in G, and a positive constant CF such that for any continuous function g on G/H with support in F , gρ can be written as Pδ (φρ) with supp φ ⊂ C and sup |φ(x)ρ(x)| ≤ CF sup |g(x)ρ(x)|. This implies: #3 # # # # |νρ (g)| = |μG,H (gρ)| = # φ(x)ρ(x) dμG (x)## G

≤ sup |φ(x)ρ(x)|μG (C) ≤ CF sup |g(x)|, where CF = μG (C)CF supx∈C |ρ(x)|. Hence, by the Riesz representation theorem, νρ is a measure on the locally compact set G/H . Observe that this can be written 3 3 g(xH ) dνρ (xH ) = g(x)ρ(x) dμG,H (x), G/H

G/H

and for each f ∈ Cc (G,H,δ),

3

μG,H (f ) =

f (x)ρ(x)−1 dνρ (xH ).

(3.3.5)

G/H

The first application of the measure νρ is the following extension: Lemma 3.3.24 The formula (3.3.5) allows to define an extension of the linear form μG,H , still denoted by μG,H , to the set of positive Borel functions f on G such that f (xh) = δ(h)f (x), h ∈ H . This extension does not depend on the choice of the rho-function ρ. Proof: Let ρ be a rho-function and f a positive Borel function such that f (xh) = δ(h)f (x), h ∈ H7. Then fρ −1 is positive and Borel, and we define μG,H (f ) by μG,H (f ) = G/H f (x)ρ(x)−1 dνρ (xH ). We need only show that this definition is independent of the choice of ρ. If ρ  is another rhofunction, then θ = ρ/ρ  is a positive Borel function such that θ (xh) = θ (x), and it is easy to check that dνρ  = θ dνρ . Hence, 3 3  −1 f (x)ρ (x) dνρ  (xH ) = f (x)ρ  (x)−1 θ (x) dνρ (xH ) G/H G/H 3 = f (x)ρ(x)−1 dνρ (xH ). G/H

3.3 Induced representations

241

The measure νρ can also be used to present an alternate realization of induced representation. We begin with an example. Example 3.3.25 Let V be a vector space over R of dimension n ∈ N, H a closed subgroup of GL(V ), and put G = V  H . Write an element x ∈ G as x = (v,h), v ∈ V , h ∈ H . Fix a Haar measure dh on H , and via a choice of basis for V , a Lebesgue measure dv on V . Define the Borel measure μG on G by 3 3 3 φ dμG = φ(v,h)| det(h)|−1 dhdv. G

V

H

It is easy to check that μG is left-invariant. Similarly H (h)−1 dhdv is rightinvariant. But a right-invariant measure is also given by G (v,h)−1 dμG (v,h), so for some c > 0, H (h)−1 dhdv = cG (v,h)−1 | det(h)|−1 dhdv. Now it is clear that G (v,e) = 1 for all v ∈ V . It follows that | det(h)| =

H (h) = δH,G (h). G (h)

Write δ = δH,G . Note this agrees with our formula above: δ(exp Y ) = |etrace adg/h Y |. From (3.2.2) we conclude that μG,H is given by 3 g(v,e) dv, g ∈ Cc (G,H,δ). μG,H (g) = V

The preceding integral is left-invariant as a form on Cc (G,H,δ), but as a form on Cc (G/H ) = Cc (V ) it is only semi-invariant: 3 3 −1 g((v0,h0 )(v,e))dv = δ(h0 ) g(v,e)dv. V

V

Let s : G/H → G be defined by s((v,h)H ) = s((v,e)H ) = (v,e); then s is a continuous section. Observe that for any x = (v,h), x = (v,e)(0,h), and a rhofunction can be defined by ρ(v,h) = δ(h). Observe that ρ(v,e) = 1, therefore the reader can verify that the measure dνρ associated to ρ on V identified with G/H is simply dv. We now realize the representation π = ind (G,H,σ ) on the space L2 (V ,Hσ ,dv) as follows: each f ∈ Cc (G,H,σ ) is determined by its restriction to V × {e}: f (v,h) = f ((v,e)(0,h)) = δ(h)1/2 σ (h)−1 f (v,e). Put F (v) = Sf (v) = f (v,e), and verify that: 3 F (v) f 2= V

2

dv.

242

Unitary representations

Hence S extends to a unitary operator S : H(G,H,σ ) → L2 (V ,Hσ ,dv) and τ = Sπ S −1 is given by τ (v,h)F (w) = σ (h)F (h−1 (w − v))δ(h)−1/2 .

(3.3.6)

If σ is the trivial representation, this is formula (3.1.3) of Example 3.1.9, describing the quasi-regular representation of G.  To generalize the preceding example, let H be a closed subgroup of G, fix left Haar measures on G and H , with the corresponding form μG,H . Let σ be a unitary representation of H , and put C˜ c (G,H,σ ) = {g ∈ C(G,Hσ ) : q(supp g) is compact, g(xh) = σ (h)−1 g(x),x ∈ G,h ∈ H }. For g ∈ C˜ c (G,H,σ ), x → g(x) 2 belongs to Cc (G/H ). Now let ρ be a rho-function on G and let Hρ (G,H,σ ) be the completion of C˜ c (G,H,σ ) with respect to the norm 3 g(x) 2 dνρ (xH ). g 2= G/H

The natural bijection U from Cc (G,H,σ ) to C˜ c (G,H,σ ) defined by f → fρ −1/2 satisfies 3 3 2 2 −1 f = f (x) ρ(x) dνρ (xH ) = Uf (x) 2 dνρ (xH ) G/H

G/H

and so extends to a unitary isomorphism U : H(G,H,σ ) → Hρ (G,H,σ ). Thus π is equivalent with πρ defined by πρ (x) = U π(x)U −1 acting in Hρ (G,H,σ ) by  1/2 ρ(y −1 x) . (3.3.7) (πρ (y)g)(x) = g(y −1 x) ρ(x) Suppose that G is simply connected and solvable, and that H is a connected subgroup with Lie algebra h. Choose a Jordan–Hölder sequence with corresponding root sequence (λj ) and let j be the character of G such that dj = λj . Recall the index set e(h), defined in Lemma 3.3.4. Define 4 j (x). (3.3.8) ρ(x) = j ∈e(h)

Then ρ is continuous, positive, and ρ(xy) = ρ(x)ρ(y) for all x,y, ∈ G. By Lemma 3.3.4, ρ(xh) = ρ(x)δH,G (h). Thus ρ is a rho-function on G. Since ρ is multiplicative, we get (πρ (y)g)(x) = g(y −1 x)ρ(y)−1/2 .

(3.3.9)

3.3 Induced representations

243

The preceding is somewhat more concrete when a Borel section s for G/H is chosen: s is a Borel measurable map s : G/H → G such that q(s(xH )) = xH , x ∈ G. In fact for each locally compact group G, and any closed subgroup H , it can be proved that such a section exists. Let s be a Borel section and let θ : G → H be the Borel map θ (x) = s(xH )−1 x. We may assume that s(eH ) = e, so θ (e) = e, θ (xh) = θ (x)h for x ∈ G, h ∈ H , and since s(s(xH )H ) = s(xH ), then θ (s(G/H )) = {e}. We also have the relation θ (y −1 x) = θ (y −1 s(xH ))θ (x). Now consider the space Bc (G,H,σ ) of all bounded Borel measurable Hσ -valued functions f on G such that q(supp f ) is compact, and such that f (xh) = δ(h)1/2 σ (h)−1 f (x), x ∈ G, h ∈ H . Similarly, let Bc (G,H,δ) be the space of bounded Borel measurable C-valued functions f on G such that q(supp f ) is compact, and such that f (xh) = δ(h)f (x), x ∈ G, h ∈ H . We have a natural bijection S : Bc (G,H,σ ) → Bc (G/H,Hσ ), where Bc (G/H,Hσ ) is the space of bounded Borel measurable functions F : G/H → Hσ having compact support. S is given by Sf (x) = f (s(xH )) with (S −1 F )(x) = δ(θ(x))1/2 σ (θ (x))−1 F (x). (Recall that we identify functions on G/H with functions on G that are constant on H -cosets.) Now for g ∈ Cc (G/H ) we have x → g(x)δ(θ (x)) belongs to Bc (G,H,δ). Observe that the function ρ(x) ˜ = δ(θ(x)) satisfies all the conditions to be a rhofunction, except the continuity property. Hence, using the argument presented after the proof of Lemma 3.3.23, we see that μG,H defines a measure ν on G/H given by 3 3 g(x)dν(xH ) = g(x)δ(θ (x))dμG,H (x). G/H

G/H

The measure ν is quasi-invariant, meaning that each translate of ν is equivalent with ν: for each y ∈ G, 3 3 g(yx)dν(xH ) = g(yx)δ(θ (x))dμG,H (x) G/H G/H 3 = g(x)δ(θ (y −1 x))dμG,H (x) G/H 3 = g(x) δ(θ (y −1 s(xH )))δ(θ (x))dμG,H (x) G/H 3 = g(x) δ(θ (y −1 s(xH )))dν(xH ). G/H

244

3

Unitary representations

Observe also that for φ ∈ Cc (G), 3 3 3 φ dμG = Pδ φ(x) dμG,H (x) = φ(xh)δ(h)−1 dh dμG,H (x) G G/H G/H H 3 3 = φ(xh)δ(h)−1 dh δ(θ (x))−1 dν(xH ) G/H H 3 3 = φ(s(xH )θ (x)h)δ(h)−1 dh δ(θ (x))−1 dν(xH ) G/H H 3 3 = φ(s(xH )h)δ(h)−1 dh dν(xH ) G/H H 3 = Pδ φ(s(xH ))dν(xH ). G/H

For f ∈ Cc (G,H,σ ) we have 3 3 2 2 f (x) σ dμG,H (x) = f = G/H

G/H

Sf (x)

2 σ dν(xH ).

It is clear that H(G,H,σ ) is the completion of Bc (G,H,σ ) so S extends to unitary isomorphism of H(G,H,σ ) with the completion H(G/H,Hσ ) of Bc (G/H,Hσ ) with respect to the norm above. One computes easily that for each x,y ∈ G and F ∈ Bc (G/H,Hσ ), Sπ(y)S −1 F (x) = (S −1 F )(y −1 s(xH )) = δ(θ (y −1 s(xH )))1/2 σ (θ (y −1 s(xH )))−1 F (y −1 s(xH )) = δ(θ (y −1 s(xH )))1/2 σ (θ (y −1 s(xH )))−1 F (y −1 x). Thus we have the following. Proposition 3.3.26 Let π = ind (G,H,σ ), and assume that there is a Borel section s : G/H → G with θ (x) = s(xH )−1 x. Then π is equivalent with the representation πs acting in H(G/H,Hσ ) by πs (y)F (x) = σ (θ (y −1 s(xH )))−1 F (y −1 x) δ(θ(y −1 s(xH ))))1/2 . (3.3.10) Finally, it is easily seen that if the section s is continuous, then ρ(x) = δ(θ(x)) = δ(s(xH )−1 x), x ∈ G

(3.3.11)

defines a rho-function on G, and proposition 3.3.26 gives a concrete realization of πρ for the rho-function of (3.3.11). Recall that if G is simply connected solvable, and H is connected, then there is always a continuous section given by a coexponential basis, but the resulting rho-function is not generally the same as that of (3.3.8). Revisiting Example 3.3.25, with s(v,h) = (v,e), we

3.3 Induced representations

245

have θ (v,h) = h and the realization (3.3.10) becomes the realization (3.3.6) shown in the example. Example 3.3.27 Referring to Example 3.3.14, we identified G/H to R by using s(q(exp xX)) = (x,0,0), thus θ(x,y,z) = (0,y,z − xy), and since δH,G = 1, the presentation in Example 3.3.14 of the representation πγ ,λ is the formula given in Proposition 3.3.26:   πγ ,λ (x1,y1,z1 )f (x) = χγ ,λ (θ ((x1,y1,z1 )−1 s(xH )))−1 f (q((x1,y1,z1 )−1 x)) = χγ ,λ ((0, − y1, − z1 + xy1 ))−1 f (x − x1 ) = eiλz1 ei(γ −λx)y1 f (x − x1 ).



Example 3.3.28 Recall the solvable Lie group E˜ 2 described in Example 1.7.14:   G = E˜ 2 = (z,t) : t ∈ R, z ∈ C with (z,t)(z,t  ) = (z + eit z,t + t  ). G is the semidirect product of the abelian Lie group N = C = {(z,0)} by the one-dimensional group R = {(0,t)}. Consider the nonconnected subgroup H = C  2πZ = {(z,2π k) : k ∈ Z} and the character: χr,τ (z,2π k) = eiRe(zr) e2iπ τ k , with r > 0 and 0 ≤ τ < 1. Clearly G/H = R/2πZ and, if s is a section s : G/H → G then s(u) = (z(u),t (u)), where u → t (u) is a section of the quotient R → R/2πZ, since there is no continuous such section, this proves there are no continuous section s. Observe that δH,G = 1. Choose the following Borel section: denote here  [x] the integral part of t 2π ). With this choice, the the real number x, and put s(q(z,t)) = (0,t − 2π formula in Proposition 3.3.26 for the representation πr,τ = ind (G,H,χr,τ ) is:  t−t 

t   1 −it 2iπ τ 2π 2π − 2π 2π f (t − t1 +2π Z). πr,τ (z1,t1 )f (t + 2πZ) = eiRe(z1 e r) e 

3.3.3 Induction in stages The following standard result, known as the induction in stages formula, is an essential tool for the construction of irreducible unitary representations of solvable Lie groups. Proposition 3.3.29 Let K ⊂ H ⊂ G be closed subgroups of G, let χ be a unitary representation of K, σ = ind(H,K,χ ), τ = ind(G,H,σ ), and π = ind(G,K,χ ). Then τ  π.

246

Unitary representations

Proof: For F ∈ Cc (G,K,χ ), put −1/2

fF (x)(h) = F (xh)δH,G (h), x ∈ G,h ∈ H . For each x ∈ G, fF (x) is continuous on H and supp fF (x) ⊂ x −1 supp F ∩H . It is clear that fF (x)(hk) = χ (k)−1 fF (x)(h)δK,H (k)1/2, so fF maps G into Cc (H,K,χ ) ⊂ H(H,K,χ ). It is also easy to check, using the definition of σ , that for each x ∈ G, h0 ∈ H , fF (xh0 ) = σ (h0 )−1 (fF (x)) δH,G (h0 )1/2 . Now to prove the continuity of fF : G → H(H,K,χ ), let α ∈ Cc (G,Hχ ) such that F = Fα = Pχ (α); then 3 Fα (xh) = K

χ (k)α(xhk)δK,G (k)−1/2 dk = FLx −1 α (h),

and 3 fFα (x) =

K

  χ (k) Lx −1 α(hk)δH,G (hk)−1/2 δK,H (k)−1/2 dk.

If x and x0 are in G, and βx = Lx −1 α − Lx −1 α, this means: 0

3 fFα (x) − fFα (x0 ) =

K

  −1/2 χ (k) βx δH,G (hk)δK,H (k)−1/2 dk.

Now α is continuous with a compact support S, so it is uniformly continuous: for each x0 , there is a compact neighborhood U of x0 such that x ∈ U implies: sup α(xh)− α(x0 h)

χ

h∈H

= sup

h∈SU −1

α(xh)− α(x0 h)

χ

= sup

−1/2

h∈SU −1

βx (h)

χ

< ε.

−1/2

Thus suph∈SU −1 δH,G (h)βx (h) χ < C1 ε, with C1 = suph∈SU −1 |δH,G (h)|. By Lemma 3.3.15, there is C = CSU −1 > 0 such that x ∈ U implies: fFα (x) − fFα (x0 )

σ

< CC1 ε.

This proves the continuity of fF for each F ∈ Cc (G,K,χ ). Finally, fF (x) = 0 implies x −1 supp F ∩ H = ∅ which means xH ∩ supp F = ∅, that is, qG,H (supp fF ) ⊂ qG,H (supp F ) which is compact by Lemma 3.3.7. Thus fF ∈ Cc (G,H,σ ).

3.3 Induced representations

247

Now x → F (x) 2 belongs to Cc (G,H,δH,G ), so by Proposition 3.3.9 we have 3 2 F (x) 2χ dμK,G (x) F π = G/K 3 3 = F (xh) 2χ δH,G (h)−1 dμK,H (h) dμH,G (x) G/H H /K 3 3 = fF (x)(h) 2χ dμK,H (h) dμH,G (x) G/H H /K 3 = fF (x) 2σ dμH,G (x) G/H

= fF

2 τ

so that F → fF is an isometry. It is immediate that for any x0 ∈ G, τ (x0 )fF = fπ(x0 )F . In order to see that τ and π are unitarily equivalent, it remains to see that the image of Cc (G,K,χ ) under F → fF is dense in Cc (G,H,σ ). Let D = {vα : α ∈ Cc (H,K,χ )} where 3 χ (k)α(hk)δK,H (k)−1/2 dk. vα (h) = K

By definition, D is a dense subset of H(H,K,χ ). By Lemma 3.3.17, if G is a Lie group, the set {fφ,v : φ ∈ Cc∞ (G), v ∈ D} (or by Remark 3.3.18 if G is not a Lie group, the set {fφ,v : φ ∈ Cc (G), v ∈ D}) is dense in H(G,H,σ ), where 3 φ(h)σ (h)v δH,G (h)−1/2 dh. fφ,v = H

We show that each fφ,v is of the form fF . Fix v ∈ D and φ ∈ Cc∞ (G) (resp. φ ∈ Cc (G)). Define F : G → Hχ by 3 φ(xh)v(h−1 )δH,G (h)−1/2 dh. F (x) = H

Let v = vα as above, and put 3 φ(xh)α(h−1 )δH,G (h)−1/2 dh. β(x) = H

It is easy to see that β is a continuous Hχ -valued function of G and that supp β is compact. A straightforward calculation, using Fubini’s Theorem (which applies since the integrand is compactly supported) and the fact that δK,H (k)−1/2 δH,G (k)−1/2 = δK,G (k)−1/2 , shows that 3 F (x) = Fβ (x) = χ (k)β(xk)δK,G (k)−1/2 dk. K

248

Unitary representations

Hence F belongs to Cc (G,K,χ ). Now for each x0 ∈ G, h0 ∈ H , fF (x0 )(h0 ) = F (x0 h0 )δH,G (h0 )−1/2 3 = φ(x0 h0 h)v(h−1 )δH,G (h)−1/2 dh δH,G (h0 )−1/2 H 3 = φ(x0 h)v(h−1 h0 )δH,G (h)−1/2 dh H 3 φ(x0 h)(σ (h)v)(h0 )δH,G (h)−1/2 dh = H

= fφ,v (x0 )(h0 ). This means that the image of the map F → fF contains the dense subset generated by the fφ,v . This completes the proof.

3.4 Elements of Mackey theory 3.4.1 System of imprimitivity The exposition here draws from the presentation of Folland in [40]. Definition 3.4.1 Let G be a locally compact group. A system of imprimitivity on G is an ordered triple  = (π,S,P ) where (a) π is a unitary representation of G on a Hilbert space Hπ , (b) S is a G-space, that is, S is a locally compact, Hausdorff space with a continuous left action of G, (c) P is a regular Hπ -projection valued measure on S such that: π(x)P (E)π(x −1 ) = P (xE), for any x ∈ G and E ∈ B(S). Let  = (π,S,P ) and   = (π ,S,P  ) be systems of imprimitivity on G, with the same G-space S. We say that  and   are equivalent if there is a unitary operator U : Hπ → Hπ  such that: U π(x) = π  (x)U,

and

U P (E) = P  (E)U,

for any x ∈ G and any E ∈ B(S). If the G-action on S is transitive, we say that  is a transitive system of imprimitivity.

3.4 Elements of Mackey theory

249

Let  = (π,S,P ) be a system of imprimitivity on G. The G-action on S defines an action both on Cc (S), and C0 (S) by (Lx φ)(y) = φ(x −1 y). Here C0 (S) is the C ∗ -algebra of continuous functions on S tending to zero at infinity, with conjugation and the sup norm. Recall that the projection valued measure P allows to define operators M(φ) acting on Hπ as follows: For each pair (f ,g) of vectors in Hπ , the map E → P (E)f ,g is a complex measure denoted by dPs f ,g, and the relation: 3 M(φ)f ,g = φ(s) dPs f ,g S

defines an operator M(φ) acting on Hπ . It is easy to see that the map M : C0 (S) → L(Hπ ) is a representation of the C ∗ -algebra C0 (S). Since  satisfies the relation in Definition 3.4.1, then by change of variable: 3 π(x)M(φ)π(x −1 )f ,g = φ(s)dPxs f ,g 3S = φ(x −1 s)dPs f ,g = M(Lx φ)f ,g. S

Conversely, if M is a representation of the commutative C ∗ algebra C0 (S) in a Hilbert space H, for each f , g in H, the formula: φ → M(φ)f ,g defines a Radon measure μf ,g on S, and for each Borel set E, the measure μf ,g (E) can be written as P (E)f ,g, where P (E) is an orthogonal projection, and P : E → P (E) is a projection valued measure on S (for the details, see, for example, Mackey [64], or Folland [40]). Moreover, if the relation π(x)M(φ)π(x −1 ) = M(Lx φ) holds for each φ ∈ C0 (S), each x ∈ G then π(x)P (E)π(x −1 ) = P (xE) for each E ∈ B(S). Therefore, a system of imprimitivity on G can be defined as a triple (π,S,M) where S is a G-space and π is a representation of G, M a representation of C0 (S), both in the same Hilbert space H, such that, for each φ ∈ C0 (S) and each x in G, π(x)M(φ)π(x −1 ) = M(Lx φ). The commutant C() of the system of imprimitivity  = (π,S,M) is the algebra of operators on Hπ commuting with all operators π(x) and M(φ). By definition, C() is a C ∗ -algebra. Let now H be a closed subgroup of G, q : G → G/H be the canonical projection, and σ a unitary representation of H on the Hilbert space Hσ .

250

Unitary representations

Consider the induced representation π = ind(G,H,σ ) acting on H(G,H,σ ), the completion of the space Cc (G,H,σ ) of continuous Hσ -valued functions f on G, such that q(supp f ) is compact in G/H , and such that for each x ∈ G and h ∈ H , f (xh) = δH,G (h)1/2 σ (h−1 )f (x). To this induced representation π is associated a canonical system of imprimitivity  = (π,G/H,M) on the G-space S = G/H , namely M is defined by: (M(φ)f )(x) = φ(q(x))f (x). Indeed M(φ) is clearly a well-defined bounded operator acting on H(G,H,σ ), M is a representation of C0 (G/H ), and for each y and x (π(x)M(φ)π(x −1 )f )(y) = φ(q(x −1 y))(π(x −1 )f )(x −1 y) = φ(x −1 q(y))f (y) = (M(Lx φ))f (y). In this case, the projection-valued measure is the map E → P (E), where P (E)f (x) = χE (q(x))f (x), where χE is the characteristic function of the Borel set E. Suppose π = ind(G,H,σ ), and  = (π,G/H,σ ) is the associated system of imprimitivity on G. Then for each T ∈ C(σ ), we define the operator T˜ on H(G,H,σ ) as follows: first if f ∈ Cc (G,H,σ ) we put: (T˜ f )(x) = T (f (x)), for any x. Observe that T˜ f is in Cc (G,H,σ ) since (T˜ f )(xh) = T (f (xh)) = δH,G (h)1/2 T (σ (h−1 )f (x)) = δH,G (h)1/2 σ (h−1 )(T˜ f )(x), moreover (T˜ f )(x) σ ≤ T f (x) σ , thus T˜ has a unique extension, still denoted T˜ to the space H(G,h,σ ) of the representation π . Clearly, T + T  = T˜ + T˜ , T< T  = T˜ T˜ , T=∗ = T˜ ∗, and T˜ ≤ T . Now T˜ is in C(). Indeed, for f ∈ Cc (G,H,σ ), (π(x)T˜ f )(y) = (T˜ f )(x −1 y) = T (f (x −1 y)) = (T˜ (π(x)f ))(y), and (M(φ)T˜ f )(y) = φ(q(y))T (f (y)) = T (φ(q(y))f (y)) = (T˜ (M(φ)f ))(y).

3.4 Elements of Mackey theory

251

We say that a closed subspace N of H(G,H,σ ) is -invariant if it is invariant under the π(x) and the M(φ) (x ∈ G, φ ∈ C0 (G/H )). Lemma 3.4.2 Let N be a closed -invariant subspace of H(G,H,σ ). Then N ∩ Cc (G,H,σ ) is dense in N . Proof: Let {ψu } be an approximate identity in Cc (G), and f ∈ N , then ψu ∗f , defined by: 3 ψu ∗ f = ψu (y)π(y)f dy G

belongs to N since N is closed and π(y)-invariant. It is a continuous function on G, since: 3 (ψu ∗ f )(x) = ψu (xy)f (y −1 ) dy. G

Thus, any element in N can be approximated by a continuous function. Take now an increasing sequence of function φn in Cc (G/H ) such that φn (q(x)) → 1 for each x ∈ G, then, for each n and u, M(φn )(ψu ∗ f ) is in N , and in Cc (G,H,σ ) and converges, in H(G,H,σ ), to (ψu ∗ f ) as n goes to ∞. This implies that ψu ∗ f , thus f is in the closure of N ∩ Cc (G,H,σ ) and proves the lemma. Lemma 3.4.3 Let M be a closed σ -invariant subspace of Hσ . Put: ˜ = Closure of {f ∈ Cc (G,H,σ ) : f (x) ∈ M for all x ∈ G}. M ˜ is a bijection between the set of σ -invariant closed subspaces Then M → M of Hσ and the set of -invariant closed subspaces of H(G,H,σ ). ˜ is -invariant; in fact, it is the space of the Proof: It is easy to verify that M representation induced from the restriction σ |M of σ to the space M: ˜ = H(G,H,σ |M ). M ˜ is one-to-one. Let M1 = M2 be two closed Let us prove that M → M σ -invariant subspaces of Hσ , and v ∈ M1 \ M2 . By Lemma 3.3.17, given ˜ 1 such that f (e) − v σ < ε. Thus ε > 0, there is f in Cc (G,H,σ |M1 ) ⊂ M f (x) is not in M2 for each x in an open neighborhood of e in G, and f is not ˜ 2. in the space M ˜ is surjective, let N be any -invariant closed To show that M → M subspace of H(G,H,σ ), and put ' & 3 −1 −1 M = Closure of vf ,φ = φ(x )f (x ) dx : f ∈ N , φ ∈ Cc (G) .

252

Unitary representations

Since f (x −1 h−1 ) = δH,G (h)−1/2 σ (h)f (x −1 ), then: 3 σ (h)vf ,φ = φ(x −1 )δH,G (h)1/2 f (x −1 h−1 ) dx 3 = φ  (x −1 )f (x −1 ) dx = vf ,φ  , where φ  (x) = δH,G (h)1/2 φ(xh), hence M is σ -invariant. ˜ = N . If f ∈ N ∩ Cc (G,H,σ ) and w ∈ M⊥ , then for We claim that M any φ ∈ Cc (G), 3 0 = vf ,φ ,w = φ(x −1 )f (x −1 ),w dx. ˜ and by Lemma This implies f (x −1 ),w = 0 for any x. Thus, f is in M, ˜ 3.4.2, N ⊂ M. To prove the other inclusion, define, for any f ∈ N , and for φ and ψ in Cc (G), the function: 3 gf ,φ,ψ (x) = δH,G (h)−1/2 ψ(xh)σ (h)vf ,φ dh. H

By Lemma 3.3.17, the space of all these functions is dense in the space ˜ = H(G,H,σ |M ). Therefore it is enough to prove that each of the functions M gf ,φ,ψ belongs to N . We compute: 3 3 gf ,φ,ψ (x) = δH,G (h)−1/2 ψ(xh)σ (h)φ(y −1 )f (y −1 ) dy dh 3H 3G = ψ(xh)φ(y −1 )f (y −1 h−1 ) dy dh H G  3 3 −1 ψ(xh)φ(y xh) dh (π(y)f )(x) dy. = 7

G

H

Put θ (x,y) = H ψ(xh)φ(y −1 xh) dh, observe that θ (xh0,y) = θ (x,y), thus θy (q(x)) = θ (x,y) is a function in Cc (G/H ), so we get: 3 gf ,φ,ψ = M(θy )π(y)f dy. G

Since f ∈ N , and N is -invariant, then for each y, M(θy )π(y)f is in N , ˜ ⊂ N, and since N is closed, it follows that gf ,φ,ψ is also in N . By density, M and the lemma is proved. Theorem 3.4.4 Let  be the canonical system of imprimitivity for the induced representation π = ind (G,H,σ ). Then the map T → T˜ is a C ∗ -algebra isomorphism between C(σ ) and C().

3.4 Elements of Mackey theory

253

Proof: Recall that the mapping T → T˜ is a homomorphism of involutive algebras from C(σ ) into C(), such that T˜ ≤ T . Let us now prove that T˜ = T . Fix ε > 0, there is v ∈ Hσ such that v = 1 and T v σ > T − ε. Choose a such that 1 − ε < a < 1 − ε/2, by Lemma 3.3.17, there is f ∈ Cc (G,H,σ ) such that 0 < f (e)−av σ < a −(1−ε). Thus f (e)−v σ < ε and f (e) σ < 1. Now there is an open neighborhood U of e in G such that f (y) σ ≤ 1 and f (y)−f (e) σ < ε, for each y ∈ U . Then for each y in U , f (y)−v σ < 2ε, Tf (y)

σ

≥ Tv

σ

− 2 T ε ≥ T (1 − 3ε) ≥ T (1 − 3ε) f (y)

σ.

Pick a function ψ ∈ Cc (G/H ) with support included in q(U ), and put g = M(ψ)f , then for any x, if g(x) = 0, there is h ∈ H and y ∈ U such that x = yh, and T g(x)

σ

= δH,G (h)1/2 T g(y)

σ

= T (1 − 3ε) g(x)

σ.

This implies that T˜ g

π

≥ δH,G (h)1/2 T (1 − 3ε) g(y)

≥ T (1 − 3ε) g

π,

σ

it follows that T˜ = T .

Therefore T → T˜ is injective. Let us prove it is surjective. Suppose first Q is a projection in C(). This means the space N = Q(H(G,H,σ )) is -invariant, thus, by Lemma 3.4.3, there is a closed σ -invariant subspace M ˜ Let P be the orthogonal projection on M, by of Hσ such that N = M. construction: Q = P˜ . Suppose now A is a self-adjoint operator in C(), by the spectral theorem, to A is associated a projection valued measure E → QE , each projection QE is a strong limit of a sequence of polynomials in A, thus is also in C(). Thus they can be written as P˜E (E ∈ B(R)). Then, 3 3 ˜ ˜ λ d Pλ = T , with T = λ dPλ . A= R

R

If A is any operator in Cc (), then A + A∗ and i(A − A∗ ) are self-adjoint in C(), so they are in the range of the map T → T˜ . Thus A itself is in this range, and this map is surjective. We saw in Example 3.3.14 that two induced representations from the same subgroup π = ind(G,H,σ ) and π  = ind(G,H,σ  ) can be unitarily equivalent, with inequivalent inducing representations σ and σ  . A consequence of Theorem 3.4.4 is that this cannot happen if the equivalence between π and π  is an equivalence of systems of imprimitivity.

254

Unitary representations

Corollary 3.4.5 Let H be a closed subgroup of the locally compact group G, and σ1 , σ2 be two unitary representations of H , let 1 , 2 be the canonical system of imprimitivity of π1 = ind(G,H,σ1 ) and π2 = ind(G,H,σ2 ). Then σ1 and σ2 are equivalent if and only if 1 and 2 are equivalent. Proof: Suppose that 1 and 2 are equivalent by the unitary operator U : Hπ1 → Hπ2 . Recall that π = π1 ⊕ π2 is the representation induced from the representation σ1 ⊕ σ2 . Its canonical system of imprimitivity is:  = (π1 ⊕ π2,G/H,M1 ⊕ M2 ) = (π,G/H,M), with M(φ) = M1 (φ) ⊕ M2 (φ). The operator U gives an operator V : Hπ1 ⊕ Hπ2 → Hπ1 ⊕ Hπ2 , by: V (f1,f2 ) = (0,Uf1 ). By construction, V ∈ C(): V π(x)(f1,f2 ) = (0,U π1 (x)f1 ) = (0,π2 (x)Uf1 ) = π(x)V (f1,f2 ), V M(φ)(f1,f2 ) = (0,U M1 (φ)f1 ) = (0,M2 (φ)Uf1 ) = M(φ)V (f1,f2 ). Thus by Theorem 3.4.4, there is a unique T : Hσ1 ⊕ Hσ2 → Hσ1 ⊕ Hσ2 in C(σ1 ⊕ σ2 ), such that V = T˜ . ∗ T is the projection on the first But we immediately see that V ∗ V = T< factor in Hπ1 ⊕ Hπ2 , this projection is also P˜1 where P1 is the projection onto the first factor in Hσ1 ⊕ Hσ2 , which is in C(σ1 ⊕ σ2 ), thus T ∗ T (v1,v2 ) = (v1,0). This implies that T (0,v2 ) 2 = T ∗ T (0,v2 ),(0,v2 ) = 0, T (v1,v2 ) = T (v1,0). T ∗ is the projection onto the second factor, Similarly, V V ∗ = T< ∗ T T (v1,v2 ) = (0,v2 ), thus T ∗ (w1,0) = 0, and (w1,0),T (v1,0) = 0, this implies that the range of T is in Hσ2 , there is S : Hσ1 → Hσ2 such that T (v1,0) = (0,Sv1 ). By definition T ∈ C(σ ) implies that S intertwines σ1 and σ2 . Moreover, S ∗ S = 1Hσ1 and SS ∗ = 1Hσ2 . Thus, S is a unitary equivalence between σ1 and σ2 . Conversely, if σ1 and σ2 are unitarily equivalent by S : Hσ1 → Hσ2 , then we saw in Proposition 3.3.19 that π1 and π2 are equivalent by U where for almost every x, (Uf1 )(x) = S(f1 (x))

3.4 Elements of Mackey theory

255

and clearly, for each φ ∈ Cc (G/H ),       U M1 (φ)f1 (x) = S φ(q(x))f1 (x) = φ(q(x))(Uf1 )(x) = M2 (φ)Uf1 (x). This proves that 1 and 2 are equivalent. Example 3.4.6 Recall the induced representations πγ ,λ of the Heisenberg group, described in Example 3.3.14: (πγ ,λ (x,y,z)f )(t) = eiλt ei(γ −λt)y f (t − x). We have observed that πγ ,λ = ind (G,H,χγ ,λ ) where if γ = 0, χγ ,λ and χ0,λ are inequivalent characters of H = R2 . If λ = 0, we saw that U πγ ,λ = π0,λ U , where (Uf )(t) = f (t + γλ ). In this case, we have the same projection valued measure in the two canonical systems of imprimitivity, namely: (P (E)f )(t) = χE (t)f (t); but U does not preserves this measure, since   γ γ γ  f t+ = χE (t)f t + = (P (E)Uf )(t). (U P (E)f )(t) = χE t + λ λ λ Since the representations πγ ,λ are irreducible, each intertwinning unitary operator has the form τ U with |τ | = 1, thus the two systems of imprimitivity are not equivalent. 

3.4.2 Irreducibility and equivalence of induced representations In this subsection, we shall prove a result which will be essential to the representation theory of solvable Lie groups. Suppose that N is a closed normal subgroup of a locally compact group G. Let ρ be a representation of N . For each x ∈ G the representation ρ x of N is defined by: ρ x (n) = ρ(x −1 nx)

(n ∈ N ).

Denote [ρ] the unitary equivalence class of the representation ρ and Gρ = {[ρ x ] : x ∈ G} the orbit of ρ and G(ρ) the stability group for this G-action: G(ρ) = {x ∈ G : ρ x  ρ}. Now suppose that the unitary dual Nˆ of N has the following separation property: ˆ then kerC ∗ (N ) ρ ⊂ kerC ∗ (N ) ρ  and kerC ∗ (N ) ρ  ⊂ kerC ∗ (N ) ρ. If [ρ] = [ρ  ] ∈ N, (3.4.1)

256

Unitary representations

It is easily seen that (3.4.1) implies that for each [ρ] ∈ Nˆ , {[ρ]} is closed in Nˆ , in the Fell topology. The proof of the following is essentially due to H. Leptin in [60]; see also [61]. Theorem 3.4.7 Let G be a locally compact group and N a closed normal subgroup for which property (3.4.1) holds. Let ρ1 and ρ2 be irreducible representations of N, and put Hi = G(ρi ). For i = 1,2, suppose that Hi is normal, and that there is a representation σi of Hi whose restriction to N is a multiple of ρi ; there are Hilbert spaces Hi such that σi = ρi ⊗ 1Hi Put πi = ind(G,Hi ,σi ), and let i = (πi ,G/Hi ,Pi ) be the system of imprimitivity associated to πi . Then we have the following. (1) If Gρ1 ∩ Gρ2 = ∅, then C(π1 |N ,π2 |N ) = {0}. (2) Suppose that ρ1 = ρ2 = ρ, and define C(P1,P2 ) as the space of operators T : Hπ1 → Hπ2 such that P2 (E)T = T P1 (E) for any Borel subset E of G/G(ρ). Then C(π1 |N ,π2 |N ) ⊂ C(P1,P2 ). Proof: Observe that since Hi is normal, the form μG/Hi is given by the Haar measure on G/Hi , and δHi ,G is equal to 1. Now there are Hilbert spaces Hi such that Hσi = Hρi ⊗ Hi and σi |N = ρi ⊗ 1Hi (i = 1,2). Denote qi the quotient map qi : G → G/Hi . Let T be an operator in C(π1 |N ,π2 |N ). We claim that for each x1 , x2 in G such that ρ1x1  ρ2x2 there are open subsets Ui of G such that xi ∈ Ui , U 1 ∩ U 2 = ∅ and (3.4.2) P2 (q2 (U2 ))T P1 (q1 (U1 )) = 0. Then we shall deduce from this identity the result in both cases: T = 0 in case (1), T ∈ C(P1,P2 ) in case (2). To prove the claim, choose φ ∈ kerC ∗ (N ) ρ1x1 \ kerC ∗ (N ) ρ2x2 . By replacing φ by λφ ∗ φ, we can suppose that φ is a positive element in C ∗ (N ), and ρ2x2 (φ) > 1. Now φ generates a sub-C ∗ algebra C ∗ (φ) in C ∗ (N ). Add an identity by putting A = C ∗ (φ) + C1. Now A is a commutative, unital C ∗ algebra, so by the Gelfand transform, it is isomorphic to the space C(Z) of continuous functions on the spectrum Z of φ in A (see for instance Dixmier [29, theorem 1.5.1]) Z is compact, included in [0, φ ]. This allows us to use functional calculus in A: for each function h, continuous on Z, there is a unique element denoted h(φ) in A and for each x, ρix (h(φ)) = h(ρix (φ)) (i = 1,2),

3.4 Elements of Mackey theory

257

where the right-hand side is defined with the usual functional calculus on the spectrum of the self-adjoint operator ρix (φ), and where we extend each ρix to A by putting ρix (1) = 1Hρi . Define the representation Lx × Rx on L1 (N ) by (Lx × Rx )ψ(n) = ψ(x −1 nx)G (x), since it is continuous, we extend it to C ∗ (N ). Then a direct computation gives ρix (φ) is equal to ρi ((Lx −1 × Rx −1 )φ). Observe that the maps: x → ρix (φ) are continuous. Thus there are two open neighborhoods Ui of xi in G such that: & ' 1 x , U2 ⊂ {x : ρ2x (φ) > 1} U1 ⊂ x : ρ1 (φ) < 2 and U 1 ∩U 2 = ∅. Moreover, if x ∈ U2 , then the intersection of the spectrum of ρ2x (φ) with [1, φ ] has a positive spectral measure. Similarly, for each x ∈ U1 , the spectrum of ρ1x (φ) is included in [0,1/2]. Consider now a continuous, positive function h ∈ C([0, φ ]) such that h(t) = 1 if t ≥ 1, and h(t) = 0 if t ≤ 12 . Then the element h(φ) of A satisfies: ρ1x (h(φ)) = 0

if x ∈ U1,

ρ2x (h(φ)) ≥ 1

if x ∈ U2 .

Now the element ψ = φh(φ) of A is in fact in C ∗ (φ) ⊂ C ∗ (N ).  Define K = x∈U1 kerC ∗ (N ) ρ1x , observe that ψ ∈ K, and ρ2x (ψ) = 0, for each x ∈ U2 . Now for any n ∈ N , ρ1x (Ln−1 × Rn−1 (ψ ∗ ψ)) = ρ1x (n−1 )ρ1x (ψ ∗ )ρ1x (ψ)ρ1x (n) = 0. This proves that Ln−1 × Rn−1 (ψ ∗ ψ) ∈ K. Since for each f ∈ Cc (G,Hi ,σi ) (i = 1,2), and since Hi is normal in G, (πi (h)f )(x) = f (h−1 x) = σi (x −1 hx)f (x) = σix (h)(f (x)), Considering the restrictions πi |N , and θ ∈ C ∗ (N ):

σix |N ,

(h ∈ Hi ).

we compute now πi (θ ), for

(πi (θ )f )(x) = σix (θ )(f (x)) = (ρix (θ ) ⊗ 1Hi )(f (x)). Therefore (π1 (K)f )(x) = 0 for each f ∈ Hπ1 and x ∈ U1 , moreover π1 (h) is commuting with any P1 (E) (E ∈ B(G/H1 )), therefore: π1 (K)P1 (q1 (U1 )) = P1 (q1 (U1 ))π1 (K) = 0. Similarly π2 (h) is commuting with any P2 (E) (E ∈ B(G/H2 )), hence P2 (q2 (U2 ))π2 (K)Hπ2 = π2 (K)P2 (q2 (U2 ))Hπ2 ⊂ P2 (q2 (U2 ))Hπ2 .

258

Unitary representations

Let us prove that the included space is dense. For that let f be in P2 (q2 (U2 ))Hπ2 , which is orthogonal to π2 (K)P2 (q2 (U2 ))Hπ2 . Let (ej )Jj=1 be an orthonormal basis of H2 . Identify Hσ2 = Hρ2 ⊗ H2 and J ⊕i=1 Hρ2 ⊗ ej . Now f becomes: f (x) =

J

fj (x) ⊗ ej ,

σ2x (h)(f (x)) =

j =1

J

ρ2x (h)(fj (x)) ⊗ ej

j =1

with fj (x) ∈ Hρ2 . Therefore, for any θ ∈ C ∗ (N ): (π2 (θ )f )(x) =

J

ρ2x (θ )(fj (x)) ⊗ ej .

j =1

Now for any n ∈ N , the element Ln−1 × Rn−1 (ψ ∗ ψ) of C ∗ (N ) is in K, thus: π2 (ψ)π2 (n)f

2

= π2 (n)f ,π2 (ψ ∗ ψ)π2 (n)f  = f ,π2 (Ln−1 × Rn−1 (ψ ∗ ψ))f  = 0.

This implies: 3

J

q2 (U2 ) j =1

ρ2x (ψ)ρ2x (n)fj (x) 2Hρ d x˙ = 0, 2

where d x˙ is the left-invariant Haar measure of the locally compact group G/H2 . This means that the positive functions x → ρ2x (ψ)ρ2x (n)fj (x) 2Hρ 2 are all vanishing for almost every x˙ ∈ q2 (U2 ). Using a dense countable subset ˙ {ni } in N, ρ2x (ψ)ρ2x (ni )fj (x) Hρ = 0 for any j , any i, and almost every x. So by continuity ρ2x (ψ)ρ2x (n)fj (x) Hρ2 = 0,

(n ∈ N, j ∈ J, a. e. x˙ ∈ q2 (U2 )).

Since if fj (x) = 0, the set {ρ2x (n)fj (x) : n ∈ N } generates a dense subspace ˙ But it is not the case, of Hρ2 , this implies that ρ2x (ψ) = 0, for almost every x. therefore fj = 0 for any j , f = 0, and we proved the equality: Closure of (P2 (q2 (U2 ))π2 (K)Hπ2 ) = P2 (q2 (U2 ))Hπ2 . Put now S = P2 (q2 (U2 ))T P1 (q1 (U1 )), then S belongs to C(π1 |N ,π2 |N ), so S ∗ is in C(π2 |N ,π1 |N ) and S ∗ π2 (K) = π1 (K)S ∗ . Thus since P1 (q1 (U1 )) π1 (K) = 0, SS ∗ π2 (K) = SP1 (q1 (U1 ))π1 (K)S ∗ = 0.

3.4 Elements of Mackey theory

259

Moreover, since π2 (K)P2 (q2 (U2 ))Hπ2 is dense in P2 (q2 (U2 ))Hπ2 , SS ∗ Hπ2 = SS ∗ P2 (q2 (U2 ))Hπ2 = 0. Therefore S = 0, and this proves the claimed relation (3.4.2). First case Gρ1 ∩ Gρ2 = ∅. In this case, to show that T = 0, we first prove that for any compact subsets Ai in G/Hi , P2 (A2 )T P1 (A1 ) = 0. ˙ y) ˙ ∈ A2 × G/H1 , there Let A2 be a compact subset of G/H2 . For each (x, and V , such that: x ˙ ∈ U , y ˙ ∈ Vx, is two open sets Ux, ˙ y˙ x, ˙ y˙ x, ˙ y˙ ˙ y˙ , and: P2 (Ux, ˙ y˙ )T P1 (Vx, ˙ y˙ ) = 0. Fix y, ˙ we extract a finite covering of A2 : A2 ⊂ ∪ri=1 Ux˙i , y˙ , put Vy˙ = ∩ri=1 Vx˙i , y˙ , we have for each f ∈ Cc (G,H,σ1 ), P2 (Ux˙i , y˙ )T P1 (Vy˙ )f = P2 (Ux˙i , y˙ )T P1 (Vx˙i , y˙ )P1 (Vy˙ )f = 0

(i = 1, . . . ,r).

This means T P1 (Vy˙ )f (x) = 0 for almost every x in Ux˙i , y˙ , and for each i, thus P2 (A2 )T P1 (Vy˙ )f (x) = 0 almost everywhere, or: P2 (A2 )T P1 (Vy˙ ) = 0. Since T ∗ is in C(π2 |N ,π1 |N ), for any compact subset A1 ⊂ G/H1 , for each y ∈ G/H2 , there is an open subset Wy˙ ⊂ G/H2 , containing y˙ such that: P1 (A1 )T ∗ P2 (Wy˙ ) = 0. Thus, P2 (Wy˙ )T P1 (A1 ) = 0, and by the above argument, for any compact sets Ai , in G/Hi , P2 (A2 )T P1 (A1 ) = 0. By the inner regularity of the Haar measures on G/Hi , then with a preceding argument, T P1 (A1 )f (x) is vanishing almost everywhere (f ∈ Hπ1 ). Thus T P1 (A1 ) = 0, similarly P1 (A1 )T ∗ = 0, for each compact set A1 implies T = 0. This concludes the proof of the first case. If Gρ1 = Gρ2 , we can replace ρ2 by ρ2x = ρ1 , and we are in the second case. Second case ρ1 = ρ2 = ρ, H1 = H2 = H . In this case, let us show that for any Borel subset E in G/H , T P1 (E) = P2 (E)T .

260

Unitary representations

/ A2 , then as above, for each Let A2 be a compact subset in G/H , and y˙ ∈ and V , such that: x˙ ∈ Ux, x˙ ∈ A2 , there are open subsets Ux, ˙ y˙ x, ˙ y˙ ˙ y˙ , y˙ ∈ Vx, ˙ y˙ , U x, ˙ y˙ ∩ V x, ˙ y˙ = ∅, and: P2 (Ux, ˙ y˙ )T P1 (Vx, ˙ y˙ ) = 0. Thus there is Vy˙ open in G/H , containing y˙ such that: Vy˙ ∩ A2 = ∅ and P2 (A2 )T P1 (Vy˙ ) = 0. Similarly, for each compact subset A1 in G/H , such that A1 ∩ A2 = ∅, and each y˙ ∈ A2 , P1 (A1 )T ∗ P2 (Wy˙ ) = 0. As in the first case, this implies P2 (A2 )T P1 (A1 ) = 0 for each A1 , A2 compact and disjoint. By the inner regularity of the Haar measures on G/H , then with the preceding argument, T P1 (A1 )f (x) is vanishing almost everywhere on the complementary space Ac1 of A1 (f ∈ Hπ1 ). Thus, for any compact A, P2 (Ac )T P1 (A) = (1 − P2 (A))T P1 (A) = 0. Replacing T by T ∗ , we get similarly: P2 (A)T P1 (Ac ) = P2 (A)T (1 − P1 (A)) = 0. Thus, P2 (A)T = P2 (A)T P1 (A) = T P1 (A) for each compact set. Employing the regularity of d x˙ again, we get: P2 (E)T = T P1 (E)

(E ∈ B(G/H )).

This achieves the proof of the theorem. Since C(π ) ⊂ C(π|N ), the following is evident: Corollary 3.4.8 With the notation and the hypothesis of Theorem 3.4.7, if σ = σ1 = σ2 , then C(π ) = C(). If ρ1 and ρ2 are two irreducible representations of N , such that Gρ1 ∩Gρ2 = ∅, then we saw that, with our separation hypothesis (3.4.1), π1 and π2 are inequivalent. Suppose now ρ1 = ρ2 = ρ. Corollary 3.4.9 Let G be a locally compact group, N ⊂ H be closed normal subgroups of G. Let ρ be an irreducible representation of N such that H = G(ρ), suppose that Nˆ has the separation property (3.4.1).

3.4 Elements of Mackey theory

261

(1) Let σ be an irreducible representation of H such that σ |N is a multiple of ρ, then π = ind (G,H,σ ) is irreducible. (2) Let σ1 and σ2 be two irreducible representations of H such that σi |N is a multiple of ρ (i = 1,2), then π1 = ind (G,H,σ1 ) is unitarily equivalent to π2 = ind (G,H,σ2 ) if and only if σ1 is unitarily equivalent to σ2 . Proof: The first point is clear since if T is in C(π ), by the preceding corollary ˜ and T is in C(). Thus by Theorem 3.4.4, there is S ∈ C(σ ), such that T = S, since σ is irreducible, then S is scalar, thus T also. Thus π is irreducible. If π1 and π2 are unitarily equivalent, there is a unitary operator T in C(π1,π2 ), and T is in C(π1 |N ,π2 |N ). By Theorem 3.4.7, T is in C(1,2 ), thus 1 is equivalent to 2 and by Corollary 3.4.5, σ1 and σ2 are equivalent. Recall that the converse holds by Proposition 3.3.19. Example 3.4.10 Recall the solvable Lie group E˜ 2 described in Examples 1.7.14 and 3.3.28:   G = E˜ 2 = (z,t) : t ∈ R, z ∈ C with (z,t)(z,t  ) = (z + eit z,t + t  ). G is the semidirect product of the abelian Lie group N = C = {(z,0)} by the one-dimensional group R = {(0,t)}. The dual Nˆ is simply the vector space n∗ , with the identification χζ (z) = eiRe(zζ ) . For each ζ = 0, the orbit Gχζ is a circle in n∗ : χζ(z,t) = χeit ζ . If ζ = 0, Gχ0 = {χ0 }. Since N is connected, Lie and abelian, condition (3.4.1) holds. Choose ζ = 0, then H = G(χζ ) = {(z,2π k) : z ∈ C, k ∈ Z}. The character χζ has many extensions to H , these extensions are the product of two characters and parametrized by τ ∈ R/Z: σζ,τ (z,2π k) = χζ (z)ητ (k) = eiRe(zζ ) e2iπ τ k . If ζ = 0, H = G, the character χ0 has also many extensions, parametrized by α ∈ R: π0,α (z,t) = eiαt . Then by Corollary 3.4.9, the representation πζ,τ = ind(G,H,σζ,τ ) is irreducible. Moreover, for each ζ = 0, σζ,τ is equivalent to σζ,τ  if and only if τ = τ  , thus, still by Corollary 3.4.9, πζ,τ is equivalent to πζ,τ  if and only if τ = τ . Finally, the same argument proves that if 0 = |ζ | = |ζ  | = 0, then πζ,τ and πζ ,τ  are inequivalent. Similarly, if ζ = 0, πζ,τ is not equivalent to π0,α and π0,α is equivalent to π0,α  if and only if α = α  . The results of this chapter do not imply that this describes the entire unitary dual of G, nevertheless, this is the case (see for instance Mackey [64]). 

262

Unitary representations

Example 3.4.11 We return to Example 3.2.18, introduced in Example 3.1.9, and revisited in Example 3.3.25 for the case where H ⊂ GL(V ). Here V is a finite-dimensional vector space over R, H is a simply connected solvable Lie group, α : H → GL(V ) is a representation of H in V , and G = V α H . We assume that α is of exponential type. For f ∈ L2 (V ), τ (v,h)f (w) = f (α(h)−1 (w − v))| det α(h)|−1/2 7 and via the Fourier transform, for F ∈ L2 (V ∗ ), τˆ (v,h) =  τˆσ (v,h) dν (σ ), where  is the cross-section for α ∗ (H )-orbits in the minimal layer  ⊂ V ∗ , and τˆσ (v,h) acts in L2 (Oσ ,ωσ ) by τˆσ (v,h)g() = χ (v)g(α ∗ (h)−1 )| det α ∗ (h)|−1/2 . Let σ ∈ , and apply Corollary 3.4.9 to N = V and ρ = χσ . Then G(σ ) = V H (σ ), and χσ extends trivially to G(σ ) by χσ (v,h) = χσ (v). By Corollary 3.4.9, πσ = ind(G,V  H (σ ),χσ ) is irreducible. We claim that πσ  τˆσ . To see this, we prove the following lemma. Lemma 3.4.12 For all h ∈ H (), δH (),H (h)−1 = δH,G (h) = | det α(h)|. Hence a rho-function for H () in H , or for V  H () in G, is −1 (v,h) = | det α(h)|−1 = | det α ∗ (h)|. ρV H (),G (v,h) = ρH,G

Proof: Let O be the H -orbit of . Let T : h/h() → V ∗ be the natural map T (Z + h()c ) = β ∗ (Z) and let U () = T (h/h()c ) = T (hc /h()c ). Then U ()  T (O)c and U is H ()-invariant.  ∗ / e}. Suppose i ∈e We claim that Vc∗ = U () ⊕ span{fi : i ∈ / bi fi = β (Z) / e : bi = 0}; then β ∗ (Z) ∈ ∈ U (). If bi = 0 for some i, then let i0 = min{i ∈ span{fi0 ,fi0 +1, . . . ,fn } = Ui0 −1 , so Z ∈ h(πi0 −1 ())c and bi0 = 0 means Z∈ / h(πi0 ())c which implies i0 ∈ e, a contradiction. This proves the claim. By Proposition 2.2.9, for i ∈ / e, and Y ∈ h(), λi (Y ) = 0. Hence for h ∈ H (), | det α ∗ (h)|U () | = | det α ∗ (h)|. But it is easily seen that | det α ∗ (h)|U () | = | det Adhc /h()c (h)|: for h ∈ H (), T Adhc /h()c (h)T −1 = α ∗ (h)|U () . Hence for h ∈ H (), | det α ∗ (h)| = | det Adhc /h()c (h)| = δH (),H (h).

3.4 Elements of Mackey theory

263

Realize the representation πσ = ind(G,V  H (σ ),χσ ) in Hρ (G,V  H (σ ),χσ ), using the rho-function ρ(v,h) = | det α ∗ (h)| = | det h|−1 . Thus for f ∈ Hρ (G,V  H (σ ),χσ ), πσ (v0,h0 )f ((0,h)) = f ((v0,h0 )−1 (0,h))ρ(v0,h0 )−1/2 −1 −1/2 = f ((0,h−1 0 h)(h v0,e))ρ(v0,h0 ) ∗ −1/2 = χσ (h−1 v0 )f (0,h−1 0 h) | det α (h0 )| ∗ −1/2 = χhσ (v0 )f (0,h−1 . 0 h)) | det α (h0 )|

This means that with the natural isomorphism of Hρ (G,V  H (σ ),χσ ) with L2 (H /H (σ ),νρ ) (where we identify G/V  H (σ ) with H /H (σ ) in the obvious way), we can write ∗ −1/2 πσ (v0,h0 )f (h) = χhσ (v0 )f (h−1 . 0 h) | det α (h)|

Define ω˜ σ on H /H (σ ) by ω˜ σ (EH ) = ωσ (Eσ ) for any Borel set E. By Lemma 3.4.12, both νρ and ω˜ σ satisfy dνρ (sxH ) = ρ(s)dνρ (xH ) and d ω˜ σ (sxH ) = ρ(s)d ω˜ σ (xH ), so they are proportional, and we choose the Haar measure on H (σ ) so that they coincide. Consider the natural isomorphism Tσ : L2 (O,dωσ ) → L2 (H /H (σ ),dνρ ) given by Tσ g(hH (σ )) = g(α ∗ (h)σ ), g ∈ Cc (O). Then πσ (v0,h0 )(Tσ g)(h) = Tσ τˆσ (v0,h0 )g(hσ ). Thus the direct integral decomposition of τˆ (or of τ ) is an irreducible decomposition. Moreover, σ  σ  implies τˆσ and τˆσ  are not equivalent. Such a decomposition is said to be multiplicity free.  Example 3.4.13 Consider now the representations Tξ,η of the Mautner group G, defined in Example 3.2.21. Writing G as the semidirect product of N = C2 by R, it is easily seen that the representation Tξ,η , for |ξ | = 0, |η| = 0 are induced from the character χξ,η (z,w) = eiRe(zξ +wη) . Now the orbit Gχξ,η in Nˆ = n∗ is the set of all χeit ξ,eiαt η , t ∈ R. We also get here H = G(χξ,η ) = N . Therefore, Corollary 3.4.9 says that each Tξ,η is an irreducible representation of G and Tξ,η and Tξ ,η are equivalent if and only if there is t such that ξ  = eit ξ and η = eiαt η.

264

Unitary representations

Defining now the torus: T2ρ,σ = {(ξ,η) : |ξ | = ρ, |η| = σ }, where ρ and σ are fixed positive numbers, we see that for each (ξ,η), with ξ = 0 and η = 0, the closure of the orbit of χξ,η in Nˆ = n∗ is the torus T2|ξ |,|η| . However, this orbit, that is the set of (ξ ,η ) such that Tξ,η and Tξ ,η are equivalent, has zero measure on the torus.  of G, acting on On the other hand, we defined the representations Tρ,σ,τ 2 2 L (T ):  (Tρ,σ,τ (z,w,x)f )(y,s) = f (ye−ix ,se−iαx )eiτ x eiRe(ρzy+wσ s) .  Recall that we proved in Example 3.2.36 that each Tρ,σ,τ (ρ > 0, σ > 0) is   is equivalent to Tρ ,σ ,τ  if and only if ρ = ρ  , σ = σ  irreducible, and Tρ,σ,τ  and τ ∈ τ  + Z + αZ. Let us now prove that Tρ,σ,τ is not equivalent to any Tξ,η . Pick the constant function f = 1, we get:  (Tρ,σ,τ (0,0,x)f )(y,s) = eiτ x f (y,s).  Suppose there is a unitary operator U intertwining Tρ,σ,τ and Tξ,η , and put g = Uf . Then, for each x, and almost every t,

(Tξ,η (0,0,x)Uf )(t) = g(t − x) = eiτ x Uf (t) = eiτ x g(t). This is clearly impossible, so there is no such U . Moreover, the inducing method coming from the use of Corollary 3.4.9 can be developed for N, χ0,η , in this case H = N  2π α Z = {(z,w,2π k/α)} and the result is an irreducible representation such that (z,0,0) is in its kernel for any z,  (z,0,0) = 1 if z = 0, then this representation is not equivalent to since Tρ,σ,τ  . The same argument proves that the method of Corollary 3.4.9 starting Tρ,σ,τ  . from χξ,0 , or χ0,0 never gives a representation equivalent to Tρ,σ,τ Therefore we see that for the Mautner group, the totality of the dual of G ˆ is cannot be obtained by this inducing method. In fact, as far as we know, G still not known (for more on this subject, see Baggett [10], and the book by E. Kaniuth and K. F. Taylor [50]). 

3.4.3 Unitary dual of a nilpotent Lie group Let G be a simply connected solvable Lie group. Recall the coadjoint action of G in the dual g∗ of its Lie algebra. If G is nilpotent, the corresponding orbits (the coadjoint orbits) are closed subsets in g∗ (see Remark 2.3.6).

3.4 Elements of Mackey theory

265

The orbit method establishes a canonical one-to-one correspondence between coadjoint orbits of G nilpotent and equivalence classes of irreducible unitary representations called the Kirillov correspondence, first proved by A.A. Kirillov in [52]. It turns out that this bijection is a homeomorphism (I. Brown [13]), and thus for a simply connected nilpotent Lie group G, the space of coadjoint orbits provides a perfect “map” of the unitary dual of G. In this setting, the Kirillov correspondence is established by an explicit construction as follows. Let π be an irreducible unitary representation of G. Then there is an analytic subgroup H of G, and a unitary character χ of H , such that π is equivalent with the representation induced by χ . Writing H = exp h and χ (exp Y ) = ei(Y ) where  ∈ g∗ , we have [h,h] ⊆ ker  and associate to π the orbit of . On the other hand let  ∈ g∗ . If h is a subalgebra of g that is also an isotropic subspace for the skew symmetric form B defined on g by B (X,Y ) = [X,Y ], then  defines a character χ of H = exp h as above. If the subalgebra h is a maximal isotropic subspace for B , then we say that h is polarization at . It can be shown that the representation induced from χ is irreducible if and only if h is a polarization at . In this case, if  = s for some s ∈ G, then it is easily seen that the representation induced from h = Ad(s)h by χ is equivalent with π. Finally, if h and h are both polarizations at , then the induced representations are also equivalent. Thus a canonical bijection K from ˆ is established. the space g∗ /G of coadjoint orbits to the unitary dual G The map K−1 can be also be obtained directly by the Kirillov character formula: let π be an irreducible unitary representation of G. Then for any Schwartz function  on g, the function φ(exp X) = (X) is in L1 (G) and π(φ) is a trace-class operator. Moreover, there is a unique coadjoint orbit O and an invariant measure μO on O such that, for any : 3 trace (π(φ)) =

O

ˆ () dμO ().

ˆ is the Euclidean Fourier transform of  Here  3 ˆ () = (X)ei(X) dX. g

Proofs of above results are found in Kirillov [52], and Pukanszky [79]; see the Corwin and Greenleaf book [16] for a detailed exposition of all relevant objects. We shall consider in the rest of the book a generalization of some of these results in the case of a simply connected solvable Lie group G. We shall need

266

Unitary representations

to use Theorem 3.4.7 with the normal subgroup N as a simply connected nilpotent Lie group. Thus we prove the following lemma. Lemma 3.4.14 Let N be a simply connected nilpotent Lie group, then the separation property (3.4.1) holds for the dual Nˆ of N . Proof: Given [ρ] ∈ Nˆ , the orbit O corresponding to [ρ] is closed. The invariant measure μO on O is unique up to a constant, and strictly positive on any nonempty open subset in O. Pick a point  ∈ O, and a compact neighborhood ˆ Since the orbit O associated to [ρ  ] is closed, V of  in O. Let [ρ  ] = [ρ] ∈ N. there is a compact neighborhood U of V in n∗ such that U ∩ O = ∅. Now fix a smooth positive function ! on g∗ , with support in U , such that !|V = 1. Define  as the inverse Fourier transform of ! and φ(exp X) = (X). Then φ ∈ L1 (N ) and trace ρ(φ) ≥ μO (V ) > 0, while ρ  (φ) = 0. Therefore: kerC ∗ (N ) ρ  ⊂ kerC ∗ (N ) ρ. Since ρ and ρ  are arbitrary, Property (3.4.1) holds. Before taking up the task of generalizing in the next chapter to the case of solvable Lie groups, we briefly illustrate these results with the well-known example of the Heisenberg Lie group. Example 3.4.15 Let G be the three-dimensional Heisenberg group (Example 1.6.4) with Lie algebra g. Denote as usual {X,Y,Z} the basis of g such that [X,Y ] = Z and (x,y,z) = exp zZ exp yY exp xX. In Example 3.2.35, we invoke the Stone–von Neumann theorem to describe the factor representations of G and show that G is type 1. From this, it is easy to see that an irreducible unitary representation π of G which is not trivial on the center (π(exp zZ) = eiλz 1Hπ , λ = 0) is equivalent with the representation π0,λ = πλ : (πλ (x,y,z)f )(t) = eiλz e−iλyt f (t − x), f ∈ L2 (R). The other irreducible representations are characters. In Example 3.3.14, we saw that πλ = ind(G,H,χλ ), where H is the analytic subgroup associated to the subalgebra h = span{Z,Y }. Let {X ∗,Y ∗,Z ∗ } be the dual basis of {X,Y,Z}. Put  = λZ ∗ , then the reader can verify the following: (1) the coadjoint orbit O of  is the plane  + Z ⊥ , (2) the subalgebra h is a polarization at , and (3) χ is the character exp(yY + zZ) → eiλz = ei(yY +zZ) . Thus the representation πλ is the representation associated to O by the Kirillov theory.

3.4 Elements of Mackey theory

267

Given a Schwartz function φ on G = R3 , the operator π(φ) is a kernel operator on L2 (R) given by 3 φ(x,y,z)eiλz e−iλyt f (t − x) dxdydz (πλ (φ)f )(t) = G  3 3 iλz −iλyt φ(t − x,y,z)e e dydz f (x) dx. = R

R2

The kernel K(x,t) of πλ (φ) is thus explicitly computed as a partial Fourier transform of φ(x,y,z): K(x,t) = (∧y,z φ)(t − x, − λt,λ). Its trace is given by 3

3 1 1 ˆ trace (πλ (φ)) = K(t,t) dt = dγ dβ. φ(β,γ ,λ) 2π |λ| R O This is the Kirillov character formula if we identify φˆ with a function on g∗ , writing ∗ ˆ ˆ + γ Y ∗ + λZ ∗ ). φ(β,γ ,λ) = φ(βX



Finally, we observe that the Kirillov theory allows for a more concrete proof of Theorem 3.4.7 in the case where the normal subgroup N is a simply connected nilpotent Lie group. We saw that the separation property holds in this case (Lemma 3.4.14). There exists  ∈ n∗ such that ρ = ind(N,P ,χ ), where P is a polarization at . Then, for each x ∈ G, ρ x  ind(N,P x ,χx· ). Pick now two points x˙1 = / O2 . x˙2 in G/H , put i = xi , and put Oi = N i . Since H = G(ρ), then 1 ∈ Since O2 is closed, there are relatively compact open subsets U1 ⊂ U1 in G such that x1 ∈ U1 , U 1 ⊂ U1 and U1  ∩ O2 = ∅. We prove that N U1  = NU 1 . Clearly, the inclusion N U 1  ⊂ N U1  holds. Let v ∈ N U1 ; there are sequences (nk ), (xk ) with nk ∈ N, xk ∈ U1 and nk xk  → v. Since U1 is relatively compact, we may assume xk → x ∈ U 1 . Since N is normal and closed, we have: nk xk  = xk nk  → v,

nk  = xk−1 nk xk  → x −1 v.

But xk−1 nk xk  is in N , and N  is closed; hence there is n ∈ N such that x −1 v = n, so v = xn = xnx −1 x ∈ N U 1 . This proves N U1  = N U 1 . Since N U1  is N -invariant, this implies that O2 ∩ N U1  = ∅. Choose then relatively compact, open subsets V2 ⊂ V2 of n∗ such that 2 ∈ V2 , V 2 ⊂ V2 ,

268

Unitary representations

and V2 ∩ N U1  = ∅, and pick a smooth positive function θ with compact support such that θ |V2 = 1, and θ |n∗ \V2 = 0. Put  = ∧−1 θ and φ(exp X) = (X). Then  is a Schwartz function on n, by the character formula, ρ x (φ) = 0 for each x ∈ U1 , and ρ x (φ) = 0 for each x such that x ∈ N V2 . Consider now the continuous map p : G → n∗ , p(x) = x. We choose then a relatively compact open subset U2 of G such that: x2 ∈ U2 ⊂ U 2 ⊂ p−1 (V2 ). In the proof of Theorem 3.4.7, we built an element ψ ∈ C ∗ (N ) such that: (a) Ln × Rn (ψ ∗ ψ) ∈ K = ∩x∈U1 kerC ∗ (N ) ρ1x , and (b) ρ2x (ψ) = 0 for almost every x ∈ U2 . When N is nilpotent, we can use the L1 (G)-function φ in the place of ψ, and the above open subsets Ui , and write with these objects the proof of the relation (3.4.2), without use of C ∗ -algebras and Gelfand theory.

4 Coadjoint orbits and polarizations

4.1 Coadjoint orbits Of singular importance for harmonic analysis on solvable Lie groups is the coadjoint representation. Denote the real linear dual of the Lie algebra g of a Lie group G by g∗ . Each element  ∈ g∗ extends to s = gc in the obvious way: (X + iY ) = (X) + i(Y ), and thus g∗ is the R-subspace of all real linear forms. For any subset k of gc put k⊥ = { ∈ g∗ : |t = 0} and for any subspace U of g∗ , put U ⊥ = ∩∈U ker . Observe that (U ⊥ )⊥ = U and if k is real, then (k⊥ )⊥ = k. The coadjoint representation Ad∗ : G → GL(g∗ ) is defined by Ad∗ (g)  =  ◦ Ad(g −1 ),  ∈ g∗,g ∈ G. Since dAd(X) = ad(X), X ∈ g, then it is easily verified that dAd∗ is the coadjoint representation ad∗ of g defined in Section 1.2: ad∗ (Y ) =  ◦ −ad(Y ) = [·,Y ], Y ∈ g. (We write [X,Y ] rather than ([X,Y ]).) As usual, ad∗ is extended to gc . For simplicity we write s = Ad∗ (s)  and X = ad∗ (X). For each , the Lie algebra of the stabilizer G() of  is the annihilator subalgebra g() = {X ∈ g : ad∗ (X) = 0} = {X ∈ g : [Y,X] = 0} and of course the complexification of g() is the annihilator s() in s = gc . Now fix  ∈ g∗ and let O = G be the coadjoint orbit of ; by Lemma 1.9.7, O is a submanifold of g∗ whose tangent space at  is T (O) = {ξX : X ∈ g} = {X : X ∈ g}. 269

270

Coadjoint orbits and polarizations

d |t=0 f (exp −tX).) The map X → X (Recall the notation (1.9.3): ξX f = dt establishes the canonical vector space isomorphism

T (O)  g/g() (Lemmas 1.9.4 and 1.9.5). Note also that the extension of the map X → X to s = gc gives a canonical complex isomorphism between the complexification T (O)c of the tangent space at  to O and s/s() (Lemma 2.2.1). Example 4.1.1 Let G be the Heisenberg group of dimension 2d + 1, with Lie algebra g (Examples 1.1.5 and 1.6.4). A basis of g is (Z,Y1, . . . ,Yd , X1, . . . ,Xd ), with nonvanishing bracket [Xi ,Yi ] = Z. Denote by Z ∗ , Yi∗ , and Xi∗ the dual basis. A direct computation shows that Xi Z ∗ = −Yi∗ , Yi Z ∗ = Xi∗,1 ≤ i ≤ d. Let  ∈ g∗ . If  = λZ ∗ with λ = 0, the orbit O of  is

O = λZ ∗ + RYi∗ + RXi∗ = λZ ∗ + z⊥, i

i

where z = RZ is the center of g. Thus g() = z and X → X maps g onto T (O) = span{Y1∗, . . . ,Yd∗,X1∗, . . . ,Xd∗ } with kernel g(). On the other hand, if (Z) = 0, then its orbit is trivial: O = {}.



For each  ∈ g∗ , define the alternate bilinear form B on g × g by B (X,Y ) = [X,Y ], X,Y ∈ g. Note that g() is the radical of the bilinear form B . So B defines a nondegenerate alternate form on g/g(), still denoted by B : B (X + g(),Y + g()) = [X,Y ]. For each s ∈ G(), Ad(s) defines a linear homomorphism on g/g(), denoted Adg/g() (s), which preserves B : B (Adg/g() (s)(X + g()),Adg/g() (s)(Y + g())) = B (X + g(),Y + g()). Hence det(Adg/g() (s)) = 1 (s ∈ G())

(4.1.1)

and trace(adg/g() (Z)) = 0 for each Z ∈ g(). The form B is extended to gc in the obvious way, and it will be useful to consider B as an alternate form on gc × gc , as well as g × g and also on

4.1 Coadjoint orbits

271

s/s() × s/s(). Fix a basis (Z1, . . . ,Zn ) of gc , and its dual basis (f1, . . . ,fn ) of g∗c , put pi () = Zi . Then the matrix of the alternate form B on gc × gc is ⎡

0 ⎢[Z2,Z1 ] ⎢ M() = ⎢ .. ⎣ .

[Z1,Z2 ] 0 .. .

[Zn,Z1 ] [Zn,Z2 ]

⎤ · · · [Z1,Zn ] · · · [Z2,Zn ]⎥ ⎥ ⎥. .. .. ⎦ . . ··· 0

Since pi (Zj ) = Zj (Zi ) = [Zi ,Zj ], this is also the matrix of the map: Z → Z considered in Section 2.2, using the basis (Zj ) and (fi ). Given a real subspace k of g, its orthogonal complement under B is denoted by k ; thus k = {X ∈ g : [X,Y ] = 0 for all Y ∈ k}. If h is a complex subspace of gc then similarly h = {W ∈ gc : [W,Z] = 0 for all Z ∈ h}. Various facts about alternate bilinear forms will be recalled in what follows. A natural symplectic structure is defined on O as follows. For each  ∈ O the above gives a nondegenerate alternate form ω on T (O) defined by ω (ξ X,ξ Y ) = B (X,Y ) = [X,Y ].

(4.1.2)

This then defines a canonical two-form ω on the manifold O, and implies that O is a manifold with even dimension; ω is called the Kirillov form. Proposition 4.1.2 Let G be any Lie group, let O be a coadjoint orbit of G. Then the Kirillov form is a nondegenerate, G-invariant, closed two-form on O. Proof: We have seen that for each  ∈ O, B is nondegenerate on g/g(), and hence ω is nondegenerate. By Lemma 1.9.4, if φ (s) = Ad∗ (s), then (dφ )s (X) = Ad∗ (s)(ξX ) = −Ad∗ (s)(X) = −(Ad(s)X)(s) = ξs

Ad(s)X

Therefore:   Ad(s)X Ad(s)Y (Ad∗ (s))∗ ω  (ξ X,ξ Y ) = ωs (ξs ,ξs ) = s[Ad(s)X,Ad(s)Y ] = ω (ξ X,ξ Y ).

.

272

Coadjoint orbits and polarizations

Lastly the two-form ω is closed, since a direct computation gives ξ X pZ = p[X,Z] , and [ξ X,ξ Y ]pZ = ξ [X,Y ] pZ if pZ () = Z. Hence, (dω) (ξ X,ξ Y ,ξ Z ) = ξ X (ω (ξ Y ,ξ Z )) − ξ Y (ω (ξ X,ξ Z )) + ξ Z (ω (ξ X,ξ Y ))− − ω ([ξ X,ξ Y ],ξ Z ) +ω ([ξ X,ξ Z ],ξ Y)− ω ([ξ Y ,ξ Z ],ξ X ) = ξ X ([Y,Z]) + ξ Y ([Z,X]) + ξ Z ([X,Y ]) − [[X,Y ],Z] − [[Z,X],Y ] − [[Y,Z],X] = 0. The preceding shows that a coadjoint orbit O of a connected Lie group is a homogeneous symplectic manifold. We can express the form ω in terms of coordinate functions pZ . Observe first that we can define ξX for any X ∈ s by putting ξX = ξX1 + iξX2 if X = X1 + iX2 and X1 , X2 ∈ g. The preceding computation still holds: if pX () = (X) (X ∈ s), then ξ X pZ = p[X,Z] , and [ξ X,ξ Y ]pZ = ξ [X,Y ] pZ . Now suppose that {U1, . . . ,Ud } is a basis of a supplementary space to s() in s. Then {ξU1 , . . . ,ξUd } is a basis of T (O)c and U

the dual basis of T (O)∗c is denoted by du1, . . . ,dud : dui (ξ j ) = δij . Observe U

that ω is determined by its value on each pair ξUi , ξ j where i < j . Note also that O carries a G-invariant volume form dκ, defined by dκ = ∧d/2 ω

(4.1.3)

where d = dim O. The associated measure κO is called the Kostant measure on O; by Proposition 4.1.2, κO is invariant. Observe also that the G-invariant measure κO is not just determined up to a constant: it is canonically associated to the orbit. Recall the invariant measure dμO on the coadjoint orbit of a simply connected nilpotent Lie group. Since it is invariant, it is proportional to dκO . We shall prove that: dμO =

1 dκO . (d/2)! (2π )d/2

The Kostant measure is determined by the value of the d-form ∧d/2 ω at . A standard computation yields the following. Lemma 4.1.3 With our notation, we have:

[Ui ,Uj ]dui ∧ duj , ω = i 0, then it is easy to check that its orbit is: O = {exp tA exp(xX + yY ) = (re−it , − ry) : t, y ∈ R}, which is not simply connected: π1 (O)  Z is generated by the path t → (re−it ,0). Using the preceding exact sequence of homotopy, we can also compute the stabilizer G() = {(z,t) ∈ E˜ 2 : t ∈ 2π Z and Im(zr) = 0}. Thus G() is not connected and the orbit O is not simply connected.



The orbit method refers to the association of a coadjoint orbit O of G to a family of equivalence classes of irreducible unitary representations of G that is in some sense canonical, and whereby certain properties of the irreducible unitary representations may correspond to geometric properties of O. This association incorporates a construction of an irreducible representation using data from O as well as other ingredients, and involves the notion of a polarization, a notion that can be expressed in both algebraic and geometric terms. In short, a polarization will be a subalgebra of the complexification gc of the Lie algebra of G that is related in a precise way to a point  ∈ O and to the symplectic structure of O. When G is simply connected nilpotent, this method is especially successful, as briefly described near the end of Chapter 3. A central goal of this book is to explicitly describe the extent to which the orbit method works for simply connected solvable Lie groups. Polarizations are essential for understanding the orbit method, and the bulk of this chapter is devoted to a detailed examination of their properties. In order to define a polarization we must recall the notion of isotropic subspace for an alternate form. Let E be a vector space over k (= R or C), and let B be an alternate bilinear form on E × E. For any subset H of E put H ⊥B = {Z ∈ E : B(Z,Y ) = 0,for all Y ∈ H }. Recall that the radical of B is the subspace radB = E ⊥B . A subspace H of E is said to be isotropic for B if H ⊂ H ⊥B . The following standard result is stated without proof.

4.1 Coadjoint orbits

277

Lemma 4.1.8 Let H be a subspace of E that is isotropic for B. Then the following are equivalent. (i) H = H ⊥B . (ii) H is maximal among all isotropic subspaces of E. (iii) dim H = (dim E + dim radB)/2. Fix a basis (Zj )1≤j ≤n for E, by which we identify E with kn . The matrix M = [B(Zi ,Zj )]1≤i,j ≤n is skew-symmetric and for any X,Y ∈ E, B(X,Y ) = t XMY . It follows that radB = nullspaceM. It is easily seen that each of the conditions of Lemma 4.1.8 is equivalent with each of the conditions (iv) dim E/H = 1/2 dim(E/rad B), or, (v) dim H = dim nullspaceM + (1/2)rankM. When G is a connected Lie group,  ∈ g∗ , and B = B , then recall that we regard the alternate form B both on g × g and gc × gc , and for a subspace k of ⊥ g (or of gc ), we denote kB by k . The orbit method that works so well for simply connected nilpotent Lie groups is complicated in the solvable case for several reasons. The first problem that arises is that for some solvable G, it may happen that there is no subalgebra h of g that is also a maximal isotropic subspace for B . The simplest example of such a group is the diamond group. Example 4.1.9 Let g be the diamond Lie algebra with basis (Z,Y,X,A) as in Examples 1.3.13 and 1.7.15. Recall that [X,Y ] = Z, and [A,X + iY ] = −i(X + iY ). Let g∗ have the dual basis, and consider  = Z ∗ . Let us show that there is no maximal isotropic subspace for B that is also a subalgebra of g. Let h be maximal isotropic for B . Then g() = span{Z,A} ⊂ h. Since h is maximal isotropic, then necessarily dim h = 3. Hence for some real numbers x and y, not both zero, h = span{Z,xX + yY,A}. But [A,xX + yY ] = −yX + xY ∈ / h, so h is not a subalgebra. Observe however that the complex subspace h = span{Z,X + iY,A} is a subalgebra and a maximal isotropic supspace of gc for the alternate form B on gc × gc .  The example of the diamond group suggests that polarizations must be defined as complex subalgebras of gc , and this is the indeed the case. The problem of an appropriate definition of polarization is taken up in the next section.

278

Coadjoint orbits and polarizations

4.2 Polarizations Let G be a solvable Lie group with Lie algebra g. For  ∈ g∗ , a suitable definition of polarization allows for a generalization of induced representations, so that we can in effect induce from a complex subalgebra that does not correspond to any Lie subgroup of G. For this purpose, there is an extra condition in the definition of polarization. Definition 4.2.1 Let  ∈ g∗ . A polarization at  is a subalgebra h of gc that is a maximal isotropic subspace for the bilinear form B , and satisfies the additional requirement that h + h is a subalgebra of gc . A g-polarization at  is a subalgebra h of g that is a maximal isotropic subspace for the bilinear form B on g. Let h be a polarization at  ∈ g∗ . Define subalgebras e and d of g by   d = (h ∩ h) ∩ g, e = h + h ∩ g. Since the subalgebras h + h and h ∩ h are real (closed under conjugation) then ec = h + h and dc = h ∩ h. Note that d = h ∩ g is the largest subalgebra of g that is contained in h, and the elements of e are just the real (and imaginary) parts of elements of h. Observe also that if h is real, then d = e, and h ∩ g is a g-polarization. An irreducible representation associated to the coadjoint orbit of  will be obtained explicitly as a subrepresentation of the unitary representation induced from a closed – but not necessarily connected – subgroup D of G whose Lie algebra is d. Our first task will be to identify the desired closed subgroup D. Consider three examples. Example 4.2.2 Let g be the Heisenberg Lie algebra with basis (Z,Y,X) as Example 1.1.5, and let g∗ have the dual basis. Put h = spanC {Z,Y }. Let  = λZ ∗ with λ = 0. Since h is a real subalgebra of gc then d = e = h ∩ g and in order to see that h is a polarization at  it is enough just to see that h is maximal isotropic for B . Since h is commutative and B (X,Y ) = λ = 0, then this is clear.  Example 4.2.3 Fix a ∈ R and let g have the basis (Z,Y,X,A) where {Z,Y,X} spans the Heisenberg Lie algebra as above, and [A,X] = X − aY , [A,Y ] = aX + Y , [A,Z] = 2Z. Again let  = λZ ∗ with λ = 0. Here we have s() = {0}, so a polarization at  must have dimension two. The subalgebra h = spanC {Z,Y } is a polarization at  and since h is real, then h = ec = dc . The ideal h = spanC {Z,X + iY } is also a polarization, and here ec = spanC {X,Y,Z} while dc = CZ. Observe that h = spanC {X + iY,A}

4.2 Polarizations

279

is a maximal isotropic subalgebra of s, but not a polarization since h + h =  spanC {Y,X,A} is not a subalgebra of s. Example 4.2.4 Let g be the diamond Lie algebra with usual basis (Z,Y,X,A) as in Example 1.3.13 so that again {X,Y,Z} spans the Heisenberg Lie algebra as above, and in the complexification, [A,X + iY ] = −i(X + iY ). Let g∗ have the dual basis and let  = Z ∗ . Let h = spanC {Z,X+iY,A}; then as in Example 4.1.9 h is a subalgebra of gc , and h is maximal isotropic. Moreover h + h = gc ; hence h is a polarization at . Note that here d = g() = spanR {Z,A} and e = g.  Now fix  ∈ g∗ . Recall that if k is a real (resp. complex) subspace of g (resp. gc ), then k is the orthogonal complement of k in g (resp. gc ) with respect to the alternate form B . The following algebraic facts are easily verified for k and l subspaces of g, or complex subspaces of gc . 

(a) k = k , so if k is real (that is, if k = k), then k is real. (b) If k is an ideal, then k is a subalgebra. (c) If k is a real subspace of g (resp. complex subspace of gc ) then (k ) = k + g() (resp. (k ) = k + g()c ). (d) k = (k)⊥ and hence (k )⊥ = k. (e) If k is an ideal, then for any s ∈ G, ks = Ad(s)k . (f) (k + l) = k ∩ l . Two further facts are now proved. Lemma 4.2.5 Let  ∈ g∗ and put m = [g,g] + g(). Then  + m⊥ = K where K = {s ∈ G : s|m = |m }. Proof: By definition of K we have K ⊂  + m⊥ . To see the opposite inclusion, note that the Lie algebra of K is k = m , and since g() ⊂ m, then k = (m ) = m. Hence (k)⊥ = m = k . But since [g,g] ⊂ m, then for X ∈ k, X ∈ m⊥ ⊂ [g,g]⊥ , so exp X =  + X. Hence  + m⊥ =  + k ⊂ K. Lemma 4.2.6 Let g be a solvable Lie algebra, let  ∈ g∗ , and let n be a subspace of g or gc . Then (n ∩ n ) = n + n . Proof: Let us prove the real case (n ⊂ g), the same argument works in the complex case (n ⊂ gc ). Applying (c) and (f) gives (n + n ) = n ∩ (n + g()).

280

Coadjoint orbits and polarizations

Since g() = g ⊂ n , then n ∩ (n + g()) = (n ∩ n ) + g(). Hence ((n + n ) ) = ((n ∩ n ) + g()) = (n ∩ n ) ∩ g = (n ∩ n ) . But since g() ⊂ n + n , then ((n + n ) ) = n + n ; so (n ∩ n ) = n + n . Now fix  ∈ g∗ and h a polarization at . Denote by D 0 and E 0 the connected Lie subgroups of G with Lie algebras d and e respectively. Recall the stabilizer subgroup G() = {s ∈ G : s = }; of course G() is closed, but G() is not necessarily connected in general (Example 4.1.7), and not necessarily contained in D 0 . In order to construct an irreducible unitary representation of G attached to the data  and h, it is crucial that we induce from a closed subgroup D that contains G(). Hence we invoke the first of several necessary assumptions for constructing irreducible unitary representations. Definition 4.2.7 Let G be a solvable Lie group with Lie algebra g, let  ∈ g∗ , and let h be a polarization in gc at . We say that h is G()-stable if for each s ∈ G(), Ad(s)h = h. Given a G()-stable polarization h, both d and e are also Ad(G())-invariant and so G() normalizes both D 0 and E 0 . Put D = G()D 0,

E = G()E 0 .

Then D and E are subgroups of G. Since g() ⊂ d ⊂ e, the Lie algebra of D (resp. E) is d (resp. e). Observe that if G were simply connected nilpotent (or even exponential), then the preceding would be unnecessary: in that case G() = exp g() is connected and already contained in D 0 , whence D = D 0 and E = E 0 . Example 4.2.8 Suppose G = E˜ 2 = C  R as in Example 4.1.7, with  = (r,0) (r > 0) with the notation of the example, the space h = spanc {X,Y } is clearly a polarization at , and it is real: d = h ∩ g = e = spanR {X,Y }. Therefore D 0 = C  {0} ⊂ E˜ 2 , but G() = {(z,t) : t ∈ 2π Z, Im(rz) = 0} = R  2π Z, therefore D = E = D 0 G() = C  2π Z.



Proposition 4.2.9 Let h be a polarization at . Then, with our notation, the following holds

4.2 Polarizations

281

(1) d = e and e = d. (2) D 0 is a closed subgroup of G. (3) Suppose that h is G()-stable. Then D is a closed subgroup of G, and D 0 is the connected component of the identity in D. Proof: Since h = h, then by (a) and Lemma 4.2.6, (h ∩ h) = h + h. For B on g × g, this gives d = e, and since g() ⊂ d, then e = (d ) = d. Let X ∈ d. Since [d,e] ⊂ e, for any Y ∈ e, we get  1 (exp X )(Y ) =  Y − [X,Y ] + [X,[X,Y ]] + · · · 2  k (−1) + (adX)k (Y )) + · · · = (Y ). k! It follows that x(Y ) = (Y ) holds for any x ∈ D 0 , and by continuity, for any x ∈ D 0 . Therefore for any X belonging to the Lie algebra of D 0 , (X)(Y ) = 0, and so X ∈ e = d. Thus both D 0 and D 0 have the same Lie algebra, and D0 = D0. Finally, suppose that h is G()-stable; then G() normalizes D 0 and D is a subgroup of G. Now for any x ∈ D, x ∈  + e⊥ and again by continuity this holds also if x belongs to the closure D of D. As above it follows that the Lie algebra of D is e = d, and the connected component of the identity in D is D 0 . This proves also that D is an open subgroup of D, thus D is a closed subgroup of D and D = D. We continue with our assumption that g is solvable,  ∈ g∗ , and h is a G()-stable polarization at . Then D is a closed subgroup of G, and hence a closed subgroup of E, and the quotient space E/D has a unique smooth manifold structure. Let π : E → E/D be the quotient map and for X ∈ e let ζ X = dπe (X). Then ker(dπ )e = d, and thus the tangent space at the identity coset π(e) is canonically identified with e/d (see Result 1.6.9). Moreover for x ∈ E, the tangent space Tx (E) is dLx e (here Lx y = xy,y ∈ E) and Tπ(x) (E/D) = (dπ )x (dLx e). Recall that the complexification of e/d is identified with ec /dc . Extending the maps dLx and (dπ )x to ec as usual, we see that the complexification of Tπ(x) (E/D) is (dπ )x (dLx ec ). Since dc = h ∩ h, then   e/d c = ec /dc  h/dc ⊕ h/dc . We use these elementary observations to define a complex structure on the manifold E/D. In what follows we denote elements in ec /dc by lower-case roman letters, writing u = dπe (U ), v = dπe (V ), for U , V in ec .

282

Coadjoint orbits and polarizations

First observe that by part (1) of Lemma 4.2.9, d is the radical of the restriction of the alternate form B to e. Thus B defines a nondegenerate alternate bilinear form on e/d, which we denote simply by B. Note that B extends by linearity to ec /dc , and for u,v ∈ h/dc – or for u,v ∈ h/ dc – B(u,v) = 0. Next define J : e/d → e/d to be the linear map whose extension to ec /dc is multiplication by the constant −i on the subspace h/dc , and +i on the subspace h/dc . Lemma 4.2.10 B is J -invariant. Proof: Let u,v ∈ ec /dc and write u = u1 +u2,v = v1 +v2 where u1,v1 ∈ h/dc and u2,v2 ∈ h/dc . Then B(J u,J v) = B(J u1 + J u2,J v1 + J v2 ) = B(−iu1 + iu2, − iv1 + iv2 ) = B(iu2, − iv1 ) + B(−iu1,iv2 ) = B(u2,v1 ) + B(u1,v2 ) = B(u1 + u2,v1 + v2 ) = B(u,v). Recall (see Kobayashi, Nomizu [56]) that an almost complex structure on a manifold M is a field (Jm )m∈M where Jm : Tm (M) → Tm (M) is a linear map satisfying Jm2 = −1 for all m ∈ M. Equivalently, J can be defined by the distribution m → Fm where Fm is the eigenspace with eigenvalue −i for Jm in the complexification Tm (M)c of the tangent space. An almost complex structure defines a complex structure on M if and only if the complex distribution F is involutive: for any pair of complex vector fields ζ , ζ  , such that Jm ζm = −iζm , Jm ζm = −iζm for all m, the bracket [ζ,ζ  ]m is still in Fm (see Kobayashi and Nomizu [56, theorem 2.8]). In this case, we say that J is integrable. We now show how h defines an involutive distribution on E/D. For x ∈ E, the idea is to define Fπ(x) = (dπ )x (dLx (h)). To show that Fπ(x) is well-defined, let y ∈ E such that π(y) = π(x), so that y = xd for some d ∈ D. Now the assumption that h is G()-stable is used: since D = G()D 0 and D 0 is the analytic subgroup of G corresponding to d, then Ad(d)ec = ec and Ad(d)h = h. A straightforward computation then shows that for all W ∈ ec ,     (dπ )y dLy W = (dπ )x dLx (Ad(d)W ) . Since Ad(d)h = h, then (dπ )x (dLx (h)) depends only upon the coset π(x). Putting Fπ(x) = (dπ )x (dLx (h)) and F π(x) = (dπ )x (dLx (h)), we get Tπ(x) (E/D)c = Fπ(x) ⊕ F π(x) .

4.3 Admissibility and the Pukanszky condition

283

Define Jπ(x) : Tπ(x) (E/D) → Tπ(x) (E/D) to be the unique real linear map whose extension to the complexification Tπ(x) (E/D)c is multiplication by −i on Fπ(x) and +i on F π(x) . Then J 2 = −1, and we have an almost-complex structure on E/D. Proposition 4.2.11 The complex distribution F = (Fπ(x) ) is involutive and hence together with J = (Jπ(x) ), defines a complex structure on E/D. Proof: By E-invariance, it is sufficient to prove that if ζ and ζ  are two vector  ∈ Fπ(x) for any x, then their bracket [ζ,ζ  ]|π(e) fields such that ζπ(x), ζπ(x) computed at the origin is in Fπ(e) . Choose then a supplementary subspace m for d in e. There are neighborhoods of π(e) in E/D and 0 in m which are diffeomorphic. We can even restrict ourselves to the case ζπ(x) = dπx (dLx Z),  = dπx (dLx Z  ), with Z = X + iY , Z  = X + iY  in h, and X, X , Y , ζπ(x) and Y  in m. In this case, [ζ,ζ  ]|π(e) = dπe ([X,X  ] − [Y,Y  ]) + idπe ([X,Y  ] + [Y,X ]) = dπe ([Z,Z  ]) ∈ dπe (h) = Fπ(e) . This proves (Fπ(x) ) is a E-invariant complex distribution on the manifold E/D.

4.3 Admissibility and the Pukanszky condition 4.3.1 The Pukanszky condition Let G be a solvable Lie group with Lie algebra g. In the preceding section, it was shown that for a polarization h ⊂ g at  ∈ g∗ , with D = D 0 G(), E = E 0 G(), the property “h is G()-stable” allows that D is a closed subgroup of G with Lie algebra d, and E/D has a canonical structure of a (connected) complex manifold. In order to use the data , h, and D to construct an irreducible unitary representation of the group G, it is necessary to examine the suborbit D inside the coadjoint orbit of , and then to introduce two additional properties of polarizations. Lemma 4.3.1 Let  ∈ g∗ , h a polarization at , d = h ∩ g, D 0 the connected subgroup of G with Lie algebra d, and D = G()D 0 . Then D is an open subset of  + e⊥ . Proof: By part (1) of Proposition 4.2.9, d = e . For Z ∈ d and Y ∈ e,   1 (exp Z)(Y ) =  Y − [Z,Y ] + [Z,[Z,Y ]] − · · · = (Y ). 2

284

Coadjoint orbits and polarizations

Hence exp Z ∈  + e⊥ and it follows that D = D 0  ⊂  + e⊥ . To see that D is open in  + e⊥ , observe that since h is a maximal isotropic subspace for B , dim(h/gc ()) = 12 dim(gc /gc ()) and dim h = 12 (dim gc + dim gc ()). This gives dim dc + dim ec = 2 dim h = dim gc + dim(gc ()). Or for the real Lie algebras: dim d + dim e = dim g + dim(g()) and dim e⊥ = dim g − dim e = dim d − dim(g()) = dim D. Observe that if h is real and H is the connected subgroup of G with Lie algebra h, and if also G() is connected, then the preceding lemma just says that H  is an open subset of  + h⊥ . This is the case in the following two simple examples where G is exponential. Example 4.3.2 Let g be the Heisenberg Lie algebra with basis (Z,Y,X) as Example 1.1.5, and recall the polarization h = span{Z,Y } at  = λZ ∗ with λ = 0. Now D = H = exp h, and an easy computation shows that exp tY  =  λZ ∗ + tλX∗ . Hence D =  + RX∗ =  + e⊥ . The preceding example illustrates the following corollary: Corollary 4.3.3 Suppose G is connected and nilpotent, let  be in g∗ , then for any polarization h at , h is G()-stable and D =  + e⊥ . Proof: Since g() ⊂ h ∩ g = d, then it is clear that Ad(exp Z)h = h for all Z ∈ g(). But since G is nilpotent, its action is of exponential type and G() = G()0 = exp g(). Hence h is G()-stable. The above also shows that G() ⊂ D 0 so D 0 = D. Now by Lemma 4.3.1, D is an open subset of  + e⊥ . But since the restriction of the coadjoint action of G to D is a unipotent action of D in the vector space g∗ , the orbit D is a closed subset of g∗ (see Pukanszky [79], this fact is also a consequence of Proposition 2.3.5 and Remark 2.3.6). Hence D is both open and closed in  + e⊥ , meaning that D =  + e⊥ . Example 4.3.4 Let g = aff(R) (see Example 4.1.6) with the usual basis (Y,A), where [A,Y ] = Y , put as before  = (1,0) and h = RA. Again h is

4.3 Admissibility and the Pukanszky condition

285

real and it is easy to check that h is a polarization at . Now D = H = exp RA and  + e⊥ = RY ∗ . But here D is a proper open subset of  + e⊥ : exp(tA) (1,0) = (e−t ,0) = e−t Y ∗ .



In Examples 4.3.2 and 4.3.4,  determines a character χ on D by χ (d) = In Example 4.3.2, the subgroup D and character χ are associated to an irreducible unitary representation of the Heisenberg group by the induction procedure defined in Section 3.3. However, if the same procedure is applied to D and χ in Example 4.3.4, the result is a unitary representation that is not irreducible. This is precisely because in the second case, D =  + e⊥ . The importance of the property D =  + e⊥ was clearly understood by L. Pukanszky, who proved its necessity for irreducibility of the associated unitary representation in the case of real polarizations (where it is H  =  + h⊥ ) in exponential Lie groups [81].

eilog(d) .

Definition 4.3.5 Let h be a polarization at a point  ∈ g∗ , and suppose that h is G()-invariant. We say that h satisfies the Pukanszky condition if D =  + e⊥ . We have the following alternate expressions of the Pukanszky condition. Proposition 4.3.6 Let h be a polarization in  ∈ g∗ . Then the following are equivalent. (1) h satisfies the Pukanszky condition. (2) The orbit G of  contains  + e⊥ . (3) For any g ∈ e⊥ , h is a polarization at the point  + g. Proof: It is clear that if (1) holds, then  + e⊥ = D ⊂ G. Suppose that (2) holds and let g ∈ e⊥ . Then the space h is clearly isotropic for the alternate form B+g . Since h is a subalgebra, the only point to prove is that h is maximal isotropic for B+g , or equivalently, that dim(g( + g)) = dim(g()). Now by (2), there is s ∈ G such that  + g = s, thus g( + g) = Ad(s)g(), and dim(g( + g)) = dim(g()), proving (3). Suppose now that (3) holds. Observe that  + e⊥ is the union of the orbits 0 D ( + g) as g runs through e⊥ . As in the proof of Lemma 4.3.1, each orbit D 0 ( + g) is an open subset of  + e⊥ , and hence both D 0  and its complement are open in  + e⊥ . Since  + e⊥ is connected, then D = D 0  =  + e⊥ , and (1) is proved.

286

Coadjoint orbits and polarizations

A simple but important consequence of the Pukanszky condition is the following technical fact. Lemma 4.3.7 Let h be any polarization at  ∈ g∗ satisfying the Pukanszky condition. Then D 0 ∩ G() = G()0 . Proof: Since h satisfies the Pukanszky condition then D = D 0  =  + e⊥ is simply connected, and hence the stabilizer of  in D 0 is connected. But it is clear that this stabilizer is D 0 ∩ G(), and so D 0 ∩ G() ⊂ G()0 . Now g() ⊂ d implies G()0 ⊂ D 0 and the lemma follows.

4.3.2 Admissibility A second desirable condition is that a polarization should be conveniently related to a real ideal in gc containing [g,g]. Given any ideal n in g, recall that nc = {Z ∈ gc : [X,nc ] = 0} is a subalgebra of gc , and the complexification of n = {X ∈ g : [X,n] = 0}. Let h be a polarization at  ∈ g∗ ; we fix the following notation: h1 = h ∩ nc, h2 = h ∩ nc, and ei = (hi + hi ) ∩ g, di = hi ∩ hi ∩ g, i = 1,2. Of course for i = 1,2, (ei )c = hi + hi , (di )c = hi ∩ hi . The reader will easily verify from the definitions that d1 and d2 are subalgebras of g, and that d1 = d ∩ n, d2 = d ∩ n . It is also easy to check that [e2,e1 ] ⊂ e1 and [d2,d1 ] ⊂ d1 , since n is an ideal. Further properties of these objects are proved in the following. Proposition 4.3.8 Let 1 = |n and suppose that h1 is maximal isotropic in nc for the bilinear form B1 = B |nc ×nc . Then the following hold. (1) (2) (3) (4)

h = h1 + h2 and e = e1 + e2 . e1 = e ∩ n, and e2 = e ∩ n ; in particular, e1 and e2 are subalgebras of g. d = d1 + d2 . e2 = n + d2 .

Proof: We begin with an observation: by virtue of the maximality assumption for h1 , and by definition of h2 , we have nc ∩ nc = n(1 )c ⊂ h1 ∩ h2 . Hence n ∩ n ⊂ h and by Lemma 4.2.6 h = h ⊂ (n + n )c .

4.3 Admissibility and the Pukanszky condition

287

To see that h = h1 + h2 , let Y ∈ h, and write Y = X + Z where X ∈ nc and Z ∈ nc . Since Y and Z both belong to h1 then so does X, and hence by our maximality assumption, X ∈ h1 . But now Z ∈ h ∩ nc = h2 . Thus Y ∈ h1 + h2 and h = h1 + h2 . Next put e1 = e ∩ n and e2 = e ∩ n . It is immediate from the definitions that ei ⊂ ei , i = 1,2 and e1 + e2 ⊂ e1 + e2 ⊂ e. But if Y ∈ e, then Y = Re W where W ∈ h, and by the preceding argument, W = W1 + W2 with W1 ∈ h1 and W2 ∈ h2 . Hence Y = Re W1 + Re W2 ∈ e1 + e2 and we have e1 + e2 = e1 + e2 = e proving (1). Moreover, e1 ∩ e2 = e1 ∩ e2 since e1 ∩ e2 ⊂ e1 ∩ e2 , and if Z ∈ e1 ∩ e2 , then Z ∈ nc ∩ nc ∩ g ⊂ h1 ∩ h2 ∩ g ⊂ e1 ∩ e2 . Hence dim e1 + dim e2 = dim e1 + dim e2 so e1 = e1 and e2 = e2 , proving (2). To show that d = d1 + d2 , we claim first that h2 is totally isotropic in nc for the restriction B2 of B to nc × nc . Let X ∈ nc ∩ h2 . By definition of nc , and h1 , we have nc ⊂ h1 , so, by part (1), X ∈ (h1 + h2 ) = h = h and hence X ∈ h2 . This proves the claim. By Proposition 4.2.9, d = e and e = d . Let Z ∈ d; using point (1), e = e1 + e2 , we write Z = X + Y where X ∈ n and Y ∈ n . For any T ∈ e1 , [Y,T ] = 0 and [Z,T ] = 0 and hence X ∈ e1 ∩ n = d1 . Similarly, thanks to our claim, Y ∈ e2 ∩ n ⊂ d2 and we have (3). Finally, since n ∩ n ⊂ e2 , then, by Lemma 4.2.6, e2 ⊂ n + n . Write Z ∈ e2 as Z = X + Y with X ∈ n, Y ∈ n . By definition of e2 , X ∈ e2 and hence Y = Z − X ∈ e2 ∩ n = d2 . Definition 4.3.9 A polarization h at a point  ∈ g∗ is admissible for the ideal n if h ∩ nc is a polarization for the restriction of  to n. Corollary 4.3.10 Let n be an ideal in g and h a polarization at  ∈ g∗ . Suppose that h ∩ nc is maximal isotropic for the restriction of B to nc × nc . Then h is admissible for n: h1 = h ∩ nc is a polarization at 1 = |n and h2 = h ∩ nc is a polarization at 2 = |n . Moreover, if h is G()-stable, then both h1 and h2 are G()-stable. Proof: Observe that part (2) of the preceding proposition shows that h is admissible for n. Note also that h2 and (e2 )c are subalgebras, and by the claim in the preceding proof, h2 is totally isotropic in nc , thus by part (b), h2 is a polarization at 2 in the subalgebra n .

288

Coadjoint orbits and polarizations

Suppose now that h is G()-stable, then for each s ∈ G(), by the connectivity of G, Ad(s)n = n, so Ad(s)h1 = h1 and Ad(s)n = ns = n , hence Ad(s)h2 = h2 . Next, we prove that admissibility and G()-stability for a polarization are sufficient for most purposes. Proposition 4.3.11 Let G be a solvable Lie group with Lie algebra g, let n be a nilpotent ideal in g that contains [g,g]. Fix a point  ∈ g∗ and let h be a polarization at  such that h is G()-stable and admissible for n. Then h satisfies the Pukanszky condition, and the subgroup E is closed. Proof: Put f = |n and D10 = exp d1 the connected subgroup of N = exp n with Lie algebra d1 . To prove the first statement, we show that  + e⊥ ⊂ G. Now since N is connected and nilpotent, h1 does satisfy the Pukanszky 0 ⊥ condition for f so f + e⊥ 1 = D1 f (Corollary 4.3.3). Hence, given g ∈  + e , 0 ⊥ there is d1 ∈ D1 such that d1 g|n = f , and hence d1 g ∈  + (n + e) . Now [g,g] ⊂ n and g() ⊂ e. Recalling the ideal m = [g,g] + g() of Lemma 4.2.5, we have m ⊂ n + e. Hence  + (n + e)⊥ ⊂  + m⊥ . By Lemma 4.2.5, we have  + m⊥ ⊂ G, so this proves the first statement. For the second, put E10 = exp e1 and let E20 be the connected subgroup with Lie algebra e2 . Recall that [e2,e1 ] ⊂ e1 ; it follows that E10 is a closed normal subgroup of E 0 and E20 E10 is an open subgroup in the connected group E 0 , thus: E10 E20 = E 0 . Hence E = E10 E20 G(). First, if A ∈ e2 , then: 1 exp A =  + A + A(A) + · · · =  + A. 2 Recall now that e2 = (e2 )⊥ , thus e2  = (e2 )⊥ = (n + d2 )⊥ . Hence E20  =  + e2  =  + (e2 )⊥ =  + (n + d2 )⊥ .

(4.3.1)

Now n + d2 is an ideal in g, thus (n + d2 )⊥ is an invariant subspace of g∗ and we can look at the action of G in g∗ /(n + d2 )⊥ . For s ∈ E, s = e1 e2 z with ei ∈ Ei0 , z ∈ G(), we have s( + (n + d2 )⊥ ) = e1 e2  + (n + d2 )⊥ = e1 ( + (n + d2 )⊥ ) ∈ E10  + (n + d2 )⊥ .

4.4 Isotropic subspaces associated to a flag

289

Or: E + (n + d2 )⊥ = E10  + (n + d2 )⊥ . Observe that the action of N on g∗ is unipotent, thus the same holds for the quotient action on g∗ /(n+d2 )⊥ , and for the E10 -action, therefore E+(n+d2 )⊥ is closed in g∗ /(n + d2 )⊥ . Next we show that E = {s ∈ G : s ∈ E + (n + d2 )⊥ }.

(4.3.2)

One inclusion is trivial: is s ∈ E, we saw that s ∈ E10  + (n + d2 )⊥ . For the reverse inclusion, suppose that s ∈ G and that s ∈ E + (n + d2 )⊥ = E10  + (n + d2 )⊥ . This means that there is e1 ∈ E10 such that s ∈ e1  + (n + d2 )⊥ . Hence e1−1 s ∈  + (n + d2 )⊥ . By (4.3.1), there is e2 ∈ E20 such that e1−1 s = e2 . Hence e2−1 e1−1 s =  so e2−1 e1−1 s ∈ G() and s ∈ E. Thus (4.3.2) is verified. But this implies that E is closed, since if sn → s with sn ∈ E, Then sn  → s, the projection map from g∗ to g∗ /(n + d2 )⊥ is continuous and open, so (sn  + (n + d2 )⊥ ) is converging to s + (n + d2 )⊥ in the closed set E( + (n + d2 )⊥ ), thus s ∈ E + (n + d2 )⊥ and by Relation (4.3.2), s ∈ E and E is closed.

4.4 Isotropic subspaces associated to a flag Let g be a solvable Lie algebra and let  ∈ g∗ . We have seen that the alternate form B plays a central role in the study of coadjoint orbits and polarizations and its matrix M() is the matrix of the action Z → Z studied in Section 2.2. We have also seen – or at least asserted – that a number of special properties of polarizations are desired. The following question arises then: Is it always possible to find a desirable polarization at the point  in general? Toward the end of providing an explicit positive answer to this question, we describe in this section an application of the flag normal form of a matrix that results in an explicit construction of a maximal isotropic subspace that is well-adapted to a given flag. Let M be a skew-symmetric n × n matrix and let N (e,h,c) = S −1 MT be its strict flag normal form. Recall that T is a unipotent upper-triangular matrix, S is a unipotent lower-triangular matrix, and the nonzero entries of N(e,h,c) = [ni,j ] are the entries ne,h(e) = c(e) with e ∈ e. We now examine

290

Coadjoint orbits and polarizations

special properties of these objects when M is skew-symmetric. Note that for any flag normal form matrix, its transpose is given by t

N (e,h,c) = N (h(e),h−1,c ◦ h−1 ).

Since M is skew-symmetric, the relation of strict equivalence is also obtained by     M = − t M = t T −1 − t N (e,h,c) t S = t T −1 N (h(e),h−1, − c ◦ h−1 ) t S. By uniqueness of the strict normal form of a matrix, we get h(e) = e, h ◦ h = ide and c ◦ h = −c. Hence N (e,h,c) is skew-symmetric, and in particular h(e) = e for all e ∈ e. Given a flag normal form equivalence class F(e,h) of n × n matrices, the subset Fskew (e,h) of all skew-symmetric elements of F(e,h) consists of those for which the function c satisfies c ◦ h = −c. Write e = e+ ∪ e− where e+ = {e ∈ e : e < h(e)} and e− = {e ∈ e : e > h(e)}. Thus the jump index function h : e → e is an involution satisfying h(e+ ) = e− . As a consequence, |e| = d is even, and we write e+ = {i1 < i2 < · · · < id/2 },

e− = {j1,j2, . . . ,jd/2 },

with jk = h(ik ). We now show that for skew-symmetric matrices, the flag normal form can be obtained symmetrically. Proposition 4.4.1 Let M be a skew-symmetric matrix. Then there is a unipotent upper-triangular matrix T = T (M) such that N (e,h,c) = t T MT . Moreover, the map M → T (M) is rational and nonsingular on F(e,h). Proof: The required matrix T is built recursively. 1. Observe that by definition of i1 = min e, the first i1 − 1 rows are identically zero. Then j1 = h(i1 ) means that the first j1 − 1 entries in the row i1 are also zero, but the entry ai1,j1 = c1 is nonzero. By skew-symmetry, the first nonzero column is therefore the i1 -column, and the first nonzero entry in this column is aj1,i1 = −c1 . We define the matrix T1 by the following column-operations: for any j , M j → M j , aj ,j M j → M j + 1 M i1 , c1 aj1,j i1 ai1,j j1 j j M → M + M − M , c1 c1

(j ≤ i1 ), (i1 < j ≤ j1 ), (j1 < j ).

4.4 Isotropic subspaces associated to a flag

291

The result of the column operations above is that in the matrix MT1 , the i1 -row has exactly one nonzero entry in the j1 -column, whose value is c1 . Also, the j1 -row has a unique nonzero entry which is in the i1 -column, with value −c1 . Note that the i1 - and j1 -columns are unchanged in MT1 . Now put M(1) = t T1 MT1 . The effect of multiplying on the left by t T1 is that the same operations performed on the columns by T1 are now performed on the rows. The result is that the rows i1 and j1 do not change, but the i1 - and j1 -columns of M(1) have exactly one nonzero entry, with values −c1 and c1 , respectively. Observe that the entries of T are rational functions, nonsingular on F(e,h). Now if d = rank M = rank M(1) = 2 then we claim that M(1) is the strict flag normal form for M. To see this, suppose that M(1)i is a row of M(1) with i ∈ / {i1,j1 }. Then Mi is a linear combination of Mi1 and Mj1 , hence all of its entries are zero except possibly in the i1 and j1 -columns. But all entries in the j1 -column (resp. the i1 column) are zero except for the entry in the i1 row (resp. the j1 row), so Mi = 0. This proves the claim. Suppose then that d > 2. Let M(1)i2 be the first row in M(1) which is linearly independent of the row M(1)i1 . If i1 < i < i2 , then arguing as in the proof of the preceding claim we see that M(1)i is identically zero. Since M(1) is skew-symmetric, then the same can be said of the columns: if i1 < i < i2 then M(1)i is identically zero. 2. Suppose we have built a skew-symmetric matrix M(k) = t (T1 . . . Tk ) M(T1 . . . Tk ), with only one nonzero entry in each column and row with indices i1, . . . ,ik and j1, . . . ,jk , and where M → T1 T2 · · · Tk is rational and nonsingular on F(e,h). If M(k) has rank 2k, then M(k) is the strict flag normal form for M. Indeed, if M(k)i is a row distinct from the rows M(k)ir ,M(k)jr ,1 ≤ r ≤ k, but belonging to their span, then every entry in M(k)i is zero except in an ir - or jr -column. But the i-th entry in any such column is zero, forcing M(k)i = 0. Assuming then that d > 2k, the first row in M(k) which is independent of the rows i1, . . . ,ik is ik+1 . As above, this implies that the rows M(k)i , / {i1, . . . ,ik }, are identically zero. In the ik+1 -row, the for i < ik+1 and i ∈ (k) entries aik+1,j are vanishing for j = j1, . . . ,jk , and j = i1, . . . ,ik . The first nonzero entry is in jk+1 , and its value is ai(k) = ck+1 . Imitating the k+1,jk+1 definition of T1 , we define Tk+1 by the column operations

292

Coadjoint orbits and polarizations

M(k)j → M(k)j , M(k)j → M(k)j +

(j ≤ ik+1 ), (k) ajk+1,j

ck+1

(ik+1 < j ≤ jk+1 ),

M(k)ik+1 ,

(k)

M(k) → M(k) + j

j

ajk+1,j ck+1

(k)

M(k)

ik+1

−

aik+1,j ck+1

M(k)jk+1 ,

(jk+1 < j ). (4.4.1)

Now put M(k + 1) = k+1 M(k)Tk+1 ; the rows and columns of M(k + 1) indexed by i1, . . . ,ik+1 and j1, . . . ,jk+1 contain a unique nonzero entry. It is clear that M → T1 T2 · · · Tk+1 is rational and nonsingular on F(e,h). 3. By definition, the matrix M(d/2) obtained at the k = d/2 step is N(e,h,c). This proves that, for any k, ck = c(ik ). It is clear from the expressions (4.4.1) that the entries in the matrix T = T1 . . . Td/2 are rational functions of the entries of M, with denominator nonvanishing on the class FC(e,h). tT

Note that if M(k) has flag normal form for some k < d/2, then Tr = I for k < r ≤ d/2. Example 4.4.2 Consider the skew-symmetric matrix: ⎡ ⎤ 0 0 1 2 ⎢0 0 3 1⎥ ⎥ M=⎢ ⎣−1 −3 0 −1⎦ . −2 −1 1 0 Then i1 = 1 and j1 = 3. We perform the column operations M 2 → M 2 −3M 1 and M 4 → M 4 + (−1)M 1 − 2M 3 , and then the corresponding row operations, to obtain ⎡ ⎤ 0 0 1 0 ⎢ 0 0 0 −5⎥ ⎥ M(1) = t T1 MT1 = ⎢ ⎣−1 0 0 0 ⎦ 0 where

⎡

1 ⎢0 T1 = ⎢ ⎣0 0

−3 1 0 0

5 0

0

⎤ 0 −1 0 0⎥ ⎥. 1 −2⎦ 0 1

Since M(1) is already in a strict flag normal form then T2 = I and M(1) = M(2) = N(e,h,c). 

4.4 Isotropic subspaces associated to a flag

293

Proposition 4.4.1 is now applied to the study of alternate bilinear forms. Let E be a vector space over k, and let B be an alternate bilinear form on E. Recall that for any subset H of E, H ⊥B = {Z ∈ E : B(Z,Y ) = 0,for all Y ∈ H }. It is easily seen that for subspaces H and K of E, (H + K)⊥B = H ⊥B ∩ K ⊥B . Fix an ordered basis (Zj ) for E with corresponding flag (Ej ) given by Ej = span{Z1, . . . ,Zj }. Let M = [B(Zi ,Zj )]1≤i,j ≤n be the matrix for B, and let N = N(e,h,c) = t T MT be the unique strict flag normal form for M, with T the n × n unipotent upper-triangular matrix constructed in Proposition 4.4.1. Observe that N = [B(T Zi ,T Zj )]1≤i,j ≤n . For each 1 ≤ i ≤ n, let B (i) be the alternate form on Ei that is the restriction of B; the matrix of B (i) for the basis (Z1, . . . ,Zi ) of Ei is clearly the i × i principal submatrix of M. For simplicity of notation, denote by M (i) the 1,...,i of any n × n matrix M. Observe that M (i) is the principal submatrix M1,...,i matrix of B (i) for the basis (Z1, . . . ,Zi ) of Ei , and that since T is unipotent upper triangular, the relation N (i) = t T (i) M (i) T (i)

(4.4.2)

holds for each 1 ≤ i ≤ n. It is clear that N (i) is the strict flag normal form for M (i) , with N = N (e(i),h|e(i) ,c|e(i) ) where e(i) := e(i,i) = {e ∈ e : e ≤ i and h(e) ≤ i}. It is also clear that for any subspace H of Ei then H ⊥B (i) = H ⊥B ∩ Ei . We will use the map T to compute orthogonal complements of certain subspaces with respect to the bilinear form B; first we prove a simple lemma. Lemma 4.4.3 Let S ⊂ {1,2, . . . ,n}, put S c = {1,2, . . . ,n} \ S, and put ES = span{Zj : j ∈ S}. Then for any unipotent upper-triangular matrix T , E = T ES ⊕ ES c .

294

Coadjoint orbits and polarizations

Proof: Define the new matrix T  by T  Zj = T Zj if j ∈ S and T  Zj = Zj if j ∈ S c . Clearly, the matrix T  is unipotent and upper triangular. Hence it is invertible and: E = T  ES ⊕ T  ES c = T ES ⊕ ES c . Let us now compute orthogonal complements of subspaces spanned by subsets of the chosen basis. Lemma 4.4.4 Let S ⊂ {1,2, . . . ,n} and put HS = T ES = span{T Zj : j ∈ S}. Then HS⊥B = span{T Zj : j ∈ / h(e ∩ S)}. More generally, for each 1 ≤ i ≤ n, if S ⊂ {1, . . . ,i} then ⊥B (i)

HS

= span{T Zj : j ≤ i, j ∈ / h(e(i) ∩ S)}.

Proof: If j ∈ / e, then the row Nj is zero which means that for all 1 ≤ i ≤ n, B(T Zi ,T Zj ) = 0 and hence (T Zj )⊥B = E. If j ∈ e, then the only nonzero entry in row Nj is nj,h(j ) so (T Zj )⊥B = span{T Zi : i = h(j )}. Thus for S ⊂ {1,2, . . . ,n}, HS⊥B = (T Zj )⊥B = (T Zj )⊥B = span{T Zj : j ∈ / h(e ∩ S)}. j ∈S

j ∈e∩S

The second assertion in the lemma is an application of the first assertion to the subspace Ei . Remark 4.4.5 Suppose that S = {i1,j1, . . . ,ir ,jr }, then it easy to prove by induction on r that for each j ∈ S, T Zj = T1 . . . Tr Zj = Zj +

r−1

at Zit + bt Zjt .

t=1

Therefore ES = span{Zj : j ∈ S} = HS = span{T Zj : j ∈ S} and: / S}. ES⊥B = span{T Zj : j ∈ A simple consequence of the above is the following. Corollary 4.4.6 The radical of B is E ⊥B = span{T Zj : j ∈ / e}. Proof: Apply the preceding lemma with S = {1,2, . . . ,n}, and recall that h(e) = e.

4.4 Isotropic subspaces associated to a flag

295

Recall that e = e+ ∪ e− as a disjoint union where e+ = {e ∈ e : e < h(e)} and e− = {e ∈ e : e > h(e)}. Write e+ = {i1 < i2 < · · · < id/2 } and e− = {j1,j2, . . . ,jd/2 } where jk = h(ik ), 1 ≤ k ≤ d/2. Finally denote by ec the indices in {1,2, . . . ,n} that do not belong to e. It is now a simple matter to compute a maximal isotropic subspace that will be well-adapted to the chosen flag. Proposition 4.4.7 Put H (i) = span{T Zj : j ∈ ec ∪ e+,j ≤ i}, 1 ≤ i ≤ n with H = H (n) . Then for each 1 ≤ i ≤ n, the subspace H (i) ⊂ Ei is a maximal totally isotropic subspace for B (i) . Moreover,

 H (i) = nullspace M (i ) . (4.4.3) i  ≤i

Proof: Observe that H = HS where S = ec ∪ e+ so h(e ∩ S) = h(e+ ) = e− . Lemma 4.4.4 now gives H ⊥B = span{T Zj : j ∈ / e− } = H . For H (i) the proof is similar: H (i) = HS for S = (ec ∪ e+ ) ∩ {1,2, . . . ,i}, e(i) ∩ S = {e ∈ e+ : e ≤ i and h(e) ≤ i}. Hence h(e(i) ∩ S) = {jk ∈ e− : jk ≤ i}. Hence j ≤ i and j ∈ / h(e(i) ∩ S) holds c + if and only if j ≤ i and j ∈ e ∪ e , if and only if j ∈ S. Thus by Lemma 4.4.4, (H (i) )⊥B (i) = H (i) . Now apply Corollary 4.4.6 to the alternate form B (i) on Ei to compute rad B (i) = nullspace M (i) = span{T Zj : j ≤ i,j ∈ / e(i) }. 

/ e(i ) if and only if j ≤ i  Now let i  such that i  ≤ i, thus j ≤ i  and j ∈ c +  and either j ∈ e or j ∈ e with h(j ) > i . Hence by definition of H (i) ,    nullspace M (i ) ⊂ H (i) for all i  ≤ i and i  ≤i nullspace M (i ) ⊂ H (i) . On the other hand, if j ∈ ec ∪ e+ and j ≤ i, then j < h(j ) so there is i  ≤ i such   / e(i ) so T Zj ∈ nullspace M (i ) . The that j ≤ i  < h(j ). This means that j ∈ formula (4.4.3) follows. As in Lemma 2.1.4, we can use the matrix T to define projections onto the maximal isotropic subspaces H (i) that are rationally dependent upon the entries in M. The following is immediate.

296

Coadjoint orbits and polarizations

Lemma 4.4.8 (1) E = rad B ⊕ Ee , and the projection onto rad B parallel to Ee is given by 2 / e, T Zj , if j ∈ Zj → 0, if j ∈ e. (2) E = H ⊕ Ee− and the projection onto H parallel to the supplementary space Ee− is given by 2 / e or j ∈ e+, T Zj , if j ∈ Zj → 0, if j ∈ e− . Since the nullspaces of M (i) depend upon the flag (Ei )i , but not upon the choice of the basis (Zi )i , the same is true for the isotropic spaces H (i) . A useful direct characterization of these spaces is given as follows. Put H0 = E and for 0 ≤ k ≤ d/2 put   / {j1, . . . jk } , Hk = span T Zj : j ∈ so that E = H0 ⊃ H1 ⊃ · · · ⊃ Hd/2 = H . Since ik < jk for all k, observe that H (ik ) = H ∩ Eik = Hk ∩ Eik . Proposition 4.4.9 The sets e+ = i = {ik : 1 ≤ k ≤ d/2}, e− = j = {jk : 1 ≤ k ≤ d/2} of indices and the subspaces Hk are recursively defined by the relations ⊥B }, ik = min{1 ≤ j ≤ n : Hk−1 ∩ Ej ⊂ Hk−1 ⊥ Hk = (Hk−1 ∩ Eik ) B ∩ Hk−1 , and jk = min{1 ≤ j ≤ n : Hk−1 ∩ Ej ⊂ Hk }.

Proof: Denote ik , jk , Hk the indices and subspaces recursively defined in the proposition. Since E ⊥B = nullspace M, then i1 = min{j : Ej ⊂ nullspace M} is just the index of the first nonzero row of M, namely i1 . Moreover, since {1,2, . . . ,i1 } ∩ e = {i1 }, then Lemma 4.4.4 implies H1 = Ei⊥1 B = span{T Zj : j = j1 } = H1 . By definition of H1 , and / H 1 } = j1 . j1 = min{j : Ej ⊂ H1 } = min{j : T Zj ∈    = ik−1 , Hk−1 = Hk−1 , and jk−1 = jk−1 . Now for any i, Suppose now ik−1

/ {j1, . . . ,jk−1 }} Hk−1 ∩ Ei = span{T Zj : j ≤ i,j ∈

(4.4.4)

4.5 Fine layering and Vergne polarizations

297

while by Lemma 4.4.4, ⊥B = span{T Zj : j ∈ ec ∪ {i1,i2, . . . ,ik−1 }}. Hk−1

But if i < ik then by the properties of the indices e+,e− , {1, . . . ,i} \ {j1, . . . ,jk−1 } ⊂ ec ∪ {i1,i2, . . . ,ik−1 } ⊥B . Therefore ik ≤ ik . Since T Zik ∈ Eik ∩ Hk−1 \ and hence Ei ∩ Hk−1 ⊂ Hk−1 ⊥B Hk−1 then ik = ik . Now again by index properties

{1, . . . ,ik } \ {j1, . . . ,jk−1 } ⊂ ec ∪ {i1,i2, . . . ,ik }, hence by Lemma 4.4.4  ⊥ (Hk−1 ∩ Eik )⊥B = span{T Zj : j ∈ {1, . . . ,ik } \ {j1, . . . jk−1 }} B = span{T Zj : j ∈ / h({i1, . . . ,ik })} = Hk . Finally, the definition of Hk and the relation (4.4.4) show that for i < jk , Hk−1 ∩ Ei ⊂ Hk , while T Zjk ∈ Hk−1 ∩ Ejk , whence jk = jk .

4.5 Fine layering and Vergne polarizations 4.5.1 Fine layering for coadjoint action Let g be a solvable Lie algebra over R. In this section we apply the construction of the preceding section to the alternate forms B ,  ∈ g∗ . At the same time, we apply the layering methods of Section 2.2 to the coadjoint representation of the simply connected group G with Lie algebra g. Fix a good Jordan– Hölder sequence (si )ni=1 for s = gc with corresponding Jordan–Hölder basis (Zi )ni=1 for s and root sequence (λi ). Recall that this means that si = spanC {Z1, . . . ,Zi } is an ideal in gc and that if si is not real, then Zi+1 = Z i . Let g∗c have the dual basis (fi = Zi∗ ), with corresponding weight sequence (−λi ) and write i = pi () = (Zi ). Recall that the dual flag defined by Ui = s⊥ i = spanC {fi+1,fi+2, . . . ,fn }, 1 ≤ i ≤ n, is a good descending flag for the coadjoint action of g. We have the natural identifications of g∗c /Ui with span{f1, . . . ,fi } and with s∗i . With these identifications, the quotient map πi : g∗c → g∗c /Ui is just the restriction map  → |si . Since [Zi ,Zj ] = pi (ad∗ (Zj )), the skew-symmetric matrix M() for B with

298

Coadjoint orbits and polarizations

respect to the ordered basis (Zi ) is also the matrix of the map X → ad∗ X with respect to the bases (Zi ) and (fi ), as described in Section 2.2. Thus, for Z = z1 Z1 + z2 Z2 + · · · + zn Zn = t [z1,z2, . . . ,zn ] and  = t [1,2, . . . ,n ] with i = (Zi ), we have ad∗ (Z) = t [1,2, . . . ,n ] where ⎡ ⎤ ⎡ 1 0 ⎢ ⎥ ⎢[Z2,Z1 ] ⎢ 2⎥ ⎢ ⎢.⎥=⎢ .. ⎣ .. ⎦ ⎣ . n

[Zn,Z1 ]

⎤⎡ ⎤ z1 · · · [Z1,Zn ] ⎢ z2 ⎥ · · · [Z2,Zn ]⎥ ⎥⎢ ⎥ ⎥⎢ . ⎥. .. .. ⎦ ⎣ .. ⎦ . . zn [Zn,Z2 ] · · · 0 [Z1,Z2 ] 0 .. .

Recall that for simplicity we frequently denote the representation ad∗ of g, as well as the representation Ad∗ of the simply connected group G, multiplicatively. Note that if M() is the zero matrix, then s = s() = s, and  is a fixed point for the coadjoint representation of G. Suppose that M() = 0, and recall from Section 2.2 that the set of jump indices for  is the set: e = e() = {1 ≤ i ≤ n : rankM()1,...,i−1 < rankM()1,...,i } and the jump index function h : e() → e() is defined by: 1,...,j

1,...,j

h(e) = min{1 ≤ j ≤ n : rankM()1,...,e−1 < rankM()1,...,e }. Recall the following: / e()} is a basis of s(), |e()| = rankM() is even, and {T ()Zj : j ∈ e() = h(e()), and h2 (e) = e for any e ∈ e(), e() = {1 ≤ i ≤ n : s(πi−1 ()) = s(πi ())} = {1 ≤ i ≤ n : si−1 = si }, For each e ∈ e(), h(e) = e, and e+ = {e ∈ e : e < h(e)}, e− = e \ e+ , e+ = {i1 < i2 < · · · < id/2 }, e− = h(e+ ) = {j1, . . . ,jd/2 }, where jk = h(ik ), (6) δ = {e ∈ e() : s(πe−1 ()) ∩ ker λe = s(πe ())}, it follows from Lemma 2.2.14 that δ ⊂ e+ .

(1) (2) (3) (4) (5)

By virtue of the results of Section 2.2, we have the ultrafine layering {e,h,δ }. Recall that each e,h,δ is a G-invariant semialgebraic set. We consider two examples. Example 4.5.1 Return to the Heisenberg Lie algebra g = h1 ; the usual basis (Z,Y,X) is a Jordan–Hölder basis for gc . Since g is nilpotent, δ = ∅ in each layer. Identify g∗ with R3 , by using the dual basis, writing  = t [1,2,3 ]. The skew-symmetric matrix M() is

4.5 Fine layering and Vergne polarizations ⎡

0 0 M() = ⎣0 0 0 1

299

⎤ 0 −1 ⎦ . 0

If 1 = 0, then e() = {2,3}, h(2) = 3, h(3) = 2, while if 1 = 0 then e = ∅. Thus we have g∗ = e,h,∅ ∪ ∅,(),∅ , where e,h,∅ = { ∈ g∗ : 1 = 0}, ∅,(),∅ = { ∈ g∗ : 1 = 0}.



Example 4.5.2 Consider now the Mautner group G = C2  R. Its Lie algebra g has a good Jordan–Hölder basis (Z,Z,W,W,A), with bracket [A,Z] = iZ, [A,W ] = iαW (α real but irrational). Here we have  = (1,2 = 1,3,4 = 3,5 ) with 1 , 3 complex and 5 real. Observe that this group is not exponential, moreover the orbits in the minimal layer are not regular, however there is a natural cross-section for each of the other two layers. This gives the following table:

e

h

(e+,e− )

δ

e,h,δ

1 = 0

{1,5}

(5,1)

({1},{5})

{1}

Undefined

1 = 0, 3 = 0

{3,5}

(5,3)

({3},{5})

{3}

1 = 5 = 0,3 > 0

∅

()

(∅,∅)

∅

1 = 3 = 0

e,h,δ

1 = 3 = 0

Let us consider here the generic layer  defined by 1 = 0. We simply identify g∗ with C2 ×R by writing  = (1,3,5 ). A direct computation gives the orbit of : O = G = {(1 eit1 ,3 eiαt1 ,5 + t2 ) : t1,t2 ∈ R}. Arguing as in Example 1.9.8, if 1 and 3 are not vanishing, since α is not rational, then the orbit O is not regular, but if 1 = 0 and 3 = 0, the orbit is homeomorphic to a generic coadjoint orbit of E˜ 2 (see Example 4.1.7), and it is regular. Thus it is possible for an ultrafine layer to contain regular and nonregular orbits. Contrary to the case of exponential groups, there is no Borel cross-section for all the orbits in this layer , this was proved by J. Glimm and E. Effros, see [44, theorem 1], [36, theorem 2.9]. In the two other ultrafine

300

Coadjoint orbits and polarizations

layers, each orbit is regular, and it is easy to describe topological cross-section for each layer.  Example 4.5.3 Consider the Lie algebra g = span{Z,Y,X,W,A} with nonvanishing brackets: [X,Y ] = Z,

[W,X] = Y,

[A,W ] = 2W,

[A,X] = −X,

[A,Y ] = Y .

The vectors (Z,Y,X,W,A) form a real good Jordan–Hölder basis of s = gc . Using the dual basis, we identify g∗ with R5 , writing  = t [1, . . . ,5 ]. Then M() is ⎡

0 0 ⎢ ⎢0 0 ⎢ M() = ⎢ ⎢0 1 ⎢ ⎣0 0 0 2

0 −1 0

0 0 −2

2 −3

0 24

0 −2 3

⎤

⎥ ⎥ ⎥ ⎥. ⎥ ⎥ −24 ⎦ 0

Suppose first that 1 = 0, then i1 = 2 and j1 = 3, the matrices T1 () and 1 ()M()T1 () are

tT

⎡

1 0 0

0

⎢0 1 0 ⎢ ⎢ T1 () = ⎢ ⎢0 0 1 ⎢ ⎣0 0 0

2 1

0 1

0 0 0 ⎡

0 0 ⎢0 0 ⎢ ⎢ ⎢ t T1 ()M()T1 () = ⎢0 1 ⎢ ⎢0 0 ⎣ 0 0

0

⎤

0

− 31 ⎥ ⎥ ⎥ 2 ⎥ , − 1 ⎥ ⎥ 0 ⎦ 1

0 −1 0

0 0

0

0

0

0 0

0

24 +

22 1

⎤

⎥ ⎥ ⎥ ⎥ 0 ⎥. 22 ⎥ −24 − 1 ⎥ ⎦ 0

2

Then the first layer is given by 1 = 0, 24 + 21 = 0. In this fine layer, e = {2,3,4,5}, with i1 = 2, j1 = 3, and i2 = 4, j2 = 5. Clearly, δ = {4}, and the cross-section in this layer is:  = { : 1 = 0, 2 = 3 = 5 = 0, 4 = ±1}.

4.5 Fine layering and Vergne polarizations

301

Similarly, the results of our construction are the following: e

e,h,δ

h

e+

δ

e,h,δ

1 = 0, 21 4 + 22 = 0

{2,3,4,5} (3,2,5,4) {2,4} {4}

1 =  0, 4 = ±1, 2 = 3 = 5 = 0

1 = 0, 2 = 0

{2,3,4,5} (5,4,3,2) {2,3} {2}

2 = ±1, 1 = 3 = 4 = 5 = 0

1 = 0, 21 4 + 22 = 0

{2,3}

(3,2)

{2}

1 = 2 = 0, 3 = 0

{3,5}

(5,3)

{3} {3}

3 = ±1, 1 = 2 = 5 = 0

1 = 2 = 3 = 0 4 = 0

{4,5}

(5,4)

{4} {4}

4 = ±1, 1 = 2 = 3 = 5 = 0

1 = · · · = 4 = 0

∅

()

∅

∅

1 =  0, 2 = 3 = 4 = 0

∅ 1 = 2 = 3 = 4 = 0 

Let now G be a connected and simply connected solvable Lie group with Lie algebra g, and denote by  the minimal layer in g∗ . Let  be the modular function of G: (s) = det(Ad(s))−1 = det(Ad∗ (s)).  Its differential is d(Y ) = trace(ad∗ (Y )) = − i λi (Y ), for Y ∈ g. Lemma 4.5.4 For each  ∈ , s() ⊂ ker(d)c . Proof: By (2.2.4) , we know that s() = span{T ()Zj : j ∈ / e}. Now by / h(e) = e, Proposition 2.2.10, if  ∈ , then for i ∈ / e, λi (T ()Zj ) = 0 for j ∈ hence for each Y ∈ g(),

λi (Y ) = 0. trace(adg() (Y )) = i ∈e /

For each s ∈ G(), Ad(s) defines a linear homomorphism on g/g(), denoted by Adg/g() (s), which preserves the symplectic form B (Y,Z) = [Y,Z]: B (Adg/g() (s)Y,Adg/g() (s)Z) = B (Y,Z).

302

Coadjoint orbits and polarizations

Hence det(Adg/g() (s)) = 1, and trace(adg/g() (Y )) = 0 for each Y ∈ g(). Extending to s() = g()c , it follows that trace(adg (Z)) = 0 for all Z ∈ s(), proving the lemma.

4.5.2 Application: Vergne polarizations We apply the constructions of Section 4.4 to obtain the following. Proposition 4.5.5 Let g be a solvable Lie algebra over R, fix a good Jordan– Hölder sequence for s and the corresponding flag (Ui ) for s∗ . For each  ∈ g∗ , there is a polarization h() at , such that the mapping  → h() is rational on each ultrafine layer . Proof: Let  = e,h,δ be an ultrafine layer and fix  ∈ . Apply the constructions of Section 4.4 to the alternate form B = B and M = M(). Recall that M (j ) is the j -th principal minor of the matrix M = M(). Observe that for each 1 ≤ j ≤ n, nullspace M (j ) = rad B|sj = sj ∩ sj = sj (πj ()). Thus via Propositions 4.4.7 and 4.4.9, we obtain a maximal isotropic subspace H of g∗ given by H =

n

nullspace M (j ) =

j =1

n

sj (πj ()) = h().

(4.5.1)

j =1

Now the entries of M() are linear functions of  and so by virtue of Proposition 4.4.1 and Lemma 4.4.8, the map  → T () is a rational function of  ∈ , regular on . Now by Proposition 4.4.7,  → h() = span{T ()Zj : j ∈ ec ∪ e+ } is also a rational map, regular on . p Also by Proposition 4.4.7, if sp is real, then j =1 sj (πj ()) is maximal isotropic for B|sp , since it is a subspace of the isotropic space h() ∩ sp , h() ∩ sp =

p

sj (πj ()).

(4.5.2)

j =1

We claim that h = h() is a subalgebra of gc . Since each sj (πj ()) is a subalgebra, then by bilinearity of B it is enough to show that for 1 ≤ j < i, [si (πi ()),sj (πj ())] ⊂ sj (πj ()). Let Xj ∈ sj (πj ()) and Xi ∈ si (πi ()). Since sj is an ideal in si , [Xi ,Xj ] ∈ sj . Given X ∈ sj , [Xj ,X] ∈ sj ⊂ si so [[Xj ,X],Xi ] = 0, while [Xi ,X] ∈ sj so [Xj ,[Xi ,X]] = 0 and [[Xi ,Xj ],X] = [[Xi ,X],Xj ] + [Xi ,[Xj ,X]] = 0. Thus [Xi ,Xj ] ∈ sj (πj ()).

(4.5.3)

4.5 Fine layering and Vergne polarizations

303

To see that h + h is a subalgebra, we prove that [h,h] ⊂ h + h. Let Z ∈ h, and W ∈ h. Again by bilinearity we may assume that Z ∈ sj (πj ()), W ∈ si (πi ()), and j ≤ i. If j < i; then Z ∈ sj +1 ⊂ si , [Z,Z] ∈ sj +1 , and the computation (4.5.3) shows that [Z,W ] ∈ si (πi ()) ⊂ h ⊂ h + h. Assume then that j = i. If sj is real, then Z ∈ sj = si , and again (4.5.3) gives [Z,W ] ∈ sj (πj ()). If sj = si is not real, then by Lemma 1.4.8: [sj +1,sj +1 ] ⊂ sj −1 . Hence [Z,W ] ∈ sj −1 . Now for any X ∈ sj −1 , [Z,X] ∈ sj −1 ⊂ sj and [W,X] ∈ sj −1 ⊂ sj . Therefore: [[Z,W ],X] = [[Z,X],W ] + [Z,[W,X]] = [[Z,X],W ] + [Z,[W,X]] = 0, and thus [Z,W ] belongs to sj −1 (πj −1 ()) ⊂ h(). This completes the proof. Definition 4.5.6 Let (sj ) be a good Jordan–Hölder sequence of ideals in s = gc and let  ∈ g∗ . The subalgebra h() defined in the preceding proposition is called the Vergne polarization at , corresponding to the flag (sj ). Here are two examples. Example 4.5.7 Recall Example 4.2.3: g has the basis (Z,Y,X,A) where [X,Y ] = Z and [A,X] = X − aY , [A,Y ] = aX + Y , [A,Z] = 2Z (a ∈ R), and where we put  = λZ ∗ with λ = 0. The subalgebra h = spanC {Z,Y } is a polarization at , but not a Vergne polarization. To see this, let (si ) be any Jordan–Hölder sequence for gc . First we have [g,g]c = nc = s3 . Since each sj is invariant under ad(A), then s2 must be the sum of two eigenspaces of ad(A) in nc . Since s2 is a Lie algebra, the only possibilities are s2 = span{Z,X + iY } or s2 = span{Z,X − iY }, and since each sj is ad(X) and ad(Y )-invariant then s1 = CZ. Then: s1 ∩ s1 = s1, s2 ∩ s2 = s2, s3 ∩ s3 = s1, and s4 ∩ s4 = 0. Thus in this example, the Vergne polarization for any Jordan–Hölder sequence (si ) the ideal s2 = h. In particular, the Jordan–Hölder sequence with s2 =  spanC {Z,X + iY } gives the polarization of Example 4.2.3. Example 4.5.8 Let g be the diamond Lie algebra having the usual basis (Z,Y,X,A) with [X,Y ] = Z, and [A,X + iY ] = −i(X + iY ). Fix the Jordan– Hölder basis (Z,X + iY,X − iY,A) (so Z1 = Z, Z2 = X + iY , Z3 = X − iY and Z4 = A). Let  = Z ∗ . Then we get s1 ∩ s1 = s1 (π1 ()) = span{Z},

304

Coadjoint orbits and polarizations

s2 (π2 ()) = span{Z,X + iY }, s3 (π3 ()) = span{Z}, s() = span{Z,A}. Thus for this Jordan–Hölder sequence, the corresponding Vergne polarization at  is h() = span{Z,X + iY,A}. Observe that the construction of the Vergne polarization depends upon the ordering of the Jordan–Hölder basis: the ordered basis Z1 = Z,Z2 = X − iY, Z3 = X + iY,Z4 = A is also a Jordan–Hölder basis and results in the Vergne polarization h = span{Z,X − iY,A}. 

4.5.3 Properties of the Vergne polarization We now show that the Vergne polarization construction yields a polarization satisfying the desired properties described in Sections 4.2 and 4.3: G()stability, admissibility, and the Pukanszky condition. For the property of admissibility, one chooses the Jordan–Hölder sequence appropriately. Accordingly, let G be a connected, solvable Lie group with Lie algebra g, and let n be an ideal containing [g,g]. Fix a good Jordan–Hölder sequence (si )ni=1 of ideals  in gc such that nc = sp for some p (Corollary 1.4.7). Let h() = i si (πi ()) be the Vergne polarization at  ∈ g∗ . We have observed that h()∩nc is maximal isotropic for Bf , where f = |n , thus by Corollary 4.3.10, h() is n-admissible. Recall also the stability subgroup G() of  for the coadjoint action of G; we have seen (Example 4.1.7) that G() is not necessarily connected. Lemma 4.5.9 The Vergne polarization h() is G()-stable. Proof: Since G is connected, we have Ad(G)si = si by Corollary 1.9.2 for any i. Suppose now s ∈ G(), so s = , thus s πi () = πi () for any i, and for any Z ∈ si (πi ()), any Y ∈ si , Ad(s)Zπi ()(Y ) = πi ()[Y,Ad(s)Z] = πi ()(Ad(s)[Ad(s −1 )Y,Z]) = s −1 πi ()[Ad(s −1 )Y,Z] = πi ()[Ad(s −1 )Y,Z] = 0. This means Ad(s)si (πi ()) = si (πi ()), and thus Ad(s)h() = h(). The following generalization of the preceding will be needed. Let n be an ideal in G with N its corresponding simply connected normal subgroup of G. Let f ∈ n∗ . Then G acts on n∗ by the restricted coadjoint action: indeed, n⊥ is an Ad∗ (G)-invariant subspace of g∗ and the quotient action of G on g∗ /n⊥ is equivalent with the action of G on n∗ via the canonical isomorphism n∗  g∗ /n⊥ . Fix  ∈ g∗ with f = |n ; a subgroup of central importance is the stability group G(f ) = {s ∈ G : s f = f }.

4.5 Fine layering and Vergne polarizations

305

Since f [X,Y ] = [X,Y ] for any X ∈ g,Y ∈ n, the Lie algebra g(f ) of G(f ) is g(f ) = {X ∈ g : [X,Y ] = 0 for all Y ∈ n} = n . The reason for our interest in the objects h2 , e2 , and d2 in Section 4.3 arises from our interest in the group G(f ), and will become more apparent in Chapter 5. Lemma 4.5.10 Let n be an ideal in g, and let (si ) be a good Jordan–Hölder sequence that includes nc = sp . Let  ∈ g∗ and put f = |n . Let h be the Vergne polarization at , corresponding to the sequence (si ), and let h1 = h∩n. Then h1 is a G(f )-stable polarization at f in sp = nc : for all s ∈ G(f ), Ad(s)h1 = h1 . Proof: Exactly as in the preceding proof, for each s ∈ G(f ) and each i ≤ p,  Ad(s)si (πi ()) = si (πi ()). Since, by Relation (4.5.2), h1 = i≤p si (πi ()), this implies that h1 is a polarization at f ∈ n∗ which is G(f )-stable. Corollary 4.5.11 Let  ∈ g∗ and (si ) be a Jordan–Hölder sequence for gc . Then the Vergne polarization h() satisfies the Pukanszky condition. Proof: Recall that the Vergne polarization can be defined recursively: thanks to Proposition 4.4.9, we have e+ () = {i1 (), . . . ,id/2 ()}, j() = (j1 (), . . . ,jd/2 ()), and s = h0 (),h1 (), . . . , hd/2 () = h(), where:   ik () = min 1 ≤ j ≤ n : sj ∩ hk−1 () ⊂ hk−1 () ,   hk () = (hk−1 ()∩sik ) ∩hk−1 () = span T ()Zj : j ∈ / {j1 (), . . . ,jk ()} ,   jk () = min 1 ≤ j ≤ n : sj ∩ hk−1 () ⊂ hk () . Let  ∈ g∗ , and let f ∈ g∗ vanishing on h(). If j ≤ i1 , then sj ⊂ h(), so if j < i1 then ( + f )[sj ,s] = [sj ,s] = 0, while ( + f )[si1 ,s] = [si1 ,s] = 0. +f Hence i1 ( + f ) = i1 (), and h1 ( + f ) = si1 = h1 (). Hence j1 ( + f ) = j1 (). Suppose now that, for t = 1, . . . ,k −1, it (+f ) = it (), ht (+f ) = ht () and jt ( + f ) = jt (). Put ik = ik (). Now ik is the first index j such that sj ∩ hk−1 () is not included in hk−1 () . We claim now that sik ∩ hk−1 () ⊂ h(). Indeed, by definition hk () ⊂ (hk−1 () ∩ sik ) , then: h() = h() ⊃ hk () ⊃ hk−1 () ∩ sik . Thus f vanishes on sik ∩hk−1 () and the above argument gives, for any j < ik , ( + f )[sj ∩ hk−1 (),hk−1 ()] = ( + f )[sj ∩ hk−1 ( + f ),hk−1 ( + f )] ⊂ f (h()) = 0,

306

Coadjoint orbits and polarizations

and ( + f )[sik ∩ hk−1 (),hk−1 ()] = 0. In other words ik ( + f ) = ik (), and similarly hk ( + f ) = (hk−1 ( + f ) ∩ sik )+f ∩ hk−1 ( + f ) = (hk−1 () ∩ sik ) ∩ hk−1 () = hk (), which immediately implies that jk ( + f ) = jk (). By induction, h(+f ) = h(), and Proposition 4.3.6 says that h() satisfies the Pukanszky condition. Fix as usual a good Jordan–Hölder sequence (si )ni=1 and associated Jordan– Hölder basis (Zi ) for gc . Recall the notations and properties of the layering indices e, h : e → e, e+ and e− , and δ for the ultrafine stratification of g∗ , and let  = e,h,δ be an ultrafine layer in g∗ , with e = ∅. For each  ∈ , let h() be the associated Vergne polarization. Now for  ∈ , recall the unipotent upper-triangular matrix T () so that t

T ()M()T () = N (e,h,c).

This means that for all 1 ≤ i,j ≤ n, [T ()Zi ,T ()Zj ] = 0 if and only if both i and j belong to e and j = h(i) (hence i = h(j ) also). By Corollary 2.2.9 (see also Remark 2.1.13), for e ∈ e, T ()Zh(e) ∈ se−1 ∩ sh(e) \ se and, since h(h(e)) = e, T ()Ze ∈ sh(e)−1 ∩ se \ sh(e) . Recall that for each i,  → T ()Zi is a nonsingular rational map on ; the elements T ()Zi are independent, and by Lemma 4.4.4, / e} ⊕ spanC {T ()Ze : e ∈ e+ }. h() = spanC {T ()Zi : i ∈

(4.5.4)

We use this basis of h() to describe certain characteristics shared by the Vergne polarizations h(), as  runs through . Recall the indices i for which si is real: I = {0 ≤ i ≤ n : si = si }. For each 1 ≤ i ≤ n, put i  = min{j ∈ I : i ≤ j } and i  = max{j ∈ I : j < i}. Since (Zi ) is a good Jordan–Hölder sequence, this means that i∈ / I %⇒ i  = i + 1 and Zi  = Z i .

4.5 Fine layering and Vergne polarizations

307

Recall that by Lemma 2.4.2, e is the disjoint union of the subsets e0 = {e : e − 1 ∈ I and e ∈ I}, e1 = {e : e ∈ / I and e + 1 ∈ / e}, / I and e + 1 ∈ e}. e2 = {e,e + 1 : e ∈ Observe that if e ∈ e∩I, then s(πe ()) is real, and by Lemma 2.4.4, if e ∈ e\I, then s(πe ()) is real if and only if e ∈ e1 . Thus if s(πe ()) is not real, then e∈ / I and e ∈ e2 . / I, and e + 1 = h(e ) with e ∈ e+ . Lemma 4.5.12 Suppose that e ∈ e+ , e ∈  Then e = e. Proof: Since e ∈ e+ , then e ≤ e. Suppose that e < e, thus h(e) > e + 1. Consider the real Lie algebra ge+1 , and recall that e(|ge+1 ) = e()(e+1) = {i ∈ e : i ≤ e + 1, h(i) ≤ e + 1}. So e ∈ / e(|ge+1 ), but e + 1 ∈ e(|ge+1 ), this is impossible by Lemma 2.4.2. Hence if e ∈ / I and e ∈ e2 ∩ e+ , then e + 1 ∈ e+ , or e + 1 = h(e). This last case, where e ∈ e+ and h(e) = e , will play a special role in what follows. Define f ⊂ e2 ∩ e+ by f = {e ∈ e+ : e ∈ / I and e + 1 = h(e)}. We now give an explicit description of the subalgebras h() ∩ h() and h() + h() associated to h(), as defined in Section 4.3. Lemma 4.5.13 Let e ∈ e+ . If s(πe ()) is real, then T ()Ze ∈ d()c = h() ∩ h(). / I. Then each of the following Suppose that s(πe ()) is not real, so that e ∈ holds. (1) If e + 1 ∈ e+ , then {T ()Ze,T ()Ze+1 } ⊂ d()c . (2) If e ∈ f (so that Zh(e) = Z e ) then [T ()Ze,T ()Zh(e) ] = [T ()Ze,T ()Ze ], and T ()Ze does not belong to h(). Proof: Suppose that s(πe ()) is real. If e ∈ I, then it is immediate that / I, then se (πe ()) = s(πe ()) ∩ se is real so T ()Ze ∈ se (πe ()) ⊂ h(). If e ∈ e ∈ e1 so s(πe+1 ()) = s(πe ()) = s(πe ()) and se ⊂ se+1 , so T ()Ze ∈ s(πe+1 ()) ∩ se+1 = se+1 (πe+1 ()) ⊂ h() and T ()Ze ∈ h() ∩ h() = d()c .

308

Coadjoint orbits and polarizations

Suppose next that s(πe ()) is not real, and that e + 1 ∈ e+ . Then e + 1 ∈ I so by the first part, T ()Ze+1 belongs to d()c . Now T ()Ze ∈ se ⊂ se+1 and h(e) > e + 1 means that T ()Ze ∈ s(πh(e)−1 ()) ⊂ s(πe+1 ()), so T ()Ze ∈ s(πe+1 ()) and hence T ()Ze ∈ s(πe+1 ()) ∩ se+1 = se+1 (πe+1 ()) ⊂ h() so T ()Ze ∈ h() ∩ h() = d()c . Finally suppose that s(πe ()) is not real and that h(e) = e +1. Since T () is unipotent upper-triangular, T ()Ze = Ze + W as well as T ()Zh(e) = Zh(e) + aZe + W  , with W and W  elements of se−1 . Since e − 1 ∈ I, then also T ()Ze = Z e + W = Zh(e) + W with W ∈ se−1 . Now T ()Zh(e) − T ()Ze = aZe + (W  − W ). But T () is invertible on se−1 so aZe + W  − W = aT ()Ze + T ()U , with U ∈ se−1 . Hence [T ()Zh(e) − T ()Ze,T ()Ze ] = [T ()U,T ()Ze ] = 0 since T ()Ze ∈ s(πe−1 ()). Now the definition of the involution h implies that [T ()Ze,T ()Zh(e) ] = 0, and so [T ()Ze,T ()Ze ] = [T ()Ze,T ()Zh(e) ] = 0. Since T ()Ze ∈ h(), / h(), so T ()Ze ∈ / h(). then T ()Ze ∈ Point (2) of the lemma can be completed by Lemma 4.5.14 Suppose that i = j are two elements in f. Then [T ()Zi ,T ()Zj ] = [T ()Zi ,T ()Zj ] = 0. Proof: Indeed, assuming that i < j , we have T ()Zi ∈ si+1 ⊂ sj −1 and by definition of T (), T ()Zj ∈ s(πj −1 ()) ⊂ s(πi+1 ()). Thus [T ()Zi , T ()Zj ] = 0 and hence [T ()Zi ,T ()Zj ] = [T Zi ,T Zj ] = 0. Corollary 4.5.15 Let  ∈  and h = h() be the Vergne polarization at , put d = d(), e = e(). Then (1) h = dc ⊕ spanC {T ()Ze : e ∈ f}, (2) ec = dc ⊕ spanC {T ()Ze, T ()Ze : e ∈ f}, / e} ⊕ spanC {T ()Ze : e ∈ e+ \ f}. (3) dc = spanC {T ()Zi : i ∈ Let us now apply Corollary 4.5.15 to compute the Kostant measure dκO . Fix  in some ultrafine layer e,h,δ , and put Ue = T ()Ze (e ∈ e), these

4.5 Fine layering and Vergne polarizations

309

vectors form a basis of a space supplementary to s() in s, therefore, using Lemma 4.1.3, the value of the Kostant volume form at  is:

 dκ = (d/2)! Pf [Ue,Ue ] due1 ∧ · · · ∧ dued . We now are able to compute each term: Denote by T ()ee , M()ee the submatrices of T (), M() corresponding to columns and rows in e. By the definition of T (),   e [Ue,Ue ] = t T ()t M()T () e = t (T ()ee )M()ee T ()ee,









therefore Pf [Ue,Ue ] = det(T ()ee )Pf [Ze,Ze ] = Pf [Ze,Ze ] .  

Denote for the moment pUe the form  →  Ue . Since the matrix [Ue,Ue ] is reduced, we get: ξUe pUe = [Ue,Ue ] = δe,h(e) [Ue,Uh(e) ]. Thus the dual basis (due ) of (ξUe ) is given by: due =

1 dpUh(e) . [Ue,Uh(e) ]

But we have dpUh(e) () = Uh(e) = (T ()Zh(e) ) =

te,h(e) Ze =

e ∈e

te,h(e) dpe (),

e ∈e

where te e are the entries of the unimodular matrix T ()ee . Finally we get the following expression of dκ :

 ±dκ = Pf [Ze,Ze ] =

(d/2)! dpe1 ∧ · · · ∧ dped e∈e [Ue,Uh(e) ] >  dpe .

(d/2)! Pf [Ze,Ze ]

e∈e

Lemma 4.5.16 On the orbit O of , the Kostant measure dκO is the invariant measure defined by the volume form whose value at  is > (d/2)!  dpe . Pf [Ze,Ze ] e∈e

310

Coadjoint orbits and polarizations

4.6 Positivity and properties of polarizations 4.6.1 Positive polarizations The final important property of polarizations that is needed is the property of positivity. Given  ∈ g∗ , define the bilinear form H on gc × gc by H (X,Y ) = 2iB (X,Y ) = 2i[X,Y ]. Observe that H is a Hermitian form, and that H (X,X) is real for all X ∈ gc . We are interested in the restriction of H to a polarization h at . Lemma 4.6.1 Let  ∈ g∗ and h any polarization at . Then dc = {Z ∈ h : H (h,Z) = 0}. Proof: We have Z ∈ h ∩ h = dc implies Z ∈ h so H (h,Z) = 2i[h,Z] = 0. If Z ∈ h and H (h,Z) = 0 then Z ∈ h so Z ∈ dc . Thus H defines a nondegenerate Hermitian form on h/dc , still denoted by H , or simply by H . Note that H (v,w) = 2iB(v,w), v, w ∈ h/dc . Definition 4.6.2 Let h ⊂ gc be any polarization at  ∈ g∗ . Then h is said to be positive if the restriction of H to h × h is positive semidefinite, that is, H (X,X) ≥ 0, holds for all X in h. Equivalently, H is a positive definite Hermitian form on h/dc . Now consider the real vector space e/d. Recall (see Lemma 4.2.10) that B defines a nondegenerate bilinear form B on e/d, thus the real bilinear form S(x,y) = B(x,Jy) is also nondegenerate on e/d. We have: Lemma 4.6.3 The form S is symmetric, nondegenerate. Moreover, S is positive definite if and only if h is a positive polarization. Proof: By Lemma 4.2.10, B is J -invariant, and hence S is symmetric: S(y,x) = B(y,J x) = −B(J 2 y,J x) = −B(Jy,x) = B(x,Jy) = S(x,y). Now for each x ∈ e/d, observe that J (x + iJ x) = −i(x + iJ x), thus x = 1/2 (x + iJ x) + 1/2(x − iJ x) is the decomposition of x ∈ e/d ⊂ ec /dc into the direct sum ec /dc = h/dc ⊕ h/dc . Since dimR (h/dc ) = 2 dimC (h/dc ) = dimR (e/d),

4.6 Positivity and properties of polarizations

311

and the map x → 1/2(x +iJ x) is injective, then it is an R-isomorphism. Now: i 1 H (x + iJ x,x + iJ x) = B(x + iJ x,x − iJ x) 4 2 = B(x,J x) = S(x,x). Therefore h is positive if and only if S is positive definite. By Lemma 4.6.1, the question of positivity depends only upon the values H (X,X) where X ∈ h \ h. Of course even the simplest Vergne polarization can have such elements, and need not be positive. Example 4.6.4 Let g be the Heisenberg Lie algebra of dimension three, having the usual basis (Z,Y,X) with [X,Y ] = Z. Put s0 = {0}, s1 = CZ, s2 = spanC {Z,X + iY }, s3 = gc ; then (si ) is a good Jordan–Hölder sequence with corresponding Jordan–Hölder basis Z1 = Z, Z2 = X + iY , Z3 = X − iY . Let  = λZ ∗ ∈ g∗ ; then the matrix M() is: ⎡

0 M() = ⎣0 0

0 0 2iλ

⎤ 0 −2iλ⎦ . 0

It is already in strict flag normal form. The open layer is  = {2,3},(3,2) = { : λ = 0}, and for all  ∈ , the Vergne polarization at  is h = spanC {Z1,Z2 } = spanC {Z,X + iY }, with d = RZ. Of course X + iY ∈ h \ h. Now since   2i aZ1 + bZ2,aZ1 + bZ2 = 2ibb[X + iY,X − iY ] = 4|b|2 λ then h is a positive polarization at  if and only if λ > 0. Writing x = X + d, y = Y + d, then h/dc = spanC {x + iy} and J x = y, Jy = −x. Thus S(x,x) = S(y,y) = B (X,Y ) = λ. Observe also that the question of positivity of h is solely determined by the value of H (X + iY,X + iY ) = 4λ. If λ < 0, h is not a positive polarization, but it is easily seen that h =  spanC {Z,X − iY } is a positive polarization at . Let g be a solvable Lie algebra over R, choose a good Jordan–Hölder basis (Zi ) for the complexification gc , and let  = e,h,δ be an ultrafine layer. By Corollary 4.5.15, h() = d()c ⊕ span{T ()Ze : e ∈ f}.

312

Coadjoint orbits and polarizations

Lemma 4.6.5 Fix  belonging to the ultrafine layer  and let h = h() be the Vergne polarization at . Then h is positive if and only if H (T ()Ze,T ()Ze ) ≥ 0

(4.6.1)

holds for all e ∈ f. Proof: Suppose that (4.6.1) holds. Let X ∈ h. By Corollary 4.5.15, we may write

ae T ()Ze X=Z+ e∈f

where Z ∈ dc . Now by Lemmas 4.5.14 and 4.6.1,  

ae T ()Ze, Z + a e T ()Ze H (X,X) = 2i Z + =2

e∈f

e∈f

|ae | i[T ()Ze,T ()Ze ] 2

e∈f

=

|ae |2 H (T ()Ze,T ()Ze ) ≥ 0.

e∈f

Hence the condition (4.6.1) implies that h is a positive polarization. The converse is clear. Now observe that by part (c) of Lemma 4.5.13, and Lemma 4.6.5, h is positive if and only if 

i T ()Ze,T ()Zh(e) = ic (e) ≥ 0 holds for all e ∈ f. Recall that the real function  → ic (ik ) does not vanish on . But if  is not connected, its sign can change. Example 4.6.6 We return to Example 4.6.4, where the open layer is  = {2,3},(3,2) . Write  = + ∪ − where + = { ∈ g∗ : λ > 0}, and − = { ∈ g∗ : λ < 0}. We saw that the Vergne polarization associated to the ordered basis (Z1 = Z,Z2 = X + iY,Z3 = Z 2 ) is positive for any  ∈ + , and not positive for  ∈ − . For  ∈ − , the permutation of the good basis, (Z1,Z2,Z3 ) → (Z1,Z 2,Z2 ) yields a new basis that is still a good Jordan–Hölder basis, where now the good Jordan–Hölder sequence is {0} ⊂ CZ1 ⊂ spanC {Z1,Z 2 } ⊂ s. With respect to this new basis, the matrix M becomes ⎡ ⎤ 0 0 0 M  () = ⎣0 0 2iλ⎦ , 0 −2iλ 0

4.6 Positivity and properties of polarizations

313

and hence the Vergne polarization corresponding to the new Jordan–Hölder  sequence is positive for any  ∈ − . To generalize the preceding example, fix a good Jordan–Hölder sequence (sj ) with an associated Jordan–Hölder basis (Zi ) for s = gc , and recall the notations and properties of the layering indices for a fine stratification of g∗ . Let  = e,h,δ be an ultrafine layer in g∗ . We have already observed that positivity of the Vergne polarization is equivalent with positivity of the functions  → c (e), e ∈ f. Write f = (e1 < e2 < · · · < em ). For each multisign ε = (ε1,ε2, . . . ,εm ) ∈ {−1,1}m , put ε = { ∈  : ic (el ) = εl ,1 ≤ l ≤ m}. We have the following. Proposition 4.6.7 Fix a good Jordan–Hölder sequence (sj ) with an associated Jordan–Hölder basis (Zj ) for s = gc , and let  = e,h,δ be an ultrafine layer in g∗ . Let ε ∈ {−1,1}m be a multisign and define ε as above. Define the new basis (Ziε )i as follows: if e = el ∈ f and εl = −1, put ε Zeε = Ze+1, and Ze+1 = Ze .

Otherwise, put Ziε = Zi . Then the sequence (sεi ), defined by sεi = spanC {Z1ε, . . . ,Ziε }, 1 ≤ i ≤ n, is a good Jordan–Hölder sequence of ideals in gc , and the associated Vergne polarization hε () is positive for all  ∈ ε . Proof: To see that each sεi is an ideal, we may assume that i ∈ f, thus i ∈ / I and by Lemma 1.4.8, [si+1,si+1 ] ⊂ si−1 , thus sεi is an ideal. Let Sε denote the linear map induced by the permutation σε = εk =−1 (ik ,ik + 1) applied to the basis (Zj ): Sε (Zj ) = Zσε (j )

(1 ≤ j ≤ n).

Observe that Sε−1 = t Sε = Sε . With respect to the new basis (Ziε ), the matrix M() = [ai,j ()] becomes M ε () = Sε M()Sε . Consider now the strict normal form of M(), N (e,h,c ) = t T ()M()T (). Observe that: Sε t T () Sε M ε () Sε T () Sε = Sε N (e,h,c )Sε (ε)

= N (e,h,c ◦ σε |e ) = N (e,h,c ),

314

Coadjoint orbits and polarizations

where, for any e = ek ∈ f, we have: 2 (ε) ic (e)

=

ic (e) > 0

if εk = +1,

ic (h(e)) = −ic (e) > 0

if εk = −1.

Now we claim that the matrix T (ε) () = Sε T ()Sε is still unipotent, upper (ε) triangular. Suppose this claim holds, then N (e,h,c ) is the strict normal form ε of M (), hence Lemma 4.6.5 applied to all  ∈ ε shows that the Vergne polarization hε (), associated to the Jordan–Hölder sequence (sεi ) is positive. To prove the claim, recall that we have written e+ = {i1 < i2 < · · · < id/2 } and jt = h(it ), and that the unipotent upper-triangular matrix T () is defined recursively as T = T () = T1 T2 . . . Td/2 by the relations (4.4.1). That means, for each k, Tk Zj = Zj ,

if j ≤ ik , (k−1)

Tk Zj = Zj +

ajk ,j

if ik < j ≤ jk ,

Zik ,

ck (k−1)

Tk Zj = Zj +

ajk ,j ck

(k−1)

Zik −

aik ,j ck

if jk < j,

Zjk ,

k−1 are the entries of the matrix t Tk−1 . . . t T1 M() T1 . . . Tk−1 . Write where ai,j now f = {e1, . . . ,em } and Sε = S1 S2 . . . Sm where St is the matrix of the permutation (et ,h(et )) = (et ,et + 1). The matrices St are commuting and if / f, Sε Tk Sε = Tk and if et = ik , it is easily seen that St Tk St = Tk . Hence, if ik ∈ ik is in f, with ik = et , then Sε Tk Sε = St Tk St . Suppose thus et = ik . This implies that jk = ik + 1, and Tk becomes:

Tk Zj = Zj ,

if j ≤ jk , (k−1)

Tk Zj = Zj +

ajk ,j ck

(k−1)

Zik −

aik ,j

Zjk ,

ck

if jk < j .

Therefore: St Tk St Zj = Zj , St Tk St Zj = Zj +

if j ≤ jk , (k−1) ajk ,j

ck

Zjk −

(k−1) aik ,j

ck

Zik ,

if jk < j .

Thus St Tk St is still unipotent upper triangular, and the claim is proved.

4.6 Positivity and properties of polarizations

315

4.6.2 Polarizations in exponential Lie algebras Let g be an exponential Lie algebra, n a nilpotent ideal containing [g,g], and  ∈ g∗ . We saw there are positive polarizations h at , such that h1 = h ∩ n is a polarization in n at f = |n . The following shows that in the exponential case, the Lie algebraic structure inside e is somewhat rigid. Recall the notation h1 = h ∩ n, e1 = (h1 + h1 ) ∩ g. Lemma 4.6.8 Suppose that g is an exponential Lie algebra, and let h be a positive polarization at  ∈ g∗ One has (1) [e,d] ⊂ d, and (2) if e = e1 + d, then [e,e] ⊂ d. Proof: Recall the nondegenerate alternate form B and the symmetric form S defined on e/d×e/d; since h is positive then S is positive definite. As in Section 4.6.1, for Y ∈ e write y = Y + d ∈ e/d. Let Z ∈ d; y → [Z,Y ] + d defines an endomorphism ade/d (Z) of e/d whose extension of ec /dc leaves h/dc and h/dc invariant. To prove (1), observe that since e = d , then for all X, Y ∈ e, 0 = [Z,[X,Y ]] = [[Z,X],Y ] + [X,[Z,Y ]] = B(ade/d (Z)x,y) + B(x,ade/d (Z)y). Moreover, ade/d (Z) commutes with J , so by the preceding, S(ade/d (Z)x,y) = B(ade/d (Z)x,J y) = −B(x,ade/d (Z)J y) = −B(x,J (ade/d (Z)y)) = −S(x,ade/d (Z)y), showing that ade/d (Z) is skew-symmetric with respect to the positive definite form S. Therefore it is diagonalizable and its eigenvalues are zero or purely imaginary. But, since g is exponential, the eigenvalues of ade/d (Z) are not purely imaginary. Hence ade/d (Z) = 0. Thus [e,Z] ⊂ d, and (1) follows. Now d being an ideal in e, we denote the Lie algebra e/d by m. Suppose now e = e1 + d, since e1 is nilpotent, then m is nilpotent, and the center z of m is nontrivial. To prove (2), we have to show that z = m. We first show that z is J -invariant. Let z ∈ z and put A = adm (J z); we must show that A = 0. We claim that A commutes with J . To see this, recall that h/dc is the set of all elements in ec /dc of the form x + iJ x, x ∈ m. Since z ∈ z, and h/dc is a subalgebra of ec /dc , then u = [y + iJy,z + iJ z] = −[Jy,J z] + i[y,J z]

(y ∈ m)

316

Coadjoint orbits and polarizations

belongs to h/dc so J u = −iu, or J (−[Jy,J z]) = Re(J u) = [y,J z], giving Ay = −J AJy and J Ay = AJy for all y ∈ m, proving the claim. Now to prove that A = 0, suppose that contrary. Since A is nilpotent, this means that there is y ∈ m such that Ay = 0 but A2 y = 0. We now derive the contradiction that S(Ay,Ay) = 0. Let Z, Y , X ∈ e such that J z = Z + d, y = Y + d, and Jy = X + d. Now since [Z,[Z,[Y,X]]] = (adZ)2 [Y,X] = [(adZ)2 Y,X] + 2[(adZ)Y,(adZ)X] + [Y,(adZ)2 X] then by linearity of  and the definition of B, B(J z,A[y,J y]) = B(A2 y,J y) + 2B(Ay,AJy) + B(y,A2 Jy). But z ∈ z and B is J -invariant, this implies that the left-hand side of the preceding is zero: B(J z,A[y,J y]) = B(J 2 z,J A[y,J y]) = −B(z,J A[y,J y]) = 0. Moreover, our choice of y and the relation J A = AJ show that B(A2 y,J y) = B(y,A2 (J y)) = B(y,J (A2 y)) = 0; hence 0 = B(Ay,AJy) = B(Ay,J Ay) = S(Ay,Ay) as desired. Thus J z is in the center z of m. Denote by zS the orthogonal complement of z with respect of the positive definite form S, we have m = z ⊕ zS , and: zB = {y ∈ m : B(z,y) = 0 (z ∈ z)} = {y ∈ m : B(J z,y) = 0 (z ∈ z)} = zS . This implies that zS is a subalgebra of m, and if zS is not zero, its center is not vanishing, there is y ∈ zS , y = 0 commuting with zS , and z, thus y ∈ z ∩ zS . This is impossible, thus zS = {0}, and m = z. Thus (2) is proved. Observe that if h is a real polarization, that is h = h, then e = d, and the Lemma 4.6.8 is trivially true. Corollary 4.6.9 Let g be an exponential Lie algebra,  ∈ g∗ , and h a positive polarization at  such that e = e1 + d. Suppose that h = h. Then e/(d ∩ ker ) is a Heisenberg Lie algebra with center d/(d ∩ ker ). Proof: Observe that if X ∈ e \ d, then [X,J X] = S(X + d,X + d) > 0, therefore d ⊂ ker . Now clearly d ∩ ker  is an ideal of e and d/(d ∩ ker ) is the center of e/(d ∩ ker ) and has dimension one.

4.6 Positivity and properties of polarizations

317

Corollary 4.6.10 Let g be a nilpotent Lie algebra,  ∈ g∗ , and h a positive polarization at . Then  + d⊥ is included in the coadjoint orbit O of . Proof: In this case, we choose n = g, so e = e1 and the conclusion of Lemma 4.6.8 holds. By Corollary 4.3.3, h satisfies the Pukanszky condition: D =  + e⊥ . If h = h, e = d and the corollary holds. Suppose now h = h. Put f = |e ∈ e∗ , then f is vanishing on d ∩ ker  and Df = {f }. Therefore, putting g(Y + ker  ∩ d) = f (Y ) (Y ∈ e), and E = exp(e), we see that the coadjoint orbit Ef in e∗ is diffeomorphic to the E/ exp(ker ∩d)-orbit of g in (e/(ker ∩d))∗ . But, in Example 4.1.1, we show that this last orbit is g + (d/ ker  ∩ d)⊥ , therefore Ef = f + d⊥ |e . Denote by π the projection g∗ → g∗ /e⊥  e∗ . We have π(E) = Ef = f + d⊥ |e , thus  + d⊥ = π −1 (f + d⊥ |e ) = π −1 (π(E)) = E + e⊥ = E( + e⊥ ) = E(D) = E. In Example 4.1.9 we saw that there are points  in the dual of the diamond Lie algebra having no real polarization. It can be shown (see Bernat et al. [11]) that, conversely, if g is a solvable Lie algebra with the property that for some  ∈ g∗ , there is no real polarization at , then there is a subalgebra g in g having a quotient that is isomorphic with the diamond algebra. Recall also that the diamond Lie algebra is not an exponential Lie algebra. We now show that if g is exponential, then every point  ∈ g∗ admits a g-polarization. Let g be an exponential Lie algebra, n be a nilpotent ideal containing [g,g] and  ∈ g∗ . We saw in Corollary 1.4.7 that there is a good Jordan–Hölder sequence of ideals (si )ni=1 in s such that sp = nc for some p, and si is real if i ≥ p. Thus the associated Vergne polarization h() at  is such that: h() = dc ⊕ span{T ()Ze : e ∈ f}, with f ⊂ {1, . . . ,p} by construction and hence, using Proposition 4.6.7, we can suppose that h() is positive. Moreover we have e1 + d = e, thus [e,e] ⊂ d. Recall now Lemmas 4.5.13 and 4.5.14: we have, for e in f, ce = i[T ()Ze,T ()Zh(e) ] = i[T ()Ze,T ()Ze ] > 0, and for e = e in f, [T ()Ze,T ()Ze ] = [T ()Ze,T ()Ze ] = 0. For e ∈ f, put Ye = Im(T ()Ze ) and define the subspace p() as:

p() = d + RYe . e∈f

(4.6.2)

318

Coadjoint orbits and polarizations

Proposition 4.6.11 The space p() is a g-polarization at , satisfying the Pukanszky condition and stable under the action of G() and p() ∩ n is stable under the action of G(f ), where f = |n . Proof: Recall that d = e and e = d, thus the preceding relations show that [p(),p()] = 0. Since [e,e] ⊂ d by Lemma 4.6.8, then [p(),p()] ⊂ d ⊂ p(), p() is a subalgebra and an isotropic subspace for B; since its real dimension is dim h(), it is maximal, therefore p() is a polarization at . Put Xe = Re(T ()Ze ) (e ∈ f), and fix any real basis Ui in d. This gives a basis (Ui ,Ye,Xe ) of e, complete it into a real basis of g. Thus we can identify e∗ to the subspace of g∗ generated by the Ui∗ , the Xe∗ and the Ye∗ . With this identification, g∗ = e∗ ⊕ e⊥ . Let now g ∈  + p()⊥ . We have:

g =+ g(Xe )Xe∗ + m, with m ∈ e⊥ . e∈f

A direct computation shows that: 4 e∈f

exp −

2g(Xe ) Ye g =  + m with m ∈ e⊥ . ce

By the Pukanszky condition for h(), there is s ∈ D such that  = s 2g(Xe ) e∈f exp − ce Ye g. Therefore, p() satisfies the Pukanszky condition. Since g is exponential, then G() is connected and [g(),p()] ⊂ p() implies that p() is G()-stable. Since the complex polarization h() is G(f )-stable, for any A ∈ g(f ), [A,d] ⊂ d, [A,d ∩ n] ⊂ d ∩ n. Let us now look at [A,Ye ] (e ∈ f). First, there are real numbers μ and a, W1 , W2 in se−1 ∩ g such that [A,Xe + iYe ] = μ(1 + ia)(Xe + iYe ) + W1 + iW2 . Since by construction Xe and Ye are in se−1 , this implies:

f [A,[Xe,Ye ]] = [A,[Xe,Ye ]] = μ[Xe + W1,Ye ] + μ[Xe,Ye + W2 ] = 2μ[Xe,Ye ]. Since A ∈ g(f ) and [Xe,Ye ] = 0, this gives μ = 0, and [A,Ye ] = W2 ∈ se−1 . But A is in se−1 , thus [A,Ye ] ∈ (se−1 ∩ se−1 ) ∩ g ⊂ h() ∩ g = d ⊂ p() ∩ n. Since G(f ) is connected, this proves that p() ∩ n is G(f )-stable.

4.7 Integral orbits

319

If we restrict ourselves to  in the minimal ultrafine layer , we can impose another property to p(), namely: Lemma 4.6.12 There is a choice of good Jordan–Hölder sequence (si ) for s such that, for any  in the minimal ultrafine layer , the g-polarization p() at  is included in ker(d). Proof: If (s) = 1 for any s, there is nothing to prove. Otherwise, ker(d) is an ideal of codimension 1 in g. We choose the sequence (si ) such that sn−1 = ker(d)c . By Lemma 4.5.4, s() ⊂ sn−1 . Thus: h() =

n

sj ∩ sj ⊂ sn−1 + s = sn−1 .

j =1

Since sn−1 is real, p() ⊂ e ⊂ sn−1 ∩ g.

4.7 Integral orbits 4.7.1 Integrality In this section, G is a simply connected solvable Lie group, with Lie algebra g. G acts on the dual g∗ of g by the coadjoint representation (s,) → s. Let H be a closed subgroup of G, with H 0 its connected component at the identity. Let χ : H → C× be a character of H : a continuous homomorphism of H into the multiplicative group C× . If |χ | = 1 then we say that χ is unitary. Observe that the differential dχ of χ is a complex linear form on h satisfying [h,h] ⊂ ker dχ , and if χ is unitary, then dχ = i for some  ∈ h∗ . It is natural to ask when the converse of the preceding observation is true: suppose that f : h → C is a linear form for which [h,h] ⊂ ker f . Since H is not assumed to be connected, then it is not always the case that there is a character χ of H such that dχ = f . In particular, for  ∈ g∗ , one always has [g(),g()] ⊂ ker , and it is of the utmost importance to understand when it is true that there is a unitary character χ of the stabilizer G() such that dχ = i. The following example illustrates that even when there is such a character χ , then it is not unique when G() is not connected. Example 4.7.1 Returning to the simply connected group G = E˜ 2 , whose Lie algebra has complexification gc = span(Z,Z,A), with [A,Z] = iZ, recall that in Example 4.1.7 we write  in g∗ as  = (ζ,α) with (Re(zZ) + aA) = Re(ζ z) + αa. Also in Example 4.1.7 we see that when r = |ζ | = 0, then

320

Coadjoint orbits and polarizations

the coadjoint orbit of  is a cylinder, and with the basis (X,Y,A), defined by Z = X + iY , X and Y in g, the stabilizer G(rX∗ ) is the abelian group exp 2πZA exp RX  Z × R. For any 0 ≤ ν < 1, the character: χν (exp 2π kA exp xX) = eirx ei2π νk satisfies dχν = irX∗ .



Example 4.7.2 Since this example was proposed by Duflo in [11], we call the corresponding simply connected Lie group the Duflo group. Let s = gc = span(T ,Z,Z,W,W,B,A) be the solvable Lie algebra such that T , A, B are real and [A,Z] = iZ, [B,W ] = iW , [A,B] = T . Write  = (τ,ζ,μ,β,α) where τ , α, β ∈ R, ζ , μ ∈ C and (tT + Re(zZ) + Re(wW ) + bB + aA) = tτ + Re(zζ ) + Re(wμ) + βb + αa. The nilradical of the simply connected group G is the vector group N , identified with R × C2 , of all elements   N = (t,z,w) = exp(tT + Re(zZ + wW )) : t ∈ R, z,w ∈ C . Now: exp aA exp bB (t,z,w)    = τ,ζ e−ia ,μe−ib,β + Im(wμ) − aτ,α + Im(zζ ) + bτ . Thus the orbit of 0 = (τ,r,R,0,0), r > 0, R > 0 is a product of two cylinders:   G0 = (τ,ζ,μ,β,α) : |ζ | = r, |μ| = R . The stabilizer of 0 is G(0 ) = {exp 2πpA exp 2π qB (t,x + 2π iqτ/r,u − 2π ipτ/R) : (p,q,t,x,u) ∈ Z2 × R3 }. Its connected component of the identity is G(0 )0 = {(t,x,u) : (t,x,u) ∈ R3 }  R3 . Suppose that χ is a character of G(0 ), with derivative i0 |g(0 ) . Then χ |G(0 )0 is given by χ (t,x,u) = ei(tτ +rx+Ru) . However, it is generally impossible to extend χ |G(0 )0 to G(0 ). Indeed if xp = exp 2πpA (0,0, − 2π ipτ/R) = exp 2pπ(A − τ/RIm(W )), and yq = exp 2π qB (0,2π iqτ/r,0) = exp 2qπ(B + τ/rIm(Z)), then 5 xp yq G(0 )0 . G(0 ) = p,q

4.7 Integral orbits

321

/ Z, On the other hand, [xp,yq ] = xp yq xp−1 yq−1 = (4pqπ 2,0,0), and, if 2πpqτ ∈ χ ([xp,yq ]) = e4π

2 ipqτ

= 1.

1 Z. Thus, it is easy to see that the extension is possible if and only if τ ∈ 2π Let us use, on the orbit G0 , the coordinate system: (a,b,α,β) → (τ,re−ia ,Re−ib,β,α). In this system of coordinates, if Z = X + iY , W = U + iV , we get ξA = −∂a + τ ∂β , ξB = −∂b − τ ∂α , ξY = r cos a ∂β , and ξV = R cos b ∂α . This is, for almost all  ∈ G0 , a basis of the tangent space to G0 at . This allows us to compute the Kirillov form ω of the orbit G0 . For instance, if cos a cos b = 0,

ω (ξ Y ,ξ V ) = ω (r cos a ∂β ,R cos b ∂α ) = rR cos a cos b ω (∂β ,∂α ) = [Y,V ] = 0, similarly, −r cos a ω (∂a ,∂β ) = ω (ξ A,ξ Y ) = ([A,Y ]) = −(X) = −r cos a. Finally, we get ω = τ da ∧ db + da ∧ dβ + db ∧ dα.



Definition 4.7.3 Let  ∈ g∗ with O = G. We say that  is integral if there is a character χ : G() → C∗ such that dχ is the restriction of i to g(). Of course if  is integral, every element in O is also integral, since G(s) = Ad(s)G(), and the map η(x) = χ (s −1 xs) is a character of G(s), whose differential is is. Thus we say that the coadjoint orbit O is integral if one (and thus any) of its elements is integral. Denote by G()0 the connected Lie subgroup with Lie algebra g(); G()0 is simply connected, and therefore there is a unique unitary character χ : G()0 → C× whose derivative is i|g() (see Result 1.6.13). Thus the problem of extending χ to a unitary character χ on G() is localized in the discrete group quotient G()/G()0 . Let us be more precise. Lemma 4.7.4 We have [G(),G()] ⊂ G()0 , and the group G()/G()0 is isomorphic to Zr , it is a free, abelian, and discrete group. Let us denote by G()ab the quotient G()/[G(),G()]. Then G()ab is isomorphic to the product G()/G()0 × G()0 /[G(),G()]. Proof: Observe that G()[G,G] is the set of s ∈ G such that s[G,G] = [G,G], recall that [G,G] is nilpotent, acting by a representation of nilpotent type on g∗ . Thus [G,G] is closed in g∗ , and hence G()[G,G] is a closed subgroup of G.

322

Coadjoint orbits and polarizations

On the other hand, for any s ∈ G() ∩ [G,G], since [G,G] is nilpotent simply connected, with Lie algebra [g,g], there is X ∈ [g,g] such that s = exp X. Since s ∈ G(), for any k ∈ Z exp kX = . This means Z is a subset of the set of solutions of the equation exp tX− = 0, but t → exp tX −  is a polynomial function, therefore this implies that for every t, exp tX ∈ G(), thus X ∈ g(), and s ∈ G()0 . Therefore G()0 ∩ [G,G] = G() ∩ [G,G]. Now:     G()/G()0 = G()/G() ∩ [G,G] / G()0 /G() ∩ [G,G]     = G()[G,G]/[G,G] / G()0 [G,G]/[G,G] . This proves that G()/G()0 is a quotient of the closed subgroup G()[G,G]/[G,G] of G/[G,G]  Rq , by a connected subgroup G()0 [G,G]/  [G,G], that is the quotient of some Zr ×Rm by some Rm , such that the quotient is discrete, this implies m = m and G()/G()0  Zr . This proves also that [G(),G()] ⊂ G()0 . Now we have:     G()/G()0 = G()/[G(),G()] / G()0 /[G(),G()]  Zr . (4.7.1) Pick s1, . . . ,sr in G(), such that the image of sj by the map G() → G()/G()0  Zr is the j -th generator of Zr , then we have 5 a s1 1 . . . srar G()0, G() = aj ∈Z

G()/[G(),G()] =

5

s1a1 . . . srar G()0 /[G(),G()].

aj ∈Z

= {s1a1 . . . srar [G(),G()] :

Define the subgroup H aj ∈ Z} of G()/[G(), G()], it is possible to prove that the map H × G()0 /[G(),G()] → G()/[G(),G()], defined by:   s1a1 . . . srar [G(),G()],u[G(),G()] → s1a1 . . . srar u[G(),G()](aj ∈ Z, u ∈ G()0 ) is an isomorphism of groups. Since H  G()/G()0 , this proves the lemma. In the Example 4.7.2, [G(),G()] = {(4π 2 k,0,0) : k ∈ Z} ⊂ G()0 , thus G()/G()0  {xp yq : p,q ∈ Z} with xp yq = yq xp , which means that in the quotient: xp yq xp yq  = xp+p yq+q  , in other words G()/G()0  Z2 .

4.7 Integral orbits

323

Moreover: (G())ab = {xp yq (t,x,u) : p,q ∈ Z, t,x,u ∈ R}/{(4π 2 k,0,0) : k ∈ Z} = {xp yq (t + 4π 2 Z,x,u) : p,q ∈ Z, t,x,u ∈ R}      (Z2 ) × (T × R2 ) = G()/G()0 × G()0 /[G(),G()] . Recall the identification: G()/G()0 = π0 (G())  π1 (G). Since it is abelian, G()/G()0 is isomorphic to the first de Rham cohomology 1 (G) of the orbit G. Thus the preceding proves that if the orbit is group HdR integral, then the set of characters χ of G() such that dχ = i is in one-to1 (G) of all characters of the group one onto correspondence with the set Hˆ dR 1 (G). More precisely, any such character χ has, in the above isomorphism, HdR 1 (G), and χ is the unique the form χ = ν × χ , where ν is a character of HdR  0 character of G() with differential i. Lemma 4.7.5 The form  is integral if and only if [G(),G()] ⊂ ker χ . Proof: Suppose  integral and let χ be a character of G(), with dχ = i|g() . Then [G(),G()] ⊂ ker χ |G()0 = ker χ , therefore [G(),G()] ⊂ ker χ . Conversely, suppose [G(),G()] ⊂ ker χ . Then χ defines a character [χ ] of G()0 /[G(),G()], thus a character [χ ] = 1 × [χ ] of G()ab , thus a character χ of G(), which extends χ defined on G()0 . Thus dχ = i. In the preceding example, [G(),G()] = {(4kπ 2,0,0) : k ∈ Z} ⊂ G()0 , and ker χ = {( 2kπ τ ,0,0) : k ∈ Z}, so the orbit is integral if and only if τ ∈ 1 2π Z. ˜ Remark 4.7.6 Consider the subgroup G() of G() defined by ˜ G() = {s ∈ G() : sxs −1 x −1 ∈ ker χ, for all x ∈ G()}. ˜ In the preceding Since χx −1  = χ (x · x −1 ) for x ∈ G, then G()0 ⊂ G(). decomposition, we get:     0 ˜ ˜ × G()0 /[G(),G()] . G()/[G(),G()] = G()/G() By the argument of Lemma 4.7.5, χ has always an extension to a character of ˜ ˜ G(). The group G() is called the reduced stabilizer of ; it was studied by Pukanszky in [82, 83]. It is easily seen via Lemma 4.7.5 that  is integral if and ˜ only if G() = G().

324

Coadjoint orbits and polarizations

Coming back to Example 4.7.2, a direct computation shows the following. 1. If τ ∈ 2. If τ ∈

1 ˜ 2π Z, then G() = G(). 1 1 c Q, then τ = 2π 2π d , with

gcd(c,d) = 1, then

˜ G() = {xp yq (t,x,u) : (p,q,t,x,u) ∈ (dZ)2 × R3 }. ˜ This can be written as G() = (dZ)2 × G()0 ⊂ Z2 × G()0 = G(). 1 ˜ 3. If τ ∈ / 2π Q, then G() = G()0 . Suppose now that the form  is integral, and h is a polarization in , with d = h ∩ g and D = D 0 G(), where D 0 is the connected subgroup of G with Lie algebra d. In this situation, any character χ of G(), with differential i|g() is extendable in a character of D. Recall that D 0 is simply connected, thus there is a unique character, still denoted χ of D 0 with differential i|d . Let us study the extension of this character to D. Lemma 4.7.7 Let  be an integral form in g∗ , and h a polarization in . Let D 0 be the connected subgroup of G with Lie algebra d = h ∩ g, and χ a character of G(), with differential i|g() . Then there is a unique character χ˜ of the group D = D 0 G(), with differential i|d and such that χ˜ |G() = χ . Proof: Uniqueness of χ˜ is evident: if such a χ˜ exists, then χ˜ (ys) = χ (y)χ (s) for any y ∈ D 0 and s ∈ G(). Conversely, such a formula is well-defined: if y1 s1 = y2 s2 , with yi ∈ D 0 and si ∈ G(), then y2−1 y1 = s2 s1−1 ∈ D 0 ∩ G(), thus χ (y2−1 y1 ) = χ (s2 s1−1 ),

χ (y1 )χ (s1 ) = χ (y2 )χ (s2 ).

Moreover the map χ˜ defined by χ˜ (ys) = χ (y)χ (s) satisfies for any yi si ∈ D, χ˜ (y1 s1 y2 s2 ) = χ˜ (y1 s1 y2 s1−1 s1 s2 ) = χ (y1 )χ (s1 y2 s1−1 )χ (s1 )χ (s2 ). But, for any X ∈ d, χ (s1 exp Xs1−1 ) = ei(Ad(s1 )X) = eiX = χ (exp X). Since D 0 is connected, χ (s1 y2 s1−1 ) = χ (y2 ), and χ˜ is a character. From now on, for each character χ of G(), with differential i|g() , we denote also χ the unique extension of χ to D.

4.7 Integral orbits

325

4.7.2 Cohomologies Let us now give a geometric interpretation of the integrality of a coadjoint orbit O = G. Fix an orbit O in g∗ . Recall that each X ∈ s = gc defines a complex vector field  → ξX , and thus ξ is a Lie algebra homomorphism from s into the Lie algebra of vector fields tangent to O. Moreover O is a symplectic manifolds for the closed 2-form ω, defined by the relation: ω (ξ X,ξ Y ) = [X,Y ]. (See Relation (4.1.2).) Recall here some results on the cohomology of a smooth manifold M. First define the de Rham cohomology as follows: the differential dα of a k-form α is defined by: dα(ξ0, . . . ,ξk ) =

k

  (−1)i ξi α(ξ0, . . . , ξ?i , . . . ,ξk )

i=0

+

(−1)i+j α([ξi ,ξj ],ξ0, . . . , ξ?i , . . . , ξ?j , . . . ,ξk ).

i 0, consider the circle M = {ζ : |ζ | = r}. Fix the following open covering U = (U1,U2,U3 ) of M: Uj = {reia : a ∈ 2πj/3 + (0,4π/3)},

j = 1,2,3.

ˇ cochain is Then each Ui,j is contractible, and U1,2,3 = ∅, thus any 1-Cech 2 2 1 a cocycle, hence H (M) = H (M,U,R) = {0}, but H (M,U,R) is one dimensional, spanned by the cochain: b1,3 = 0, b2,3 = 0, b1,2 = −2π . We saw that b is a cocycle, but not a coboundary, it is easy to see that any cocycle b differs from a multiple of b by a coboundary dc: H 1 (M,U,R) = span{[b]}. On the other hand, H 1 (M) = span{[β = da]}. In the preceding proof, we associate to the closed non exact form β = da = dζ iζ the class of b. Indeed, consider the functions fj (eia ) = a well-defined on Uj , with 2j3π < a < (2j +4)π (j = 2,3) and f1 (eia ) = a + 2π. Then on each Uj , β|Uj = dfj , and 3 on U1,2 , f2 − f1 = −2π, on U1,3 , f3 − f1 = 0, on U2,3 , f3 − f2 = 0. That means, in the isomorphism between H 1 (M,U,R) and H 1 (M), we identify the class [b] with the class [da].  Remark that if U  = (Uj ) is another open contractible covering of M, for any closed 2-form α the same construction gives a collection of 1-forms βj defined on Uj , a collection of functions γj1,j2 defined on Uj1,j2 , a collection of constant cj 1,j2,j3 associated to Uj1,j2,j3 . Suppose Ui1,i2,i3 ∩ Uj1,j2,j3 = ∅, our ˇ = (c + db)j1,j2,j3 . This means that the Cech construction proves that c j1,j2,j3

cohomology does not depend on the choice of the covering U. So we put:

Definition 4.7.10 We say that a closed k-form is integral if its image in HCk (M,U,R) is in HCk (M,U,Z) for a covering U . Example 4.7.11 Let r1 > 0 and r2 > 0, consider the manifold M = M1 × M2 × R2 where M1 = {r1 eia1 : a1 ∈ R/2πZ}, M2 = {r2 eia2 : a2 ∈ R/2πZ}, parametrized by the map (a1,a2,β,α) → (r1 eia1 ,r2 eia2 ,α,β). Consider also the 2-form τ da1 ∧ da2 on M. We define an open contractible covering × Ui(2) × R2 }, where the open subset Ui(k) is the subset U = {U(i1,i2 ) = Ui(1) 1 2 Ui defined in the preceding example for Mk . We choose also the lexicographic ordering on the set I = {1,2,3} × {1,2,3} of indices in U . Let us now compute ˇ the Cech cohomology class of τ da1 ∧ da2 in HC2 (M,U,R). With the notations of Example 4.7.9, we get:

4.7 Integral orbits

329

On each U(i1,i2 ) , τ da1 ∧ da2 = d(τfi1 (a1 )da2 ), following the proof of (1) (1) Proposition 4.7.8, we put fi (a1,a2,α,β) = fi (a1 ) and β(i1,i2 ) = τfi1 da2 . On each U(i1,i2 ),(j1,j2 ) , we have with obvious notations:  (1) (2)  (1) (1) β(j1,j2 ) − β(i1,i2 ) = τ (fj1 − fi1 )da2 = τ d bi1,j1 fj2 . (1)

(2)

We thus put γ(i1,i2 ),(j1,j2 ) = τ bi1,j1 fj2 . On each U(i1,i2 ),(j1,j2 ),(k1,k2 ) , we have: γ(j1,j2 ),(k1,k2 ) − γ(i1,i2 ),(k1,k2 ) + γ(i1,i2 ),(j1,j2 )   (2) (1) (2) (1) (2) = τ bj(1) f − b f + b f . i1,k1 k2 i1,j1 j2 1,k1 k2 Now U(i1,i2 ),(j1,j2 ),(k1,k2 ) = ∅ and (i1,i2 ) < (j1,j2 ) < (k1,k2 ) only if i1 = j1 < k1 or i1 < j1 = k1 , moreover the right-hand side is nonvanishing only if {1,2} ⊂ {i1,j1,k1 }. This implies (i1,j1,k1 ) = (1,1,2) or (1,2,2). In the first (1) (2) (2) case, the right-hand side is τ b1,2 (fk2 − fk2 ) = 0, and in the second case, it (1)

(2)

(2)

(1) (2)

is τ b1,2 (−fk2 + fj2 ) = −τ b1,2 bj2,k2 . ˇ Thus the Cech cohomology class of τ da1 ∧ da2 is [c], where the only nonvanishing ci,j,k are (1) (2)

c(1,i2 ),(2,1),(2,2) = −τ b1,2 b1,2 = −4π 2 τ . τ c da1 ∧ da2 is [ 2π ]. It is in HC2 (M,U,Z), if Thus the cohomology class of 2π 1 and only if τ is in 2π Z. Let us now come back to the coadjoint orbit O = G0 of the Duflo group studied in Example 4.7.2, the orbits being parametrized by the cylinder above, with a1 = a, a2 = b. Recall the Kirillov form ω on O is:

ω = τ da1 ∧ da2 + da1 ∧ dβ + da2 ∧ dα.

  Observe that da1 ∧ dβ + da2 ∧ dα is an exact form, it is −d βda1 + αda2 , therefore [ω] = [τ da1 ∧ da2 ], and O is an integral orbit if and only if the class ω ] is integral. We shall see that this is a general result.  [ 2π

4.7.3 Line bundles on a coadjoint orbit Now consider a simply connected Lie group G and one of its coadjoint orbits O = G0  G/G(0 ). Suppose that O is integral: there is a unitary character χ of G(0 ) whose differential is i0 . Then it is easy to build a natural complex line bundle on G0 = G/G(0 ). To do this define an equivalence relation on G × C: for (s,z) ∈ G × C, say that (s,z) ≡ (sh,χ (h)−1 z),

330

Coadjoint orbits and polarizations

for all h ∈ G(0 ). The quotient space G ×χ C = {[s,z] : (s,z) ∈ G × C} naturally has the structure of a manifold. Indeed, recall that if g(0 ) is the Lie algebra of G(0 ), with m a supplementary space for g(0 ) in the Lie algebra g of G, then for V a sufficiently small open neighborhood of 0 in m, the map ϕs : Y → s exp Y 0 (Y ∈ V ) is a local chart around s0 in O. Then we define a local chart around [s,z], of the form V ×C → G×χ C by putting ψs : (Y,z ) → [s exp Y,z ]. This defines a unique structure of a smooth manifold on G ×χ C. Moreover the surjective map π : G ×χ C → O, given by π : [s,z] → s0 is well-defined and smooth. In fact π gives a structure of a line bundle over O. Indeed, let U = ϕs (V ) be the domain of a local chart in O, then π −1 (U ) is homeomorphic to U × C through the smooth map: θU : [s exp Y,z] → (s exp Y,z),

Y ∈ V , z ∈ C.

The map θU is called a local trivialization of π. Moreover if U  = ϕs  (V  ) is the domain of another local chart, the map θU  ◦θU−1 : (U ∩U  )×C → (U ∩U  )×C, has the following form: θU  ◦ θU−1 (s exp Y 0,z) = θU  [s exp Y,z] = θU  [s  exp Y  h,z] = θU  [s  exp Y ,χ (h)z] = (s exp Y 0,χ (h)z), if h ∈ G(0 ) is defined by s  exp Y  h = s exp Y . Thus, for each  ∈ U ∩ U  , θU  ◦ θU−1 (,z) = (,βU,U  ()z), with |βU,U  ()| = 1. Choose now an open contractible covering U = (Ui ) such that each Ui is a domain for some local chart ϕsi : Vi → O, Vi ⊂ m and the nonempty sets Ui1,...,ik are contractible. Let us now denote βi,j the map βUi ,Uj . Since Ui,j is contractible, we can choose a smooth function bi,j : Ui,j → R such that βi,j () = eibi,j () for any  ∈ Ui,j . We choose also bj,i = −bi,j . Consider now a nonempty set Ui,j,k . For each  ∈ Ui,j,k , each z ∈ C, we have: θUi ◦ θU−1 (,z) = (,eibi,k () z) = θUi ◦ θU−1 ◦ θUj ◦ θU−1 (,z) k j k = (,ei(bj,k ()+bi,j ()) z). Therefore (bj,k − bi,k + bi,j )() ∈ 2πZ and since it is a continuous function on a connected set, there is a constant ci,j,k ∈ Z such that 2π ci,j,k = (bj,k − bi,k + bi,j )(). An immediate computation gives dci,j,k,l = cj,k,l − ˇ cohomology, ci,k,l + ci,j,l − ci,j,k = 0. Thus c is a 2-cocycle in the Cech 2 and [c] ∈ HC (O,U,Z). Let us now prove that [c] is the image of the class ω 2 (O), where ω is the Kirillov form on O. ] ∈ HdR [ 2π

4.7 Integral orbits

331

First, we define a 1-form αˆ on the manifold G × C× , putting z = reiθ with r > 0, the real space tangent at (s,z) to G × C× is generated by the vectors ˆ sX ) = s0 (X), α(∂ ˆ θ ) = 1 and α(∂ ˆ r ) = 0. Now ξsX , ∂θ and ∂r . So we put α(ξ if p is the canonical quotient map p : G × C → G ×χ C, p(s,z) = [s,z], then p −1 ([s,z]) = (sh,χ (h)−1 z) : h ∈ G(0 ) , thus the kernel of dp(s,z) is generated by the tangent vectors ηX , with X ∈ g(), where: # # d ## d ## X −1 η(s,z) = # (s exp tX,χ (exp tX) z) = # (s exp tXs −1 s,e−i0 (tX) z) dt t=0 dt t=0 = (ξsAd(s)X − 0 (X)∂θ )(s,z) . Thus:  X  αˆ η(s,z) = s0 (Ad(s)(X)) − 0 (X) = 0, and αˆ defines by projection a 1-form α on G ×χ C× . Let us denote by αi the restriction of α to π −1 (Ui )∩G×χ C× . We now use the local trivialization θUj , calling αj the form αj ◦ dθUj . By definition, if s = sj exp Yj , with Yj ∈ Vj , s0 ∈ Uj then   #  X  d ##  = s0 (X), (exp −tXs exp Y ,z) = αˆ ξ(s,z) αj j j # dt t=0 αj (∂θ ) = 1,

αj (∂r ) = 0.

This proves that the form αj − dθ is vanishing on ∂θ and ∂r , and we can define X ) = s (X). a form γj on Uj such that γj ◦ dπ = αj − dθ , we get γj (ξs 0 0 Suppose now s0 ∈ Ui,j , or s = si exp Yi h, with h ∈ G(0 ), in the same local trivialization θUj , we get the following expression of αi :   #   X d ##  ibi,j (exp −tXs0 ) X αi (exp −tXsj exp Yj ,e z) = αˆ ξ(s,z) + ξs b ∂ # 0 i,j θ dt t=0   X ), αi ∂θ ) = 1, αi ∂r ) = 0. = s0 (X) + dbi,j (ξs 0 Thus the form αj − αi defines the form γj − γi on Ui,j such that αj − αi = (γj − γi ) ◦ dπ , and γj − γi = dbi,j . This proves there is a unique 2-form ω on O such that, on each Ui , ω = dγi , and since, on each Ui,j,k , bj,k − bi,k + bi,j = 2π ci,j,k , by definition the ˇ cohomology class de Rham cohomology class [ω ] corresponds to the Cech X [2π c]. Finally, on Uj , we saw that γj (ξ ) = (X), thus, for any X, Y ∈ g, ω (ξ X,ξ Y ) = ξX ((Y )) − ξY ((X)) − γj ([ξ X,ξ Y ]| ) = [X,Y ] = ω (ξ X,ξ Y ).

332

Coadjoint orbits and polarizations

ω ], where ω is the Kirillov form Thus, if O is integral, the cohomology class [ 2π 2 is the integral cohomology class [c] ∈ HC (O,U,Z). In fact the converse holds (see B. Kostant [57]) and this gives a geometric characterization of the integrality of a coadjoint orbit O, in terms of its Kirillov form ω:

Theorem 4.7.12 The coadjoint orbit O is integral if and only if the 2-form 1 2π ω is integral.

5 Irreducible unitary representations

In this chapter, we embark upon the task of the construction of irreducible unitary representations of a simply connected solvable Lie group G. As indicated in Section 4.1, the goal is to canonically associate to a coadjoint ˆ of G. A key requirement however is ˆ O of the dual space G orbit O a subset G that O be integral: for each  ∈ O there is a unitary character χ of G() such that dχ = i|g() . Specifically, given the following data, 1. a point  ∈ g∗ that is integral, 2. a polarization h at  such that a. h is positive, b. h is admissible for some nilpotent ideal n containing [g,g], c. h is G()-stable, d. h ∩ n is G(|n )-stable, and 3. a character χ of G() such that dχ = i|g() , we define a unitary representation ind(G,,h,χ ) in Section 5.1 by a method which generalizes the inducing construction of Chapter 3. After a crucial extension procedure that is described in Section 5.2, we show in Section 5.3 that the representation ind(G,,h,χ ) is irreducible. Moreover, the equivalence class of this representation is independent of the choice of a polarization h satisfying properties a–d, while the map χ → [ind(G,,h,χ )] is one-to-one. Finally, we show that for s ∈ G, if  = s, h = Ad(s)h and χ  = χ (s −1 · s), then ind(G,,h,χ ) and ind(G,,h,χ  ) are equivalent. Thus ˆ O = {[ind(G,,h,χ )] : dχ = i|g(), ∈ O}. G

333

334

Irreducible unitary representations

5.1 Holomorphic induction For a simply connected nilpotent Lie group G, every irreducible unitary representation is equivalent with a representation of the form ind(G,H,χ ), where the Lie algebra h of H is a g-polarization at a point  and χ is the unique unitary character of the closed subgroup H = exp h with dχ = i|h . Thus, the inducing construction described in Chapter 3 leads to the Kirillov correspondence, whereby each coadjoint orbit corresponds to one and only one class of irreducible unitary representations. For exponential solvable Lie groups, this approach is successful in almost the same way. Since the coadjoint action is of exponential type, the stabilizers G() for this action are connected (Proposition 1.9.13), and each  ∈ g∗ is therefore integral. Moreover, each  ∈ g∗ admits a g-polarization h that satisfies the Pukanszky condition (Proposition 4.6.11), and in this case we will see that the unitary representation ind(G,H,χ ), with χ as above, is irreducible. As a result, in the case where G is exponential, there is again a canonical one-to-one correspondence between coadjoint orbits and equivalence classes of irreducible unitary representations. For simply connected solvable groups in general, the situation is more complicated. We have seen an example of a nonexponential solvable Lie group G – the diamond group – for which almost all  ∈ g∗ have no real polarization. Even worse, we have seen an example of a solvable group G and a point  ∈ g∗ that is not integral: such that G() is not connected and for which there is no unitary character defined on the stabilizer G(). In this section we define and study a more general notion of induction which is needed for the case where there is no real polarization.

5.1.1 Definition of holomorphic induction To define holomorphic induction for a solvable Lie group, let g be a solvable Lie algebra over R, and let G be a solvable Lie group with Lie algebra g. (It is not assumed that G is connected.) Fix  ∈ g∗ and let h ⊂ gc be a G()-stable polarization at . Recall we put e = (h + h) ∩ g and d = h ∩ g. Let E 0 and D 0 be the corresponding connected Lie subgroups of G; note that since g() ⊂ d, then G()0 ⊂ D 0 . Recall that D = D 0 G() and E = E 0 G() are subgroups of G, that D is closed (Lemma 4.2.9), and that neither D nor E is necessarily connected. For Y ∈ e, define the left-invariant differential operator R(Y ) on C ∞ (G) by # d ## R(Y )φ = φ(· exp tY ) dt #t=0

5.1 Holomorphic induction

335

and extend R to ec by linearity. Since [d,d] ⊂ ker , then the formula χ (exp Z) = ei(Z) defines a unitary character χ of D 0 . We now invoke the assumption of integrality: suppose that χ |G()0 extends to some unitary character χ of G(); by Corollary 4.7.7, then χ extends to a unitary character of D which we still denote by χ . Now let H(G,D,χ ) be the Hilbert space of the induced representation ind(G,D,χ ) as in Chapter 3. Recall that if F ∈ H(G,D,χ ) then for any d ∈ D, F (· d) = δD,G (d)1/2 χ (d)−1 F , and so if F ∈ C ∞ (G), then for any Y ∈ dc ,   1 (5.1.1) R(Y )F = −i(Y ) + trace adgc /dc (Y ) F . 2 The idea of holomorphic induction is to extend the condition (5.1.1) to all of h. First we prove a lemma. Lemma 5.1.1 The character δD,E is trivial. Proof: We have seen that every element d ∈ D fixes the restriction of  to e. Also, the alternate form B defines a nondegenerate alternate form on e/d: B(X + d,Y + d) := [X,Y ]. Now the linear mapping Ade/d (d) leaves the form B invariant, and hence δD,E (d) = det Ade/d (d) = 1. In order to extend (5.1.1) to Y ∈ h, define hol,h C(G,D,χ ) to be the subspace of H(G,D,χ ) consisting of those functions F ∈ C ∞ (G) ∩ H(G,D,χ ) satisfying   1 (5.1.2) R(Y )F = −i(Y ) + trace adgc /dc (Y ) F 2 for all Y ∈ h. Let hol,h H(G,D,χ ) denote the closed subspace of H(G,D,χ ) that is the completion of the subspace hol,h C(G,D,χ ). Since the condition (5.1.2) is left-invariant, then hol,h H(G,D,χ ) is an invariant subspace for the unitary representation ind(G,D,χ ). Denote the restriction of ind(G,D,χ ) to the invariant subspace hol,h H(G,D,χ ) by hol,h ind(G,D,χ ). This is the unitary representation defined by holomorphic induction. Now suppose that E = E 0 G() is a closed subgroup of G. Since ec contains h, the holomorphic condition (5.1.2) can just as well be applied to functions on E. To describe the representation hol,h ind(G,D,χ ), it will be useful to first consider hol,h ind(E,D,χ ); in fact, as will be shown below, there is a very explicit description of the space hol,h H(E,D,χ ) when G is nilpotent. Since δD,E = 1 (Lemma 5.1.1) then hol,h H(E,D,χ ) is the completion of the subspace hol,h C(E,D,χ ) of functions in C ∞ (E) ∩ H(E,D,χ ) satisfying R(Y )ϕ = −i(Y ) ϕ

336

Irreducible unitary representations

for all Y ∈ h. Of course the representation hol,h ind(E,D,χ ) also acts by left translation. Moreover, the following shows that induction in stages works also for holomorphic induction. Lemma 5.1.2 Let  ∈ g∗ be integral and fix a character χ of G() such that dχ = i|g() . Let h be a polarization at , and suppose that E = E 0 G() is closed. Set τ = hol,h ind(E,D,χ ). Then hol,h ind(G,D,χ )  ind(G,E,τ ). Proof: Recall the unitary intertwining map, defined on Cc (G,D,χ ) as F → fF in the proof of Proposition 3.3.29: −1/2

fF (x)(h) = F (xh)δE,G (h), x ∈ G, h ∈ E. Denote this map by U . We must show that U maps hol,h H(G,D,χ ) onto H(G,E,τ ). First we show that U maps hol,h C(G,D,χ ) into H(G,E,τ ). Let F ∈ hol,h C(G,D,χ ), fix x ∈ G and put v = fF (x). Then for any Y ∈ e, # # d ## d ## R(Y )v(y) = # v(y exp tY ) = # F (xy exp tY )δE,G (y exp tY )−1/2 dt t=0 dt t=0 1 = R(Y )F (xy)δE,G (y)−1/2 − trace(adg/e (Y ))F (xy)δE,G (y)−1/2 . 2 If W ∈ h, then R(W )F (xy) = (−i(W ) + 12 trace(adgc /dc (W )))F (xy). Hence 

 1 R(W )v(y) = −i(W ) + trace(adgc /dc (W )) F (xy)δE,G (y)−1/2 2 1 − trace(adgc /ec (W ))F (xy)δE,G (y)−1/2 2  1 = −i(W ) + trace(adec /dc (W )) F (xy)δE,G (y)−1/2 2   1 = −i(W ) + trace(adec /dc (W )) v(y), 2 showing that U (F ) belongs to H(G,E,τ ). To finish the proof it is enough to show that U (hol,h C(G,D,χ )) is dense in H(G,E,τ ). Put F = {fφ,v : φ ∈ Cc∞ (G),v ∈ hol,h C(E,D,χ )}. By definition hol,h C(E,D,χ ) is dense in the Hilbert space hol,h H(E,D,χ ), so by Lemma 3.3.17, the space F is dense in H(G,E,τ ). We

5.1 Holomorphic induction

337

show that U (hol,h C(G,D,χ )) includes F, and this is proved in exactly the same way as the proof of Proposition 3.3.29: given fφ,v ∈ F, put 3 F (x) = φ(xy)v(y −1 )δE,G (y)−1/2 dy. E

We have seen in loc. cit. that F ∈ Cc (G,D,χ ) and U (F ) = fφ,v ; now it remains to show that F ∈ hol,h C(G,D,χ ), and this amounts to a computation of R(W )F for W ∈ h. First let Y ∈ e; then we have 3 φ(x exp tY y)v(y −1 )δE,G (y)−1/2 dy F (x exp tY ) = E

3 = 3

φ(xy)v(y −1 exp tY )δE,G (exp −tY y)−1/2 dy

E

φ(xy)v(y −1 exp tY )δE,G (y)−1/2 dy δE,G (exp tY )1/2 .

= E

Taking the derivative at t = 0 gives 3   φ(xy) R(Y )v(y −1 ) δE,G (y)−1/2 dy R(Y )F = E   3 1 −1 −1/2 φ(xy)v(y )δE,G (y) dy + trace adg/e Y . 2 E Since v ∈ hol,h C(E,D,χ ), then R(W )v = −i(W )v holds for each W ∈ h. Hence   1 R(W )F = −i(W ) + trace adg/e (W ) F . 2 Thus F belongs to hol,h C(G,D,χ ) and the proof is finished. For the construction of irreducible unitary representations of a simply connected solvable Lie group G, the method of holomorphic induction will be applied for polarizations satisfying the special conditions described in Sections 4.2, 4.3, and 4.6 and recalled at the very beginning of the chapter. In particular, we will see that the assumption that h is a positive polarization insures that the Hilbert space hol,h H(G,D,χ ) is nontrivial. An important ingredient will be the application and analysis of holomorphic induction to a simply connected nilpotent subgroup N of G that contains [G,G], and a thorough understanding of holomorphic induction for a nilpotent group is an essential aspect of the development of the general theory. We take up this task in the next subsection, and it is there that the importance of positivity of h will become clear.

338

Irreducible unitary representations

In what follows a slightly simpler notation will also be used: C(G,,h,χ ) = hol,h C(G,D,χ ), H(G,,h,χ ) = hol,h H(G,D,χ ), and ind(G,,h,χ ) = hol,h ind(G,D,χ ). Observe that the representation ind(G,,h,χ ) depends only on the restriction |e of  to e. Thus, for any subgroup K with Lie algebra k of G such that e ⊂ k, and any character χK of G()∩K, we will write ind(K,,h,χK ) instead of ind(K,|k,h,χK ).

5.1.2 The nilpotent case If G is simply connected and nilpotent and  ∈ g∗ , then by Corollary 4.3.3, every polarization at  is G()-stable and satisfies the Pukanszky condition. Moreover, for every  there is a g-polarization, and the orbit method may be fully implemented via g-polarizations. The following theorem, originally proved by A. A. Kirillov [52], shows that there is a canonical bijection ˆ known as the Kirillov map, or Kirillov correspondence. K : g∗ /G → G, A comprehensive exposition of this work, together with many other aspects of harmonic analysis on nilpotent Lie groups, is found the book of L. Corwin and F. Greenleaf, [16]. Theorem 5.1.3 Let G be a simply connected nilpotent Lie group, let  ∈ g∗ , and let p be a g-polarization at , with P = exp p and χ the character of P with dχ = i|p . Each of the following holds. (1) ind(G,P ,χ ) is irreducible. (2) Let p be another g-polarization at , with P  = exp p with χ the character of P  for which dχ = i|p . Then ind(G,P ,χ ) is equivalent with ind(G,P ,χ ). (3) Let  = s for some s ∈ G so that Ad(s)p is a g-polarization at s, and χs = χ (s −1 · s) is the character of sP s −1 with d(χs ) = is|Ad(s)p . Then ind(G,sP s −1,χs ) is equivalent with ind(G,P ,χ ). Thus if G is a simply connected real nilpotent Lie group, an element K(O) ˆ is canonically associated to each coadjoint orbit O ⊂ g∗ by of the dual G inducing from a unitary character χ of an analytic subgroup, where −idχ is the restriction of an element of O. Now suppose that G is solvable, let  ∈ g∗ , and let O be the coadjoint orbit of . If there is no real polarization at  (see Example 4.2.4), then it is necessary to choose a positive complex polarization and carry out holomorphic induction. The proof that this method yields an irreducible unitary representation requires that we consider a holomorphically

5.1 Holomorphic induction

339

induced representation of a normal, nilpotent subgroup of G. Accordingly, the goal of this subsection is to prove the following. Theorem 5.1.4 Let G be a simply connected nilpotent Lie group, let  ∈ g∗ , and let O be the coadjoint orbit of . Let h ⊂ gc be a polarization at  with χ the character of D = exp d such that dχ = i|d . Assume that the polarization h is positive for . Then ind(G,,h,χ ) is irreducible and belongs to the equivalence class K(O). The proof of this theorem will consist of a number of steps. Fix a positive polarization h at  with h = h. Recall the real subalgebras dc = h ∩ h and ec = h+h, and the Hermitian form H on h, giving a nondegenerate Hermitian form H on h/dc . Choose W1, . . . ,Wm in h such that {Wl + dc : 1 ≤ l ≤ m} is an H -orthogonal basis for h/dc , and set m = spanC {Wl : 1 ≤ l ≤ m}. Put λl =

1 i H (Wl ,Wl ) = [Wl ,Wl ]. 4 2

(5.1.3)

Note that none of the elements Wl are real: Wl = Wl . Write Wl = Xl + iYl with Xl ,Yl ∈ g, and put Zl = [Xl ,Yl ] =

i [Wl ,W l ] 2

so that λl = (Zl ) > 0, 1 ≤ l ≤ m. Recall that d = dc ∩g, e = ec ∩g. Of course since G is nilpotent, D = D0 = exp d, E = E0 = exp e, h is G()-stable and satisfies the Pukanszky condition. Recall that by Lemma 4.6.8 and Corollary 4.6.9, [e,e] ⊂ d and e/(d ∩ ker ) is a Heisenberg Lie algebra. In particular, Zl ∈ d \ ker , 1 ≤ l ≤ m. Put K = exp(d ∩ ker ) and let χ be the unique character of D with differential i|d ; then K ⊂ ker χ and E/K is a simply connected Heisenberg group (Corollary 4.6.9). The following is an immediate consequence of these observations and the Baker–Campbell–Hausdorff formula. Lemma 5.1.5 (1) The exponential mapping exp : (m + m) ∩ g → E induces a bijection (m + m) ∩ g → E/D. (2) Let Y,Y  ∈ e; then 1 exp Y exp Y  = exp(Y + Y  ) exp [Y,Y  ] mod K. 2 Now let ρ = ind(E,,h,χ ) be the holomorphically induced representation of E by  and h, and for simplicity put Hρ = H(E,,h,χ ). By Lemma 5.1.2, ind(G,E,ρ)  ind(G,,h,χ ).

340

Irreducible unitary representations

Note that since E and G are nilpotent, all modular functions are trivial, and there is a G-invariant Borel measure μG/E on G/E. Denote by qG/E : G → G/E the canonical map. Recall (see Chapter 3) that ind(G,E,ρ) acts by left translation in the completion H(G,E,ρ) of the space Cc (G,E,ρ) of all functions F ∈ C(G,Hρ ) for which F (xy) = ρ(y)−1 F (x), y ∈ E and qG/E (supp(F )) is compact in G/E, with respect to the norm given by 3 F 2= F (x) 2 dμG/E (x) < ∞. G/E

The proof of Theorem 5.1.4 will require an explicit description of the space Hρ . Recall the space C(E,,h,χ ) of functions ϕ in C ∞ (E) ∩ H(E,D,χ ) satisfying R(Y )ϕ = −i(Y ) ϕ, Y ∈ h. As above E/D also has an E-invariant Borel measure μE,D . In fact, let α : Cm × D → E be defined by   (5.1.4) α(w,d) = exp Re(w1 W1 ) + · · · + Re(wm Wm ) d. By Lemma 5.1.5, α is a homeomorphism, and for simplicity we write now α(w,d) = (w,d). Note that for ϕ ∈ H(E,D,χ ), ϕ(w,d) = χ (d)−1 ϕ(w,e). It is clear that w → (w,e)D is a homeomorphism αE/D of Cm with E/D. Moreover, for any function  on E that is constant on left D-cosets, ((w0,d0 )(w,e)) = (w0 + w,e) so, with dw denoting the Lebesgue measure on Cm , 3 3 ((w0,d0 )(w,e))dw = (w0 + w,e)dw. Cm

Cm

Thus we take μE,D to be the measure dw, and the norm is given by 3 |ϕ(w,e)|2 dw. ϕ 2=

·

on C(E,,h,χ )

Cm

For wl = xl + iyl ∈ C, xl ∈ R, yl ∈ R, put ∂l = 2

∂ ∂ ∂ ∂ ∂ = −i , and ∂ l = +i , 1 ≤ l ≤ m. ∂wl ∂xl ∂yl ∂xl ∂yl

With the coordinates (w,d) for E, we regard ∂l , ∂ l as differential operators on C ∞ (E). Lemma 5.1.6 Let ϕ be any smooth function in H(E,D,χ ). Then for each 1 ≤ l ≤ m,   1 R(Wl )ϕ = ∂ l + λl wl ϕ. 2

5.1 Holomorphic induction Proof: Write Wl = Xl + iYl , wl = xl + iyl , and Re(wW ) = for brevity. Since ϕ is constant on K-cosets, Lemma 5.1.5 gives

341  l

Re(wl W l )

ϕ((w,d) exp tXl ) = ϕ(exp Re(wW )d exp tXl ) = ϕ((w1,w2, . . . ,wl + t, . . . ,wm,d) exp(−(yl t/2)Zl )) = ϕ(w1, . . . ,wl + t, . . . ,wm,d)eiλl yl t/2 so that

 R(Xl )ϕ(w,d) =

Similarly,

 R(Yl )ϕ(w,d) =

and

 ∂ iλl + yl ϕ(w,d). ∂xl 2

 ∂ iλl − xl ϕ(w,d), ∂yl 2

  1 R(Wl )ϕ = R(Xl )ϕ + iR(Yl )ϕ = ∂ l + λl wl ϕ. 2

Proposition 5.1.7 Put γl = (Wl ), 1 ≤ l ≤ m. The subspace C(E,,h,χ ) is precisely the set of all functions ϕ ∈ H(E,D,χ ) ∩ C ∞ (E) satisfying   1 ∂ l ϕ(w,d) = −iγl − λl wl ϕ(w,d), 1 ≤ l ≤ m. 2 Proof: Observe that the holomorphic condition (5.1.2) holds for a smooth function ϕ ∈ H(E,D,χ ) if and only if R(Wl )ϕ = −iγl ϕ, 1 ≤ l ≤ m.

(5.1.5)

The result follows from Lemma 5.1.6 and a direct calculation. An immediate consequence of the preceding is the following. Let  ∈  + d⊥ , and put γl =  (Wl ), 1 ≤ l ≤ m. Then  belongs to the orbit of  and determines the same character χ on D as does . Moreover h is a positive polarization at  , and the map ϕ → ϕ eiRe((γ −γ

 )w)

on H(E,D,χ ) defines an isometric isomorphism of C(E,,h,χ ) onto C(E,,h,χ ) intertwining ρ with ind(E,,h,χ ). The equivalence of ind(G,,h,χ ) and ind(G,,h,χ ) then follows from Proposition 3.3.19 and Lemma 5.1.2. Henceforth we assume then that  is chosen such that γl = 0, 1 ≤ l ≤ m.

342

Irreducible unitary representations

Define ϕ0 on E by ϕ0 (w,d) = χ (d)−1

m 4

e−λl wl wl /4 .

l=1

Since each λl is positive (here is where the fact that h is a positive polarization is used), 3 3 2 2 |ϕ0 (w,e)|2 dw = e−(λ1 |w1 | +···+λm |wm | )/2 dw < ∞. (5.1.6) Cm

Cm

Evidently ϕ0 (xd) = χ (d)−1 ϕ0 (x) for all d ∈ D, hence ϕ0 ∈ H(E,D,χ ). Now for each 1 ≤ l ≤ m, ∂ l ϕ0 = − 12 λl wl ϕ0 so by Proposition 5.1.7, ϕ0 belongs to C(E,,h,χ ). Now let p : Cm → C be holomorphic on Cm , and consider the function ϕ on E defined by ϕ(w,d) = p(w)ϕ0 (w,d). Observe that 1 ∂ l ϕ = p ∂ l ϕ0 = − λl wl ϕ, 2 and

3

3

Cm

|ϕ(w,e)|2 dw = =

3

Cm Cm

|p(w)|2 |ϕ0 (w,e)|2 dw |p(w)|2 e−(λ1 |w1 |

2 +···+λ |w |2 )/2 m m

dw.

(5.1.7)

Again by Proposition 5.1.7, ϕ belongs to C(E,,h,χ ) if and only if the integral (5.1.7) is finite. Let A = {p : Cm → C : p is holomorphic on Cm, and p 2A < ∞} where

3 p A= 2

Cm

|p(w)|2 e−(λ1 |w1 |

2 +···+λ

m |wm |

2 )/2

dw.

Observe that A is an inner product space where 3 2 2 p1,p2  = p(1 w)p2 (w)e−(λ1 |w1 | +···+λm |wm | )/2 dw. Cm

Lemma 5.1.8 The map T : A → C(E,,h,χ ) defined by T p(w,d) = p(w)ϕ0 (w,d) is an isometric isomorphism. Proof: It remains only to prove that the map T is surjective. But this is almost immediate: given ϕ ∈ C(E,,h,χ ), put p(w) = ϕ(w,d)ϕ0 (w,d)−1 . It is clear that p is well-defined and holomorphic since

5.1 Holomorphic induction

343

  1 ∂ l ϕ0 (w,d)−1 = λl wl ϕ0 (w,d)−1 . 2 Evidently p 2A = ϕ

2

< ∞ so p ∈ A and T p = ϕ.

The space A with the norm · A is known as the Segal–Bargmann space, or the Bargmann–Fock space (where ordinarily λl = 1,1 ≤ l ≤ m.) The next step is to prove that A is complete. We introduce the following notation. For w ∈ Cm write its polar form w = (r1 eit1 ,r2 eit2 , . . . ,rm eitm ) as reit . Given a function g of a single complex variable, the function on Cm defined by (u1,u2, . . . ,um ) → g(u1 )g(u2 ) · · · g(um ) will be written as g(u). For instance eλ1 t1 +···+λm tm = eλt ,

λ = λ1 λ2 . . . λm,

or

ϕ0 (w,d) = χ (d)−1 e−λ|w|

2 /4

.

For any 0 < r ≤ ∞, let U (r) = {w ∈ Cm : |wl | < r,1 ≤ l ≤ m}. Lemma 5.1.9 For any bounded subset S of Cm , there is a constant CS such that, for any p ∈ A, sup |p(z)| ≤ CS p A . z∈S

Proof: Given a bounded subset S, choose R > 1 such that S ⊂ U (R − 1). The Cauchy integral formula yields 1 p(z) = (2π )m =

1 (2π )m

3 [0,2π ]m

p(r1 eit1 ,r2 eit2 , . . . ,rm eitm )

m 4 l=1

3 p(reit ) [0,2π ]m

irl eitl dt1 . . . dtm rl eitl − zl

ireit dt reit − z

for z ∈ S, and r1 ≥ R,. . . , rm ≥ R. Now integrate with respect to rl ,1 ≤ l ≤ m on [R,∞): 1 2 (2π )m e−λR /2 p(z) = λ =

3

3

[0,2π ]m

3

Cm \U (R)

p(reit )e−λr [R,∞)m

p(w)e−λ|w|

2 /2

2 /2

ireit drdt reit − z

isign(w) dw. w−z

344

Irreducible unitary representations

# # # # Since # wl 1−zl # ≤ 1,1 ≤ l ≤ m, then 3 1 2 2 (2π )m e−λR /2 |p(z)| ≤ |p(w)|e−λ|w| /2 dw m λ C 3 1/2 3 −λ|w|2 /2 ≤ e dw Cm

=

2 −λ|w|2 /2

Cm

|p(w)| e

1/2 dw

(2π )m p A, λ

and the estimate |p(z)| ≤ eλR

2 /2

p A

holds for all z ∈ S. For each multi-index k = (k1,k2, . . . ,km ) with kl ∈ {0,1,2, . . .}, denote by mk the monomial km = wk . mk (w1,w2, . . . ,wm ) = w1k1 w2k2 · · · wm

Observe that the linear span of the family (mk )k is included in A, and since A consists of holomorphic functions, then each function p ∈ A can be written as  p = k ak mk where (ak )k is a multi-indexed sequence of complex numbers, and where convergence is uniform on bounded subsets of C m . Lemma 5.1.10 For any multi-indices j and k with j = k, 3 2 w j w k e−λ|w| /2 dw = 0, U (r)

and hence the family (mk )k is a pairwise orthogonal family in A. Proof: For any distinct multi-indices j and k, 3 m 3 4 2 2 j w j w k e−λ|w| /2 dw = wl l wl kl e−λ|wl | /2 dwl U (r)

l=1

and when jl = kl , 3 3 jl kl − 12 λl |wl |2 wl w l e dwl = |wl | 0, R > 0, we get nc = span{T ,Z,Z,W,W }, n(f ) = n. Recall that G() was computed in Example 4.7.2. If xp = exp 2πpA(0,0,2π ipτ/R) and yq = exp 2π qB(0,2π iqτ/r,0), then G() = ∪xp yq G()0 . Now g(f ) = n, hence 2 = f , and this gives: 5 xp yq N, Z(2 ) = G(f ). G(f ) = p,q∈Z

Here, q = span{τ X − rT ,τ U − RT ,Y,V }, therefore G(f )0 /Q is one dimensional, abelian, but G(f )/Q is not connected and not abelian. More precisely [xp yq n,xp yq  n ] = exp 4π 2 (pq  − p q)T , thus [G(f )/Q,G(f )/Q] = exp 4π 2 ZT . Consider now the case  = T ∗ = (1,0,0,0). In this case n(f ) = n and g(f ) = g, 2 = , moreover g() = n = span{T ,X,Y,U,V }, and G() = N = Z(2 ), and q = span{X,Y,U,V }. So G(f )0 /Q = span{A + Q, B + Q,T + Q}. Since [A,B] = T , then G(f )/Q = G(f )0 /Q is a Heisenberg group of dimension three.  Denote both canonical maps g(f ) → g(f )/q and G(f ) → G(f )/Q by p. For s ∈ G(f ) put s˜ = p(s) ∈ G(f )/Q. Since q ⊂ ker 2 , we have ˜2 ∈ =2 = s˜ ˜2 for s ∈ G(f ). Observe that (g(f )/q)∗ such that 2 = ˜2 ◦ p, and s d2 = h2 ∩ g(f ) contains q, and we denote p(h2 ), p(d2 ) by h˜ 2 , d˜ 2 , respectively. The connected subgroup of G, with Lie algebra d2 is denoted by D20 . Observe that by Lemma 5.2.3, n(f ) ⊂ z(2 ) ⊂ d2 , so N (f ) ⊂ Z(2 )0 ⊂ D20 . On the other hand, D20 /Q = D˜ 2 is the connected Lie subgroup of the simply connected nilpotent Lie group G(f )0 /Q with Lie algebra d˜ 2 . Suppose now that h is positive. By Corollary 4.3.10, h2 = h ∩ g(f ) is a polarization of g(f ) at 2 , which is still positive. Let χ˜2 be the unique unitary character of D˜ 2 determined by ˜2 . Now h˜ 2 is a positive polarization at ˜2 , so by Theorem 5.1.4, the unitary representation η˜ = ind(G(f )0 /Q, ˜2, h˜ 2,χ ˜ ) 2

is irreducible. Now χ2 = χ˜2 ◦ p is the unique unitary character of D20 with dχ2 = i2 |d2 . The unitary representation

5.2 Construction of irreducible representations

355

η = ind(G(f )0,2,h2,χ2 ) satisfies η = η˜ ◦ p and is an irreducible representation of G(f )0 . Suppose next that  is integral: there is a character χ of G() whose differential is i|g() . Let M be a subgroup of G(f ) such that G(f )0 ⊆ M ⊆ G(f ). Note that the Lie algebra of M is g(f ), that M 0 = G(f )0 , that G(f )0 is normal in M, that M is closed in G(f ) and hence in G, and finally, that M/G(f )0 is discrete. By restriction, χ is a character of G() ∩ M. Thus we may define ηM,χ = ind(M,2,h2,χ |G()∩M ).

(5.2.2)

Observe that by definition of holomorphic induction, the restriction of χ to G() ∩ M has a unique extension to the group DM = D20 M(2 ) = D20 (Z(2 ) ∩ M) as a character having differential i2 |d2 . By a slight abuse of notation, this character of DM is also denoted by χ . Note that χ2 is determined by 2 , so the representation η depends only upon 2 and h2 . On the other hand, if Z(2 ) ∩ M is not connected, then the character χ is not determined by 2 even though it satisfies dχ = i2 |z(2 ) . Let us now show that, nevertheless, ηM,χ is also irreducible, by using Mackey theory. To do this, we shall need to use the forthcoming Lemma 5.3.8: ηs = ind(G(f )0,s2,sh2,χs2 ) and η˜ s˜ = ind(G(f )0 /Q, s˜ ˜2, s˜ h˜ 2,(χ˜ 2 )s˜ ) hold for each s ∈ G(f ). So we determine now the subgroup M(η) = {s ∈ M : ηs  η}. Lemma 5.2.7 Let M be a subgroup of G satisfying G(f )0 ⊆ M ⊆ G(f ). Then M(η) = G(f )0 (M ∩ G()). Proof: Let s ∈ M such that ηs  η. Then η˜ s˜  η. ˜ By the Kirillov correspondence for simply connected nilpotent Lie groups, s˜ ˜2 belongs to the G(f )0 / Q-orbit of ˜2 , so we have y˜ ∈ G(f )0 such that s˜ ˜2 = y˜ ˜2 . Now pick y ∈ G(f )0 such that p(y) = y; ˜ then s2 = y2 , which gives s ∈ G(f )0 G(). Thus s ∈ G(f )0 G() ∩ M = G(f )0 (M ∩ G()), since G(f )0 ⊂ M. Conversely, suppose s = yz with y ∈ G(f )0 and z ∈ M ∩ G(), then z ∈ M ∩ G() ⊂ Z(2 ), so s˜ ˜2 = y˜ z˜ ˜2 = y˜ ˜2 belongs to the G(f )0 /Q-orbit ˜ and ηs  η. of ˜2 , therefore η˜ s˜  η, Now Z(2 ) = N (f )G(), and N (f ) ⊂ M, thus Z(2 ) ∩ M = N (f )(G() ∩ M), and DM = D20 N (f )(G() ∩ M) = D20 (G() ∩ M).

356

Irreducible unitary representations

Recall the irreducible representation η = ind(G(f )0,2,h2,χ ) of G(f )0 . We have G(f )0 ⊆ M(η) ⊆ M where M(η) = {s ∈ M : ηs  η} = G(f )0 (M ∩ G()) by Lemma 5.2.7. Note that G(f )0 is an open subgroup of both M and M(η). Put now η0 = ind(M(η),2,h2,χ |M(η) ). Proposition 5.2.8 We have the following. (1) The representation η0 is an extension of η to M(η). Moreover the representation ηM,χ is equivalent with ind(M,M(η),η0 ) and is an irreducible unitary representation of M. (2) Let χ and χ  be characters of DM with differential i2 |d2 . If the irreducible representations ηM,χ and ηM,χ  are equivalent, then χ = χ  . (3) Let 2 ∈ g(f )∗ , h2 a polarization at 2 in g(f ), and χ  the corresponding 2

character of (D2 )0 , denote η the corresponding representation of G(f )0 , if the G(f )-orbits of 2 and 2 in g(f )∗ are disjoint, then the  corresponding representations ηM,χ , ηM,χ  are inequivalent.

Proof: By Lemma 5.1.2, ηM,χ  ind(M,M(η),η0 ). We claim that the restriction of functions on M(η) to G(f )0 defines an isomorphism between Hη0 and Hη . To see this, let ψ ∈ Cc (M(η),DM ,χM ), and put rψ = ψ|G(f )0 . Then there is α ∈ Cc (G(f )0 ) such that: 3 2 |rψ(u)| = α(uy) |detAdg(f )/d2 (u)| dμD 0 (y). 2

D20

Observe that since g(f )/q is a nilpotent algebra, then detAdg(f )/d2 (u) = detAd(g(f )/q)/(d2 /q) (u) = 1. Since G(f )0 is open in M(η) and D20 = DM ∩ G(f )0 is an open subgroup of DM , then we may regard α as a continuous function with compact support in M(η); moreover, for u ∈ G(f )0 , 3 3 α(uy) dμDM (y) = α(uy) dμD 0 (y) = |ψ(u)|2, D20

DM

2

and for s ∈ M ∩ G(), 3 α(usy) dμDM (y) = |ψ(u)|2 = |ψ(us)|2 . DM

5.2 Construction of irreducible representations

This means rψ

2

3 =

G(f )0

357

3 α(u) dμG(f )0 (u) =

α(u) dμM(η) (u) = ψ

2

.

M(η)

Thus r extends to an isomorphism of H(M(η),DM ,χM ) onto H(G(f )0,D20,χ ). Now, each u ∈ M(η) has a neighborhood of the form G(f )0 s, for some s ∈ G() ∩ M, and ψ(u) = rψ(us −1 )χ (s)−1 . It is thus clear that ψ ∈ H(M(η), DM ,χM ) is smooth if and only if rψ is smooth. Similarly, since by Corollary 4.3.10, h2 is G()-stable, then for any s ∈ G(), a direct computation proves that R(Y )ψ(vs) = R(Ad(s)Y )ψ(v)χ (s)−1 . Hence if rψ satisfies the holomorphic condition on G(f )0 , then   1 R(Y )ψ(vs) = −i(Y ) + traceg(f )c /d2c (Y ) χ (s)−1 ψ(v). 2 Suppose now u ∈ M(η) and u = vs with v ∈ G(f )0 and s ∈ G() ∩ M,   1 R(Y )ψ(u) = R(Y )ψ(vs) = −i(Y ) + traceg(f )c /d2c (Y ) ψ(u). 2 The holomorphic condition (5.1.2) is fulfilled in C(M(η),DM ,χM ) if and only if it is fulfilled in C(G(f )0,D20,χ ). This proves the claim. Since both η0 and η act by left translation, it is clear that r intertwines η0 |G(f )0 and η. This means that η0 is equivalent with an extension of η. Observe that for each s0 ∈ G() ∩ M, each t ∈ G(f )0 , (η0 (s0 )rψ)(t) = χ (s0 )rψ(s0−1 ts0 ).

(5.2.3)

˜ η˜ 0 , η˜ M,χ respectively Now regard η, η0 and ηM,χ as representations η, of G(f )0 /Q ⊂ M(η)/Q ⊂ M/Q; since G(f )0 /Q is a simply connected nilpotent Lie group, its representations satisfy relation (3.4.1). Observe that η˜ is an irreducible unitary representation of G(f )0 /Q, for which ˜ = M(η)/Q. (M/Q)(η) ˜ = {˜s ∈ M/Q : η˜ s˜  η} Moreover η˜ 0 is an extension of η˜ and η˜ M,χ = ind(M/Q,M(η)/Q,η0 ). By Corollary 3.4.9, η˜ M,χ , and hence ηM,χ are irreducible, and part (1) is proved. Turning to part (2), let χ and χ  characters of DM such that dχ = dχ  = i2 |d2 . Suppose that ηM,χ  ηM,χ  , and put η0 = ind(M(η),2,h2,χ  ). Since DM = D20 (M ∩ G()) and χ and χ  must agree on D20 , then it is enough to show that χ and χ  agree on M ∩ G(). Given s ∈ M ∩ G(), t ∈ G(f )0 , Lemma 5.2.4, part (4) shows that −1 s tst −1 ∈ Q. This means that

358

Irreducible unitary representations ˜ −1 ts) = η(t) ˜ = η(t), η0 (s)−1 η(t)η0 (s) = η(s −1 ts) = η(s

so η0 (s) commutes with all η(t), t ∈ G(f )0 . Since η is irreducible, the Schur Lemma says that η0 (s) is a scalar operator. Its value is easily computed: for a continuous nonzero function ϕ ∈ Hη , we have ϕ(ty) = ϕ(t), t ∈ G(f )0 , y ∈ Q, so (η0 (s)ϕ)(t) = ϕ(s −1 t) = ϕ(ts −1 ) = χ (s)ϕ(t). Thus η0 (s) = χ (s) 1Hη , and similarly η0 (s) = χ  (s) 1Hη . Now by Corollary 3.4.9, if η˜ M,χ  η˜ M,χ  , then η0  η0 , and hence χ (s) = χ  (s) holds for each s ∈ M ∩ G(). Suppose now there is some 2 , h2 , χ2 as in (3). Suppose that 2 is not in ˜ then for any the G(f )-orbit of 2 . Then η˜ is not in the G(f )/Q-orbit of η, s ∈ G(f ), the extensions η0s and η0 are not equivalent. Thus, by the remark  before Corollary 3.4.9, ηM,χ and ηM,χ  are inequivalent.

5.2.2 Irreducible unitary representations of intermediate groups We continue to fix  ∈ g∗ with f = |n . Recall that our goal is to construct, when  is integral, an irreducible unitary representation of G that is canonically associated to the coadjoint orbit of . By Theorem 5.1.3, there is a unique equivalence class K(Ad∗ (N )f ) of irreducible unitary representations of N associated with the orbit Ad∗ (N )f . Given ρ ∈ K(Ad∗ (N )f ), an important first step in this construction will be the extension of ρ to a subgroup of N G(f ) containing N . Then using this extension, we will be able to construct irreducible unitary representations of subgroups N M where G(f )0 ⊆ M ⊆ G(f ). To do this, we need the following result: Lemma 5.2.9 Let N be a simply connected nilpotent Lie group, γ an automorphism of N and D a γ -stable connected subgroup of N . Then for each function g ∈ Cc (N/D), 3 3 g(γ (x))|det γn/d | dμN,D (x) = g(x) dμN,D (x). N/D

N/D

Proof: Call the subgroup generated by γ in Aut(N ), it is abelian, unimodular and if G = N  , H = D  , by Example 3.3.6, we saw that the form μG,H on Cc (G,H,δH,G ) is defined by μN,D . Moreover δH,G (x,γ k ) = k |. Hence the function ρ(x,γ k ) = | det γ k | is a rho-function for H |det γn/d n/d and G:

5.2 Construction of irreducible representations

359

  k+r | = ρ(x,γ k )δH,G (y,γ r ). ρ (x,γ k )(y,γ r ) = ρ(xγ k (y),γ k+r ) = |det γn/d But we have N/D = G/H , thus for each g ∈ Cc (N/D) = Cc (G/H ), 3     g(γ (x)) dμN,D (xD) = μG,H g (e,γ )(x,1) ρ(x,1) N/D    = μG,H g(x,1)ρ (e,γ )−1 (x,1)   = μG,H g(x,1)ρ(x,1) |det γn/d |−1 3 = g dμN,D |det γn/d |−1 . N/D

Let ρ be any irreducible unitary representation of N corresponding to Ad∗ (N )f . Let h be a positive, G()-stable polarization at , admissible for n and such that nc ∩ h is G(f )-stable. Recall we know that such an h exists (Lemma 4.5.9, Lemma 4.5.10, Proposition 4.6.7). As before put h1 = h ∩ nc , and let ρ1 be the irreducible representation of N holomorphically induced from 1 and h1 . By Theorem 5.1.4, ρ1 is equivalent with ρ; write ρ1 = SρS −1 where S : Hρ → Hρ1 is unitary. Given s ∈ G, the representation ρ s = ρ(s −1 · s) is also an irreducible representation of N , and (s,ρ) → ρ s defines an action of G on the unitary dual Nˆ of N . We will see that when s ∈ G(f ), then ρ s and ρ are equivalent. Fix s ∈ G(f ). For ϕ ∈ Cc (N,D1,χf ), put C1 (s)ϕ(x) = ϕ(s −1 xs)| det Adn/d1 (s)|−1/2 . Then for any y ∈ D1 , χf (s −1 ys) = χs f (y) = χf (y) so (C1 (s)ϕ)(xy) = ϕ(s −1 xs(s −1 ys))| det Adn/d1 (s)|−1/2 = χf (y)−1 |ϕ(s −1 xs)| det Adn/d1 (s)|−1/2 showing that C1 (s)ϕ belongs to Cc (N,D1,χf ). Since, by Lemma 5.2.9, 3 |ϕ(s −1 xs)|2 | det Adn/d1 (s)|−1 dμN,D1 (xD1 ) N/D1 3 = |ϕ(x)|2 dμN,D1 (xD1 ), N/D1

then C1 (s) extends to an isometry of L2 (N,D1,χf ) into itself. If ϕ is in C(N, f ,h1,χf ), then for each Y ∈ h1 , C1 (s)(R(Y )ϕ) = R(Ad(s −1 )Y )(C1 (s)ϕ) = −i f (Ad(s −1 )Y )(C1 (s)ϕ) = −i f (Y )(C1 (s)ϕ),

360

Irreducible unitary representations

proving that C1 (s) maps H(N, f ,h1,χf ) into itself. Since C1 (s −1 ) = C1 (s)−1 , then C1 (s) is a unitary operator on H(N, f ,h1,χf ) and it is easy to check that C1 (s)ρ1s (n)C1 (s)−1 = ρ1 (n), n ∈ N . Thus for each s ∈ G(f ), C1 (s) is an intertwining operator for the representations ρ1 and ρ1s . Note that C(s) = S −1 C1 (s)S now similarly intertwines ρ and ρ s . Moreover, it is clear that s → C(s) (or C1 (s)) is a unitary representation of G(f ). Consider now the semidirect product N  G(f ), with the product (n,s)(n,s  ) = (nsn s −1,ss  ). For (n,s) ∈ N  G(f ) define ρ ext (n,s) = ρ(n)C(s). Then ρ ext ((n,s)(n,s  )) = ρ ext (nsn s −1,ss  ) = ρ(n(sn s −1 ))C(ss  ) = ρ(n)ρ(sn s −1 )C(s)C(s  ) = ρ(n)C(s)ρ(n )C(s  ) = ρ ext (n,s)ρ ext (n,s  ) and ρ ext is a unitary representation of N  G(f ) in Hρ . Since ρ is irreducible, then so is ρ ext . Note that for ϕ ∈ Hρ we have ρ ext (n,s)ϕ(n0 ) = ϕ(s −1 n−1 n0 s)| det Adn/d1 (s)|−1/2 .

(5.2.4)

Now the reader may observe that the construction of ρ ext depends upon the realization of ρ as a representation induced from a positive polarization h at f in nc that is G(f )-stable. The fact that there is always at least one such polarization is not completely satisfactory: one would like to know that the construction of ρ ext in (5.2.4) is canonical in the following sense. Suppose that h1 and h2 are two positive G(f )-stable polarizations in nc for f , and let ρi = ind(N,hi , f ,χf ) be the corresponding induced representations. We have seen that ρ1 and ρ2 are equivalent: there is a unitary isomorphism T : Hρ1 → Hρ2 such that T ρ1 (n) = ρ2 (n)T holds for all n ∈ N . However, at this point we do not even know that ρ1ext and ρ2ext are equivalent. In fact the following shows that the same operator T effects this equivalence. Theorem 5.2.10 Let h1 and h2 be two positive polarizations in nc at f that are G(f )-stable, and let ρ1 and ρ2 be corresponding irreducible representations. Let γ be an automorphism of N that fixes both f and hi : f ◦ γ = f , and γ (hi ) = hi , i = 1, 2, and define Ci : H(N, f ,hi ,χf ) → H(N, f ,hi ,χf ) by Ci ϕ(n) = ϕ(γ −1 (n)) |det γn/di |−1/2 . Let T : Hρ1 → Hρ2 be any unitary intertwining operator for ρ1 and ρ2 . Then T C 1 = C2 T .

5.2 Construction of irreducible representations

361

The proof of this theorem is quite lengthy, and so its proof is the subject of Section 5.3.2. Applying Theorem 5.2.10, with γ the inner automorphism of s ∈ G(f ) restricted to N, we get that an intertwining operator T : Hρ1 → Hρ2 for ρ1 and ρ2 is also intertwining for their extensions to N  G(f ), that is, T ρ1ext (n,s)T −1 = ρ2ext (n,s) holds for all (n,s) ∈ N  G(f ). We continue to assume that M is a subgroup of G(f ) with G(f )0 ⊆ M ⊆ G(f ). Since the orbit N f is closed, the subgroup N G(f ) = {s ∈ G : s f ∈ N f } is closed in G. Now N M is open in N G(f ), so it is closed in N G(f ), and hence NM also in G. The Lie algebra of N M is n + n , and by Proposition 4.3.8, h ⊂ (n + n )c . The next task is to construct an irreducible representation of N M corresponding to ,h and to a character χ with dχ = i|g() . Consider the stabilizer (N M)(|n+n ) = {a ∈ N M : (a − )|n+n = 0}. Now nm ∈ NM belongs to N M(n+n ) if and only if, for each X ∈ n, Y ∈ n , (X + Y ) = nm(X + Y ) = nm(X) + nm(Y ) = n(X) + m−1 nm(m−1 Y ) = n(X) + (Ad(m−1 )Y ) = n(X) + m(Y ). Here we use the facts that Ad(m−1 )Y ∈ n , and m−1 nm ∈ N . Thus (N M)(|n+n ) = N (f )M(2 ) = N (f )(Z(2 ) ∩ M) = N (f )(G() ∩ M). Now recalling the analytic group D 0 corresponding to the Lie algebra d, we have d ⊂ n + n , D 0 = D1 D20 and D 0 ⊂ N M. So DN M = D 0 (N M)(|n+n ) = D 0 (G() ∩ M) = D1 DM . Now assume that there is a unitary character χ of G() ∩ M, and hence of DM , such that dχ = i|d2 . We will show below that there is a unitary character χN M of DN M whose restrictions to D1 and DM are the characters χf and χ , respectively. Then it is natural to consider the holomorphically induced representation ind(N M,,h,χN M ); that this representation is irreducible will follow from its relation to the representation ρ ext (recall that ρ ext is an extension of the N -representation associated to N f ) and the representation ηM,χ defined in (5.2.2). First, extend ηM,χ to N  M by putting ηM,χ (x,u) = ηM,χ (u), x ∈ N , u ∈ M. For simplicity, from now on, we denote by ρ the canonical extension ρ ext of ρ to N  M. Thus we have two irreducible unitary representations of N  M: ρ and ηM,χ . Recall that elements of the Hilbert space Hρ are given by functions

362

Irreducible unitary representations

on N, while elements of the Hilbert space HηM,χ are given by functions on M. Now consider the inner tensor product ρ ⊗ ηM,χ : the unitary representation of N  M acting on elementary tensors in Hρ ⊗ HηM,χ by (ρ ⊗ ηM,χ )(x,u)(ϕ ⊗ ψ) = ρ(x,u)ϕ ⊗ ηM,χ (x,u)ψ. Since ρ|N is an irreducible representation of N , the restriction to N of ρ ⊗ηM,χ is a multiple of ρ|N , by Lemma 3.2.28, this implies that C(ρ ⊗ ηM,χ ) ⊂ 1 ⊗ L(HηM,χ ). Since ηM,χ is irreducible, this commutant is reduced to the scalar operators and ρ ⊗ ηM,χ is irreducible. Using the equality N (f ) = N ∩ G(f ) = N ∩ M it is easy to check that the natural map p : N  M → N M defined by (n,z) → nz has kernel diagN(f ) = {(u,u−1 ) : u ∈ N (f )}. The following shows that the kernel of ρ ⊗ ηM,χ contains diagN (f ). This means that ρ ⊗ ηM,χ gives rise to an irreducible unitary representation of N M. Proposition 5.2.11 One has the following. (1) diagN(f ) ⊂ ker(ρ ⊗ ηM,χ ), and hence the representation ρ ⊗ ηM,χ defines a representation σM,χ of the closed subgroup N M of G. (2) There is a unique character χN M of DN M extending χf on D1 and χ on DM , and the representation σM,χ is equivalent to ind(N M,,h,χN M ). Proof: To see that the kernel of the unitary representation ρ ⊗ ηM,χ of N M contains diagN (f ), observe that formula (5.2.4) shows that ρ(u,u−1 ) = χf (u)1 and since 2 |n(f ) = f |n(f ) and s f = f , ηM,χ (u,u−1 )ψ(s) = ηM,χ (u−1 )ψ(s) = ψ(us) = ψ(s(s −1 us)) = χ2 (s −1 us)−1 ψ(s) = χf (s −1 us)−1 ψ(s) = χf (u)−1 ψ(s). Hence we have a unitary representation σM,χ of N M satisfying ρ ⊗ ηM,χ = σM,χ ◦ p. Coming to assertion (2), we first observe that D1 ∩ DM ⊂ N (f ) ⊂ D1 ∩ D20 ⊂ D1 ∩ DM . This implies that if y1 y2 = y1 y2 with y1 , y1 in D1 , and y2 , y2 in DM , then y1−1 y1 = y2 (y2 )−1 is in D1 ∩ DM = N (f ). Hence χf (y1−1 y1 ) = χ (y2 (y2 )−1 ). Thus the relation: χN M (y1 y2 ) = χf (y1 )χ (y2 )

(y1 ∈ D1, y2 ∈ DM )

5.2 Construction of irreducible representations

363

defines a complex valued map on the group DN M = D1 DM . This map is a character since: χN M (y1 y2 y1 y2 ) = χN M (y1 (y2 y1 y2−1 )y2 y2 ) = χf (y1 )χf (y2 y1 y2−1 )χ (y2 )χ (y2 ) = χf (y1 )χy −1 f (y1 )χ (y2 )χ (y2 ) = χN M (y1 y2 )χN M (y1 y2 ). 2

This proves existence and uniqueness of χN M extending χf and χ . Next we prove that the representation σM,χ is equivalent with the induced representation τ = ind(N M,|n+n ,h,χN M ) in several steps. For ϕ ∈ Hρ and ψ ∈ HηM,χ , put T (ϕ ⊗ ψ)(xu) = ϕ(x)ψ(u)| det Adn/d1 (u)|1/2, x ∈ N,u ∈ M. Consider only smooth functions ϕ and ψ, compactly supported modulo respectively D1 and DM . 0. The function f = T (ϕ ⊗ ψ) is a well-defined function on N M. Indeed, if xu = x  u , then (x  )−1 x = u u−1 ∈ N (f ), therefore u−1 (u u−1 )u is also in N(f ), and: f (xu) = ϕ(x)ψ(u)| det Adn/d1 (u)|1/2 = ϕ(x  u u−1 )ψ(u)| det Adn/d1 (u)|1/2 = χf (u u−1 )−1 ϕ(x  )ψ(u)| det Adn/d1 (u)|1/2 = χf (u−1 (u u−1 )u)−1 ϕ(x  )ψ(u)| det Adn/d1 (u)|1/2 = χf (u−1 u )−1 ϕ(x  )ψ(u)| det Adn/d1 (u)|1/2 = ϕ(x  )χ (u−1 u )−1 ψ(u)| det Adn/d1 (u)|1/2 = ϕ(x  )ψ(u )| det Adn/d1 (u )|1/2 = f (x  u ). 1. The image of T is in the space of smooth functions f on N M such that, for any y ∈ DN M , f ((xu)y) = χN M (y)−1 δDNM ,NM (y)1/2 f (xu), and for any W ∈ h,

 1 R(W )f (xu) = −i((W ) + trace adn+g(f )/d1 +d2 (W ) f (xu). 2 

Indeed, if f = T (ϕ ⊗ ψ), and y1 ∈ D1 , y2 ∈ DM , f (xuy1 y2 ) = f (xuy1 u−1 uy2 ) = ϕ(xuy1 u−1 )ψ(uy2 )| det Adn/d1 (uy2 )|1/2 1/2  = χu−1 f (y1 )−1 χ (y2 )−1 δDM ,M (y2 )| det Adn/d1 (y2 )| f (xu).

364

Irreducible unitary representations

Recall now the relation N M = N  M/diagN (f ), so at the level of Lie algebras: n + g(f ) = n × g(f )/k where k = {(Z, − Z) : Z ∈ n(f )}. This is also a quotient of G(f )-modules, we get: # # δDNM ,NM (y1 y2 ) = #det Adn+g(f )/d1 +d2 (y2 )# # # (5.2.5) = #det Adn×g(f )/d ×d (y2 )# 1

2

= | det Adn/d1 (y2 )|δDM ,M (y2 ). Therefore: f (xuy1 y2 ) = χN M (y1 y2 )−1 δDNM ,NM (y1 y2 )1/2 f (xu). Similarly, for any Y1 ∈ e1 , # # d ## d ## f (xu exp tY1 ) = # ϕ(x exp tAd(u)Y1 )ψ(u)| det Adn/d1 (u)|1/2 dt #t=0 dt t=0 therefore, for any W1 ∈ h1 , (R(W1 )f )(xu) = (−i f (Ad(u)W1 ))f (xu) = −i f (W1 )f (xu), and, if Y2 ∈ e2 , # # d ## d ## f (xu exp tY ) = ϕ(x)ψ(u exp tY2 )|det Adn/d1(u exp tY2 )|1/2 . 2 dt #t=0 dt #t=0 Thus, if W2 ∈ h2 , arguing as above:  1 (R(W2 )f )(xu) = −i2 (W2 ) + trace adg(f )/d2 (W2 ) 2  1 + trace adn/d1 (W2 ) f (xu) 2   1 = −i2 (W2 ) + trace adn+g(f )/d1 +d2 (W2 ) f (xu). 2 This finishes the proof of step 1. 2. T (ϕ ⊗ ψ) is in H(N M,n+n ,h,χN M ) and T is unitary. First, for each ϕ and ψ, there are compactly supported, continuous functions α1 , α2 on N and M such that 3 2 |ϕ(x)| = α1 (xy1 ) dμD1 (y1 ), D 3 1 α2 (uy2 )δDM ,M (y2 )−1 dμDM (y2 ). |ψ(u)|2 = DM

5.2 Construction of irreducible representations

365

We put α(x,u) ˆ = α1 (x)α2 (u)| det Adn (u)| and consider the function: 3 −1 α(xu) = α((x,u)(v,v ˆ )) dμN (f ) (v) N (f ) 3 = α1 (xuvu−1 )α2 (uv −1 )| det Adn (uv −1 )| dμN (f ) (v). N (f )

This function is well-defined on N M, since if xu = x  u , with x, x  in N and u, u in M, then u u−1 ∈ N (f ), so u−1 u = u−1 (u u−1 )u ∈ N (f ), and putting v  = vu−1 u , using the right invariance of the Haar measure of the nilpotent group N (f ), we get 3 α(xu) = α1 (x  u vu−1 )α2 (uv −1 )| det Adn (uv −1 )| dμN (f ) (v) N (f ) 3 = α1 (x  u v  (u )−1 )α2 (u (v  )−1 )| det Adn (u (v  )−1 )| dμN (f ) (v  ) N (f )  

= α(x u ). Then |T (ϕ ⊗ ψ)(xu)|2 = |ϕ(x)|2 |ψ(u)|2 | det Adn/d1 (u)| 3 α1 (xy1 ) dμD1 (y1 )| det Add1 (u)|−1 | det Adn (u)| = D1 3 α2 (uy2 )δDM ,M (y2 )−1 dμDM (y2 ) × DM 3 = α1 (xuy1 u−1 )α2 (uy2 )| D1 ×DM

× det Adn (u)|δDM ,M (y2 )−1 dμD1 (y1 )dμDM (y2 ) 3 α1 (xuy1 u−1 )α2 (uy2 )| det Adn (uy2 )| = D1 ×DM   × | det Adn/d1 (y2 )|−1 δDM ,M (y2 )−1   × | det Add1 (y2 )|−1 dμD1 (y1 )dμDM (y2 ) . Since, by Corollary 3.1.3, the Haar measure of D1 DM is | det Add1 (y2 )|−1 dμD1 (y1 )dμDM (y2 ), this gives: 3 α((x,u)(y ˆ |T (ϕ ⊗ ψ)(xu)|2 = 1,y2 )) D1 DM   × | det Adn/d1 (y2 )|−1 δDM ,M (y2 )−1 dμD1 DM (y1,y2 ).

366

Irreducible unitary representations

Now as for N M, we have DN M = D1 DM = D1  DM /diagN (f ), and δD1 DM ,N M (y2 ) = δDNM ,NM (y2 ). Now the Haar measure dμDNM (d1 d2 ) of DN M can be identified to the invariant form μD1 DM ,N(f ) . This means 3 D1 DM

3

=

β(y1,y2 )dμD1 DM (y1,y2 ) 3 β((y1,y2 )(v,v −1 ))dμN (f ) (v)dμDNM (y1 y2 )

DNM

N (f )

for any continuous, compactly supported function β on D1  DM . Therefore, with (5.2.5), 3 3 −1 −1 α((x,u)(y ˆ |T (ϕ ⊗ ψ)(xu)|2 = 1,y2 )(v,v ))δDNM ,NM (y2 ) DNM

N (f )

× dμN (f ) (v) dμDNM (y1 y2 ) 3 α(xuy1 y2 )δDNM ,NM (y2 )−1 dμDNM (y1 y2 ). = DNM

This means that the square of the norm of T (ϕ ⊗ ψ) can be computed as an integral of α over N M, again using Corollary 3.1.3: 3 2 α(xu) dμN M (xu) T (ϕ ⊗ ψ) = 3N M 3 −1 = α((x,u)(v,v ˆ )) dμN (f ) (v) dμN M (xu) N M N (f ) 3 = α(x,u)dμ ˆ NM (x,u) NM 3 α(x,u)| ˆ det Adn (u)|−1 dμN (x)dμM (u) = N×M 3 3 α1 (x) dμN (x) α2 (u) dμM (u) = ϕ 2 ψ 2 . = N

M

Therefore T is an isometry and its range contains all the projection of continuous functions α1 (x)α2 (u), with αi continuous with compact support. This space being dense in Cc (N  M), for the topology of uniform convergence on any compact set, T is unitary. 3. T intertwines σM,χ and τ . Indeed, 

 (σM,χ )(x0 u0 )ϕ ⊗ ψ (x,u) −1 −1/2 = ϕ(u−1 ⊗ ψ(u−1 0 x0 xu0 )| det Adn/d1 (u0 )| 0 u).

5.2 Construction of irreducible representations

367

Therefore    T (σM,χ )(x0 u0 )ϕ ⊗ ψ (xu) −1 −1/2 1/2 = ϕ(u−1 ψ(u−1 0 x0 xu0 )| det Adn/d1 (u0 )| 0 u)| det Adn/d1 (u)| −1 −1 −1 1/2 = ϕ(u−1 0 x0 xu0 )ψ(u0 u)| det Adn/d1 (u0 u)|

= T (ϕ ⊗ ψ)((x0 u0 )−1 xu)   = τ (x0 u0 )T (ϕ ⊗ ψ) (xu). This completes the proof of the proposition. Observe that, if M = G(f )0 , then ηM,χ = η; in this case the character is fixed: χ = χ2 is determined by . Accordingly, we simply denote by σ the representation σG(f )0,χ defined in the proposition. On the other hand, if M = G(f ), Lemma 3.2.28 has two important consequences. Proposition 5.2.12 Let  and  be two elements of g∗ such that |n =  |n , let h and h two positive polarizations in  and  , admissible to n, and let χ be a character of G(), χ  a character of G( ), extended to D and D  , such that dχ = i|g() , χ  = i |g( ) . Suppose that the associated representations  σG(f ),χ and σG( f ),χ  of N G(f ) are equivalent then; there is x ∈ G such that  x =  and χ x = χ  .   Proof: Let us first prove that σG(f ),χ  σG( f ),χ  implies ηG(f ),χ  ηG(f ),χ  . The restriction to N of these representations is ρ ⊗ 1. By Lemma 3.2.28, the intertwining operators between σG(f ),χ and σG(f ),χ  have the form 1 ⊗ S, with , and, using the definition of ρ, S : HηG(f ),χ → Hη  G(f ),χ

C(s) ⊗ S ◦ ηG(f ),χ (s) = (1 ⊗ S) ◦ (C(s) ⊗ ηG(f ),χ (s)) = (1 ⊗ S) ◦ σG(f ),χ (s)  = (ρ(s) ⊗ ηG( f ),χ  (s)) ◦ (1 ⊗ S)  = C(s) ⊗ (ηG( f ),χ  (s) ◦ S)  holds for each s ∈ G(f ). Thus ηG(f ),χ  ηG( f ),χ  .

By Proposition 5.2.8, there is s ∈ G(f ) such that s|n+g(f ) = n+g(f ) , thus s ∈  + (g(f ) + n)⊥ . Thus by Lemma 4.2.5 this implies s ∈ G , hence there is x ∈ G such that x =  . Using conjugation by x, we can suppose that  =  , and ηG(f ),χ  ηG(f ),χ  . By Proposition 5.2.8, part (b), χ = χ  .

368

Irreducible unitary representations

And similarly: Proposition 5.2.13 Let u be in G(f ), the following are equivalent: (i) σ u  σ , (ii) ηu  η, (iii) u ∈ G(f )0 G(). Proof: Recall σ ◦ p = ρ ⊗ η (Proposition 5.2.11). By Lemma 5.2.7, (ii) and (iii) are equivalent. We saw that, for each u ∈ G(f ), ρ u  ρ, thus, by definition of σ , (ii) implies (i). Finally if (i) holds, then ρ ⊗ η = σ ◦ p  σ u ◦ p = ρ u ⊗ ηu  ρ ⊗ ηu . Hence there is an unitary operator T : Hρ ⊗ Hη → Hρ ⊗ Hη , commuting with all the σ (n) = ρ(n) ⊗ 1, n ∈ N . Now Lemma 3.2.28 implies there is S : Hη → Hη such that T = 1 ⊗ S, thus for each s ∈ G(f )0 , C(s) ⊗ S ◦ η(s) = C(s) ⊗ (ηu (s) ◦ S). Thus ηu  η, and (ii) is equivalent to (i). Notation From now on, put M = G(f )0 G(). Observe that, by Lemma 5.2.7, we have now M(η) = G(f )0 G() = M, therefore, with the notation of Proposition 5.2.8, ηM,χ coincides with η0 , it is an extension of η. Proposition 5.2.13 says that the stability subgroup G(f )(σ ) of σ in G(f ) coincides with M. Since N stabilizes ρ and η, it easy to prove that similarly: N G(f )(σ ) = N M. Here is a summary of results above, expressed by treating equivalent representations as identical. By Proposition 5.2.8, the representations ηM,χ and ηG(f ),χ are irreducible, ηM,χ is an extension of η, and ηG(f ),χ is the induced representation ind(G(f ),M,ηM,χ ). Taking the tensor product of these representations with suitable extensions of ρ, we get the irreducible representations σ , σM,χ and σG(f ),χ , and by Proposition 5.2.11, σM,χ  ind(N M,,h,χ ) and σG(f ),χ  ind(NG(f ),,h,χ ). As shown in the following diagram:       N M N  G(f ) N  G(f )0 ext ind → → ρ ⊗ ηM,χ ρ⊗η ρ ⊗ ηG(f ),χ ↓p 

NG(f )0 σ

↓p 

ext

→



NM σM,χ

↓p 

ind

→



N G(f ) σG(f ),χ

(5.2.6) 

  G → π ind

5.3 Orbit method for solvable groups

369

where σM,χ is an extension of σ and, by Lemma 3.3.21, σG(f ),χ is induced from σM,χ : σG(f ),χ  ind(NG(f ),N M,σM,χ ). Since E = E 0 G() ⊆ N G(f ), then Lemma 5.1.2 gives σG(f ),χ  ind(NG(f ),,h,χ )  ind(NG(f ),E,ind(E,,h,χ )). At the right of the diagram is the representation π = ind(G,NG(f ),σG(f ),χ ), and by Proposition 3.3.29 and Lemma 5.1.2, we get: π  ind(G,E,ind(E,,h,χ ))  ind(G,,h,χ ).

5.3 Orbit method for solvable groups In this section, we show that the representation π just defined is irreducible and we study relations to the character χ and the polarization h.

5.3.1 Construction of an irreducible unitary representation of G Let us summarize the ingredients used so far to describe the representation π . Recall that n is a nilpotent ideal in g containing [g,g], N is the connected subgroup of G with Lie algebra n and  is a point in g∗ , and f is the restriction of  to the n, 2 = |g(f ) . 1. ρ is the irreducible representation of N associated to the orbit N f in n∗ . 2. h is a positive G()-stable polarization at , admissible for n, such that h ∩ nc is G(f )-stable, and h2 = h ∩ g(f )c . 3.  ∈ g∗ is integral: there is a character χ of G() whose differential is i|g() . 4. Fix the character χ of G(), denote still by χ its unique extension to D such that dχ = i|d . Recalling the groups M = G(f )0 G(), and G(f ), containing G(), with Lie algebra n , then h is a subalgebra of nc + nc , and the corresponding D-groups coincide: D = D 0 G() = DN M = DN G(f ) . 5. The corresponding holomorphically induced representations are : σM,χ = ind(N M,,h,χ ),

σG(f ),χ = ind(NG(f ),,h,χ ).

They are irreducible, and σG(f ),χ = ind(NG(f ),N M,σM,χ ). Each of the groups G(f )0 , M, and G(f ) has Lie algebra g(f ) = n . Let us now present some examples.

370

Irreducible unitary representations

Example 5.3.1 Recall again the universal covering G = E˜ 2 of the motion group of the plane (see Example 1.6.6), where g is the span of {A,X,Y } and [A,X] = Y , [A,Y ] = −X. Then n = RX + RY , we choose  = rX∗ , with r > 0. Then g() = RX,

G() = exp RX exp 2π ZA,

and g(f ) = n,

G(f )0 = N G(f )0 = N,

N M = N exp 2π ZA = N G(f ) = G(f ).

We choose the real polarization h = nc , then D 0 = N , and D = G(f ). Since G()/G()0  Z, then  is integral. The character χ of D 0 with differential i|d is defined by χ (exp(xX + yY )) = eirx . The extensions of this character to D = G(f ) are characterized by a number 0 ≤ ν < 1: χ ν (exp(xX + yY ) exp(2π kA)) = ei(rx+2π νk) . Now N is abelian and N M = N G(f ) so the representation σM,χ ν = σG(f ),χ ν is a character: for each ν, σG(f )0 = χ,

σG(f ),χ = χ ν .

Finally π is the representation πr,ν = ind(G,G(f ),χ ν ). We shall prove that this representation πr,ν is irreducible. It is realized on the space Hπr,ν of measurable functions f such that f (s + 2π ) = e−2iπ ν f (s) almost everywhere and: 3 2π 2 |f (s)|2 ds < ∞, f = 0

by, as in Example 3.1.13, the element (z,t) of E˜ 2 is represented by: (πr,ν (z,t)f )(s) = eiRe(re

−is z)

f (s − t).



Example 5.3.2 Let us now come back to Example 4.7.2: g = span(T ,Z,Z, W,W,A,B) with T , A, B real and [A,Z] = iZ, [B,W ] = iW , [A,B] = T . Put Z = X + iY , W = U + iV , choose  = τ T ∗ + rX∗ + RU ∗ , with r, R > 0 and τ = 0. Then g() = spanR (T ,X,U ),

G()0 = {exp(tT + xX + uU ) : t,x,u ∈ R},

and G() = G()0 exp 2π ZA exp 2π ZB ,

5.3 Orbit method for solvable groups

371

where A = A − Rτ V , B  = B + τr Y . Here, there is no choice for n, it is n = spanR (T ,X,Y,U,V ) = [g,g]. Again N is abelian, G(f )0 = N , the polarization h at  is h = nc . 1 Z. In Example 4.7.2, it is shown that  is integral if and only if τ ∈ 2π Suppose this is the case, then we can extend χ to G(). The possible extensions are as follows: pick a pair (μ,ν) such that 0 ≤ μ,ν < 1 and define χ μ,ν by: χ μ,ν (exp(tT +xX+uU )exp 2pπA exp 2qπZB  ) = ei(τ t+rx+Ru) ei2π(μp+νq) . The unique extension to χ μ,ν to M = G(f ) = N exp 2π ZA exp 2π ZB  is still denoted χ μ,ν : χ μ,ν (exp(tT + xX + yY + uU + vV ) exp 2pπA exp 2qπZB  ) = ei(τ t+rx+Ru) ei2π(μp+νq) . As in the preceding example, D = G(f ) and σG(f ),χ μ,ν = χ μ,ν . Hence, here: π = ind(G,,h,χ μ,ν ) = ind(G,G(f ),χ μ,ν ). As in the preceding example, we shall prove that π is irreducible.



Example 5.3.3 The following example is just to show that the situation G(f )0  M  G(f ) is possible. Recall Example 4.7.2 of the Lie algebra span(T ,Z = X + iY,Z,W = U + iV ,W,A,B) be the solvable Lie algebra such that T , A, B are real and [A,Z] = iZ, [B,W ] = iW , [A,B] = T . Add two real elements, and put: s = gc = spanC (S,T ,Z,Z,W,W,A,B,C),

[C,B] = S.

We choose n = [g,g] = spanR (S,T ,X,Y,U,V ), and  = σ S ∗ + τ T ∗ + rX∗ + RU ∗ , with σ , τ = 0, and r > 0 and R > 0. It is easy to verify that the Lie algebra g(f ) is abelian, it is g(f ) = n + RC. Therefore G(f )0 = N exp RC. On the other hand the Lie algebra g() is  σ  g() = spanR S,T ,X,U,C − V . R The groups G(f ) and G() are not connected. Explicitly, they are:   τ  τ  G(f ) = G(f )0 exp 2π Z A − V exp 2π Z B + Y R r and  τ  G() = G()0 exp 2π Z A − V . R

372

Irreducible unitary representations

Thus,

  τ  τ  G(f )0  M = G(f )0 exp 2π Z A− V  G(f ) = M exp 2π Z B + Y . R r Remark that  is integral, the character χ of G()0 can be extended to a 1 / 2π Z, it cannot be extended to G(f ) character of G(f )0 , or M, but, if τ ∈ (see Example 4.7.2).  Now we turn to the irreducibility of the representation π = ind(G,,h,χ ). Consider the irreducible representations σM,χ , and σG(f ),χ ; since σG(f ),χ  ind(NG(f ),N M,σM,χ ), then it is realized on the space H(NG(f ),N M,σM,χ ) Since NG(f )/NM is discrete, its Haar measure is simply the counting mea8 sure, and this space is unitary equivalent to s˙∈G(f )/M  HσM,χ , by choosing representatives s ∈ s˙ , and through the map f → f (s) . Now the of σG(f ),χ to N is a multiple of ρ. Indeed, if n ∈ N , for  restriction  any f = f (s) , (σG(f ),χ (n)f )(s) = f (ss −1 n−1 s) = σM,χ (s −1 ns)(f (s)) = ρ(s −1 ns)(f (s)) = C(s −1 )ρ(n)C(s)(f (s)). This means that: σG(f ),χ |N 



C(s)−1 ρC(s).

s˙ ∈G(f )/M

Thus the restriction of σG(f ),χ to N is equivalent to a direct sum of irreducible representations, each of them being equivalent to ρ, that is to say σG(f ),χ |N is a multiple of ρ. Moreover the stability group G(ρ) of ρ is exactly the set of s ∈ G such that s f belongs to the N-orbit of f , that is G(ρ) = N G(f ). Recall that the separation property (3.4.1) holds for the nilpotent connected Lie group N, therefore by Corollary 3.4.9, π = ind(G,NG(f ),σG(f ),χ )  ind(G,,h,χ ) is irreducible. Moreover, if χ = χ  , then σG(f ),χ and σG(f ),χ  are inequivalent. Again, by Corollary 3.4.9, ind(G,,h,χ ) and ind(G,,h,χ  ) are inequivalent. Summarizing, we proved: Theorem 5.3.4 (Auslander and Kostant, [9]) Let G be a simply connected, solvable Lie group, g its Lie algebra, n a nilpotent ideal of g containing [g,g]. Suppose we are given the following data: (a) an integral element  in the dual g∗ of g, with f the restriction of  to n, (b) a positive G()-stable polarization h at , admissible for n, and such that h ∩ nc is stable under the adjoint action of G(f ).

5.3 Orbit method for solvable groups

373

Then for each character χ of G() such that dχ = i|g() , the representation π = ind(G,,h,χ ) is irreducible, and the map χ → [π] associating to χ the equivalence class of the representation π is one-to-one. It remains to prove the following points: the preceding construction is independent of the choice of the polarization h, and the class of the representation π depends only on the coadjoint orbit of , and the choice of the character χ . This result will be Proposition 5.3.11. The first step is now to prove Theorem 5.2.10, that is to prove that the extension of ρ defined just before it is independent of the polarization h ∩ nc of n at f .

5.3.2 The extension is canonical In this section, we return to the proof of Theorem 5.2.10. First we consider a special case, of independent interest. Proposition 5.3.5 Let N be a nilpotent, simply connected Lie group with Lie algebra n, with dim n > 1, let f ∈ n∗ and γ an automorphism of N (and n) fixing f : f ◦ γ −1 = f . Suppose: (a) the center z of n is one dimensional and f |z does not vanish, and (b) there is no noncentral, abelian and γ -stable ideal a in n. Then n is a Heisenberg algebra, and there is only one γ -invariant positive polarization at f . Proof: Since dim z = 1, then z is included in any ideal of n, since dim n > 1, z ⊂ [n,n]  n. Consider the center a of [n,n]. It is an abelian ideal since [n,[n,a]] = [[n,n],a] = 0 implies that [n,a] ⊂ z ⊂ a. Moreover it is γ invariant. Thus, by assumption (b), a is central: a = z. Observe that z is γ -invariant. Suppose now [n,n] = z. By the Engel theorem for the n module [n,n]/z, there is Y in [n,n] \ z such that [n,Y ] ⊂ z. But the Jacobi relation implies that such a Y is in a, which is impossible, thus [n,n] = z. Put now V = ker f , so that n = V ⊕ z. The automorphism γ leaves z invariant and induces a mapping γ on V  n/z. Denote H (Y,Z) = 2iBf (Y,Z) (Y , Z in Vc ). Let r be the radical of H : r = {Y ∈ Vc : H (Vc,Y ) = 0}. Observe that r = {0} indeed, if Y ∈ r, then for each Z ∈ Vc , Z ∈ Vc , 2i f [Y,Z] = H (Y,Z) = 0, thus [Y,Z] = 0, and Y ∈ V ∩ z = {0}.

374

Irreducible unitary representations

Let α be an eigenvalue of γ , and V α the corresponding eigenspace in Vc . Put: (V α )⊥H = {Z ∈ Vc : H (V α ,Z) = 0}. If 0 = Y ∈ V α ∩ (V α )⊥H , then H (Y,Y ) = 0, thus f [Y,Y ] = 0 which implies [Y,Y ] = 0 and z ⊕ spanR (Re(Y ),Im(Y )) is a γ -stable noncentral abelian ideal in n. This is impossible thus V α ∩ (V α )⊥H = 0. Since H is nondegenerate, this implies Vc = V α ⊕ (V α )⊥H . Observe that (V α )⊥H is γ -stable, it follows  α that γ is diagonalizable in Vc , Vc = α∈R V , where R is the set of eigenvalues of γ . Moreover γ has no real eigenvalue, since if α was real and Y ∈ V α \{0}, then z+RY is an abelian noncentral, γ -invariant ideal in n. This is impossible. Now, we saw that the real continuous function Y → H (Y,Y ) does not vanish on V α \ {0}. Thus, for each α, this function is either always positive or always negative on V α \{0}. Put R + = {α ∈ R : H (Y,Y ) > 0 (Y ∈ V α \{0})}, R − = {α ∈ R : H (Y,Y ) < 0 (Y ∈ V α \ {0})}. Since γ is real, R − = R + . Now if α and β are in R, and Y ∈ V α , Z ∈ V β , then H (Y,Z) = 2i f [Y,Z] = 2i f [γ (Y ),γ (Z)] = βα2i f [Y,Z] = βαH (Y,Z). Therefore |α| = |β| = 1, and if α = β, βα = 1, and then H (Y,Z) = 0 and [Y,Z] = 0, [V α ,V β ] = 0.  This proves that z⊕ α∈R + V α is a γ -stable ideal in nc which is an isotropic subspace for Bf . Since 2i f [Y,Y ] > 0 for each Y ∈ V α \ {0}, it is a maximal isotropic subspace, that is a positive, γ -stable polarization at f . Now let h be any γ -stable positive polarization at f , h = z ⊕ W for some γ -stable subspace W of Vc . It follows that there is a subset RW ⊂ R such that  W = α∈RW (V α ∩ W ). Since h is positive, W ∩ V α = 0 for each α ∈ R − , and W ⊂ ⊕α∈R + V α . But the dimension of these two subspaces of Vc are 1/2 dim Vc , thus h=



V α ⊕ z.

α∈R +

This proves the uniqueness of the γ -invariant, positive polarization. Finally n is a Heisenberg algebra: for any orthonormal basis (Wj = Xj + iYj ) in W for the form H , (Xj ,Yj ,Z) is a basis of n, such that Z ∈ z, f (Z) = 1 and: [Xj ,Yk ] = f [Xj ,Yk ]Z =

1 1 H (Wj ,Wk )Z = δj,k Z. 4 4

5.3 Orbit method for solvable groups

375

Turning to the proof of Theorem 5.2.10, let N be a simply connected nilpotent Lie group with Lie algebra n, f ∈ n∗ , and let γ be an automorphism of N that fixes f . Choose a realization ρf of the irreducible representation of N associated to the N -orbit of f . We want to extend ρf by defining an operator C = C(γ ) such that Cρf (γ −1 (n))C −1 = ρf (n), for each n ∈ N . A way to do that is to choose a γ -stable positive polarization h1 in nc at f , then to build ρ1 = ind(N, f ,h1,χf ), which is equivalent to ρf : there is S1 : Hρ1 → Hρf such that ρf (n)S1 = S1 ρ1 (n), for each n ∈ N, to put C1 ϕ(n) = ϕ(γ −1 (n))| det γn/d1 |−1/2, and, coming back to ρf , the operator C = S1 C1 S1−1 satisfies the desired relation. Theorem 5.2.10 says that this extension is canonical. More precisely, if ρ2 , C2 are obtained in the same way, using another γ -stable positive polarization h2 at f , then S2 C2 S2−1 = S1 C1 S1−1 . This is equivalent to saying that for some (and hence for any) intertwining operator T : Hρ1 → Hρ2 between ρ1 and ρ2 , T C 1 = C2 T . In this case we say that C1 and C2 are equivalent. Proof of Theorem 5.2.10: Let h1 and h2 be γ -stable positive polarizations at f , with ρi = ind(N, f ,hi ,χf ), and let Ci be the extension operators defined on H(N, f ,hi ,χf ) by Ci ϕ(n) = ϕ(γ −1 (n))| det γn/di |−1/2 . To show that C1 and C2 are equivalent, we proceed by induction on dim n. If [n,n] = 0, the Lie algebra is abelian and there is only one polarization: hi = nc . Similarly, if f |[n,n] = 0, nc is the unique polarization at f . In both cases, C1 = C2 , and we are done. Suppose then that f |[n,n] = 0, and that the theorem holds for any nilpotent Lie algebra n such that dim n < dim n. Step 1: Reduction to the case of one-dimensional center. Let z be the center of n, and z0 = ker f ∩ z. By construction z0 is a γ -stable ideal, included in any polarization h at f . Suppose z0 = 0. Put n˜ = n/z0 , γ and f give γ˜ and f˜ on n˜ . Then h˜ i = hi /z0 is a γ˜ -stable positive polarization of n˜ at f˜ . If Z0 is the connected subgroup of N with Lie algebra z0 , and p : N → N/Z0 = N˜ , denote by ρi and ρ˜i the corresponding irreducible representations of N and of N˜ . The spaces Hρi , Hρ˜i are canonically isomorphic, and ρi = ρ˜i ◦ p.

376

Irreducible unitary representations

Moreover the extensions Ci and C˜ i of ρi and ρ˜i , coincide. Now if T is an intertwining operator for ρ1 and ρ2 , then T intertwines ρ˜1 and ρ˜2 . By the induction hypothesis C˜ 2 ◦ T = T ◦ C˜ 1 . The same is true for C1 and C2 . Similarly, if there is a γ -stable ideal b in n, included in ker f ∩ h1 ∩ h2 , with B = exp b, as above the representations ρi are ρ˜i ◦ p, with p : N → N/B, thus the extensions of ρ1 and ρ2 satisfy C2 T = T C1 . Therefore we now suppose that dim z = 1, and there is no γ -stable ideal b, included in ker f ∩h1 ∩h2 . Hence z ⊂ [n,n], and f |z = 0. By Proposition 5.3.5, we may also assume that there is a γ -stable abelian, noncentral ideal a in n. Step 2: Reduction to equivalence between h and h , with a ⊂ h, a ⊂ h . Let a be an abelian noncentral γ -stable ideal. In this step, we show that it will be enough to prove the following: if h is a positive, γ -stable polarization h at f , then there is a positive γ -stable polarization h that contains a, and such that the extensions C(γ ), C  (γ ) for ρ = ind(N, f ,h,χf ), ρ  = ind(N, f ,h,χf ) are equivalent. Suppose that the preceding statement holds. If h1 does not contain a, then replacing h1 by h1 , we may assume that a ⊂ h1 ; similarly, we may assume that a ⊂ h2 . Now suppose that this is the case: a ⊂ h1 and a ⊂ h2 . Observe that af is a γ -stable subalgebra of n and that dim af < dim n. Let K = exp af be the connected subgroup of N corresponding to af . By the induction hypothesis, the theorem holds for the extensions Bi = Bi (γ ) of the representations τi = ind(K, f |af ,hi ,χf ) (i = 1,2): for any intertwining operator S : Hτ1 → Hτ2 , SB1 = B2 S. By induction in stages, ρi = ind(N, f ,hi ,χf ) is unitarily equivalent to ind(N,K,τi ) (i = 1,2), via the unitary operator Ri : Hρi → H(N,K,τi ) defined by   (x ∈ N, k ∈ K). (Ri F )(x) (k) = F (xk) Now recall the equivalence S˜ : H(N,K,τ1 ) → H(N,K,τ2 ) given by ˜ )(x) = S(f (x)) (Sf

(x ∈ N ).

˜ 1 is an intertwining operator for ρ1 and ρ2 . Recall also that, Thus T = R2−1 SR if we put δK,Di (γ ) = | det γk/di |, then Bi , Ci = Ci (γ ) are: (Bi ϕ)(k) = ϕ(γ −1 (k))δK,Di (γ )−1/2,

(Ci F )(x) = F (γ −1 (x))δN,Di (γ )−1/2 .

Then we have   Ri Ci F (x) (k) = F (γ −1 (x)γ −1 (k))δN,K (γ )−1/2 δK,Di (γ )−1/2   = Bi Ri F (γ −1 (x)) (k)δN,K (γ )−1/2 .

5.3 Orbit method for solvable groups

Hence

377

  ˜ 1 C1 F (x) = SB1 R1 F (γ −1 (x)) δN,K (γ )−1/2 SR   ˜ 1 F (γ −1 (x)) δN,K (γ )−1/2 = B2 SR   = B2 R2 T F (γ −1 (x)) δN,K (γ )−1/2 = R2 C2 T F (x).

Thus T C1 = C2 T . The following example illustrates the statement assumed in Step 2 above: for a given polarization h, a new polarization h containing a is constructed where the extension is seen to be equivalent. Example 5.3.6 Let n be the Heisenberg algebra of dimension five. Fix a basis (Z,Y1,Y2,X1,X2 ) with usual nonvanishing brackets [Xi ,Yi ] = Z. Put f = Z ∗ , and define γ on the Lie algebra n by γ (Z) = Z, γ (Y1 ) = −Y1, γ (Y2 ) = αY2, γ (X1 ) = −X1, γ (X2 ) = α −1 X2, where α is a nonzero real number. It is easily seen that γ is an automorphism. A γ -stable, noncentral abelian ideal a is a = z + RU , where U is an eigenvector of γ . Choose a = spanR (Z,Y1 ). A positive, γ -stable polarization that does not contain a is h = spanC (Z,X1 + iY1,Y2 ). Put ρ(f ,h) = ind(N,h, f ,χf ). A positive γ -stable polarization h containing a, for which the extension of ρ(f ,h ) = ind(N, f ,h,χf ) is equivalent with that of ρ(f ,h), is given by h = spanC (Z,Y1,Y2 ). Observe that ρ(f ,h) is equivalent to ρ = ind(N,E,τ ), where E is the group with Lie algebra e = spanR (Z,X1,Y1,Y2 ), and τ = ind(E, f ,h,χf ) (Lemma 5.1.2). It is realized on the space H(N,E,τ ) of functions ψ : N → A, where A is the space of holomorphic functions on C defined just before Lemma 5.1.8. Identify H(N,E,τ ) with the space Hρ of measurable functions F (w,x2 ), w ∈ C, x2 ∈ R, holomorphic in the variable w and such that: 3 3 2 2 F = |F (w,x2 )|2 e−|w| /2 dw dx2 < ∞. R C

With this identification, the extension C of ρ to γ is (CF )(w,x2 ) = F (−w,α −1 x2 )|α|−1/2 .

378

Irreducible unitary representations

Similarly, the representation ρ  = ind(N, f ,h,χf ) is realized in L2 (R2 ), and the extension C  of ρ  to γ is (C  ϕ)(x1,x2 ) = ϕ(−x1,α −1 x2 )|α|−1/2 . By Proposition 5.1.15, an intertwining operator between ρ  and ρ is given by 3 2 2 ex1 w e−x1 /2 ϕ(x1,x2 ) dx1, (T ϕ)(w,x2 ) = e−w /4 R

and it is straightforward to check that T C  = CT holds.



Step 3: Construction of h . Let h be a positive, γ -stable polarization at f and a a minimal, γ -stable, noncentral abelian ideal in n; assume that a is not included in h. Let us now proceed to build the new polarization h containing a. Since z ⊂ a, then by minimality of a, γ induces a linear map on a/z with either one real eigenvalue, or two nonreal, conjugate eigenvalues. It follows that the dimension of a is 2 or 3. Now, because f is nontrivial on [n,n], af has codimension 1 or 2. By construction, the ideal [n,a] is nonzero, and γ -stable, strictly included in a, thus by minimality of a, [n,a] = z. A similar proof gives h ∩ ac = zc . Recall the subalgebras associated to h: e = (h + h) ∩ n, d = h ∩ n. Recall that, since n is nilpotent, [e,e] ⊂ d (see Lemma 4.6.8). Put now: h0 = h ∩ afc , e0 = (h0 + h0 ) ∩ n, d0 = h0 ∩ n, and h = h0 + ac, e = (h + h ) ∩ n, d = h ∩ n. Each of these subalgebras is γ -stable. To see that h is a polarization, observe first that h0 ∩ac = h∩ac = zc , so Bf defines a bilinear form on (h/h0 )⊕(ac /zc ) which is nondegenerate. Hence dim(h/h0 ) = dim(ac /zc ). Thus: dim h = dim h0 + dim ac − dim(ac ∩ h0 ) = dim h0 + dim(ac /zc ) = dim h0 + dim(h/h0 ) = dim h. Since n(f ) ⊂ h = hf , then by properties of orthogonality,   (h )f = hf0 ∩ afc = (h ∩ afc )f ∩ afc ⊇ hf + (afc )f ∩ afc = (h + ac ) ∩ afc = h ∩ afc + ac = h, hence h is a polarization, and since h is positive, and h = h0 + ac , it is easy to check that h is positive. Define k = e + e . Observe that in fact k = e + a. We claim that d0 ∩ ker f is a γ -stable ideal in k.

5.3 Orbit method for solvable groups

379

Let Y ∈ d0 ∩ ker f and let X + A ∈ k with X ∈ e and A ∈ a. Since ef = d (Proposition 4.2.9), we have f [X,Y ] = 0 and since d0 is a subset of af , f [A,Y ] = 0. Thus [X + A,Y ] ∈ ker f . To see that [X + A,Y ] ∈ d0 , recall that [n,n] ⊂ af , thus [X,Y ] ∈ af ∩ [e,d] ⊂ af ∩ d = d0 . On the other hand [A,Y ] ∈ z ⊂ d0 . Thus [X + A,Y ] ∈ d0 and so d0 ∩ ker f is an ideal in k. Since d0 ∩ ker f is clearly γ -stable, the claim is proved. Since both h and h are included in k, then by the argument of Step 2, we may assume k = n. Now by the reduction argument of Step 1 we may assume that d0 ∩ ker f = 0, and hence d0 = z. As a consequence of this reduction, we have h0 = w ⊕ zc , with iBf (W,W ) > 0 for each nonvanishing W ∈ w, and h = w + ac with dc = h ∩ h = ac + w ∩ w = ac . Recall that [n,a] ⊂ z, thus [n,n] ⊂ af . Observe also that, n being nilpotent, [e,e] ⊆ d, so [e,e] ⊆ d ∩ af = d0 , therefore [n,n] = [e + a,e + a] ⊂ z. Now [e0,e0 ] ⊆ d ∩ af = d0 = z; recalling that dim z = 1, this means that either e0 is z (when w = {0}), or e0 is a Heisenberg algebra. Step 4: Construction of a small dimensional Heisenberg subalgebra m. Observe that the definition of h0 shows that h0 is a γ -stable subalgebra with f [h0,h0 ] = 0. Since h0 = w + zc and (e0 )c = h0 + h0 , then h0 is a polarization in (e0 )c at f |e0 , clearly positive. We now build a convenient supplementary space for h0 in h. Pick a basis (Xi ) for a supplementary space of h0 in h, such that Xi ∈ ker f . Since Bf gives a duality between h0 /zc and h0 /zc , for each i there is Yi ∈ h0 ∩ ker f such that f [Xi ,W ] = f [Yi ,W ] for any W ∈ h0 . Thus f Ti = Xi − Yi belongs to h0 ∩ ker f . Put t = spanC (Ti ): it is a supplementary space for h0 in h, it is included in ker f and orthogonal to h0 for Bf . Next we claim that t + ac = t + ac , that is, that t + ac is a real subalgebra. Since h/h0 is isomorphic to the dual of ac /zc via the bilinear form Bf , and ac /zc is a real vector space, the same holds for h/h0 . Thus for each T in t, there is T  ∈ t defining the same linear form on ac , which means that T − T  ∈ afc . f

Now T ∈ t ⊂ h0 , and T  ∈ h ⊂ hf0 , thus

T − T  ∈ hf0 ∩ afc = (h0 + ac )f = (h )f = h . f



Exchanging T and T  , we get similarly T − T  ∈ h0 ∩ afc = h . Thus  T − T  ∈ h ∩ h = dc = ac . Hence t + ac contains T , for each T ∈ t. This proves our claim that t + ac is real, meaning that we have m ⊂ n with mc = t + ac . Note that t + t + ac = t + ac and the sum t + ac is direct since t ∩ ac ⊂ (h ∩ ac ) ∩ t = zc ∩ t ⊂ h0 ∩ t = {0}.

380

Irreducible unitary representations

Note that since [n,n] ⊂ a, then m is an ideal of n. The duality between t  h/h0 and ac /zc proves that m is a Heisenberg algebra with center z, and that dim m = 2 dim a − 1, which is either 3 or 5. The subalgebra hm = t + zc is a polarization in m at f |m ; put dm = hm ∩ m. Put also hm = ac , it is a real polarization at f |m . Step 5: n = e0 + m, [m,e0 ] = 0, and n is a Heisenberg algebra. Define bc = ac ∩ ker f , so dc = ac = bc ⊕ zc . Then nc = ec + ac = h + h + ac = h0 + h0 + t + t + ac = (e0 )c + t + ac = (e0 )c + t + bc . To see that this sum is direct, let X ∈ (e0 )c , T ∈ t and Y ∈ bc such that X + T + Y = 0. Then T = −X − Y ∈ t ∩ afc ⊂ h ∩ afc ∩ t = h0 ∩ t = {0}. Hence T = 0 and Y = −X ∈ (e0 )c ∩ bc ⊂ ((e0 )c ∩ ac ) ∩ bc ⊂ ((e0 )c ∩ (e0 )fc ) ∩ bc = zc ∩ bc = {0}. Thus nc = (e0 )c ⊕ t ⊕ bc

and

mc = t + ac = t ⊕ bc ⊕ zc .

Therefore (e0 )c ∩ mc = zc , and these algebras being real, e0 ∩ m = z. Moreover, since e0 ⊂ af , [(e0 )c,mc ] = [(e0 )c,t] ⊂ [nc,nc ] = zc . But t is, for the form Bf , orthogonal to h0 and to h0 , then (e0 )c is orthogonal to t. Hence [(e0 )c,t] ⊂ zc ∩ ker f = {0}, which means that [e0,m] = {0}. Now recall that either e0 = z or e0 is a Heisenberg algebra with center z, and that m is a Heisenberg algebra with center z. It follows that n = e0 + m is a Heisenberg algebra with center z. Step 6: Reduction to the case n = m. Consider the direct product Lie algebra n˜ = e0 × m, with the subalgebras h˜ = h0 × hm,

h˜  = h0 × ac = h0 × hm .

Define the direct product N˜ = E0 × M of the connected groups associated to e0 and m, extend γ and f to N˜ and n˜ , as: γ˜ (x,m) = (γ (x),γ (m)),

f˜ (W,Y ) = f (W ) + f (Y ).

5.3 Orbit method for solvable groups

381

The map p : n˜ → n, p(W,Y ) = W + Y is a morphism of Lie algebras, with kernel {(X, − X) : X ∈ z}; extend p to a morphism still denoted p : N˜ → N , with kernel {(z,z−1 ) : z ∈ Z} where Z is the connected subgroup with Lie ˜ h˜  are γ˜ -stable positive polarizations at algebra z. It is easy to check that h, ˜f . Put: ˜ ˜ ), ˜ f˜ , h,χ ρ˜ = ind(N, f

˜ f˜ , h˜ ,χ ˜ ). ρ˜  = ind(N, f

Associated to these representations, there are the extensions C(γ˜ ), C  (γ˜ ). Observe that, if ρ = ind(N, f ,h,χf ), then Hρ˜ and Hρ are canonically ˜ ˜ ) → H(N, f ,h,χf ): ˜ f˜ , h,χ isomorphic through the well-defined map U : H(N, f (U F )(xm) = F (x,m),

(U −1 )(x,m) = (xm)

(x ∈ E0, m ∈ M).

Computations similar to those in the proof of Proposition 5.2.11 show that U C(γ˜ ) = C(γ )U . In the same way, we have a canonical isomorphism U  : Hρ˜  → Hρ  such that U  C  (γ˜ ) = C  (γ )U  . With these identifications we have: ρ = ρ˜ ◦ p,

ρ  = ρ˜  ◦ p.

Suppose for the moment that dim m < dim n. Then by the induction  (γ ) of ρ hypothesis the extensions CM (γ ) and CM M = ind(M, f |m,hm,χf )   and ρM = ind(M, f |m,hm,χf ) to γ |M are equivalent: there is an intertwining operator S : HρM → HρM such that  SCM (γ ) = CM (γ )S.

(5.3.1)

Define then T : Hρ˜ → Hρ˜  by (T F )(x,m) = S(F x )(m); here the function F x (m) = F (x,m) is in HρM for almost every x by construction. Then T intertwines ρ˜ and ρ˜ and   (T C(γ˜ )F )(x,m) = S (C(γ˜ )F )x (m) −1/2

−1/2

−1/2

−1/2

−1

= δ(γ )e0,z δ(γ )m,dm S(F γ (x) )(γ −1 (m))   −1 −1/2 = δ(γ )e0,z S(CM (γ )F γ (x) ) (m)   −1 −1/2  (γ )S)F γ (x) (m) = δ(γ )e0,z (CM = δ(γ )e0,z δ(γ )m,ac S(F γ = (C  (γ˜ )T F )(x,m),

−1 (x)

)(γ −1 (m))

382

Irreducible unitary representations

showing that the intertwining operator U  T U −1 intertwines C(γ ) and C  (γ ). Thus, if dim m < dim n, C(γ ) and C(γ  ) are equivalent. Therefore the only remaining case to prove is m = n. Step 7: End of the proof. We are now in the case where n is a Heisenberg algebra of dimension three or five, containing a γ -stable, abelian, noncentral ideal a, with nc = ac + t, h = t + zc a positive, γ -stable polarization not containing a, and h = ac is a real γ -stable polarization at f . Put b = ker f ∩ a, so a = b ⊕ z, and b is γ -stable. We first consider the case where h is real. Case (a): Suppose h real. There is c ⊂ ker f such that h = cc + zc and h ∩ a = (c ∩ b ) + z is a γ -stable abelian ideal, strictly included in a, thus c ∩ b = {0}, ker f = c ⊕ b is a direct sum of two γ -stable totally isotropic subspaces. Now the form Bf gives a form B on c × b which is nondegenerate and γ -invariant: Bf (γ X,Y ) = Bf (X,γ −1 Y ). This form gives a pairing between c and b . Realize the representation ρ = ind(N,D,χf ) in the space L2 (b ), by identifying a function ϕ ∈ H(N,D,χf ) to the function ϕ˜ on b : ϕ(Y ˜ ) = ϕ(exp Y ). By abuse of notation just write ϕ = ϕ. ˜ Then C(γ )ϕ(Y ) = ϕ(γ −1 Y )|det −1/2  . Realize similarly ρ = ind(N,D ,χf ) and C  (γ ) in L2 (c). A direct γ b | computation proves that the map FB : L2 (b ) → L2 (c) defined by 3 FB ϕ(X) = ϕ(Y )eiBf (Y,X) dY is an intertwining operator between ρ and ρ  . For instance, if X0 ∈ c, then (ρ(exp X0 )ϕ)(Y ) = e−i f [Y,X0 ] ϕ(Y ) and 3 FB (ρ(exp X0 )ϕ)(X) = ϕ(Y )eiBf (Y,X−X0 ) dY   = (FB ϕ)(X − X0 ) = ρ  (exp X0 )(FB ϕ) (X). On ker f , γ is a symplectic transformation, thus | det γ | = 1 = | det γc | | det γb |, and: 3 FB C(γ )ϕ(X) = ϕ(γ −1 Y )|det γb |−1/2 eiBf (Y,X) dY 3 = ϕ(Y )|det γb |1/2 eiBf (γ Y,X) dY 3 −1 = |det γc |−1/2 ϕ(Y )eiBf (Y,γ X) dY = C  (γ )FB ϕ(X).

5.3 Orbit method for solvable groups

383

Case (b): Now suppose h = h. We claim that h + h = nc . Since nc = t + ac , and t ⊂ h, it is enough to show that ac ⊂ h + h. If this were not the case, then by minimality of a, (h + h) ∩ ac = zc . Pick X ∈ h \ h, there is X ∈ t, Y ∈ ac such that X = X + Y . Thus X − X = Y belongs to ac ∩ (h + h) = zc . Hence, X ∈ t + zc = h, a contradiction. Given Y in b , write Y = 1i (W − W  ) with W and W  in t. Thus Re(W ) = Re(W  ), and 1i (W − W  ) belongs to t; since t ∩ n = {0}, then W = W  . This means that each Y ∈ b is the imaginary part of some W ∈ t. Suppose now dim n = 5; the proof in the case where dim n = 3 is a simplification of what follows, and is left to the reader. Pick Y1 ∈ b , Y1 = 0. Choose W1 = X1 + iY1 ∈ t. Then 4f [X1,Y1 ] = Hf (W1,W1 ) > 0. Since dim b = 2, then we have Y2 = 0 in b such that f [X1,Y2 ] = 0, and the same construction gives W2 = X2 + iY2 ∈ t. Then f [W1,W2 ] = 0, which implies [X1,X2 ] = 0. By normalizing the two vectors W1 and W2 , we can suppose Hf (Wi ,Wi ) = 4, and it follows that (Z,Y1,Y2,X1,X2 ) is the usual basis of a Heisenberg Lie algebra, with nonvanishing brackets [Xi ,Yi ] = Z, and f = Z ∗ . Now write γ W1 = aW1 + bW2,

γ W2 = cW1 + dW2,

since h and h (thus also t and b ) are γ -stable, and since γ is real, spanR (X1,X2 ) and spanR (Y1,Y2 ) = b are invariant and a, b, c and d are real numbers. Recall that γ is fixing f , and so its restriction to b is an isometry with respect to the form Hf . Remark that moreover, since  the vectors a b W1 , W2 are orthogonal with equal norms, then the matrix c d of γt is an orthogonal matrix. Moreover this matrix is also the matrix of γb and γc if c = span{X1,X2 } and | det γc | = | det γb | = 1. With this choice of basis for n, the representations ρ = ind(N, f ,h,χf ) and ρ  = ind(N, f ,h,χf ) are exactly the representations denoted respectively ρ and ρr in Proposition 5.1.15, with m = 2, and λ1 = λ2 = 1. By Proposition 5.1.15, the intertwining map V −1 : Hρ  → Hρ can be written: 3 V −1 ϕ(exp X) = (2π )−2 ϕ(exp X exp Y ) dY with X ∈ c, Y ∈ b . Now with γ (exp X) = exp γc (X), we have (C  (γ )V −1 ϕ)(exp X) = (V −1 ϕ)(exp γc−1 (X)) 3 = (2π )−2 ϕ(exp γc−1 (X) exp Y ) dY

384

Irreducible unitary representations

= (2π )−2 = (2π )−2

3 3

ϕ(exp γc−1 (X) exp γb−1  (Y )) dY ϕ(γ −1 (exp X exp Y )) dY

= (V −1 C(γ )ϕ)(exp X). This completes the proof of Theorem 5.2.10.

5.3.3 Parametrizing representations associated to orbits Recall Theorem 5.3.4: let G be a connected and simply connected solvable Lie group, with Lie algebra g, n a nilpotent ideal in g, containing [g,g],  an integral point in the dual g∗ of g, f the restriction of  to n. Fix a character χ of G() with differential i|g() , choose a positive G()-stable polarization h at , admissible for n, and such that h ∩ nc is stable under the adjoint action of G(f ). Then π = ind(G,,h,χ ) is irreducible. Corollary 5.3.7 The equivalence class of the representation π is independent of the choice of the polarization h satisfying the above conditions. Proof: Suppose that h and h are two polarizations at  satisfying the above properties. Put h1 = h∩nc , h1 = h ∩nc , h2 = h∩nf , h2 = h ∩nf . By Theorem 5.1.4, the representations ρ = ind(N, f ,h1,χf ) and ρ  = ind(N, f ,h1,χf ) are equivalent. By the preceding proof of Theorem 5.2.10, the extensions of ρ and ρ  to N  M are also equivalent. Consider now 2 = |nf , and the associated representations η = ind(G(f )0, 2,h2,χ2 ) and η = ind(G(f )0,2,h2,χ2 ). Recall the normal subgroup Q in G(f )0 such that G(f )0 /Q is simply connected and nilpotent, with q : G(f )0 → G(f )0 /Q. By Theorem 5.1.3, η ◦ q and η ◦ q are equivalent. Thus η and η are equivalent; let T : Hη → Hη be a unitary intertwining operator. By Lemma 5.2.7, we have M = G(f )0 G() is the stabilizer of η in G(f ) for the action of G(f ) on the dual of G(f )0 . Following the proof of Proposition  = 5.2.8 for the group G(f ), we see that ηM,χ = ind(M,2,h2,χ ) and ηM,χ   ind(M,2,h2,χ ) are extensions of η and η . More precisely, they are realized respectively on Hη and Hη , and the relation (5.2.3) says that, for each s0 ∈ G() and t ∈ G(f )0 , (ηM,χ (s0 )ϕ)(t) = χ (s0 )ϕ(s0−1 ts0 ),

 (ηM,χ (s0 )ψ)(t) = χ (s0 )ψ(s0−1 ts0 ).

 . Since these expressions are identical, then T intertwines ηM,χ and ηM,χ  Now consider σM,χ = ind(N M,2,h2,χ ) and σM,χ = ind(N M,2,h2,χ ). If p : N  M → N M is the canonical map, since σM,χ ◦ p = ρ ⊗ ηM,χ and

5.3 Orbit method for solvable groups

385

   ◦p = ρ  ⊗ηM,χ are equivalent, σM,χ and σM,χ are also equivalent. Finally σM,χ  ) by Proposition 3.3.19, π = ind(G,N M,σM,χ ) and π  = ind(G,N M,σM,χ are equivalent.

Thanks to Corollary 5.3.7, we put now, for any  integral and any character χ of G() such that dχ = i, [π,χ ] = [ind(G,,h,χ )] , where h is a positive polarization at , which is G()-stable, admissible for n and such that h ∩ nc is G(f )-stable. It will be important to now consider the following situation: let γ be a continuous automorphism of G, so that dγ is an automorphism of g and dγ ∗ : g∗ → g∗ is defined by dγ ∗ () =  ◦ dγ −1 . Suppose that  is integral and h is a G()-stable polarization at . Recall that if π is a representation of G, then the representation π γ is defined by π γ = π ◦ γ −1 . Observe (more generally) that if D is a closed subgroup of G and χ is a representation of D, then χ γ = χ ◦ γ −1 is a representation of γ (D). Lemma 5.3.8 Let χ be a character of D = D 0 G() for which dχ is the restriction of i to d. Then ind(G,,h,χ )γ  ind(G,dγ ∗ (),dγ (h),χ γ ). Proof: Write h = dγ (h),  = dγ ∗ (), χ  = χ γ and D  = γ D. By Lemma   3.1.4, We have left-invariant 7 measures 7 dμ on G, dν on D, and dν on D . The   measure dμ defined by G ϕdμ = G ϕ ◦ γ dμ (ϕ ∈ Cc (G)) is left-invariant, thus proportional to dμ: there is cγ > 0 such that 3 3 ϕ(γ (x)) dμ(x) = cγ ϕ(x) dμ(x). G

G

Similarly, there is aγ > 0 such that for each f ∈ Cc (D  ), 3 3 f (γ (y)) dν(y) = aγ f (y  ) dν  (y  ). D

D

Now, for Y ∈ g or Y ∈ d, and y ∈ D, Ad(γ (y))dγ (Y ) = dγ (Ad(y)Y ), and thus dγ −1 Ad(γ (y))dγ = Ad(y),

(dγ |d )−1 Ad(γ (y))|d (dγ |d ) = Ad(y)|d .

Therefore: G (γ (y)) = G (y),

D  (γ (y)) = D (y)

and

Let us write the last relation as δ  (γ (y)) = δ(y).

δD ,G (γ (y)) = δD,G (y).

386

Irreducible unitary representations

Now let f be a function in Cc (G,D,χ ), then f ◦ γ −1 is by definition in Cc (G,D ,χ  ). Recall that |f |2 is in Cc (G,D,δ) and there is φ ∈ Cc (G) such that 3 |f (s)|2 = Pδ (φ)(s) = φ(sy) δ(y)−1 dν(y) (s ∈ G). D

Thus |f ◦ γ −1 |2 is in Cc (G,D ,δ  ), and since Pδ  (φ ◦ γ −1 )(s) =

3

φ(γ −1 (sy  )) δ  (y  )−1 dν  (y  ) 3 1 φ(γ −1 (s)y) δ(y)−1 dν(y) = aγ D 1 1 (Pδ φ)(γ −1 (s)) = |f (γ −1 (s))|2, = aγ aγ D

then |f ◦ γ −1 |2 = aγ Pδ  (φ ◦ γ −1 ). Thus we have 3 G/D 

|f ◦ γ −1 |2 dμG,D  = aγ

3

Pδ  (φ ◦ γ −1 ) dμG,D  3 3 aγ = aγ φ ◦ γ −1 dμ = φ dμ cγ G G 3 aγ |f |2 dμG,D . = cγ G/D G/D 

This shows that f ◦ γ −1 is in H(G,D ,χ  ) if and only if f ∈ H(G,D,χ ). Similarly f ∈ C(G,,h,χ ) if and only if f ◦ γ −1 ∈ C(G,,h,χ  ). We leave the verification of this point to the reader. A c Finally define C : H(G,,h,χ ) → H(G,,h,χ  ) by (Cf ) = aγγ f ◦γ −1 ; then C is unitary and easily seen to be intertwining for ind(G,,h,χ )γ and ind(G,,h,χ  ). Now, suppose that γ is the inner automorphism t → sts −1 , then for each t ∈ G, π γ (t) = π s (t) = π(s)−1 π(t)π(s), so π γ is equivalent to π . For any  ∈ g∗ , which is integral, and any character χ of G(), recall the notation [π,χ ] = [ind(G,,h,χ )]. We get: Corollary 5.3.9 Let  be an integral point in the dual g∗ of g, and χ a character of G() such that dχ = i. Let s ∈ G, then [πs,χ s ] = [π,χ ].

5.3 Orbit method for solvable groups

387

Recall now Relation 4.7.1: the stability group G() can be written as: 5 a s1 1 . . . skak G()0 . G() = a∈Zk

Therefore each character χ of G() such that dχ = i is characterized by ν = (ν1, . . . ,νk ) ∈ Tk such that χ (si ) = νi , so, for z ∈ G()0 , a

χ (s1a1 . . . sk k z) = ν a χ (z). In Lemma 4.7.4, we saw that G()/[G(),G()] is isomorphic to G()/G()0 × G()0 /[G(),G()], accordingly we now write χ = ν ⊗ χ . The set of characters χ such that dχ = i is thus parametrized by Tk . Now for each s ∈ G, the character χ s of G(s) = sG()s −1 is: χ s ((s1 )a1 . . . (sk )ak z ) = ν a χs (z ), where si = ssi s −1 . Corollary 5.3.10 Let  be an integral point in g∗ and O its coadjoint orbit. ˆ given by ν → [π,ν⊗χ ] is one-to-one. Thus this map Then the map Tk → G ˆ O of classes of irreducible representation gives a parametrization of the set G k associated to O by T . This corollary can be considered to provide a first parametrization of a large ˆ of G. subset in the unitary dual G More precisely, put  = { ∈ g∗ :  is integral}, then  is G-stable. For each  ∈ , denote by X() the set of unitary characters χ of G() such that dχ = i|g() . Then we put X() = {(,χ ) :  ∈ , χ ∈ X()}. We say that (,χ ) ∼ (,χ  ) if and only if there is s ∈ G such that  = s, χ  = χ s . This is an equivalence relation on X(). Denote by [,χ] the equivalence class of (,χ ). By virtue of Theorem 5.3.4, Corollary 5.3.7, and Corollary 5.3.10, there is a well-defined canonical map AK defined by AK[,χ ] = [π,χ ]. Denote p the projection map p : [,χ] → G, we have the following diagram: p

X()/ ∼ −→ ⏐ ⏐ C AK ˆ G. Proposition 5.3.11 The map AK is injective.

/G

388

Irreducible unitary representations

Proof: It is enough to show that if O = G and O = G are two distinct integral coadjoint orbits in g∗ , then π,χ is not equivalent to π,χ  for any χ , χ . Let f = |n , f  =  |n . If Gf = Gf  , then the corresponding representations ρ and ρ  of N are not in the same G-orbit in Nˆ . Thus by Corollary 3.4.9, π,χ and π,χ  are inequivalent. Suppose now Gf = Gf  , by conjugation, we can suppose that now f = f  , ρ = ρ  = ρf and also π,χ = ind(G,NG(f ),σG(f ),χ ) and π,χ  =   ind(G,NG(f ),σG( f ),χ  ) are equivalent. Since σG(f ),χ and σG(f ),χ  restricted to  N are multiples of ρf , then by Corollary 3.4.9, σG(f ),χ is equivalent to σG( f ),χ  , thus by Proposition 5.2.12, there is s in G such that s =  , and χ s = χ  . This proves the proposition. ˆ int the image of the map AK, we can now complete the Denoting by G preceding diagram:

ˆ int is a fiber bundle over /G with fiber where q = p ◦ AK−1 . Thus G −1 ˆ q (O) = GO . If G is any simply connected solvable group, its coadjoint orbits are not necessarily integral or regular (see Examples 4.7.2 and 3.4.13). The main result of Auslander and Kostant in [9] is that if every coadjoint orbit of G is integral and regular, then the map AK is bijective, i.e., gives a parametrization of the ˆ of G. They also prove that G is type 1 if and only if its coadjoint unitary dual G orbits are integral and regular. Here is an example of the parametrization of the dual of a type 1 solvable Lie group. Example 5.3.12 We come back to the diamond Lie algebra (see Example 1.3.13) with basis (Z,Y,X,A) and commutation relation [X,Y ] = Z, [A,X + iY ] = −i(X+iY ). Put U = X+iY , U = X−iY , so that (Z,U,U,A) is a good Jordan–Hölder basis for s. The ideal n is here n = [g,g] = span{Z,U,U } ∩ g. An element  ∈ g∗ is written:  = (λ,μ,α) ∈ R × C × R, with (zZ + Re(uU ) + aA) = λz + Re(uμ) + αa.

5.3 Orbit method for solvable groups

389

Then the layers  = e,h,δ , with cross-sections  and the positive Vergne polarizations are given by the following table:



e

h

δ



∅

(λ,0,α)

h span{Z,U,A} (λ > 0)

λ = 0

{2,3} (3,2)

span{Z,U,A} (λ < 0) λ = 0, μ = 0 {2,4} (4,2) {2} (0,r,0) r > 0 λ=μ=0

∅

()

∅

(0,0,α) α ∈ R

span{Z,U,U } g

As in Example 1.7.15, the diamond group G is the semidirect product of the Heisenberg group N by the real line.

Case of the minimal layer The stability group of 0 in the cross-section 1 of the minimal layer 1 = {2,3},(3,2),∅ is G(0 ) = {exp(zZ + aA) : z,a ∈ R}. Therefore G() is connected for each  ∈ 1 , and hence X() = {χ }, and (,χ ) →  is a bijection from X(1 ) onto 1 , therefore X(1 )/ ∼ can be identified with 1 . We carry out the constructions described in this chapter to ˆ Fix determine, for each 0 ∈ 1 , the corresponding equivalence class in G. 0 ∈ 1 . Recall from Examples 4.1.9 and 4.2.4 that there is no real polarization at 0 , which means that an irreducible unitary representation associated to 0 cannot be obtained as an induced representation described in Chapter 3: holomorphic induction is required. We saw in Example 4.5.8 that the Vergne polarization associated to the choice (Z,U,U,A) of Jordan–Hölder basis is h = span{Z,U,A}; now 2iBσ (U,U ) = 4λ, so h is positive if and only if λ > 0. If λ > 0, we put W = U , h0 = h, while if λ < 0, we put W = U and use the positive polarization h0 = h, associated to the basis (Z,U,U,A) at 0 . In any case, h0 = span{Z,W,A} is

390

Irreducible unitary representations

a positive polarization, and we write an element of G as (z,w,a) = exp(zZ + Re(wW )) exp aA. The group D is G(0 ) and E = G. On the other hand, here f = 0 |n and it is easy to prove that G(f ) = G(0 ), so with the notation used in the diagram (5.2.6), we have here M = G(f )0 G(0 ) = G(0 ) and N M = G. Then ρ is the representation in K(N f ) defined by: ρ = ind(N, f ,h ∩ nc,χf ). Recall that we gave an explicit description of this representation. Using Lemma 5.1.8, Corollary 5.1.13, and Relation 5.1.10, ρ is realized in a space A of holomorphic functions. Specifically, A  H(N, f ,h ∩ nc,χf ) is the space of entire functions p on C such that: 3 2 2 p = |p(u)|2 e−|λ||u| /2 du < ∞, C

and ρ acts in A by   2 ρ(z,w)p (u) = eiλz e−|λ||w| /4 e|λ|wu/2 p(u − w). Put (z,a) = exp(zZ + aA) ∈ M. Let us now define the representation ρ ext of N  M, that is define the operators C(z,a) : A → A, we get: C(z,a)p(w) = p(e−isign(λ)a w), and ρ ext (z,w;z,a) = ρ(z,w)C(z,a). On the other hand, since M is abelian, the representation η is just a character. With our notation, this is: η(z,a) = eiλz eiαa = χ0 (z,a) = χ (z,a), and by definition η = ηM,χ = ηG(f ),χ  . The diagonal subgroup in N  M is diagN(f ) = {(exp zZ, exp −zZ) = (z,0,0),(−z,0) : z ∈ R}. Here clearly (ρ ext ⊗ η)(diagN (f )) = 1, this allows us to define the representation σ = σM,χ = σG(f ),χ . Here since N G(f )0 = G, we get σ = π . Denote this representation by πλ,α ; this construction gives the irreducible unitary representation such that AK([0,χ ]) = [πλ,α ], it is:   πλ,α ((z,w) exp aA)p(u) = ρ(z,w)η(0,a)C(0,a)p (u) = eiλz eiαa e−|λ||w|

2 /4

e|λ|wu/2 p(e−isign(λ)a (u − w)).

Case of the second layer Let  = (0,μ,α) be in 2 = {2,4},(4,2),{2} . We thus adapt the computations in Example 5.3.1 to G. First G() is not connected, if  = (0,μ,α), it is: G() = {exp(zZ + Re(uU )) exp 2π kA : z ∈ R, Im(uμ) = 0, k ∈ Z} = G()0 exp 2π ZA,

5.3 Orbit method for solvable groups

391

and g(f ) = n, G(f )0 = N , but G(f ) = N exp 2π ZA. Since the polarization here is nc , it is real and D = E = N exp 2πZA. As in Example 5.3.1,  is integral, χ is defined on D 0 = N by: χ (exp(zZ + Re(uU ))) = eiRe(uμ), and it can be extended to the characters χ ν indexed by ν ∈ R/Z such that: χ ν (exp(zZ + Re(uU )) exp 2π kA) = eiRe(uμ) e2iπ νk . This means that X() = {χ ν : ν ∈ R/Z}, and   X(2 ) = {(,χ ν ) = (0,μ,α),χ ν : μ ∈ C \ {0}, α ∈ R, ν ∈ R/Z}. For each s = (z,w,a) ∈ G, we verify immediately that s = s(0,μ,α) = (0,eia μ,α + Im(wμ)) and (χ ν )s (exp(zZ + Re(uU )) exp 2π kA) = eiRe(ue

ia μ)

e2iπ νk . 

This means that the equivalence relation ∼ on X(2 ) is (,χ ν ) ∼ (,χ ν ) if and only if there is s ∈ G such that  = s and ν  = ν. Hence the set X(2 )/ ∼ is identified to 2 × (R/Z), where 2 = {2,4},(4,2),{2} . Fix now 0 = (0,r,0) ∈ 2 (r > 0), and ν ∈ R/Z, and denote χ = χ ν . Using the notation of the diagram (5.2.6), we get here M = G(f ) = N M, thus the representation ρ = ρ ext is here simply χ , it coincides with ηM,χ = ηG(f ),χ . By construction, it coincides also with the representations σM,χ and σG(f ),χ . Finally, the representation π = ind(G,0,nc,χ ) in the diagram (5.2.6) is the representation denoted πν0 in Example 5.3.1. So we get that AK[0,χ ν ] = [πν0 ], and πν0 acts on the space Hπν of measurable functions f on R such 0

that f (t + 2π ) = e−2iπ ν f (t) almost everywhere, and 3 2π 2 f = |f (t)|2 dt < ∞, 0

as follows: 

 −it πν0 ((z,w) exp aA)f (t) = eiRe(re w) f (t − a).

Clearly, the map AK on the last layer gives the characters of G, on the form: π ((z,w) exp aA) = eiαa

( = (0,0,α) ∈ 3 = ∅,(),∅ = 3 ).



Now we illustrate the Auslander–Kostant characterization of type 1 solvable groups by two examples of simply connected solvable Lie groups which are not type 1.

392

Irreducible unitary representations

Example 5.3.13 Suppose that G = C2  R is the Mautner group defined in Example 3.2.21. It is easy to verify that each coadjoint orbit of the Mautner group is integral. However, we saw in Example 4.5.2 that many orbits are not regular. Recall that the representation Tξ,η defined in Example 3.2.21 is the representation induced from the character: χξ,η (u,v) = eiRe(uξ +vη), and thus Tξ,η = ind(G,C2,χξ,η ) (see Example 3.4.13), and the reader can prove that if ξ and η are not vanishing, it is the representation AK(O) where O is the (integral) coadjoint orbit of (ξ,η,0) in g∗ . In Example 4.5.2, we saw that this orbit is not regular. Recall also we prove in Example 3.2.36 that the Mautner group is not type 1. Now, if ξ or η is vanishing, the corresponding orbits are regular, and the Auslander–Kostant map is easy to describe. Finally, it is easy to verify that the  described in Example 3.2.21 and Example irreducible representations Tρ,σ,τ 3.4.13 are not in the range of the map AK.  Example 5.3.14 Similarly, we saw that the Duflo group G in Examples 4.7.2, 4.7.11, 5.2.6, and 5.3.2 is such that each coadjoint orbit of G is regular, but these orbits are generally not integral. Let us verify here that this group is not type 1. Recall its Jordan–Hölder basis (T ,Z,Z,W,W,B,A), with [A,Z] = iZ, [B,W ] = iW and [A,B] = T . Put (t,z,w) = exp(tT + Re(zZ + wW )) and (t,z,w;b) = (t,z,w) exp bB, (t,z,w;b,a) = (t,z,w) exp bB exp aA. Let us build some irreducible representations of G. Define L = exp l, where l is the ideal span(T ,Z,Z,W,W,B) ∩ g of g. Since the commutation relations in l are simply [B,W ] = iW , then H is the direct product of K = (R × C) with Lie algebra k = span(T ,Z,Z) ∩ g by E˜ 2 , with Lie algebra span(W,W,B) ∩ g. We consider the representations of L associated to the orbit of f = (τ,ζ,R,0) with τ = 0, R > 0 and |ζ | = r > 0. Since L(f )0 = exp(k + RRe(W )), and L(f ) = L(f )0 exp 2π ZB, we extend χf to a character of L(f ) by putting χ ν (exp 2π kB) = ν k , where |ν| = 1. By the Auslander–Kostant theory, the representation ρf ,χ ν = ind(L,KL(f ),χ ν ) is irreducible, moreover ρf ,χ ν  ρ   ν  with f  = (τ ,ζ ,μ,β  ) if and only f ,χ

if τ = τ  , ζ = ζ  , |μ | = R, and ν  = ν. Using the argument in the proof of Lemma 3.4.14, we can prove that the separation Property (3.4.1) holds for ρf ,χ ν and ρf ,χ ν  if τ = τ  or ζ = ζ  or |μ | = R. Suppose now τ = τ  , ζ = ζ  , and |μ | = R. In this case, we can suppose μ = R, but ν = ν  . In this case, consider a function ψ ∈ L1 (L), with support in

5.3 Orbit method for solvable groups

393

exp(k + Re(CW ) + (−π/2,π/2)B) such that ρf ,χ ν (ψ) = ρf ,χ ν  (ψ) is not vanishing (it is easy to find such a function). Thus ρf ,χ ν (L(exp 2π B)ψ − νψ) = 0, ρf ,χ ν  (L(exp 2π B)ψ − νψ) = (ν  − ν)(exp 2π B)π(ψ) = 0. This means that the separation Property (3.4.1) is still holding in this case. 1 Q, choose ζ such that |ζ | = r, realize the representation Fix now τ ∈ / 2π ρf ,χ ν on a space of functions φ on R such that φ(t − 2π ) = νφ(t), and 7 2π 2 0 |φ(t)| dt < ∞:   iu ρf ,χ ν (t,z,w;b)φ (u) = eitτ eiRe(zζ ) eiRRe(e w) φ(u − b). Put now ν = ei2π ξ , and ψ(u) = e−iξ u φ(u). Then ψ(u − 2π ) = = ψ(u). The function ψ is now in L2 (T), and the representation ρf ,χ ν becomes:    iu ρf ,χ ν (t,z,w;b)ψ (u) = eitτ eiRe(zζ ) eiRRe(e w) eiξ u φ(u − b) eiξ(u−2π ) ei2π ξ φ(u)

= eitτ eiRe(zζ ) eiRRe(e

iu w)

eiξ b ψ(u − b).

Then the group G acts on the representation ρf ,χ ν by exp RA, ρ a = ρf ,χ ν (exp −aA · exp aA). A direct computation gives (recall that f = (τ,ζ,R,0)):  a   ρ (t,z,w;b)ψ (u) = ρf ,χ ν (t − ab,e−ia z,w;b)ψ (u) = eitτ eiRe(ze ζ ) eiRRe(e w) eiξ b e−iτ ab ψ(u − b)   = ρa f ,χ ν  (t,z,w;b)ψ (u), ia

iu

1 where a f = (τ,eia ζ,R,0), and ν  = νe−2iπ τ a . Since τ ∈ / 2π Q, we get  G(ρf ,χ ν ) = L, and Corollary 3.4.9 proves that, for each complex number ζ and ν, the representation

πζ,ν = ind(G,L,ρf ,χ ν ) is irreducible. Moreover, πζ,ν  πζ ,ν  if and only if there is a ∈ R such that ζ  = ζ eia and ν  = e−2iπ τ a ν. Now the representation πζ,ν acts on L2 (R × T) as follows:  

 πζ,ν (t,z,w;b,a)f (s) (u)   = eiξ b ρf ,χ ν (t − bs,eis z,eiu w;0) f (s − a) (u − b)   is iu = eiξ b eiτ (t−bs) eiRe(e zζ ) eiRRe(e w) f (s − a) (u − b). We write now (f (s))(u) = f (s,u).

394

Irreducible unitary representations

7⊕ πζ,χ ν dμ(ζ,ν) As in Example 3.2.36, we look at7 the representation π = ⊕ Hπζ,χ ν dμ(ζ,ν) identified as in Examacting on the direct integral H = ple 3.2.9 with the space H = L2 (T2 × (R × T)) with usual Lebesgue Following the method used for the Mautner group, we put Vy = 7measure. ⊕ πζ,ν (0,0,0;0, − y)dμ(ζ,ν), we get:    Vy πζ,ν (t,z,w;b,a)Vy−1 f (s,u) = πζ,ν (t,z,w;b,a)Vy−1 f (s + y) (u)   is iy iu = eiξ b e−ibyτ eiτ (t−bs) eiRe(e ze ζ ) eiRRe(e w) Vy−1 f (s + y − a) (u − b) = πeiy ζ,e2iπξ e−2iπyτ (t,z,w;b,a)f (s,u) = πeiy ζ,νe−2iπyτ (t,z,w;b,a)f (s,u). We put (Wy f )ζ,ν = Vy−1 (feiy ζ,νe−2iπyτ ), the same computation gives: Wy ∈ C(π ), then any element in the center of C(π ) is a multiplication operator commuting with Wy and Vy , thus it is the multiplication by a function θ (ζ,ν) such that θ (ζ,ν) = θ (eiy ζ,νe−2iπyτ ) almost everywhere. We saw in Example 3.2.36, that this implies that π is a factor representation, and it is not a type 1 factor since for any (ζ,ν) the set of (ζ ,ν  ) such that πζ,ν  πζ ,ν  has measure zero. This proves that G is not a type 1 group. 

5.3.4 The exponential case In this section, we suppose that G is an exponential Lie group. By Proposition 1.9.13, G() is connected for each  ∈ g∗ ; hence every point  ∈ g∗ is integral and X() = {χ }. Note also that Proposition 1.9.13 shows that each coadjoint orbit of an exponential Lie group G is regular. Referring to the diagram before Proposition 5.3.11, we have  = g∗ , the p ˆ =G ˆ int map X(g∗ )/ ∼ −→ g∗ /Ad∗ (G) is therefore a bijection, and since G ∗ −1 ∗ ˆ we have a bijection K = q from g /Ad (G) onto G. This mapping is often called the Kirillov mapping: A. A. Kirillov [52] proved that when G is nilpotent, then this canonical map is a bijection between ˆ Later O. Takenouchi [87] extended this result to the class of g∗ /G and G. exponential groups. More precisely, starting with the g-polarization p() at  studied in Proposition 4.6.11, we immediately see that p() satisfies all our conditions, therefore the representation: π = ind(G,P (),χ ), where P () = exp p(), is irreducible. Moreover, K(O) = [π ], for each  ∈ O.

5.3 Orbit method for solvable groups

395

Using a good Jordan–Hölder basis (Zi ) such that nc = span{Z1, . . . ,Zp }, and Zi ∈ g if i > p, we consider an ultrafine layer  = e,h,δ with its crosssection  = e,h,δ . For each σ ∈ , we have the real polarization p(σ ). Let us now study the continuity of the map σ → p(σ ) defined on . Recall the basis of the Vergne polarization h(σ ) given in (4.5.4): {T (σ )Zi : i ∈ ec ∪ e+ }. Denote e = (h(σ ) + h(σ )) ∩ g, d = h(σ ) ∩ g. Then, the basis for dc is {T (σ )Zi : i ∈ ec ∪ (e+ \ f)}. Fix σ ∈ , choose a real basis for d contained in {Re(T (σ )Zi ), Im(T (σ )Zi ) i ∈ ec ∪ (e+ \ f)}, denote {X1 (σ ), . . . ,Xt (σ )} this basis, for each i, Xi (σ ) is either the real part or the imaginary part of some T (σ )Zj . Using Relation (4.6.2), we complete this basis by the set: {Ye (σ ) = Im(T (σ )Ze ) : e ∈ f} = {Xt+1 (σ ), . . . ,Xr (σ )}, in order to define a basis of p(σ ). Observe that for any e ∈ f, by construction, e ≤ p, thus Ye (σ ) is in n ∩ e = e1 . This means that e = e1 + d. By Lemma 4.6.8, we have [p(σ ),e] ⊂ [e,e] ⊂ d. Let P (σ ) be the connected subgroup of G with Lie algebra p(σ ). Recall that (λi ) is the sequence of roots of g and i (exp Y ) = eλi (Y ) . Lemma 5.3.15 The function δP (σ ),G does not depend on σ , more precisely, for each h ∈ P (σ ), 4 δP (σ ),G (h) = |e (h)|. e∈e−

Proof: Recall that, by Lemma 3.3.4, we have: 4   |j (h)|, with e(p(σ )) = j : sj ⊂ sj −1 + p(σ )c . δP (σ ),G (h) = j ∈e(p(σ ))

Let us now determine e(p(σ )). It is easy to see that if j − 1 and j are in I, then j ∈ e(p(σ )) if and only if j ∈ e− . Suppose now j − 1 ∈ I and j ∈ / I. If j ∈ / f, then j ∈ e− if and only if j ∈ e(p(σ )), and j + 1 ∈ e− if and only if j + 1 ∈ e(p(σ )). If j ∈ f, then j ∈ e(p(σ )) and j + 1 is not in e(p(σ )). This implies e(p(σ )) = (e− \ h(f)) ∪ f. Thus e(p(σ )) is independent of σ , and we can write: 4 4 4 δP (σ ),G (h) = |j (h)| |j (h)| = |j (h)|, j ∈e− \h(f)

j ∈f

j ∈e−

because, for each j ∈ f, |j (h)| = |h(j ) (h)|. Suppose now (σk ) is a sequence of points in  which converges to σ ∈ , let us make a choice of the real or imaginary part of T (σ )Zj , denote

396

Irreducible unitary representations

Xi (σ ) this choice (1 ≤ i ≤ r). Then, by continuity, we can suppose that {X1 (σk ), . . . ,Xr (σk )} is still a basis of p(σk ) for every k. Using this basis, we define a sequence of maps hk : Rr → G: $ % $ %

hk (t1, . . . ,tr ) = exp ti Xi (σk ) , h(t1, . . . ,tr ) = exp ti Xi (σ ) . i

i

Lemma 5.3.16 Let K be a compact subset of G, then ∪k h−1 k (K) is a bounded subset of Rr . Proof: Complete the basis {X1 (σ ), . . . ,Xr (σ )} into a basis {X1 (σ ), . . . ,Xr (σ ), Xr+1 (σ ), . . . ,Xn (σ )} of g. Fix the norm on g: ; ; n ; ; ; ; xs Xs (σ ); = sup |xs |. ; ; 1≤s≤n ; s=1

Since exp is a homeomorphism, there is C > 0 such that exp Y ∈ K implies Y ≤ C. Choose any sequence (tk = [tk,i ]) in Rr such that hk (tk ) ∈ K for each k; we show that (tk ) is bounded. Decompose the vectors Xi (σk ) using the basis Xs (σ ), we get: Xi (σk ) =

r

ak,j,i Xj (σ ) +

j =1

n

bk,s,i Xs (σ )

(1 ≤ i ≤ r).

s=r+1

By construction the r × r matrices Ak = [ak,j,i ] are converging to the identity matrix I , and the (n − r) × r matrices Bk = [bk,s,i ] are converging to 0. Now we have: ⎞ ⎛ r r r n



⎝ tk,i Xi (σk ) = ak,j,i tk,i Xj (σ ) + bk,s,i tk,i Xs (σ )⎠ i=1

i=1

=

r

j =1

j =1

(Ak tk )j Xj (σ ) +

s=r+1 n

(Bk tk )s Xs (σ ).

s=r+1

Since Ak → I we can suppose that Ak is invertible and write: sup |tk,i | = tk ≤ A−1 k

Ak tk .

i

≤ 1 + ε, and this gives: For k sufficiently large, A−1 k ; ; r ; ; 1 ; ; tk,i Xi (σk ); ≥ sup |(Ak tk )j | = Ak tk ≥ tk . ; ; 1≤j ≤r ; 1+ε i=1

5.3 Orbit method for solvable groups  Since hk (tk ) = exp ri=1 tk,i Xi (σk ) ∈ K, then tk sufficiently large. This proves the lemma.

397

≤ (1 + ε)C for any k

Denote χσ the character of P (σ ) = exp p(σ ) defined by χσ (exp Y ) = eiσ (Y ) . Recall now that the space Cc (G,P (σ ),χσ ) is defined in Relation (3.3.4) as the space of functions on G: fφ,σ (y) 3 φ(yh)χσ (h)δP (σ ),G (h)−1/2 dh = P (σ )

3 =

Rr

φ(yh(t))χσ (h(t))δP (σ ),G (h(t))

#  #  # 1 − e−adp(σ ) ( i ti Xi (σ )) ##  # dt, #det # adp(σ ) ( i ti Xi (σ )) #

−1/2 #

where φ is an element of Cc (G) with support Kφ . Therefore, putting Yσ (t) =  i ti Xi (σ ), we get: 3 fφ,σ (y) =

Rr

φ(y exp Yσ (t))eiσ (Yσ (t)) p(Yσ (t)) dt,

#  −adp(σ ) (Y )  # # # with p(Y ) = δP (σ ),G (exp Y )−1/2 # det 1−e #. Since [p(σ ),e] ⊂ d, adp(σ ) (Y ) then for any Y ∈ p(σ ) and any i ∈ f ∪ h(f), λi (Y ) = 0. This gives: # # # 1 − e−adp(σ ) (Y ) ## # #= #det # adp(σ ) (Y ) #

4 i∈ec ∪(e+ \f)∪h(f)

# # # 1 − e−λi (Y ) # # # #= # # λi (Y ) #

# # 4 ## 1 − e−λi (Y ) ## #. # # λi (Y ) # c +

i∈e ∪e

(5.3.2) Thus, by Lemma 5.3.15, p(Y ) = δP (σ ),G (exp Y ) =

4

|e−λi (Y )/2 |

i∈e−

#  # # 1 − e−adp(σ ) (Y ) ## # #det # # adp(σ ) (Y ) # # 4 ## 1 − e−λi (Y ) ## #. # # λi (Y ) # + c

−1/2 #

i∈e ∪e

Now, by definition of δP (σ ),G : δP (σ ),G (exp Y )−1/2 = G (exp Y )1/2 P (σ ) (exp Y )−1/2 4 = G (exp Y )1/2 |eλi (Y )/2 |. i∈e+ ∪ec

398

Irreducible unitary representations

So: p(Y ) = G (exp Y )

1/2

4

|e

λi (Y )/2

i∈e+ ∪ec

= G (exp Y )1/2

# # 4 ## 1 − e−λi (Y ) ## | # # # λi (Y ) # + c i∈e ∪e

# # 4 ## eλi (Y )/2 − e−λi (Y )/2 ## #. # # # λi (Y ) + c

i∈e ∪e

Observe that even if the polarization p(σ ) depends upon σ , the function p itself is defined on g and does not depend on σ . Suppose that (σn ) is a sequence converging to σ , then Lemma 5.3.16 has the following corollary: Corollary 5.3.17 Suppose that σk → σ in , then for each φ ∈ Cc (G), the sequence (fφ,σk ) converges to fφ,σ uniformly on compact subsets of G. Proof: Let V be any compact subset of G, Kφ the support of φ, and K = V −1 Kφ . By Lemma 5.3.16, there is a compact subset C of Rr containing ∪k h−1 σk (K), thus, for any k, the integral defining fφ,σk or fφ,σ is an integral on C: 3 fφ,σ (y) =

φ(y exp Yσ (t))eiσ (Yσ (t)) p(Yσ (t)) dt

(y ∈ V ).

C

If k → ∞, then Yσk (t) → Yσ (t) uniformly on C, thus φ(y exp Yσk (t)) p(Yσk (t)) tends to φ(y exp Yσ (t))p(Yσ (t)) uniformly in y ∈ V , t ∈ C. Moreover, |σk (Yσk (t)) − σ (Yσ (t))| ≤ |σn (Yσk (t)) − σ (Yσk (t))| + |σ (Yσk (t) − Yσ (t))| ≤ σk − σ

sup t∈C, k∈N

Yσk (t) + |σ (Yσk (t) − Yσ (t))|

is tending to zero uniformly in t ∈ C, thus eiσk (Yσk (t)) → eiσ (Yσ (t)) uniformly in t ∈ C, and fφ,σk (y) → fφ,σ (y) uniformly in y ∈ V . Recall the ultrafine layering {e,h,δ } for the coadjoint action in g∗ described in Proposition 2.2.20, and the cross-section e,h,δ of e,h,δ defined in ˆ and the ˆ e,h,δ = K(e,h,δ /G) of G Proposition 2.5.11, we get a layering G restriction of the Kirillov map to e,h,δ /G can be viewed as a bijective map: ˆ e,h,δ , Ke,h,δ : e,h,δ → G

Ke,h,δ (σ ) = [πσ ].

5.3 Orbit method for solvable groups

399

Proposition 5.3.18 Let G be an exponential Lie group and e,h,δ an ultrafine layer in g∗ , with cross-section e,h,δ . Then the map Ke,h,δ is continuous from ˆ e,h,δ . e,h,δ to G Proof: Let (σn ) be sequence converging to σ ∈  = e,h,δ . Fix a coexponential basis (Y1, . . . ,Yd/2 ) for h = p(σ ), then for n sufficiently large, (Y1, . . . ,Yd/2 ) is still a basis coexponential to hn = p(σn ) in g. This allows us to identify each space Hπσn to H = L2 (Rd/2 ) as in Lemma 3.3.10. Fix φ ∈ Cc (G), we identify fφ,σ with the L2 -function v: v(s) = fφ,σ (exp s1 Y1 . . . exp sd/2 Yd/2 ). Then, with H = exp h, we get: πσ (x)v,v = fφ,σ (x −1 ·),fφ,σ  3 3 φ(x −1 yh)φ(yh )χσ (hh−1 )δ(hh )−1/2 dhdh dμG,H (y) = G/H H ×H 3 3 = φ(x −1 yh h1 )φ(yh )χσ (h1 )δ(h h1 h )−1/2 dh1 dh dμG,H (y) G/H H ×H 3 3 = fφ,σ (x −1 yh )δ(h )−1 φ(yh ) dh dμG,H (y) G/H H 3 = fφ,σ (x −1 y)φ(y) dy. G

Similarly, we identify fφ,σn to the L2 -function vn , and: 3 fφ,σn (x −1 y)φ(y) dy. πσn (x)vn,vn  = G

We saw in Corollary 5.3.17 that the sequence (fφ,σn ) is converging uniformly on compact subsets to fφ,σ . Therefore for any compact set K, if x ∈ K and y ∈ Kφ = supp(φ), x −1 y is in the compact K −1 Kφ and πσn (x)vn,vn  converges to πσ (x)v,v uniformly on K. On the other hand, with the same argument, for any compactly supported function w, we have: 3 (fφ,σn − fφ,σ )(exp s1 Y1 . . . exp sd/2 Yd/2 )w(s) ds → 0. vn − v,w = supp(w)

This implies that vn,v → v,v, thus: vn − v

2

= vn − v,vn − v = vn,vn  − vn,v − v,vn  + v,v → 0.

400

Irreducible unitary representations

We proved that vn → v and πσn (x)vn,vn  → πσ (x)v,v uniformly on compact subsets. Using Relation (3.1.4) this implies that: πσn (x)v,v → πσ (x)v,v uniformly on compact subsets, for any v = fφ,σ . Since the space of these vectors is dense, by Corollary 3.1.24, this proves the proposition. Example 5.3.19 Let G be the Heisenberg group with dimension three. In Example 4.1.1, we saw that the coadjoint orbits are either two-dimensional planes, or points: O = λZ ∗ + RY ∗ + RX∗ = {(λ,γ ,β) : γ ,β ∈ R} if λ = 0, O = {γ Y ∗ + βX∗ } = {(0,γ ,β)}. There are two layers: the minimal one is the union of two-dimensional orbits:  = {2,3},(3,2),∅ , and the second one is the union of zero-dimensional orbits:  = ∅,(),∅ . The cross-section for the first layer is:  = {2,3},(3,2),∅ = {(λ,0,0) : λ = 0}. The cross-section for the second layer is   =  . To the first layer is associated the usual infinite-dimensional representations πλ (see Example 3.1.8), and to the second layer, the characters χγ ,β such that χγ ,β (exp(aZ + bY + cX) = ei(γ b+βc) . Proposition 5.3.18 proves that the maps λ → [πλ ] and (γ ,β) → [χγ ,β ] are continuous. In fact, they are clearly bicontinuous since if [πλn ] → [πλ ] (λ = 0), then it is easy to verify that πλn (0,0,z) = eiλn z → eiλz = πλ (0,0,z) for any z ∈ R, thus λn → λ. Similarly, if [πλn ] → [χγ ,β ], λn → 0 and if [χγn,βn ] → [χγ ,β ], the same argument shows that (γn,βn ) → (γ ,β). We want now to prove that if λ → 0, then for any (γ ,β), [πλ ] → [χγ ,β ]. First recall that πλ is unitarily equivalent to the representation denoted πγ ,λ in Example 3.2.16, such that: (πγ ,λ (x,y,z)f )(t) = eiγ y eiλz e−iλyt f (t − x). Fix ε > 0 and a compact subset K of G. Let f ∈ L2 (R), and f0 ∈ Cc (R) such that f − f0 < ε there is η > 0 (depending on f0 ) such that |λ| < η implies: sup{|eiλ(z−yt) − 1| : (x,y,z) ∈ K, t ∈ supp(f0 )} < ε.

5.3 Orbit method for solvable groups

401

Therefore: # # 3 # # #πγ ,λ (x,y,z)f0,f0  − eiγ y f0 (t − x)f0 (t) dt # < ε|Lx f0,f0 | # # if |λ| < η for each (x,y,z) ∈ K. Thus: # # 3 # # #πγ ,λ (x,y,z)f ,f  − eiγ y f (t − x)f (t) dt # # # < 2( f + f0 )ε + |Lx f0,f0 |)ε   < 2(2 f + ε) + ( f + ε)2 ε. In other words, for any f , any ε > 0, there is ηf > 0 (depending of f ) such that: # # #πγ ,λ (x,y,z)f ,f  − eiγ y Lx f ,f # < ε. We make now an unitary transform by using Fourier transform: 3 1 f (t)eitu dt, fˆ(u) = √ 2π getting:

# # 3 # # #πγ ,λ (x,y,z)f ,f  − eiγ y fˆ(u)eiux fˆ(u) du# < ε. # #

Now fix β ∈ R. There is α > 0 such that sup{|eiux − eiβx | : (x,y,z) ∈ K} < ε if |u − β| ≤ α, we choose f such that suppfˆ ⊂ [β − α,β + α], fˆ(u) ≥ 0 and fˆ = 1, thus with this choice of f , if |λ| < ηf , # # sup{#πγ ,λ (x,y,z)f ,f  − ei(γ y+βx) # : (x,y,z) ∈ K} < 2ε. This proves that [πλ ] = [πγ ,λ ] is in the neighborhood U(1,K,2ε) of [χγ ,β ] if  |λ| < ηf . The preceding example shows that for the Heisenberg group G, the Kirillov ˆ is not only a bijection, but also a homeomorphism map K from g∗ /G onto G ∗ ˆ the Fell topology. when we put on g /G the quotient topology, and on G This is a general result, proved for any nilpotent simply connected Lie group by I. Brown in [13], and generalized to any exponential Lie group G in the book [61] by H. Leptin and J. Ludwig.

6 Plancherel formula and related topics

Let G be a simply connected solvable Lie group with Lie algebra g. Given an integral element  ∈ g∗ , and a character χ of G() extending the character χ on G()0 , it is shown in Chapter 5 that there is a suitable polarization h ⊂ gc at  such that the holomorphically induced representation ind(G,,h,χ ) is irreducible. This construction, combined with further results from Chapter 5, ˆ ˆ O of the dual G then associates to the integral coadjoint orbit O of  a subset G ˆ O are realized as above and parametrized in a canonical way. The elements of G by the various unitary characters of G() extending χ . We have seen a number of examples, including the simply connected cover of the motion group in the plane, where G() is not connected, and the example of the diamond group shows that even when G() is connected it is sometimes necessary to use a polarization h that is not real: h = h. When G is exponential, we have seen that both of these issues vanish: the stabilizer G() is connected for every  ∈ g∗ (so every  is integral), and one can always find a suitable real polarization. Though nonreal polarizations have proved useful objects of study in the theory of exponential groups, every irreducible unitary representation of an exponential solvable Lie group is equivalent with one induced from a closed connected subgroup H of G as in Section 3.3. A number of nonexponential groups have this property as well. In the present chapter, not only are coadjoint orbits integral, but it is assumed that real polarizations exist. With these assumptions we first consider any coadjoint orbit O that admits a real polarization h at  ∈ O satisfying the Pukanszky condition, and we derive expressions for the canonical measure κO that are naturally associated to the structure of the orbit. For much of this chapter it is assumed that G is exponential. Given a choice of good Jordan– Hölder basis, the layering and orbital parametrizations on Chapter 2 are used

402

6.1 Invariant measure on a coadjoint orbit

403

to derive orbital character and semicharacter formulae, and then to write a concrete Plancherel formula as an integral over the cross-section  for the minimal Zariski open layer.

6.1 Invariant measure on a coadjoint orbit Let G be a simply connected solvable Lie group with Lie algebra g, and let O be a coadjoint orbit in g∗ . Fix 0 ∈ O, and let h be a real polarization at 0 , meaning as always that h = h, so that (in the notation of Chapter 4) d = h ∩ g = e is a g-polarization. Assume moreover that G(0 ) is connected. Our object of study here will be the unitary representation induced from the character χ0 of the closed connected subgroup H corresponding to e in the manner of Section 3.3. We now change notation so as to better match Chapter 3: write h in place of h ∩ g, so that H is the closed connected subgroup whose Lie algebra is the g-polarization h. Recall that h⊥ = { ∈ g∗ : |h = 0}. Assume that h fulfills the Pukanszky condition: H 0 = 0 +h⊥ ⊂ O. Since [h,h] ⊂ ker 0 , h⊥ is an h-module. On g/h, the determinant of the H -action is: δ(h) = δH,G (h) =

det Adg (h) = det Adg/h (h). det Adh (h)

Now h⊥ is naturally identified with (g/h)∗ by defining m(X + h) = m(X), m ∈ h⊥ , and thereby the action of H on h⊥ is identified with the dual action of H on (g/h)∗ . Therefore, the determinant of the dual action of H on h⊥ is det(m → hm) = δ(h)−1 . Choose a coexponential basis (X1, . . . ,Xp ) for h in g (Proposition 1.7.10); note that with d = dim O, p = d/2. Define the basis (Xj∗ ) for h⊥ by Xj∗ (h) = 0,

Xj∗ (Xj  ) = δjj  .  Thus each element m ∈ h⊥ is written as m = mj Xj∗ , and we let dm be the Lebesgue measure on h⊥ so obtained. Now H /G(0 )  H 0 = 0 + h⊥ , and each element of O is written uniquely as exp t1 X1 . . . exp tp Xp (0 + m) where tj ∈ R, m ∈ h⊥ . Thus we obtain a measure on O, called the mixed measure defined for F ∈ Cc (O) by 3 3 F (exp tX(0 + m)) dmdt, (6.1.1) Rp

h⊥

where exp tX(0 + m) = exp t1 X1 . . . exp tp Xp (0 + m).

404

Plancherel formula and related topics

Lemma 6.1.1 Let G be a simply connected solvable Lie group with Lie algebra g, fix 0 ∈ g∗ , such that G(0 ) is connected, and let h be a g-polarization at 0 satisfying the Pukanszky condition. Let (X1, . . . ,Xp ) be a coexponential basis for h in g. Then the orbital measure given by (6.1.1) is G-invariant. Proof: Since [h,h] ⊂ ker 0 , then h0 = 0 + m(h), with m(h) ∈ h⊥ . Thus for all h ∈ H and s ∈ G, 3 3 F (sh(0 + m)) dm = δ(h) F (s(0 + m)) dm h⊥

h⊥

7 holds for all positive F ∈ Cc (O), and f : s → h⊥ F (s(0 + m)) dm is a positive Borel function satisfying f (sh) = δ(h)f (s). By Lemma 3.2.19 the left-invariant form μG,H on Cc (G,H,δ) extends to such functions f , and by Lemma 3.3.10, μG,H can be defined by 3 3 μG,H (f ) = f dμG,H = f (exp t1 X1 · · · exp tp Xp ) dt. Rp

G/H

Accordingly the measure defined by (6.1.1) on the coadjoint orbit O is given by 3 3 3 3 F (exp tX(0 + m)) dmdt = F (s(0 + m)) dm dμG,H (s). Rp

h⊥

G/H

h⊥

The G-invariance of this measure follows from the G-invariance of the form μG,H : let s0 ∈ G. Then, by linearity, for any F ∈ Cc (O), 3 3 F (s0 s(0 + m)) dm dμG,H (s) G/H h⊥ 3 3 = F (s(0 + m)) dm dμG,H (s). G/H

h⊥

Next consider the volume form dtdm = dt1 ∧ · · · ∧ dtp ∧ dm1 ∧ · · · ∧ dmp on O defined on O by the homeomorphism above. The G-invariant mixed measure defined by (6.1.1) is the measure associated to this form. To compare the mixed measure to the Kostant measure κO , it is enough to compare the value of the two corresponding volume forms at the point 0 . Lemma 6.1.2 The following holds: dtdm =

(−1)p(p+1)/2 dκ0 . p!

6.1 Invariant measure on a coadjoint orbit

405

Proof: Regard h⊥ as the subspace of the space tangent at 0 to O generated by the tangent vectors ∂mj |0 . Since H 0 = 0 + h⊥ , for each 1 ≤ j ≤ p there Y

is Yj in h such that ξ0j = ∂mj |0 . Then spanR {Y1, . . . ,Yp } is a supplementary space to g(0 ) in h, and {X1, . . . ,Xp,Y1, . . . Yp } is a basis of a suppleX

mentary space to g(0 ) in g. Now the form dti is such that dti (ξ0 j ) = δij (as X

in Example 4.1.5) and dti (∂mj |0 ) = 0. On the other hand dmi (ξ0 j ) = 0 and Y

dmi (ξ0j ) = dmi (∂mj ) = δij . This means that (dt1, . . . ,dtp,dm1, . . . ,dmp ) is the dual basis of X

Y

(ξX0 1 , . . . ,ξ0 p ,ξY01 , . . . ,ξ0p ). Now  → mi () = (Xi ) is the Xi∗ -coordinate Y

function, so ξ0j mi = 0 [Xi ,Yj ], and so: Y

0 [Yj ,Xi ] = −ξ0j mi =

# d ## mi (0 + tXj∗ ) = δij . dt #t=0

Therefore the Kirillov form at 0 is:

ω 0 = 0 [Yi ,Yj ] dmi ∧ dmj i > ± dpe = dpe ∧ (dpe ∧ dpe+1 ). e∈e

e∈e\e2

e∈e2 \I

Observe that by Proposition 2.5.14, if e = ek ∈ e \ e2 , then pe depends only on z1, . . . ,zk , while if e = ek ∈ e2 \I, pe and pe+1 depend on z1, . . . ,zk+1 . Hence ∂pe to compute ∧e∈e dpe , we need only consider the partials ∂zkk |z∗ for ek ∈ / e2 and # # # # ∂pek+1 ## ∂pek+1 ## ∂pek ## ∂pek ## , , , ∂zk #z∗ ∂zk+1 #z∗ ∂zk #z∗ ∂zk+1 #z∗ for ek ∈ e2 \ I. Using the expressions of pi in Proposition 2.5.14, we get: if ek ∈ e0 \ δ, ∂pe ∂pe ∂pe ∂pek then ∂zkk = 1, while if ek ∈ e2 \ I, then ∂zkk = 1, ∂zk+1 = i and ∂zk+1 = 1, k # ∂pek+1 ∂pek # ∗ ∂zk+1 = −i. Now if ek ∈ e1 \ δ, then ∂zk #z∗ = ck (z ,σ ) = ck (σ ). Finally, if # ∂pe # ek ∈ δ, then ∂zkk # ∗ = (1 + iaek )bek (σ ). Hence ±

> e∈e

z

dpe = (−2i)|e2 |/2

4 ek ∈e1 \δ

ck (σ )

4 ek ∈δ

(1 + iaek )bek (σ )

> k

dzk .

408

Plancherel formula and related topics

Recall that |ck (σ )| = |bek (σ )| = 1, therefore, we get the following comparison between the real volume forms. Proposition 6.1.4 Let G be an exponential Lie group with Lie algebra g. Choose a good Jordan–Hölder basis (Zi ) for gc , with dual basis (fi ) and good descending flag (Ui ) for the coadjoint action in g∗ . Carry out the constructions of Chapter 2 for the coadjoint action, and let  = e,h,δ be the minimal layer, with explicit cross-section . Fix σ ∈  and let O be the coadjoint orbit of σ . Choose a covering set O in , containing σ and let P : Z ×  ∩ O →  be the parametrization defined in Proposition 2.5.8. Then the Kostant measure dκO on O is the measure defined by the G-invariant volume form on O whose value at σ is: d 2|e2 |/2 ek ∈δ |1 + iaek |(d/2)! > #

# dzk . #Pf σ [Ze,Ze ] # k=1 Put e (s)

= e∈e e (s), and let Pfe () be the Pfaffian of the skew-symmetric matrix [Ze,Ze ] e,e ∈e . Then the Kostant measure on O is given by 3

-

3 |1 + iaek |(d/2)! # # F () dκO () = F ()dμσ () #Pfe (σ )# O O 2|e2 |/2 ek ∈δ |1 + iaek |(d/2)! # # = #Pfe (σ )# 3 × F (P (z,σ ))|∗e (g(z,σ ))|−1 dz. 2|e2 |/2

ek ∈δ

Z

Example 6.1.5 Consider again the Heisenberg group of dimension three. Choose the good Jordan–Hölder basis (Z,Y,X) as before, and carry out the procedure of Chapter 2. The open layer is  = {(λ,γ ,β) : λ = 0}, where e = {2,3}, h = (3,2), and δ = ∅. The covering of  is just , the crosssection is  = {(λ,0,0) : λ = 0} and the parameter set Z = R2 . One finds easily that for σ ∈ , P (z1,z2,σ ) = (λ,z1,z2 ), Pfe (λ,0,0) = −λ and hence the Kostant measure is given by dκO = |λ|−1 dz1 dz2 . Now consider the good Jordan–Hölder basis (Z,X + iY,X − iY ). Then the open layer , the jump set e, the cross-section , and the parameter set Z all remain the same.Since P is written with real coordinates,  P is also the same: P (z1,z2,σ ) = P1 (z1,z2,σ ),P2 (z1,z2,σ ),P3 (z1,z2,σ ) = (λ,z1,z2 ). However, we now have e = e2 and the Pfaffian uses the complex basis (Z,X + iY,X−iY ). Hence Pfe (λ,0,0) = [X+iY,X−iY ] = −2iλ, while 2|e2 |/2 = 2. Of course the result is the same: dκO = 2|e2 |/2 |Pfe (λ,0,0)|−1 dz1 dz2 =  |λ|−1 dz1 dz2 .

6.2 Character formula

409

Example 6.1.6 We return to Example 4.1.5: gc has a good Jordan–Hölder basis (Z,Z,Y,X,X,A) with [A,X] = −(1 + i)X,[A,Z] = −(1 + i)Z, and [X,Y ] = Z. The minimal ultrafine layer is e,h,δ where e = {1,3,4,6},h = (6,4,3,1), δ = {1}. A typical element σ in the cross-section  for the minimal ultrafine layer  is: σ = (λ,λ,0,λix, − λix,0) where |λ| = 1 and x ∈ R. There is only one nonempty open covering set O = . For all σ ∈ , the parametrization z → P (z,σ ) maps Z = (0,+∞)× R × R × R diffeomorphically onto O and is given by  = P (z,σ ) = (λz11+i ,λz11−i ,z2,λz1i (z3 + ixz1 ),λz1−i (z3 − ixz1 ),z4 ). Comparing with the parametrization of Example 4.1.5 we see that z1 = |∗1 (exp t1 A)| = |∗4 (exp t1 A)| while ∗3 (exp t1 A) = ∗6 (exp t1 A) = 1. It follows that 4 |∗e (g(z,σ ))|−1 = z1−2 . e∈e

√ Now (d/2)! = 2, ek ∈δ |1 + iaek | = |1 + i| = 2, e2 = ∅, and |Pfe (σ )| = √ | − (1 + i)λ2 | = 2. Hence by Proposition 6.1.4 3 3 f () dκO () = 2 f (P (z,σ )) z1−2 dz.  O

Z

6.2 Character formula Let (π,H) be an irreducible unitary representation of a separable locally compact group G. If dim H < ∞, then the character associated to π is the function on G defined by χπ (s) = trace(π(s)). When G is compact, every irreducible unitary representation is finite dimensional, and much of representation theory can be cast in terms of the theory of characters. When G is a simply connected solvable Lie group, we have seen that irreducible unitary representations are generally not finite dimensional, and thus trace(π(s)) has no meaning. However the notion of trace can be given a meaning if we consider not the unitary operators π(s), but rather the bounded operators π(φ), where φ ∈ L1 (G). Indeed, the set of all φ ∈ C ∗ (G) such that [π ] → trace(π(φ)) ˆ is an important two-sided ideal in C ∗ (G). The is finite and continuous on G generalized notion of character is deeply connected to the so-called primitive

410

Plancherel formula and related topics

ideal space Prim(G), and related results are numerous and deep; we mention here a selection of pioneering work: R. Godement [45, 46], L. Pukanszky [82, 84], N.V. Pedersen [74, 73], and R. Lipsman [62]. Recall again the example of the Heisenberg group G. Following Examples 3.2.35 and 3.4.15, each irreducible unitary representation π of G is either a unitary character, or is equivalent to the unitary representation πλ acting in L2 (R) by (πλ (x,y,z)f )(t) = eiλz e−iλyt f (t − x), f ∈ L2 (R). Given a Schwartz function φ on G = R3 , the operator πλ (φ) is an integral operator given by the kernel K(x,t) = (∧y,z φ)(t − x, − λt,λ). When G is identified with R3 as in Example 3.4.15, the Euclidean Fourier transform φˆ of φ can be regarded as a function on g∗ . Referring also to Examples 4.1.4 and 6.1.5, one finds that πλ (φ) is trace class with trace given by 3 3 1 1 1 ˆ trace(πλ (φ)) = (6.2.1) dγ dβ = φ(β,γ ,λ) φˆ dκO . 2π O |λ| 2π O In fact for arbitrary simply connected nilpotent Lie groups G, if π is an irreducible unitary representation of G then φ → trace(π(φ)) is finite on Cc∞ (G) and defines a tempered distribution (there is a natural notion of Schwartz functions on G). By contrast, if G is the affine group, and π is an irreducible unitary representation corresponding to one of the open orbits, then it is easy to find φ ∈ Cc∞ (G) for which π(φ) is not trace class (see Example 6.2.1). Generally, it is known that for a large class of groups G, and for large classes of unitary representations π , the ideal of functions φ ∈ L1 (G) for which π(φ) is trace class is nontrivial. In fact, for exponential solvable Lie groups, there are explicit constructions, by M. Duflo [31], and N.V. Pedersen [74], of smooth functions ψ for which π(ψ) is trace class. Although such constructions are beyond the scope of this book, we do show how to compute the character formula for an irreducible unitary representation π under suitable conditions, and in the case where G is exponential and π is irreducible, we use the orbital parameters developed in Chapter 2 to derive explicit character formulas that generalize the formula (6.2.1).

6.2.1 Computation of the kernel For an irreducible unitary representation π of a nonnilpotent exponential solvable Lie group, the operator π(φ) may not be a trace class operator, even when φ ∈ Cc∞ (G).

6.2 Character formula

411

Example 6.2.1 Consider the group G = Aff(R) as in Example 1.6.5, realized as R  R by (y,t)(y ,t  ) = (y + et y ,t + t  )

(y,y ,t,t  ∈ R).

An irreducible representation associated to the coadjoint orbit of Y ∗ can be realized on L2 (R) by: (π(y,t)f )(s) = eie

−s y

f (s − t).

A formal computation describes the operator π(ψ) as 3 3 −s ψ(y,t)eie y f (s − t)e−t dydt = K(s,t)f (t) dt, (π(ψ)f )(s) = R2

where

R

3 K(s,t) =

R

ψ(y,t − s)eie

−s y

dyet−s = ∧y ψ(e−s ,t − s)et−s .

Here ∧y ψ is the partial Fourier transform of ψ with respect to the variable y. Since π(ψ) is bounded, we may say that π(ψ) is an integral operator. Now suppose that ψ = φ ∗ ∗ φ, with φ smooth and compactly supported. Since π(ψ) = π(φ)∗ π(φ) is positive definite, then K(s,s) ≥ 0. The general theory (see the book by Kato [51]) says that π(ψ) is trace class if and only if 7 K(s,s) ds is finite, whence R 3 K(s,s) ds. trace π(ψ) = But if ∧y ψ(0,0) = 1, then

7

R K(s,s)

R

ds = +∞.



Our strategy for the character formula is to compute the kernel K of the operators π(φ ∗ ∗ φ), and to write, as in the example: 3 trace π(φ ∗ ∗ φ) = K(s,s) ds, where it is understood that if π(φ ∗ ∗ φ) is not trace class, then its trace is +∞. Such an expression is called the character formula for the class of the representation π. We begin with a general computation. Let G be a locally compact group, let H be a closed subgroup, and let χ be a character of H . Fix left Haar measures μG for G and μH for H and put δ(h) = H (h)/G (h) as usual. For brevity, write dh = dμH (h) and dt for dμG (t). Recall the space Cc (G,H,δ) of continuous functions f on G satisfying g(xh) = δ(h)g(x),x ∈ G,h ∈ H ,

412

Plancherel formula and related topics

and supp(g)H is compact in G/H . Recall also the projection Pδ : Cc (G) → Cc (G,H,δ), and the G-invariant form μG,H on Cc (G,H,δ) defined by (3.3.3): 3 3 3 φ(sh)δ(h)−1 dh dμG,H (s), φ ∈ Cc (G). μG,H (Pδ φ) = φ(s) ds = G

G/H H

Let ψ ∈ Cc (G) be of the form ψ = φ ∗ ∗ φ for some φ ∈ Cc (G), and let π = ind(G,H,χ ). We compute a kernel for the operator π(ψ) acting in H(G,H,χ ). For f ∈ Cc (G,H,χ ), π(ψ)f is an element of H(G,H,χ ) defined pointwise by 3 3 (π(ψ)f )(s) = ψ(t)(f )(t −1 s) dt = ψ(st)f (t −1 ) dt G G 3 = G (t)−1 ψ(st −1 )f (t) dt G 3 3 = G (th)−1 ψ(sh−1 t −1 )f (th)δ(h)−1 dh dμG,H (t) G/H H  3 3 −1 −1 −1 −1 −1 −1/2 G (t) ψ(sh t )χ (h) G (h) δ(h) dh = G/H

H

× f (t) dμG,H (t). With the observation that δ(h)−1/2 = G (h)H (h)−1 δ(h)1/2 , and d(h−1 ) = H (h)−1 dh, we continue:  3 3 −1 −1 −1/2 (π(ψ)f )(s) = G (t) ψ(sht )χ (h)δ(h) dh f (t) dμG,H (t) G/H H 3 = Kψ (s,t) f (t) dμG,H (t), G/H

with: Kψ (s,t) = G (t)

−1

3

ψ(sht −1 )χ (h)δ(h)−1/2 dh, s,t ∈ G.

H

Observe that for each t ∈ G, Rt −1 ψ ∈ Cc (G) and Kψ (s,t) is given by (3.3.4): K(s,t) = fRt −1 ψ , so for each t ∈ G, Kψ (sh,t) = δ(h)1/2 χ (h)−1 Kψ (s,t). Also for each s ∈ G and f ∈ Cc (G,H,χ ), t → Kψ (s,t)f (t) ∈ Cc (G,H,δ). Finally, observe that the function F : s → Kψ (s,s) is positive. Indeed, the same computation, replacing ψ by φ gives that π(φ) is a kernel operator with kernel Kφ (s,t) and that the function F satisfies: 3 |Kφ (s,t)|2 dμG/H (t). F (s) = Kψ (s,s) = G/H

6.2 Character formula

413

It is easy to check that the function F satisfies F (sh) = δ(h)F (s) for any s ∈ G and h ∈ H : fix h0 ∈ H and write 3 −1 −1/2 ψ(sh0 hh−1 dh Kψ (sh0,sh0 ) = G (sh0 )−1 0 s )χ (h)δ(h) H

= G (s)−1 G (h0 )−1 H (h0 ) 3 −1 −1/2 ψ(shs −1 )χ (h−1 dh × 0 hh0 )δ(h0 hh0 ) H 3 = G (s)−1 δ(h0 ) ψ(shs −1 )χ (h)δ(h)−1/2 dh = δ(h0 )Kψ (s,s). H

Since F (s) = Kψ (s,s) is a positive Borel function satisfying F (sh) = δ(h) F (s) for any s ∈ G and h ∈ H , then Lemma 3.3.24 shows that 3 μG,H (F ) = Kψ (s,s) dμG,H (s) G/H

is defined. Since in general μG,H is not a measure, we need to show that the aforementioned theorem for integral operators due to Mercer [11, 51] still holds in this setting. This is done using a rho-function ρ: the space of π is unitarily equivalent to L2 (G/H,dνρ ) by the map f → fρ −1/2 (see Relation (3.3.5)). In this realization, the operator π(ψ) becomes π˜ (ψ)(ρ −1/2 f )(s) = ρ(s)−1/2 π(ψ)(f )(s) 3 ρ(s)−1/2 Kψ (s,t)f (t)ρ(t)−1 dνρ (t), = G/H

so its kernel is (s,t) → ρ(s)−1/2 Kψ (s,t)ρ(t)−1/2 . We conclude that π(ψ) is trace class if and only if the integral with respect to dνρ of ρ(s)−1 Kψ (s,s) is finite, and in this case the trace of this operator is the value of the integral. We sum up the preceding as follows. Lemma 6.2.2 Let G be a separable locally compact group, with H a closed subgroup and χ a character of H , and let π = ind(G,H,χ ). Let ψ ∈ Cc (G) be of the form ψ = φ ∗ ∗φ with φ ∈ Cc (G). Then π(ψ) is a trace class operator if and only if the positive function F (s) = Kψ (s,s) has a finite value for the form μG,H . If π(ψ) is trace class, then 3 Kψ (s,s) dμG,H (s) trace (π(ψ)) = G/H 3 3 −1 = G (s) ψ(shs −1 )χ (h)δ(h)−1/2 dh dμG,H (s). G/H

H

414

Plancherel formula and related topics

6.2.2 The exponential case Let G be an exponential solvable Lie group with Lie algebra g. Let (sj ) be a good Jordan–Hölder sequence of ideals in s = gc such that for some p, sp is a nilpotent ideal of s, containing [s,s], and si is real if i ≥ p and sn−1 ⊂ ker dG . Fix 0 ∈ g∗ . Given this data, we construct the layering and determine the ultrafine layer e,h,δ containing 0 . We construct a g-polarization as in Proposition 4.6.11; as indicated above we now denote this g-polarization by h, and let H = exp h be the corresponding closed connected subgroup of G. In Proposition 4.6.11 it is shown that h satisfies the Pukanszky condition. Recall that e = e+ ∪ e− where e+ = {e ∈ e : e < h(e)}. By Lemma 5.3.15, for h = exp Y , δ(h) = δH,G (h) =

4

4

|e (h)| =

e∈e−

|eλe (Y ) |,

e∈e−

and this implies that for Y ∈ h, 4

H (exp Y ) =

|e−λi (Y ) |.

i∈ec ∪e+

Fix a Lebesgue measure dY on h (this can be done with any basis of h). By Lemma 3.1.2, a left Haar measure on H is given by # # # 1 − e−adh (Y ) ## # # dY, #det # adh (Y ) # and since Relation (5.3.2) is: # # # 1 − e−adh (Y ) ## # #= #det # adh (Y ) #

# # 4 ## 1 − e−λi (Y ) ## # # # λi (Y ) # c +

i∈e ∪e

then # # 4 ## 1 − e−λi (Y ) ## f (h) dh = f (exp Y ) # dY . # # λi (Y ) # h c + 3

3 H

i∈e ∪e

Since δ(h)−1/2 dh = H (h)−1/2 G (h)1/2 dh, then the inner integral of Lemma 6.2.2 becomes

6.2 Character formula 3





ψ ◦ exp (Ad(s)Y )e

h

=

3



i(Y )

δ(exp Y )

−1/2

415

# # 4 ## 1 − e−λi (Y ) ## # dY # # λi (Y ) # c +

i∈e ∪e



ψ ◦ exp (Ad(s)Y )ei(Y ) G (exp Y )1/2 # # 4 4 ## 1 − e−λi (Y ) ## × |eλi (Y )/2 | # dY # # λi (Y ) # c + c + h

3

i∈e ∪e



i∈e ∪e

# # 4 ## eλi (Y )/2 − e−λi (Y )/2 ## # dY . # # # λi (Y ) + c



ψ ◦ exp (Ad(s)Y )ei(Y ) G (exp Y )1/2

= h

i∈e ∪e

Now for all X ∈ g put

# # 4 ## eλi (X)/2 − e−λi (X)/2 ## p(X) = G (exp X) # # # # λi (X) i∈e+ ∪ec # # 4 # 2 sinh(λi (X)/2) # #. # = G (exp X)1/2 # # λi (X) + c 1/2

(6.2.2)

i∈e ∪e

We can then write 3 3   ψ ◦ exp (Ad(s)Y )ei(Y ) p(Y )dY . ψ(shs −1 )χ (h)δ(h)−1/2 dh = H

h

 Finally put !(X) = ψ ◦ exp (X)p(X), X ∈ g. Since p(X) is an Ad(G)invariant function on g, we can write for X ∈ g,   ψ ◦ exp (Ad(s)X)p(X) = !(Ad(s)X). Hence



3

ψ(shs −1 )χ (h)δ(h)−1/2 dh =

H

3 !(Ad(s)Y )ei(Y ) dY h

and Kψ (s,s) = G (s)−1

3 !(Ad(s)Y )ei(Y ) dY . h

Next, fix a coexponential basis (X1, . . . ,Xd/2 ) for h in g and put m = span{X1, . . . ,Xd/2 }. (Recall that d is the dimension of the coadjoint orbit O of , so the codimension of the polarization h is d/2.) Identifying m with Rd/2 via the basis (Xi ), write dX for Lebesgue measure on m, and observe that dY dX is a Lebesgue measure on g. Let dm be the Lebesgue measure on m∗ where we use the dual basis (Xi∗ ). Identify m∗ with h⊥ , so that each dual basis element Xi∗ vanishes on h. Identifying h∗ with m⊥ write

416

Plancherel formula and related topics

f = |h and write g ∈  + h⊥ as g = f + m. Bring in the Euclidean Fourier transform on m, so that by Fourier inversion, 3 !(AdsY ) ei(Y ) dY h 3 3 3 1 = !(Ads(Y + X)) ei(Y ) eim(X) dXdm dY (2π )d/2 h m∗ m 3 3 3 1 = !(Ads(Y + X)) ei f (Y )+im(X) dXdY dm (2π )d/2 m∗ h m 3 3 1 ∗ = G (s) !(Y + X) eiAd (s)(f +m)(Y +X) dY dX dm (2π )d/2 m∗ g 3 1 ∗ ˆ = G (s) (s)(f + m)) dm. !(Ad (2π )d/2 h⊥ Since 0 = f + m0 and dm is translation invariant, we have 3 1 ˆ Kψ (s,s) = !(s( 0 + m)) dm. (2π )d/2 h⊥

(6.2.3)

Finally we turn to the outer integral of Lemma 6.2.2. By Lemma 3.3.10, we can write the form dμG,H as: 3 3 f dμG,H = f (exp s1 X1 . . . exp sd/2 Xd/2 ) ds. G/H

Rd/2

Using Lemma 6.1.2, the trace formula of Lemma 6.2.2 becomes 3 3 1 ˆ f + m)) dm dμG,H (s) trace (π(ψ)) = !(s( (2π )d/2 G/H h⊥ 3 1 ˆ = !() dκO (). (2π )d/2 (d/2)! O Proposition 6.2.3 Let G be an exponential Lie group with Lie algebra g, let (Zi ) be a good Jordan–Hölder basis for gc , and carry out the layering procedure and constructions of Chapter 2 for the coadjoint action. Let O be the coadjoint orbit of a point  ∈ g∗ . Let e,h,δ be the ultrafine layer containing O, and form the function p(X) as in (6.2.2). Let h be a g-polarization at  as in Proposition 4.6.11, satisfying the Pukanszky condition and let π be the irreducible representation π = Ind(G,H,χ ). Finally, let φ ∈ Cc∞ (G). Then we have the following. (1) Put ψ = φ ∗ ∗ φ. Then the function !(X) = (ψ ◦ exp)(X)p(X) has a ˆ which is nonnegative on O. Fourier transform !

6.2 Character formula

417

ˆ is integrable on O with respect to (2) π(φ ∗ ∗ φ) is trace class if and only if ! the canonical measure κO . In this case, we have the equality: 3 1 ˆ dκO, trace π(φ ∗ ∗ φ) = ! (2π )d/2 (d/2)! O where d = dim O. (3) Suppose that e,h,δ is the minimal ultrafine layer in g∗ , and let σ be the corresponding point in the cross-section for e,h,δ . Put Pfe (σ ) = Pf(σ [Ze,Ze ]e,e ∈e ) and ∗e (s) = e∈e ∗e (s). Let μσ be the G-invariant measure on O defined above. Then 3 2#e2 /2 ek ∈δ |1 + iaek | ∗ ˆ dμσ . # # ! trace(π(φ ∗ φ)) = (2π )d/2 #Pfe (σ )# O In particular the trace can be computed by 2#e2 /2 ek ∈δ |1 + iaek | ∗ # # trace π(φ ∗ φ) = (2π )d/2 #Pfe (σ )# 3 ˆ (z,σ ))|∗e (gd (z,σ )|−1 dz. !(P × Z

Example 6.2.4 We return to Examples 4.1.5 and 6.1.6. Recall the good Jordan–Hölder basis (Z,Z,Y,X,X,A) of s, and let O be the coadjoint orbit of σ = (λ,λ,0,iλx, − iλx,0). The Vergne polarization at σ is real (the index set f is empty) and the g-polarization of Proposition 4.6.11 is h = span{Re(Z),Im(Z),Y,λ1 X2 − λ2 X1 }. Here e+ ∪ ec = {1,2,3,5} and the corresponding roots λi are zero on nc = span{Z,Z,Y,X,X}, so its enough to compute the weight function p on RA. We get G (exp tA) = e4t and for i = 1,2,5, 2| sinh(λi (tA)/2)/λi (tA)| = 2| sinh(t (1 + i)/2)/t (1 + i)|. Hence # # # 2 sinh(t (1 + i)/2) #3 # , p(tA) = e2t ## # t (1 + i) and for ψ = φ ∗ ∗ φ with φ ∈ Cc (G), we get !(W + tA) = ψ(exp(W + tA))p(tA). Finally the trace of π(ψ) is given by: 3 3 1 1 ˆ ˆ (z,σ ))z−2 dz. trace(π(ψ)) = !() dκσ () = !(P 1 (2π )2 2! O (2π )2 Z ˆ (z,σ )) > 0, then trace(π(ψ)) = Note that, as in Example 6.2.1, if limz1 →0 !(P +∞. 

418

Plancherel formula and related topics

6.3 Semicharacters and the Plancherel formula In this section, we derive a generalization of the usual Plancherel theorem ˆ = g∗ : on Rn . Suppose that G = (Rn,+). Then G is identified with g and G every irreducible unitary representation π of G is a character (Example 3.1.6) π = χ (x) = ei(x) for some  ∈ (Rn )∗ . Hence, as in Example 3.1.15, the corresponding group algebra representation π : L1 (G) → C is just the evaluation of the Fourier transform at : 3 ˆ φ(x)ei(x) dx. π(φ) = φ() = G

Viewed in this way, the Plancherel theorem establishes a canonical unitary isoˆ by which the left regular morphism P : L2 (Rn ) → L2 ((Rn )∗,dμ), P : φ → φ, 7 n representation of R is equivalent with the direct integral (Rn )∗ χ dμ() and where the Plancherel measure is dμ() = (2π )−n d. Put ψ = φ ∗ ∗ φ for φ ∈ Cc (Rn ). Then the identity φ 2 = P φ 2 is equivalent to 3 1 ˆ ψ(0) = ψ() d. (2π )n (Rn )∗ ˆ The value φ() may be viewed as the trace of the “operator” π(ψ) on the Hilbert space C. Let us now look to the character formula of the preceding section to extend this analogy to exponential Lie groups. To place this in context, consider a separable locally compact group G and assume that G is type 1. As usual equip G with a left Haar measure. As observed in Section 3.2.3, since G is type 1, Theorem 3.2.32 implies that the direct product of the left and right regular representations has a unique decomposition into irreducible unitary representations. The isomorphism implementing this decomposition should be the Plancherel transform. To see how this should work, let π be an irreducible unitary representation of G, and regard π as an involutive algebra homomorphism of L1 (G) as in Section 3.1.3. We observed there that for each s ∈ G, π(Ls φ) = π(s)π(φ) and π(Rs φ) = G (s)−1 π(φ)π(s)−1 . Let L and R be the left and right regular representations of G respectively, and recall that for φ ∈ L2 (G), R(s)φ = Rs φ G (s)1/2 . Hence for φ ∈ L2 (G) ∩ L1 (G), π(L(s)φ) = π(s)π(φ) while π(R(s)φ) = π(φ)π(s)−1 G (s)−1/2 . These observations together with the preceding remarks about the Plancherel formula for Rn suggest that, if G is unimodular, then a suitable generalization of the Fourier transform of a function φ ∈ L2 (G) ∩ L1 (G) should be a map ˆ In the of the form φˆ : σ → πσ (φ), for σ in a suitably large subset  of G. 1/2 nonunimodular case, we should look for a map of the form σ → πσ (φ)Dσ

6.3 Semicharacters and the Plancherel formula

419

where Dσ is a (generally unbounded) positive operator on the Hilbert space of πσ such that πσ (s)Dσ = Dσ πσ (s)(s)−1 . A. Kleppner and R. Lipsman describe the measure class for this decomposition in terms of the Plancherel measure class of a closed normal subgroup [54, 55]. Results there are then used by M. Duflo and C. Moore in [32] to describe the Plancherel transform in the terms indicated above. Specifically, ˆ a μ-conull Borel subset  of G, ˆ and meathere is a measure μ on G, surable fields {πσ }σ ∈ and {Dσ }σ ∈ , where πσ ∈ σ , and Dσ is a positive 1/2 1/2 semi-invariant operator on Hπσ , such that the closure [Dσ πσ (φ)Dσ ] of 1/2 1/2 Dσ πσ (φ)Dσ is trace class for μ-almost every σ ∈ , and 3   trace [Dσ1/2 πσ (φ)Dσ1/2 ] dμ(σ ) φ(e) = 

holds for any smooth, compactly supported function φ on G. Once the Haar measure on G is fixed, the product {Dσ dμ(σ )}σ is canonical, and it is this product that may be called the Plancherel measure for G. The the Duflo–Moore operators, and the maps operators Dσ are often called 1/2 1/2  φ → trace [Dσ πσ (φ)Dσ ] may be called semicharacters. Now suppose that G is exponential. Fix a good Jordan–Hölder basis (Zi ) for gc , with dual basis (fi ) and carry out the layering procedure and constructions for the cross-sections and parametrizations P (z,σ ) as in Chapter 2, for the coadjoint action of G (and g) in g∗ . Recall that for G exponential, the coadjoint orbits are in canonical one-to-one correspondence with the elements ˆ Thus ˆ that is, we have a canonical bijection K : g∗ /G → G. of the dual G, ˆ is naturally parametrized by the union of the layer-wise cross-sections conG structed in the layering procedure. When considering the Plancherel transform, it might be expected that the cross-section  for the minimal Zariski open layer ˆ e,h,δ , would be a suitable domain for the e,h,δ , and the corresponding subset G ˆ First we construct the Duflo–Moore operators, and Plancherel transform on G. prove an orbital semicharacter formula.

6.3.1 Semicharacter formula for generic orbits Let G be an exponential Lie group with Lie algebra g, and as usual, put s = gc , and let n be a nilpotent ideal of g containing [g,g]. Let (sj ) be a good Jordan– Hölder sequence for s, such that sp = nc for some p and sn−1 ⊂ ker dG . ∗ Let (λi ) be the corresponding root sequence. With Ui = s⊥ i ⊂ s , we have a ∗ good descending flag of submodules in U = s for the coadjoint action. When an associated good Jordan–Hölder basis is fixed, then the dual basis is a good flag basis for the flag (Ui ). Accordingly, the weight sequence for the coadjoint

420

Plancherel formula and related topics

action is (λ∗i ) where λ∗i = −λi , 1 ≤ i ≤ n. Carrying out the constructions of Chapter 2, let  = e,h,δ be the minimal ultrafine layer for the coadjoint action, with cross-section . Fix σ ∈ , construct the g-polarization h at σ as in Proposition 4.6.11, and let H = exp h. Let π be an irreducible unitary representation of G corresponding to the orbit of σ . We have seen that it is not necessarily true that the operator π(φ) is trace class for all φ ∈ Cc∞ (G). However, one has more success when considering operators of the form π(φ)Aσ , where Aσ is a positive semi-invariant, densely defined operator. This is not surprising in light of our remarks about the Plancherel formula. In [73], as well as in subsequent work, N. V. Pedersen developed a formal notion of semicharacter; here we informally say that a semicharacter formula is an orbital trace formula associated to an equivalence class of irreducible unitary representations wherein a positive semi-invariant, densely defined operator is used. By Lemma 4.6.12, H ⊂ ker G , so for every function f ∈ Cc (G,H,χσ ), the function Dσ f defined by (Dσ f )s) = G (s)f (s) is also in Cc (G,H,χσ ), and thus we have a linear map Dσ : Cc (G,H,χσ ) → Cc (G,H,χσ ) defined by Dσ = G f . If G is nontrivial then Dσ extends to an unbounded, denselydefined operator in H(G,H,χσ ). Let φ ∈ Cc∞ (G), with ψ = φ ∗ ∗ φ. Recall that the operator πσ (ψ) is an integral operator with kernel: 3 −1 ψ(sht −1 )χσ (h)δ(h)−1/2 dh. Kψ (s,t) = G (t) H 1/2

1/2

If we apply π(ψ) to Dσ f , then apply Dσ , again, we still get an integral operator defined by 3   1/2 1/2 Kψ (s,t) f (t) dμG,H (t), Dσ πσ (ψ)Dσ f (s) = G/H

where: Kψ (s,t) = G (s)1/2 G (t)−1/2

3

ψ(sht −1 )χσ (h)δ(h)−1/2 dh. H

Hence 3 Kψ (s,s) =

ψ(shs −1 )χσ (h)δ(h)−1/2 dh = G (s)Kψ (s,s).

H

We have seen above that Kψ (s,s) ≥ 0 and for h ∈ H , Kψ (sh,sh) = δ(h)Kψ (s,s). The preceding together with the fact that H ⊂ ker G shows that these

6.3 Semicharacters and the Plancherel formula

421

properties also hold for Kψ : Kψ (s,s) ≥ 0 and Kψ (sh,sh) = δ(h)Kψ (s,s), h ∈ H . By the generalized Mercer theorem, we conclude the following. 7 1. Suppose that G/H Kψ (s,s) dμG,H (s) is finite. Then the integral operator 7 Tψ defined by Tψ f = G/H Kψ (s,t)f (t)dμG,H (t) is a bounded, trace-class operator on H(G,H,χσ ). Moreover, Tψ agrees with 1/2

1/2

Dσ πσ (ψ)Dσ on the dense subspace Cc (G,H,χσ ). Hence the operator 1/2 1/2 1/2 1/2 Dσ πσ (ψ)Dσ is closeable, and its closure [Dσ πσ (ψ)Dσ ] is Tψ . Finally, 3 trace [Dσ1/2 πσ (ψ)Dσ1/2 ] = Kψ (s,s) dμG,H (s). (6.3.1) G/H

1/2

1/2

2. Suppose that Dσ πσ (ψ)Dσ is closeable, and that its closure 1/2 1/2 [Dσ πσ (ψ)Dσ ] is a bounded trace-class operator. Then 1/2 1/2 [Dσ πσ (ψ)Dσ ] is7an integral operator on H(G,H,χσ ) with kernel Kψ (s,t), and hence G/H Kψ (s,s) dμG,H (s) is finite and (6.3.1) holds. Recall Relation (6.2.3), with 0 = σ : 3 1 ˆ · (σ + m)) dm. Kψ (s,s) = !(s (2π )d/2 h⊥ Replacing Kψ by Kψ , we get: 3 Kψ (s,s) dμG,H (s) G/H 3 3 1 ˆ · (σ + m))G (s) dm dμG,H (s) = !(s (2π )d/2 G/H h⊥ 3 3 1 ˆ = !(exp tX · (σ + m))G (exp tX) dmdt. (2π )d/2 Rd/2 h⊥ Bring in formula (6.1.2) and Proposition 6.1.4; we conclude that for any positive Borel function F on O, 3 3 3 2|e2 |/2 ek ∈δ |1+iaek | F (exp tX(σ + m)) dmdt = F ()dμσ (). |Pfe (σ )| Rd/2 m∗ O Now apply the preceding with F defined on O by F (exp tX(σ + m)) = ˆ !(exp tX(σ + m))G (exp tX), for  = exp tX(σ + m), recall the unique element s() ∈ S(σ ) such that  = s()σ . Then for some h ∈ H , σ = s()−1 exp tXhσ,

422

Plancherel formula and related topics

and, since G() ⊂ H ⊂ ker G , then G (s()) = G (exp tX). Hence, ˆ F (exp tX(σ + m)) = F (s()σ ) = !(s()σ )G (s()) and, using the orbital measure ωσ of Lemma 2.3.14, 3 Kψ (s,s) dμG,H (s) G/H 3 3 1 = F (exp tX(σ + m))dmdt (2π )d/2 Rd/2 m∗ 3 2#e2 /2 ek ∈δ |1 + iaek | # # = F () dμσ () (2π )d/2 #Pfe (σ )# O 3 2#e2 /2 ek ∈δ |1 + iaek | ˆ # # = !(s()σ )()G (s())G (s())−1 dωσ () (2π )d/2 #Pfe (σ )# O 3 2#e2 /2 ek ∈δ |1 + iaek | ˆ # # = !()()dω σ (). (2π )d/2 #Pfe (σ )# O Taking a covering set O containing σ , we may also compute 3 Kψ (s,s) dμG,H (s) G/H

=

-

ek ∈δ |1 + iaek | # # d/2 (2π ) #Pfe (σ )#

2#e2 /2

3 Z

ˆ (z,σ )) !(P

4

|∗i (g(z,σ ))|dz.

i ∈e /

This gives the semicharacter formula: Proposition 6.3.1 Let G be an exponential Lie group with Lie algebra g, let (Zi ) be a good Jordan–Hölder basis for gc , let e,h,δ be the minimal layer for the coadjoint action in g∗ , and let Pfe (σ ) = Pf(σ [Ze,Ze ]e,e ∈e ). Fix σ ∈ , and let h be a g-polarization at σ satisfying the Pukanszky condition, and such that H = exp h ⊂ ker G . Then 3 2#e2 /2 ek ∈δ |1 + iaek | 1/2 1/2 ˆ # # trace [Dσ πσ (ψ)Dσ ] = !() dωσ (), (2π )d/2 #Pfe (σ )# O where !(X) = ψ(exp X)p(X), and dωσ is the orbital measure of Lemma 2.3.14. Example 6.3.2 We return to Example 6.2.1, where the two-dimensional affine group is realized as R  R, and the Lie algebra is g = span{Y,A} with [A,Y ] = Y . With (Y,A) as the good Jordan–Hölder basis, the minimal layer

6.3 Semicharacters and the Plancherel formula

423

is  = e,h,δ with e = {1,2},h = (2,1), δ = {1}, and the covering is trivial: O = . Write an element of g∗ using the dual basis as  = γ Y ∗ + αA∗ = (γ ,α). The cross-section for  is  = {(±1,0)} and the parametrization of Proposition 2.5.8 is P (z,(±1,0)) = (±z1,z2 ), where z1 > 0,z2 ∈ R. One computes that g(z,σ ) = exp(− ln z1 A) exp −z2 Y and ∗ (g(z,σ )) = z1−1 . A natural polarization at σ = (ε,0) satisfying the Pukanszky condition is h = RY , so H = {(y,0) : y ∈ R}, and the irreducible unitary representation πσ is realized in L2 (R) by πσ (y,t)f (s) = eiεe

−s y

f (s − t).

Here G ((y,t)) = e−t , and Dσ f (s) = e−s f (s). We compute p(yY + tA) = e−t/2

2 sinh(t/2) 1 − e−t = . t t

Now e2 = ∅, |1 + ia1 | = 1, and |Pfe (σ )| = |ε| = 1, so 3 ∞3 1 1 ˆ !(εz trace(πσ (φ ∗ ∗ φ)) = 1,z2 ) dz2 dz1 2π 0 z1 R with !(yY +tA) = (φ ∗ ∗φ)(exp(yY +tA))(1−e−t )/t. Recall that in Example 6.2.1, we saw that if ∧y !(0,0) = 0, the above integral is infinite. Bring in the Duflo–Moore operators Dσ and we get 3 ∞3   1 ˆ trace [Dσ1/2 πσ (φ ∗ ∗ φ)Dσ1/2 ] = !(εz 1,z2 )dz2 dz1 . 2π 0 R Thus we see that [Dσ πσ (φ ∗ ∗ φ)Dσ )] is trace class for all φ ∈ Cc∞ (G), and  in fact the semi-invariant measure ωσ defines a tempered distribution. 1/2

1/2

Example 6.3.3 Returning to Examples 4.1.5, 6.1.6, and 6.2.4. Here ec = {2,5} ∗ 2 and so i ∈e / |i (g(z,σ ))| = z1 . The irreducible unitary representation πσ is 2 2 realized in L (R ) so that Dσ f (s1,s2 ) = e4s2 f (s1,s2 ). With !(X) = (φ ∗ ∗ φ)(exp X)p(X) as before, we get 3 1 ˆ trace [Dσ1/2 πσ (φ ∗ ∗ φ)Dσ1/2 ] = !() dωσ () (2π )2 O 3 1 ˆ (z,σ ))z12 dz. !(P = (2π )2 Z Recall the function P (z,σ ): P (z,σ ) = (λz11+i ,λz11−i ,z2,λz1i (z3 + ixz1 ),λz1−i (z3 − ixz1 ),z4 ).

424

Plancherel formula and related topics

We choose the following real coordinates for  ∈ g∗ : (v1 v2,v3,v4,v5,v6 ) = (Re(1 ),Im(1 ),3,Re(4 ),Im(4 ),6 ). Observe that z12 = Re(1 )2 + Im(1 )2 = v12 + v22 , z2 = 3 = v3 , z4 = 6 = v6 , and z3 = Re(λz1−i (Re(4 ) + iIm(4 )), thus: z32 ≤ |λz1−i (Re(4 ) + iIm(4 ))|2 = Re(4 )2 + Im(4 )2 = v42 + v52 . ˆ By construction, the function !() is the Fourier transform of a smooth compactly supported function, thus it is a Schwartz function in the v-variables. Thus for any k, there is C such that ˆ 1, . . . ,v6 )|(v12 + v22 )(1 + v12 + v22 + v32 + v42 + v52 + v62 )k < C. sup |!(v

v∈R6

This implies sup z∈(0,∞)×R3

ˆ (z,σ ))|z12 (1 + z12 + z22 + z32 + z42 )k < C. |!(P

Therefore the integral converges for any σ in the cross-section  of .



Example 6.3.4 Consider now the nonexponential group E˜ 2 . We saw in Example 5.3.1 that, to the generic orbit of σ = rX∗ (see the notation in Example 5.3.1), is associated the family of irreducible unitary representations (πr,ν )0≤ν