Geometric and Harmonic Analysis on Homogeneous Spaces -- TJC 2017, Mahdia, Tunisia, December 17–21 [1 ed.] 978-3-030-26561-8

Table of contents :
Preface......Page 6
Majdi Ben Halima a Brilliant Mathematician Turned off at Early Age......Page 8
Homage to Majdi Ben Halima 1979–2016......Page 11
Contents......Page 12
About the Editors......Page 13
1 Position of the Problems......Page 14
2.1 The General Context......Page 16
2.2 Monomial Representations......Page 17
2.3 Case of a Normal Subgroup of Codimension 1......Page 18
2.5 An Algebra of G-Invariant Differential Operators......Page 19
3 Monomial Representations of Discrete Type......Page 20
4.1 On Some Relative Indices......Page 29
4.2 An Algorithm of Construction of Co-exponential Bases......Page 30
4.3 A Trace Formula Starting From the Root System......Page 36
5 A Proof of Convergence......Page 39
6 Concrete Plancherel Formula......Page 46
6.1 Computing the Matrix Coefficients......Page 47
6.2 An Intertwining Formula......Page 48
6.3 Proof of Theorem 3......Page 51
7 Invariant Differential Operators......Page 60
8 Polarizations......Page 65
References......Page 67
1 Introduction......Page 69
2.1 Topologically Finitely Generated Groups......Page 71
2.2 Frattini Subgroups......Page 72
3 The Class of Self-Chabauty Isolated Groups......Page 74
4 A Necessary and Sufficient Condition for Connected Locally Compact Groups......Page 77
5 Locally Compact Totally Disconnected [SIN]-Groups......Page 79
References......Page 81
Quantization of Color Lie Bialgebras......Page 83
1.1 Graded and Color Vector Spaces......Page 84
1.2 Color Lie Algebras......Page 85
1.3 Color Lie Bialgebras......Page 87
1.4 Topologically Free Modules......Page 88
1.5 Quasitriangular Color Quasi-Hopf Algebras......Page 89
1.6 Quantized Universal Enveloping Algebras......Page 90
2.2 Drinfeld Category......Page 91
3 Quantization of Color Lie Algebras......Page 92
3.1 Quantization of mathfrakg+ and mathfrakg-......Page 97
4 Quantization of Triangular Color Lie Bialgebras......Page 98
5.1 Topological Spaces......Page 99
5.2 Manin Triples......Page 100
5.3 Equicontinuous mathfrakg-Modules......Page 101
5.4 Tensor Functor F......Page 102
5.6 The Algebra Uh(mathfrakg+)......Page 103
6 Simple Color Lie Bialgebras of Cartan Type......Page 104
References......Page 106
1 Introduction......Page 107
2.1 Frobenius Lie Algebras......Page 109
2.2 Orbit Method......Page 110
2.3 Square-Integrable Representation......Page 111
2.4 Analysis on a Semi-direct Product Group......Page 112
3.1 The First Case......Page 114
3.2 The Second Case......Page 117
3.3 The Third Case......Page 118
References......Page 120
1 Introduction......Page 122
2 A Holomorphically Induced Representation of Boidol's Group......Page 125
2.2 Case of mathcalH(mathfrakh, F, 0)neq{0}......Page 128
References......Page 131
1 Introduction......Page 132
2 Detailed Description of the Main Result......Page 136
2.4 The Case mathfrakg=mathfrakso(2r,1) (rge2)......Page 137
2.6 The Hermitian Type......Page 138
2.8 Split Lie Groups with Simply-Laced......Page 139
3.1 The Chevalley Restriction Theorem......Page 140
3.2 The Algebra of Invariant Differential Operators......Page 141
3.3 Joint Eigenfunctions and Integral Formulas......Page 142
3.4 The Multiplicity Function κπ......Page 144
3.5 The Radial Part of the Casimir Operator......Page 145
3.6 Radial Parts of General Invariant Differential Operators......Page 146
4 Heckman–Opdam Hypergeometric Functions......Page 148
4.1 Hypergeometric Differential Operators......Page 149
4.2 Definition of the Hypergeometric Functions......Page 150
4.3 Regularity Conditions on k......Page 151
5.1 Coincidence of Differential Operators......Page 155
5.2 The Associated Split Semisimple Subgroup......Page 158
5.3 Simplifying Matching Conditions......Page 159
6 Case-by-Case Analysis......Page 160
6.2 Complex Simple Lie Groups......Page 161
6.4 The Case mathfrakg=mathfraksp(p,q)......Page 162
6.5 The Case mathfrakg=mathfrakso(p,q)......Page 164
6.6 The Hermitian Type......Page 168
6.7 The Case is of Type F4......Page 170
6.8 Split Lie Groups with Simply-Laced......Page 171
7 Spherical Transforms......Page 172
7.2 Harish-Chandra's c-Function......Page 173
7.3 The π-Spherical Transform......Page 174
7.4 Inversion Formulas and Plancherel Formulas......Page 176
References......Page 178
1 Introduction......Page 180
2.1 Realization of Exceptional Lie Group of Type G2......Page 183
2.2 Realization of Spinor Group Spin(7,mathbbC)......Page 184
2.4 Unit Sphere and Complex Unit Sphere......Page 185
2.5 Transitive Actions on Unit Spheres......Page 186
3.1 G2-Action on S(Im(mathfrakCmathbbC))......Page 187
3.2 G2-Action on G2(mathbbC)/SL(3,mathbbC)......Page 188
3.3 Lie Algebra mathfraka1......Page 190
4.1 Spin(7)-Action on S(mathfrakCmathbbC)......Page 191
4.2 Spin(7)-Action on Spin(7,mathbbC)/G2(mathbbC)......Page 192
4.3 Lie Algebra of widetildeA0......Page 193
5 Cartan Decomposition for (SO(7,mathbbC),G2(mathbbC))......Page 194
6.1 Verification of (V.1)......Page 196
6.2 Verification of (S.1)......Page 197
6.5 Remark......Page 199
References......Page 200
1 Introduction......Page 202
2 Multisymplectic Vector Spaces and Manifolds, and Lagrangian Submanifolds......Page 204
3 Standard Multisymplectic Vector Spaces and Manifolds......Page 207
4 Normal Forms of Lagrangian Submanifolds of Standard Multisymplectic Manifolds......Page 211
References......Page 216
1 Introduction......Page 217
2 First Approach to Conjecture 1......Page 219
3 The Nilpotent Case......Page 221
4.2 The ``Saturated'' Case Unsolved......Page 222
4.3 The Normal Polarization Case......Page 224
4.4 Example......Page 225
5 Dixmier and Kirillov Maps......Page 226
References......Page 227

Citation preview

Springer Proceedings in Mathematics & Statistics

Ali Baklouti Takaaki Nomura Editors

Geometric and Harmonic Analysis on Homogeneous Spaces TJC 2017, Mahdia, Tunisia, December 17–21

Springer Proceedings in Mathematics & Statistics Volume 290

Springer Proceedings in Mathematics & Statistics This book series features volumes composed of selected contributions from workshops and conferences in all areas of current research in mathematics and statistics, including operation research and optimization. In addition to an overall evaluation of the interest, scientific quality, and timeliness of each proposal at the hands of the publisher, individual contributions are all refereed to the high quality standards of leading journals in the field. Thus, this series provides the research community with well-edited, authoritative reports on developments in the most exciting areas of mathematical and statistical research today.

More information about this series at http://www.springer.com/series/10533

Ali Baklouti Takaaki Nomura •

Editors

Geometric and Harmonic Analysis on Homogeneous Spaces TJC 2017, Mahdia, Tunisia, December 17–21

123

Editors Ali Baklouti Department of Mathematics, Faculty of Sciences at Sfax Sfax University Sfax, Tunisia

Takaaki Nomura Faculty of Mathematics Kyushu University Fukuoka, Japan

ISSN 2194-1009 ISSN 2194-1017 (electronic) Springer Proceedings in Mathematics & Statistics ISBN 978-3-030-26561-8 ISBN 978-3-030-26562-5 (eBook) https://doi.org/10.1007/978-3-030-26562-5 Mathematics Subject Classification (2010): 22D05, 54B20, 22E40, 22E25, 22E27, 22E45, 33C67, 43A90, 53D05, 53D12, 53C12, 15A04, 17A99, 17D99, 22E46, 32M05, 22E60, 14M17, 81S10 © Springer Nature Switzerland AG 2019 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Preface

This book presents a series of important contributions focusing on harmonic analysis and representation theory of Lie groups. All were originally presented at the 5th Tunisian–Japanese conference entitled “Geometric and Harmonic Analysis on Homogeneous Spaces and Applications”, which was held at Mahdia in Tunisia from December 17 to 21, 2017 and was dedicated to the memory of the brilliant Tunisian mathematician Majdi Ben Halima, who passed away at such a young age and contributed so significantly to this series of Tunisian–Japanese conferences. We deeply regret this tragic loss to our community. All of the contributions presented will be of high interest to harmonic analysts worldwide. Examples of the topics covered include: harmonic analysis for four-dimensional real Frobenius Lie algebras; quantization of color Lie bialgebras; self-Chabauty-isolated locally compact groups; Lagrangian submanifolds of standard multisymplectic manifolds; spherical functions for small K-types; and the Poisson characteristic variety of unitary irreducible representations of exponential Lie groups. Two of the contributions are devoted to the scientific life of Majdi Ben Halima, who was passionate about finding new problems and developing bridges between ideas and theories. The contributions selected for publication were subjected to a peer review process by specialists and underwent modifications as required by the referees. They are without exception of a high standard, equivalent to that in the first-class mathematical periodicals. The conference attracted 86 participants, and 29 talks were given by the foremost researchers from both Tunisia and Japan as well as from several other countries, including France. Lively mathematical discussions were held during and after the talks, and new joint works by some of the participants are currently in progress. The details of the conference are available at the following web page: https://www2.math.kyushu-u.ac.jp/*tnomura/Mahdia/. This series of Tunisian-Japanese conferences started with the conclusion of an agreement in June 2007 regarding academic cooperation between the Faculty of Science of Sfax University in Tunisia and the Faculty of Mathematics at Kyushu University in Japan. The agreement between the faculties was extended for 5 years v

vi

Preface

in 2012, with new participation of the Institute of Mathematics for Industry, Kyushu University, and then extended once again in 2017 for a further 5 years. Previous conferences in the series were held at Kerkennah Islands, Sousse, Hammamet, and Monastir in 2009, 2011, 2013, and 2015, respectively. The 6th conference is already planned for Djerba in December 2019. All of our conferences are supported by Grants-in-Aid for Scientific Research in Japan (JSPS), the Tunisian Mathematical Society, the Mediterranean Institute for the Mathematical Sciences (MIMS), the Ministry of Higher Education, Scientific Research and Technology in Tunisia, the Faculty of Sciences of Sfax University, and École Doctorale, Sciences Fondamentales, Sfax University. It is our pleasant duty to express our most sincere gratitude to these organizations for continuously supporting our activity. We are always thinking about how to involve younger generations to promote continuation of this academic exchange. Sfax, Tunisia Fukuoka, Japan

Ali Baklouti Takaaki Nomura

Majdi Ben Halima a Brilliant Mathematician Turned off at Early Age

It is a tremendous honor to write this tribute to Majdi BEN HALIMA, an intimate Friend, Colleague, Teacher, Scholar, and a Researcher. In whatever role I knew him, from whatever vantage point, he stood apart as someone special! Majdi was committed to research focused on harmonic analysis and representation theory of Lie groups. He was passionate about finding new problems and enabling bridges between ideas and theories. In addition to his devotion to his work and to the improvement of research, he always found time for his colleagues, his friends, and his laboratory companions. Majdi Ben Halima was born on June 24, 1979 in the city of Sfax. He achieved his undergraduate studies in Mathematics at the Faculty of Sciences of Sfax in 2002 with excellent grades. He was amended the first prize for graduating with excellence from the Ministry of Higher education, Scientific Research and Technology in Tunisia. He then moved to the University of Paul-Verlaine-Metz in France, where he obtained his Master degree in Mathematics in 2003, after he got a Tunisian Government scholarship for Master and doctoral studies in France. In 2006, he defended his Ph.D. thesis entitled: “Invariant differential operators on homogeneous

vii

viii

Majdi Ben Halima a Brilliant Mathematician Turned off at Early Age

spaces: Branching rules and applications” under the supervision of Prof. Tilmann Wurzbacher. Right after, he pursued his researches on similar problems, which include: • Branching rules for compact Lie groups and spectra of invariant differential operators • Construction of fuzzy homogeneous spaces and applications • Orbit method for certain classical Lie groups and Lie subgroups. In 2013 Majdi defended his Habilitation thesis at the Faculty of Sciences of Sfax, after having published several important articles. The list of his publications includes: 1. M. Ben Halima, Branching rules for unitary groups and spectra of invariant differential operators on complex Grassmannians, J. Algebra., 318 (2007), 520– 552. 2. M. Ben Halima, Spectrum of the Hodge Laplacian on complex Grassmannian Gr2 ðCm þ 2 Þ; Bull. Sci. Math., 132 (2008), 19–36. 3. M. Ben Halima, Spectrum of twisted Dirac operators on the complex projective space P2q þ 1 ðCÞ; Comment. Math. Univ. Carolin., 49 (2008), 437–445. 4. M. Ben Halima and T. Wurzbacher, Fuzzy complex Grassmannians and quantization of line bundles, Abh. Math. Semin. Univ. Hambg., 80 (2010), No. 1, 59–70. 5. M. Ben Halima, Construction of certain fuzzy flag manifolds, Rev. Math. Phys., 5 (2010), 533–548. 6. M. Ben Halima, Generalized Littlewood-Richardson rule and sum of coadjoint orbits of compact Lie groups, Bull. Sci. Math., 135 (2011), 345–352. 7. M. Ben Halima and A. Rahali, On the dual topology of a class of Cartan motion groups, J. Lie Theory., 22 (2012), No. 2, 491–503. 8. M. Ben Halima and A. Rahali, Dual topology of the Heisenberg motion groups, Indian J. Pure. App. Math., 45 (2014), 513–530. 9. M. Ben Halima, Coadjoint orbits of certain motion groups and their coherent states, J. Nonlinear Math. Phys., 20 (2013), 420–430. 10. M. Ben Halima and A. Rahali, Separation of unitary representations of Euclidien motion groups, Not Mat., 35 (2015), 15–22. 11. M. Ben Halima and Massaoud Anis, Corwin-Greenleaf multiplicity function for compact extensions of Rn : Int. J. Math., 26 (2015), No. 10. Majdi was an active member of our Laboratory. He regularly attended our seminars and also participated in several international events. He was an inspiring figure of our department, his serious, deep thinking, and pretty quiet character made of him unanimously a best friend for the whole staff. He very often came to my office and we talked together about teaching, research, family, society problems, and even some intimate issues. For what concerns mathematics, he was the brilliant theorist who made enduring contributions and inspired many researchers to pursue

Majdi Ben Halima a Brilliant Mathematician Turned off at Early Age

ix

his alluring way. We did engage many regular discussions together about some problems related to visible actions on complex solvable homogeneous spaces and many preliminary results were obtained. Following a first suggestion of mine, we co-supervised together the Ph.D. thesis of Ayman Rahali, who is at present an Assistant Professor at the University of Kairouan. The defense ceremony turned on during the third Tunisian-Japanese conference on Geometric and Harmonic Analysis on Homogeneous spaces and Applications TJC3, hosted in Hammamet in 2013. The papers [7, 8, 10] follow from a fruitful collaboration, in which Majdi was the most perseverant engine. He then started the supervision of the Ph.D. thesis of the researcher Anis Massaoud, acting now as “Professeur-Agrégé” at the University of Gafsa and they published together the article [11] above. Majdi passed away on February 04, 2016, before the accomplishment of the project, after he fell sick from a severe disease. As Anis expressed his great enthusiasm, I was offered the immense honor to carry on the rest of this work. The publication [11] was as a starting milestone of a new issue in the theory of branching rules of unitary representations, and left behind many open problems, tackled by Anis in the next chapters of the thesis. We will all have our own personal and proper memories, of the legacy he left in our hearts and our lives, and it is very hard for me today to be up here, imagining his alluring behavior, trying my best to focus on the glad memories Majdi brought to us, rather than the fact that he is no longer here with us today. I do offer my heartfelt respects upon the passing of the talented scientist, to his grieving family, all harmonic analysts worldwide and the scientific community. Ali Baklouti Sfax, Tunisia [email protected]

Homage to Majdi Ben Halima 1979–2016

As I understand, the name Majdi signifies glory, pride, grandeur, immortality.

In retrospective these words are a perfectly fitting description of the life of the shy and polite, young man who entered my office in Metz in France in November 2002. Despite his late arrival to the last year of courses of the Master of Mathematics cycle, he quickly showed his great capacities and wrote an excellent Master thesis on index theory of invariant differential operators under my supervision in 2003. We continued our common path with the work on his Ph.D. on branching rules for linear representations of compact Lie groups with applications to geometric differential operators. His thesis was brilliant, concise, clear, and full of new insights, and it taught me a lot on the difficult subject of branching rules. His thesis defense took place in June 2009 and he then went back to his beloved hometown Sfax in Tunisia, where he occupied from September 2009 until his untimely death the position of an assistant professor in Mathematics and started a never failing professional career. Let me mention that he always took greatest care of his teaching, as well as of the written and oral presentations of his research. He received his Habilitation degree in Sfax in May 2013 during a ceremony I had the good fortune to be present. But not only his work went on with great success, he also founded a happy family with his wife Afef and their two children Yacine and Emna. I still relive the shock I got when I remember how he told me that he had to cope with a very serious disease. He never gave up fighting for his life, giving us an unforgettable image of strength in the hope to continue to be with his family and based on his faith. Majdi, we will not forget you! December 2017

Tilmann Wurzbacher

xi

Contents

Monomial Representations of Discrete Type of an Exponential Solvable Lie Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ali Baklouti, Hidenori Fujiwara and Jean Ludwig

1

Self-Chabauty-isolated Locally Compact Groups . . . . . . . . . . . . . . . . . . Hatem Hamrouni and Firas Sadki

57

Quantization of Color Lie Bialgebras . . . . . . . . . . . . . . . . . . . . . . . . . . . Benedikt Hurle and Abdenacer Makhlouf

71

Harmonic Analysis for 4-Dimensional Real Frobenius Lie Algebras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Edi Kurniadi and Hideyuki Ishi

95

An Example of Holomorphically Induced Representations of Exponential Solvable Lie Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Junko Inoue Spherical Functions for Small K-Types . . . . . . . . . . . . . . . . . . . . . . . . . 121 Hiroshi Oda and Nobukazu Shimeno A Cartan Decomposition for Non-symmetric Reductive Spherical Pairs of Rank-One Type and Its Application to Visible Actions . . . . . . . 169 Atsumu Sasaki Lagrangian Submanifolds of Standard Multisymplectic Manifolds . . . . 191 Gabriel Sevestre and Tilmann Wurzbacher The Poisson Characteristic Variety of Unitary Irreducible Representations of Exponential Lie Groups . . . . . . . . . . . . . . . . . . . . . . 207 Ali Baklouti, Sami Dhieb and Dominique Manchon

xiii

About the Editors

Ali Baklouti is Full Professor of Mathematics at the University of Sfax, Tunisia. He received his Ph.D. in Mathematics from the University of Metz (France) in 1995. He was elected as the President of the Tunisian Mathematical Society for two consecutive terms (April 2016–March 2019 and April 2019–March 2022), and he has been the Deputy Director of the Mediterranean Institute of Mathematical Sciences since January 2012. He was also nominated as a member of the “Tunisian Academy of Sciences, Letters and Arts: Beit Elhikma” in December 2016. He has published over 70 papers in peer-reviewed national and international journals and proceedings, as well as a number of book chapters. He is Co-Editor-in-Chief of the Tunisian Journal of Mathematics and an editorial board member for a number of well-known journals. He has delivered several invited talks at national and international conferences. Takaaki Nomura was a Full Professor of Mathematics at Kyushu University in Japan, and has been a Professor Emeritus since April 2019. He is currently affiliated with Osaka City University, Advanced Mathematical Institute. He has published about 30 original papers in peer-reviewed national and international journals, as well as several survey papers and proceeding articles written both in English and in Japanese. He has written three mathematical books in Japanese, and delivered several invited talks at national and international conferences.

xv

Monomial Representations of Discrete Type of an Exponential Solvable Lie Group Ali Baklouti, Hidenori Fujiwara and Jean Ludwig

Abstract Let G be an exponential solvable Lie group, H an analytic subgroup of G and χ a unitary character of H . We study some problems related to the induced representation τ = indGH χ of G when τ has multiplicities either finite or infinite of discrete type. In particular, we are interested in the Plancherel formula for τ and the commutativity problem due to Duflo (Open problems in representation theory of Lie groups, edited by T. Oshima, Katata, Japan 1986, [12]) for the algebra Dτ (G/H ) of G-invariant differential operators on the fiber space associated to the data (H, χ ) over the base space G/H . We give in particular an example where this problem can admit a negative solution in the frame of exponential solvable Lie groups. Keywords Orbit method · Irreducible representations · Penney distribution · Plancherel formula · Differential operator 1991 Mathematics Subject Classication 22E27

1 Position of the Problems This work is a continuation of the previous works [14, 16] as a first objective, [8, 17] as a second. It matters to invest more in the harmonic analysis on the solvable homogeneous spaces through monomial representations (which are induced from This work has been partially supported by the JSPS through the subvention No 20540194 and the DGRSRT through the Research Lab: LR 11E S52. A. Baklouti (B) Département de Mathématiques, Faculté des Sciences de Sfax, Route de Soukra, 3038 Sfax, Tunisie e-mail: [email protected] H. Fujiwara Faculty of Science and Technology for Humanity, Kinki University, Iizuka 820-8555, Japan e-mail: [email protected] J. Ludwig Laboratoire LMAM, UMR 7122, Université de Metz, Ile du Saulcy, 57045 Metz Cedex 01, France e-mail: [email protected] © Springer Nature Switzerland AG 2019 A. Baklouti and T. Nomura (eds.), Geometric and Harmonic Analysis on Homogeneous Spaces, Springer Proceedings in Mathematics & Statistics 290, https://doi.org/10.1007/978-3-030-26562-5_1

1

2

A. Baklouti et al.

unitary characters of analytic subgroups). The orbit method due to Kirillov turns out to be a crucial ingredient for our study. The frame of our work is limited to the connected and simply connected exponential solvable Lie groups G = exp(g) with Lie algebra g. Let H = exp h be a closed connected subgroup of G and χ = χ f , f ∈ g∗ a unitary character of H . We consider the monomial representation τ = indGH χ of G which is irreducible if and only if the subalgebra h is a polarization at f of g which verifies the Pukanszky condition. We disintegrate this representation into irreducible components  τ

⊕ Gˆ

m(π )π dμ(π )

(1)

for a certain measure μ on the unitary dual Gˆ of G and a multiplicity function m(π ) on Gˆ with values in N ∪ {+∞} (cf. [2] for concrete disintegration in the nilpotent case). Then beyond the completely solvable case, we ignore until now how to characterize the finiteness of these multiplicities μ−almost everywhere, except certain particular cases (cf. [4]). When G is nilpotent, it is well known that the multiplicity function is either finite and uniformly bounded or uniformly equal to the ∗ infinity μ-almost everywhere on the affine space  = f + h⊥,g . In the first case, the Plancherel formula due to Penney [21] is well described as follows. The measure μ can be chosen in such a manner that for any π in the spectrum of τ , there exist some distributions aπk (1 ≤ k ≤ m(π )) such that for all ϕ ∈ Cc∞ (G), f

ϕ H (e) =

 m(π)  Gˆ k=1

π(ϕ)aπk , aπk dμ(π ),

(2)



where f ϕ H (g)

=

ϕ(gh)χ f (h)dh (g ∈ G) H

with a Haar measure dh on H . Moreover, each distribution is explicitly written as an integral of a certain translated of ϕ on a homogeneous space coming from the context of the problem. Beyond the nilpotent case and except certain particular cases (cf. [16]), such a formula would be very useful in many related problems. The first objective of this paper is to prove this formula for all the monomial representations τ with multiplicities of discrete type, namely finite or infinite of discrete type. We shall propose to concertize this formula by making explicit the Penney’s distributions aπk which come directly from the disintegration of the Dirac measure δτ for τ . At least for the formal level, these distributions are well described. Their existence as generalized integrals of continuous functions is far from being evident, contrarily to the situation of nilpotent Lie groups. The objective of section 5 of this paper is to show the existence of such integrals. In the sixth section, we shall prove this Plancherel formula in this setting. Especially, when these multiplicities are infinite, we shall give a precise meaning to the above formula.

Monomial Representations of Discrete …

3

The second objective of this paper is the study of the algebra Dτ (G/H ) of the G-invariant differential operators on the fiber space associated to the data (H, χ ) over the base space G/H . When G is nilpotent, it is well known that τ has finite multiplicities if and only if Dτ (G/H ) is a commutative algebra (cf. [3, 8, 17]). By means of the Plancherel formula obtained when τ is of discrete type, we are able to extend our study and prove that the algebra Dτ (G/H ) is again commutative in this situation. Here too, we expect to obtain results relative to the polynomial conjecture which consists in showing that this algebra is isomorphic to the algebra of the H invariant polynomial functions on the affine space . It is the case for example when the subgroup H is normal (see [3] and Example 6 of Sect. 7). Far from the nilpotent context, we produce a situation (see Example 7) showing that the problem of Duflo (cf. [12]) can have a negative solution in the exponential solvable context. When h is a polarization at f , the algebra Dτ (G/H ) is trivial (cf. [3]) and τ is of discrete type. In this setting, we show in the last section of this paper a kind of Frobenius reciprocity, which characterizes each multiplicity m(π ) of τ as the dimension of a certain space of distributions which are H −semi-invariant (with the 1/2 factor χ f  H,G ).

2 Preliminaries and Notations 2.1 The General Context In this paper, G denotes an exponential solvable Lie group with Lie algebra g. Let dg be a left Haar measure on G and G the modular function of G so that we have   ϕ(gx −1 )dg = G (x) ϕ(g)dg (x ∈ G) G

G

for any function ϕ belonging to the space K (G) of the continuous functions on G with compact support, we have G (x) = | det(Adx)|−1 (x ∈ G). We introduce a closed subgroup H of G with Lie algebra h and denote by  H,G the character of H with values in R+ defined by  H,G (h) =

 H (h) (h ∈ H ). G (h)

Hence it comes that, for X ∈ h,  H,G (exp X ) = exp(Tr adg/h X ).

4

A. Baklouti et al.

Let K (G, H ) be the space of the numerical continuous functions ϕ on G with compact support modulo H and which verify ϕ(gh) =  H,G (h)ϕ(g) for all g ∈ G and h ∈ H . the group G acting on K (G, H ) by left translation, we know that there exists, up to a scalar multiplication, one and only one G-invariant positive linear form. We denote it by νG,H or more simply ν and write it under an integral form 



νG,H (ϕ) =

ϕ(g)dν(g) = G/H

ϕ(g)dνG,H (g). G/H

If  H = G on H , νG,H is nothing but a G-invariant measure on the homogeneous space G/H and we shall often denote it by d g. ˙ Let K be a closed subgroup of H . We have the following lemma dealing with the formula of transitivity (cf. [6], Chap. V): Lemma 1 Let K ⊂ H be closed subgroups of G and ϕ a νG,K -integrable function. Then the set of all the g ∈ G such that the function on H : h → ϕ(gh)−1 H,G (h) is not ν H,K -integrable is νG,H -negligible. The function defined on G by 

ϕ(gh)−1 H,G (h)dν H,K (h)

g → H/K

is νG,H -integrable and, up to a normalisation, we have the formula 





ϕ(x)dνG,K (x) = G/K

dνG,H (g) G/H

H/K

ϕ(gh)−1 H,G (h)dν H,K (h).

2.2 Monomial Representations Let χ be a unitary character of H and τ = indGH χ the monomial representation of G. There exists a linear form f on g such that f vanishes√on the derived algebra [h, h] of h and that χ is written as χ (exp X ) = ei f (X ) (i = −1, X ∈ h). In this case, χ is also written as χ f . In order to describe this representation, let K (G, H, f ) be the space of the numeric continuous functions ϕ on G with compact support modulo H , 1/2 satisfying ϕ(gh) = χ f (h) H,G (h)ϕ(g) for all g ∈ G and h ∈ H . The group G acts on K (G, H, f ) by left translation. For ξ, η ∈ K (G, H, f ) we have the G-invariant scalar product 

ξ, η =

ξ(g)η(g)dν(g), G/H

(3)

Monomial Representations of Discrete …

5

what permits us to realize the representation τ by left translation in the space L 2 (G/H, χ f ), the completion of K (G, H, f ) with respect to the norm deduced from the scalar product (3). Recall first the canonical central decomposition of τ (cf. [15]), described in the framework of the orbit method. In what follows we shall often identify an equivalence class of a unitary represenˆ for a unitary irreducible tation by one of its representatives, writing simply π ∈ G, ˆ assorepresentation of G. Recall that the Kirillov–Bernat mapping θ : g∗ → G, ciates to each ∈ g∗ a (class of) unitary irreducible representation of G. Moreover, the induced mapping θ¯ = θ¯G : g∗ /G → Gˆ is a bijection (cf. [6]). Let now h⊥ be the annihilator of h in g∗ and μ˜ a finite positive measure on the affine space  = f + h⊥ equivalent to the Lebesgue measure on . We regard μ˜ as a measure on g∗ and take its image μ by the mapping θ . This defines the measure μ in the decomposition (1) of τ up to equivalence. Theorem 1 In the disintegration (1) of τ , the multiplicity function is given in the following way: The multiplicity m(π ) is the number of the connected components of . When this condition  ∩ (π ) if each component is a variety of dimension dim (π) 2 is not satisfied, m(π ) is equal to +∞. In any case, m(π ) is obtained as the number of the H -orbits contained in  ∩ (π ). In the case where the group G is connected and simply connected nilpotent, it is well known that the multiplicity function is either finite and uniformly bounded μ-almost everywhere on , or uniformly equal to the infinity.

2.3 Case of a Normal Subgroup of Codimension 1 Let G 0 = exp(g0 ) be a normal subgroup of codimension 1 in G. With respect to the restriction p : g∗ → g∗0 , a coadjoint orbit  of G is either saturated or nonsaturated in the direction g⊥ 0 and it occurs μ-almost everywhere the same eventuality of this alternative (cf. [18]). If  is saturated, p() is composed by a one-parameter ¯ family {ωt }t∈R of G 0 -orbits, dim  = dim ωt + 2 and θ¯G ()  indG G 0 θG 0 (ωt ) for all t ∈ R. On the other hand, if  is non-saturated, ω = p() is a G 0 -orbit, p gives a diffeomorphism from  onto ω, p −1 (ω) is composed by a one-parameter family {t }t∈R of G-orbits and the following holds: ¯ indG G 0 θG 0 (ω) 



⊕ R

θ¯G (t )dt.

6

A. Baklouti et al.

2.4 The Plancherel Formula We suppose in this subsection that the representation τ = indGH χ f has finite multiplicities. For a unitary representation ρ of G, we shall denote by Hρ its Hilbert space, Hρ∞ the space of the C ∞ -vectors equipped with the usual topology and Hρ−∞ the anti-dual space of Hρ∞ (cf. [7, 22]). For a ∈ Hρ±∞ and b ∈ Hρ∓∞ , we denote by

a, b the value of a at the point b, what leads to b, a = a, b. Given a closed subgroup K of G and a character λ of K , we put  −∞  K ,λ = {a ∈ Hρ−∞ ; ρ(k)a = λ(k)a, k ∈ K }. Hρ If e designates the neutral element of G, we see that the Dirac measure δτ : Hτ∞  ψ → ψ(e) ∈ C   H,χ f 1/2 H,G supplies an element of Hτ−∞ . Then, according to the disintegration of τ , δτ is disintegrated  δτ 

⊕ Gˆ

m(π) 

 aπk

dμ(π )

(4)

k=1

  H,χ f 1/2 H,G with aπk ∈ Hπ−∞ (cf. [21]). This says that, for any ϕ ∈ Cc∞ (G), the formula (2) holds with  f −1/2 ϕ H (g) = ϕ(gh)χ f (h) H,G (h)dh (g ∈ G). H

2.5 An Algebra of G-Invariant Differential Operators Our second objective in this paper is the study of the algebra Dτ (G/H ) of G-invariant differential operators on the fibre space associated to the data (H, χ ) with the base space G/H . Let U(g) be the enveloping algebra of gC . The elements of U(g) acting from right as left invariant differential operators. Let aτ be the subspace {Y + i f (Y ) − 1/2tr adg/h Y ; Y ∈ hC } of U(g) and let U(g)aτ be the left ideal of U(g) generated by aτ . Put finally U(g, τ ) = {W ∈ U(g); [h, W ] ⊂ U(g)aτ }.

Monomial Representations of Discrete …

7

Then, we know [8, 12] that, under the right action of the elements of U(g), the algebra Dτ (G/H ) is isomorphic to U(g, τ )/U(g)aτ . We know that the following problem was positively solved [17] in the nilpotent case. For ∈ g∗ , let B be the bilinear form on g defined by B (X, Y ) = ([X, Y ]) and let g( ) be the radical of B . Problem (Duflo [12]). The algebra Dτ (G/H ) is commutative if and only if generically on  the subspace h + g( ) is Lagrangian with respect to the form B .

3 Monomial Representations of Discrete Type Definition 1 We say that τ is of discrete type (or has multiplicities of discrete type) if each connected component of  ∩ (π ) is an H -orbit μ-almost everywhere. As an abuse of language, we often say that the multiplicities m(π ) are of discrete type. When G is completely solvable, a monomial representation is of discrete type if and only if it has finite multiplicities [5]. We also know [15] that τ is of discrete type if and only if h + g( ) is μ-almost everywhere a Lagrangian subspace for B . Independently of the fact that the monomial representation indGH χ f is of discrete type, we have the following lemma. Lemma 2 Let g be a real Lie algebra, h a subalgebra of g and n an ideal of g. Let f ∈ g∗ such that f, [h, h] = {0}. We put m 1 = inf dim(g( f + )) ∈h⊥

m 2 = inf dim(g(( f + )|n )) ∈h⊥

m 3 = inf dim(n ∩ g( f + )). ∈h⊥

Then the sets Ui ⊂ f + h⊥ (i = 1, 2, 3) constituted of ∈  such that dim(g( )) = m 1 , resp. dim(g( |n )) = m 2 , resp. dim(g( ) ∩ n) = m 3 are Zariski open sets. For every ∈ U1 , we have [g( ), g( )] ⊂ h ∩ g( ), and for every ∈ U2 ∩ U3 we have [g( |n ), g( ) ∩ n] ⊂ h ∩ g( ) ∩ n. Proof A part of this lemma is already found in ([6], Chap II). We can also apply the proof of the Proposition 1.11.7 of [9]. We here give another simple proof whose idea comes from [11]. Let d = dim g − m 1 . For ∈ U1 , we take a basis Z = {Z 1 , . . . , Z n } of g such that the matrix ⎛

⎞ 0

, [Z 1 , Z 2 ] . . . ...

, [Z 1 , Z d ] ⎜

, [Z 2 , Z 1 ] 0 ... ...

, [Z 2 , Z d ] ⎟ ⎜ ⎟ ⎜ ⎟ . .. . . . .. .. .. .. M( ) = ⎜ ⎟ . ⎜ ⎟ ⎝

, [Z d−1 , Z 1 ] ... ... 0

, [Z d−1 , Z d ] ⎠

, [Z d , Z 1 ] ... . . .

, [Z d , Z d−1 ] 0

8

A. Baklouti et al.

is of rank d, namely invertible. Then there is a neighbourhood V of 0 in h⊥ such that the corresponding matrix M( + ) is also invertible for every  ∈ V. For r = d + 1, . . . , n, let ξr () = (ξr1 (), . . . , ξrd ()) ∈ Rd , for which ξr ()M( + ) = (−a1,r (), . . . , −ad,r ()) with ak,r () =

+ , [Z r , Z k ]. Then, the vectors X r () = Z r +

d 

ξrj ()Z j , r = d + 1, . . . , n

j=1

form a basis of g( + ) and these vectors depend rationally on . So, for ∈ U1 and two arbitrary elements x, y of g( ), there exist two differentiable curves x(t), y(t) ∈ g( + t) defined in a neighbourhood of t = 0 and satisfy x(0) = x, y(0) = y. Deriving the relation

+ t, [x(t), y(t)] = 0 at t = 0, we obtain

, [x, y] +

, [x  (0), y] +

, [x, y  (0)] = 0, from which , [x, y] = 0. This means that [x, y] ∈ h. Likewise, for ∈ U2 ∩ U3 and x ∈ g( |n ), y ∈ g( ) ∩ n, there exist two differentiable curves x(t) ∈ g(( + t)|n ), y(t) ∈ g( + t) ∩ n in a neighbourhood of t = 0, satisfying x(0) = x, y(0) = y. Then, for t small enough, we have

+ t, [x(t), y(t)] = 0. Deriving this relation at t = 0, we have

, [x, y] +

, [x  (0), y] +

, [x, y  (0)] = 0, from which we get , [x, y] = 0. This means that [x, y] ∈ h.



Let S : {0} = g0 ⊂ g1 ⊂ · · · ⊂ gn−1 ⊂ gn = g, dim g j = j (0 ≤ j ≤ n), be a good sequence of subalgebras of g. That is to say, if g j is not an ideal of g, g j±1 are ideals of g and the adjoint action of g on g j+1 /g j−1 is irreducible. If g j and g j−1 are ideals of g, we have a root α j : g → R of g: ad(X )(X j ) − α j (X )X j ∈ g j−1 , X j ∈ g j , X ∈ g.

Monomial Representations of Discrete …

9

If g j is not an ideal of g, we take a subspace v j of dimension 2 of g j+1 such that g j+1 = v j + g j−1 . Then there is a homomorphism α j of g into the algebra of the endomorphisms of v j such that ad(X )v − α j (X )v ∈ g j−1 , X ∈ g, v ∈ v j . For ∈ g∗ , let

g j ( j ) = {X ∈ g j ;

, [X, g j ] = {0}}

with j = |g j (1 ≤ j ≤ n). Then, b = b[ ] = bS [ ] =

n 

g j ( j )

j=1

turns out to be a polarization at , called a Vergne polarization (with respect to S), and b[ ] satisfies [6] the Pukanszky condition.  such Let us return to our monomial representation τ of discrete type. Let π ∈ G that  ∩ (π ) decomposes into finite connected components as  ∩ (π ) =

m(π) 

Ck ,

k=1

where Ck is a H -orbit of dimension (1/2) dim (π ). Taking our previous works [1, 13–16], into account, we proceed as follows. We fix ∈ (π ) and take there a Vergne polarization b = b[ ] to realize π = indGB χ with B = exp b. Let us choose gk ∈ G (1 ≤ k ≤ m(π )) satisfying gk · ∈ Ck . A candidate of our generalized vector aπk would be 

aπk , ϕ =

−1/2

H/(H ∩gk Bgk−1 )

ϕ(hgk )χ f (h) H,G (h)dνk (h) ϕ ∈ Hπ∞ .

(5)

The first problem on the possibility to consider this integral concerns the relation of traces. For the simplicity, take gk = e. Our problem is the convergence of the integral 

aπ , ϕ =

−1/2

H/(H ∩B)

ϕ(h)χ f (h) H,G (h)dν(h).

(6)

Thus, we expect that the following trace relation holds for X ∈ b[ ] ∩ h: Tr adb[ ]/(b[ ]∩h) X + Tr adh/(b[ ]∩h) X = 0.

(7)

The next example shows that this is not always the case even if h + g( ) is Lagrangian.

10

A. Baklouti et al.

Example 1 Let g = X, Y1 , Y2 , Z , CR ; [X, Y1 ] = Y2 , [X, Y2 ] = Z and [Y1 , C] = C. Take h = Y1 , Y2 , Z R and = Z ∗ . Then g( ) = Y1 , Z , CR and for a choice of S, b[ ] = Y1 , Y2 , Z , CR . Clearly, h + g( ) is Lagrangian for the bilinear form B but Tr adb[ ]/(b[ ]∩h) Y1 + Tr adh/(b[ ]∩h) Y1 = 1. So, we introduce the following definition. Definition 2 Let g be an exponential solvable Lie algebra, h a subalgebra of g and n a nilpotent ideal of g containing [g, g]. Let f ∈ g∗ such that f, [h, h] = {0} and S : {0} ⊂ g1 ⊂ · · · ⊂ gn = g a good sequence of subalgebras passing through n. For ∈ , we denote by b[ ] the Vergne polarization at associated to S. Then, is said to be generic with respect to n and S, or merely generic, if ∈ U1 ∩ U2 ∩ U3 and if dim(b[ ] ∩ h) = inf dim(b[ f + ] ∩ h). ∈h⊥

Let  be the set of the roots of g which vanish on every g( |n ) with a generic ∈ . Let g0 = ∩α∈ ker α. We say that ∈  is strongly generic if is generic and if α(g( |n )) = {0} for every root α not contained in . Remark. From Lemma 2, for every generic linear form ∈ , we have [g( |n ), g( ) ∩ n] ⊂ h and [g( ), g( )] ⊂ h. Moreover, the set of the strongly generic elements of  contains a Zariski open set of . Indeed, U1 ∩ U2 ∩ U3 is a Zariski open set of  and g( |n ) admits a basis which varies rationally in on a Zariski open set of . On the other hand, the set of the linear forms ∈  verifying dim(b[ ] ∩ h) = inf dim(b[ f + ] ∩ h) ∈h⊥

is expressed as a set of the linear forms which do not vanish a finite family of polynomials on . Proposition 1 With the preceding notations, let h0 = h ∩ g0 and 0 = f |g0 + (h0 )⊥ ⊂ g∗0 . Let C be a composition sequence passing through n, which does not change all the Vergne polarizations b[ ] for ∈  generic, constructed from the starting composition sequence. Then, for every strongly generic ∈ , its restriction 0 to g0 is generic (with respect to h0 , C ∩ g0 and n). Proof In fact, for every strongly generic ∈ , we have: • g( |n ) = g0 ( 0 |n ),

Monomial Representations of Discrete …

• dim(g0 ( 0 )/g( )) = dim(g/g0 ), • dim((g0 ( 0 ) ∩ n)/g( ) ∩ n) = dim(g/g0 ), • h ∩ b[ ] = h0 ∩ b[ ].

11



The following proposition intends to show the trace relation (7) for the generic elements. Proposition 2 We keep the notations of the previous definition. Let ∈  generic with respect to n and S. Then the trace relation (7) holds on b[ ] ∩ h. Proof We proceed by induction on p = dim G + dim G/H . Let C : {0} = g0 ⊂ g1 ⊂ · · · ⊂ gd = g be the composition sequence of g determined by S such that gk = n. For p ∈ {1, 2} or more generally if g is abelian, our assertion is clear. Let j > 0 be the smallest index such that g j ⊂ g( ). If j > k, then n ⊂ g( ), b[ ] is an ideal of g and b = b[ ] is contained in g( |n ). Replacing h by h = h + n, we have [b[ ], n] ⊂ h by Lemma 2 because is generic and b[ ] ⊂ g( |n ). Hence, Tr adb[ ]/(b[ ]∩h) X = Tr adh/(b[ ]∩h) X = 0 for every X ∈ b[ ] ∩ h. So, we can suppose hereafter j ≤ k, namely g j ⊂ n. If j > 1 and if vanishes on a non-trivial ideal a of g contained in g j−1 , then a ⊂ g( ). If a is included in h and so in b ∩ h, everything passes to the induction. If not, we can replace h by h = h + a and apply the induction hypothesis. In fact, we have b[ ] ∩ h = b[ ] ∩ h + a, from which h /(b[ ] ∩ h )  h/(b[ ] ∩ h) and

b[ ]/(b[ ] ∩ h)  b[ ]/(b[ ] ∩ h ) + a/(a ∩ h).

Using the induction hypothesis, we have, for every X ∈ b[ ] ∩ h, Tr adb[ ]/(b[ ]∩h ) X + Tr adh/(b[ ]∩h) X = 0, as X is a sum of a nilpotent vector and an element of g( |n ) and Tr ada/(a∩h) X = 0 because of [g( |n ), a] ⊂ b ∩ h. Taking everything into account, we can suppose that either j = 1 or j = 2 where g1 is central. We are now in the situation where (g1 ) = {0} and the root α 1 of g on g1 is non-zero. Let g(1) = {X ∈ g; [X, g1 ] = {0}}. Then g(1) is an ideal of g of codimension 1 containing g( |n ) + n. Let S (1) be the good sequence determined by the sequence g j ∩ g(1) . Let h(1) = h ∩ g(1) + g1 and (1) = |g(1) . Then (1) is

12

A. Baklouti et al.

generic with respect to h(1) and S (1) . Since b ∩ h ⊂ g(1) , we can apply the induction hypothesis to h(1) , S (1) , (1) and n. It remains for us to study the case where g1 = RZ , Z being central, Z ∈ h and

(Z ) = 1. Let d be the minimal non-central ideal g2 . If d = RZ ⊕ y with an ideal y of dimension 1 or 2, then we proceed as before according to (y) = 0 or not. Finally, if any of the preceding cases does not happen, the subalgebra g(1) = d = {X ∈ g; B (X, Y ) = 0, Y ∈ d} is proper and contains b. Let S (1) be the good sequence of g(1) obtained from the sequence S. The subalgebra b is a polarization at (1) = |g(1) . Let n(1) = n ∩ g(1) . If h ⊂ g(1) , set h(1) = h ∩ g(1) + d. Then we easily verify that (1) is generic with respect to h(1) , n(1) , S (1) . The induction hypothesis applied to g(1) assures us of the trace relation Tr adb[ ]/(b[ ]∩h(1) ) X + Tr adh(1) /(b[ ]∩h(1) ) X = 0 for every X ∈ b[ ] ∩ h, which is nothing but the desired relation. Indeed, b[ ] ∩ h(1) = b[ ] ∩ h + d and we have Tr add/(d∩g1 ) X + Tr adh/(h∩g(1) ) X = 0 for any X ∈ b[ ] ∩ h. Now suppose that h ⊂ g(1) . Let again h(1) = h + d and (1) = |g(1) . If d ⊂ h, (1) is generic and we can apply the induction hypothesis. If the root α 2 of g on g2 /g1 vanishes on h, then, considering the subalgebra h(1) , (1) is again generic with respect to n(1) and S (1) . Hence it suffices to apply the induction hypothesis, because Tr adb[ ]/(b[ ]∩h) X = Tr adb[ ]/(b[ ]∩h(1) ) X, Tr adh(1) /(b[ ]∩h(1) ) X = Tr adh/(b[ ]∩h) X. If the root α 2 does not vanish on h and if d ⊂ h, then d ∩ h = RZ . Let  ∈ h⊥ such that |d = 0. Since is generic, we have dim(b[ ]) = dim(b[ + t]), dim(b[ ] ∩ h) = dim(b[ + t] ∩ h) for t small enough. If there is U ∈ b[ ] ∩ h such that α 2 (U ) = 0, this means that there exists Ut ∈ b[ + t] ∩ h such that α 2 (Ut ) = 0 for t = 0 small enough. But then / g(1) , which is contradictory with the fact that h ⊂ g(1) . Therefore, α 2 (b[ ] ∩ Ut ∈ h) = {0} and we finally obtain the relation Tr adb[ ]/(b[ ]∩h) X + Tr adh/(b[ ]∩h) X = 0 for X ∈ b[ ] ∩ h.



Monomial Representations of Discrete …

13

Proposition 3 For ∈ , put b = b[ ] and B = exp b. Let gk ∈ G such that k = Ad∗ (gk ) ∈  and let bk = Ad(gk )b and Bk = gk Bgk−1 . Let us consider for ϕ ∈ K (G, B, ) the function −1/2

(h) = ϕ(hgk )χ f (h) H,G (h), h ∈ H. If the trace relation (7) is verified for k and h, then we have for x ∈ H ∩ Bk , and h ∈ H, (hx) =  H ∩Bk ,H (x)(h). Proof The subalgebra bk is a Vergne polarization at k associated with the good sequence Ad(gk )(S). Then the trace relation (7) says that  Bk ∩H,H (x) = −1 Bk ∩H,Bk (x), namely,

−1/2

 Bk ∩H (x) H

(8)

1/2

(x) =  Bk (x), x ∈ Bk ∩ H.

Hence, −1/2

(hx) = ϕ(hxgk )χ f (hx) H,G (hx) −1/2

= χ (gk−1 xgk )−1 χ f (x) B,G (gk−1 xgk ) H,G (hx)χ f (h)ϕ(hgk ) 1/2

1/2

−1/2

=  Bk (x) H

−1/2

(x) H,G (h)χ f (h)ϕ(hgk ) −1/2

=  Bk ∩H,H (x) H,G (h)χ f (h)ϕ(hgk ) (using equation (8)) =  Bk ∩H,H (x)(h), x ∈ Bk ∩ H h ∈ H.  This proves that the expression (6) has at least formally a meaning. We still need to prove the convergence of the interval. If the simple product H Bk is a closed subset of G, as the homogeneous space H/(H ∩ Bk ) is homeomorphic to H Bk /Bk , the integral (6) would be convergent for ϕ ∈ Hπ∞ continuous with compact support modulo B. Unfortunately, the simple product in question is not always closed as the following example shows. Example 2 Let g = X, T, A, Z R ; [X, T ] = −T, [X, A] = A, [T, A] = Z . Take h = RX and f ∈ g∗ such that f |h = 0. Let = α X ∗ + βT ∗ + γ A∗ + λZ ∗ (α, β, γ , λ ∈ R) be an arbitrary element of g∗ . By simple calculations, we see that G· =

   βγ uw X ∗ + uT ∗ + w A∗ + λZ ∗ ; u, w ∈ R α− + λ λ

14

A. Baklouti et al.

when λ = 0. Thus, G· ∩ h⊥ = {uT ∗ + w A∗ + λZ ∗ ; uw = ζ } with ζ = βγ − λα. Take again = uT ∗ + w A∗ + λZ ∗ (λ = 0) in h⊥ . If uw = 0, h + g( ) is a Lagrangian subspace for the bilinear form B and each connected component of G· ∩ h⊥ is an H -orbit, where H = exp h. Then τ = indGH χ f has finite multiplicities uniformly equal to 2. Now, let us consider the Jordan–Hölder sequence of ideals of g, S : {0} ⊂ RZ ⊂ RZ + RA ⊂ RZ + RA + RT ⊂ g. Then at the point , the associated Vergne polarization is bS [ ] = b[ ] = R(λX − wT ) + RA + RZ . Putting

 (ζ = uw), π = πζ,λ = indGB[ ] χ ∈ G 

we have τ

⊕

R2

2πζ,λ dζ dλ.

Hence τ has finite multiplicities, but the simple product H B[ ], B[ ] = exp(b[ ]), is a proper open subset of G. Definition 3 Let G be an exponential solvable Lie group with Lie algebra g, H = exp h a closed connected subgroup of G and ∈ g∗ which vanishes on [h, h]. Let b be a subalgebra of g subordinated to and satisfying Tr adb/(b∩h) X + Tr adh/(b∩h) X = 0, X ∈ b ∩ h. Let B = exp b. We can define the linear form a ,b,h on the vector space 1/2

D(B, H, ) := {ϕ : G → C; continuous, ϕ(gc) =  H,B∩H (c)χ (c)ϕ(g), c ∈ B ∩ H, g ∈ G,  −1/2 ϕ1 = |ϕ(h)| H,G (h)dν(h) < ∞} H/(H ∩B)

by the (convergent) integral 

a ,b,h , ϕ = (see Proposition 3).

−1/2

H/(H ∩B)

ϕ(h)χ (h) H,G (h)dν(h)

Monomial Representations of Discrete …

15

Our objective is to prove the convergence of the integral (6) which will allow us to describe our Plancherel formula in the sequel. More precisely, we go to prove the following theorem since the explicit determination of Hπ∞ is unknown. Theorem 2 Let G be an exponential solvable Lie group with Lie algebra g, H = exp h a closed connected subgroup of G, f ∈ g∗ vanishing on [h, h] and τ = ind GH χ f a monomial representation of discrete type. Let C be a composition sequence of g passing through a nilpotent ideal n which contains [g, g] and S a good sequence of subalgebras of g constructed from C. For any strongly generic ∈ , the integral 

a ,b,h , ϕ = aπ , ϕ =

−1/2

H/(H ∩B)

ϕ(h)χ f (h) H,G (h)dν(h)

(9)

is well defined and is convergent for any ϕ ∈ K (G, B, ). Namely, K (G, B, ) ⊂ D(B, H, ). Here, b = b[ ] denotes the Vergne polarization at with respect to S,  B = exp b and π = ind GB χ ∈ G. Corollary 1 For any strongly generic ∈  and any ϕ ∈ K (G, B, f ), the function defined by 

−1/2

f

g → ϕ H,B (g) =

H/(H ∩B)

ϕ(gh)χ f (h) H,G (h)dν(h)

is continuous on G. f

Proof In fact, since the mapping Cc (G) → K (G, B, f ); ξ → ξ B defined by 

−1/2

f

ξ B (g) =

B

ξ(gb)χ (b) B,G (b)db, g ∈ G f

is surjective, there is a function ϕ0 ∈ Cc (G) such that ϕ = (ϕ0 ) B . Then for any compact set K of G, we can take a function ψ0 ≥ 0 in Cc (G) such that |ϕ0 (kg)| ≤ ψ0 (g) for all g ∈ G, k ∈ K . Then  −1/2 |ϕ0 (kgb)| B,G (b)db |ϕ(kg)| ≤ B  −1/2 ≤ ψ0 (gb) B,G (b)db =: ψ(g). B

The functions |ϕ|2 , ψ 2 belonging to K (G, B), Theorem 2 gives 

−1/2

H/(H ∩B)

|ϕ(kgh)| H,G (h)dν(h) ≤



−1/2

H/(H ∩B)

ψ(gh) H,G (h)dν(h) < ∞

16

A. Baklouti et al.

for all k ∈ K , g ∈ G. An application of Lebesgue’s dominated convergence theorem f  gives the continuity of the function ϕ H,B on G.

4 A Basis for h/(h ∩ b) 4.1 On Some Relative Indices Let g be an exponential solvable Lie algebra and n a nilpotent ideal of g which contains [g, g]. Let us take a composition sequence C : {0} = g0 ⊂ g1 ⊂ · · · ⊂ gd−1 ⊂ gd = g, 1 ≤ dim(g j /g j−1 ) ≤ 2 (1 ≤ j ≤ d), where gk = n for a certain k. For every j ∈ {1, . . . , d}, we determine a subspace w j of g j so that g j = w j ⊕ g j−1 . We denote by α j : g → End(w j ) the irreducible representation of g in w j given by the adjoint action on g j /g j−1 . By abuse of a notation, we shall write dim(α j ), which is actually 1 or 2, to designate the dimension of the space of the root α j in question. Let D = { j ∈ {1, . . . , d}; g j /g j−1 is 2 dimensional} and r : {0, . . . , d} → {0, . . . , n} the mapping defined by r (0) = 0 and inductively on j = 1, . . . , d by  r ( j − 1) + 1 if j ∈ / D, r ( j) = r ( j − 1) + 2 otherwise. This gives us a good sequence of subalgebras (gr )rn=1 in the following way. Each r ∈ {1, . . . , n} is written as r = r ( j) or r = r ( j) − 1 for a certain j ∈ {1, . . . , d}. For every j ∈ {1, . . . , d} \ D, take a vector Z j ∈ g j \ g j−1 and put w j = RZ j and Z r ( j) = Z j . For j ∈ D, we have a linear form a j : g → R, a real number ω = 0 and two vectors X j , Y j ∈ g j \ g j−1 such that, for T ∈ g, α j (T )(X j ) ≡ a j (T )(X j − ωY j ), α j (T )(Y j ) ≡ a j (T )(Y j + ωX j ) modulo g j−1 . Put w j = RX j ⊕ RY j and Z r ( j)−1 = X j , Z r ( j) = Y j ∈ g j \ g j−1 . Then the vectors

Monomial Representations of Discrete …

17

{Z r , r = r ( j), j ∈ {1, . . . , d} \ D and r = r ( j), r ( j) − 1, if j ∈ D} give us a basis B of g and also a good sequence of subalgebras S = (gr )rn=1 . Denote by b the Vergne polarization at with respect to S. For a subspace v of g, let ICv = {i ∈ {1, . . . , d}; vi = v ∩ gi = vi−1 }. Likewise, let ISv = {r ∈ {1, . . . , n}; vr = v ∩ gr = vr −1 }. For a linear form on g and X ∈ g, we write (X ) or often

, X , to designate the value of at X .

4.2 An Algorithm of Construction of Co-exponential Bases Let g be an exponential solvable Lie algebra, h a subalgebra of g subordinate to f such that indGH χ f is of discrete type and a strongly generic ∈ . In this part, we suppose that no root of g vanishes on g( |n ). We intend to construct an algorithm which determines the Vergne polarization b = bS [ ] at associated with a good sequence of subalgebras S related to the composition sequence C, a coexponential basis of h modulo b ∩ h and a coexponential basis of g modulo b. This construction is principally built on the sequence C and the consequent root system. We now construct at each step of the induction an intermediate subalgebra g(i) which converges at the end of the algorithm to the polarization b, a subalgebra h(i) taking the position of h at each step, and the sets of indices J, L which indicate the positions of our desired coexponential basis. For more details on these different constructions, we advise the readers to consult the reference [20]. Start from our composition sequence C = (g j )dj=1 such that gk = n. Let ∈ g∗ and let B = {Z 1 , . . . , Z n } be a Jordan–Hölder basis of g extracted from the sequence S, namely Z j ∈ g j \ g j−1 . For j ∈ IS g( ) j ∈ IS ), the vector Z j is taken in h ∩ b (or g( ) respectively).

h∩b

(or

2.1. Initial step Set g(0) = g, h(0) = h, C (0) = C, (0) = and L = {1, . . . , n}. 2.2. First step of the algorithm (1) Let us consider the first index j1 such that a = g j1 ⊂ g( ). If this index does not exist, then g = g( ) and the situation is trivial. If j1 > k, then n ⊂ g( ). The polarization b at and g( ) are ideals of g and g/g( ) is abelian. This situation is also easy to treat.

18

A. Baklouti et al.

If j1 = 1, then (g j1 ) = {0} and a = g j1 , whose dimension is 1 or 2, is not central. Then, g(1) = a is the kernel of the root α 1 , which is an ideal of codimension 1 in g containing b + [g, g]. Since is strongly generic, this case does not occur. (2) Now, let k ≥ j1 > 1, then necessarily (g j1 ) = {0}. Suppose first that

(g j1 −1 ) = {0}. If the dimension of α j1 is 1, take Y1 ∈ g j1 such that (Y1 ) = 1. Then, Y1 ∈ gr ( j1 ) ( r ( j1 ) ) ⊂ b. Furthermore, [X, Y1 ] = α j1 (X )Y1 modulo g j1 −1 and

([X, Y1 ]) = α j1 (X ) for all X ∈ g. Hence, b ⊂ ker(α j1 ). So, this case does not occur. The same happens if the dimension of α j1 is 2. (3) In this way, we can suppose that (g j1 −1 ) = {0}. We find C1 ∈ g j1 −1 with

(C1 ) = 1 and Y1 ∈ g j1 \ g j1 −1 with (Y1 ) = 0. We also find a non-zero linear form β1 : g → R such that [A, Y1 ] = α j1 (A)Y1 + β1 (A)C1 mod g j1 −1 ∩ ker , A ∈ g. Then β1 (A) =

, [A, Y1 ], A ∈ g. If ker β1 is an ideal of g, then g(1) = a is an ideal of codimension 1 in g containing b[ ] + [g, g]. We set h(1) = h + a if h ⊂ a and h(1) = h ∩ a + a otherwise. In this last case, take a vector U1 ∈ h \ a . In order to construct a basis of g/b[ ], take as first vector V1 = U1 if h ⊂ a and otherwise an arbitrary vector V1 of g \ a . The starting composition sequence C 0 of g(0) is replaced by the new sequence C 1 of g(1) . Let (1) = |g(1) and n(1) = n ∩ g(1) . We remark that α j1 = 0 in this case, because otherwise β1 does not vanish on [g, g]. (4) Now, if ker(β1 ) is not an ideal, we set g(1) = ker(β1 ), n(1) = n ∩ g(1) and take the smallest index r1 such that gr1 ⊂ ker(β1 ) and V1 ∈ gr1 such that β1 (V1 ) = 1. If h ⊂ g(1) , we also choose the smallest index s1 such that h ∩ gs1 ⊂ ker(β1 ) and U1 ∈ h ∩ gs1 with β1 (U1 ) = 1. If h ∩ ker(β1 ) ⊂ ker(α j1 ), we can find U1 in n. Set h(1) = h + g j1 if h ⊂ ker(β1 ) and h(1) = h ∩ g(1) + g j1 otherwise. (5) Suppose now that dim(α j1 ) = 2. Then g j1 ⊂ b and we have two vectors Y1 , Y1 ∈ (g j1 ∩ ker ) \ g j1 −1 , C1 ∈ g j1 −1 verifying (C1 ) = 1 and two linear forms β1 , β1 such that, for A ∈ g, [A, Y1 ] = α j1 (A)(Y1 − ωY1 ) + β1 (A)C1 mod g j1 −1 ∩ ker , [A, Y1 ] = α j1 (A)(Y1 + ωY1 ) + β1 (A)C1 mod g j1 −1 ∩ ker . Moreover, the linear forms β1 , β1 are linearly independent with α j1 . Indeed, if β1 , β1 are linearly dependent, there exists a constant λ ∈ R such that Y1 − λY1 ∈ g( ). Taking A ∈ g( |n ) \ ker(α j1 ), we see that Y1 , Y1 belong to g( ) since is strongly generic. This is impossible. Set g(1) = ker(β1 ) ∩ ker(β1 ), which is of codimension 2, and n(1) = n ∩ g(1) .

Monomial Representations of Discrete …

19

(6) Let r1 be the minimal index such that gr1 ⊂ g(1) . Then, it is easy to see dim(gr1 /gr1 −1 ) = 2 and the existence of two vectors V1 , V1 ∈ gr1 such that gr1 = gr1 −1 + RV1 + RV1 and

β1 (V1 ) = 1, β1 (V1 ) = 0, β1 (V1 ) = 0, β1 (V1 ) = 1.

Similarly, we set h(1) = h + g j1 if h ⊂ g(1) and h(1) = h ∩ g(1) + g j1 otherwise. If h ⊂ g(1) , let s1 be the smallest index such that h ∩ gs1 ⊂ g(1) . If further h(1) ⊂ ker(α j1 ), we have two vectors U1 , U1 ∈ gs1 such that h ∩ gs1 = h ∩ gs1 −1 + RU1 + RU1 and

β1 (U1 ) = β1 (U1 ) = 1, β1 (U1 ) = β1 (U1 ) = 0,

(10)

modifying U1 by a multiple of U1 if necessary. Moreover, we can choose U1 , U1 in n. If h is not contained in g(1) and if α j1 vanishes on h(1) , the dimension of h/(h ∩ g(1) ) is 1 or 2. We find again the minimal index s1 and a vector U1 or the supplementary  minimal index s1 and two vectors U1 ∈ gs1 and U1 ∈ gs1 with the relations (10). But, we can no longer say that U1 or U1 , U1 are in n. 2.3. Second step of the algorithm Let L 1 = {1, ..., n} \ {r ( j1 )}, resp. L 1 = {1, ..., n} \ {r ( j1 ), r ( j1 ) − 1}. Take as before (1) = |g(1) and replace in the preceding reasoning g by g(1) , n by n(1) . In the cases (4) and (6), we shall make supplementary changes. First we repeat the construction and consider as before the smallest index j2 such that g(1) ∩ g j2 ⊂ g(1) ( (1) ). Suppose first that the root α j2 is of dimension 1. So, we are in the case 4). Take Y2 ∈ g j2 ∩ g(1) ∩ ker \ g j2 −1 , C2 ∈ g j2 −1 ∩ g(1) verifying

, C2  = 1. Then, for A ∈ g, [A, Y2 ] = α j2 (A)Y2 + β2 (A)C2 mod g j2 −1 ∩ ker , where β2 is the linear form given by β2 (A) =

, [A, Y2 ]. Being free to replace C2 by C2 + ζ Y1 (resp. C2 + ζ Y1 + ζ  Y1 ) with a well chosen ζ ∈ R (resp. ζ, ζ  ∈ R), we can suppose

, [U1 , C2 ] = 0 (resp.

, [U1 , C2 ] = 0,

, [U1 , C2 ] = 0).

20

A. Baklouti et al.

Likewise, being free to modify Y2 by a multiple of Y1 (resp. a linear combination of Y1 , Y1 ), we can admit

, [U1 , Y2 ] = 0, resp.

, [U1 , Y2 ] = 0. Set g(2) = g(1) ∩ ker(β2 ) and h(2) = h(1) ∩ ker(β2 ) + g(1) ∩ g j2 if h(1) ⊂ g(2) , otherwise h(2) = h(1) + g(1) ∩ g j2 . We obtain the maximal index r2 such that gr2 −1 ∩ g(1) ⊂ g(2) and a vector V2 ∈ gr2 ∩ g(1) satisfying

, [V2 , Y2 ] = 1. In the same way, if h(1) ⊂ g(2) , we have an index s2 and an element U2 ∈ h ∩ gs2 ∩ g(1) such that h(1) ∩ gs2 −1 ⊂ g(2) and

, [U2 , Y2 ] = 1. Finally, put L 2 = L 1 \ {r ( j2 )}. In the case where the dimension of α j2 is 2, we shall have the indices r2 , s2 and occasionally s2 . Also the vectors V2 , V2 , U2 and occasionally U2 etc. If further h(2) ⊂ ker(α j2 ), we can admit that U2 (resp. U2 and U2 ) belongs to n. Finally, set L 2 = L 1 \ {r ( j2 ), r ( j2 ) − 1}. 2.4. Intermediate step of the algorithm We continue this process until we arrive to the number l for which g(l) = b. Let

(i) = |g(i) . For 1 ≤ i ≤ l, we find the smallest index ji such that g ji ⊂ g(i−1) ( (i−1) ), the root α ji , the vectors Ci , Yi (resp. Ci , Yi , Yi ) in g ji ∩ g(i−1) such that (Ci ) = 0 and (Yi ) = 0 (resp. (Yi ) = (Yi ) = 0), and the linear form βi (resp. βi , βi ) such that, for A ∈ g, [A, Yi ] ≡ α ji (A)Yi + βi (A)Ci mod g ji −1 ∩ ker , respectively [A, Yi ] ≡ α ji (A)(Yi − ωYi ) + βi (A)Ci [A, Yi ] ≡ α ji (A)(Yi + ωYi ) + βi (A)Ci modulo g ji −1 ∩ ker . Besides, we get the subalgebra g(i) = g(i−1) ∩ ker(βi ) (resp. g(i) = g(i−1) ∩ ker(βi ) ∩ ker(βi ), the smallest index ri such that gri ∩ g(i−1) ⊂ g(i) and the vector Vi (resp. Vi , Vi ) in gri ∩ g(i−1) such that βi (Vi ) = 1 (resp. βi (Vi ) = βi (Vi ) = 1, βi (Vi ) = βi (Vi ) = 0). Set h(i) = h(i−1) ∩ g(i) + g ji ∩ g(i−1) if h(i−1) ⊂ g(i) . Otherwise, h(i) = h(i−1) + g ∩ g(i−1) . Then just as before, if h(i−1) ⊂ g(i) , we find the minimal index si (resp. si , si ) and Ui (resp. Ui , Ui ) in h(i−1) such that gsi ∩ h(i−1) ⊂ h(i) and ji

βi (Ui ) = 1 (resp. βi (Ui ) = βi (Ui ) = 1, βi (Ui ) = βi (Ui ) = 0). We have also the relations:

Monomial Representations of Discrete …

21

, [Ui , Y j ] = δi, j (1 ≤ i, j ≤ l), h(i−1) = h(i)

, [Ui , C j ] = 0, ∀ j ≥ i.

(11)

Then,

, [h, C j ] = {0}

(12)

for all j. Set finally L i = L i−1 \ {r ( ji )} or L i = L i−1 \ {r ( ji ), r ( ji ) − 1} according to the situation. We can impose that Ui (resp. Ui , Ui ) is contained in n if α ji (h(i) ) = {0}. 2.5. Final result of the algorithm Here we resume the results at the end of the development of the preceding algorithm. We present these results in the form of lemmas. Suppose that g = g( ). We begin with: Lemma 3 There is a sequence of subalgebras g = g(0)  g(1)  · · ·  g(s) = b and a sequence of integers 2 ≤ j1 ≤ · · · ≤ js ≤ d such that, putting (i) = |g(i) (1 ≤ i ≤ s − 1), we have g ji+1 −1 ∩ g(i) ⊂ g(i) ( (i) ), g ji+1 ∩ g(i) ⊂ g(i) ( (i) ) and g(i+1) = g(i) ( (i) |g ji+1 ∩g(i) ). We define the subalgebra h(i) of g(i) inductively on 0 ≤ i ≤ s by setting h(0) = h and h(i) = h(i−1) ∩ g(i) + g ji ∩ g(i−1) for i > 0. For 0 ≤ i ≤ s, we denote by S (i) the trace of S on g(i) and put n(i) = n ∩ g(i) , G (i) = exp(g(i) ), H (i) = exp(h(i) ), f (i) = f |g(i) and  (i) = f (i) + (h(i−1) )⊥ . Lemma 4 (i) f |[h(i) ,h(i) ] = {0} so that we can consider the character χ (i) of H (i) (i) with differential i f |h(i) and the monomial representation τ (i) = ind GH (i) χ (i) . (ii) τ (i) is of discrete type. (iii) (i) is an element of  (i) strongly generic with respect to h(i) , S (i) and n(i) . (iv) |g ji+1 −1 = 0. (v) g ji ⊂ b ⊂ g(i) . (vi) B induces a g(i+1) -invariant duality between g(i) /g(i+1) and g ji+1 /g ji+1 −1 . In particular, the vector space g(i) /g(i+1) is of dimension 1 or 2. (vii) h(i) = h ∩ g(i+1) + g j1 + g j2 ∩ g(1) + · · · + g ji ∩ g(i−1) . In particular, we have h(i) /(h(i) ∩ g(i+1) ) = (h ∩ g(i) )/(h ∩ g(i+1) ). (viii) No root of g vanishes on g(i) ( (i) |n(i) ). We set Ik = {1 ≤ i ≤ s; dim g ji /g ji−1 = k}, Jk = {1 ≤ i ≤ s; dim(h ∩ g(i−1) )/ (h ∩ g(i) ) = k} for k = 1, 2, I = I1  I2 = {1, . . . , s} and J = J1  J2 ⊂ I . For i ∈ I (resp. i ∈ J ), we denote by ri (resp. si ) the smallest index such that gri ∩ g(i−1) ⊂ g(i) (resp. gsi ∩ h(i−1) ⊂ g(i) ) and if i ∈ J2 verifies dim(gsi ∩ h(i−1) )/(g(i) ∩ gsi ∩  h(i−1) ) = 1, we denote by si the smallest integer such that gsi ∩ h(i−1)  gsi ∩ h(i−1) .

22

A. Baklouti et al.

Lemma 5 There is C ∈ g j1 −1 verifying

, C = 1, and for i ∈ I1 (resp. i ∈ I2 ) there is Yi ∈ (g ji ∩ g(i−1) ) \ g(i) (resp. Yi , Yi ∈ (g ji ∩ g(i−1) ) \ g(i) and wi ∈ R \ {0}), βi ∈ g∗ (resp. βi , βi ∈ g∗ ), Vi ∈ (gri ∩ g(i−1) ) \ g(i) (resp. Vi , Vi ∈ (gri ∩ g(i−1) ) \ g(i) ) such that, on the one hand, if i ∈ I1 , [X, Yi ] = α ji (X )Yi + βi (X )C, X ∈ g, modulo g ji −1 ∩ ker , (Yi ) = 0, βi (Vi ) = 1, and if i ∈ I2 , [X, Yi ] = α ji (X )(Yi + wi Yi ) + βi (X )C, [X, Yi ] = α ji (X )(Yi − wi Yi ) + βi (X )C, X ∈ g, modulo g ji −1 ∩ ker , (Yi ) = (Yi ) = 0, βi (Vi ) = βi (Vi ) = 1, βi (Vi ) = βi (Vi ) = 0. On the other hand, if i ∈ J1 , there is Ui ∈ (h ∩ g(i−1) ∩ gsi ) \ g(i) verifying βi (Ui ) = 1. If i ∈ J2 and dim((gsi ∩ h(i−1) )/(g(i) ∩ gsi ∩ h(i−1) )) = 2, there are Ui , Ui ∈ (h ∩ g(i−1) ∩ gsi ) \ g(i) such that βi (Ui ) = βi (Ui ) = 1 and βi (Ui ) = βi (Ui ) = 0. If i ∈ J2 and dim((gsi ∩ h(i−1) )/(g(i) ∩ gsi ∩ h(i−1) )) = 1, there are  Ui ∈ (h ∩ g(i−1) ∩ gsi ) \ g(i) and Ui ∈ (h ∩ g(i−1) ∩ gsi ) \ (h ∩ gsi ) satisfying     βi (Ui ) = βi (Ui ) = 1 and βi (Ui ) = βi (Ui ) = 0. Moreover, if i ∈ J1 ,

, [Ui , C] = 0, and if i ∈ J2 ,

, [Ui , C] =

, [Ui , C] = 0. We have

, [U j , Yi ] =

, [U j , Yi ] =

, [U j , Yi ] =

, [U j , Yi ] = 0, when these expressions have a sense. Let i ∈ J satisfying α j1 (h(i) ) = {0}. Then, we have dim((gsi ∩ h(i−1) )/(g(i) ∩ gsi ∩ h(i−1) )) = dim((h ∩ g(i−1) )/(h ∩ g(i) )) and we can choose Ui ∈ n (resp. Ui , Ui ∈ n) if i ∈ J1 (resp. i ∈ J2 ). For i ∈ I1 (resp. i ∈ I2 ) satisfying α ji = 0, the linear forms α ji and βi (resp. βi , βi ) are linearly independent. Definition 4 Let J = {1 ≤ i ≤ d; h(i−1) = h(i) }. Remark that, for i ∈ J , the dimension of h(i) /(h(i−1) ∩ g(i) ) is 1 or 2. Let B = exp b be the Vergne polarization at . For s ∈ B, we set  ji e−i Tr(α (log s)) , δ (s) = δ ,S (s) = i∈J

where i = 1 if dim(α ji ) = 1 or dim(α ji ) = 2 and dim(h(i) /(h(i−1) ∩ g(i) ) = 2. Otherwise, i = 1/2. Corollary 2 The process described by the above algorithm gives us the following families of coexponential bases:

Monomial Representations of Discrete …

23

{Vi , i ∈ I1 }  {Vi , Vi , i ∈ I2 } is a coexponential basis of g/b, {Ui , i ∈ J1 }  {Ui , Ui , i ∈ J2 } is a coexponential basis of h/(b ∩ h), {Yi , i ∈ I1 }  {Yi , Yi , i ∈ I2 } is a coexponential basis of b/g( ), {Yi , i ∈ I1 }  {Yi , Yi , i ∈ I2 }  {C} is a coexponential basis of b/(g( ) ∩ ker ). Corollary 3 We can describe the quotient space H/(H ∩ B) and the integral  H/(H ∩B)

ϕ(h)dν H,H ∩B (h)

for a continuous function ϕ on H with compact support modulo H ∩ B in the following way: H/H ∩ B =



exp (RU j ) (or



j∈J

 H/(H ∩B)



j∈J



and

ϕ(h)dν H,H ∩B (h) =



R# J



or H/H ∩B

ϕ(h)dν H,H ∩B (h) =

with #J =

exp (RU j ) exp (RU j ))



R# J

ϕ(

ϕ(





exp(u i Ui ))

i∈J



du i ,

i

exp(u i Ui ) exp(u i Ui ))

i∈J



du i du i



i

dim(h(i) /(h(i−1) ∩ g(i) )).

i∈J

4.3 A Trace Formula Starting From the Root System We now prove a relative trace formula, which will turn out to be very useful in the sequel. Proposition 4 Let C be a composition sequence of g and h a subalgebra of g. Let

∈  generic such that the subspace  h = h + g( ) is Lagrangian for the alternating bilinear form B . We denote by b = b[ ] the Vergne polarization at determined by C. Let U = T + V be in h with T ∈ b and V ∈ n. Assume further that βi (U ) = 0, resp. βi (U ) = βi (U ) = 0, i ∈ J, i ≥ 2 with preceding notations. Then,

24

A. Baklouti et al.

Tr adg/h (U ) − Tr adg/b (T ) = 2



Tr α ji (T ).

i∈J

Proof We can assume that no root of g vanishes on g( |n ). We use the composition sequence C and the decomposition g j = w j + g j−1 , 1 ≤ j ≤ d. We choose w j in g( ) g( ) if j ∈ IC , and in b if j ∈ ICb . If h ∩ b ∩ g j ⊂ g j−1 , we determine a subspace j v in h ∩ b ∩ g j such that h ∩ b ∩ g j = v j + (h ∩ b ∩ g j−1 ). For i ∈ J , we take a subspace ui ∈ h ∩ gsi such that h ∩ gsi = ui + (h ∩ gsi −1 ) (and   eventually with si and ui such that h ∩ gsi = ui + (h ∩ gsi −1 )). Then, for Ui ∈ ui , i ∈ h∩b J and for Z i ∈ vi , i ∈ IC , we have ad(U )Ui =

 i  ∈J

ad(U )Z i =

μi  ,i (Ui ) +   

 i  ∈J

∈ ui 

σi  ,i (Z i ) +    ∈ ui 

 b∩h i  ∈IC

νi  ,i (Ui ),   



b∩h i  ∈IC

∈ vi 

τi  ,i (Z i ) .    ∈ vi 

Let us determine the vectors μi,i (Ui ), i ∈ J . The relations (11), where we choose Ci so that we have [Ui , Yi ] = α ji (Ui )Yi + βi (Ui )Ci , and the properties of U tell us that, for 1 < i ∈ J , we have

, [[U, Ui ], Yi ] =

, [U, [Ui , Yi ]] −

, [Ui , [U, Yi ]] =

, [U, α ji (Ui )Yi + βi (Ui )Ci ] −

, [Ui , α ji (U )Yi + βi (U )Ci ] mod ker( |g ji −1 ).

(13)

Now utilizing (11) and (12),

, [[U, Ui ], Yi ] = α ji (Ui )βi (U ) − α ji (U )

, [Ui , Yi ] mod ker( |g ji −1 ) = −α ji (U ).

If dim α ji = 2, we have in the same way:

, [[U, Ui ], Yi ] =

, [U, [Ui , Yi ]] −

, [Ui , [U, Yi ]] =

, [U, α ji (Ui )(Yi − ωYi ) + βi (Ui )Ci ] −

, [Ui , α ji (U )(Yi − ωYi ) + βi (U )Ci ] mod ker( |g ji −1 ) = −α ji (U ), and eventually

, [[U, Ui ], Yi ] = −α ji (U ). In this way, μi,i (Ui ) = −α ji (U )Ui .

(14)

Monomial Representations of Discrete …

25

The indices si , si (i ∈ J ) being minimal, we can agree that the vectors in ui have no h∩b component in the subspaces v j , j ∈ IC . Hence we see that τ j, j (Z j ) = α j (T )(Z j ), j ∈ IC

h∩b

.

(15)

Formulas (14) and (15) permit to write the trace formula in the following way. Tr adg/h (U ) − Tr adg/b (T ) = Tr adb (T ) − Tr adh (U )   = Tr α ji (T ) + Tr α j (T ) g( )

i∈{1,...,l}

+



j∈IC

Tr α ji (T ) −



Tr α i (T ).

h∩b

i∈J

i∈IC

This gives us: Tr adb (T ) − Tr adh (U ) = 2





Tr α ji (T ) +

i ∈J,i∈{1,...,l} /

i∈J

+



Tr α ji (T )



Tr α (T ) − j

g( ) j∈IC

Tr α j (T ).

(16)

h∩b j∈IC

 h∩b As

, [Ui , Yi ] = δi,i  for all i ∈ J , we have IC ∩ J = ∅. On the other hand, as  h is Lagrangian, we have the relations dim  h = dim b and



dim(h(i) /(h(i−1) ∩ g(i) )) = dim(h/(h ∩ b)) = dim( h/( h ∩ b)) = dim(b/( h ∩ b)).

i∈J

These equalities say that 

Tr α j (T ) =



Tr α ji (T ).

i ∈J,i∈{1,...,l} /

 (h∩b)/g( )

j∈IC

Moreover, by Lemma 2, 

Tr α j (T ) =

  h∩b

h∩b

j∈IC

=

g( )/(h∩g( )

Tr α j (T )

 h∩b j∈IC

=

Tr α j (T )

j∈IC

j∈IC





Tr α j (T ) −



 h∩b/g( ) j∈IC

Tr α j (T ) +

 g( ) j∈IC

Tr α j (T ).

26

A. Baklouti et al.

Finally, 

Tr α ji (T ) +

i ∈J,i∈{1,...,l} /



Tr α j (T ) =

g( ) j∈IC



Tr α j (T )

h∩b j∈IC



and so we obtain the result from (16).

5 A Proof of Convergence We keep our notations and prove in this section Theorem 2. Our proof will be carried out in different steps and, in each case, by an induction on p = dim G + dim G/H . If G is abelian or more generally nilpotent, the theorem is evident. If the ideal g0 of Definition 2 is different from g, then G 0 = exp(g0 ) is a normal subgroup of G containing the polarization B = exp(b[ ]) and the normal subgroup N = exp n. Besides, the modular functions G and G 0 coincide on G 0 . Hence, if H ⊂ G 0 , we have the identity 



−1/2

H/(H ∩B)

|ϕ(h)| H,G (h)dν(h) =

−1/2

H/(H ∩B)

|ϕ(h)| H,G 0 (h)dν(h).

Applying the induction hypothesis to (G 0 , H ), we have the result through Proposition 1. If h ⊂ g0 , we take a subspace u of h such that h = u ⊕ (h ∩ g0 ) and write as before h0 = h ∩ g0 , H0 = exp(h0 ). Then, 

−1/2

H/(H ∩B)

|ϕ(h)| H,G (h)dν(h)

 

−1/2

−1/2

|ϕ((exp U )h)| H,G (exp U ) H0 ,G 0 (h)dν(h)dU   −1/2 −1/2 |ϕ((exp U )h)| H0 ,G 0 (h)dν(h)dU. ≤  H,G (exp U )

=

u

H0 /(H0 ∩B)

u

H0 /(H0 ∩B)

This again permits us to apply the induction hypothesis, because the linear form

0 = |g0 is strongly generic and the function U →

−1/2  H,G (exp U )



−1/2

H0 /(H0 ∩B)

|ϕ((exp U )h)| H0 ,G 0 (h)dν(h)

is compactly supported and continuous by the dominated convergence theorem of Lebesgue. Indeed, we can make use of an another continuous function with compact support modulo B having the necessary covariance relation whose variables are separated in exp U and h ∈ H0 and the modulus dominates |ϕ|.

Monomial Representations of Discrete …

27

Thus, we can suppose that the roots of g do not vanish on g( |n ). Recall the composition sequence C = (g j )1≤ j≤d of g which passes through n = log N containing [g, g] and we utilize the preceding constructions and notations. Let us prove the following stronger assertion: for any generic ∈  and any ϕ ∈ K (G, B, ), we have  sup δ ,S (s)

s∈B[ ] m∈N

−1/2

H/(H ∩B)

|ϕ(mshs −1 )| H,G (h)dν(h) < ∞.

(17)

Everything is clear if p = 1 or if N ⊂ G( ), namely j1 > k. In this case, the function δ ,S is the constant 1 and [G, G] ⊂ N ⊂ G( ) ⊂ B. Hence H ∩ B is normal in H , H B is a closed subgroup of G and  sup δ ,S (s)

s∈B[ ] m∈N

−1/2

H/(H ∩B)

|ϕ(mshs −1 )| H,G (h)dν(h) =



−1/2

H/(H ∩B)

|ϕ(h)| H,G (h)d(h)

< ∞.

(18)

So, we can suppose j1 ≤ k. a. We can suppose g j1 −1 ⊂ h. Indeed, if this is not the case, we take h1 = h + which is also subordinate to and put H1 = exp(h1 ). The trace formula (7) g holds also for (b, h1 ), because [g( |n ), g j1 −1 ] ⊂ h as is generic and g j1 −1 ⊂ n. On the other hand,  H,G (h) =  H1 ,G (h)χ (h)(h ∈ H ), where χ is a real character of G verifying χ (exp U ) = eTr ada/(a∩h) (U ) , U ∈ h, j1 −1

with a = g j1 −1 and χ | B = 1 (as is generic). Then, the function χ −1/2 ϕ belongs to the space of the representation π . Besides, 

−1/2

H/(H ∩B)

|ϕ(mshs −1 )| H,G (h)dν(h)



=

−1/2

H1 /(H1 ∩B)

|χ −1/2 (mshs −1 )ϕ(mshs −1 )| H1 ,G (h)dν(h),

which essentially gives us the desired property. In the cases (3) and (5), the subalgebra g(1) is an ideal of g containing [g, g] + b[ ]. Therefore, G (1) = exp(g(1) ) is a normal subgroup of G containing the polarization B = exp(b[ ]) and the normal subgroup N (1) = exp(n(1) ). Besides, the modular functions G and G (1) coincide on G (1) . In the cases where H ⊂ G (1) , we hence have the identity

28

A. Baklouti et al.

 S := sup δ ,S (s)

−1/2

H/(H ∩B)

s∈B[ ] m∈N

|ϕ(mshs −1 )| H,G (h)dν(h)



−1/2

= sup δ ,S (1) (s)

H (1) /(H (1) ∩B)

s∈B[ ] m∈N

|ϕ(mshs −1 )| H,G (1) (h)dν(h).

Applying the induction hypothesis to (G (1) , H = H (1) ), we get the result. If h ⊂ g(1) , we take a subspace u of dimension 1 or 2 in h such that h = u ⊕ (h ∩ g(1) ) and we write as before h(1) = (h ∩ g(1) ) + g j1 and n(1) = n ∩ g(1) . Here we treat only the case where u is of dimension 2 and u = RA + RX with X ∈ n. We then have 

 S =

sup s∈B[ ] m (1) ∈N (1) ,x∈R

δ ,S (s)

R2

−1/2

H (1) /(H (1) ∩B)

 H,G (exp(a A)) −1/2

|ϕ(exp(x X )m (1) s exp(x  X ) exp (a A)h (1) s −1 )| (1) (1) (h (1) )dν(h (1) )dad x  H ,G    −1/2 ≤ sup  H,G (exp(a A)) sup δ ,S (1) (s) x∈R R2

H (1) /(H (1) ∩B)

s∈B[ ] m (1) ∈N (1)

|ϕ(exp(x X )m (1) s exp(x  X ) exp (a A)h (1) s −1 )|



−1/2 (h (1) )dν(h (1) ) dad x  . H (1) ,G (1)

For m (1) ∈ N (1) , x  , a ∈ R, s ∈ B[ ], h (1) ∈ H (1) , we write m (1) s exp(x  X ) exp(a A)h (1) s −1 = exp(x  X ) exp(a A)b(x  , a, s, m (1) )m (1) sh (1) s −1 with b(x  , a, s, m (1) ) ∈ [G, G] ⊂ N (1) . Since G is the topological product of R2 with G (1) , we can take a function ϕ1 ∈ Cc (G/G (1) ) and a function ϕ2 ∈ K (G (1) , B, f ) so that |ϕ(ex p(x X ) exp(a A)g (1) )| ≤ |ϕ1 (exp(x X ) exp(a A))||ϕ2 (g (1) )|, g (1) ∈ G (1) , x, a ∈ R.

This again allows us to apply the induction hypothesis to (G (1) , H (1) ) and we have  S ≤ sup x∈R



R2

−1/2  H,G (exp(a A))

 sup δ ,S (1) (s)

s∈B[ ] m (1) ∈N (1)

H (1) /(H (1) ∩B)

|ϕ(exp(x X ) exp(x  X )

 −1/2 × exp(a A)b(x  , a, s, m (1) )m (1) sh (1) s −1 )| H (1) ,G (1) (h (1) )dν(h (1) dad x     −1/2 ≤ sup  H,G (exp(a A))|ϕ1 (exp((x + x  )X ) exp(a A))|d x  da x∈R R2   |ϕ2 (b(x  , a, s, m (1) )m (1) sh (1) s −1 )| × sup δ ,S (1) (s) s∈B[ ] m (1) ∈N (1) ,x∈R

−1/2

H (1) /(H (1) ∩B)

×  H (1) ,G (1) (h (1) )dν(h (1) )



Monomial Representations of Discrete …

29



 −1/2  H,G (exp(a A))|ϕ1 (exp((x + x  )X ) exp(a A))|d x  da x∈R R2    −1/2 |ϕ2 (m (1) sh (1) s −1 )| H (1) ,G (1) (h (1) )dν(h (1) ) × sup δ ,S (1) (s) ≤



sup

H (1) /(H (1) ∩B)

s∈B[ ] m (1) ∈N (1)

< ∞.

It remains to study the cases (4) and (6). We consider the first index j1 such that g j1 ⊂ g( ). We already know that j1 > 1. We begin with the case: j −1

b. Case where dim(α j1 ) = 1. Take Y1 ∈ g j1 \ g j1 −1 . Let g01 = g j1 −1 ∩ ker j −1 and C1 ∈ g j1 −1 \ g01 if |g j1 −1 = 0. As j1 > 1 we can always suppose (Y1 ) = 0. We then have a linear form β1 : g → R such that j −1

[A, Y1 ] ≡ α j1 (A)Y1 + β1 (A)C1 mod g01 , A ∈ g. Let V1 ∈ g verifying β1 (V1 ) = 1. With the notations of Sect. 4.2, put g(1) = ker β1 . We have the relation [T, V1 ] = −α j1 (T )V1 mod g(1) . Starting from the sequence (g j ∩ g(1) ) j , we construct a composition sequence C (1) and a good sequence of subalgebras S (1) of g(1) . Let be generic, then as we saw in the proof of Proposition 2, (1) is generic in (g(1) )∗ and bS (1) [ (1) ] = bS [ ] ⊂ g(1) . b.1. Case where h ⊂ g(1) . Set h(1) = h + RY1 . Let g0 = ker(α j1 ) ∩ g(1) which is an ideal of codimension 1 in g(1) , h0 = h ∩ g0 and h(1) 0 = h0 + RY1 . Clearly, h0 is an ideal of h and of h(1) . On the other hand, as the root α j1 vanishes on h0 , we have 0 −1/2

 H0 ,G (h 0 ) = e1/2α

j1 (log(h 0 ))

−1

−1

−1

−1/2

2  H02,G (1) (h 0 ) =  H02,G (1) (h 0 ) = G (1) (h ) H0 ,G (1) (h 0 ) ,G 0

−1/2

for h 0 ∈ H0 = exp(h0 ). For ϕ ∈ K (G, B, ), the function (G (1) ,G ϕ)|G (1) is an element of K (G (1) , B, (1) ). As n ⊂ g(1) , we write n = RX + n(1) , N (1) = exp(n(1) ), where n(1) = n ∩ g(1) and X ∈ g \ g(1) . Then G = exp(RX )G (1) and we can write every m ∈ N as m = exp(x X )m  with x ∈ R and m  ∈ N (1) . Let H0(1) = exp(h(1) 0 ). (1) (1) If B[ ] ∩ H ⊂ G 0 = exp(g0 ), then h(1) + g ( ) is Lagrangian and we have 0 

−1/2

H/H ∩B

|ϕ(mshs −1 )| H,G (h)dν(h) = ×



−1/2 |G (1) ,G ϕ(exp(x X )m  sh 0 s −1 )| H0 /H0 ∩B −1/2  H0 ,G (1) (h 0 )dν(h 0 )

30

A. Baklouti et al.

 =

−1/2

H0(1) /H0(1) ∩B

|G (1) ,G ϕ(exp(x X )m  sh 0 s −1 )|

−1/2 (h 0 )dν(h 0 ). H0(1) ,G (1)

×

This allows us to conclude by use of the induction hypothesis. Indeed, we take a function ψ ∈ K (G, B, ) whose modulus dominates the modulus of ϕ and whose variables are separated in x ∈ R and g  ∈ G (1) . In this way, we see that the root associated to U ∈ h \ ker(α j1 ) has no influence on the expression of δ . So, we can apply the induction hypothesis to G (1) , H0(1) , (1) for ψ to get also the result for ϕ. Suppose now that B ∩ H ⊂ G 0 and H ⊂ G 0 . Then it is clear that  H,G =  H (1) ,G (1) G (1) ,G and G , G (1) coincide on H (1) ∩ B. Using these facts, we see that the function −1/2 h  → |ϕ(h  )| H (1) ,G (1) (h  ) is in the space K (H (1) , H (1) ∩ B) and satisfies from Proposition 3 the necessary trace property. Hence our integral converges by similar arguments as in the precedent cases. The case where H ⊂ G 0 is easier and is similarly treated. Now, we hereafter suppose h ⊂ g(1) . Put h(1) = h ∩ g(1) + g j1 . Using the notations of Sect. 4.2, we choose an element U = U1 ∈ h verifying β1 (U ) = 1 and βi (U ) = 0 (resp. βi (U ) = βi (U ) = 0) for any i > 1, i ∈ J . b.2. Case where h ⊂ g(1) and α j1 vanishes on h(1) . We have a vector U = U1 ∈ h \ h(1) which is written, if U is not in n, as U = T + V with T ∈ b[ ] and V ∈ [g, g] such that [T, V ] ≡ −α j1 (T )V modulo g(1) . We therefore obtain a decomposition g = RV ⊕ g(1) and G = exp(RV )G (1) . In the same way, we have  e−uα 1 (T ) − 1  exp(uU ) = exp V n(u) exp(uT ), u ∈ R, −α j1 (T ) j

where n(u) is an analytic upon u and it belongs to G (1) ∩ [G, G] and e−α 1 (uT ) − 1 −α j1 (T ) j

must be replaced simply by u if α j1 (T ) = 0. From Proposition 4 we have −1/2

1/2

 B,G (exp(uT )) H,G (exp(uU )) = δ (exp(uU )). On the other hand, we have  H (h) =  H (1) (h), G (h) = G (1) (h) for h ∈ H ∩ G (1) . Thus, the transitivity rule of ν·,· gives us

Monomial Representations of Discrete …





−1/2

H/H ∩B

31

ϕ(h) H,G (h)dν(h) =

−1/2

R

 H,G (exp(uU ))

 ×  = ×

−1/2

H (1) /(H (1) ∩B)

ϕ(exp (uU )h  ) H,G (h  )dν(h  )du

−1/2

R



 H,G (exp(uU )) −1/2

H (1) /(H (1) ∩B)

ϕ(exp (uU )h  ) H (1) ,G (1) (h  )dν(h  )du.

for continuous functions ϕ on H with compact support modulo B ∩ H . Hence we obtain, for ϕ ∈ K (G, B, ),  K (s, m) = δ (s)  = δ (s) = e−α

−1/2

H/(H ∩B)

R

|ϕ(mshs −1 )| H,G (h)dν(h)



du

j1 (log s)

−1/2

H (1) /(H (1) ∩B)



δ (1) (s)

R



du

−1/2

|ϕ(ms exp(uU )h  s −1 )| H,G (exp(uU )) H (1) ,G (1) (h  )dν(h  ) H (1) /(H (1) ∩B)

j    e−α 1 (uT ) − 1 −α j1 (log s)    × ϕ m exp V n(s, u) exp(uT )sh  s −1 exp(−uT )  e j 1 −α (T )

−1/2

−1/2

×  H,G (exp(uU )) B,G (exp(uT )) H (1) ,G (1) (h  )dν(h  ) 1/2

with n(s, u) ∈ N ∩ G (1) . As an immediate consequence of these relations, we obtain 

 K (s, m) =

R

du δ (1) (s)

H (1) /(H (1) ∩B)

j    e−α 1 (uT ) − 1 −α j1 (log s)     −1 e × ϕ m exp V n(s, u) exp(uT )sh s exp(−uT )  −α j1 (T )

e−α

j1 (uT )

e−α

j1 (log s)

−1/2

δ (1) (exp(uT )) H (1) ,G (1) (h  )dν(h  ).

As we can suppose that α j1 (T ) = 0 so that we can choose T such that α j1 (T ) = −1. In such a case, we obtain, for m = exp(x V )m  with x ∈ R and m  ∈ N (1) , 



dv δ (1) (s) J H (1) /(H (1) ∩B)     × ϕ exp((x + v)V )n(s, f (v), m) exp( f (v)T )sh  s −1 exp(− f (v)T ) 

K (s, m) =

−1/2

× δ (1) (exp( f (v)T )) H (1) ,G (1) (h  )dν(h  ), where J is an interval of R, n(s, f (v), m) ∈ N (1) and f (v) = log(1 + veα 1 (log s) ) which is well defined on J . In this way, j

32

A. Baklouti et al.



 sup K (s, m) ≤ s∈B( ) m∈N

sup s∈B( ) r ∈R, m  ∈N (1)

R

dv δ (1) (s exp(r T ))

H (1) /(H (1) ∩B)

   −1/2  × (G (1) ,G ϕ) exp(vV )n(s, r, m  ) exp(r T )sh  s −1 exp(−r T )  −1/2

×  H (1) ,G (1) (h  )dν(h  ) < ∞, applying the induction hypothesis to g(1) , h(1) and (1) . b.3. Case where α j1 is non-zero on h(1) . In this case we can suppose U ∈ n. Write j  G = exp(RU )G (1) . Keeping the same notations, we have  H (h  ) = e2α 1 (log h )  H (1) j1  (h  ) and G (h  ) = eα (log h ) G (1) (h  ) for any h  ∈ H (1) and hence  H,G (h  ) = eα

j1 (log h  )

 H (1) ,G (1) (h  ).

For any continuous function ϕ on H with compact support modulo B ∩ H , the transitivity formula of ν·,· gives us 

 

−1/2

H/H ∩B

ϕ(h) H,G (h)dν(h) =

eα

j1 (log h  )

H (1) /(H (1) ∩B)

R

ϕ(exp (uU )h  )

−1/2

×  H,G (h  )dν(h  )du   1 j1  = e 2 α (log h ) ϕ(exp (uU )h  ) H (1) /(H (1) ∩B) −1/2  H (1) ,G (1) (h  )dν(h  )du R

× = ×

 

−1/2 (G (1) ,G ϕ)(exp (uU )h  ) H (1) /(H (1) ∩B) −1/2  H (1) ,G (1) (h  )dν(h  )du. R

So, for ϕ ∈ K (G, B, ) we obtain with m = exp(xU )m  , x ∈ R, m  ∈ N (1) :  δ (s)

−1/2

H/(H ∩B)

|ϕ(mshs −1 )| H,G (h)dν(h) = ×

 R

du e−α

j1 (log s)

 δ (1) (s)

−1/2 |(G (1) ,G ϕ)(exp((x

H (1) /(H (1) ∩B)

+ ue−α

j1 (log s)

)U )

−1/2

× m  (s, u, m)sh  s −1 )| H (1) ,G (1) (h  )dν(h  )   = du δ (1) (s)

H (1) /((H (1) ∩B)) −1/2 |(G (1) ,G ϕ)(exp((x + u)U ) −1/2 m  (s, u, m)sh  s −1 )| H (1) ,G (1) (h  )dν(h  ), R

× ×

Monomial Representations of Discrete …

33

with m  (s, u, m) ∈ N (1) . Therefore, we get the result by applying the induction hypothesis to g(1) , h(1) and (1) . c. Case where dim(α j1 ) = 2. Exactly as before, we have two vectors Y1 , Y1 ∈ (g ∩ ker ) \ g j1 −1 , a vector C1 ∈ g j1 −1 verifying (C1 ) = 1 and two linear forms β1 , β1 such that j1

[A, Y1 ] ≡ α j1 (A)(Y1 − ωY1 ) + β1 (A)C1 [A, Y1 ] = α j1 (A)(Y1 + ωY1 ) + β1 (A)C1 modulo g j1 −1 ∩ ker for A ∈ g. Moreover, the linear forms β1 , β1 are linearly independent with α j1 . Then, we see that the situation where h ⊂ g(1) = ker(β1 ) ∩ ker(β1 ) is treated similarly to the situation b.1. Hence, suppose that h ⊂ g(1) . If the root α j1 is non-zero on h(1) , then dim(h/(h ∩ g(1) )) = 2 and we can choose two linearly independent vectors U1 , U1 in h ∩ n outside of g(1) . Then,  H,G (h  ) = j  j  e2α 1 (log h )  H (1) ,G (1) (h  ) and G (1) ,G (h  ) = e−2α 1 (log h ) for any h  ∈ H (1) . On the other hand, we have j1 δ (s) = δ (1) (s)e−2α (log s) for s ∈ B[ ]. This is clearly sufficient to compute the terms coming out from the integral, in which we consider the two variables coming from the vectors U1 , U1 . If the root α j1 vanishes on h(1) , then dim(h/(h ∩ g(1) )) could be 1 or 2. In the case where this dimension is 2, δ (s) = δ (1) (s)e−2α

j1 (log s)

for all s ∈ B[ ]. This compensates the change of variables at the level of the expression j j  e−uα 1 (T1 ) − 1 −α j1 (log s)   e−uα 1 (T2 ) − 1 −α j1 (log s)  e e exp V1 exp V2 , −α j1 (T1 ) −α j1 (T2 ) where Ui = Ti + Vi , Ti ∈ b, Vi ∈ [g, g] for i = 1, 2. While, when dim(h/(h ∩ g(1) )) = 1, we have j1 δ (s) = δ (1) (s)e−α (log s) by Definition 4. Therefore, we see that everything goes just as in the case b.2.

6 Concrete Plancherel Formula Consulting the references [14–16], we would like to describe explicitly the Plancherel formula for τ = indGH χ f , supposed always to be of discrete type. We propose to prove:

34

A. Baklouti et al.

Theorem 3 We keep the same hypotheses and notations. Fixing from now on a good sequence of subalgebras, we can choose the measure μ used in the of  disintegration k k

π(ϕ)a , a τ , supposed to be of discrete type, in such a manner that the sum m(π) π π k=1 is well defined μ-almost everywhere and we have the formula f ϕ H (e)

=

 m(π)   G k=1

π(ϕ)aπk , aπk dμ(π )

for all ϕ ∈ Cc∞ (G). The proof of this theorem will be divided in several independent steps.

6.1 Computing the Matrix Coefficients f

Let ∈  be strongly generic. For ψ ∈ K (G, B, ), the function ψ H,B is continuous f by Corollary 1 and we have ϕ H ∈ K (G, H, f ) for any ϕ ∈ Cc∞ (G). This means 

 G/H

f |ϕ H (g)|dνG,H (g)

−1/2

H/(H ∩Bk )

|ψ(ghgk )| H,G (h)dν H,H ∩Bk (h) < +∞.

Using of Fubini’s theorem, we get  ϕ(g) aπk , π(g −1 )ψdg

π(ϕ)aπk , ψ = G     dνG,H (g) ϕ(gh  )−1 = H,G (h )dh G/H H  −1/2 × ψ(gh  hgk )χ f (h) H,G (h)dν H,H ∩Bk (h) H/(H ∩Bk )   f −1/2 ϕ H (g)dνG,H (g) ψ(ghgk )χ f (h) H,G (h)dν H,H ∩Bk (h) = G/H H/(H ∩Bk )  f ϕ H (g)ψ(ggk )dνG,H ∩Bk (g) = G/(H ∩Bk )   f dνG,B (g) ϕ H (gbgk−1 )ψ(gb)−1 = B,G (b)dν B,B∩gk−1 H gk (b) G/B



=

B/(B∩gk−1 H gk )

−1/2

ϕ H (gbgk−1 )χ (b) B,G (b)dν B,B∩gk−1 H gk (b), ψ. f

B/(B∩gk−1 H gk )

Since ψ ∈ Hπ∞ has a compact support modulo B, these calculations have a sense. Indeed, using Theorem 2,

Monomial Representations of Discrete …

35

 f

|ϕ H (g)ψ(ggk )|dνG,H ∩Bk (g)   f dνG,H (g) |ϕ H (gh)ψ(ghgk )|−1 = H,G (h)dν H,H ∩Bk (h) G/H H/(H ∩Bk )   f −1/2 = |ϕ H (g)|dνG,H (g) |ψ(ghgk )| H,G (h)dν H,H ∩Bk (h) < +∞. G/(H ∩Bk )

H/(H ∩Bk )

G/H

Therefore, 

−1/2

ϕ H (gbgk−1 )χ (b) B,G (b)dν B,B∩gk−1 H gk (b). f

(π(ϕ)aπk )(g) =

B/(B∩gk−1 H gk )

As an immediate consequence of these observations, we can deduce that

π(ϕ)aπk , aπk  is formally written as follows: 

π(ϕ)aπk , aπk  =



=

−1/2

H/(H ∩Bk )

−1/2

H/(H ∩Bk )



π(ϕ)aπk (h)χ f (h) H,G (h)dν H,H ∩Bk (h) χ f (h) H,G (h)dν H,H ∩Bk (h)

B/(B∩gk−1 H gk )

 =  ×

−1/2

ϕ H (hgk bgk−1 )χ (b) B,G (b)dν B,B∩gk−1 H gk (b) f

×

−1/2

H/(H ∩Bk )

χ f (h) H,G (h)dν H,H ∩Bk (h) f

Bk /(Bk ∩H )

−1/2

ϕ H (hb)χgk · (b) Bk ,G (b)dν Bk ,Bk ∩H (b).

In the sequel we intend to give a meaning to these matrix coefficients by using the induction and intertwining operators. We shall attack case by case every step of the induction. Besides, as in [10], every ϕ ∈ Cc∞ (G) is written as a finite linear combination of functions of type ψ ∗ ∗ψ for some ψ ∈ Cc∞ (G) and

π(ψ ∗ ∗ψ)aπk , aπk  = π(ψ)aπk 2 ≥ 0. This allows us to establish equalities between certain integrals in what follows.

6.2 An Intertwining Formula Here we suppose that there exists a vector subspace g of codimension 1 in g which contains n + g( ) + h for all in a non-empty Zariski open set V of , where as before n is a nilpotent ideal of g containing [g, g]. Let j0 ≤ k0 ≤ n be the smallest

36

A. Baklouti et al.

index k verifying gk ⊂ g . Cutting the sequence S at the level of g , we have the good sequence: S  : g1 ⊂ · · · ⊂ gk0 −1 ⊂ gk0 ⊂ · · · ⊂ gn−1 = g ⊂ g, where gk = gk+1 ∩ g for k0 ≤ k ≤ n − 1. Let b = b [ ] = bS  [ 0 ] for ∈ V where

0 = |g ∈ (g )∗ and B  = exp(b ). We put G  = exp(g ). Suppose first that b[ ] ⊂ g generically in . In this case we have bS [ ] = bS  [ 0 ]  for almost every ∈ V, τ  = indGH χ 0 has multiplicities of discrete type, G  is a normal subgroup of G and the modular functions G , G  coincide on G  . The  Penney’s distributions aπ for π and aπ  for π  = indGB χ defined by (9) are the same. Now, we are in the case where b[ ] ⊂ g generically in . Writing g ( 0 ) = RX ( ) + g( ), we see that b [ 0 ] = RX ( ) + (b[ ] ∩ g ) and hence

b [ 0 ] = RX ( ) + (b[ ] ∩ b [ ]).

Since h + g( ) is a Lagrangian subspace, the element X ( ) belongs to h + g( ). So we write X ( ) = T ( ) + V ( ) with T ( ) ∈ h and V ( ) ∈ g( ). Remark that T ( ) ∈ / n because b[ ] ⊂ g . Fixing for the moment, write simply X, T, V instead of X ( ), T ( ), V ( ). Let π = indGB χ and π  = indGB χ . Then, as b [ ] = RT + (b[ ] ∩ b [ ]), the intertwining operator R from π to π  is given by  (Rϕ)(g) =

ϕ(g exp(t T ))eit (T ) e− 2 Tr adg/b [ ] T dt t

R

for ϕ ∈ Hπ continuous with compact support modulo B. The following lemma allows us to conclude our proof of convergence in this context because R is an intertwining isomorphism. Lemma 6 Let aπ (resp. aπ  ) be the Penney’s distribution for π (resp. π  ) defined as in (9). Then aπ = aπ  ◦R. Proof Since B (T, [g, g]) = {0} and [T, g] ⊂ g , using the trace formula (7), we have Tr adg/b [ ] T = −Tr adb [ ]/g ( 0 ) T = −Tr adb [ ]/(b [ ]∩(h+g ( 0 ))) T − Tr ad(b [ ]∩(h+g ( 0 )))/g ( 0 ) T = Tr adh/(h∩b [ ]) T − Tr ad(h∩b [ ])/(h∩g ( 0 )) T. Therefore by the transitivity property of ν·,· ,

Monomial Representations of Discrete …   ≤

H/(H ∩B  )

37

  − t Tr adg/b [ ] T −1/2 | ϕ(h exp(t T ))e−it (T ) e 2 dt |χ (h)| H,G (h)dν(h) R



H/(H ∩B  ) R

|ϕ(h exp(t T ))|e

− 2t Tr adg/b [ ] T

−1/2

dt H,G (h)dν(h)

  t Tr ad  [ ]) T +Tr ad(h∩b [ ])/(h∩g ( )) T −1/2 h/(h∩b 2 0 |ϕ(exp(t T )h)| H,G (h)dν(h)e dt R H/(H ∩B  )

  =

  =

  =  =

t Tr ad h/(h∩g ( 0 )) T dt

−1/2

R H/(H ∩B  )

|ϕ(exp(t T )h)| H,G (h)dν(h)e 2 −1/2

R H/(H ∩B  )

t

|ϕ(exp(t T )h)| H,G (h)dν(h)e− 2 Tr adg/h T dt −1/2

H/(H ∩B)

|ϕ(h)| H,G (h)dν(h)

< ∞.

Hence the function R(ϕ) is in the domain of definition of the distribution aπ  and 

 (aπ  ◦R)(ϕ) =

H/(H ∩B  )

R

ϕ(h exp(t T ))e−it (T ) e

− 2t Tr adg/b [ ] T

−1/2

dtχ (h) H,G (h)dν(h).

This means that  (aπ  ◦R)(ϕ) =

 H/(H ∩B  )

ϕ(h exp(t T ))e−it (T ) e− 2 (Tr adh/(h∩b [ ]) T ) t

R

−1/2

t

× e 2 Tr ad(h∩b [ ])/(h∩g ( 0 ))) T dtχ (h) H,G (h)dν(h)   −1/2 = ϕ(exp(t T )h)χ (h) H,G (h)dν(h) R

H/(H ∩B  )

−it (T )

× e  = R

R

t

−1/2

H/(H ∩B  )

−it (T )

× e  =

e 2 (Tr adh/(h∩b [ ]) T +Tr ad(h∩b [ ])/(h∩g ( 0 )) T ) dt ϕ(exp(t T )h)χ (h) H,G (h)dν(h)

t

e 2 Tr adh/(h∩g ( 0 )) T dt −1/2

H/(H ∩B  )

ϕ(exp(t T )h)χ (h) H,G (h)dν(h)

× e−it (T ) e− 2 Tr adg/h T dt t

=

H/(H ∩B)

−1/2

ϕ(h)χ (h) H,G (h)dν(h) = aπ (ϕ). 

38

A. Baklouti et al.

6.3 Proof of Theorem 3 We proceed by induction on m = dim G + dim G/H and follow closely the proof of the disintegration of τ given in [23]. If m = 1, everything is evident and we suppose that m > 1. Since g is solvable, it contains at least an ideal g0 containing [g, g] and of codimension 1 in g. Put G 0 = exp(g0 ). We shall treat separately the following cases. 3.1. Case where g0 is an ideal of saturation containing h. In this case we have almost everywhere saturated orbits with respect to g0 . Put f 0 = p( f ) = f |g0 ∈ g∗0 and 0 = f 0 + h⊥,g0 = { 0 ∈ g∗0 ; 0 |h = f 0 |h }. Since h + g( ) is almost everywhere Lagrangian, the representation τ0 = indGH0 χ f0 is of discrete type. Let  ⊕

τ0 

0 G

m 0 (π0 )π0 dμ0 (π0 )

be the canonical central decomposition of τ0 . The subgroup G 0 being normal, G acts  is identified with G 0 /G. Besides, 0 by usual way and up to a μ-negligible set G on G the disintegration of μ0 relative to μ, which is the image of μ0 by the projection 0 /G leads to 0 → G q:G  τ  indG G 0 τ0   =

⊕  G

⊕

0 G ⊕

 dμ(π )

  m 0 (π0 ) indG G 0 π0 dμ0 (π0 )

Gˆ0

 π  m 0 (π0 ) indG G 0 π0 dμ0 (π0 ),

where the fiber measure μπ0 is carried on q −1 (π ). From what we have seen until now, the representation  ⊕  π  m 0 (π0 ) indG G 0 π0 dμ0 (π0 ) 0 G



is factorial and m(π )π 

⊕ 0 G

 π  m 0 (π0 ) indG G 0 π0 dμ0 (π0 )

for μ-almost all π . It follows that τ = indGH χ f is of discrete type if and only if τ0 = indGH0 χ f is carried on discrete points such that m 0 (π0 ) are of discrete type. These are hence given by the number of the connected components of 0 ∩ ω(π0 ), ω(π0 ) denoting the G 0 -orbit corresponding π0 . From these observations we deduce that q −1 (π ) is a discrete set and

Monomial Representations of Discrete …

39



m(π ) =

m 0 (π0 ).

π0 ∈q −1 (π)

Thus, for each connected component C of  ∩ (π ), p(C) coincides with a connected component of 0 ∩ ω(π0 ) for a certain π0 ∈ q −1 (π ). This component being 0 )) , C is a manifold of desired dimension. a manifold of dimension dim(ω(π 2 Let us return to the proof of the Plancherel formula (2). Going back to the Sect. 6.2,  ( ∈ ). Then, taking we take the Vergne polarization b ⊂ g0 realizing π = π ∈ G into account the expression of the coefficients π(ϕ)aπk , aπk  as in the Sect. 6.1, we see that these coefficients exist by applying directly the induction hypothesis at the level of G 0 for the coefficients π0 (ϕ)aπk 0 , aπk 0 . Indeed, 

π(ϕ)aπk , aπk =

−1/2

H/(H ∩Bk )



χ f (h) H,G (h)dν H,H ∩Bk (h) −1/2

f

×

Bk /(Bk ∩H )

 = 

ϕ H (hb)χgk · (b) Bk ,G (b)dν Bk ,Bk ∩H (b) −1/2

H/(H ∩Bk )

χ f (h) H,G 0 (h)dν H,H ∩Bk (h) f

×

Bk /(Bk ∩H )

−1/2

ϕ H (hb)χgk · (b) Bk ,G 0 (b)dν Bk ,Bk ∩H (b),

which converges by the induction hypothesis, where ϕ = ϕ|G 0 . Then from what we have observed, we are able to interpret the Plancherel formula for τ0 which is supposed by the induction hypothesis and described for the test function ϕ as our searched (2). More precisely, the induction hypothesis assures us that the  0formula (π0 ) k k

π (ϕ)a sum m 0 π0 , aπ0  has a sense μ0 -almost everywhere and the Plancherel k=1 formula  m 0 (π0 ) f

π0 (ϕ)aπk 0 , aπk 0 dμ0 (π0 ) ϕ H (e) = 0 G k=1

holds. As μ0 is described by the integral decomposition f

ϕ H (e) =

 m(π)   G k=1

⊕  G

μπ0 dμ(π ), we arrive to

π(ϕ)aπk , aπk dμ(π ).

This allows us to conclude in this case. 3.2. Case where h + [g, g] = g. Let g0 be an ideal of codimension 1 in g containing h + [g, g] and G 0 = exp(g0 ). If g0 is an ideal of saturation, we are in the preceding case 3.1 which has been settled. So we attack the case where almost all  the restriction the orbits are non-saturated with respect to g0 . For almost all π ∈ G,  is identified with π0 of π to G 0 is irreducible, m(π ) = m(π0 ) and the Borel space G

40

A. Baklouti et al.

0 × R up to a negligible set. The induction hypothesis assures the existence of a G measure μ0 which establishes the concrete Plancherel formula for τ0 . Then, under our identification, the measure μ = μ0 × dλ, dλ being a Lebesgue measure on R will suit us. In fact, we can take gk in G 0 and at every point

∈  such that p(

) = 0 = gk · |g0 ∈ g∗0 ,

) = we can take an element X = X ( 0 ), depending on 0 in g so that we have g(

(X )) supplies our identification RX ⊕ g0 ( 0 ) and the Borelian mapping π → (π0 ,  and G 0 × R. Besides, it is possible to suppose that Bk = exp(RX )gk B0 gk−1 between G with B0 = exp(b0 ), b0 being the Vergne polarization of g0 at |g0 ∈ g∗0 . In this way, the homogeneous space Bk /(Bk ∩ H ) is isomorphic to the product of R and the homogeneous space B0k /(B0k ∩ H ), where B0k = gk B0 gk−1 . Writing b ∈ Bk as b = exp(x X )b0 with x ∈ R and b0 ∈ B0k , we have 

π(ϕ)aπk , aπk =

−1/2

H/(H ∩Bk )



χ f (h) H,G (h)dν H,H ∩Bk (h) −1/2

f

×

Bk /(Bk ∩H )

 =

−1/2

H/(H ∩B0k )

 ×

ϕ H (hb)χgk · (b) Bk ,G (b)dν Bk ,Bk ∩H (b)

B0k /(B0k ∩H )

χ f (h) H,G 0 (h)dν H,H ∩B0k (h)  −1/2 f  B k ,G (exp(x X ))ϕ H (h exp(x X )b0 )χgk · (b0 ) R

0

0

−1/2 χ (exp(x X )) B k ,G (b0 )d xdν B0k ,B0k ∩H (b0 ) 0 0



f

=

B0k /(B0k ∩H )

−1/2

ϕ H (hb0 )χgk · (b0 ) B k ,G (b0 )dν B0k ,B0k ∩H (b0 ), 0

0

which converges by the induction hypothesis, where  ϕ (g1 ) =

R

−1/2

 B k ,G (exp(x X ))ϕ(g1 exp(x X ))χ (exp(x X ))d x. 0

0

This allows us to conclude that the coefficients π(ϕ)aπk , aπk  exist from their expressions as in the Sect. 6.1. Now, putting λ =

(X ), we can get the formula (2) with the help of the Fourier inversion formula for R applied between d x and dλ, in order to conclude the concrete Plancherel formula for τ . 3.3. Case where h + [g, g] = g. In this situation, it turns outthat, μ-almost evk k erywhere, the multiplicities of τ are finite and therefore the sum m(π) k=1 π(ϕ)aπ , aπ  has a meaning. It is left for us to establish the Plancherel formula. Remark that it suffices to show the formula (2) for h containing the center z of g. Otherwise, let Z be a central element which is not contained in h. Put h = h + RZ and H (1) = exp(h ). For arbitrary λ ∈ R, let f λ ∈  such that f λ (Z ) = λ and

Monomial Representations of Discrete …

41

˜ ) = λ}. λ = { ˜ ∈ ; (Z Hence we see here that the coefficients π(ϕ)aπk , aπk  exist from their expressions as in Sect. 6.1. Suppose the concrete Plancherel formula for τ  = indGH (1) χ fλ described on λ with a measure μλ . We integrate this formula with respect to the measure dλ f f in order to pass from ϕ Hλ(1) to ϕ H , applying there the Fourier inversion formula for R. Next, we rewrite the right member, passing from Bk /(Bk ∩ H (1) ) to Bk /(Bk ∩ H ) and from λ to  by the inversion formula once again. In other respects, if h contains an ideal a of g on which f vanishes, it suffices to apply the induction hypothesis to the quotient group G/ exp a. All these points will be supposed in the sequel. It follows that dim z ≤ 1. Let a be a minimal non-central ideal. We find that a is abelian and dim a ≤ 3. The adjoint action of g on a/(a ∩ z) gives us a root or its real part λ ∈ g∗ and there always exists X ∈ h verifying λ(X ) = 1. The kernel of λ, denoted by g0 , is an ideal of g and we have g = RX ⊕ g0 . Now we put A = exp a, G 0 = exp(g0 ) and go to examine different cases. (I) Suppose that a ∩ z = {0}. We find h ∩ a = {0}. Let h = h ∩ g0 + a and (1) H = exp(h ). Reasoning at the level of the subgroup K = exp k, k = h + a, we see immediately that the situation is entirely similar to one realized in the examples of small dimensions examined in [23]. If dim a = 1, let a = RY and f ± ∈  such that f ± (Y ) = ±1. It is evident that f ± ([h , h ]) = {0}. Let τ± be the monomial representations of G induced from the unitary characters χ± of H (1) coming from f ± . As g = h + [g, g], the result concerning the disintegration is obtained [23] from the relation  τ  τ+ ⊕ τ− 

⊕  G

 m + (π )π dν+ (π ) ⊕

⊕  G

m − (π )π dν− (π )

with the notations well understood. But, we know that intertwining operators R± from τ± to τ are given by the formulas 

−1/2

(R± ψ)(g) = H/H0

ψ(gh)χ± (h) H,G (h)dν H,H0 (h) (g ∈ G)

with H0 = exp(h0 ) where h0 = h ∩ g0 . We denote by aπk,± the Penney’s distributions associated to τ± . Then, an application of the transitivity of ν·,· and a simple verification show aπk = R± ◦aπk,± and π(ϕ)aπk = R± (π(ϕ)aπk,± ) for ϕ ∈ Cc∞ (G). In this case, the coefficients π(ϕ)aπk , aπk  have a sense as

π(ϕ)aπk , aπk  = R± (π(ϕ)aπk,± ), R± ◦ aπk,±  = π(ϕ)aπk,± , aπk,± . Put a± = δτ ◦R± , namely a± : Hτ∞±  ψ →

 H/H0

−1/2 ˙ ψ(h)χ f± (h) H,G (h)d h.

42

A. Baklouti et al.

Appropriately normalized, a± satisfy

τ (ϕ)δτ , δτ  = τ+ (ϕ)a+ , a+  + τ− (ϕ)a− , a−  for any ϕ ∈ Cc∞ (G). Then formula (2) is obtained as in [16]. If dim a = 2, let Y1 , Y2 ∈ a such that the three elements X, Y1 , Y2 verify the bracket relations [X, Y1 ] = Y1 − αY2 , [X, Y2 ] = Y2 + αY1 (0 = α ∈ R). Let  θ ∈  be such that

 θ |a = (cos θ )Y1∗ + (sin θ )Y2∗ .

Trivially,  θ ([h , h ]) = {0} and  τ



⊕ [0,2π]

indGH (1) χθ dθ





⊕

dθ [0,2π]

⊕  G

m θ (π )π dνθ (π ).

(19)

Put θ =  θ + h ⊥ and θ0 =  θ + k⊥ . Take now a coadjoint orbit  of G which does not vanish on a. We then have dim(g( ) ∩ a) = 1 for ∈ . For such an , there exists a unique θ ∈ [0, 2π ) such that  ∩  = exp(RX ) · (θ0 ∩ ), θ ∩  = A·(θ0 ∩ ), A = exp(a). Taking these observations into account, the above formula (19) is interpreted as the desired disintegration. According to this disintegration, we form an intertwining operator Rθ from τθ = indGH (1) χθ to τ by the formula 

−1/2

(Rθ ψ)(g) = H/H0

ψ(gh)χθ (h) H,G (h)d h˙ (g ∈ G).

We next put aθ = δτ ◦Rθ , namely aθ :

Hτ∞θ

  ψ → H/H0

−1/2 ˙ ψ(h)χ f (h) H,G (h)d h.

Reasoning at the level of the subgroup K , we directly verify as in the case of the 1/2 ax + b group that aθ belongs to the space (Hτ−∞ ) H,χ f  H,G . Example 2 in [15] then θ shows  ϕ H (e) = τ (ϕ)δτ , δτ  = (2π )−2

2π

f

τθ (ϕ)aθ , aθ dθ

(20)

0

for any ϕ ∈ Cc∞ (G). As in the preceding case, we show the existence of the coefficients π(ϕ)aπk , aπk  through the intertwining operator.

Monomial Representations of Discrete …

43

Let us continue the proof of formula (2). A direct computation gives us, for g ∈ G,   (τθ (ϕ)aθ )(g) =



H (1)

R

t  −1/2 ϕ(gh exp (t  X ))χ f (h) H,G (h)eit f (X ) e− 2 (Tr adg X +Tr adh X ) dhdt  .

Under the unique expression of h ∈ H in the form h = h 0 exp(s X ) with h 0 ∈ H0 and s ∈ R, we have dh = e−sTr adh X dh 0 ds. Let ϕ ∈ Cc∞ (G). For t ∈ R arbitrary fixed, we form t ∈ Cc∞ (G 0 ) by  t (g0 ) =

R

ϕ(exp(t X )g0 exp(t  X ))eit





f (X ) − t2 (Tr adg X +Tr adh X )

e

dt 

and put 

(t )θH (1) (g0 ) =



−1/2

H (1)

t (g0 h  )χθ (h  ) H (1) ,G 0 (h  )dh  (g0 ∈ G 0 ).

From the induction hypothesis, we see 



(t )θH (1) (e) =

dμθ (π )

 G m θ (π)  k=1

−1/2

H (1) /(H (1) ∩Bk )

χθ (h) H (1) ,G 0 (h)Pθk (t, h)dν Bk ,H (1) ∩Bk (h)

with  Pθk (t, h) =

 

Bk /(H (1) ∩Bk )

= ×



Bk /(H (1) ∩Bk )

−1/2

 

× ×e

χgk · (b) Bk ,G 0 (b)dν Bk ,H (1) ∩Bk (b) −1/2

H (1)

=

−1/2

(t )θH (1) (hb)χgk · (b) Bk ,G 0 (b)dν Bk ,H (1) ∩Bk (b)

t (hbh  )χθ (h  ) H (1) ,G 0 (h  )dh  −1/2

χgk · (b) Bk ,G 0 (b)dν Bk ,H (1) ∩Bk (b)  −1/2   χθ (h ) H (1) ,G 0 (h ) ϕ(exp(t X )hbh  exp(t  X ))

Bk /(H (1) ∩Bk )

H (1)



R

it  f (X ) − t2 (Tr adg X +Tr adh X )

e

dt  .

Taking the transitivity of ν·,· into account, we deduce from (20) that

44

A. Baklouti et al. f ϕ H (e)

×

= τ (ϕ)δτ , δτ  = (2π )

k=1

×  ×

× 

dθ

 G

dμθ (π )

H/(H ∩Bk )

χθ (h) H,G (h)dν H,H ∩Bk (h) 

−1/2

Bk /(H ∩Bk )

χgk · (b) Bk ,G (b)dν Bk ,H ∩Bk (b)

dh 0 H0

−1/2

R





2π

−1/2

 G



ϕ(hbh 0 exp(t  X ))χ f (h 0 exp(t  X )) H,G (h 0 exp(t  X ))e−t Tr adh X dt 

 =

 0

m(π) 



−2

dμ(π )

m(π) 

−1/2

H/(H ∩Bk )

k=1

χ f (h) H,G (h)dν H,H ∩Bk (h)

−1/2

Bk /(H ∩Bk )

χgk · (b) Bk ,G (b)dν Bk ,H ∩Bk (b) −1/2

× H

ϕ(hbh  )χ f (h  ) H,G (h  )dh  =

 m(π)   G k=1

π(ϕ)aπk , aπk dμ(π ).

(II) Suppose that a ∩ z = z = {0}. Then, h ∩ a = a or h ∩ a = z. Suppose first a ⊂ h. We introduce the proper subalgebra g1 = a f , the orthogonal of a with respect to B f , which contains h. From the induction hypothesis applied to the subgroup G 1 = exp(g1 ), it follows that  τ1 = indGH1 χ f  Then,

 τ

indG G 1 τ1



⊕ 1 G

⊕ 1 G

m 1 (ρ)ρdμ1 (ρ).

m 1 (ρ)(indG G 1 ρ)dμ1 (ρ).

(21)

Here ρ| A being a multiple of χ f | A , these indG G 1 ρ are all irreducible and inequivalent each other. On the other hand, A· = + (a )⊥ for any ∈ g∗ . Now, formula (21) gives us the disintegration. Our formula (2) holds for τ1 by the induction hypothesis −1/2 applied to the test function ϕ1 (g1 ) = ϕ(g1 )G 1 ,G (g1 ) (g1 ∈ G 1 ). The existence of k k the coefficients π(ϕ)aπ , aπ  is assured by the induction hypothesis, exactly as in the case 3.1. We now consider the case where h ∩ a = z. Suppose first that h0 = h ∩ g0 ⊂ g1 . θ , what depends on dim a. As [h0 , a] = {0}, the subalgebra h is subordinate to f ± or  The above arguments followed by a last regrouping of terms under the coadjoint action of G lead us to the desired formula (2). Let us examine the last case where h0 ⊂ g1 . Let again h0 = h ∩ g1 , H0 = exp(h0 ), h = h0 + a and H (1) = exp(h ). Evidently, f ([h , h ]) = {0}. We know that an intertwining operator R from τ  = indGH (1) χ f to τ is obtained by the formula

Monomial Representations of Discrete …

45



−1/2

(Rψ)(g) = H/H0

ψ(gh)χ f (h) H,G (h)dν H,H0 (h) (g ∈ G).

Here also we utilize as before the intertwining operator to assure the existence of the coefficients π(ϕ)aπk , aπk . Putting a  = δτ ◦R and reasoning at the level of the subgroup K , we see that 

a :

Hτ∞



−1/2

 ψ → H/H0

ψ(h)χ f (h) H,G (h)dν H,H0 (h) −1/2

) H,χ f  H,G . Lemma 2 in [16] says that up to a gives us an element of the space (Hτ−∞  normalization f ϕ H (e) = τ (ϕ)δτ , δτ  = τ  (ϕ)a  , a   for all ϕ ∈ Cc∞ (G). Finally, the formula (2) is established just as in [16]. This finishes the proof of Theorem 3. Example 3 Let us recall all the data of Example 2. We denote by H the Hilbert space of π . Since G = exp(RX )B[ ], H is identified with L 2 (R) through the mapping (t) = φ(exp(t T ))(t ∈ R). If we define 



−1/2

a ,  = H

φ(h)χ f (h) H,G (h)dh =

R

φ(exp(x X ))d x

for  ∈ L 2 (R), the relation exp(x X ) = exp

w

  x (1 − e−x )T exp (λX − wT ) λ λ

gives 

  w  x (1 − e−x )T exp (λX − wT ) d x φ exp λ λ R   xζ x w = (1 − e−x ) e−i λ e− 2 d x  λ R  ∞   ζ log y dy w = (1 − y) e−i λ √ .  λ y 0

a ,  =

On the other side, simple computations show (λX − wT )· = λt T · = −

d + dt



 λ − iζ  2

d , A· = i(w − λt), Z · = iλ. dt

46

A. Baklouti et al.

So, the space of the C ∞ -vectors of π is identified with the Schwartz space S(R) of the rapidly decreasing C ∞ -functions. Since a is a tempered distribution, we conclude that  H,χ f 1/2  H,G a ∈ H −∞ . Now, for ϕ ∈ Cc∞ (G), put  f ϕ H (g)

= H

−1/2 ϕ(gh)χ f (h) H,G (h)dh

 =

R

ϕ(g exp(x X ))d x.

Now, G· ∩ h⊥ has two connected components of i (i = 1, 2), where

1 = ζ T ∗ + A∗ + λZ ∗ and 2 = −ζ T ∗ − A∗ + λZ ∗ . Then, we find the formula: 

π 1 (ϕ)a 1 , a 1  =  =

 R R

dx  dx

−1/2

f

B[ 1 ]

ϕ H (exp(x X )b)χ 1 (b) B[ 1 ],G (b)db f

R3

ϕ H (exp(x X ) exp(t (λX − T )) exp(a A) exp(z Z )) λt

× e−itζ eia eiλz e 2 dtdadz     −x −tλ     − 1) e (e f T exp ae x etλ A exp(z Z ) dx ϕ H exp = λ R R3 λt

× e−itζ eia eiλz e 2 dtdadz     −x −tλ   − 1) e (e f = T exp(b A) exp(z Z ) dx ϕ H exp λ R R3 × e−itζ eibe

−x e−tλ

λt

eiλz e−x e− 2 dtdbdz.

In the same way, 

π 2 (ϕ)a 2 , a 2  =



 dx

R

R3

f ϕH



  e−x (1 − e−tλ ) exp T exp(b A) exp(z Z ) λ

itζ −ibe−x e−tλ iλz −x − λt2

×e e

e

e

e

dtdbdz.

Finally, the measures being appropriately normalized, we arrive to the Plancherel formula for τ :

Monomial Representations of Discrete …

47



 R2

( π 1 (ϕ)a 1 , a 1  + π 2 (ϕ)a 2 , a 2 )dζ dλ =

f

ϕ H (exp(b A) exp(z Z ))  −x  −x × eibe + e−ibe eiλz e−x dbd xdzdλ    −x −x f = e−x dbd x ϕ H (exp(b A)) eibe + e−ibe 2 R  f f = ϕ H (exp(b A))eiby dbd y = ϕ H (e), R4

R2

from the ordinary Fourier inversion formula.

7 Invariant Differential Operators We still keep the notations introduced until now and suppose that our monomial representation τ = indGH χ f has multiplicities of discrete type. By means of the disintegration formula (4) of δτ , we immediately ascertain, as in the case of finite multiplicities (cf. [19]), that the mapping  τ (ϕ)δτ →

⊕  G

m(π) 

 π(ϕ)aπk

dμ(π ) (ϕ ∈ Cc∞ (G))

k=1

gives us an intertwining operator between τ and its disintegration (1). Here, we are interested to study the Duflo’s problem. Let W ∈ U(g, τ ) and let W act on τ (ϕ)δτ ∈ Hτ∞ from right and transport by the intertwining operator this action into the disintegration. This action is decomposable into factorial parts. This means that,  there exists a complex matrix A(π ) = (ai j ) of order m(π ) for μ-almost all π ∈ G, verifying m(π)  R(W )(π(ϕ)aπk ) = ak j π(ϕ)aπj (22) j=1

for all ϕ ∈ Cc (G), where R(W ) denotes the right action of W . Indeed, according to the disintegration (1) the operator U = R(W ) is decomposed into an integral ⊕ U = G Uπ dμ(π ), which allows us to admit the existence of a continuous operator k,l Uπ (k, l ∈ N) in the space Hπk,∞ . Actually it suffices to consider the composition  p,∞ of Uπ with the partial l-projection of the space m(π) into Hl,∞ π . Here the p=1 Hπ p,∞ ∞ spaces Hπ (1 ≤ p ≤ m(π )) denote the spaces of the C vectors corresponding to the isotypic components of the decomposition of τ . On the other hand, it is easy to ascertain that the operators Uπk,l (k, l ∈ N) commute with the action of G, which allows us to ascertain that these operators are scalars from Corollary 3.5 in [22]. At this stage we can admit that μ-almost everywhere in  all the double classes Bgi H = B[ ]gi H (1 ≤ i ≤ m(π )) have the same dimension. In fact, it suffices to consider the H -invariant Zariski open set { ∈ ; dim(h + b[ ]) is maximal}.

48

A. Baklouti et al.

Each term from equation (22), being a C ∞ -function, we can evaluate each term at the point e ∈ G. Let j = k. Taking a test function ϕ which, as well as its!derivatives to a certain determined order, vanishes on the union of the double classes i= j Bgi−1 H , we find that ai j = 0 except i = j. Indeed, each double class in the union being locally −1 closed, Bg −1 j H can intersect the closure Bgi H only on its frontier for i  = j. Thus, it results that π(W )aπk = PW (gk · )aπk

with a H -invariant function PW defined almost everywhere on . The mapping U(g, τ )  W → PW ( ) induces a homomorphism from the algebra Dτ (G/H ) into the algebra of the H invariant functions on . Besides, the Plancherel formula for τ implies that this homomorphism is injective. Summarizing, we establish one implication of Duflo’s problem. Theorem 4 If τ = ind GH χ f has multiplicities of discrete type, the algebra Dτ (G/H ) is commutative. Corollary 4 Suppose that τ = ind GH χ f has multiplicities of discrete type. Let g0 be an ideal of codimension 1 in g such that the G-orbits are μ-almost everywhere saturated in the direction g⊥ 0 . Then, the algebra U(g, τ ) is contained in U(g0 ) modulo the kernel U(g)aτ . Proof Let g = RX + g0 . If U(g, τ ) is not contained in U(g0 ), there exists an element W having the form W = Xa + b with a, b ∈ U(g0 ) and a belonging to U(g, τ ) \ U(g)aτ . In fact, if W = X m am + X m−1 am−1 + · · · + a0 ∈ U(g, τ ) with / U(g)aτ , then it is clear that m Xam + am−1 has ai ∈ U(g0 )(0 ≤ i ≤ m) and am ∈ the same property. Note from now on W = Xa + b. Generally, we take in the Zariski open set of  given by Theorem 2 and let  We consider the associated Penney’s distribution a . We realize π π = θG ( ) ∈ G. G as π = ind B χ , B = exp b, by means of a Vergne polarization at and we calculate the differential operator W ,  



π(W )a , ψ =

H/(H ∩B)

J

 ∂ J ψ¯ −1/2 λ J (h) J (h)  H,G (h)χ f (h)dν(h) ∂x

with multi-indices J and functions λ J (h). Since  J

λ J (h)

∂J ∂ =c + A, ∂x J ∂ x1

Monomial Representations of Discrete …

49

where c is a non-zero complex number, x1 being the first coordinate, and where A designates a partial differential operator concerning the coordinates (x2 , . . . , x p ), p = dim(G/B). Then it is evident that π(W )a is not a multiple of a , which is absurd. Remark here that π(W ) is a differential operator on the representation space which  we express by means of the coordinates (x1 , . . . , x p ). Example 4 Let us come back to Example 2 with all the notations used there. Let W ∈ U(g). As X ∈ aτ , W takes the form W =

   m

 cmp,q T p Aq

Z m (cmp,q ∈ C).

p,q

Then, W belongs to U(g, τ ) if and only if cmp,q = 0 if p = q. Namely, W is a polynomial in Z and T A. Let ∈  such that a is well defined. Let us calculate (T A)·a . For φ ∈ Hπ∞ , 

(T A)·a , φ = a , (AT )·φ =

R

(AT )·φ(exp(x X ))d x

  d (T ·φ)(exp(t A) exp(x X )) d x R dt t=0    d −x =− (T ·φ)(exp(x X ) exp(te A)) d x R dt t=0   d  ite−x (A)  =− (T ·φ)(exp(x X ))e  dx R dt t=0  −x = −i (A) e (T ·φ)(exp(x X ))d x 

=−

 = −i (A) Supposing (A) = 0, we put α = putation shows

R

e−x R

(Z ) .

(A)

  d φ(exp(−t T ) exp(x X )) d x. dt t=0

Then, α X − T ∈ b[ ]. Here, a simple com-

exp(−t T ) exp(x X ) = exp(y X ) exp s(α X − T ) with eαs = 1 + αte x , y + αs = x for t ∈ R in a neighbourhood of 0. Therefore, αs = log(1 + αte x ). So, we have,    d

(T A)·a , φ = −i (A) e−x φ(exp(y X ) exp s(α X − T )) d x dt R t=0    −x d − αs −is (T )  dx = −i (A) e φ(exp(y X ))e 2 e  dt R t=0

50

A. Baklouti et al.



(T )  d  φ(exp(y X ))(1 + αte x )−1/2−i α  d x dt R t=0    

(T ) −x x = −i (A) e αe φ(exp(x X )) −1/2 − i α R " x dφ (exp(x X )) d x + αe dx    i dφ =

(Z ) − (T ) (A) (exp(x X ))d x. φ(exp(x X ))d x − i (Z ) 2 R R dx 

= −i (A)

e−x

Since     1 − e−x x x T e− 2 ei α (T ) = 0, lim φ(exp(x X )) = lim φ exp x→±∞ x→±∞ α we finally get



 i

(Z ) − (T ) (A) a , φ. 2



 i

(Z ) − (T ) (A) a . 2

(T A)·a , φ = Namely, (T A)·a = Now, it follows that our mapping

U(g, τ )  W → PW ( ) induces an isomorphism from the algebra Dτ (G/H ) onto the algebra C[] H of the H -invariant polynomial functions on . Example 5 Take now h = RT, H = exp h and f ∈ g∗ such that f |h = 0 in the same example. Put = α X ∗ + βY ∗ + λZ ∗ ∈ h⊥ (α, β, λ ∈ R) and compute the orbit G· . Writing g = exp(x X ) exp(t T ) exp(a A) ∈ G with x, t, a ∈ R, we have g −1 · (X ) = (g·X ) = α − ae x β − atλ, g −1 · (T ) = (g·T ) = −aλ, g −1 · (A) = (g·A) = βe x + tλ. Hence, G· = x ∗ X ∗ + t ∗ T ∗ + a ∗ A∗ + λZ ∗ , x ∗ − α = for λ = 0. So,

G· ∩ h⊥ = α X ∗ + RA∗ + λZ ∗ = H · .

t ∗a∗ λ

Monomial Representations of Discrete …

51

It follows that τ = indGH χ f is without multiplicity. On the other side, the algebra U(g, τ ) is modulo U(g)aτ the algebra of the polynomials in two variables X and Z . For φ ∈ Hπ∞ , put (t) = φ(exp(t T )) (t ∈ R) and consider the Penney’s distribution  a (φ) =

R

(t)dt.

Then,   d φ(exp(x X ) exp(t T )) dt R R dt x=0         d β β  dt φ exp te−x + (1 − e−x ) T exp x X − T =  λ λ R dt x=0        d β x φ exp te−x + (1 − e−x ) T e− 2 eiαx  dt = dt λ R x=0      β d (t) dt = (−1/2 + iα) (t) − t − λ dt R  # $∞    β = (−1/2 + iα) (t)dt − t− (t) − (t)dt λ R R −∞  = (1/2 + iα) (t)dt = (1/2 + iα) a (φ). 

(X ·a )(φ) = −



X ·φ(exp(t T ))dt =

R

In consequence, X ·a = (1/2 + iα) a . After all, our algebra Dτ (G/H ) is once again isomorphic to the algebra C[] H of the H -invariant polynomial functions on . Example 6 Let G be an exponential solvable Lie group. Consider the case where H is normal in G. Then it is well known that τ is either without multiplicity or with multiplicities uniformly equal to the infinity almost everywhere on . Then, it is shown in [3] that the algebras Dτ (G/H ) and C[] H are isomorphic when τ is without multiplicity. Far from the nilpotent context, the following example shows that the Duflo’s problem can have a negative solution in the exponential framework. Example 7 When G = exp g is no longer nilpotent, the algebra Dτ (G/H ) can be very small. Let

52

A. Baklouti et al.

g = T, X, Y, Z R : [T, X ] = X, [T, Y ] = Y, [T, Z ] = 2Z , [X, Y ] = Z . ⎧⎛ 2 ⎫ ⎞ ⎨ a x z ⎬ G = ⎝ 0 a y ⎠ ; a > 0, x, y, z ∈ R ⎩ ⎭ 0 01

Therefore,

is completely solvable. Take f = 0 ∈ g∗ and h = RT . Then, it follows immediately that g( ) = {0} for ∈ g∗ verifying (Z ) = 0. It results that τ = indGH 1 has infinite multiplicities of continuous type. However, we see that the algebra Dτ (G/H ) is trivial. Indeed, each monomial X p Y q Z r ( p, q, r ∈ N) is an eigenvector for the action ad T with the eigenvalue p + q + 2r , what immediately leads our assertion. Thus, Theorem 4 is the best possible generalization to exponential Lie groups of the CorwinGreenleaf result for nilpotent groups.

8 Polarizations As an example, we here treat a particular case of monomial representations of discrete type, where h is a polarization at f ∈ g∗ which does not satisfy the Pukanszky condition. Recall the results of [23]. We designate by /G the set of the coadjoint orbits of G whose intersection with the affine subspace  are non-empty open set of . It turns out that /G is a finite set. For  ∈ /G, consider the connected components of  ∩ , each of them is a H -orbit and their number m() is finite. The result of [23] says that τ = indGH χ f is decomposed in a direct sum τ

⊕ 

m()π(),

∈/G

 where we wrote π() at the place of θ¯ () ∈ G. −∞ For an intertwining operator R : Hπ() → Hτ , we define a R ∈ Hπ() by ∞ a R (ψ) = δτ , R(ψ) = R(ψ)(e) (ψ ∈ Hπ() ).

A simple calculation shows that  H,χ f 1/2  H,G −∞ a R ∈ Hπ() and the mapping R → a R is anti-linear and injective. Hence,   H,χ f 1/2 H,G −∞ dim Hπ() ≥ m()

Monomial Representations of Discrete …

53

(for more general result, see [21]).   H,χ f 1/2 H,G −∞ Now let a ∈ Hπ() . For any ϕ ∈ Cc∞ (G), a direct calculation shows f

π()(ϕ)a = π()(ϕ H )a. Therefore, the mapping f

f

τ (ϕ)(δτ ) = ϕ H → π()(ϕ H )a = π()(ϕ)a gives us an intertwining operator from the space Hτ∞ =

⊕ 

∞ m()Hπ()

∈/G ∞ into the space Hπ() . Let

δτ 

⊕ 

m(π) 

∈/G

 k aπ()

k=1

be the concrete Plancherel formula described with the Penney’s distributions aπk ∞ (cf. [16]). Denote by I(Hτ∞ , Hπ() ) the space of the intertwining operators. For  ∈ /G, let us consider the projections ∞ (1 ≤ j ≤ m()) p : Hτ∞ → Hπ() j

defined by

p (τ (ϕ)δτ ) = π()(ϕ)aπ() (ϕ ∈ Cc∞ (G)). j

j

Then, we know [22] that ∞ )= I(Hτ∞ , Hπ()

m() 

j

C p .

j=1

In this way, we arrive to a variant of the Frobenius reciprocity, which was studied in [16] under an additional condition. Theorem 5 Assume that h is a polarization at f ∈ g∗ . Then for  ∈ /G, we obtain  H,χ f 1/2  H,G −∞ . m() = dim Hπ() On the other side, for  ∈ / /G, we have

54

A. Baklouti et al.



−∞ Hπ()

 H,χ f 1/2 H,G

= {0}.

Remark. When h is a polarization at f ∈ g∗ , we easily verify (cf. [3]) that the algebra Dτ (G/H ) is trivial, namely Dτ (G/H ) = C1. Acknowledgements The authors would like to thank the Referee for having proposed many valuable comments and suggestions to improve the final form of the paper.

References 1. Arnal, D., Fujiwara et, H., Ludwig, J.: Opérateurs d’entrelacement pour les groupes de Lie exponentiels, Amer. J. Math. 118, 839–878 (1996) 2. Baklouti, A., Ludwig, J.: Désintégration des représentations monomiales des groupes de Lie nilpotents. J. Lie Theory 9, 157–191 (1999) 3. Baklouti et, A., Fujiwara, H.: Opérateurs différentiels associés à certaines représentations unitaires des groupes de Lie résolubles exponentiels. Compositio. Math. 139, 29–65 (2003) 4. Baklouti, A., Hamrouni, H., Khlif, F.: Analysis of some monomial representations of exponential solvable Lie groups. Russ. J. Math. Phys. 13(4), 363–379 (2006) 5. Baklouti, A., Hamrouni, H.: The multiplicity function of mixed representations on completely solvable lie groups. Tokyo. J. Math. 30(1), 41–55 (2007) 6. Bernat, P., et al.: Représentations des groupes de Lie résolubles. Dunod, Paris (1972) 7. Cartier, P.: Vecteurs différentiables dans les représentations unitaires des groupes de Lie. Lecture Notes in Mathematics, vol. 514, pp. 20–34. Springer, Berlin (1975) 8. Corwin, L., Greenleaf, F.P.: Commutativity of invariant differential operators on nilpotent homogeneous spaces with finite multiplicity. Comm. Pure Appl. Math. 45, 681–748 (1992) 9. Dixmier, J.: Algèbres enveloppantes. Gauthier-Villars, Paris (1974) 10. Dixmier, J., Malliavin, P.: Factorisations de fonctions et de vecteurs indéfiniment différentiables. Bull. Sci. Math. 102, 305–330 (1978) 11. Duflo, M.: Personal Communication 12. Duflo, M.: Open problems in representation theory of Lie groups, edited by T. Oshima. pp. 1–5. Katata, Japan (1986) 13. Fujiwara, H.: Certains opérateurs d’entrelacement pour des groupes de Lie résolubles exponentiels et leurs applications. Mem. Fac. Sci. Kyushu Univ. Ser A 36, 13–72 (1982) 14. Fujiwara, H.: Représentations monomiales des groupes de Lie nilpotents. Pacific J. Math. 127, 329–351 (1987) 15. Fujiwara, H.: Représentations monomiales des groupes de Lie résolubles exponentiels, The orbit method in representation theory. In: Duflo, M., Pedersen, N.V., Vergne, M. (eds.) Proceedings of a conference in Copenhagen, pp. 61–84. Birkhaüser, Boston (1990) 16. Fujiwara et, H., Yamagami, S.: Certaines représentations monomiales d’un groupe de Lie résoluble exponentiel. Adv. St. Pure Math. 14, 153–190 (1988) 17. Fujiwara, H., Lion, G., Magneron, B., Mehdi, S.: Commutativity criterion for certain algebras of invariant differential operators on nilpotent homogeneous spaces. Math. Ann. 327, 513–544 (2003) 18. Grélaud, G.: Désintégration des représentations induites des groupes de Lie résolubles exponentiels, Thèse de 3e cycle, Univ. de Poitiers (1973) 19. Lipsman, R.: The Penney-Fujiwara Plancherel formula for homogeneous spaces, in representation theory of lie groups and lie algebras. In: Proceedings of Fuji-Kawaguchiko Conference, pp. 120–139. World-Scientific Publishing Co., Singapore (1992)

Monomial Representations of Discrete …

55

20. Leptin, H., Ludwig, J.: Unitary Representation Theory of Exponential Lie Groups. W. De Gruyter, Berlin (1994) 21. Penney, R.: Abstract Plancherel theorem and a Frobenius reciprocity theorem. J. Funct. Anal. 18, 177–190 (1975) 22. Poulsen, N.S.: On C ∞ -vectors and intertwining bilinear forms for representations of lie groups. J. Funct. Anal. 9, 87–120 (1972) 23. Vergne, M.: Étude de certaines représentations induites d’un groupe de Lie résoluble exponentiel. Ann. Sci. Éc. Norm. Sup. 3, 353–384 (1970)

Self-Chabauty-isolated Locally Compact Groups Hatem Hamrouni and Firas Sadki

Abstract Let G be a locally compact group. We denote by SUB (G) the space of closed subgroups of G equipped with the Chabauty topology. The group G is called self-Chabauty-isolated if the point G is isolated in SUB (G). In this paper we are interested in the following question: Give necessary and sufficient conditions for the group G to be a self-Chabauty-isolated. Keywords Locally compact group · Lie group · Pro-Lie group · Discrete subgroup · Chabauty topology · Frattini subgroup 1991 Mathematics Subject Classification 22D05 · 54B20 · 22E40

1 Introduction Let G be a locally compact group with identity element e. The Chabauty topology on the set SUB (G) of all closed subgroups of G has a subbase given by sets of the following form O1 (K ) = {H ∈ SUB (G) | H ∩ K = ∅} O2 (V ) = {H ∈ SUB (G) | H ∩ V = ∅} , where V and K run respectively, over all open and compact subsets of G. The Chabauty topology is named after Claude Chabauty, who introduced it in [5] to generalize Mahler’s compactness criterion to lattices in locally compact groups. If G is locally compact, then SUB (G) is compact. There is ample literature on the Chabauty topology, see for example [2, 3, 7, 15].

H. Hamrouni (B) · F. Sadki Department of Mathematics, Faculty of Sciences at Sfax, Sfax University, B.P. 1171, 3000 Sfax, Tunisia e-mail: [email protected] © Springer Nature Switzerland AG 2019 A. Baklouti and T. Nomura (eds.), Geometric and Harmonic Analysis on Homogeneous Spaces, Springer Proceedings in Mathematics & Statistics 290, https://doi.org/10.1007/978-3-030-26562-5_2

57

58

H. Hamrouni and F. Sadki

In this space, each closed subgroup H of G has a neighborhood base consisting of sets def

U(H ; K , W ) = {L∈SUB (G) | L∩K ⊆W H and H ∩K ⊆W L} ,

(1.1)

where K ranges through the set K of all compact subsets of G and W through the set U(e) of all neighborhoods of the identity. In particular G ∈ SUB (G) has a neighborhood base consisting of sets U(G; K , W ) = {L∈SUB (G) | K ⊆W L}

(1.2)

K ∈ K and W ∈ U(e). The following definition is given in [12, Definition]. Definition 1.1 (Self-Chabauty-isolated group) A locally compact group G is selfChabauty-isolated if the point G is isolated in SUB (G) with the Chabauty topology. In this paper we are interested in the following question: Question 1.2 Give necessary and sufficient conditions for a locally compact group G to be a self-Chabauty-isolated. Firstly, we note, as the subspace of topologically finitely generated subgroups (see Definition 2.1) is dense in SUB (G) (is a folklore result which appears, for example, as Proposition 2.6 in [16]), every isolated point in SUB (G) is topologically finitely generated. In particular, every self-Chabauty-isolated group is topologically finitely generated. Being topologically finitely generated is not in general a sufficient condition for a locally compact group to be self-Chabauty-isolated, as the group of reals shows (see Examples 3.3 (3)). However, in the discrete groups setting we have Proposition 1.3 ([11]) A discrete group G is self-Chabauty-isolated if and only if it is finitely generated. Next, we recall the following characterization result for locally compact abelian self-Chabauty-isolated group, due to Yves de Cornulier [6, Lemma 5.2]. Proposition 1.4 Let G be a locally compact abelian group. Then G is self-Chabautyisolated if and only if G has the form D × H , where D is discrete and finitely generated and H is a product of finitely many p-adic integer groups. Concerning the connected Lie groups, we have the following proposition which was proved in [1, Proposition 2.2]. Proposition 1.5 Let G be a connected Lie group. Then G is self-Chabauty-isolated if and only if G is topologically perfect (that is; the commutator subgroup of G is dense in G). Example 1.6 (The special linear group) For every n ≥ 2, the special linear group SLn (R) belongs to the class of self-Chabauty-isolated groups. For a background and discussion on the question, we refer to [6, 11, 13, 17].

Self-Chabauty-isolated Locally Compact Groups

59

2 Background on Topological Groups 2.1 Topologically Finitely Generated Groups The subgroup generated by a finite subset {x1 , . . . , xn } of a group G is denoted by x1 , . . . , xn . Definition 2.1 (Topologically finitely generated groups) A topological group G is said to be topologically finitely generated if there exists a positive integer m such that G = x1 , x2 , . . . , xm , for some subset {xl , x2 , . . . , xm } ⊆ G. The following result is very simple. Lemma 2.2 Every topologically finitely generated locally compact group is compactly generated. Proof Let G be a topologically finitely generated locally compact group, S a finite subset of G such that G = S, and V a compact neighborhood of e in G. We have G ⊇ S ∪ V  ⊇ S V ⊇ S = G. Then G = S ∪ V .



Lemma 2.3 Let G be a topologically finitely generated group and S be a finite set of G such that G = S. If H is a subgroup of G with a finite index m in G then H ∩ S has a finite index less than or equal to m in S. Proof This follows from the fact that the set S/(H ∩ S) and SH/H are equipotent and SH/H ⊆ G/H .  Proposition 2.4 Let G be a topological group and H a closed subgroup of finite index in G. Then the two conditions are equivalent: (1) G is topologically finitely generated; (2) H is topologically finitely generated. Proof (1) ⇒ (2): Let S be a finite set of G such that G = S. By Lemma 2.3, the subgroup H ∩ S has a finite index in S and so H ∩ S is finitely generated (see [24, p. 36, 1.6.11]). Let F be a finite set of G such that S ∩ H = F. On the other hand, as H is open in G then F = S ∩ H = H ∩ S = H ∩ G = H. Hence, the group H is topologically finitely generated. (2) ⇒ (1): Let S be a finite subset such that H = S and let F be a finite subset of G such that π(F) = G/H , where π : G → G/H is the canonical projection. It clear  that G = S ∪ F.

60

H. Hamrouni and F. Sadki

Proposition 2.5 Let G be a topologically finitely generated group. For every n ∈ N, the number of closed subgroups of index n is finite. Proof For any closed subgroup H of index n in G there is a continuous morphism θ H : G → Sym(G/H ) from G into the symmetric group of the discrete finite set G/H such that θ H (g) = id G/H if and only if g ∈ H . Then the morphisms corresponding to different closed subgroups of index n are all different and so the result follows from the fact that the number of continuous morphisms from a topologically finitely generated group to a given finite group is finite. 

2.2 Frattini Subgroups Definition 2.6 Let G be a topological group. (1) An element g of G is an openwise non-generator of G if G = X  for each subset X in G such that G = g, X  and X  is an open subgroup of G. (2) An element g of G is a topological non-generator of G if G = X  for each subset X in G such that G = g, X . In the sequel we often use the next elementary statement: Lemma 2.7 Let G be a topological group, A and B two subsets of G. Then A, B = A, B = A, B. As an immediate consequence we have Proposition 2.8 Let G be a topological group. (1) An element g of G is an openwise non-generator of G if and only if G = H for each open subgroup H of G such that G = g, H . (2) An element g of G is a topological non-generator of G if and only if G = H for each closed subgroup H of G such that G = g, H . Definition 2.9 Let G be a topological group. (1) The set of all topological non-generators of G is called the topological Frattini subgroup of G and it is denoted by Frat (G). (2) The set of all openwise non-generators of G is called the openwise Frattini subgroup of G and it is denoted by TFrat (G). Remark 2.10 For any topological group G the following containment G 0 ⊆ TFrat (G) holds, since every open subgroup of G contains G 0 .

Self-Chabauty-isolated Locally Compact Groups

61

Remark 2.11 For any topological group G, Frat (G) ⊆ TFrat (G). The following example shows the inclusion may be strict. For the additive group R we have TFrat (R) = R and Frat (R) = {0}. Proposition 2.12 In a topological group G, the openwise Frattini subgroup TFrat (G) of G coincides with the intersection of the maximal open subgroups of G. Proof Let g be an openwise non-generator of G and H a maximal open subgroup of G. If g ∈ / H , then G = g, H  = g, H  and so G = H ; this is a contradiction since H = G. Now, let g ∈ TFrat (G), X a subset of G such that G = g, X  and X  is open in / X . Consider the family G. We show that g ∈ X . Suppose on the contrary that g ∈ A of all open subgroups M of G such that g ∈ / M and X  ⊆ M. It is clear that A is non empty and inductive and so it has a maximal element N . Remark that N is a maximal open subgroup of G; indeed, if K is a subgroup of G containing N and such that K = N , then g ∈ K and so K = G. Since g ∈ / N, g ∈ / TFrat (G), which leads to a contradiction. Recall that a locally compact group is called a [SIN]-group if it possesses a basis of identity neighborhoods which are invariant by conjugation.  Proposition 2.13 Let G be a locally compact group. Then the following assertions are equivalent: (1) G is a totally disconnected [SIN]-group; (2) The compact open normal subgroups form a basis of identity neighborhoods. (3) G is a strict projective limit of discrete groups. If G is compactly generated, the assertions are equivalent to: def  (4) Res(G) = {H | H open normal subgroup of G} = {e}. Proof See Lemma 3.5 of [21] and Corollary 4.1 of [4].



Remark 2.14 Examples of such groups include (1) Totally disconnected abelian locally compact groups, (2) profinite groups, (3) compactly generated totally disconnected nilpotent locally compact groups [26, Theorem, page 144], (4) a totally disconnected group G possesses a compact open normal subgroup Q which is topologically finitely generated ([4], Remark 4.2). Lemma 2.15 In a locally compact totally disconnected [SIN]-group G, the closure of a subgroup L in G is the intersection of the open subgroups of G containing L. Proof Since the set  of compact open normal subgroups of G is a fundamental  system of neighborhoods of the identity e, then L = H ∈ H L.  Proposition 2.16 In every locally compact totally disconnected [SIN]-group we have TFrat (G) = Frat (G).

62

H. Hamrouni and F. Sadki

Proof Let g ∈ TFrat (G) and X a subset of G such that G = X, g. If H is an open subgroup of G which contains X  then G = H, g and therefore H = G, since g ∈ TFrat (G). It follows by Lemma 2.15 that G = X , and so g ∈ Frat (G). Then TFrat (G) ⊆ Frat (G) and therefore TFrat (G) = Frat (G). A subgroup of topological group G is topologically characteristic if it is invariant under all continuous automorphisms of G. It is clear that Frat (G) is a topologically characteristic subgroup of G.  Proposition 2.17 For any surjective morphism f : G → H of topological groups the containment f (TFrat (G)) ⊆ TFrat (H ) holds. In particular, TFrat (G) is a topologically characteristic subgroup of G. Proof Let g ∈ TFrat (G) and L an open subgroup of H such that H =  f (g), L. It will be to show that L = H . As    f (g), L =  f (g), f ( f −1 (L)) = f g, f −1 (L) and g, f −1 (L) contains ker( f ), then g, f −1 (L) = G and so f −1 (L) = G. Hence L = H , as required.  Proposition 2.18 If G is a topologically finitely generated group, H closed subgroup of G, and G = H Frat (G), then H = G. Proof Let S = {g1 , . . . , gm } ⊆ G such that G = S. For every i = 1, . . . , m, there exists (h i , ai ) ∈ H × Frat (G) such that gi = h i ai . It is clear that G = h 1 , . . . , h m , a1 , . . . , am , and so G = h 1 , . . . , h m , since a1 , . . . , am ∈ Frat (G). Then G ⊆ H and hence G = H . 

3 The Class of Self-Chabauty Isolated Groups Let [SCI] be the class of locally compact self-Chabauty-isolated groups. A locally compact group belonging to the class [SCI] will often be called [SCI]-group. Proposition 3.1 For a locally compact group G the following conditions are equivalent: (1) G is self-Chabauty-isolated; (2) there exist a compact subset K of G and a neighborhood U of e such that for every H ∈ SUB (G), if K ⊆ U H then H = G. (3) there exist open subsets U1 , . . . , Un of G such that for every choice of n elements xi ∈ Ui , i = 1, . . . , n, the subgroup x1 , . . . , xn  is dense in G; (4) there exist an open neighborhood U of e and g1 , . . . , gn ∈ G such that for every choice of n elements xi ∈ gi U , i = 1, . . . , n, the subgroup x1 , . . . , xn  is dense in G;

Self-Chabauty-isolated Locally Compact Groups

63

(5) there exist a neighborhood V of e, g1 , . . . , gn ∈ G such that for every H ∈ SUB (G), if {g1 , . . . , gn } ⊆ V H then H = G. (6) there exist a neighborhood V of e, g1 , . . . , gn ∈ G such that for every topologically finitely generated subgroup H of G, if {g1 , . . . , gn } ⊆ V H then H = G. Proof (1) ⇐⇒ (2): G is self-Chabauty-isolated if and only if {G} is open in SUB (G), or equivalently, if and only if there exist a compact subset K of G and a neighborhood U of e such that U(G; K , U ) = {G}. The rest of the proof follows from (1.2). (2) =⇒ (3): By hypothesis we have U(G; K , U ) = {G}. Let U1 , . . . , Un be open subsets of G such that n  O2 (Ui ) = U(G; K , U ). i=1

For every choice of n elements xi ∈ Ui , i = 1, . . . , n, the subgroup x1 , . . . , xn  belongs to U(G; K , U ) and so x1 , . . . , xn  = G. n (3) =⇒ (4): Let (g1 , . . . , gn ) ∈ i=1 Ui , and let U be a symmetric neighborhood of the identity such that, gi V ⊆ Ui , for all i ∈ {1, . . . , n}. (4) =⇒ (5): Let V = U −1 and let H be a closed subgroup of G such that {g1 , . . . , gn } ⊆ V H . It follows that, for every i = 1, . . . , n, H ∩ gi V −1 = ∅ and hence, by (4), the closed subgroup H contains a dense subgroup of G and so H = G. (5) ⇒ (6): Trivial. (6) ⇒ (2): Let H be a closed subgroup of G such that {g1 , . . . , gn } ⊆ V H . There exists {h 1 , . . . , h m } ⊆ H such that {g1 , . . . , gn } ⊆ V {h 1 , . . . , h m }. Then {g1 , . . . , gn } ⊆ V h 1 , . . . , h m  and so, by (6), h 1 , . . . , h m  = G. As h 1 , . . . , h m  ⊆ H , H = G.  Remark 3.2 In view of (1) ⇐⇒ (3), we remark that “self-Chabauty-isolated” here was called “strongly finitely generated” in [17]. Example 3.3 (Examples and non-examples of Self-Chabauty-isolated groups) (1) (The group of integers) Z ∈ [SCI], because it satisfies the condition (3) of Proposition 3.1. (2) (The group of p-adic integers) Let p be aprime number and  let Z p be the group   of p-adic integers. As the mapping φZ p : n1 | n ∈ N \ {0} ∪ {0} → SUB Z p defined by p n−1 Z p if t = n1 , φZ p (t) = {0} if t = 0 is a homeomorphism, Z p ∈ [SCI] [9, Examples 5].

64

H. Hamrouni and F. Sadki

(3) (The additive real group) Since every neighborhood in R contains a rational number, R does not satisfy the condition (3) of Proposition 3.1 and so it is not a self-Chabauty-isolated group (for another proof, it suffices to observe that the sequence of closed subgroups ( n1 Z)n≥1 converges to R in SUB (R), see Proposition 1.7 of [14]). (4) (The one dimensional torus) The one dimensional torus T = R/Z is not a selfChabauty-isolated group, because the sequence of closed subgroups ( n1 Z/Z)n≥1 converges to T in SUB (T). A topological space is called σ-compact if it is a countable union of compact subsets. Corollary 3.4 Every locally compact [SCI]-group is topologically finitely generated. In particular, every locally compact [SCI]-group is compactly generated (and so σ-compact). Proof This follows from Proposition 3.1 and Lemma 2.2.



Remark 3.5 If H is any closed subgroup of an [SCI]-group G, it is not necessary that H should be [SCI]-group (see Examples 3.3). Proposition 3.6 Let G be a compact group. Then G is self-Chabauty-isolated if and only if there exists a neighborhood U of the identity such that G = U L for every closed subgroup L of G with L = G. Proof Since G is compact, the set {U(G; G, W ) | W ∈ U(e)} is a neighborhood base of G in SUB (G). Consequently, G is self-Chabautyisolated if and only if there is a neighborhood U of the identity such that {G} = U(G; G, U ). In view of (1.2), our assertion is now obvious.  Proposition 3.7 Let G be an [SCI]-group. If H is a locally compact group such that there exists a continuous surjective morphism p : G → H , then H is an [SCI]-group. In particular, any Hausdorff quotient group of an [SCI]-group is an [SCI]-group. Proof By Corollary 3.4, G is σ-compact and so, by the Open mapping Theorem for locally compact groups (see [18, p. 42, Theorem 5.29] or [20, Corollary 2.7]), the mapping p is open. By Proposition 1.2 of [14] the mapping SUB ∗ ( p) : SUB (H ) → SUB (G), defined by SUB∗ ( p) (L) = p −1 (L), is continuous (see also Proposition  2.5 of [15]) and therefore SUB∗ ( p)−1 ({G}) = {H } is open in SUB (H ). Proposition 3.8 Let G be a locally compact group and H an open normal subgroup of G. If H and G/H are both [SCI]-groups then G is also an [SCI]-group.

Self-Chabauty-isolated Locally Compact Groups

65

Proof Let K be a compact subset of H containing e, U an open neighborhood of e in H such that, for each L ∈ SUB (H ), if K ⊆ U L then L = H . On the other hand, as G/H is a discrete [SCI]-group, there exist a finite set S of G such that G/H = π(S), where π : G → G/H is the canonical projection. Define def

D = K ({e} ∪ S). Let L ∈ SUB (G) such that D ⊆ U L. As K ⊂ D, K ⊆ U L ∩ H = U (L ∩ H ) and so L ∩ H = H . Then H ⊆ L. As S ⊆ U L = L, then H S ⊆ L H = L and so G = L.  Remark 3.9 Combining Propositions 3.7 and 3.8, if a locally compact group G contains an open normal [SCI]-subgroup H , then G ∈ [SCI] if and only if G/H ∈ [SCI]. The following proposition is due to Gelander and Levit [12, Proposition 6.3] Proposition 3.10 Let G be a locally compact group and H an open subgroup of G. Assume that there are only finitely many intermediate subgroups H ≤ L ≤ G. If H is self-Chabauty-isolated then G is self-Chabauty-isolated as well. Proof We present here another proof of this result. As H is open in G, the mapping φ : SUB (G) → SUB (H ) , L → L ∩ H is continuous (Proposition 1.2 of [14]) and so, φ−1 ({H }) is an open finite subset of SUB (G). Then, since SUB (G) is Hausdorff, each element of φ−1 ({H }) is an isolated point in SUB (G). As G ∈ φ−1 ({H }) the result follows. The above assumption holds in particular when H is maximal or of finite index in G. 

4 A Necessary and Sufficient Condition for Connected Locally Compact Groups As an immediate consequence of Proposition 1.4 we have Corollary 4.1 Every locally compact abelian [SCI]-group is totally disconnected. Proof We present here an alternative direct proof of this result. Let U be a neighborhood of e in G. In view of Lemma 3.4, G is compactly generated and so, by Theorem 9.6 of [18], U contains a compact subgroup H such that G/H is topologically isomorphic with Ra × Tb × Zc × F where a, b and c are nonnegative integers and F is a finite abelian group. On the other hand, by Proposition 3.7, G/H ∈ [SCI] and so, by (3) and (4) of Examples 3.3, a = b = 0. Then G/H is discrete and so H is

66

H. Hamrouni and F. Sadki

open in G. Consequently, every neighborhood of e in G contains an open subgroup and so G is totally disconnected.  Let G be a group. For a, b ∈ G write [a, b] = aba −1 b−1 . Such an expression is called a commutator of G. The commutator subgroup of G is defined to be the subgroup D (G) generated by all the commutators [a, b] of G. Proposition 4.2 Let G be a locally compact group with identity component G 0 . If G ∈ [SCI] then G 0 ⊆ D (G). Proof By Proposition 3.7, the locally compact abelian group G/D (G) ∈ [SCI] and  so it is totally disconnected. Then G 0 ⊆ D (G). Definition 4.3 (Topologically perfect group) A topological group is called topologically perfect if its commutator subgroup is dense. Remark 4.4 (1) G is topologically perfect if and only if it has no non-trivial continuous homomorphism to an abelian topological group. (2) The class of topologically perfect groups is closed under continuous homomorphic images. In particular, the class of topologically perfect groups is stable under taking quotients by normal subgroups. Corollary 4.5 A necessary condition for a connected locally compact group G to be an [SCI]-group is that it is topologically perfect. This condition is sufficient for connected Lie groups. The requirement that G is a Lie group is crucial for Proposition 1.5 as explained by the next example. To state this example, we need some facts about commutator subgroups in compact groups. Proposition 4.6 Let G be a compact group with identity component G 0 . (1) If G/G 0 is topologically finitely generated and H a closed normal subgroup of H , then the commutator subgroup [H, G] is closed in G. In particular, the commutator subgroup of a connected compact topological group is closed. (2) If G is connected, then the function G × G → D (G) , (x, y) → [x, y] = x yx −1 y −1 is surjective. Proof For (1), see Theorem 1.1 of [22]. For (2), we refer to Theorem 9.2 of [19].  Example 4.7 Let G be a compact connected perfect Lie group and let def

G=



G = Map(N, G).

k∈N

Then G is a connected compact perfect group which is not an [SCI]-group. Indeed, it is clear that G is connected and compact. By Proposition 4.6 the group G is perfect. The sequence of proper closed subgroups

Self-Chabauty-isolated Locally Compact Groups

67

G (n) = {(xk ) ∈ G | (∀ k ≥ n + 1) xk = e} def

converges to G and so G is not isolated in SUB (G). For a topological group G we denote by G 0 the connected component of the identity e. A topological group G is said to be almost connected if the factor group G/G 0 of G modulo the identity component G 0 is compact. Proposition 4.8 Let G be an almost connected Lie group. If the identity component G 0 ∈ [SCI], then G ∈ [SCI]. Proof As G 0 has a finite index in G, the result follows from Proposition 3.8.



We denote by CoLie (G) the set of compact subgroups N of G such that G/N is a Lie group. Proposition 4.9 Let G be a locally compact connected group. Then the following are equivalent. (1) G ∈ [SCI]; (2) G is topologically perfect and there is N ∈ CoLie (G) such that N H = G for every closed subgroup H of G with H = G. Proof (1) =⇒ (2): By Proposition, for every N ∈ CoLie (G) the connected Lie group G/N ∈ [SCI]. As {G/N } is a fundamental system of   open neighborhoods of G/N in SUB (G/N ), then p −1 (G/N ) | N ∈ CoLie is a fundamental system (G) N of open neighborhoods of G in SUB (G). As {G} is open in SUB (G), there exists N ∈ CoLie (G) such that {G} = p −1 N (G/N ) = {L ∈ SUB (G) | L N = G}. (2) =⇒ (1): As G/N is a topologically perfect connected Lie group, {G/N } is open in SUB (G/N ). The continuity of the mapping SUB (π) : SUB (G) → SUB (G/N ) , L → π(L) where π : G → G/N is the canonical projection (Corollary 2.4 of [15]), implies that the set {L ∈ SUB (G) | L N = G} = {G} is open in SUB (G) and so G is an isolated point in SUB (G). 

5 Locally Compact Totally Disconnected [SIN]-Groups Recall that for a profinite group G, Frat (G) = TFrat (G) (see Proposition 2.16) and so Definition 2.9 coincides with the classical definition of the Frattini subgroup for profinite groups (see Sect. 2.8, pages 52–53, of [23]). Proposition 5.1 (Characterization of profinite [SCI]-groups) Let G be a profinite group. Then G is self-Chabauty-isolated if and only if its Frattini subgroup Frat (G) is open in G.

68

H. Hamrouni and F. Sadki



Proof See Theorem 5.6 of [10].

The following characterization of profinite groups with open Frattini subgroup can be found in [10, Theorem 5.6] (see also [25, Proposition, page 148]). Proposition 5.2 Let G be a profinite group. The following properties are equivalent: (1) There is an open normal subgroup U of G such that HU = G for every closed normal subgroup H of G with H = G. (2) The Frattini subgroup Frat (G) of G is open. (3) The prime numbers dividing the order of G are finite in number, and the Sylow subgroups of G are (topologically) finitely generated.  (4) There is an open normal subgroup N of G with N = p∈P N p , where P is a finite set of primes, and each N p is a finitely generated pro- p-group. For the following consequence of Proposition 5.1 recall that a pro- p group is a profinite group in which every open normal subgroup has index equal to some power of p. Corollary 5.3 A pro- p group is self-Chabauty-isolated if and only if it is topologically finitely generated. Proof This follows from the fact that a pro- p group G is topologically finitely generated if and only if TFrat (G) is open in G (see Proposition 1.4 of [8] or Proposition 2.8.10 of [23]).  Proposition 5.4 Let G be a topologically finitely generated locally compact group. If Frat (G) is open then G ∈ [SCI]. Proof Let S be a finite set of G such that G = S. For any H ∈ U(G; S, Frat (G)) we have S ⊆ H Frat (G). As H Frat (G) is an open subgroup of G, S ⊆ H Frat (G) and hence G = H Frat (G). By Proposition 2.18, H = G. Then U(G; S, Frat (G)) = {G}, and the proof is complete. The following is a generalization of Proposition 5.1.  Proposition 5.5 Let G be a topologically finitely generated totally disconnected locally compact [SIN]-group. Then the following statements are equivalent: (1) G ∈ [SCI]; (2) Frat (G) is open; (3) G has an omissible compact open normal subgroup; that is, there is a compact open normal subgroup N such that N H = G for every closed subgroup H of G with H = G. Proof (1) =⇒ (2): There exist a compact subset K of G and an open normal compact subgroup N of G such that ∀ H ∈ SUB (G) ,

K ⊆ N H =⇒ H = G.

Self-Chabauty-isolated Locally Compact Groups

69

Let g ∈ N and let X be a subset of G such that G = g, X . We have K ⊆ N g, X  ⊆ N N , X  ⊆ N X  and so X  = G. Then g is a non-generator element of G and therefore N ⊆ Frat (G). Consequently, Frat (G) is an open subgroup of G. The implication (2) =⇒ (1) follows from Proposition 5.4. (1) ⇐⇒ (3): Follows from Proposition 3.1. 

References 1. Abert, M., Bergeron, N., Biringer, I., Gelander, T., Nikolov, N., Raimbault, J., Samet, I.: On the growth of L 2 -invariants for sequences of lattices in Lie groups. To appear in Annals of Mathematics 2. Benedetti, R., Petronio, C.: Lectures on hyperbolic geometry. Springer, Berlin (1992) 3. Bridson, M., De La Harpe, P., Kleptsyn, V.: The Chabauty space of closed subgroups of the three-dimensional Heisenberg group. Pacific J. Math. 240, 1–48 (2009) 4. Caprace, P.-E., Monod, N.: Decomposing locally compact groups into simple pieces. Math. Proc. Camb. Philos. Soc. 150, 97–128 (2011) 5. Chabauty, C.: Limite d’ensemble et géométrie des nombres. Bull. Soc. Math. France 78, 143– 151 (1950) 6. De Cornulier, Y.: On the Chabauty space of locally compact abelian groups. Algebr. Geom. Topol. 11, 2007–2035 (2011) 7. De la Harpe, P.: Spaces of closed subgroups of locally compact groups. arXiv:0807.2030v2 [math GR]. Last accessed 12 Nov 2008 8. Dixon, J.D., Du Sauty, M.P.F., Mann, A., Segal, D.: Analytic- p groups, 2nd edn. Cambridge University Press, Cambridge (2003) 9. Fisher, S., Gartside, P.: On the space of subgroups of a compact group I. Topology Appl. 156, 862–871 (2009) 10. Gartside, P., Smith, P.: Counting the closed subgroups of profinite groups, J. Group Theory 13, 41–61 (2010) 11. Gelander, T.: Lecture notes on invariant random subgroups and lattices in rank one and higher rank. arXiv:1503.08402 12. Gelander, T., Levit, A.: Invariant random subgroups over non-archemedean local fields. arXiv:1503.08402 13. Glasner, Y., Kitroser, D., Melleray, J.: From isolated subgroups to generic permutation representations. J. London Math. Soc. 94, 688–708 (2016) 14. Haettel, T.: L’espace des sous-groupes fermés de R × Z. Algebr. Geom. Topol. 10, 1395–1415 (2010) 15. Hamrouni, H., Kadri, B.: On the compact space of closed subgroups of locally compact groups. J. Lie Theory 23, 715–723 (2014) 16. Hamrouni, H., Kadri, B.: On the connectedness of the Chabauty space of a locally compact prosolvable group. Adv. Pure Appl. Math. 6, 97–111 (2015) 17. Hamrouni, H., Firas, S.: On the continuity of the centralizer map of a locally compact group. J. Lie Theory 26, 117–134 (2016) 18. Hewitt, E., Ross, K.A.: Abstract Harmonic Analysis I. Springer, Berlin (1963) 19. Hofmann, K.H., Morris, S.A.: The structure of compact groups. De Gruyter, Berlin (1998). Revised and Augmented, 3rd edn (2013)

70

H. Hamrouni and F. Sadki

20. Hofmann, K.H., Morris, S.A.: Open mapping theorem for topological groups. Topol. Proc. 31, 533–551 (2007) 21. Hofmann, K.H., Morris, S.A., Stroppel, M.: Varieties of topological groups, Lie groups and SIN-groups. Colloq. Math. LXX, 151–163 (1996) 22. Nikolov, N., Segal D.: On normal subgroups of compact groups. J. Eur. Math. Soc. 16, 597–618 (2014) 23. Ribes, L., Zalesskii, P.A.: Profinite Groups, 2nd edn. Springer, Berlin (2011) 24. Robinson, D.J.S.: A course in the theory of groups (Second Edition), Graduate Texts in Mathematics, vol. 80. Springer, Berlin (1982) 25. Serre, J.P.: Lectures on the Mordell-Weil theorem. Vieweg, Braunschweig (1989) 26. Willis, G.: Totally disconnected, nilpotent, locally compact groups. Bull. Austral. Math. Soc. 55, 143–146 (1997)

Quantization of Color Lie Bialgebras Benedikt Hurle and Abdenacer Makhlouf

Abstract The main purpose of this paper is to study Quantization of color Lie bialgebras, generalizing to the color case the approach by Etingof–Kazhdan which was considered for superbialgebras by Geer. Moreover we discuss Drinfeld category, Quantization of Triangular color Lie bialgebras and Simple color Lie bialgebras of Cartan type. Keywords Quantization · Color Lie bialgebra · Color Hopf algebra

Introduction Lie bialgebras appeared in the Eighties mostly due to Drinfeld [2, 3] and SemenovTian-Shansky [17], who introduced Poisson–Lie groups and discussed the relationships with the concept of a classical r -matrix and Yang–Baxter equation. The Lie algebra of a Poisson–Lie group has a natural structure of Lie bialgebra, the Lie group structure gives the Lie bracket as usual, and the linearization of the Poisson structure on the Lie group gives the Lie bracket on the dual of the Lie algebra. Deformations and their relationships with cohomology of Lie algebras were discussed by Nijenhuis– Richardson, following Gerstenhaber’s approach, in [14] while cohomology of Lie bialgebras where studied first in [13]. One of the main problems formulated by Drinfeld in quantum group theory was the existence of a universal quantization for Lie bialgebras, which arose from the problem of quantization of Poisson Lie groups. This issue was solved by Etingof–Kazhdan in [6] using the methods and ideas from [11].

B. Hurle · A. Makhlouf (B) IRIMAS-département de mathématiques, Université de Haute-Alsace, 6 bis rue des Frères Lumière, 68093 Mulhouse, France e-mail: [email protected] B. Hurle e-mail: [email protected] © Springer Nature Switzerland AG 2019 A. Baklouti and T. Nomura (eds.), Geometric and Harmonic Analysis on Homogeneous Spaces, Springer Proceedings in Mathematics & Statistics 290, https://doi.org/10.1007/978-3-030-26562-5_3

71

72

B. Hurle and A. Makhlouf

Lie superbialgebras and Poisson–Lie supergroups were studied in [1]. Quantization functors have been constructed for Lie superbialgebras and group Lie bialgebras in [5, 8]. Color Lie algebras are a natural generalization of Lie superalgebras where the grading is defined by any abelian group and a commutation factor. They have become an interesting subject of mathematics and physics. The first study and cohomology of color Lie algebras were considered by Scheunert in [15, 16]. Moreover, various properties were studied in color setting, see [9] and references therein. We aim in this paper to discuss quantization of color Lie bialgebras, by adapting the method of Etingof–Kazhdan [6], which was already considered in the supercase by Geer in [8]. In the first section, we provide some preliminaries about colored structures and quantized universal enveloping algebras. In Sect. 2 Drinfeld associator and category are discussed. Section 3 includes the main results about Quantization of color Lie bialgebras. Section 4 deals with triangular color Lie bialgebras and in Sect. 5 a second quantization is presented. The last section provides a discussion about simple color Lie bialgebras of Cartan Type.

1 Preliminaries In this section we give the basic definitions of color vector space, Lie bialgebra and so on. We also make some remarks regarding category theory, which we will need later to use in the construction of Etingof–Kazhdan. In fact, we will need a bit of enriched category theory see e.g. [12]. Throughout the paper K will be a fixed field of characteristic 0.

1.1 Graded and Color Vector Spaces Definition 1 (Commutation factor) Let Γ be an abelian group. Then a map ε : Γ × Γ → K× is called an anti-symmetric bicharacter or commutation factor if ε( f + g, h) = ε( f, h)ε(g, h),

(1)

ε( f, g + h) = ε( f, g)ε( f, h), ε( f, g)ε(g, f ) = 1.

(2) (3)

We note that the product of two commutation factors is again a commutation factor. Let Γ  be an abelian group, and for each g ∈ Γ let Vg be a vector space, then we call V = g∈Γ Vg a Γ -graded vector space. Definition 2 A (Γ, ε) color vector space is a Γ -graded vector space V , together with a commutation factor ε. In the following a color vector space will always be with respect to a fixed Γ and ε.

Quantization of Color Lie Bialgebras

73

Given an abelian group Γ and a commutation factor ε on it, one can define the category grVecΓ ε of (Γ, ε)-color vector spaces. This category is closed monoidal, where the tensor product is given by (V ⊗ W )i =



V j ⊗ Wk ,

(4)

j+k=i

and the internal homs, called the graded morphisms, are given by Homgr(V, W ) =



Homgri (V, W ),

(5)

i∈Γ

Homgri (V, W ) = {φ : V → W linear|φ(V j ) ⊂ W j+i }.

(6)

It is clear that Homgr(V, W ), is again a color vector space. The morphisms in the ordinary category grVecΓ ε are given by the graded morphisms of degree 0. We further define the flip τV ⊗W : V ⊗ W → W ⊗ V by v ⊗ w → ε(|v|, |w|)w ⊗ v, if the involved spaces are clear, we simply write τ . With this grVecΓ ε becomes a symmetric monoidal category. We also note that grVecΓ ε is an enriched category over Vec. Another, maybe nicer, way to define it is to consider the category Γˆ , enriched over Vec, which has as objects the elements of Γ , and as morphisms Hom(g, g) = K and Hom(g, h) = 0 for g = h for all g, h ∈ Γ . One can make Γ into a monoidal category by defining the tensor product to be the addition. Then one can consider the associator to be trivial. A commutation factor gives a symmetry on this category. With this structure the Vec-functor category Hom(Γˆ , Vec) is again a symmetric monoidal category and isomorphic to grVecΓ ε . In the following we will mostly deal with a fixed grading group and commutation factor and simply write grVec instead of grVecΓ ε . Throughout this paper, we will use the Koszul rule, this means given graded morphisms φ and ψ we have (φ ⊗ ψ)(x ⊗ y) = ε(ψ, x)φ(x) ⊗ ψ(y), with ε(ψ, x) denoting ε(|ψ|, |x|) for short and where | · | is the degree.

1.2 Color Lie Algebras In this section we give the basic definitions of color Lie algebras and bialgebras, for more details see e.g. [9]. Definition 3 (Color Lie algebra) For a group Γ and a commutation factor ε, a (Γ, ε)-color Lie algebra is a Γ -graded vector space g with a graded bilinear map [·, ·] : g × g → g of degree zero, such that for any homogeneous elements a, b, c ∈ g

74

B. Hurle and A. Makhlouf

[a, b] = −ε(a, b)[b, a], j (a, b, c) := ε(c, a)[a, [b, c]] + ε(a, b)[b, [c, a]] + ε(b, c)[c, [a, b]] = 0.

(7) (8)

The second equation is called color Jacobi identity. It can also be written as [a, [b, c]] = [[a, b], c] + ε(a, b)[b, [a, c]],

(9)

which shows that the adjoint representation ada (b) := [a, b] for a ∈ g is a color Lie algebra derivation. Using σ (a ⊗ b ⊗ c) = ε(a, bc)b ⊗ c ⊗ a, β(a, b) = [a, b] and the Koszul rule, it can also be written as β(β ⊗ id)(id + σ + σ 2 ) = 0.

(10)

A morphism φ of color Lie algebras (g, [·, ·]) and (h, [·, ·] ) is a morphism of color vector spaces such that [φ(x), φ(y)] = φ([x, y]).

(11)

We will usually only consider morphism of degree zero. An ideal of a color Lie algebra g is a graded subspace i such that [i, g] ⊂ i. We call a color Lie algebra simple if it has no color Lie ideal. Note that it can have a non graded ideal. If i ⊂ g is a color Lie ideal then the quotient g i is again a color Lie algebra. A color Lie subalgebra is a graded subspace h such that [h, h] ⊂ h. Let A be a Γ -graded associative algebra and ε a commutation factor then [a, b] = ab − ε(a, b)ba

(12)

defines a color Lie algebra structure. We denote the corresponding color Lie algebra by A L . So we get especially a Lie bracket on the graded homomorphisms of a color vector space, for which we have: Proposition 4 The color derivations Der(A) of a color algebra A form a color Lie subalgebra of Homgr(A). Definition 5 (Universal enveloping algebra) For a color Lie algebra g one defines the universal enveloping algebra, or short UEA, U (g) by the tensor algebra T (g) modulo the ideal generated by elements of the form x y − ε(x, y)yx − [x, y]

(13)

for x, y ∈ g ⊂ U (g). Theorem 6 ([15]) The universal enveloping algebra U (g) is an (filtered) associative color algebra. With the graded commutator it is a color Lie-algebra, with g as a color Lie subalgebra. It has the universal property that is for any color algebra A

Quantization of Color Lie Bialgebras

75

and color Lie algebra homomorphism f : g → A L there exists a unique algebra homomorphisms such that f = g|g . In fact one has the structure of a color Hopf algebra on U (g). The coproduct is given by Δ(x) = x ⊗ 1 + 1 ⊗ x for x ∈ g ⊂ U (g) and extended to the rest of U (g) by the universal property stated in the theorem above. The Lie algebra g is precisely formed by the primitive elements in U (g), where an element x ∈ U (g) is called primitive if it satisfies Δ(x) = 1 ⊗ x + x ⊗ 1.

1.3 Color Lie Bialgebras For the definition of color Lie bialgebras, we first need to define the adjoint action on tensor powers of g. Let a ∈ g and x ⊗ y ∈ g ⊗ g then, we set a · (x ⊗ y) = (a · x) ⊗ y + ε(a, x)x ⊗ (a · y).

(14)

This can be generalized to higher tensor products. Definition 7 (Color Lie bialgebra) A color Lie bialgebra is a color Lie algebra g with a color antisymmetric cobracket δ : g → g ⊗ g of degree 0, such that the compatibility condition δ([a, b]) = a · δ(b) − ε(a, b)b · δ(a)

(15)

holds and δ satisfies the co-Jacobi identity given by (id + σ + σ 2 )(δ ⊗ id)δ = 0.

(16)

It is called quasitriangular, if there exits an r ∈ g ⊗ g, such that r + τ (r ) is g invariant and satisfies CYB(r ) = 0 with CYB = [r12 , r13 ] + [r12 , r23 ] + [r13 , r23 ].

(17)

It is called triangular if in addition, r = −τ (r ). We give an example of a color Lie bialgebra which is not a super-Lie algebra, the three dimensional color sl2 Lie algebra g. It is Z32 graded, but the only nonvanishing summands are g(1,1,0) =< e1 >, g(1,0,1) =< e2 >, g(0,1,1) =< e3 >. The commutation factor is given by ε((i 1 , i 2 , i 3 ), ( j1 , j2 , j3 )) = (−1)i1 j2 +i1 j3 +i2 j1 +i2 j3 +i2 j3 +i3 j1 . The Lie bracket is given by [e1 , e2 ] = e3

[e3 , e1 ] = e2

[e2 , e3 ] = e1 .

The other brackets are given by antisymmetry. The aim is to find a cobracket on g such that we get a color Lie bialgebra. Because of the grading, it must hold that

76

B. Hurle and A. Makhlouf

δ(e1 ) = γ1 e2 ∧ e3

δ(e2 ) = γ2 e3 ∧ e1

δ(e3 ) = γ3 e1 ∧ e2

For every choice of the γi this satisfies the co-Jacobi identity. But as a calculation shows this only gives a color Lie bialgebra if γi = 0 for i = 1, 2, 3. Definition 8 (Color Manin triple) A color Manin triple is a triple (p, p+ , p− ), where p is a color Lie algebra, p± are color Lie subalgebras of p and p = p+ ⊕ p− as color vector spaces, with a non-degenerate invariant symmetric inner product (·, ·) on p, such that p± are isotropic, i.e. (p± , p± ) = 0. Invariant here means that ([a, b], c) + ε(a, b)(b, [a, c]) = 0,

(18)

and symmetric means that (a, b) = ε(a, b)(b, a).

(19)

Note that the invariance can also be written as ([b, a], c) = (b, [a, c]).

(20)

Theorem 9 Let g be a color Lie bialgebra and set p+ = g, p− = g∗ and p = p+ ⊕ p− . Then (p, p+ , p− ) is a color Manin triple. Conversely any finite-dimensional color Manin triple p gives rise to a Lie bialgebra structure on p+ . In the following let {xi }i be a basis of g and {α i }i be the corresponding dual basis of g∗ , i.e. α i (x j ) = δ ij , where δ is the Kronecker delta. Proposition 10 (Double) If p is finite dimensional, there is also the structure of a color Lie bialgebra on p given by the r -matrix r = xi ⊗ α i . For this we have δ(x) = x · r = [x, xi ] ⊗ α i + ε(i, x)xi ⊗ [x, α i ]. This color Lie bialgebra is called the double of g, and it is denoted by D(g). For explicit proofs, see e.g. [9]. On D(g) we define the Casimir element Ω = r + τ ◦ r , which is invariant, i.e. x · Ω = 0 for all x ∈ g. We consider the category of all (Γ, ε) color Lie bialgebras LBAΓ ε . The morphisms are given by the graded morphisms, which preserve the color Lie bracket and cobracket. Note that this category is not enriched over grVec, since the sum of two morphisms is in general not a morphism again.

1.4 Topologically Free Modules For the quantization we need Γ -graded K[[λ]]-modules, where K[[λ]] denotes the ring of formal power series over K. So we first recall some basics from the nongraded case. A K[[λ]]-module is called topologically free if it is isomorphic to one

Quantization of Color Lie Bialgebras

77

ˆ K[[λ]] for a K-vector space V . Given a K-module V , we will simply of the form V ⊗ ˆ denotes the completed tensor product with respect to denote it by V [[λ]]. Here ⊗ the filtration by λ or the λ-adic topology. A Γ -graded K[[λ]]-module V = Vi is called  free, if all Vi are free. This means ˆ K[[λ]]. there exists a Γ -graded vector space W = i∈Γ Wi such that Vi = Wi ⊗ We will simply write V = W [[λ]]. In general this is not equivalent to the statement  that i∈Γ Vi is free. But it is equivalent if only finitely many Vi are nonzero.

1.5 Quasitriangular Color Quasi-Hopf Algebras Definition 11 (Color quasi-Hopf algebra) A color quasi-Hopf algebra is an associative color algebra H with a multiplication μ, a coproduct Δ, a unit 1, a counit ε and an invertible associator Φ ∈ H ⊗3 all of degree 0, which satisfy ∀x, y ∈ H : Δ(x y) = Δ(x)Δ(y) (ε ⊗ id)Δ = id = (id ⊗ ε)Δ Φ(Δ ⊗ id)Δ = (id ⊗ Δ)ΔΦ Φ1,2,34 Φ12,3,4 = Φ2,3,4 Φ1,23,4 Φ1,2,3

(compatibility), (counit), (quasi- coassociativity), (Pentagon identity),

(id ⊗ ε ⊗ id)Φ = 1 ⊗ 1. Here we used the shorthand notation Φ1,23,4 = (id ⊗ Δ ⊗ id)Φ, Φ2,3,4 = 1 ⊗ Φ and similarly for the others. It also has an antipode S, which satisfies μ(id ⊗ S)Δ = id = μ(id ⊗ S)Δ. One could also allow for left and right unitors, but we will not do so. Note that since these equations use the product in H ⊗ H , defined by (a ⊗ b)(c ⊗ d) = ε(b, c)ac ⊗ bd for a, b, c, d ∈ H , they depend on the commutation factor ε. We assume also the operations to be of degree zero for two reasons, it is easier to handle this way categorically, and actually all operations have to be of degree zero, because they respect the unit. A color quasi-Hopf algebra, where Φ = 1 ⊗ 1 ⊗ 1 is simply called a color Hopf algebra. A color quasi-Hopf algebra H is called quasitriangular, if there exists also an r -matrix R ∈ H ⊗2 , of degree 0, such that −1 −1 R13 Φ213 R12 Φ123 , (id ⊗ Δ)R = Φ231

(21)

−1 R23 Φ123 , Φ312 R13 Φ132

(22)

(Δ ⊗ id)R = RΔ

opp

= ΔR.

(23)

Here Δopp (x) = τ Δ(x), Φ312 = τ H,H ⊗H Φ and similarly for the other permutations. It is called triangular if R21 R = id.

78

B. Hurle and A. Makhlouf

From Eq. (22) and the fact the R is invertible, it follows that (ε ⊗ id)R = 1 = (id ⊗ ε)R, so R is automatically of degree 0. Two quasitriangular quasi-Hopf algebras H and H  are called twist equivalent if there exists an invertible element J ∈ H ⊗2 of degree 0 and an algebra isomorphism θ : H → H  , such that (ε ⊗ id)J = 1 = (id ⊗ ε)J, Δ = J −1 ((θ ⊗ θ )Δθ −1 (x))J, −1 −1 J1,23 θ (Φ)J12,3 J1,2 , Φ  = J2,3 −1 R  = J21 R J.

In the following we will be mostly interested in the case, where H = H  and θ is the identity. If J satisfies the first identity above, one can define a new twist equivalent quasitriangular quasi-Hopf algebra by using the identities as definitions for Δ , μ and R. Theorem 12 The category of modules over a quasitriangular color quasi-Hopf algebra is a braided monoidal category, enriched over grVec. And if two Hopf algebras are twist equivalent, the category of modules are tensor equivalent, i.e. it exists an invertible monoidal functor between them. There is a category of ((quasi)-triangular) color quasi-Hopf algebras, where the morphisms preserve the product, coproduct, unit, counit and the associator. In the case of (quasi-)triangular-Hopf algebras they also preserve the r -matrix. Since the morphism satisfies φ(1) = 1 they have to be of degree 0 if they are homogeneous. So this category is only an ordinary one.

1.6 Quantized Universal Enveloping Algebras Definition 13 (QUEA) Let H be a topological free color Hopf  K[[λ]]-algebra. Then it is called a quantized universal enveloping algebra, if H λH ∼ = U (g), for a color Lie algebra g. If g is a color Lie bialgebra then H is a quantization of it if in addition δ=

1 (Δ − Δopp ) mod λ. λ

(24)

Let H be a quantization of a quasitriangular color Lie bialgebra. Then (H, R) is called a quasitriangular quantization if (H, R) is a quasitriangular Hopf algebra and R ≡ 1 ⊗ 1 + λr mod λ2 .

Quantization of Color Lie Bialgebras

79

2 Drinfeld Category 2.1 Associator Following [6], let Tn be the algebra over K generated by symmetric elements ti j for 1 ≤ i, j ≤ n, satisfying the relations ti j = t ji , [ti j , tlm ] = 0 if i, j, l, m are distinct and [ti j , tik + t jk ] = 0. For disjoint sets P1 , . . . , Pn ⊂ {1, . . . , m}, there exists a unique homomorphism Tn → Tm defined on generators by ti j → p∈Pi ,q∈P j t pq . We denote it by X → X P1 ,...,Pn . For Φ ∈ T3 , the relation Φ1,2,34 Φ12,3,4 = Φ2,3,4 Φ1,23,4 Φ1,2,3

(25)

λ

is called the pentagon relation, and for R = e 2 t12 ∈ T2 [[λ]] the relations −1 R2,3 Φ1,2,3 R12,3 = Φ3,1,2 R1,3 Φ1,2,2

R1,23 =

−1 −1 Φ2,3,1 R12,3 Φ2,1,3 R12,3 Φ1,2,3

(26) (27)

are called the hexagon relations. There is the following well known theorem due to Drinfeld [4]. Theorem 14 There exists an associator over Q. This means that there is also an associator for every field, which includes the rational numbers. In the following we will fix such an associator for K. For a color Lie algebra g, with a symmetric invariant element Ω = Ω1 ⊗ Ω2 , we can define a map from Tn to End(M1 ⊗ · · · ⊗ Mn ), by setting ti j → Ωi j . Here Ωi j is 1 ⊗ · · · ⊗ Ω1 ⊗ · · · ⊗ Ω2 ⊗ · · · ⊗ 1 with the components of Ω in the i-th and j-th factor in the tensor product. If i > j, we have that Ωi j = τ Ω ji . So the Ωi j satisfy the same relations as the ti j since Ω is invariant. As in the non-color case we get the following theorem. Theorem 15 We get a quasitriangular color quasi-Hopf algebra (U (g)[[λ]], Δ, ε, Φ, R), which we denote by Ag,t .

2.2 Drinfeld Category Let g+ be a finite dimensional color Lie algebra, and g = D(g+ ) = g+ ⊕ g− be the Drinfeld double of g+ with its Casimir Ω. Since Ω is invariant and symmetric, we get a quasitriangular color quasi-Hopf algebra Ag,Ω . We define the category Mg , whose objects are g-modules and whose morphisms are given by (28) HomM g (V, W ) = Homg (V, W )[[λ]].

80

B. Hurle and A. Makhlouf

Note: We consider this category to be enriched over grVec so the homomorphisms here are graded in general. We equip Mg with the usual tensor product, symmetry given by βV,W : V ⊗ W → W ⊗ V, v ⊗ w → τ exp( λΩ )v ⊗ w and associator 2 ΦV,W,U : (V ⊗ W ) ⊗ U → V ⊗ (W ⊗ U ), v ⊗ w ⊗ u → Φ · (v ⊗ w ⊗ u). (29) This is called Drinfeld category. In fact it is just the category of modules over the quasitriangular color quasi-Hopf algebra Ag,Ω .

3 Quantization of Color Lie Algebras Let A be the category of topological free graded K[[λ]]-modules, considered as grVec-category. Let g+ be a finite dimensional color Lie algebra and g = D(g+ ) be its double. We consider the Verma modules M+ = U (g) ⊗U (g+ ) c+

M− = U (g) ⊗U (g− ) c− ,

and

where c± is the trivial one dimensional U (g± )-module, concentrated in degree 0. The module structure on M± comes from its definition as Verma module by acting on U (g). Remark 1 It should be clear that given a graded algebra A, a right A-module M and a left A-module N , the tensor product M ⊗ A N is well defined and again a graded vector space. If M, was in fact a B-A-bimodule, it is a left B-module, and similarly for N . There is also a nice definition for the tensor product over A using a coend. A graded algebra A can be considered as a category A enriched over grVec with only one object, which we denote by ∗. A left (resp. right) A-module is precisely a functor from A (resp. A opp ) to grVec. Then the tensor product over A can be defined as  M ⊗ A N :=

a∈A

M(a) ⊗ N (a),

(30)

where the integral symbol denotes a coend. Note: By the PBW Theorem, we have an isomorphism U (g+ ) ⊗ U (g− ) ∼ = U (g) of vector spaces, which is given by the multiplication in U (g) and in general U (g ⊕ h) ∼ = U (g) ⊗ U (h) as vector space. This implies that (31) M± = U (g∓ )1± , with 1± in M± . So M± are free U (g∓ )-modules.

Quantization of Color Lie Bialgebras

81

Lemma 1 There exists an isomorphism φ : U (g) → M+ ⊗ M− of U (g)-modules of degree 0 given on generators by 1 → 1+ ⊗ 1− . Proof It is well defined by the universal property of U (g), as the extension of x → x1+ ⊗ 1− + 1+ ⊗ x1− . It is an isomorphism since U (g) and M± can be regarded as free connected coalgebras and φ clearly is an isomorphism on the primitive elements. Next we define a grVec-functor F : Mg → A by F(V ) = HomM g (M+ ⊗ M− , V ).

(32)

Since this is just a Hom functor, its definition on morphisms is clear. The isomorphism from Lemma 1 gives an isomorphism ΨV : F(V ) → V [[λ]], f → f (1+ ⊗ 1− ).

(33)

So the functor F is naturally isomorphic to the “forgetful” functor. We want to show that F is a tensor functor, for this we need a natural transformation J : ⊗K[[λ]] ◦ (F ⊗ F) → F ◦ ⊗M g , which also satisfies JU ⊗V,W ◦ (JU,V ⊗ idW ) = JU,V ⊗W ◦ (idU ⊗ JV,W ). Define i ± : M± → M± ⊗ M± by 1± → 1± ⊗ 1± and extended as g-module morphism. Clearly i ± is of degree 0. Lemma 2 The map i ± is coassociative, i.e. Φ ◦ (i ± ⊗ id) ◦ i ± = (id ⊗ i ± ) ◦ i ± . Proof Following [6, Lemma 2.3]. We only prove the identity for i + , since the proof for i − is analogous. Let x ∈ M+ . Then since the comultiplication in U (g− ) is coassociative, we have (i + ⊗ id)i + x = (id ⊗ i + )i + x.

(34)

Φ · (i + ⊗ id)i + x = (i + ⊗ id)i + x,

(35)

It is enough to show but since Φ is g invariant by definition, it is enough to show this for x = 1+ . This means Φ · 1+ ⊗ 1+ ⊗ 1+ = 1+ ⊗ 1+ ⊗ 1+ . Which follows directly from the fact that Ω annihilates 1+ ⊗ 1+ and the definition of Φ. We define J by −1 −1 JV,W (v, w) = (v ⊗ w) ◦ Φ1,2,34 ◦ (id ⊗ Φ2,3,4 ) ◦ β2,3 ◦ (id ⊗ Φ2,3,4 ) ◦ Φ1,2,34 ⊗ (i + ⊗ i − )

for v ∈ F(V ), w ∈ F(W ). Since all involved maps are of degree 0, J is also of degree 0.

82

B. Hurle and A. Makhlouf

The maps JV,W (v, w) can be represented by a diagram i + ⊗i −

Φ

→ M+ ⊗ ((M+ ⊗ M− ) ⊗ M− ) M+ ⊗ M− −−−→ (M+ ⊗ M+ ) ⊗ (M− ⊗ M− )) − β2,3

Φ

v⊗w

−→ M+ ⊗ ((M− ⊗ M+ ) ⊗ M− ) − → (M+ ⊗ M− ) ⊗ (M+ ⊗ M− ) −−→ V ⊗ W Actually in the graded case there is a better definition ⊗

JV,W : Hom(M+ ⊗ M− , V ) ⊗ Hom(M+ ⊗ M− , W ) − → Hom(M+ ⊗ M− ⊗ M+ ⊗ M− , V ⊗ W ) Hom(i + ⊗i − ,·)

Hom(β23 ,·)

−−−−−−−→ Hom(M+ ⊗ M+ ⊗ M− ⊗ M− , V ⊗ W ) −−−−−−−−−→ Hom(M+ ⊗ M− , V ⊗ W )

Theorem 16 The functor F together with the natural transformation J forms a tensor functor. Proof One only needs the check the equation F(ΦU V W ) ◦ JU ⊗V,W (JU,V ⊗ id) = JU,V ⊗W ◦ (id ⊗ JV,W ).

(36)

The proof given for this in [7] is diagrammatic so it also holds in the color case. Let End(F) be the color algebra of natural endomorphisms of F. In fact what we mean here is  Hom(F(V ), F(V )). (37) End(F) = V ∈M g

This is the end in the category grVec enriched over itself, so it is a color vector space. We need to use this definition since in the “classical” one the natural transformations must consist of morphisms of degree 0. A natural transformation in this sense consists of a family ηV of graded maps V → V , which satisfy the “normal” relation for a natural transformation modified with the commutation factor, this means ηW F( f ) = ε(η, f )F( f )ηV .

(38)

We say that η ∈ End(V ) is of degree i if all ηV are of degree i. So End(F) =  g∈Γ End g (F) is a graded vector space, where End g (F) consists of the natural transformations of degree g. Proposition 17 There is a canonical color algebra isomorphism Θ : U (g)[[λ]] → End(F), x → x·

(39)

where x· on the right stands for the action of x induced on every U (g)-module. Proof End(F) and F(M+ ⊗ M− ) are isomorphic due to the Yoneda Lemma for enriched categories. Since M+ ⊗ M− is isomorphic to U (g) as U (g)-module, End(F) is in fact isomorphic to U (g).

Quantization of Color Lie Bialgebras

83

Following [8]: We can identify F(V ) with V [[λ]]. The map Θ is injective since the action of U (g) on itself is injective. We want to show that it is surjective. Let η ∈ End(F), and we identify ηV with a map V [[λ]] → V [[λ]]. We define x := ηU (g) (1). We claim that ηV = x·. For y ∈ U (g) we define r y ∈ End(U (g)) by r y (z) = ε(y, z)zy for z ∈ U (g). We have ηU (g) (y) = ηU (g) (r y 1) = ε(x, y)r y ηU (g) (1) = ε(x, y)r y x = x y.

(40)

So we get ηU (g) = x·. Similarly one shows that ηV = x· for any free g-modules V and since any g-module is a quotient of a free one the claim follows. We define J ∈ (U (g) ⊗ U (g))[[λ]] by −1 J = (φ −1 ⊗ φ −1 )(Φ1,2,34 (1 ⊗ Φ2,3,4 )β2,3 (id ⊗ Φ2,3,4 )Φ1,2,34 (1+ ⊗ 1+ ⊗ 1− ⊗ 1− )).

(41) This means JU (g),U (g) (φ −1 ⊗ φ −1 )(1+ ⊗ 1− ). With this the natural transformation J can be identified with the action of the element J . Proposition 18 Using Ψ as defined in Eq. (33), we have J · (v ⊗ w) = ΨV ⊗W (JV,W (ΨV−1 (v) ⊗ ΨW−1 (w)))

(42)

for v ∈ V [[λ]], w ∈ W [[λ]]. Proof For each v ∈ V [[λ]], we define f v : M+ ⊗ M− → V by f v (x) = ε(v, x)x · v. Then f v (1+ ⊗ 1− ) = v, since 1+ ⊗ 1− is the unit in M+ ⊗ M− . So we have f v = Ψ −1 (v). Let θ1 ⊗ θ2 := (φ ⊗ φ)J ∈ (M+ ⊗ M− )⊗2 . Then the right hand side gives (JV,W (ΨV−1 (v) ⊗ ΨW−1 (w)))(1+ ⊗ 1− ) = ( f v ⊗ f w )(θ1 ⊗ θ2 ) = ε(w, θ1 ) f v (θ1 ) ⊗ f w (θ2 ) (43) = ε(w, θ1 )ε(v, θ1 )ε(w, θ2 )φ −1 (θ1 ) · v ⊗ φ −1 (θ2 ) · w. (44)

And the left hand side gives φ −1 (θ1 ) ⊗ φ −1 (θ2 )(v ⊗ w) = ε(θ2 , v)φ −1 (θ1 ) · v ⊗ φ −1 (θ2 ) · w. This equal since J and with this θ1 ⊗ θ2 are of degree 0. Lemma 3 We have

λ J ≡ 1 + r mod λ2 . 2

(45)

 Proof Recall that r = m i ⊗ pi , where the pi are a basis of g+ and the m i are the corresponding dual basis of g− . We have τ (r ) · (1− ⊗ 1+ ) = 0, since pi acts trivially on 1+ . So we have using Φ ≡ 1 mod λ2

84

B. Hurle and A. Makhlouf

J ≡ (φ −1 ⊗ φ −1 )(eλΩ23 /2 (1+ ⊗ 1− ⊗ 1+ ⊗ 1− ) mod λ2 λ ≡ 1 + (φ −1 ⊗ φ −1 )(r23 + τ (r23 ))(1+ ⊗ 1− ⊗ 1+ ⊗ 1− ) mod λ2 2 λ −1 ≡ 1 + (φ ⊗ φ −1 )(1+ ⊗ pi 1− ⊗ m i 1+ ⊗ 1− ) mod λ2 2 λ ≡ 1 + pi φ −1 (1+ ⊗ 1− ) ⊗ m i φ −1 (1+ ⊗ 1− ) mod λ2 2 λ ≡ 1 + r mod λ2 . 2 Definition 19 We can now define a color Hopf algebra H on U (g)[[λ]] by Δ = J −1 Δ0 J, ε = ε0 and S = Q S0 Q −1 , with Q = μ(S0 ⊗ id)J . We give the corresponding color Hopf algebra structure on End(V ) under the isomorphisms Θ. For this we first need. Lemma 4 We have End(F) ⊗ End(F) = End(F ⊗ F)

(46)

as an algebra. Proof In the classical case this is obvious, but it is not so in our case, since we consider the enriched natural transformations. Here it follows from the definition as an end. There is a natural coproduct on End(F) given by −1 Δ(a)V,W = JV,W aV ⊗W JV,W for a ∈ End(F)

(47)

and End(F) becomes a bialgebra with it. In fact it is a Hopf algebra. This follows directly from the fact that (F, J ) is a tensor functor, which gives that J is a twist. The twisted coproduct is precisely the one given here and the twisted Φ is trivial. We have that H ∼ = End(V ) as Hopf algebra. Proposition 20 The Hopf algebra H is a quantization of the color Lie bialgebra g. Moreover we define an r -matrix on H by λ

R = (J opp )−1 e 2 Ω .

(48)

Then (H, R) is a quasitriangular quantization of (g, r ).  Proof By definition H λH is isomorphic to U (g) as color Hopf algebra. From Lemma 3 and the definition of the coproduct it follows that Δ1 (x) = 21 r Δ0 (x) +

Quantization of Color Lie Bialgebras

85

opp 1 Δ (x)r = 21 [Δ0 (x), r ], so we have δ(x) = Δ1 (x) − Δ1 (x) 2 0 = [Δ0 (x), r ], since r + τ (r ) = Ω, which is g invariant.

= 21 [Δ0 (x), r − τ (r )]

R is an r -matrix, because it is obtained by twist, or explicit computation. From Lemma 3, we immediately get R ≡ 1 ⊗ 1 + λ( 21 r − 21 τ (r ) + 21 Ω) ≡ 1 ⊗ 1 + λr mod λ2 .

3.1 Quantization of g+ and g− As shown before, we have End(F) ∼ = End(M+ ⊗ M− ), since both are isomorphic to U (g) as color algebras. So we can define Uλ (g+ ) = F(M− ) and embed it into H via the map i : F(M− ) → End(M+ ⊗ M− ) given by i(x) = (id ⊗ x) ◦ Φ ◦ (i + ⊗ id)

(49)

for x ∈ F(M− ). Then i is injective, and satisfies i(x) ◦ i(y) = i(z)

(50)

for x, y ∈ F(M− ) and z = x ◦ (id ⊗ y) ◦ Φ ◦ (i + ⊗ id) ∈ F(M− ). So Uλ (g+ ) is a color subalgebra of H . We next want to show that it is indeed a color Hopf subalgebra, for this we need the following proposition. Proposition 21 The r -matrix of H is polarized which means that R ∈ Uλ (g+ ) ⊗ Uλ (g− ) ⊂ H ⊗ H . Proof Following [7, Lemma 19.4]. The defining equation of R is equivalent to −1 R ◦ β23 ◦ (i + ⊗ i − ) = β23 ◦ (i + ⊗ i − )

(51)

in Hom(M+ ⊗ M− , M+ ⊗ M− ⊗ M+ ⊗ M− ), where we regard R as an element in Hom(M+ ⊗ M− ⊗ M+ ⊗ M− , M+ ⊗ M− ⊗ M+ ⊗ M− ). Using the counit we have β23 (i + ⊗ i − ) = (id ⊗ εid ⊗ id ⊗ εid⊗)β34 (id ⊗ i + ⊗ i − ⊗ id)(i + ⊗ i − )

(52)

and using Eq. (51) in the middle and then the coassociativity of i + and i − we further get β23 (i + ⊗ i − ) = (id ⊗ εid ⊗ id ⊗ εid⊗)(id ⊗ R ⊗ id)β34 (id ⊗ i + ⊗ i − ⊗ id)(i + ⊗ i − ) = (id ⊗ εid ⊗ id ⊗ εid⊗)(id ⊗ R ⊗ id)β34 (i + ⊗ id ⊗ id ⊗ i − )(i + ⊗ i − ).

86

B. Hurle and A. Makhlouf

So we have proved R = (id ⊗ ε ⊗ id ⊗ id ⊗ ε ⊗ id) ◦ (id ⊗ R ⊗ id) ◦ (i + ⊗ id ⊗ id ⊗ i − ).

(53)

We define maps p± : Uh (g∓ )∗ → Uh (g± ) by p+ ( f ) = (id ⊗ f )(R) resp. p− (g) = (g ⊗ id)(R).

(54)

Let im p± be the image of p± and U± be the color algebra generated by it, then we have Lemma 5 U± is closed under the coproduct and the antipode. Proof Uses the hexagon identity. Lemma 6 We have Uh (g± ) ⊗K[[λ]] K((λ)) = U± ⊗K[[λ]] K((λ)). Proof The proof in [7, Lemma19.5] works also in the color case. Theorem 22 Uh (g± ) is a color Hopf algebra and a quantization of g+ . Proof From the previous lemmas it follows that, it is a color Hopf subalgebra of Uh .

4 Quantization of Triangular Color Lie Bialgebras Let a be a not necessarily finite dimensional triangular color Lie bialgebra, then we define g+ := {(1 ⊗ f )(r ), f ∈ a∗ }, g− := {( f ⊗ 1)(r ), f ∈ a∗ } and g = g+ ⊕ g− . One can identify g− with g+ via the map χ ( f ) = ( f ⊗ id)(r ). Then one can define a Lie bracket on g, such that for x, y ∈ g± it is the Lie bracket in g± and for x ∈ g+ , y ∈ g it is defined by [x, y] := (ad∗ x)(y) − ε(x, y)(ad∗ y)(x).

(55)

One can define a map π : g → a, such that the restriction to g+ and g− is the embedding. With this one has π([x, y]) = [π(x), π(y)].

(56)

Proposition 23 The bracket on g defined above actually is a color Lie bracket, and the natural pairing gives an invariant inner product, so in fact a color Manin triple. Let Ma be the category of a-modules, with morphisms given by Hom(V, W ) = Homa (V, W )[[λ]], this again can be viewed as a category enriched over grVec. Using morphisms π one can define the pullback functor π ∗ : Ma → Mg to the Drinfeld category of g. One can also pullback the monoidal structure along this functor. Ω := r + τ (r ) is g invariant, this is needed to pullback the monoidal structure.

Quantization of Color Lie Bialgebras

87

Using the pullback functor π ∗ and the Verma modules defined before one can define a functor F : Ma → A by F(V ) := HomM h (M+ ⊗ M− , π ∗ (V )).

(57)

F is again isomorphic to the forgetful functor, and so we have H := End(F) = U (a[[λ]]). In the same way as before one can define a tensor structure on F, and with this a deformed bialgebra on H . Note that if a was in fact triangular then, we have Ω = 0 and the Hopf algebra is also triangular. So essentially following the construction in the previous section, one gets the following Theorem 24 Any quasitriangular color Lie bialgebra admits a quasitriangular quantization Uh (g), and if g is triangular, the quantization is also triangular.

5 Second Quantization of Color Lie Bialgebras 5.1 Topological Spaces Let F be a space of functions into a topological space, then the weak topology is the initial topology with respect to the evaluation maps. Let V and W be topological vector spaces. We need a topology for Hom(V, W ). We use the weak one, for which a basis is given by { f ∈ Hom(V, W )| f (vi ) ∈ Ui , i = 1, . . . , n}U1 ,...,Un ,v1 ,...,vn ,

(58)

where Ui are open sets in W and vi are elements in V . Let K be a field of characteristic zero, with the discrete topology, and V a topological vector space over K. Then its topology is called linear if the open subspaces form a basis of neighborhoods of 0. Let V be a topological vector space with a linear  topology, then V is called separated if the map V → proj limU open subspace ( V U ) is injective, this is e.g. the case when V is discrete, i.e. 0 is an open set. All topological vector spaces we consider will be linear and separated, so we will just call them topological vector spaces. If V is finite dimensional than the weak topology on Hom(V, K) is the discrete topology. In general a neighborhood basis of zero is given by finite-codimensional subspaces. A topological vector space is called complete if the map V →

 proj lim ( V U ) U open subspace

is surjective.

88

B. Hurle and A. Makhlouf

Let V and W be topological vectors spaces, we define the topological tensor product by   ˆ = proj lim V V  ⊗ W W  , (59) V ⊗W where V  , W  run over open subspaces of V (resp. W ). With this we have: Proposition 25 Complete vector spaces with continuous linear maps form a symmetric monoidal category. Topology and Grading

 A topological color vector space, is a color vector space V = i∈Γ Vi , where each space Vi is a topological vector space. A linear map between topological color vector spaces is continuous if each homogeneous part is continuous. For a graded vector space we say that it is complete, separated or has a given property if every space Vi has this property. The tensor product can also be defined by replacing the usual tensor product over vector spaces by the completed one. We note that the tensor product involves the direct sum j∈Γ Vi ⊗ Wi− j , for which a priori it is not clear whether it is complete. But it turns out that here since the considered topologies are linear, this is the case. In fact using the construction, which defines graded vector spaces as functors, one just replaces the category of vector spaces by the category of complete vector spaces and gets a category of graded complete vector spaces, which is again monoidal due to Proposition 25.

5.2 Manin Triples Let a be a color Lie bialgebra with discrete topology, i.e. each ai is equipped with the discrete topology, and a∗ its dual, with the weak topology. Since a is discrete a∗ is the full graded dual. The cocommutator defines a continuous Lie bracket on a∗ . We have a natural topology on a ⊕ a∗ and the above defines a continuous Lie bracket with respect to this topology. Let g be a Lie algebra, with a nondegenerate inner product ·, · and g+ and g− be two isotropic Lie subalgebras, i.e. g+ , g+  = 0, such that g = g+ ⊕ g− . Then the inner product defines an embedding g− → g∗+ . To get a topology on g, we equip g+ with the discrete topology and g− with the weak topology. If the Lie bracket on g is continuous in this topology we call (g, g+ , g− ) a Manin triple. To every color Lie bialgebra one can associate a color Manin triple by (a ⊕ a∗ , a, a∗ ), and conversely every color Manin triple gives a Lie bialgebra on g+ .

Quantization of Color Lie Bialgebras

89

5.3 Equicontinuous g-Modules Let M be a topological vector space and {A x }x∈X be a family in End M. Then {A x }x∈X is equicontinuous if for all open neighborhoods U of 0 in M there exists an open neighborhood V such that A x V ⊂ U for all x ∈ X . Definition 26 Let M be a complete topological color vector space. Then we call M an equicontinuous g-module if there is a continuous color Lie algebra morphism π : g → End(M) such that {π(g)} is an equicontinuous family. ˆ N is again an For two equicontinuous g-modules M, N , we have that M ⊗ equicontinuous g-module. Further on (V ⊗ W ) ⊗ U can be identified with V ⊗ (W ⊗ U ) and V ⊗ W with W ⊗ V by the flip. So we can define the symmetric monoidal category M0e of equicontinuous g-modules. We define again the Verma modules M± by M± = Indgg± 1 = U (g) ⊗U (g± ) 1. Lemma 7 The module M− equipped with the discrete topology is an equicontinuous g-module. Proof This is true in the non-graded case, see e.g. [6], so it also holds in the color case since it can be checked in each degree. We want to define a topology on M+ , for this we first define a topology on U (g− ). We have Un (g− ) ∼ = ⊕k≤n S k g− . We equip S k g− with the weak topology com∗ ing from the embedding into (g⊗k + ) . This gives a topology on Un (g− ). Finally we put on U (g− ), and with this on M+ , the topology coming from the inductive limit lim Un (g) = U (g). Lemma 8 For all g ∈ g the map π M+ (g) : M+ → M+ is continuous.

 Next we need a topology on M+∗ . For this we note that Un (g− )∗ ∼ = k≤n S k g+ , so we equip Un (g− )∗ with the discrete topology. Since U (g− )∗ is the projective limit of Un (g− )∗ , it carries the corresponding topology.

Lemma 9 M+∗ is an equicontinuous g-module. However M+ is not equicontinuous in general. There is a Casimir element. It corresponds to the identity under the isomorphism ˆ a → End(a). of a∗ ⊗ Let Mgt be the category of equicontinuous g-modules and morphisms HomM gt (V, W ) = Homg (V, W )[[λ]],

(60)

where the Hom on the right denotes the continuous g-module morphisms. −1 ∈ Hom(V ⊗ W, W ⊗ V ) We define a natural transformation γ by γV,W = βW,V e for V, W ∈ M . Using the completed tensor product, we can define the structure of a braided monoidal category on Mgt using Φ and γ similarly to Sect. 3.

90

B. Hurle and A. Makhlouf

5.4 Tensor Functor F ˆ K[[λ]] is again a Let V be a complete color space over K. Then the space V ⊗ complete color space, and carries a natural structure of a topological color K[[λ]]module. A K[[λ]]-module is called complete if it is isomorphic to V [[λ]] for a complete color space V . Let A c be the category of complete color K[[λ]]-modules. It is equipped with a ˜ tensor product, which we denote by V ⊗W for V, W ∈ A c , see e.g. [6, Sect. 8.1]. With this we define  the tensor product of two complete color K[[λ]]-modules V, W ˜ )i = ˜ by (V ⊗W j∈Γ (V j ⊗Wi− j ). We define a functor F from the category Mgt of equicontinuous g-modules to the category of complete color vector spaces A c , by V → Hom(M− , M+∗ ⊗ V ) We can define a comultiplication on M+∗ by i +∗ : M+∗ ⊗ M+∗ → M+∗ : i +∗ ( f ⊗ g)(x) := ( f ⊗ g)(i + (x))

(61)

for f, g ∈ M+∗ . This is continuous. Proposition 27 There is an isomorphisms ΨV : F(V ) → V for all V ∈ Mgt natural in V , given by (62) f → (ev(1+ ) ⊗ id) f (1− ) Proof This follows from Frobenius reciprocity. This shows that F is natural isomorphic to the forgetful functor. We now want to define a tensor structure on the functor F. Similar to (41) we define, JV ⊗W (v ⊗ w) by i−

Φ

v⊗w

∗ ∗ ∗ ∗ M− − → M− ⊗ M− −−→ (M+ ⊗ V ) ⊗ (M+ ⊗ W) − → M+ ⊗ ((V ⊗ M+ ) ⊗ W) γ23

Φ

∗ ⊗id⊗id i+

∗ ∗ ∗ ∗ ∗ −→ M+ ⊗ ((M+ ⊗ V) ⊗ W) − → (M+ ⊗ M+ ) ⊗ (V ⊗ W ) −−−−−→ M+ ⊗ (V ⊗ W )

That is (without associators) JV W (v ⊗ w) = (i +∗ ⊗ id ⊗ id) ◦ (id ⊗ γ ⊗ id) ◦ (v ⊗ w) ◦ i − .

(63)

Again one can write down a version without using explicitly the maps v, w, which do not exist in the categorical sense for the graded case. Lemma 10 We have Φ ◦ (i − ⊗ id) ◦ i − = (id ⊗ i − ) ◦ i −

(64)

Quantization of Color Lie Bialgebras

and

Φ ◦ (i +∗ ⊗ id) ◦ i +∗ = (id ⊗ i +∗ ) ◦ i +∗

91

(65)

i.e. i − and i +∗ are coassociative in Hom(M− , (M− )⊗3 ) resp. Hom(M+∗ , (M+∗ )⊗3 ). Proposition 28 The maps JV W are isomorphisms and define a tensor structure on F. Proof They are isomorphisms because they are isomorphisms modulo λ.

5.5 Quantization of Color Lie Bialgebras Let H = End(F) be the algebra of endomorphisms of the fiber functor F, where End(F) is again to be understood in the enriched sense. Let H0 be the endomorphism algebra of the forgetful functor from M0e to the category of complete color vector spaces. The algebra H is naturally isomorphic to H0 [[λ]]. Let F 2 : M e × M e → A c be the bifunctor defined by F 2 (V, W ) = F(V ⊗ W ) and H 2 = End(F 2 ) then H ⊗ H ⊂ H 2 but not necessarily H 2 = H ⊗ H . H has a “comultiplication” Δ : H → H 2 , defined by Δ(a)V W (v ⊗ w) = JV−1W aV ⊗W JV W (v ⊗ w).

(66)

5.6 The Algebra Uh (g+ ) For x ∈ F(M− ) we define m + (x) ∈ End(F) by m + (x)(v) = ε(x, v)i +∗ ⊗ id ◦ Φ −1 ◦ (id ⊗ v) ◦ x

(67)

for v ∈ F(V ). We define Uh (g+ ) ⊂ H as the image of m + . We have m + is an embedding since it is so modulo λ. Proposition 29 Uh (g+ ) is a subalgebra of H . Proof Since i +∗ is coassociative we get ∗ ⊗ id)Φ −1 (id ⊗ i ∗ ⊗ id)Φ −1 (id ⊗ id ⊗ v)(id ⊗ y)x m + (x)m + (y)(v) = ε(x, yv)ε(y, v)(i + + 2,3,4 ∗ ⊗ id)(i ∗ ⊗ id ⊗ id)Φ −1 Φ −1 Φ −1 (id ⊗ id ⊗ v)(id ⊗ y)x = ε(x, y)ε(x, v)ε(y, v)(i + + 1,2,3 1,23,4 2,3,4 ∗ ⊗ id)(i ∗ ⊗ id ⊗ id)Φ −1 Φ −1 (id ⊗ id ⊗ v)(id ⊗ y)x = ε(x, y)ε(x y, v)(i + + 12,3,4 12,3,4 ∗ ⊗ id)Φ −1 (id ⊗ v)(⊗i ∗ ⊗ id)Φ −1 (id ⊗ y)x = ε(x, y)ε(x y, v)(i + + ∗ ⊗ id)Φ −1 (id ⊗ v)z = ε(x y, v)(i +

= i(z)v,

92

B. Hurle and A. Makhlouf

where z = (⊗i +∗ ⊗ id)Φ −1 (id ⊗ y)x. Proposition 30 The algebra Uh (g+ ) is closed under the coproduct. Proof The proof in [7, Sect. 21.2] is pictorial so it can also be used in the color case. The element Δ(m + (x)) is uniquely defined by the equation (i +∗ ⊗ id ⊗ id) ◦ (id ⊗ i +∗ ⊗ id ⊗ id) ◦ γ34 ◦ (id ⊗ v ⊗ w) ◦ (id ⊗ i − ) = (i +∗ ⊗ id ⊗ id) ◦ γ23 ◦ Δ(m + (x))(v ⊗ w) ◦ i −

(68)

for v ∈ F(V ), w ∈ F(W ). We want to get (i +∗ ⊗ id ⊗ id) ◦ γ23 ◦ (i +∗ ⊗ id ⊗ i +∗ ⊗ id) ◦ (id ⊗ v ⊗ id ⊗ w) = (i +∗ ⊗ id ⊗ id) ◦ (i +∗ ⊗ id ⊗ id ⊗ id) ◦ γ34 ◦ (id ⊗ v ⊗ w) ◦ (i +∗ ⊗ id ⊗ id) ◦ γ23 . Theorem 31 Uh (g+ ) is a quantization of g+ , so for every color Lie bialgebra there exists a quantized universal enveloping algebra.

6 Simple Color Lie Bialgebras of Cartan Type In the case of Lie superalgebras, there are the so called classical simple ones of type A-G. Let A = (Ai j )i, j∈I be a Cartan matrix, I = {1, . . . , s} and τ ⊂ I the set corresponding to odd roots. Let g be the Lie superalgebra generated by h i , ei and f i for i ∈ I . We can put a Zs grading on it as follows. We denote by z i the i-th generator. The elements h i are all of degree zero and deg( f i ) = − deg(ei ) = z i . So we consider g to be graded by the root system. The commutation factor is given by ε0 (z i , z j ) = −1 if either i ∈ τ or j ∈ τ and ε(z i , z j ) = 1 else. The generators satisfy the relations [8, 18] [h i , h j ] = 0, [h i , e j ] = Ai j e j , [h i , f j ] = −Ai j f j , [ei , e j ] = δi j h i

(69)

and the so called super classical Serre-type relations [ei , e j ] = [ f i , f j ] = 0 (ad ei )

1+|Ai j |

e j = (ad f i )1+|Ai j | f j = 0 if i = j, i ∈ /τ

[em , [em−1 , [em , em+1 ]]] = [ f m , [ f m−1 , [ f m , f m+1 ]]] = 0 for m − 1, m, m + 1 ∈ I, Amm = 0 [[[em−1 , em ], em ], em ] = [[[ f m−1 , f m ], f m ], f m ] = 0 if the Cartan matrix is of type B and τ = m, s = m.

The relations respect the Zs -grading so g can be considered as a Zr -graded algebra.

Quantization of Color Lie Bialgebras

93

For a set of constants εi j ∈ K× , i, j = 1, . . . , s. We define σ (a, b) = s ai b j for a, b ∈ Zr . Then σ : Zr × Zr → K× is a bicharacter and ε (a, b) = i, j=1 εi j 1 σ (a, b)σ (b, a) a commutation factor. We set ε = ε ε0 . For x, y ∈ g, we define [x, y] = σ (x, y)[x, y]. With this bracket g becomes a ε color Lie algebra. For more details on this construction see [9]. There are r matrices on these Lie algebras, see e.g. [10]. Not all ofthem given r there  respect the Z -grading,+ but the standard r-matrices given by r = h i ⊗ h i + α∈Δ+ eα ⊗ f α do. Here Δ denotes the set of positive roots. For these Lie superalgebras there is a well known quantization given by the so called Drinfeld-Jimbo type superalgebras. To define them we first need 

m+n n

= t

n−1

m+n−i

t

i=0

t i+1

− t −m−n+i . − t −i−1

(70)

Assume that the Cartan matrix A is symmetrizable that is there are non-zero rationals number d1 , . . . , ds such that di Ai j = d j A ji . Set q = eλ/2 and qi = edi . Let U (g) be the C[[λ]] superalgebra generated by h i , ei and f i , i = 1, . . . s and relations [h i , h j ] = 0, [h i , e j ] = Ai j e j , [h i , f j ] = −Ai j f j [ei , e j ] = δi j

q di h i − q −di h i qi − qi−1

ei2 = 0 for i ∈ I, Aii = 0 [ei , e j ] = 0, i, j ∈ I, i = j, Ai j = 0 1+|Ai j |

 k=0

(−1)k



1 + |Ai j | 1+|Ai j |−k ei e j eik = 0, for 1 ≤ i, j ≤ s, i = j, i ∈ /τ k t

em em−1 em em+1 + em em+1 em em−1 + em−1 em em+1 em + em+1 em em−1 em (q + q −1 )em em−1 em+1 em = 0, for m − 1, m, m + 1 ∈ I, Amm = 0 3 2 2 3 − (q + q −1 − 1)em em−1 em − (q + q −1 − 1)em em−1 em + em em−1 = 0 em−1 em−1

if the Cartan matrix is of type B and τ = m, s = m,

and the same relations where ei is replaced by f i . For more details on this super quantized universal enveloping algebra see e.g. [18]. Note that all relations are compatible with the Zr -grading. Again we set x y = σ (x, y)x y, and get so a ε-color algebra.

94

B. Hurle and A. Makhlouf

We define a comultiplication on U (g) by specifying it on generators as Δ(ei ) = ei ⊗ q di h i + 1 ⊗ ei , Δ( f i ) = f i ⊗ q −di h i + 1 ⊗ f i , Δ(h i ) = h i ⊗ 1 + 1 ⊗ h i , ε(h i ) = ε( f i ) = ε(ei ) = 0. Here the ε does not appear, but it does if one computes the comultiplication of other elements, since it appears in the definition of the multiplication on U (g) ⊗ U (g). Therefore we get a color Hopf algebra structure on U (g).

References 1. Andruskiewitsch, N.: Lie superbialgebras and Poisson-Lie supergroups. Abh. Math. Semin. Univ. Hamburg 63, 147–163 (1993) 2. Drinfeld, V.G.: Hamiltonian lie groups, lie bialgebras and the geometric meaning of the classical Yang-Baxter equation. Sov. Math. Dokl. 27(1), 68–71 (1983) 3. Drinfeld, V.G.: Quasi-Hopf algebras. Algebra i Analiz 1(6), 114–148 (1989) 4. Drinfel’d, V.G.: On quasitriangular quasi-Hopf algebras and on a group that is closely connected with Gal(Q/Q). Algebra i Analiz 2(4), 149–181 (1990) 5. Enriquez, B., Halbout, G.: Quantization of Γ -Lie bialgebras. J. Algebra 319(9), 3752–3769 (2008) 6. Etingof, P., Kazhdan, D.: Quantization of Lie bialgebras. I. Selecta Math. (N.S.) 2(1), 1–41 (1996) 7. Etingof, P., Schiffmann, O.: Lectures on Quantum Groups, 2nd edn. Lectures in Mathematical Physics, vol. 0. International Press, Somerville (2002) 8. Geer, N.: Etingof-Kazhdan quantization of Lie superbialgebras. Adv. Math. 207(1), 1–38 (2006) 9. Hurle, B., Makhlouf, A.: Color Lie bialgebras: Big bracket, cohomology and deformations. Geometric and Harmonic Analysis on Homogeneous Spaces and Applications, pp. 69–115. Springer International Publishing, Cham (2017) 10. Karaali, G.: Constructing r-Matrices on simple Lie superalgebras. J. Algebra 282, 83–102 (2004) 11. Kazhdan, D., Lusztig, G.: Tensor structures arising from affine Lie algebras, III. J. AMS 7, 335–381 (1994) 12. Kelly, G.M.: Basic Concepts of Enriched Category Theory. London Mathematical Society Lecture Note Series, vol. 64. Cambridge University Press, Cambridge (1982) 13. Lecomte, P.B.A., Roger, C.: Modules et cohomologie des bigèbres de Lie. C. R. Acad. Sci. Paris Sér. I Math. 310, 405–410 (1990) 14. Nijenhuis, A., Richardson, R.W.: Deformation of homomorphisms of Lie group and Lie algebras. Bull. Am. Math. Soc. 73, 175–179 (1967) 15. Scheunert, M.: Generalized Lie algebras. J. Math. Phys. 20(4), 712–720 (1979) 16. Scheunert, M., Zhang, R.B.: Cohomology of Lie superalgebras and their generalizations. J. Math. Phys. 39(9), 5024–5061 (1998) 17. Semenov-Tian-Shansky, M.A.: What is a classical r -matrix? Funct. Anal. Appl. 17(4), 259–272 (1983) 18. Yamane, H.: Quantized enveloping algebras associated with simple Lie superalgebras and their universal R-matrices. Publ. Res. Inst. Math. Sci. 30(1), 15–87 (1994)

Harmonic Analysis for 4-Dimensional Real Frobenius Lie Algebras Edi Kurniadi and Hideyuki Ishi

Abstract A real Frobenius Lie algebra is characterized as the Lie algebra of a real Lie group admitting open coadjoint orbits. In this paper, we study irreducible unitary representations corresponding to open coadjoint orbits for each of 4-dimensional Frobenius Lie algebras. We show that such unitary representations are squareintegrable, and their Duflo–Moore operators are closely related to the Pfaffian of the Frobenius Lie algebra. Keywords Frobenius Lie algebras · Square-integrable representations · Duflo-Moore operators MSC Classification 22E45 · 22D10 · 43A32

1 Introduction The theory of the orbit method claims that there exists a natural correspondence between the unitary dual of a Lie group and the space of coadjoint orbits. In view of its philosophy (cf. [8]), it seems reasonable to expect that open coadjoint orbits correspond to discrete summands of the left regular representation, that is, squareintegrable representations. Indeed, Lipsman [9] showed that, for an almost algebraic group G, there is a one-to-one correspondence between square-integrable representations of G and open orbits in a certain G-space. Lipsman’s results were established in a framework of a sophisticated version of the orbit method, and the “open orbits” E. Kurniadi · H. Ishi (B) Graduate School of Mathematics, Nagoya University,Furo-cho, Chikusa-ku, Nagoya 464-8602, Japan e-mail: [email protected] E. Kurniadi e-mail: [email protected] E. Kurniadi Department of Mathematics of FMIPA, Universitas Padjadjaran, Sumedang, Indonesia © Springer Nature Switzerland AG 2019 A. Baklouti and T. Nomura (eds.), Geometric and Harmonic Analysis on Homogeneous Spaces, Springer Proceedings in Mathematics & Statistics 290, https://doi.org/10.1007/978-3-030-26562-5_4

95

96

E. Kurniadi and H. Ishi

in [9] did not necessarily mean open coadjoint orbits. Notice that, if G is a compact Lie group, all the irreducible unitary representation is square-integrable, whereas G has no open coadjoint orbit. Going back to a rather primitive perspective in the orbit method, we shall study square-integrability of unitary representations corresponding to open coadjoint orbits. As is seen later, the Lie algebra of a real Lie group admitting open coadjoint orbits is nothing else but a real Frobenius Lie algebra. In general, the center of a real Frobenius Lie algebra g is trivial, so that the adjoint representation ad : g → End(g) is faithful. Let G be the subgroup of G L(g) corresponding to ad(g). We regard g as the Lie algebra of G. For f ∈ g∗ , we denote by  f the coadjoint orbit Ad∗ (G) f ⊂ g∗ through f . We pose the following conjectures: Conjecture 1 If  f is open in g∗ , there exists a polarization p ⊂ g at f such that G π f := Indexp p ν f is a square-integrable representation, where ν f is a one-dimensional representation of the group exp p ⊂ G defined by ν f (exp X ) := e2πiX, f  for X ∈ p. Since the Frobenius Lie algebra g is even dimensional, the Pfaffian Pf g := d Pf([X i , X j ])i,d j=1 ∈ S(g) of g is defined up to a constant multiple, where {X i }i=1 (d := dim g) is a basis of g. Let s : S(g) → U (g) be the symmetrization operator. For a unitary representation π of G, i d/2 dπ(s(Pf g )) is a symmetric operator. Conjecture 2 Let (π, Hπ ) be a square-integrable representation of G. Then i d/2 dπ(s(Pf g )) is essentially self-adjoint, and the Duflo–Moore operator Cπ of π equals a constant multiple of the operator |i d/2 dπ(s(Pf g ))|−1/2 on Hπ . Namely, there exists a positive constant cπ > 0 such that Cπ = cπ |i d/2 dπ(s(Pf g ))|−1/2 . For the case where G is exponential solvable, a statement similar to Conjecture 2 was claimed by Duflo and Raïs [4, Théorème 5.3.8], where Conjecture 1 was also proved at the same time. We also notice that, since a Frobenius Lie algebra is not necessarily almost algebraic, Lipsman’s work [9] does not imply our conjectures. Csikós and Verhóczki [2] classified all the 4-dimensional Frobenius Lie algebras over any field whose characteristic is not equal to 2. Based on this classification, we shall confirm Conjectures 1 and 2 for each of real 4-dimensional Frobenius Lie algebras in this paper. Moreover, we determine a constant cπ for all the representations corresponding to open coadjoint orbits. A main tool of our argument is a previous work by the second named author [6, 7] about harmonic analysis on a semi-direct product G = N  H of a unimodular group N and a closed subgroup H of Aut(N ). In that work, square-integrable subrepresentations of the quasi-regular representation of G on L 2 (N ) are studied by using the operator Fourier transform for N . In particular, the Duflo–Moore operators of the representations are described in a simple way. Applying the results to our cases, where N is the 3-dimensional Heisenberg group or 2-dimensional vector group, we can easily compare the Duflo–Moore operators and the differential representation of the element s(Pf g ) ∈ U (g).

Harmonic Analysis for 4-Dimensional Real Frobenius Lie Algebras

97

2 Preliminaries 2.1 Frobenius Lie Algebras d Let g be a d-dimensional Lie algebra over a field F, and {X i }i=1 a basis of g. We denote by Mg a d × d alternating matrix of g-entry defined by Mg := ([X i , X j ])i,d j=1 . Then det Mg ∈ S(g) does not depend on the choice of the basis {X i } up to a constant multiple. The Lie algebra g is said to be a Frobenius Lie algebra if det Mg is not identically zero [10]. Let us regard det Mg ∈ S(g) as a polynomial function on the space g∗ of Fvalued linear forms on g. Then we have (det Mg )( f ) = det ([X i , X j ], f )i,d j=1 ∈ F for f ∈ g∗ . In other words, (det Mg )( f ) equals the determinant of the alternating form B f : g × g  (X, Y ) → [X, Y ], f  ∈ F

with respect to the basis {X i }. In particular, (det Mg )( f ) = 0 if and only if B f is non-degenerate. Therefore, if g is a Frobenius Lie algebra, then d = dim g is even. In this case, the Pfaffian P f g = Pf Mg ∈ S(g) is not identically zero. Let ϕ : g → g be a Lie algebra automorphism, which is naturally extended to an algebra automorphism on S(g). Then we have (1) ϕ(Pf g ) = (detϕ)Pf g . Indeed, the left-hand side is the Pfaffian of the alternating matrix d  d  ϕ([X i , X j ]) i, j=1 = [ϕ(X i ), ϕ(X j )] i, j=1 , the matrix expreswhich is equal to t Aϕ Mg Aϕ , where Aϕ = (ai j ) ∈ Mat(d, F) is  sion of ϕ with respect to the basis {X i }, that is to say, ϕ(X j ) = nj=1 ai j X i . Since det Aϕ = det ϕ, we obtain (1). The 4-dimensional Frobenius Lie algebras over F are classified by Csikós and Verhóczki [2] when char F = 2. Theorem 3 ([2]) Any 4-dimensional Frobenius Lie algebra g over F with char F = 2 is isomorphic to one of the following: (1) g I : [X 1 , X 4 ] = [X 2 , X 3 ] = −X 1 , [X 2 , X 4 ] = −X 2 /2, [X 3 , X 4 ] = −X 3 /2, (2) g I I (τ ), τ ∈ F : [X 1 , X 4 ] = [X 2 , X 3 ] = −X 1 , [X 2 , X 4 ] = −X 3 , [X 3 , X 4 ] = −X 3 + τ X 2 , (3) g I I I (ε), ε ∈ F× : [X 1 , X 3 ] = [X 2 , X 4 ] = −X 1 , [X 1 , X 4 ] = εX 2 , [X 2 , X 3 ] = −X 2 . The Frobenius Lie algebras g I I I (ε) and g I I I (ε ) are isomorphic if and only if there exists a ∈ F for which ε = a 2 ε. In what follows, we consider only real Lie algebras, that is, the case F = R. Let G be a connected real Lie group, and g the Lie algebra of G. The coadjoint action

98

E. Kurniadi and H. Ishi

of g ∈ G on f ∈ g∗ is defined by X, Ad∗ (g) f  := Ad(g −1 )X, f  for X ∈ g. We write  f for the coadjoint orbit Ad∗ (G) f ⊂ g∗ through f . Then the openness of  f is equivalent to that B f is non-degenerate. Therefore G admits an open coadjoint orbit if and only if g is a real Frobenius Lie algebra, and in this case, we have 

f =



 f ∈ g∗ ; Pf g ( f ) = 0 .

 f :open

Moreover, substituting ϕ = Ad(a) for a ∈ G to (1), we obtain a covariance of Pf g under the adjoint action of G: Ad(a)(Pf g ) = (det Ad(a)) Pf g

(a ∈ G).

(2)

Let s : S(g) → U (g) be the symmetrization map. Namely, s is a linear map whose image of any monomial Y1 Y2 . . . Ym ∈ S(g) (Yk ∈ g, k = 1, . . . , m) is given by s(Y1 Y2 . . . Ym ) :=

1  Yσ(1) · Yσ(2) · · · Yσ(m) ∈ U (g). m! σ∈S m

It is known that s is Ad(G)-equivariant, so that (2) tells us that Ad(a)s(Pf g ) = (det Ad(a)) s(Pf g ) (a ∈ G). Therefore, for any representation π of G, we have π(a) ◦ dπ(s(Pf g )) ◦ π(a)−1 = (det Ad(a)) dπ(s(Pf g ))

(a ∈ G).

(3)

2.2 Orbit Method Let G be a connected Lie group, and g the Lie algebra of G. A subalgebra p of g is called a polarization at f ∈ g∗ if p is a Lagrangian subspace with respect to the alternating form B f . Assume that there exists a one-dimensional unitary representation ν f : exp p → U (1) such that ν f (exp X ) := e2πiX, f  for X ∈ p. The Pukanszky condition for a polarization p at f means that the coadjoint orbit  f contains the affine subspace f + p⊥ of g∗ . Bernat [1] showed that, if G is simply connected and exponential solvable, then for each f ∈ g∗ there exists a polarization p satisfying G the Pukanszky condition at f , and the unitary representation π f := Indexp p ν f of G is irreducible. Moreover, the equivalence class [π f ] does not depend on the choice of such polarization p, and the map f → [π f ] induces a one-to-one correspondence ˆ from the orbit space g∗ /Ad∗ (G) onto the unitary dual G.

Harmonic Analysis for 4-Dimensional Real Frobenius Lie Algebras

99

Although a Frobenius Lie algebra g is not necessarily exponential solvable, we G shall consider a unitary representation Indexp p ν f defined from a polarization p at ∗ f ∈ g satisfying the Pukanszky condition when  f is an open coadjoint orbit.

2.3 Square-Integrable Representation Let G be a locally compact group. An irreducible unitary representation (π, Hπ ) of G is said to be square-integrable if there exists a non-zero vector v0 ∈ Hπ for which |(v0 |π(g)v0 )Hπ |2 dg < ∞, G

where dg denotes the left Haar measure on G. Such vector v0 is called an admissible vector of the representation π. Duflo and Moore [3] showed that, if π is squareintegrable, then there exists a positive self-adjoint linear operator Cπ densely defined on Hπ such that (i) v0 ∈ Hπ is admissible if and only if v0 ∈ dom Cπ , and (ii) for any u 1 , u 2 ∈ Hπ and v1 , v2 ∈ dom Cπ , one has (u 1 |π(g)v1 )Hπ (π(g)v2 |u 2 )Hπ dg = (u 1 |u 2 )Hπ (Cπ v2 |Cπ v1 )Hπ .

(4)

G

Following [5], we call Cπ the Duflo–Moore operator of π, whereas, instead, the formal degree K π := Cπ−2 is considered in [3]. By changing variable g  = ga with a fixed a ∈ G at the integral in (4), we obtain (u 1 |u 2 )Hπ (Cπ v2 |Cπ v1 )Hπ = (u 1 |π(g  )π(a)v1 )Hπ (π(g  )π(a)v2 |u 2 )Hπ G (a) dg  G

= G (a)(u 1 |u 2 )Hπ (Cπ π(a)v2 |Cπ π(a)v1 )Hπ , where G denotes the modular function of G. Therefore, if v1 ∈ dom Cπ , then π(a)v1 ∈ dom Cπ , and we have Cπ2 v1 = G (a) π(a)−1 ◦ Cπ2 ◦ π(a)v1 thanks to the self-adjointness of Cπ . Moreover, since Cπ is positive, we obtain π(a) ◦ Cπ ◦ π(a)−1 = G (a)1/2 Cπ

(a ∈ G).

(5)

Note that, if G is a Lie group, then G (a) = det Ad(a)−1 . Comparing (3) and (5), we pose Conjecture 2 (see [3, Lemma 1]).

100

E. Kurniadi and H. Ishi

Taking an admissible vector v0 ∈ dom Cπ , we have an isometric embedding Wv0 : Hπ → L 2 (G) defined by Wv0 v(g) := (v|π(g)v0 )/Cπ v0  (v ∈ Hπ , g ∈ G). It is easy to check that the map Wv0 , called a continuous wavelet transform, is an intertwining operator from π into the left-regular representation. In this way, we see that a square-integrable representation is a subrepresentation of the left-regular representation, and vice-versa.

2.4 Analysis on a Semi-direct Product Group We shall review some results in [6] used in this paper, arranging a part of notation and presentation for simplicity. Let N be a separable locally compact unimodular (not necessarily commutative) group, H a closed subgroup of Aut(N ), and G the semidirect product group N  H . Let us denote the action of h ∈ H on n ∈ N by h · n. Let dn be a Haar measure on N , and δ(h) (h ∈ H ) the positive number for which d(h · n) = δ(h) dn (n ∈ N ). When a left Haar measure dh on H is given, a left Haar measure dg on G is defined by dg := δ(h)−1 dhdn (g = nh, n ∈ N , h ∈ H ). The quasi-regular representation L of G over the homogeneous space G/H  N is realized on L 2 (N ) by L(h)φ(n 0 ) := δ(h)−1/2 φ(h −1 · n 0 ), L(n)φ(n 0 ) := φ(n −1 n 0 ) (φ ∈ L 2 (N ), h ∈ H, n, n 0 ∈ N ). The Plancherel measure μ on the unitary dual Nˆ is determined by the following abstract Plancherel formula: |φ(n)|2 dn = πλ (φ)2HS dμ(λ) (φ ∈ L 2 (N ) ∩ L 1 (N )), N

Nˆ

where (πλ , Hλ ) is a realization of each λ ∈ Nˆ , and  · HS denotes the HilbertSchmidt norm of an operator. Let BHS (H λ ) be the space of Hilbert-Schmidt operators ⊕ on Hλ for λ ∈ Nˆ , and let F : L 2 (N ) → Nˆ BHS (Hλ ) dμ(λ) the Fourier transform for 2 1 N , which is defined as

⊕a unitary completion of the correspondence L (N ) ∩ L (N )  φ → (πλ (φ))λ∈ Nˆ ∈ Nˆ BHS (Hλ ) dμ(λ). We define an action of H on Nˆ by h · λ := [πλ ◦ h −1 ] (h ∈ H, λ ∈ Nˆ ). Let Oλ∗ denote the H -orbit H · λ ⊂ Nˆ through λ ∈ Nˆ . Assume that there exists λ0 ∈ Nˆ such that

Harmonic Analysis for 4-Dimensional Real Frobenius Lie Algebras

101

(i) μ(Oλ∗0 ) > 0, (ii) The stabilizer Hλ0 := { h ∈ H ; h · λ0 = λ0 } is compact, (iii) The map H/Hλ0  h Hλ0 → h · λ0 ∈ Oλ∗0 is a homeomorphism, where the topology of Oλ∗0 is induced from the Fell topology of Nˆ . By definition, πh·λ and πλ ◦ h −1 is equivalent, so that we have a unitary intertwining operator C(h, λ) : Hλ → Hh·λ such that C(h, λ) ◦ πλ (h −1 · n) = πh·λ (n) ◦ C(h, λ)

(n ∈ N ).

(6)

The operator C(h, λ) is determined by (6) uniquely up to a U (1)-multiple. Then we have F[L(n)φ](λ) = πλ (n) ◦ Fφ(λ) (n ∈ N ), F[L(h)φ](λ) = δ(h)1/2 C(h, h −1 · λ) ◦ Fφ(h −1 · λ) ◦ C(h, h −1 · λ)−1 (h ∈ H ) for φ ∈ L 2 (N ) and almost all λ ∈ Nˆ . For h, h  ∈ H and λ ∈ Nˆ , there exists ch h  λ ∈ U (1) such that C(hh  , λ) = ch h  λ C(h, h  · λ) ◦ C(h  , λ). In particular, we have a projective representation Hλ0  h → C(h, λ0 ) ∈ U (Hλ0 ) of the compact group Hλ0 . Let Vλ0 ⊂ Hλ0 be an irreducible subspace of the projective representation. For λ = h · λ0 ∈ Oλ∗0 (h ∈ H ), define Vλ := C(h, λ0 )Vλ0 ⊂ Hλ . Let L0 (N ) be a subspace of L 2 (N ) defined by  L0 (N ) :=

φ ∈ L (N ) ; 2

Image Fφ(λ) ⊂ Vλ (a.a. λ ∈ Oλ∗0 ) Fφ(λ) = 0 (otherwise)

.

Then L0 (N ) is an irreducible subspace of the quasi-regular representation (L , L 2 (N )). Moreover, the following result was obtained. Theorem 4 ([6]) The representation (L , L0 (N )) is square-integrable. The Duflo– Moore operator C0 of this representation is given by F[C0 φ](λ) = (dim Vλ0 )−1/2 D0 (λ)1/2 Fφ(λ)

(φ ∈ L0 (N ), λ ∈ Oλ∗0 ),

where D0 : Oλ∗0 → R>0 is a continuous function defined in such a way that

Oλ∗

ψ(λ)D0 (λ) dμ(λ) = H

0

ψ(h · λ0 ) H (h)−1 dh (ψ ∈ Cc (Oλ∗0 )).

Let us introduce a Hilbert space   BHS (Hλ0 , Vλ0 ) := T ∈ BHS (Hλ0 ) ; Image T ⊂ Vλ0 ,

(7)

102

E. Kurniadi and H. Ishi

on which we define a unitary representation 0 of a group G λ0 := N  Hλ0 by 0 (n)T := πλ0 (n) ◦ T, 0 (h)T := C(h, λ0 ) ◦ T ◦ C(h, λ0 )−1 (n ∈ N , h ∈ H, T ∈ BHS (Hλ0 , Vλ0 )).

(8)

Proposition 5 The representation (L , L0 (N )) is equivalent to the induced representation Ind G G λ 0 . 0

3 4-Dimensional Cases From Theorem 3, we get a list of 4-dimensional real Frobenius Lie algebras: (1) g I : [X 1 , X 4 ] = [X 2 , X 3 ] = −X 1 , [X 2 , X 4 ] = −X 2 /2, [X 3 , X 4 ] = −X 3 /2, (2) g I I (τ ), τ ∈ R : [X 1 , X 4 ] = [X 2 , X 3 ] = −X 1 , [X 2 , X 4 ] = −X 3 , [X 3 , X 4 ] = −X 3 + τ X 2 , (3) g I I I (ε), ε = ±1 : [X 1 , X 3 ] = [X 2 , X 4 ] = −X 1 , [X 1 , X 4 ] = εX 2 , [X 2 , X 3 ] = −X 2 . By definition, the Pfaffian of 4-dimensional Frobenius Lie algebra is given by Pf g := [X 1 , X 2 ][X 3 , X 4 ] − [X 1 , X 3 ][X 2 , X 4 ] + [X 1 , X 4 ][X 2 , X 3 ] ∈ S(g), so that we have Pf g I = X 12 , Pf g I I (τ ) = X 12 , Pf g I I I (ε) = −X 12 − εX 22 . In this section, we shall study open coadjoint orbits and corresponding unitary representations for each case, and confirm Conjectures 1 and 2. Eventually, the constant cπ equals 2π for all cases.

3.1 The First Case The Lie algebra g I has a 3-dimensional ideal n := X 1 , X 2 , X 3  which is isomorphic to the Heisenberg Lie algebra. Let G I be the connected Lie subgroup of G L(g I ) corresponding to ad(g I ). Then G I is the semi-direct product N  H , where N = exp n is a simply connected 3-dimensional Heisenberg group, and H = exp RX 4 is a one-dimensional subgroup of G I . We shall determine open coadjoint orbits in g∗I . First we note that if the coadjoint orbit  f = Ad∗ (G I ) f is open, then Ad∗ (N ) f is necessarily 3-dimensional. For X = a X 1 + bX 2 + cX 3 ∈ n (a, b, c ∈ R), the matrix expression of ad(X ) ∈ End(g I ) with respect to (X 1 , . . . , X 4 ) is

Harmonic Analysis for 4-Dimensional Real Frobenius Lie Algebras

103

⎛

⎞ 0 c −b −a ⎜ 0 0 −b/2⎟ ⎜ ⎟, ⎝ 0 −c/2 ⎠ 0 so that the matrix expression of Ad∗ (exp X ) ∈ G L(g∗I ) with respect to the dual basis (X 1∗ , . . . , X 4∗ ) of g∗I is ⎛ ⎞ 1 ⎜−c 1 ⎟ ⎜ ⎟ ⎝ b 0 1 ⎠. a b/2 c/2 1 Thus, for f = αX 1∗ + β X 2∗ + γ X 3∗ + δ X 4∗ ∈ g∗I with α, β, γ, δ ∈ R∗ , the N -orbit Ad∗ (N ) f is 3-dimensional if and only if α = 0, and in this case, Ad∗ (N ) f = αX 1∗ + X 2∗ , X 3∗ , X 4∗ . On the other hand, the matrix expression of Ad∗ (exp t X 4 ) ∈ G L(g∗I ) is ⎛ ⎜ ⎜ ⎝

e−t

⎞ e−t/2 e

⎟ ⎟, ⎠

−t/2

(9)

1 so that N -orbits Ad∗ (N ) f are mapped each other by the H -action in a way Ad∗ (exp t X 4 )(αX 1∗ + X 2∗ , X 3∗ , X 4∗ ) = e−t αX 1∗ + X 2∗ , X 3∗ , X 4∗ . Therefore, we see that there exist just two open coadjoint orbits ±X 1∗ = (±R>0 )X 1∗ + RX 2∗ + RX 3∗ + RX 4∗ in g∗I . The commutative subalgebra p := X 1 , X 2  of g I is a polarization satisfying Pukanszky condition at both X 1∗ and −X 1∗ . We denote by π± the induced repreGI ∗ sentation Indexp p ν±X 1 of G I . We shall realize π± as a subrepresentation of the quasi-regular representation (L , L 2 (N )) so as to apply Theorem 4 to our π± . For this purpose, we shall recall basic facts about harmonic analysis over the Heisenberg group N . The unitary dual Nˆ consist of two types of representations of N . The first type is one-dimensional representation νβ X 2∗ +γ X 3∗ parametrized by (β, γ) ∈ R2 given by νβ X 2∗ +γ X 3∗ (exp(a X 1 + bX 2 + cX 3 )) = e2πi(βb+γc) . The second type is an infinite dimensional unitary representation σα parametrized by α ∈ R \ {0} characterized by σα (exp a X 1 ) = e2πiαa Id for a ∈ R. In this way, Nˆ is identified with R2 ∪ (R \ {0}). Only the representation σα of second type involves with the Plancherel formula.

104

E. Kurniadi and H. Ishi

Indeed, we have for φ ∈ L 1 (N ) ∩ L 2 (N ), |φ(n)|2 dn = σα (φ)2H S dμ(α), N

R\{0}

where dn is a Haar measure on N given by dn = da db dc for n = exp(a X 1 + bX 2 + cX 3 ) ∈ N , and dμ is the Plancherel measure give by dμ(α) := |α| dα. Let us fix a realization of σα for each α ∈ R \ {0}. When α = ±1, we define σ±1 to be a standard Schrödinger representation on L 2 (R) given by σ±1 (exp a X 1 )ψ(x) = e±2πia ψ(x), σ±1 (exp bX 2 )ψ(x) = e±2πibx ψ(x), σ±1 (exp cX 3 )ψ(x) = ψ(x + c)

(ψ ∈ L 2 (R), x ∈ R).

N Note that σ±1 is equivalent to the induced representation IndexpX ν ∗ . The action 1 ,X 2  ±X 1 ˆ of H on R \ {0} ⊂ N is given by

h · α = e−t α (h = exp t X 4 , α ∈ R \ {0}). Keeping in mind this action, for a general α ∈ R \ {0}, define a representation σα of N on L 2 (R) by σα (n) := σε (h −1 · n) (n ∈ N ), where h ∈ H and ε = ±1 are unique elements for which α = h · ε. Thanks to these realizations, all the intertwining operators C(h, λ) in (6) are taken as the identity 2 The Fourier transform F for N is a unitary map from L 2 (N ) operator

on L (R). 2 onto R\{0} BHS (L (R)) dμ(α), and the quasi-regular representation (L , L 2 (N )) is described via F as F[L(n)φ](α) = σα (n) ◦ Fφ(α), F[L(exp t X 4 )φ](α) = et Fφ(et α)

(10) (11)

(φ ∈ L 2 (N ), n ∈ N , t ∈ R, α ∈ R \ {0}). The stabilizer subgroup H±1 of H at ±1 ∈ R \ {0} is trivial, so that any onedimensional subspace Rψ0 of L 2 (R) is an irreducible H±1 -subspace, where ψ0 ∈ L 2 (R) is a fixed non-zero vector. Define  (a.a. α ∈ ±R ) Image Fφ(α) ⊂ Rψ 0 >0 L± (N ) := φ ∈ L 2 (N ) ; . Fφ(α) = 0 (otherwise) Theorem 6 (i) The subrepresentation (L , L± (N )) of the quasi-regular representation (L , L 2 (N )) is irreducible and square-integrable. The Duflo–Moore operator C± of the representation is described as

Harmonic Analysis for 4-Dimensional Real Frobenius Lie Algebras

105

F[C± φ](α) := |α|−1 Fφ(α) (a.a. α ∈ ±R>0 ). GI ∗ (ii) The representation (L , L± (N )) of G I is equivalent to π± = Indexp p ν±X 1 corre∗ sponding to the open coadjoint orbit ±X 1∗ ⊂ g I .

Proof. We obtain the assertion (i) by just applying Theorem 4. In fact, the equation (7) becomes ±R>0

ψ(α)|α|−2 dμ(α) =

R

ψ(±e−t ) dt

because dμ(α) = |α|dα and the Haar measure dh on the commutative group H = exp RX 4 is given by dh = dt (h = exp t X 4 ∈ H, t ∈ R). For the assertion (ii), we see from Proposition 5 that the representation (L , L± (N )) is equivalent to Ind GN I σ±1 . Indeed, the space BHS (Hλ0 , Vλ0 ) in (8) is equal to L 2 (R) ⊗ Rψ0  L 2 (R), and the representation (0 , BHS (Hλ0 , Vλ0 )) is exactly the Schrödinger representation (σ± , L 2 (R)) of N  H± = N in this case. On the other hand, we N ∗ know that σ± is equivalent to Indexp p ν±X 1 . Therefore, the assertion follows from the theorem on induction by stages.  Theorem 6 tells us that Conjecture 1 is true. Now we examine Conjecture 2. By (10), we have F[d L(X 1 )φ](α) = 2πiαFφ(α) (a.a. α ∈ ±R>0 ). Thus, recalling Pf g I = X 12 , we have F[i 2 d L(s(Pf g I ))φ](α) = 4π 2 α2 Fφ(α). Therefore Conjecture 2 is true with cπ = 2π.

3.2 The Second Case Similarly to the first case, the Lie algebra g I I (τ ) is equal to the direct sum n ⊕ h of the Heisenberg Lie algebra n := X 1 , X 2 , X 3  and a one-dimensional subalgebra h = RX 4 . The Lie group G I I (τ ) ⊂ G L(g I I (τ )) corresponding to ad(g I I (τ )) is isomorphic to the semi-direct product N  H of the Heisenberg Lie group N := exp n and a one-dimensional group H := exp RX 4 . In this case, the action of H on N depends on the parameter τ . Despite this difference, we can apply exactly the same argument as the one for g I in Sect. 3.1 to g I I (τ ). In conclusion, we get Theorem 7 (i) There exists just two open orbits ±X 1∗ . G I I (τ ) (ii) The representation π± := IndexpX ν ∗ is realized as a subrepresentation of 1 ,X 2  ±X 1 2 the quasi-regular representation (L , L (N )). (iii) π± is irreducible and square-integrable. (iv) Conjecture 2 holds with cπ± = 2π.

106

E. Kurniadi and H. Ishi

3.3 The Third Case The Lie algebra g I I I (ε) is the direct sum n ⊕ h of a commutative ideal n := X 1 , X 2  and a commutative subalgebra h := X 3 , X 4 . The matrix expressions of ad(a X 1 + bX 2 ) and ad(cX 3 + t X 4 ) with respect to the basis (X 1 , . . . , X 4 ) are ⎛

⎞ −a −b ⎜ 0 −b εa ⎟ ⎜ ⎟ ⎝ ⎠ 0 0 0

⎛

⎞ c t ⎜−εt c ⎟ ⎜ ⎟, ⎝ 0 ⎠ 0

and

The Lie group G I I I (ε) ⊂ G L(g I I I (ε)) corresponding to ad(g I I I (ε)) is the semidirect product N  H with ⎧⎛ ⎫ ⎞ ⎪ ⎪ ⎪ 1 −a −b ⎪ ⎨ ⎬ ⎜ 1 −b εa ⎟ ⎟ ; a, b ∈ R , N= ⎜ ⎝ ⎠ 1 ⎪ ⎪ ⎪ ⎪ ⎩ ⎭ 1 and when ε = −1, ⎧⎛ c e cosh t ec sinh t ⎪ ⎪ ⎨⎜ c e sinh t ec cosh t H= ⎜ ⎝ 1 ⎪ ⎪ ⎩

⎞

⎫ ⎪ ⎪ ⎬

⎟ ⎟ ; c, t ∈ R , ⎠ ⎪ ⎪ ⎭ 1

and when ε = 1, ⎧⎛ c e cos t ec sin t ⎪ ⎪ ⎨⎜ c −e sin t ec cos t H= ⎜ ⎝ 1 ⎪ ⎪ ⎩

⎞

⎫ ⎪ ⎪ ⎬

⎟ ⎟ ; c, t ∈ R . ⎠ ⎪ ⎪ ⎭ 1

The matrix expression of Ad∗ (exp (a X 1 + bX 2 )) with respect to (X 1∗ , . . . , X 4∗ ) is ⎛

1

⎞

⎜ 1 ⎟ ⎜ ⎟ ⎝a b 1 ⎠ . b −εa 1

Harmonic Analysis for 4-Dimensional Real Frobenius Lie Algebras

107

For f = αX 1∗ + β X 2∗ + γ X 3∗ + δ X 4∗ ∈ g I I I (ε)∗ , the N-orbit Ad∗ (N ) f equals 

 αX 1∗ + β X 2∗ + (γ + αa + βb)X 3∗ + (δ − εβa + αb)X 4∗ ; a, b ∈ R ,

     a α β a which is 2-dimensional if and only if the linear map → is b −εβ α b   α β non-singular, that is, det = α2 + εβ 2 = 0. In this case, we have −εβ α Ad∗ (N ) f = αX 1∗ + β X 2∗ + X 3∗ , X 4∗ . And these orbits are mapped each other by the H -action. When ε = −1, the set  αX 1∗ + β X 2∗ ; α2 − β 2 = 0 is decomposed into four orbits   Ad∗ (H )(±X 1∗ ) = αX 1∗ + β X 2∗ ; α2 − β 2 > 0, ±(α + β) > 0 , and

  Ad∗ (H )(±X 2∗ ) = αX 1∗ + β X 2∗ ; α2 − β 2 < 0, ±(α − β) < 0 .

Therefore, there exist four open coadjoint orbits ±X i∗ (i = 1, 2) in g I I I (−1)∗ .   When ε = 1, the set αX 1∗ + β X 2∗ ; α2 + β 2 = 0 is an H -orbit Ad∗ (H )X 1∗ . Therefore, there exists just one open orbits  X 1∗ in g I I I (1)∗ . We shall study the unitary representations of G I I I (ε) corresponding to open coadjoint orbits. Let us consider the case ε = −1 first. Then p := n = X 1 , X 2  is a polarization satisfying the Pukanszky condition at all ±X i∗ (i = 1, 2). DeG I I I (−1) fine πi,± := Indexp ν±X i∗ (i = 1, 2). We realize πi,± as subrepresentations of the p quasi-regular representation (L , L 2 (N )) of G I I I (−1), and apply Theorem 4. Note that N is a 2-dimensional vector group, and Nˆ is identified with n∗ = X 1∗ , X 2∗  via the correspondence αX 1∗ + β X 2∗ → ναX 1∗ +β X 2∗ . Then the Fourier transform for N coincides with the ordinary Fourier transform: Fφ(αX 1∗

+

β X 2∗ )

:=

R2

e2πi(αa+βb) φ(exp(a X 1 + bX 2 )) dadb,

and the Plancherel measure dμ coincides with the Euclidean measure as dμ(αX 1∗ + β X 2∗ ) = dα dβ. The action of H on Nˆ ≡ n∗ is exactly the coadjoint action, which is already discussed. The quasi-regular representation (L , L 2 (N )) is described as F[L(n))φ](αX 1∗ + β X 2∗ ) = e2πi(αa+βb) Fφ(αX 1∗ + β X 2∗ ), F[L(h)φ](αX 1∗ 2

+

β X 2∗ )

∗

= e Fφ(Ad (h c

−1

)(αX 1∗

+

β X 2∗ ))

(φ ∈ L (N ), n = exp(a X 1 + bX 2 ) ∈ N , h = exp(cX 3 + t X 4 ) ∈ H ).

(12) (13)

108

E. Kurniadi and H. Ishi

Define   / Ad∗ (H )(±X i∗ )) . Li,± (N ) := φ ∈ L 2 (N ) ; Fφ(ξ) = 0 (a.a. ξ ∈ Applying Theorem 4, we obtain the following result. Theorem 8 The subrepresentation (L , Li,± (N )) (i = 1, 2) of the quasi-regular representation (L , L 2 (N )) of G I I I (−1) is irreducible and square-integrable. Its Duflo–Moore operator Ci,± is given by F[Ci,± φ](αX 1∗ + β X 2∗ ) = |α2 − β 2 |−1/2 Fφ(αX 1∗ + β X 2∗ ).

(14)

If αX 1∗ + β X 2∗ = Ad∗ (exp(cX 3 + t X 4 ))(±X i∗ ), then dcdt = |αdαdβ 2 −β 2 | . Thus, the formula (14) follows from (7). On the other hand, we have by (12) F[i 2 d L(s(Pf g I I I (ε) ))φ](αX 1∗ + β X 2∗ ) = F[i 2 d L(X 12 − X 22 )φ](αX 1∗ + β X 2∗ ) = 4π 2 (α2 − β 2 )Fφ(αX 1∗ + β X 2∗ ). Therefore, Conjecture 2 holds with cπi,± = 2π. Next we consider the case ε = 1. In this case, the H -orbit Ad∗ (H )X 1∗ is open dense in Nˆ ≡ n∗ . Therefore the quasi-regular representation (L , L 2 (N )) itself is irreducible and square-integrable by Theorem 4. Moreover, this representation is G I I I (1) equivalent to π X 1∗ := Indexp p ν X 1∗ , where p := n = X 1 , X 2 , so that we confirm Conjecture 1. Conjecture 2 is verified by exactly the same argument as in the case ε = −1 above. In fact, the constant cπ X ∗ equals 2π for this case, too. 1

Acknowledgements The present authors are very grateful to Professor Michel Duflo for suggesting that we study Frobenius Lie algebras. They thank Professor Hidenori Fujiwara for his interest to this work, and thank the referee for helpful comments. And they express their gratitude to Professors Ali Baklouti and Takaaki Nomura for invitation to the fifth Tunisian-Japanese conference and its wonderful hospitality. The first author would like to thank The Research, Technology Directorate General of Higher Education (RISTEK-DIKTI), Ministry of Research, Technology and Higher Education of Indonesia. This research is partially supported by KAKENHI 16K05174.

References 1. Bernat, P., et al.: Représentations des groupes de Lie résolubles. Dunod, Paris (1972) 2. Csikós, B., Verhóczki, L.: Classification of Frobenius Lie algebras of dimension ≤ 6. Publ. Math. Debrecen. 70, 427–451 (2007) 3. Duflo, M., Moore, C.C.: On the regular representation of nonunimodular locally compact group. J. Funct. Anal. 21, 209–243 (1976) 4. Duflo, M., Raïs, M.: Sur l’analyse harmonique sur les groupes de Lie résolubles. Ann. Sci. École Norm. Sup. 9, 107–144 (1976) 5. Grosmann, A., Morlet, J., and Paul, T.: Transforms associated to square integrable group representations: I. General results. J. Math. Phys. 26, 2473–2479 (1985)

Harmonic Analysis for 4-Dimensional Real Frobenius Lie Algebras

109

6. Ishi, H.: Wavelet transform for semi-direct product groups with not necessarily commutative normal subgroups. J. Fourier. Anal. Appl. 12, 37–52 (2006) 7. Ishi, H.: Continuous wavelet transforms and non-commutative Fourier analysis. RIMS Kôkyûroku Bessatsu B 20, 173–185 (2010) 8. Kirillov, A.A.: Lectures on the orbit method (Graduate Studies in Mathematics, vol. 64). American Mathematical Society, Providence (2004) xx+408 9. Lipsman, R.L.: An orbital perspective on square-integrable representations. Indiana Univ. Math. J. 34, 393–403 (1985) 10. Ooms, A.I.: On Frobenius Lie algebras. Comm. Algebra 8, 13–52 (1980)

An Example of Holomorphically Induced Representations of Exponential Solvable Lie Groups Junko Inoue

Abstract We discuss a holomorphically induced representation ρ = ρ( f, h) of Boidol’s group (split oscillator group) G from a real linear form f of the Lie algebra g of G and a one-dimensional complex subalgebra h of gC given by (2.2) in Sect. 2.ρ is a subrepresentation of the regular representation of G with the Plancherel measure ν. For ν-almost all irreducible representations π of G, the spaces of generalized vectors satisfying the semi-invariance associated with f and h are one-dimensional subspaces. On the other hand, according to the choice of f , there are two cases that (1) ρ vanishes, and (2) ρ is non-zero. Keywords Holomorphically induced representation, Solvable Lie group, Plancherel formula

1 Introduction In this article we study holomorphically induced representations of exponential solvable Lie groups focusing on Frobenius reciprocity in distribution sense associated with Penney’s abstract Plancherel theorem. Let G = exp g be an exponential Lie group with Lie algebra g and f ∈ g∗ be a real linear form, which is extended to gC by complex linearity. We denote by B f the skew symmetric bilinear form on gC defined by B f (X, Y ) := f ([X, Y ]), X, Y ∈ gC . Let h ⊂ gC be an isotropic complex subalgebra for B f , that is, h satisfies f ([h, h]) = {0}.

This work was supported by JSPS KAKENHI Grant Number JP17K05280. J. Inoue (B) Education Center, Organization for Educational Support and International Affairs, Tottori University, Tottori 680-8550, Japan e-mail: [email protected] © Springer Nature Switzerland AG 2019 A. Baklouti and T. Nomura (eds.), Geometric and Harmonic Analysis on Homogeneous Spaces, Springer Proceedings in Mathematics & Statistics 290, https://doi.org/10.1007/978-3-030-26562-5_5

111

112

J. Inoue

Let d := g ∩ h, D = exp d be the corresponding subgroup, χ f be the unitary character of D defined by χ f (exp X ) = ei f (X ) , X ∈ d. We denote E := (h + h) ∩ g noting that E is not necessarily a subalgebra. Take δ ∈ E ∗ such that δ|d =

1 tr ad g/d and δ|E∩n = 0, 2

(1.1)

where n is the nilradical of g, and extend it to EC by complex linearity. From a triple (h, f, δ), we define a holomorphically induced representation ρ = ρ(h, f, δ), which is a subrepresentation of ind GD χ f , by the following Definition 1.1. Let us recall that on the space of continuous functions ψ on G with compact support modulo D satisfying ψ(x y) =

 D (y) ψ(x) ∀y ∈ D, ∀x ∈ G, G (y)

where G and  D are the modular functions of G and D, respectively, there exists a positive left invariant linear functional  ψ(g) dμG,H

ψ → G/H

uniquely up to a positive constant factor. (See [2, Chap. V], [5, Chap. 3].) Definition 1.1 Let C ∞ (h, f, δ) be the space of C ∞ functions φ on G such that 1. φ(g y) = χ f (y)−1



 D (y) G (y)

1/2 φ(g), ∀g ∈ G, ∀y ∈ D,

(1.2)



2.

φ 2 :=

|φ(g)|2 dμG,D (g) < ∞,

(1.3)

G/D

3. R(X )φ = (−i f (X ) + δ(X ))φ, ∀X ∈ h,

(1.4)

where R denotes the action as the left invariant vector field: R(X )φ(g) :=

 d φ(g exp(t X ))t=0 , dt

X ∈ g,

which is extended to gC by complex linearity. Let H(h, f, δ) be the completion of C ∞ (h, f, δ) and define a representation ρ of G on H(h, f, δ) by the left translation

An Example of Holomorphically Induced Representations …

113

ρ(g)φ(x) := φ(g −1 x), g, x ∈ G. for φ ∈ H(h, f, δ). Let us remark that the property (1.1) of the linear form δ ∈ E ∗ makes (1.2) compatible with (1.4). Suppose H(h, f, δ) = {0}, and denote the space of C ∞ vectors by H(h, f, δ)∞ and its anti-dual space by H(h, f, δ)−∞ . Elements of the anti-dual space H(h, f, δ)−∞ are called generalized vectors. Then the anti-linear form aρ ∈ H(h, f, δ)−∞ defined by H(h, f, δ)∞ φ → aρ , φ := φ(e), where e is the unit element of G, satisfies the semi-invariance ρ(X )aρ = (i f (X ) + δ(X ))aρ ,

X ∈ h.

⊕ Suppose ρ is decomposed into a direct integral G m(π)π dμ(π) of irreducible rep on Hilbert spaces Hπ , where dμ is a Borel measure on G  and resentations π ∈ G m(π) is the multiplicity of π. Then by Penney’s abstract Plancherel theorem [12], aρ is decomposed into a direct integral of generalized vectors aπ of π, and each aπ satisfies the semi-invariance π(X )aπ = (i f (X ) + δ(X ))aπ ,

X ∈ h.

(1.5)

Let (Hπ−∞ )h, f,δ := {a ∈ Hπ−∞ ; π(X )a = (i f (X ) + δ(X ))a, Then we have

X ∈ h}.

dim(Hπ−∞ )h, f,δ ≥ m(π)

(1.6)

(1.7)

for μ-almost all π [12]. Our problem of ‘reciprocity’ is as follows: Problem 1.2 (See [5, Chap. 10]) Does the equality dim(Hπ−∞ )h, f,δ = m(π)

(1.8)

hold for μ-almost all π? In particular, suppose h is real, that is, h = h. Then, we have that ρ = ind GD χ f is a monomial representation. The space of semi-invariant vectors is described by  1/2

(Hπ−∞ ) D,χ f  D,G

:= a ∈

Hπ−∞ ;



 D (h) π(h)a = χ f (h) G (h)

1/2

a, ∀h ∈ D . (1.9)

114

J. Inoue

(See [5, Chap. 10].) When G is a connected and simply connected nilpotent Lie group, it is proved by Fujiwara [4] that the reciprocity holds for all monomial representations ind GD χ f . Concerning general exponential groups, it is still an open problem; so far, we can find some affirmative results and examples, e.g. [1, 3, 5–7]. Let us consider general cases where h = h. In [13], Penney gave a modified definition of holomorphically induced representation ρ, which can be described by ρ = ρ(h, f, 21 tr ad g/d ) in terms of our definition, for constructing non-zero irreducible representations. Let us briefly recall his results: He obtained that under the assumption that h is a positive polarization at f satisfying the Pukanszky condition, the representation ρ is equivalent to the irreducible representation π f corresponding to the coadjoint orbit Ad∗ (G) f by the Kirillov–Bernat mapping [13, Theorem 1]. The proof reduces to that for the case that h is totally complex. Suppose furthermore, h is totally complex. Then there exists a non-zero C ∞ -vector v such that π f (X )v = i f (X )v for all X ∈ h. The vector v is called a Frobenius vector, and the space of Frobenius vectors is one-dimensional [13]. In the proof, the matrix element associated with the Frobenius vector v gives an intertwining operator between π f and ρ. Concerning general isotropic subalgebras, Magneron’s work [10, 11] on holomorphic induction associated with complex involutions is the first result treating those from a complex subalgebra which is not a polarization. Let G = exp g be a connected and simply connected nilpotent Lie group. Suppose the complex subalgebra h is a fixed point set of an involution σ of gC . Let ρ = ρ(h, 0, 0). Then Magneron obtained a necessary and sufficient condition for ρ = 0 in terms of coadjoint orbits, and that  and that ρ is decomposed into a multiplicity-free dim(Hπ−∞ )h ≤ 1 for all π ∈ G, direct integral of those irreducible representations π that dim(Hπ−∞ )h = 1. We also have some affirmative examples for the Problem 1.2 concerning holomorphic induction associated with (h, f, δ) where h is not necessarily a polarization in [8, 9]. Let ν be the Plancherel measure for the monomial representation ind GD χ f . If the  then ρ vanishes because of the inspace (Hπ−∞ )h, f,δ = {0} for ν-almost all π ∈ G, equality (1.7). In Sect. 2, we show a phenomenon that the converse does not hold. We also give an example of a non-zero holomorphic induction ρ and its decomposition into a direct integral of irreducible representations. We apply the idea of using matrix elements associated with semi-invariant vectors developed by Penney [13] and Magneron [10, 11].

2 A Holomorphically Induced Representation of Boidol’s Group Let G be the connected and simply connected Lie group with Lie algebra g = R-Span{T, X, Y, Z } defined by [T, X ] = X, [T, Y ] = −Y, [X, Y ] = Z .

(2.1)

An Example of Holomorphically Induced Representations …

Let

f = γ X ∗ , γ ∈ R, δ = 0,

V = X + i T ∈ gC , h = CV,

115

(2.2)

where {T ∗ , X ∗ , Y ∗ , Z ∗ } is the dual basis in g∗ . We study the holomorphic induction ρ := ρ(h, f, 0). Since h ∩ h = {0}, the representation ρ is a subrepresentation of the regular representation of G. The subset  := {l ∈ g∗ ; l(Z ) = 0} of g∗ is open dense and Ad∗ (G)-invariant, and the space / Ad∗ (G) consists of 2-dimensional coadjoint orbits parametrized by  := {l = t ∗ T ∗ + z ∗ Z ∗ ; t ∗ , z ∗ ∈ R, z ∗ = 0}.

(2.3)

By the orbit method, we have that the regular representation is supported on the  corresponding to  by the Kirillov–Bernat mapping. generic subset of G We shall show that there are two cases of H(h, f, 0) = {0} and H(h, f, 0) = {0} according to f , and that in both cases, the spaces of semi-invariant generalized vectors are one-dimensional for all representations πl corresponding to l ∈ : Theorem 2.1 Let G = exp g be the connected and simply connected Lie group with Lie algebra g = R-Span{T, X, Y, Z } defined by (2.1), and let V = X + i T , h = CV , f = γ X ∗ ,  be as (2.2) and (2.3). Then we have the following. 1. If γ ≥ 0, then the space H(h, f, 0) = {0}. For all l ∈ , we have h, f,0

dim Hπ−∞ = 1. l On the other hand, we have that nonzero semi-invariant generalized vectors al are not elements of Hπl for all l ∈ . 2. If γ < 0, then the space H(h, f, 0) = {0}. For all l ∈ , we have h, f,0

= 1. dim Hπ−∞ l Furthermore, we have that semi-invariant generalized vectors al are elements of h, f,0

Hπl , that is, Hπ−∞ ⊂ Hπl for all l ∈ . l The representation ρ = ρ(h, f, 0) decomposes into a multiplicity-free direct integral over : ρ=

⊕ 

πl dl,

(2.4)

where dl is a Lebesgue measure on the open dense subset  of RT ∗ + RZ ∗ . Proof Let l = t ∗ T ∗ + z ∗ Z ∗ ∈ . Then we have that the Lie algebra of the stabilizer of coadjoint action at l is g(l) = R-Span{T, Z }, and taking a Pukanszky polarization b = R-Span{T, Y, Z } at l, we realize the irreducible representation πl corresponding to the orbit G · l by πl = ind GB χl , where χl is the unitary character associated with l. By the diffeomorphism

116

J. Inoue

R4 (t, x, y, z) → g(t, x, y, z) := exp(t T ) exp(x X ) exp(yY ) exp(z Z ) ∈ G, we identify G with R4 and transfer a Lebesgue measure dtd xdydz on g = R4 to a left invariant measure dg on G. Identifying G/B with R by the mapping R s → exp(s X )B ∈ G/B, we describe the representation space of πl by Hπ := L 2 (R) and the action of G  B (exp(t T )) = exp( tr ad g/b (t T )) = et , we have for and g as follows: Noting that  G (exp(t T )) g(t, x, y, z) = exp(t T ) exp(x X ) exp(yY ) exp(z Z ) ∈ G and φ = φ(s) ∈ Cc∞ (R), ∗

(y(x−se−t )+z) it ∗ t − 2t

e e φ(−x + se−t ),   dφ 1 + it ∗ − φ(s), πl (T )φ(s) = −s ds 2 dφ πl (X )φ(s) = − , ds πl (Y )φ(s) = −i z ∗ sφ(s), πl (Z )φ(s) = i z ∗ φ(s).

πl (g(t, x, y, z))φ(s) = ei z

(2.5) (2.6) (2.7) (2.8) (2.9)

Thus the space of C ∞ -vectors Hπ∞l is the space of Schwartz functions S (R). Suppose al = al (s) is a distribution of Cc∞ (R) satisfying πl (V )al = (i f (V ) + δ(V ))al . Then for f = γ X ∗ and δ = 0 we have −

    dal dal 1 − i −s + it ∗ − al (s) = iγal (s), ds ds 2

   dal 1 ∗ = (1 + is) t + i −γ + al (s). (1 + s ) ds 2 2

Writing β = −γ + 21 , we have al (s) = e(t

∗

+iβ)arctan (s)

it ∗

β

(1 + s 2 ) 2 (1 + s 2 )− 2

(2.10)

up to a constant factor. Since Hπ∞l = S (R), the anti-linear form al : φ →

R

al (s)φ(s) ds, φ ∈ Cc∞ (R)

extends to a continuous anti-linear form on Hπ∞l . Thus we have

−∞ h, f,0 = Cal Hπl

(2.11)

An Example of Holomorphically Induced Representations …

117

for all l ∈ . Furthermore, we have that al ∈ L 2 (R) if and only if β > 21 .

2.1 Case of H(h, F, 0) = {0} Suppose f (X ) = γ ≥ 0, that is, β ≤ 21 . We shall show that the space H(h, f, 0) = {0}. Writing e := RX + RT and remarking that e is a subalgebra of g, let E = exp e be the corresponding subgroup. Suppose φ ∈ H(h, f, 0), and let 1/2

G (h) φ(gh) for g ∈ G and h ∈ E, where  E is the modular function φg (h) :=   E (h) of E. Then taking a left Haar measure dμ E on E, we have 





|φ|2 dg = G

dμG,E (g) 

G/E



E

dμG,E (g)

= G/E

G (h) |φ(gh)|2 dμ E (h)  E (h) |φg (h)|2 dμ E (h),

E

G (exp(t T ) exp(x X )) = det Adg/e (exp(t T ))−1 = et . For almost all g ∈ G, we where   E (exp(t T ) exp(x X )) have that the function φg (h) on E is square integrable and

 R(V )φg (h) = −i

 1 φg (h). f (V ) − 2

Let f  = f − 21 X ∗ . By Rossi–Vergne [14, Sect. 4B], we have that the holomorphically induced representation of the ax + b group E from (h, f  , 0) is non-zero if and only if f  (X ) < − 21 . But we have f  (X ) = γ − 21 ≥ − 21 , thus φg = 0. Therefore, we have that H(h, f, 0) = {0}.

2.2 Case of H(h, F, 0)  = {0} Suppose f (X ) = γ < 0, that is, β > 21 . Then the semi-invariant vector (2.10) is a square integrable function on R. Let ∗

it ∗

β

a 0 (t ∗ ; s) : = e(t +iβ)arctan (s) (s 2 + 1) 2 (1 + s 2 )− 2

a 0 (t ∗ ) 2 : = |a 0 (t ∗ ; s)|2 ds R al (s) := a(t ∗ , z ∗ ; s) : = |z ∗ |a 0 (t ∗ ; s)/ a 0 (t ∗ ) .

(2.12) (2.13) (2.14)

118

J. Inoue

∗ ∗ Letting dl be a Lebesgue measure on  naturally  ⊕ obtained from RT + RZ , we consider the multiplicity-free direct integral  πl dl of irreducible representah, f,0

tions πl over . Since dim Hπ−∞ = 1 for all l ∈ , the representation ρ is a l ⊕ subrepresentaion of  πl dl. We shall show that ρ is actually described by the di⊕ rect integral  πl dl. We apply the idea of using matrix elements associated with semi-invariant vectors, which is developed by Penney [13] and Magneron [10, 11]. Let φ = φ(l, s) = φ(t ∗ , z ∗ ; s), where l = t ∗ T ∗ + z ∗ Z ∗ ∈ , s ∈ R, be a compactly supported smooth function on  × R. We define a function σφ = σφ (g) on G by

σφ (g) =



φ(l, s), πl (g)al (s)dl, g ∈ G.

(2.15)

Then it formally satisfies R(V )σφ = −i f (V )σφ . We shall prove that σφ is square integrable on G and that φ → σφ extends to an ⊕ intertwining operator between  πl dl on L 2 (, L 2 (R))  L 2 ( × R) and ρ on H(h, f, 0). Let F = F(g) = F(t, x, y, z), where g = g(t, x, y, z) ∈ G, be a compactly supported smooth function on G. We shall calculate σφ (g)F(g) dg.

I := G

Recalling that πl (g(t, x, y, z))−1 φ(l, s) = e−i z

∗

(−sy+z) −it ∗ t

e

t

e 2 φ(l, et (s + x))

and letting FˆY Z (t, x, η, ζ) :=

R2

ei(η y+ζz) F(t, x, y, z) dydz, t, x, η, ζ ∈ R,

we have I =

πl (g)−1 φ(l, s), al (s) dl F(g) dg G 

πt ∗ T ∗ +z ∗ Z ∗ (g(t, x, y, z))−1 φ(t ∗ , z ∗ ; s), a(t ∗ , z ∗ ; s) dt ∗ dz ∗ F(t, x, y, z) dtd xdydz = R4  t ∗ ∗ e−i z (−sy+z) e−it t e 2 φ(t ∗ , z ∗ ; et (s + x))a(t ∗ , z ∗ ; s) dsdt ∗ dz ∗ F(t, x, y, z) dtd xdydz = R4  R t ∗ e−it t e 2 φ(t ∗ , z ∗ ; et (s + x))a(t ∗ , z ∗ ; s) FˆY Z (t, x, sz ∗ , −z ∗ ) dsdt ∗ dz ∗ dtd x = R2

=

R2

 R



 R

∗

t

e−it t e− 2 φ(t ∗ , z ∗ ; x)

√ a 0 (t ∗ ; s) |z ∗ | ˆ FY Z (t, e−t x − s, sz ∗ , −z ∗ ) dsdt ∗ dz ∗ dtd x.

a 0 (t ∗ )

An Example of Holomorphically Induced Representations …

119

For t, x, s, t ∗ ∈ R and z ∗ ∈ R \ {0}, let  t |z ∗ |e− 2 FˆY Z (t, e−t x − s, sz ∗ , −z ∗ ), ∗ ˆ T (t ∗ , x, s, z ∗ ) := eit t (t, x, s, z ∗ ) dt.  (t, x, s, z ∗ ) :=

R

Then we have

I =

R2

=

R





∗

R

e−it t (t, x, s, z ∗ )φ(t ∗ , z ∗ ; x)

φ(t ∗ , z ∗ ; x)

R

ˆ T (−t ∗ , x, s, z ∗ ) 

a 0 (t ∗ ; s) ds dt ∗ dz ∗ dtd x

a 0 (t ∗ )

a 0 (t ∗ ; s) ds dt ∗ dz ∗ d x.

a 0 (t ∗ )

For z ∗ = 0, we have

∗

R3



|(t, x, s, z )| dtd xds = 2

R3

| FˆY Z (t, x, s, −z ∗ )|2 dtd xds.

Thus we have the inequality 2   a 0 (t ∗ ; s)   ∗ ∗ ˆ T (−t , x, s, z ) ds  dt ∗ dz ∗ d x   0 (t ∗ )

 

a ×R R     |a 0 (t ∗ ; s)|2 ˆ T (−t ∗ , x, s, z ∗ )|2 ds | ds dt ∗ dz ∗ d x ≤ 0 (t ∗ ) 2

a R ×R R ∗ ∗ 2 ∗ ∗ ˆ T (−t , x, s, z )| ds dt dz d x = | ×R R = 2π |(t, x, s, z ∗ )|2 dsdtd xdz ∗ R3 ×(R\{0}) = 2π | FˆY Z (t, x, s, −z ∗ )|2 dtd xdsdz ∗



=

R3 ×(R\{0}) (2π)3 F 2L 2 (G) .

and we have 3

|I | ≤ (2π) 2 φ L 2 (×R) F L 2 (G) . Hence, we have that σφ ∈ L 2 (G), and the mapping φ → σφ extends to a bounded  intertwining operator on L 2 ( × R) to H( f, h, 0). This implies (2.4).

120

J. Inoue

References 1. Benoist, Y.: Multiplicité un pour les espaces symétriques exponentiels. Mém. Soc. Math. France (N.S.) (15), 1–37 (1984) 2. Bernat, P., Conze, N., Duflo, M., Lévy-Nahas, M., Rais, M., Renouard, P., Vergne, M.: Représentations des groupes de Lie résolubles. Monographies de la Société Mathématique de France, no. 4. Dunod, Paris (1972) 3. Fujiwara, H.: Représentations monomiales des groupes de Lie résolubles exponentiels. The Orbit Method in Representation Theory, pp. 61–84. Birkhäuser, Basel (1990) 4. Fujiwara, H.: Une réciprocité de Frobenius. Infinite Dimensional Harmonic Analysis III, pp. 17–35. World Scientific Publishing, Hackensack, NJ (2005) 5. Fujiwara, H., Ludwig, J.: Harmonic Analysis on Exponential Solvable Lie Groups. Monographs in Mathematics. Springer, Tokyo (2015) 6. Fujiwara, H., Yamagami, S.: Certaines représentations monomiales d’un groupe de Lie résoluble exponentiel. Representations of Lie groups, Kyoto, Hiroshima, 1986. Advanced Studies in Pure Mathematics, vol. 14, pp. 153–190. Academic Press, Boston, MA (2014) 7. Inoue, J.: Semi-invariant vectors associated to decompositions of monomial representations of exponential Lie groups. J. Math. Soc. Jpn. 49(4), 647–661 (1997) 8. Inoue, J.: Holomorphically induced representations of some solvable Lie groups. J. Funct. Anal. 186(2), 269–328 (2001) 9. Inoue, J.: Holomorphically induced representations of exponential solvable semi-direct product groups R  Rn . Adv. Pure Appl. Math. 6(2), 113–123 (2015) 10. Magneron, B.: Représentations induites holomorphes des groupes de Lie nilpotents et involutions complexes. C. R. Acad. Sci. Paris, Ser. I Math. 317, 37–42 (1993) 11. Magneron, B.: Involutions complexes et vecteurs sphériques associés pour les groupes de Lie nilpotents réels. Astérisque, no. 253 (1999) 12. Penney, R.: Abstract Plancherel theorems and a Frobenius reciprocity theorem. J. Funct. Anal. 18, 177–190 (1975) 13. Penney, R.: Holomorphically induced representations of exponential Lie groups. J. Funct. Anal. 64, 1–18 (1985) 14. Rossi, H., Vergne, M.: Representations of certain solvable Lie groups on Hilbert spaces of holomorphic functions and the application to the holomorphic discrete series of a semisimple Lie group. J. Funct. Anal. 13, 324–389 (1973)

Spherical Functions for Small K-Types Hiroshi Oda and Nobukazu Shimeno

Abstract For a connected semisimple real Lie group G of non-compact type, Wallach introduced a class of K-types called small. We classify all small K-types for all simple Lie groups and prove except just one case that each elementary spherical function for each small K-type (π, V ) can be expressed as a product of hyperbolic cosines and a Heckman–Opdam hypergeometric function. As an application, the inversion formula for the spherical transform on G ×K V is obtained from Opdam’s theory on hypergeometric Fourier transforms. Keywords Small K-types · Spherical functions · Hypergeometric functions 1991 Mathematics Subject Classification 22E45 · 33C67 · 43A90

1 Introduction Let G be a connected real semisimple Lie group with finite center. Let G = KAN be an Iwasawa decomposition of G. K-bi-invariant C ∞ functions on G are called spherical functions and are important in the analysis of functions on the Riemannian symmetric space G/K. In particular, a key role is played by those spherical functions that are elementary. Here a spherical function φ is called elementary if it is non-zero and satisfies the functional equation

The first author was supported by JSPS KAKENHI Grant Number 18K03346. H. Oda Faculty of Engineering, Takushoku University, 815-1 Tatemachi, Hachioji, Tokyo 193-0985, Japan e-mail: [email protected] N. Shimeno (B) School of Science and Technology, Kwansei Gakuin University, 2-1 Gakuen, Sanda, Hyogo 669-1337, Japan e-mail: [email protected] © Springer Nature Switzerland AG 2019 A. Baklouti and T. Nomura (eds.), Geometric and Harmonic Analysis on Homogeneous Spaces, Springer Proceedings in Mathematics & Statistics 290, https://doi.org/10.1007/978-3-030-26562-5_6

121

122

H. Oda and N. Shimeno

 φ(xky)dk = φ(y)φ(x)

for any x, y ∈ G

K

(dk is the normalized Haar measure on K), or equivalently, if it takes 1 at 1G ∈ G and is a joint eigenfunction of the algebra D of the invariant differential operators on G/K (cf. [16, 22]). It is well known that the spherical transform (also called the Harish-Chandra transform) defined by elementary spherical functions essentially gives the irreducible decomposition of L2 (G/K). Now suppose (π, V ) is any irreducible unitary representation of K (a K-type for short). When we consider the analysis of sections of the vector bundle G ×K V in a parallel way to the case of G/K (which corresponds to the trivial K-type), there naturally appears a notion of elementary spherical functions for (π, V ). Unfortunately the general theory for such functions, which has been developed by [13, 14, 39, 41] and others, has some inevitable complexity. But it can be considerably reduced when the algebra Dπ of the invariant differential operators on G ×K V is commutative (cf. [2]). We know from [7, Theorem 3] that Dπ is commutative if and only if V decomposes multiplicity-freely as an M -module (M is the centralizer of A in K). In what follows we assume that (π, V ) satisfies this condition. Definition 1.1 An EndC V -valued C ∞ function φ on G is called π-spherical if φ(k1 gk2 ) = π(k2−1 )φ(g)π(k1−1 ) for any g ∈ G and k1 , k2 ∈ K.

(1.1)

The space of π-spherical functions is denoted by C ∞ (G, π, π). A π-spherical function φ is called elementary when it is non-zero and satisfies  φ(xky) Tr π(k)dk = K

1 φ(y)φ(x) for any x, y ∈ G. dim V

(1.2)

As we see in Sect. 3.2 in detail, Dπ naturally acts on C ∞ (G, π, π). Theorem 1.2 ([2, Theorem 3.8]) A given φ ∈ C ∞ (G, π, π) is elementary if and only if it takes id V at 1G and is a joint eigenfunction of Dπ . The theorem is certainly central when we investigate analytical properties of elementary π-spherical functions. However, to get even a little of explicit results, two more things seem necessary, namely, the structure of Dπ and a version of Chevalley restriction theorem for C ∞ (G, π, π). As shown below, both of them are available if (π, V ) is small in the sense of Wallach. Definition 1.3 ([40, Sect. 11.3]) A K-type (π, V ) is called small if V is irreducible as an M -module. In this paper we restrict ourselves to the study of elementary π-spherical functions for small K-types. So hereafter let (π, V ) be a small K-type. First, we have a lot of the same results as in the case of the trivial K-type. For example, there is a generalization of the Harish-Chandra isomorphism by Wallach (Theorem 3.1):

Spherical Functions for Small K-Types

123

∼ → S(aC )W . γ π : Dπ − Here S(aC ) is the symmetric algebra for the complexification of the Lie algebra a of A. (As usual, when a capital English letter denotes a Lie group, the corresponding German small letter denotes its Lie algebra.) W is the Weyl group defined, as usual, ˜ /M with M ˜ being the normalizer of A in K. S(aC )W is the subalgebra of by W = M S(aC ) consisting of the W -invariants. Furthermore we have Theorem 1.4 For any λ ∈ a∗C (the dual linear space of aC ), there exists a unique φπλ ∈ C ∞ (G, π, π) such that φπλ (1G ) = id V

and D φπλ = γ π (D)(λ) φπλ for any D ∈ Dπ .

(1.3)

Moreover, φπλ is real analytic on G. Thus the elementary π-spherical functions are parameterized by λ ∈ W \a∗C . The proof of the theorem is given in Sect. 3.3, together with two integral formulas for φπλ . Now, since one easily sees from (1.1) the restriction φ|A of any φ ∈ C ∞ (G, π, π) to A takes values in EndM V and since the C-algebra EndM V is isomorphic to C by Schur’s lemma, φ|A is identified with a C-valued C ∞ function on A. Let Υ π (φ) ∈ ∼ → A. C ∞ (a) be the composition of φ|A with exp : a − Theorem 1.5 (the Chevalley restriction theorem) The restriction map Υ π is a linear bijection between C ∞ (G, π, π) and C ∞ (a)W . The proof is given in Sect. 3.1. Through this bijection, (1.3) is translated into a condition on Υ π (φπλ ) to be a joint eigenfunction of a commuting family of differential operators on a. The family consists of the π-radial parts of D ∈ Dπ . The π-radial part of the Casimir operator, which has a prominent role in the family, is expressed in a uniform way by use of a parameter κπ associated with (π, V ) (Theorem 3.9). The parameter κπ , whose precise definition is given in Sect. 3.4, is a W -invariant function on the set Σ = Σ(g, a) of restricted roots. (In general, a Weyl group invariant function on a root system is called a multiplicity function.) Of course, we could proceed further with our study analogously to the case of the trivial K-type, which would include calculation of the c-function for each individual (π, V ) by rank-one reduction. But we take an alternative route, attaching importance to the fact that in almost all cases the system of differential equations for Υ π (φπλ ) coincides with a hypergeometric system of Heckman and Opdam [20] up to twist by a nowhere-vanishing function. In general, their hypergeometric system is defined for any triple of a root system Σ  in a∗ , a multiplicity function k on Σ  , and λ ∈ a∗C . (Σ  spans a∗ by definition. Throughout the paper a root system is assumed crystallographic.) If k satisfies a certain regularity condition, their system has a unique Weyl group invariant analytic solution F(Σ  , k, λ) such that F(Σ  , k, λ; 0) = 1. The solution F(Σ  , k, λ) is called a hypergeometric function associated to the root system Σ  , which is thought of as a deformation of the elementary spherical function for the trivial K-type by an arbitrary complex root multiplicity k. When dim a∗ = 1,

124

H. Oda and N. Shimeno

F(Σ  , k, λ) reduces to a Jacobi function, which is studied by Koornwinder and Flensted-Jensen prior to [20] (cf. [25]). A Jacobi function is essentially a Gauss hypergeometric function. We review the definition and fundamental properties of F(Σ  , k, λ) in more detail in Sect. 4. To state our main result, we fix some notation. Let Σ + be the positive system corresponding to N . For any α ∈ Σ let gα be the restricted root space and put mα = dim gα . Then m : Σ  α → mα ∈ Z>0 is a multiplicity function on Σ. We put δ˜G/K =

    sinh α mα    ||α||  .

(1.4)

α∈Σ +

Here we consider a and a∗ as inner product spaces by the Killing form B(·, ·) of g. Likewise, for any root system Σ  in a∗ and multiplicity function k on Σ  we put ˜  , k) = δ(Σ

  sinh(α/2) 2kα    ||α/2||  +

(1.5)

α∈Σ

where Σ + is some positive system of Σ  . The main result of this paper is the following. Theorem 1.6 Suppose (π, V ) is a small K-type of a non-compact real simple Lie group G with finite center. If G is a simply-connected split Lie group G˜ 2 of type G 2 , we further suppose π is not the small K-type π2 specified in Theorem 2.2. Then there exist a root system Σ π in a∗ and a multiplicity function kπ on Σ π such that −2 ˜ π , kπ ) 21 F(Σ π , kπ , λ) for any λ ∈ a∗ . Υ π (φπλ ) = δ˜G/K δ(Σ C 1

(1.6)

The proof of the theorem is divided into two large steps. As the first step, we derive in Sect. 5 a simple condition on Σ π and kπ for the validity of (1.6) under the assumption that Σ π ⊂ Σ ∪ 2Σ (Proposition 5.9). This condition consists of only a few equations between kπ , m and κπ . Thus κπ encodes all the information on (π, V ) needed for our purpose. As the second step, we determine the values of κπ for each small K-type (π, V ) of each non-compact simple real Lie group G in order to find a pair of Σ π and kπ using the condition in the first step. The existence of such a pair is actually confirmed in each case except π2 of G˜ 2 . In this process all small K-types are classified for each G. This generalizes a result of Lee which classifies small K-types for each split real simple Lie group (cf. [27, 28]). The case-by-case analysis in this step is carried out in Sect. 6. We also prove in Sect. 6.9 that in the case of π2 of G˜ 2 , (1.6) never holds for any choice of Σ π and kπ . As a detailed explanation of Theorem 1.6, the concrete information obtained in the second step is summarized in Sect. 2. That is, the classification of small K-types and one or two possible choices of Σ π and kπ for each small K-type (except π2 of G˜ 2 ). − 21 ˜ π , kπ ) 21 in (1.6) extends Now, for each our choice of Σ π and kπ , the factor δ˜G/K δ(Σ to a nowhere-vanishing real analytic function on a. Indeed, this can be written as a

Spherical Functions for Small K-Types

125

product of hyperbolic cosines (Proposition 5.4), whose concrete form in each case is also presented in Sect. 2. If G is of Hermitian type, a small K-type is nothing but a one-dimensional unitary representation of K (Sect. 2.6, Theorem 6.11). Hence in this case Theorem 1.6 restates results of [19, Chap. 5] and [36]. If G has real rank one, then the hypergeometric function in (1.6) is a Jacobi function. Hence in the rank one case, one can see some known results are essentially equivalent to Theorem 1.6. For example, a result of [12] for the one-dimensional K-types of the universal cover of SU(p, 1), that of [3] for the lowest-dimensional non-trivial small K-types of Spin(2p, 1) (p ≥ 2) and that for the small K-types of Sp(p, 1) obtained by [8, 37, 38]. According to Oshima [34], a commuting family of W -invariant differential operators on a is necessarily equal to a system of hypergeometric differential operators up to a gauge transform under the conditions: (1) W is of a classical type; (2) the symbols of the operators span S(aC )W (complete integrability); and (3) the operators have regular singularities at every infinity. In view of this, our result is not so surprising since the family of π-radial parts of D ∈ Dπ satisfies (2) and (3). Also, the exceptional case in Theorem 1.6 suggests the possibility that there might be a new class of completely integrable systems associated with the Weyl group of type G 2 . Now, Formula (1.6) enables us to reduce a large part of analytic theory for elementary π-spherical functions to the one for Heckman–Opdam hypergeometric functions. For example, Harish-Chandra’s c-function for G ×K V equals a scalar multiple of the c-function for Heckman–Opdam hypergeometric functions (Theorem 7.2), and the π-spherical transform (the spherical transform for G ×K V ) is a composition of a multiplication operator and a hypergeometric Fourier transform introduced by [32] (Theorem 7.4). Using them we can obtain the explicit formula of Harish-Chandra’s c-function and the inversion formula of the π-spherical transform. These applications are discussed in Sect. 7.

2 Detailed Description of the Main Result In this section we list all small K-types for each non-compact real simple Lie group G up to equivalence. (Actually the classification is given for each real simple Lie algebra of non-compact type since a small K-type for G is always lifted to that for any finite cover of G.) In addition, for each small K-type π other than the one exception stated in Sect. 1, we present one or two combinations of Σ π and kπ for which (1.6) is valid. Though there may be some or infinitely many other choices of such Σ π and kπ (cf. Sect. 6.2), we do not pursue all the possibilities. The results of this section will be proved in Sect. 6.

126

H. Oda and N. Shimeno

2.1 The Trivial K-Type First of all, the trivial K-type (π, V ) is small for any G. In this case, (1.6) holds with − 21 ˜ π , kπ ) 21 = 1 and Υ π (φπ ) = Σ π = 2Σ and kπ : Σ π  2α → m2α , whence δ˜G/K δ(Σ λ F(Σ π , kπ , λ). In the rest of this section we basically treat only non-trivial small K-types.

2.2 Simple Lie Groups Having No Non-trivial Small K-Type There is no non-trivial small K-type in each of the following cases: • • • • • •

G is a complex simple Lie group; g sl(p, H) (p ≥ 2); g sp(p, q) (p ≥ q ≥ 2); g so(2r + 1, 1) (r ≥ 1); g e6(−26) (E IV); g f4(−20) (F II).

2.3 The Case g = sp(p, 1) (p ≥ 1) Suppose G = Sp(p, 1) (p ≥ 1) and K = Sp(p) × Sp(1). Then G is simplyconnected. Let pr 1 and pr 2 be the projections of K to Sp(p) and Sp(1) respectively. For the irreducible representation (πn , Cn ) of Sp(1) SU(2) of dimension n = 1, 2, . . . , the K-type πn ◦ pr 2 is small. If p = 1 then πn ◦ pr 1 is also a small K-type. There are no other small K-types. Let Σ = {±α, ±2α} if p ≥ 2 and Σ = {±2α} if p = 1. Let π = πn ◦ pr 2 if p ≥ 2 and π = πn ◦ pr 1 or πn ◦ pr 2 if p = 1. Then putting Σ π = {±2α, ±4α}, kπ2α = 2p − 1 ± n and kπ4α = 21 ∓ n, we have (1.6) and −1 ˜ π , kπ ) 21 = (cosh α)−1∓n . δ˜ 2 δ(Σ G/K

2.4 The Case g = so(2r, 1) (r ≥ 2) Suppose G = Spin(2r, 1) (r ≥ 2) and K = Spin(2r). Then G is simply-connected. For s = 0, 1, 2, . . . , the irreducible representation πs± of K = Spin(2r) with highest weight (s/2, . . . , s/2, ±s/2) in the standard notation is small. There are no other small K-types. Let Σ = {±α}. Let π = πs± . Then putting Σ π = Σ ∪ 2Σ = − 21 ˜ π , kπ ) 21 = {±α, ±2α}, kπα = −s and kπ2α = r + s − 21 , we have (1.6) and δ˜G/K δ(Σ α s (cosh 2 ) .

Spherical Functions for Small K-Types

127

2.5 The Case g = so(p, q) (p > q ≥ 3) Suppose G is the double cover of Spin(p, q) (p > q ≥ 3). Thus K = Spin(p) × Spin(q) and G is simply-connected. Let pr 1 and pr 2 be the projections of K to Spin(p) and Spin(q) respectively. We may assume Σ = {±ei | 1 ≤ i ≤ q} ∪ {±ei ± ej | 1 ≤ i < j ≤ q} for some orthogonal basis {ei | 1 ≤ i ≤ q} of a∗ with ||e1 || = · · · = ||eq ||. (i) Let σ denote the spin representation of Spin(q) if q is odd, and either of two half-spin representations of Spin(q) if q is even. Then π = σ ◦ pr 2 is a small K-type. For this π, we can choose Σ π = {±2ei | 1 ≤ i ≤ q} ∪ {±ei ± ej | 1 ≤ and kπ±ei ±ej = 21 so that (1.6) holds and i < j ≤ q} and kπ with kπ±2ei = p−q 2 1  1 −2 ei −ej ei +ej − 1 ˜ π , kπ ) 2 = 2. δ˜G/K δ(Σ 1≤i 0 (∀α ∈ Σ  ) we have lim et(−λ+ρ(k))(H ) F(Σ  , k, λ; tH ) = c(Σ  , k, λ).

t→∞

142

H. Oda and N. Shimeno

Proof Immediate from (4.7) and the previous corollary.



Remark 4.11 The prototype of the theorem is a similar result for Opdam’s generalized Bessel functions obtained by Dunkl, de Jeu and Opdam [11, Sect. 4]. They also give for each type of irreducible Σ  a very explicit description of the singular set of those k which do not satisfy (2). By the theorem this singular set equals K(Σ  ) \ Kreg (Σ  ). The rest of this subsection is devoted to the proof of Theorem 4.7. For d = 0, 1, 2, . . . let Sd (aC ) be the subspace of S(aC ) ( P(a)) consisting of homogeneous polynomials of degree d . Since T¯ k (H ) is a homogeneous operator of degree −1 for ˆ any H ∈ a \ {0}, Sd (aC ) ⊥ Se (a

C ) with respect to (·, ·)k if d  = e. Now any f ∈ A0 decomposes into the sum f = d ≥0 fd of its homogeneous parts fd ∈ Sd (aC ). We define ord f = min{d | fd = 0} ∈ N ∪ {∞} and put Aˆ0,>d = {f ∈ Aˆ0 | ord f > d }. Then we have  Se (aC ). Aˆ0 /Aˆ0,>d S≤d (aC ) := e≤d

The next lemma is easily observed: Lemma 4.12 Suppose f ∈ Aˆ0 \ {0} with d = ord f . Then for any H ∈ a, ord(Tk (H )f ) ≥ d − 1 and Tk (H )f ≡ T¯ k (H )fd

(mod Aˆ0,>d −1 ).

For d = 0, 1, 2, . . . let ·, ·dk be the restriction of ·, ·k to S≤d (aC ) × S≤d (aC ). Since dim S≤d (aC ) < ∞, the left and right radicals of ·, ·dk have the same dimension. Let us consider an auxiliary condition: (3) ·, ·dk is non-degenerate for each d = 0, 1, 2, . . . . Proof of (2)⇔(3)⇔(3) ⇒(4). Suppose (2). For any D ∈ S(aC ) \ {0} with deg D = d , we can take f ∈ Sd (aC ) so that (D, f )k = 0. Then from the lemma above we have D, f k = (Tk (D)f )(0) = (T¯ k (D)f )(0) = (D, f )k = 0, proving (3). Next, one easily sees (3)⇒(3) ⇒(4) since S≤d (aC ) ⊥ Aˆ0,>d with respect to ·, ·k by the lemma. To deduce (2) from (3) , take an arbitrary f ∈ Sd (aC ) \ {0} with d = 0, 1, 2, . . . . Then (3) assures the existence of D ∈ S≤d (aC ) such that D, f k = 0. Since ord f =  d , the lemma again implies (D, f )k = D, f k = 0. The implication (1)⇒(6)⇒(7) is obvious. ˜  , k0 , λ; H ) is a non-trivial function in Lemma 4.13 For any k0 ∈ K(Σ  ), F(Σ (λ, H ) ∈ a∗C × a. Proof Because there is a non-empty open subset Uk0 ⊂ a∗C such that (4.7) holds for any (λ, H ) ∈ Uk0 × a− and k = k0 . 

Spherical Functions for Small K-Types

143

Hence by Theorem 4.5 we have (5)⇒(1).  Proof  of (7)⇒(3). Suppose l-Rad k  D = 0. Since l-Radk is an ideal of S(aC ), D := w∈W  w(D) is a non-zero W -invariant element in l-Rad k . Now suppose (7) is true. Then we can find λ ∈ Z such that D (λ) = 0. But for a function f ∈ Aˆ0 of (HG1 )–(HG3 ) we have D , f k = (Tk (D )f )(0) = D (λ)f (0) = D (λ) = 0. This  contradicts that D ∈ l-Radk . The proof is complete if we show (4)⇒(5). To do so, we needs the graded Hecke algebra H = H(Σ + , k) by [29]. The algebra H is isomorphic to CW  ⊗ S(aC ) as a C-linear space. Here the group algebra CW  of W  and S(aC ) are identified with subalgebras of H by w → w ⊗ 1 and D → 1 ⊗ D. These two subalgebras relate to each other as follows: for any w ∈ W  and D ∈ S(aC );

w·D =w⊗D H · rα = rα · rα (H ) − (kα + 2k2α )α(H )

for any simple root α ∈ Σ + and any H ∈ aC .



The center of H equals S(aC )W [29, Theorem 6.5]. Thanks to [5, Theorem 2.4], H  w ⊗ D → w · Tk (D) ∈ EndC Aˆ0 defines an algebra homomorphism (also denoted by Tk (·)). Let Cv0 be a one-dimensional right trivial W  -module. Then Cv0 ⊗CW  H is a right H-module. Identifying S(aC ) with Cv0 ⊗CW  H by D → v0 ⊗ D, we consider ·, ·k is a bilinear form on Cv0 ⊗CW  H × Aˆ0 . Observe that ·h, ·k = ·, h·k for any h ∈ H. For d = 0, 1, 2, . . . , v0 ⊗ S≤d (aC ) is a W  -submodule of Cv0 ⊗CW  H and v0 ⊗ S≤d (aC )



v0 ⊗ S≤d −1 (aC ) Sd (aC ) with natural right W  -module structure.

From this one easily sees Lemma 4.14 The subspace of Cv0 ⊗CW  H consisting of the right W  -invariants is  v0 ⊗ S(aC )W . Proof of (4)⇒(5). Let λ ∈ a∗C and suppose f ∈ Aˆ0 satisfies (HG1 ), (HG2 ) and ˜ ∈ f (0) = 0. Take an arbitrary D ∈

S(aC ) . Then by the last lemma there exists D  ˜ = 1  w∈W  v0 ⊗ D · w. Hence we have S(aC )W such that v0 ⊗ D #W D, f k = v0 ⊗ D, f k =

1  1  v0 ⊗ D, w · f k = v0 ⊗ D · w, f k  #W #W    w∈W

w∈W

˜ f k = (Tk (D)f ˜ )(0) = D(λ)f ˜ = v0 ⊗ D, (0) = 0.

This shows f ∈ r-Radk .



144

H. Oda and N. Shimeno

5 Matching Conditions We return to the setting of Sect. 3. Thus (π, V ) is a small K-type of a connected noncompact real semisimple Lie group G with finite center. The purpose of this section is to get an easy and concrete condition on a root system Σ π and a multiplicity function kπ for the validity of (1.6).

5.1 Coincidence of Differential Operators Let Σ  is a root system in a∗ (a = Lie A) and k a multiplicity function on Σ  . In addition, we suppose Σ  ⊂ Σ ∪ 2Σ and the Weyl group W  for Σ  equals W . Let Σ + := Σ  ∩ (Σ + ∪ 2Σ + ). Then the notation in Sect. 3 and that in Sect. 4 are fully compatible. Note that the algebra R  , which is generated by (1 − eα )−1 (α ∈ Σ + ), is a subalgebra of R, which is generated by (1 ± eα )−1 (α ∈ Σ + ), and that M  = M ∩ R. Proposition 5.1 With δ˜G/K in (1.4) it holds that −2 2 ◦ (Δπ (Ωg − π ) + ||ρ||2 ) ◦ δ˜G/K (5.1) δ˜G/K

2 π  mα ||α|| −κα 2 − mα − 2m2α + 4κπα . + =Ωa + 4 sinh2 α2 sinh2 α + 1

1

α∈Σ

Proof Letting Σ  = 2Σ and k2α = 21 mα (α ∈ Σ), we have L(Σ  , k) = Ω +



˜  , k) = δ˜G/K and ρ(k) = ρ. mα coth α Hα , δ(Σ

α∈Σ +

Hence it follows from (3.10) and the Proposition 4.2 that 1

1

−2 2 ◦ (Δπ (Ωg − π ) + ||ρ||2 ) ◦ δ˜ G/K = Ωa + δ˜ G/K

 mα ||α||2 2 − mα − 2m2α κπα . − 4 sinh2 α cosh2 α2 α∈Σ +

Using the equality 1 cosh2 we get the proposition.

α 2

=

1 sinh2

α 2

−

4 , sinh2 α 

Let us consider general Σ  and k again. The algebra homomorphism γρ(k) : R ⊗ S(aC ) → S(aC ) defined by (4.3) can be regarded as a part of the algebra homomorphism 

Spherical Functions for Small K-Types

145 f (λ) →f (λ+ρ(k))

projection

γρ(k) : R ⊗ S(aC ) = S(aC ) ⊕ M ⊗ S(aC ) −−−−−→ S(aC ) −−−−−−−−−→ S(aC ). Lemma 5.2 The subalgebra     (R ⊗ S(aC ))W,L(Σ ,k) := D ∈ (R ⊗ S(aC ))W  [L(Σ  , k), D] = 0 

coincides with (R  ⊗ S(aC ))W,L(Σ ,k) . Proof By the same argument as in [19, Sect. 1.2], we can prove the restriction of  γρ(k) to (R ⊗ S(aC ))W,L(Σ ,k) is injective homomorphism into S(aC )W . Hence the  lemma follows from Proposition 4.3. (Recall W = W  by assumption.) Proposition 5.3 Suppose for a choice of Σ  and k the equality 1

1

˜  , k)− 2 = δ˜ 2 ◦ (Δπ (Ωg − π ) + ||ρ||2 ) ◦ δ˜ − 2 ˜  , k) 2 ◦ (L(Σ  , k) + (ρ(k), ρ(k))) ◦ δ(Σ δ(Σ G/K G/K 1

1

(5.2) holds, namely, the operators in (4.2) and (5.1) coincide. Then we have  ˜  , k)− 21 δ˜ 2 ◦ Δπ (Dπ ) ◦ δ˜− 2 δ(Σ ˜  , k) 21 . (R  ⊗ S(aC ))W,L(Σ ,k) = δ(Σ G/K G/K 1

1

Moreover, for any D ∈ U (gC )K it holds that   1 1 ˜  , k) 21 = γ π (D). ˜  , k)− 21 δ˜ 2 ◦ Δπ (D) ◦ δ˜− 2 δ(Σ γρ(k) δ(Σ G/K G/K Proof Define an algebra homomorphism τ : U (gC )K → (R ⊗ S(aC ))W by ˜  , k)− 21 δ˜ 2 ◦ Δπ (D) ◦ δ˜− 2 δ(Σ ˜  , k) 21 . τ (D) = δ(Σ G/K G/K 1

1

Suppose D ∈ U (gC )K . Then Proposition 3.10 implies γ π (D) = γρ ◦ Δπ (D), while 1 1   ˜  , k)− 21 δ˜ 2 ◦ E ◦ δ˜− 2 δ(Σ ˜  , k) 21 = γρ (E) for any E ∈ one easily sees γρ(k) δ(Σ G/K G/K R ⊗ S(aC ). Thus we have γρ(k) ◦ τ (D) = γ π (D), the second assertion of the proposition. Now it holds that   [L(Σ  , k), τ (D)] = τ [Ωg − π + ||ρ||2 − (ρ(k), ρ(k)), D] = 0. 



This shows τ (U (gC )K ) ⊂ (R ⊗ S(aC ))W,L(Σ ,k) = (R  ⊗ S(aC ))W,L(Σ ,k) . But these two subalgebras actually coincide by Theorem 3.1, Proposition 4.3 and the second assertion.  Since the functions sinh−2 if and only if

α 2

(α ∈ Σ ∪ 2Σ) are linearly independent, (5.2) holds

  − mα κπα + 21 m α 1 − 21 m α − mα + 2κπα = kα (1 − kα − 2k2α ) for any α ∈ Σ ∪ 2Σ. 2

2

2

(5.3)

146

H. Oda and N. Shimeno

Here we suppose mα = κπα = 0 for α ∈ / Σ and kα = 0 for α ∈ / Σ . −2 ˜  , k) 21 ∈ A (areg ) extends to a real analytic Proposition 5.4 The function δ˜G/K δ(Σ function on some open set U containing areg ∪ {0} if and only if 1

mα + m2α = kα + k2α + k4α for any α ∈ Σ \ 2Σ. 2

(5.4)

If this is the case then −2 ˜  , k) 21 = δ˜G/K δ(Σ 1

 α∈Σ + \2Σ +

cosh

α 2

−kα

(cosh α)k4α −

m2α 2

.

(5.5)

−2 ˜  , k) 21 extends to a nowhere-vanishing real analytic function δ(Σ In particular, δ˜G/K on a taking 1 at 0 ∈ a. 1

Proof The first statement is immediate since for each α ∈ Σ ∪ 2Σ we have the expansion

α (α/2)2 (α/2)4 sinh(α/2) = 1+ + + ··· . ||α/2|| ||α|| 3! 5! Next, (5.5) holds since for each α ∈ Σ + \ 2Σ +   mα   m2α        sinh α − 2  sinh(2α) − 2  sinh(α/2) kα  sinh α k2α  sinh(2α) k4α            ||α||   ||2α||   ||α/2||   ||α||   ||2α||    m2α  mα sinh α kα = | sinh α|− 2 | sinh α cosh α|− 2  | sinh α|k2α | sinh α cosh α|k4α cosh(α/2) 

m2α α −kα = cosh (cosh α)k4α − 2 . 2

 Theorem 5.5 Suppose (5.3) and (5.4) hold for a choice of Σ  and k. Then k ∈ Kreg (Σ  ) and it holds that −2 ˜  , k) 21 F(Σ  , k, λ) for any λ ∈ a∗ . Υ π (φπλ ) = δ˜G/K δ(Σ C 1

1

(5.6)

˜  , k)− 21 δ˜ 2 Υ π (φπ ) extends to a Proof Suppose λ ∈ a∗C . By Proposition 5.4, δ(Σ G/K λ real analytic function on a taking 1 at 0 ∈ a. This is clearly W -invariant. Also, it follows from Corollary 3.13 and Proposition 5.3 that this function satisfies (HG2) 1 ˜  , k)− 21 δ˜ 2 Υ π (φπ ) satisfies (HG1)–(HG3) for any λ ∈ a∗ . in Sect. 4.1. Thus δ(Σ G/K C λ Hence k ∈ Kreg (Σ  ) by the implication (6)⇒(1) in Theorem 4.7. Finally, (5.6) follows from Corollary 4.8. 

Spherical Functions for Small K-Types

147

In the notation of Theorem 1.6 our result is summarized as follows. Let Σ π be a root system in a∗ and kπ a multiplicity function on Σ π . Then (1.6) holds if the following conditions are satisfied: (MC1) Σ π ⊂ Σ ∪ 2Σ; (MC2) Equation (5.3) holds with (Σ  , k) = (Σ π , kπ ); (MC3) Equation (5.4) holds with (Σ  , k) = (Σ π , kπ ). Note that if (MC3) is true, then each root in Σ is proportional to some root in Σ π . Hence the Weyl group of Σ π equals W under (MC1) and (MC3). It is not so hard to observe that under (MC1), (1.6) holds only if both (MC2) and (MC3) are true. (We do not use this fact in the paper.) Conditions (MC1)–(MC3) will be further simplified after we look into the structure of (π, V ) more precisely.

5.2 The Associated Split Semisimple Subgroup Let b be a Cartan subalgebra of m = Lie M . Then bC + aC is a Cartan subalgebra of gC . A root μ for (gC , bC + aC ) is called real when μ|bC = 0. We denote the set of real roots by Σreal , which is naturally identified with a subset of Σ. A restricted root α ∈ Σ belongs to Σreal if and only if mα is odd (cf. [21, Chap. X, Exercises F]). Now  gbα gb = a + b + α∈Σreal

is a reductive subalgebra of g. Its semisimple part is gsplit := [g , g ] = asplit + b

b

 α∈Σreal

gbα



 asplit := RHα ,

(5.7)

α∈Σreal

which is a split semisimple Lie algebra with Cartan subalgebra asplit [24, Chap. VII, §5]. The restricted root system of gsplit is identified with Σreal . Let G split be the analytic subgroup for gsplit (the associated split semisimple subgroup). Let M0 denote the identity component of M . If we put Ksplit = K ∩ G split , Msplit = M ∩ Ksplit , then Msplit is the centralizer of asplit in Ksplit . Furthermore Msplit normalizes M0 , and M = M0 Msplit (cf. [24, Theorem 7.52]). Proposition 5.6 The restriction of (π, V ) to M0 is isomorphic to the direct sum of some copies of an irreducible representation of M0 : V |M0 U ⊕r . Let μ be an extremal bC -weight of U and put Vμ = {v ∈ V | π(H )v = μ(H )v (∀H ∈ b)}.

(5.8)

Then π(Ksplit )Vμ ⊂ Vμ and Vμ is irreducible as an Msplit -module. That is, (π|Ksplit , Vμ ) is a small Ksplit -type of G split with dimension r.

148

H. Oda and N. Shimeno

Proof Let √ μ be the highest weight of V |M0 with respect to a lexicographical order of −1b∗ and define Vμ by (5.8). Since Ksplit ⊂ G b , we have π(Ksplit )Vμ ⊂ Vμ . Let E ⊂ Vμ be an irreducible Msplit -submodule. Since Msplit normalizes M0 , π(U (mC ))E is M -stable and hence is equal to V . By the highest weight theory, E = Vμ ∩ π(U (mC ))E = Vμ . Thus Vμ is an irreducible Msplit -module. By the highest weight theory again, we see if U is an irreducible M0 -module with highest weight  μ then V |M0 U ⊗r with r = dim Vμ . Corollary 5.7 For any α ∈ Σ with even mα we have κπα = 0. Proof Let Δm be the root system for (mC , bC ). Suppose α ∈ Σ has an even root multiplicity. Then α ∈ / Σreal and all bC -weights of mC -module (gα )C are not zero. Hence by the representation theory of complex reductive Lie algebra, any bC -weight of (gα )C (or equivalently, that of {Xα + θXα | Xα ∈ (gα )C }) is outside the root lattice ZΔm . Let μ be as in Proposition 5.6. Then by the proposition, μ − λ belongs to ZΔm for any bC -weight λ of V . But this means the difference of μ and any bC -weight of the M -submodule {π(Xα + θXα )v | Xα ∈ gα , v ∈ V } ⊂ V is also inside ZΔm . It is possible only when {π(Xα + θXα )v | Xα ∈ gα , v ∈ V } = {0}. Hence Proposition 3.8  implies κπα = 0. Corollary 5.8 Let Vμ be as in Proposition 5.6 and put (πμ , Vμ ) = (π|Ksplit , Vμ ). For π any α ∈ Σ with mα = 1, we have α ∈ Σreal and κπα = καμ . Proof Suppose α ∈ Σ with mα = 1 is given. One has α ∈ Σreal and gα ⊂ gsplit . 2 B(Xα , θXα ) = 1. Then κπα id V = π(Xα + θXα )2 by defiTake Xα ∈ gα so that − ||α|| 2 nition. Note the normalization condition for Xα is rewritten as [Xα , [Xα , θXα ]] = 2Xα , π which is common to both g and gsplit . Hence καμ id Vμ = πμ (Xα + θXα )2 = π(Xα +  θXα )2 |Vμ = κπα id Vμ .

5.3 Simplifying Matching Conditions Proposition 5.9 Let R be a complete system of representatives for the W -orbits of Σ \ 2Σ. Let Σ π be a root system in a∗ satisfying (MC1). Then a multiplicity function kπ on Σ π satisfies (MC2) and (MC3) if and only if the following are valid: (1) for any α ∈ R with m2α = 0  ⎧ mα − 1 ± (mα − 1)2 − 4mα κπα ⎪ π ⎪ , ⎨ kα = 2  2 π ⎪ ⎪ ⎩ kπ = 1 ∓ (mα − 1) − 4mα κα ; 2α 2 (2) for any α ∈ R with m2α > 0

Spherical Functions for Small K-Types

149

⎧ π kα = 0, ⎪ ⎪ ⎪  ⎪ ⎪ ⎨ π mα + m2α − 1 ± (m2α − 1)2 − 4m2α κπ2α k2α = , 2 ⎪  ⎪ ⎪ π 2 ⎪ ⎪ ⎩ kπ = 1 ∓ (m2α − 1) − 4m2α κ2α , 4α 2 or κπ2α

⎧ π kα = mα + m2α − 1, ⎪ ⎪ ⎨ 1 1 mα + m2α and = m2α − , kπ2α = 1 − ⎪ 4 2 2 ⎪ ⎩ π k4α = 0.

Proof Suppose α ∈ R and m2α > 0. Then mα is even (cf. [21, Chap. X, Exercises F]) and κπα = 0 by Corollary 5.7. Thus those parts of (5.3) and (5.4) that relate to α, 2α and 4α are reduced to: ⎧ 0 = kπα (1 − kπα − 2kπ2α ), ⎪ ⎪ ⎪ ⎪ ⎨ −m2α κπ + 1 mα (1 − 1 mα − m2α ) = kπ (1 − kπ − 2kπ ), 2α 2α 2α 4α 2 2 (5.9) π π π 1 1 ⎪ m (1 − 2 m2α + 2κ2α ) = k4α (1 − k4α ), ⎪ 2 2α ⎪ ⎪ ⎩ 1 (mα + m2α ) = kπα + kπ2α + kπ4α . 2 In addition, since Σ π is a root system, either kπα or kπ4α is zero. Hence by an elementary argument (5.9) is still reduced to the condition in (2). The condition in (1) for α ∈ R  with m2α = 0 is obtained in a similar way.

6 Case-by-Case Analysis In this section, all the results in Sect. 2 will be proved through case-by-case analysis. We start with some preparation. Let G be a non-compact real simple Lie group with finite center. Note that G is connected by definition. Lemma 6.1 Suppose k1 is an ideal of k such that k1 ⊂ m. Then k1 = {0}. Proof By assumption, one has for any k ∈ K [k1 , Ad(k)a] = Ad(k)[Ad(k −1 )k1 , a] = Ad(k)[k1 , a] ⊂ Ad(k)[m, a] = {0}. But since s =

 k∈K

Ad(k)a, k1 is an ideal of g, which must be {0} since k1 ⊂ k = g. 

Corollary 6.2 The Lie algebra k is generated by {Xα + θXα | Xα ∈ gα (α ∈ Σ)}.

150

H. Oda and N. Shimeno

Proof Note k = m ⊕ α∈Σ {Xα + θXα | Xα ∈ gα } and {Xα + θXα | Xα ∈ gα } is M stable for each α ∈ Σ. It follows that the Lie subalgebra k0 generated by {Xα + θXα | Xα ∈ gα (α ∈ Σ)} is an ideal of k. Hence the orthogonal complement k1 of k0 in k with respect to B(·, ·) is an ideal satisfying the assumption of Lemma 6.1. Thus  we get k1 = {0} and k0 = k.

6.1 The Trivial K-Type Proposition 6.3 A small K-type (π, V ) is trivial if and only if κπα = 0 for any α ∈ Σ.

(6.1)

Proof Note Ker k π := {X ∈ k | π(X ) = 0} is an ideal of k (and in particular, it is a subalgebra). If we assume (6.1), then Ker k π = k by Proposition 3.8 and Corollary 6.2, showing π is trivial. The converse is clear from Proposition 3.8.  The result on the trivial K-type stated in Sect. 2.1 readily follows from (6.1) and Proposition 5.9.

6.2 Complex Simple Lie Groups Let G be a complex simple Lie group and (π, V ) a small K-type of G. Then for any α ∈ Σ we have mα = 2, and hence κπα = 0 by Corollary 5.7. Thus by Proposition 6.3 (π, V ) is the trivial K-type. Moreover the right-hand side of (5.1) equals Ωa . From [22, Chap. IV, Sect. 5, No. 2] we have Υ

π

(φπλ )



sgn(w)ewλ α∈Σ + ρ(Hα )

w∈W = . wρ α∈Σ + λ(Hα ) w∈W sgn(w)e

Put Σ π = cΣ and kπ ≡ 1 with any c > 0. Then one easily has for any λ ∈ a∗C  + F(Σ , k , λ) = α∈Σ π

π

α∈Σ +

cρ (Hα ) 2

λ(Hα )



w∈W w∈W

sgn(w)ewλ cρ

sgn(w)ew 2

,



cρ 1 ρ(Hα ) w∈W sgn(w)ew 2 ˜ π , kπ ) 21 =  α∈Σ + cρ ˜δ − 2 δ(Σ

. G/K wρ α∈Σ + 2 (Hα ) w∈W sgn(w)e Hence (1.6) holds for infinitely many combinations of Σ π and kπ .

Spherical Functions for Small K-Types

151

6.3 Other Simple Lie Groups Having No Non-trivial Small K-Type We have no non-trivial small K-type for those real simple Lie groups G with the following Lie algebras: g sl(p, H) (p ≥ 2) so(2r + 1, 1) (r ≥ 1) e6(−26) (E IV) Σ Ap−1 A1 A2 mα 4 2r 8 sp(p) so(2r + 1) f4 k (so(3, 1) sl(2, C), sl(2, H) so(5, 1)) f4(−20) (F II) g Σ {±α, ±2α} ((BC)1 ) 8 mα 7 m2α so(9) k The argument for the first three cases is the same as for the complex case. Suppose g = f4(−20) and (π, V ) is a small K-type of G. Let α is a short restricted root. Then it follows from Corollary 5.7 and Proposition 3.8 that Xα + θXα ∈ Ker k π \ {0} for any Xα ∈ gα \ {0}. Now since Ker k π is an ideal of the simple Lie algebra k so(9), one has Ker k π = k and hence (π, V ) is the trivial K-type.

6.4 The Case g = sp(p, q) Suppose G = Sp(p, q) (p ≥ q ≥ 1) and K = Sp(p) × Sp(q). Then G is connected, simply-connected Lie group and M = M0 (cf. [24, Appendix C, Sect. 3]). Let H = R + Ri + Rj + Rk be the field of quaternions. We use the following realization:  

 A B  A ∈ gl(p, H), C ∈ gl(q, H), t A = −A, t C = −C , ∈ gl(p + q, H)  tB C

   A Op,q  t A = −A, t C = −C sp(p) ⊕ sp(q), A ∈ gl(p, H), C ∈ gl(q, H), k= Oq,p C  ⎫ ⎧ ⎞ ⎛ Oq,p−q diag(a1 , . . . , aq )  Oq,q ⎬ ⎨ ⎠  a = (a1 , . . . , aq ) ∈ Rq , Op−q,q Op−q,p−q Op−q,q a = H (a) := ⎝  ⎭ ⎩  diag(a1 , . . . , aq ) Oq,p−q Oq,q ⎫ ⎧⎛  ⎞  m1 , . . . , mq ∈ H, ⎪ ⎪ Oq,q  ⎬ ⎨ diag(m1 , . . . , mq ) Oq,p−q  m + m = 0 (1 ≤ i ≤ q), ⎠ ⎝ Op−q,q Y Op−q,q m= i  i  ⎪ ⎪ ⎩ Oq,q Oq,p−q diag(m1 , . . . , mq )  Y ∈ gl(p − q, H), t Y = −Y ⎭ 

g=

su(2)q ⊕ sp(p − q).

152

H. Oda and N. Shimeno

Define ei ∈ a∗ by ei (H (a)) = ai (i = 1, . . . , q). Then Σ ⊂ {±ei , ±2ei | 1 ≤ i ≤ q} ∪ {±ei ± ej | 1 ≤ i < j ≤ q} and the multiplicity of each restricted root is as follows: sp(p, q) (p ≥ q ≥ 1) g real rank q 4(p − q) mshort := m±ei (1 ≤ i ≤ q) 4 (q ≥ 2) mmiddle := m±ei ±ej (1 ≤ i < j ≤ q) mlong := m±2ei (1 ≤ i ≤ q) 3 Let pr 1 and pr 2 be the projections of K to Sp(p) and Sp(q) respectively. Theorem 6.4 If p ≥ q ≥ 2 then G = Sp(p, q) has no non-trivial small K-type. Suppose p ≥ q = 1. Then for the irreducible representation (πn , Cn ) of Sp(1) SU(2) πn ◦pr 2 = of dimension n = 1, 2, . . . , πn ◦ pr 2 is a small K-type of G = Sp(p, 1) with κshort 2 πn ◦pr 2 n −1 0 and κlong = − 3 . If p > q then all the small K-types are constructed in this way. If p = q = 1 then the other small K-types are constructed in the same way as above but using pr 1 instead of pr 2 and κπn ◦pr1 = κπn ◦pr2 for any n = 1, 2, . . . . Proof Suppose first p ≥ q ≥ 2. Then we have a restricted root vector ⎛

Xe1 −e2

0 −1 ⎜ 1 0 ⎜ ⎜ ⎜ Op−2,2 ⎜ =⎜ ⎜ 0 −1 ⎜ ⎜ −1 0 ⎜ ⎝ Oq−2,2

O2,p−2 Op−2,p−2 O2,p−2 Oq−2,p−2

⎞ 0 −1 O2,q−2 ⎟ −1 0 ⎟ ⎟ Op−2,2 Op−2,q−2 ⎟ ⎟ ⎟ ∈ ge −e . 1 2 ⎟ 0 −1 O2,q−2 ⎟ ⎟ 1 0 ⎟ ⎠ Oq−2,2 Oq−2,q−2

Observe that Xe1 −e2 + θXe1 −e2 belongs to neither sp(p) nor sp(q). Thus there is no proper ideal of k that contains Xe1 −e2 + θXe1 −e2 . Now, for any small K-type (π, V ) of G, Xe1 −e2 + θXe1 −e2 ∈ Ker k π by Corollary 5.7 and Proposition 3.8. This means Ker k π = k and hence (π, V ) is the trivial K-type. Next suppose p > q = 1 Then πn ◦ pr 2 is small since pr 2 (M ) = Sp(1). Also, πn ◦pr 2 πn ◦pr 2 = 0 by Corollary 5.7. To calculate κlong take a root vector κshort

X2e1

⎛ ⎞ i O1,p−1 −i 1⎝ Op−1,1 Op−1,p−1 Op−1,1 ⎠ ∈ g2e1 , = 2 i O1,p−1 −i

which is normalized as in Lemma 3.6. Under √ √ c + d√ −1 b −1 √ ∈ su(2), sp(1)  bi + cj + d k → −c + d −1 −b −1

Spherical Functions for Small K-Types

153

√

− −1 √0 pr 2 (X2e1 + θX2e1 ) maps to . Hence πn ◦ pr 2 (X2e1 + θX2e1 ) −1 0 √ 2 πn ◦pr 2 = − n 3−1 . −1 diag(n − 1, n − 3, . . . , −n + 1) and from Lemma 3.6 one has κlong Take any Xe1 ∈ ge1 \ {0}. We have Xe1 + θXe1 ∈ Ker k (πn ◦ pr 2 ) by Corollary 5.7 and Proposition 3.8. Since Ker k (πn ◦ pr 2 ) = sp(p) for n ≥ 2, we see Xe1 + θXe1 ∈ sp(p) and sp(p) is generated by Xe1 + θXe1 as an ideal of k. Now, for any small K-type (π, V ) of G, Xe1 + θXe1 ∈ Ker k π by the same reason as above. This means sp(p) ⊂ Ker k π and π equals some πn ◦ pr 2 . Finally suppose p = q = 1. Then M = SU(2) is diagonally embedded to K = SU(2) × SU(2). Since any K-type is given as the exterior tensor product πm  πn of two irreducible representations of SU(2), its restriction to M equals the interior tensor product πm ⊗ πn . By the representation theory of SU(2), πm ⊗ πn is irreducible if and only if either πm or πn is trivial. Hence {πn ◦ pr i | i = 1, 2, n = 1, 2, . . .} is the complete set of small K-types. The values of κπn ◦pri are calculated in the same way as in the previous case.  The result of Sect. 2.2 follows from this theorem and what we discussed in Sects. 6.2–6.4. Also, Theorem 6.4 and Proposition 5.9 easily imply the result of Sect. 2.3.

6.5 The Case g = so(p, q) Suppose g = so(p, q) (p ≥ q ≥ 1). (We exclude the cases so(1, 1) R and so(2, 2)

sl(2, R)⊕2 .) Under the natural inclusion so(p, q) ⊂ sp(p, q), k, a and m are identified with the intersections of so(p, q) and those for sp(p, q). In particular, k = so(p) ⊕ so(q) and ⎫ ⎧⎛ ⎞ ⎬ ⎨ Oq,q Oq,p−q Oq,q  m = ⎝Op−q,q Y Op−q,q ⎠  Y ∈ so(p − q) so(p − q). ⎭ ⎩ Oq,q Oq,p−q Oq,q  One has Σ ⊂ {±ei | 1 ≤ i ≤ q} ∪ {±ei ± ej | 1 ≤ i < j ≤ q} and the multiplicity of each restricted root is as follows: so(p, q) (p ≥ q ≥ 1) g real rank q p−q mshort := m±ei (1 ≤ i ≤ q) 1 (q ≥ 2) mlong := m±ei ±ej (1 ≤ i < j ≤ q) Taking some finite covering group of G if necessary, we may assume K = K1 × K2 with k1 := Lie K1 so(p) and k2 := Lie K2 so(q). Furthermore, if ki so(r) with r ≥ 3, then we may assume Ki Spin(r) (i = 1, 2). The projections K → Ki and k → ki are denoted by pr i (i = 1, 2).

154

H. Oda and N. Shimeno

Theorem 6.5 ([28, Theorem 1, Lemmas 4.2, 4.3]) (i) Suppose p = q ≥ 3. Then a non-trivial K-type π is small if and only if it is equivalent to σ ◦ pr i with i = 1, 2 and a (half-)spin representation σ of Ki . For such π, κπshort = 0 and κπlong = − 14 . (ii) Suppose p = q + 1 with odd q ≥ 3. Then there are three non-trivial small K-types: π = σ ◦ pr 1 with either of two half-spin representations σ of K1 = Spin(p) and π = σ ◦ pr 2 with the spin representation σ of K2 = Spin(q). One has κπshort = −1, κπlong = − 41 in the former case and κπshort = 0, κπlong = − 14 in the latter case. (iii) Suppose p = q + 1 with even q ≥ 4. Then a non-trivial K-type π is small if and only if it is equivalent to σ ◦ pr 2 with a half-spin representation σ of K2 = Spin(q). For such π, κπshort = 0 and κπlong = − 41 . We generalize this result to all cases in the subsequent two theorems. Theorem 6.6 Suppose p is even, p ≥ 4 and q = 1. Then K = K1 = Spin(p). Fix a Cartan subalgebra and a system of positive roots of kC . For s = 0, 1, 2, . . . , let πs± be the irreducible representation of K = Spin(p) with highest weight (s/2, . . . , s/2, ± πs± = − s(s+p−2) . s/2) in the standard notation. Then πs± is a small K-type with κshort p−1 There are no other small K-types. Remark 6.7 We already studied so(2r + 1, 1) (r ≥ 1) in Sect. 6.3 and so(4, 1) sp(1, 1) in Theorem 6.4. Also, so(2, 1) sl(2, R) su(1, 1) will be covered in Sect. 6.6. Let Eij denote a matrix whose entry is 1 in the (i, j)-position and 0 elsewhere. Let Fij = Eij − Eji . Proof of Theorem 6.6. Suppose first π is small. Then by Proposition 5.6 there is only one isotypic component in the restriction of π to m = so(p − 1). In view of the branching law for so(p) ↓ so(p − 1) (cf. [15, Theorem 8.1.4]), it is possible only when π is equivalent to some πs± in the theorem. Conversely, let π = πs± with representation space V . Then π is small since π|M0 is irreducible by the branching law. To calculate κπshort we take restricted root vectors X (i) := Fi,1 − Ei,p+1 − Ep+1,i (2 ≤ i ≤ p) for e1 ∈ Σ. They constitute an orthonormal basis of ge1 , so that κπshort id V = 2  (i) √ 1 p + θX (i) . Now, we assume H1 , . . . , H p2 with Hi := −1F2i,2i−1 i=2 π X p−1 (1 ≤ i ≤ 2p ) constitute a basis of the Cartan subalgebra of kC and Δ+ k := {εi ± εj | 1 ≤ i < j ≤ 2p } is the system of positive roots, where we let {εi } be the dual basis of {Hi }. For i = 2, 3, . . . , 2p take root vectors √ √ 1 (F2i−1,1 + −1F2i−1,2 + −1F2i,1 − F2i,2 ) ∈ (kC )ε1 +εi , 2 √ √ 1 := (F2i−1,1 + −1F2i−1,2 − −1F2i,1 + F2i,2 ) ∈ (kC )ε1 −εi . 2

Xε1 +εi := Xε1 −εi

Spherical Functions for Small K-Types

155

Then one has for i = 2, 3, . . . , 2p [Xε1 +εi , Xε1 −εi ] = [Xε1 +εi , Xε1 −εi ] = [Xε1 +εi , Xε1 −εi ] = [Xε1 +εi , Xε1 −εi ] = 0, [Xε1 +εi , Xε1 +εi ] = −(H1 + Hi ), X

(2i−1)

X

+ θX

(2i)

(2i−1)

+ θX

(2i)

[Xε1 −εi , Xε1 −εi ] = −(H1 − Hi ),

= Xε1 +εi + Xε1 −εi + Xε1 +εi + Xε1 −εi , √ = − −1(Xε1 +εi − Xε1 −εi − Xε1 +εi + Xε1 −εi ).

From these we calculate in U (kC )

p 2   √  (2i−1)  (i) 2 2  2 X X + θX (i) = (−2 −1H1 )2 + + θX (2i−1) + X (2i) + θX (2i) p

i=2

i=2

≡

−4H12

− 2(p − 2)H1

mod U



 (kC )−α + U (kC ) (kC )α .

α∈Δ+ k

α∈Δ+ k

Applying this to the highest weight vector of V , we obtain (p − 1)κπshort = −s(s + p − 2).  One easily has the result of Sect. 2.4 by Theorem 6.6 and Proposition 5.9. Theorem 6.8 (i) Suppose p > q = 2. Then a K-type π is small if and only if it is equivalent to τ ◦ pr 2 with a one-dimensional representation τ of K2 . For such π, κπshort = 0. (ii) Suppose p is even and q is odd with p > q ≥ 3. Then one has the same results as in the case of Theorem 6.5 (ii). (iii) Suppose p > q ≥ 3 and either q or p − q is even. Then a non-trivial K-type π is small if and only if it is equivalent to σ ◦ pr 2 with a (half-)spin representation σ of K2 = Spin(q). For such π, κπshort = 0 and κπlong = − 14 . Proof Choosing a Cartan subalgebra b ⊂ m suitably, we may assume gsplit

 ⎧⎛ ⎫ ⎞  B A Oq,p−q ⎨  A, C ∈ so(q), ⎬ = ⎝Op−q,q Op−q,p−q Op−q,q ⎠ ∈ gl(p + q, R) 

so(q, q) ⎩ tB  B ∈ Mat(q, q, R)⎭ Oq,p−q C

if p − q is even, and gsplit

⎧⎛  ⎫ ⎞  A ∈ so(q + 1), ⎪ ⎪ A Oq+1,p−q−1 B ⎨  ⎬  = ⎝Op−q−1,q+1 Op−q−1,p−q−1 Op−q−1,q ⎠ ∈ gl(p + q, R)  B ∈ Mat(q + 1, q, R), ⎪  ⎪ tB ⎩ ⎭ Oq,p−q−1 C  C ∈ so(q)

so(q + 1, q)

if p − q is odd. Suppose p ≥ 3. Let (τ , V ) be a non-trivial irreducible representation of K2 and consider the K-type (π, V ) := (τ ◦ pr 2 , V ). If q = 2 then this is always small since V

156

H. Oda and N. Shimeno

is one-dimensional. For q ≥ 3, we assert that (π, V ) is small if and only if τ is a (half)spin representation σ of K2 = Spin(q) and that κπlong = − 41 for such (π, V ). Indeed, if τ = σ, then (σ ◦ pr 2 |Ksplit , V ) is a small K-type by Theorem 6.5 and (σ ◦ pr 2 |Msplit , V ) is irreducible. Since Msplit ⊂ M , (π, V ) is small. Conversely, if (π, V ) = (τ ◦ pr 2 , V ) is small, then Vμ in Proposition 5.6 equals V since m ⊂ k1 acts on V trivially. Hence (τ ◦ pr 2 |Ksplit , V ) is a small Ksplit -type and is non-trivial since pr 2 (Ksplit ) = pr 2 (K). It then follows from Theorem 6.5 that τ is a (half)-spin representation. We also have κπlong = − 41 by Corollary 5.8. Next, note p > q in all cases of the theorem and we have a restricted root vector Xe1 := Fq+1,1 − Eq+1,p+1 − Ep+1,q+1 ∈ ge1 ,

(6.2)

for which Xe1 + θXe1 ∈ k1 \ {0}. It is easy to check there is no proper ideal of k1 = so(p) that contains Xe1 + θXe1 . (Note so(p) is simple for p = 3, 5, 6, . . . .) Hence by Proposition 3.8, a small K-type (π, V ) is written as (π, V ) = (τ ◦ pr 2 , V ) for some irreducible representation τ of K2 if and only if κπshort = κπe1 = 0. It follows from Corollary 5.7 that if p − q is even then all small K-type are of this type. We claim the same thing holds if q is even. To show this, we may assume p is odd. If (p, q) = (3, 2), then so(3, 2) sp(2, R) and the claim in this case will be shown in the first paragraph of the proof of Theorem 6.11. Suppose (p, q) is a general combination of an odd p and an even q (p > q ≥ 2) and let (π, V ) be any small K-type. Let Vμ be as in Proposition 5.6. Since (π|Ksplit , Vμ ) is a small Ksplit -type, it follows from Theorem 6.5 (iii) or the claim for (p, q) = (3, 2) that k1 ∩ ksplit = so(q + 1) acts trivially on Vμ . Note that F2,1 ∈ k1 ∩ ksplit commutes with mC and that V = π(U (mC ))Vμ by Proposition 5.6. Thus F2,1 acts trivially on V . By the simplicity of k1 = so(p), we have k1 ⊂ Ker k π, which proves our claim. Note that (i) and (iii) are already proved up to this point. In order to show (ii), suppose p is even, q is odd and p > q ≥ 3. Thanks to Theorem 6.5 (ii), we may also assume p − q ≥ 3. Let (π, V ) is a small K-type such that π|K1 is non-trivial. We assert that π|K2 is trivial. In fact, if k1 ∩ ksplit = so(q + 1) acts trivially on Vμ in Proposition 5.6, then the same argument as in the last paragraph implies π|K1 is trivial, a contradiction. Thus the action of k1 ∩ ksplit = so(q + 1) on Vμ is non-trivial and hence that of k2 ∩ ksplit = k2 = so(q) is trivial by Theorem 6.5 (ii). (We also see κπlong = − 41 by Corollary 5.8.) Since V = π(U (mC ))Vμ and since k2 commutes with mC , k2 acts trivially on V , proving the assertion. Therefore there exists a non-trivial irreducible representation τ of k1 = so(p) such that π = τ ◦ pr 1 . Now, by Proposition 5.6 there is only one isotypic component in the restriction of (τ , V ) to m = so(p − q). In view of the branching laws for the orthogonal Lie algebras (cf. [15, Theorems 8.1.3, 8.1.4]), it is possible only when τ is a half-spin representation. Conversely, let (σ, V ) be any of two half-spin representations of K1 = Spin(p). The proof is complete if we can show π = σ ◦ pr 1 is small and κπshort = −1. Let  : Spin(p) → SO(p) be the canonical projection. Then

Spherical Functions for Small K-Types

pr 1 (M ) = 

−1

157

() '

 q  diag(m1 , . . . , mq ) Oq,p−q  mi = 1, g ∈ SO(p − q)  mi = ±1, Op−q,q g  i=1

q p  + *    ⊃ −1 diag(m1 , . . . , mp )  mi = ±1, mi = mi = 1 , i=1

pr 1 (M ) ∪ pr 1 (M ) · 

i=q+1

−1

(diag(−1, . . . , −1)) p  + *   mi = 1 . ⊃ −1 diag(m1 , . . . , mp )  mi = ±1, i=1

 *  Now V is irreducible under the action of −1 diag(m1 , . . . , mp )  mi = ±1, + p . (In fact, (σ, V ) is a small ‘K-type’ of the double cover of SL(p, R) i=1 mi = 1 by Theorem 2.1.) But since −1 (diag(−1, . . . , −1)) is contained in the center of Spin(p), V is irreducible as a pr 1 (M )-module. This proves the smallness of π = σ ◦ pr 1 . Finally, we can directly check Xe1√in (6.2) is normalized as in Lemma  3.6 and the eigenvalues of π(Xe1 + θXe1 ) are ± −1. Thus κπshort = −1. All the results stated in Sect. 2.5 follow from Theorem 6.8 (ii), (iii) and Proposition 5.9.

6.6 The Hermitian Type Let G be a non-compact real simple Lie group of Hermitian type. There exists a central element Z ∈ k such that J = ad(Z) is a complex structure of s = g−θ . Let 2e1 , . . . 2el be the longest roots in Σ + . Then Σ ⊂ {±ei , ±2ei | 1 ≤ i ≤ l} ∪ {±ei ± ej | 1 ≤ i < j ≤ l}. Put mlong = m±2ei (1 ≤ i ≤ l), mmiddle = m±ei ±ej (1 ≤ i < j ≤ l) and mshort = m±ei (1 ≤ i ≤ l). Their values are listed below: g su(p, p) sp(p, R) so∗ (4p) (p ≥ 2) so(p, 2) (p ≥ 3) e7(−25) (E VII) real rank l p p p 2 3 0 0 0 0 0 mshort 4 p−2 8 mmiddle 2 (p ≥ 2) 1 (p ≥ 2) 1 1 1 1 1 mlong (su(1, 1) sp(1, R) sl(2, R), su(2, 2) so(4, 2), sp(2, R) so(3, 2), so∗ (8) so(6, 2)) su(p, q) (p > q ≥ 1) so∗ (4p + 2) (p ≥ 1) e6(−14) (E III) g real rank l q p 2 2(p − q) 4 8 mshort 2 (q ≥ 2) 4 (p ≥ 2) 6 mmiddle 1 1 1 mlong (su(3, 1) so∗ (6))

158

H. Oda and N. Shimeno

Take X2ei ∈ g2ei so that − ||2e2i || B(X2ei , θX2ei ) = 1 (1 ≤ i ≤ l). 2

Lemma 6.9 By replacing X2ei with −X2ei if necessary, we have 1 (X2ei + θX2ei ) + Y 2 i=1 l

Z=

(6.3)

for some Y ∈ b. Here b is a Cartan subalgebra of m.

Proof It is well known that t := li=1 R(X2ei + θX2ei ) + b is a Cartan subalgebra of k. Since Z ∈ t, there exist constants c1 , . . . , cl ∈ R and Y ∈ b such that Z = c1 (X2e1 + θX2e1 ) + · · · + cl (X2el + θX2el ) + Y . Since 

     X2ei , X2ej = X2ei , θX2ej = H2ei , X2ej = 0 (i = j),

    X2ei , θX2ei = − ||2e2i ||2 H2ei , H2ei , X2ei = ||2ei ||2 X2ei , we have for i = 1, . . . , l −H2ei = J 2 H2ei = ad(Z)2 H2ei = −4ci2 Hαi − ci ||2ei ||2 ([Y , X2ei ] − [Y , θX2ei ]) and hence [Y , X2ej ] = [Y , θX2ej ] = 0 and ci = ± 21 .



Corollary 6.10 One has g±2ei ⊂ gM for i = 1, . . . , l. Proof This is clear from (6.3) since Ad(m)Z = Z for any m ∈ M .



Theorem 6.11 Suppose (π, V ) is a small K-type. Then V is one-dimensional and κπshort = κπmiddle = 0.

p Proof First we assume g = sp(p, R). Then m = {0} and t = i=1 R(X2ei + θX2ei ) is a Cartan subalgebra of k. Since t ⊂ kM by Corollary 6.10, M is a finite subgroup of the Cartan subgroup corresponding to t and in particular is Abelian. Thus any small K-type π is one-dimensional. We claim κπmiddle = 0 for such π. Indeed, [k, k] equals the orthogonal complement (RZ)⊥ of RZ in k with respect to B(·, ·). Since  α:middle

{Xα + θXα | Xα ∈ gα } ⊥

p 

R(X2ei + θX2ei ),

i=1

one sees by Lemma 6.9 that {Xα + θXα | Xα ∈ gα } ⊂ [k, k] for any middle α. Since [k, k] ⊂ Ker k π, our claim follows from Proposition 3.8. Next, suppose g is a general simple Lie algebra of Hermitian type and (π, V ) is a small K-type. We claim κπshort = κπmiddle = 0. This was shown in the previous

Spherical Functions for Small K-Types

159

paragraph for g = sp(p, R) and in Theorem 6.8 (i) for g = so(p, 2). For the remaining cases, the claim follows from Corollary 5.7. Now, by Proposition 3.8, π(Xα + θXα ) = 0 for any restricted root vector Xα of any α ∈ Σ with short or middle length. On the other hand, by Corollary 6.10, each π(X2ei + θX2ei ) is a scalar operator. Hence by Corollary 6.2, π(X ) for any X ∈ k is a scalar operator. This proves V is onedimensional.  √ Let z and π0 ∈ −1z∗ be as in Sect. 2.6. If we identify π0 with one-dimensional representation of k, then it follows from [19, Proposition 5.3.2] that √ π0 (X2ei + θX2ei ) ∈ {± −1} for i = 1, . . . , l. Hence if (the differentiation of) a small K-type π is written as π = νπ0 for some ν ∈ Q, then κπlong = −ν 2 . This and Proposition 5.9 imply the result stated in Sect. 2.6.

6.7 The Case Σ is of Type F4 Let G be a simply-connected real simple Lie group with Σ of type F4 . As in Sect. 2.7, we exclude the complex simple Lie group of type F4 . Thus g is one of f4(4) , e6(2) , e7(−5) and e8(−24) . Among these f4(4) is of split type. Let Σshort and Σlong be as in Sect. 2.7. From [1] one sees there exists a sequence of embeddings f4(4) ⊂ e6(2) ⊂ e7(−5) ⊂ e8(−24) .

(6.4)

The following table summarizes some necessary data on these Lie algebras: f4(4) (F I) e6(2) (E II) e7(−5) (E VI) e8(−24) (E IX) g mshort 1 2 4 8 1 1 1 1 mlong k sp(3) ⊕ su(2) su(6) ⊕ su(2) so(12) ⊕ su(2) e7 ⊕ su(2) Thus in any case K is the product of two simple compact groups. Let K = K1 × K2 with K2 = SU(2). Let pr i : K → Ki be the projection (i = 1, 2). Theorem 6.12 ([28, Theorem 1, Lemmas 4.2, 4.3]) Suppose g = f4(4) . Let (σ, C2 ) be the irreducible representation of SU(2) of dimension 2. Then π = σ ◦ pr 2 is the only non-trivial small K-type. Moreover, κπshort = 0 and κπlong = − 41 . We generalize this to Theorem 6.13 The last theorem also holds for g = e6(2) , e7(−5) and e8(−24) . π Proof Suppose g = f4(4) and let π = σ

◦ pr 2 be as in Theorem 6.12. Since κshort = 0, it follows from Proposition 3.8 that α∈Σshort {Xα + θXα | Xα ∈ gα } ⊂ Ker k π = k1 .

160

H. Oda and N. Shimeno



Since α∈Σshort {Xα + θXα | Xα ∈ gα } is the orthogonal complement of α∈Σlong {Xα + θXα | Xα ∈ gα } in k with respect to B(·, ·) and k1 is that of k2 , one has 

k2 ⊂

{Xα + θXα | Xα ∈ gα }.

α∈Σlong

Now let g be one of e6(2) , e7(−5) and e8(−24) . We use with the obvious meanings the notation G  , K  , k , gα , and so on. Fix an embedding G → G  so that k ∩ g = k and a = a. We claim k2 su(2) is an ideal of k . Indeed, since gα ( = gα ) commutes with m for each α ∈ Σlong , 

,



[m , k2 ] ⊂ m ,



{Xα + θXα | Xα ∈ gα } = {0}.

(6.5)

α∈Σlong

This proves our claim since m and k generate the Lie algebra k by Theorem 3.7. Thus k2 = k2 su(2) and K2 = K2 . Since k = (k1 ∩ k) ⊕ k2 is a decomposition of k into two ideals, we have k1 = k1 ∩ k and K1 ⊂ K1 . Hence π extends to a K  -type π  = σ ◦ pr 2 . This is small since M ⊂ M  . Since gα ⊂ gα for any α ∈ Σ, we have  κπ = κπ by Lemma 3.6. Conversely, let ν be any non-trivial small K  -type. Then it follows from Corollary

5.7 and Proposition 3.8 that α∈Σshort {Xα + θXα | Xα ∈ gα } ⊂ Ker k ν ∩ k1 . By the simplicity of k1 , k1 ⊂ Ker k ν and there exists a non-trivial irreducible representation τ of K2 = SU(2) such that ν = τ ◦ pr 2 . Now we claim pr 2 (M  ) = pr 2 (M ). Indeed, since  gα ⊂ g, gsplit = a + α∈Σlong  ⊂ M . On the other hand, since (6.5) implies m ⊂ we have G split ⊂ G and Msplit      ) = pr 2 (Msplit ) ⊂ pr 2 (M ) = k1 , one has M0 ⊂ K1 . Hence pr 2 (M  ) = pr 2 (M0 Msplit pr 2 (M ). Since the opposite inclusion is obvious, we get the claim. Thus ν|K = τ ◦ pr 2 is a small K-type. This is non-trivial since dim τ > 1. By Theorem 6.12 we conclude τ = σ. 

These two theorems and Proposition 5.9 imply the result of Sect. 2.7.

6.8 Split Lie Groups with Simply-Laced Σ Let G be one of the simply-connected split real simple Lie groups of type Al (l ≥ 2), Dl (l ≥ 3) and El (l = 6, 7, 8). Let (π, V ) be a small K-types listed in Theorem 2.1. Then it follows from [28, Lemma 4.2] that κπα = − 41 for any α ∈ Σ. (For the type Dl case, we already know this by Theorem 6.5 (i).) Thus Proposition 5.9 implies the result of Sect. 2.8.

Spherical Functions for Small K-Types

161

6.9 The Split Lie Group of Type G 2 Let G = G˜ 2 , the simply-connected split real simple Lie group of type G 2 . We use the same notation as in Sect. 2.9. Let Σshort  Σlong be the division of Σ according to the root lengths. From [28, Lemmas 4.2, 4.3] the values of κπ1 and κπ2 are as follows: π

π1

π2

κπshort κπlong

− 41 − 41

− 94 − 14

Hence if π = π1 then we have the same result as for the split simply-laced case. Suppose π = π2 and let us prove we cannot find any combination of Σ π and kπ for which (1.6) holds. To do so, assume (1.6) holds for some (Σ π , kπ ). Then −1 ˜ π , kπ ) 21 is non-singular at 0 and hence for each α ∈ Σ there exists β ∈ Σ π δ˜ 2 δ(Σ G/K

1

2 which is proportional to α. This implies Σ π is of type G 2 . Now δ˜G/K Υ π (φπλ ) = 1 π π π π ˜ δ(Σ , k ) 2 F(Σ , k , λ) ∈ A (areg ) is an eigenfunction of both (5.1) and (4.2) with (Σ  , k) = (Σ π , kπ ). Hence

 α∈Σshort ∩Σ +

||α||2 16



32 + − sinh2 α2 sinh2 α 4



||α||2

α∈Σlong ∩Σ +

=

16 sinh2 α2

 kπα (1 − kπα − 2kπ )||α||2 2α α∈Σ π+

4 sinh2 α2

+C

for some constant C. Since the members of {sinh−2 α2 | α ∈ Σ ∪ 2Σshort ∪ Σ π } ∪ {1} are linearly independent in A (areg ), we have kπα (1 − kπα − 2kπ2α ) = 0 for each α ∈ Σ ∪ 2Σshort . Hence Σ π ⊃ Σ ∪ 2Σshort , a contradiction. The results in Sect. 2.9 are thus proved.

7 Spherical Transforms Let G be a non-compact real simple Lie group with finite center and (π, V ) a small K-type. In this section we apply our main formula (1.6) to the calculation of HarishChandra’s c-function for G ×K V and the theory of π-spherical transform. We note each combination of Σ π and kπ in Sect. 2 is chosen so that Σ π ⊂ Σ ∪ 2Σ.

(7.1)

162

H. Oda and N. Shimeno

7.1 Weight Functions ˜ π , kπ ) defined by (1.4) and (1.5) are normalized The weight functions δ˜G/K and δ(Σ 1 − ˜ π , kπ ) 21 in (1.6) takes 1 at 0 ∈ a. In the literature, so that δ˜ 2 δ(Σ G/K

δG/K =



|2 sinh α|

 π    α 2kα  and δ(Σ , k ) = 2 sinh 2  π+ π

mα

α∈Σ +

π

α∈Σ

are often used. Lemma 7.1 Suppose (1.6) and (7.1) are valid for Σ π and kπ . Then −2 ˜ π , kπ ) 21 = 2e(Σ π ,kπ ) δ − 2 δ(Σ π , kπ ) 21 δ˜G/K δ(Σ G/K 1

1

(7.2)

with π



π

e(Σ , k ) =

α∈Σ + \2Σ



m2α π π kα − k4α + . 2 +

(7.3)

Proof By a calculation similar to the one for (5.5) we have −1



2 δG/K δ(Σ π , kπ ) 2 = 1

2 cosh

α∈Σ + \2Σ +

α 2

−kπα

π

(2 cosh α)k4α −

The lemma then follows from this and (5.5).

m2α 2

. 

7.2 Harish-Chandra’s c-Function Let a+ := {H ∈ a | α(H ) > 0 for any α ∈ Σ + } and a∗+ = {λ ∈ a∗ | λ(α∨ ) > 0 for √ any α ∈ Σ + }. For λ ∈ a∗+ + −1a∗ put π



c (λ) =

N¯

e−(λ+ρ)(H (¯n)) π(κ(¯n))d n¯

(7.4)

where the Haar measure d n¯ on N¯ := θN is normalized so that  e−2ρ(H (¯n)) d n¯ = 1. N¯

The integral in (7.4) absolutely converges and defines an EndM V -valued holomorphic function known as Harish-Chandra’s c-function. This satisfies for any H ∈ a+

Spherical Functions for Small K-Types

and λ ∈ a∗+ +

√

163

−1a∗ lim et(−λ+ρ)(H ) φπλ (etH ) = cπ (λ).

(7.5)

t→∞

These things are shown using the integral formula (3.7) in the same way as in the case of the trivial K-type (cf. [22, Chap. IV, Sect. 6, No.6]), or are deduced as a special case of the asymptotic behavior of Eisenstein integrals (cf. [41, Theorem 9.1.6.1], [23, Theorem 14.7, (14.29)]). It is known that cπ (λ) extends to a meromorphic function on a∗C . We regard cπ (λ) as a C-valued function by EndM V C. Theorem 7.2 Suppose (1.6) and (7.1) are valid for Σ π and kπ . With e(Σ π , kπ ) in (7.3) we have π π (7.6) cπ (λ) = 2e(Σ ,k ) c(Σ π , kπ , λ). (Recall c(Σ π , kπ , λ) is defined by (4.5).) Proof Note that −1

2 lim etρ(H ) δG/K (tH ) = lim e−tρ(k

t→∞

t→∞

π

)(H )

δ(Σ π , kπ ; tH ) 2 = 1 1

and that e−λ+ρ Υ π (φπλ ) = 2e(Σ

π

,kπ )

 ρ − 21  −ρ(kπ ) π 1 e δG/K e δ(Σ π , kπ ) 2 e−λ+ρ(k ) F(Σ π , kπ , λ). 

Hence (7.6) follows from (7.5) and Corollary 4.10.

7.3 The π-Spherical Transform Let Cc∞ (G, π, π) be the subspace of C ∞ (G, π, π) consisting of the compactly supported π-spherical functions. For φ1 ∈ C ∞ (G, π, π) and φ2 ∈ Cc∞ (G, π, π) define the convolution φ1 ∗ φ2 ∈ C ∞ (G, π, π) by 

φ1 (g −1 x)φ2 (g)d g,

(φ1 ∗ φ2 )(x) = G

where d g is a Haar measure on G. Since (π, V ) is a small, Cc∞ (G, π, π) is a commutative algebra by [7, Theorem 3]. For φ ∈ Cc∞ (G, π, π) we define its π-spherical transform by  ˆ φπ (g −1 )φ(g)d g = (φπ ∗ φ)(1G ), (7.7) φ(λ) = G

λ

λ

which is, by (3.1), a holomorphic function on a∗C taking values in EndK V C. For each λ ∈ a∗C

164

H. Oda and N. Shimeno

ˆ Cc∞ (G, π, π)  φ → φ(λ) ∈C π ˆ is an algebra homomorphism since one has φπλ ∗ φ = φ(λ)φ λ by Theorem 1.4. Now we normalize the Haar measure dH on a so that for any compactly supported continuous K-bi-invariant function ψ on G   1 ψ(g)d g = ψ(eH ) δG/K (H )dH #W a G

(cf. [22, Ch. I, Theorem 5.8], [13, Proposition 2.4.6]). Since the trace of the integrand of (7.7) is K-bi-invariant, we have using (3.4)  1 ˆ Υ π (φπλ )(−H ) Υ π (φ)(H ) δG/K (H )dH φ(λ) = #W a  1 = Υ π (φπ−λ )(H ) Υ π (φ)(H ) δG/K (H )dH . #W a Hence by Theorem 1.5 the π-spherical transform is identified with the integral transform  1 f (H ) Υ π (φπ−λ )(H ) δG/K (H )dH (7.8) f → fˆ (λ) := #W a for f ∈ Cc∞ (a)W . Here Cc∞ (a)W = {f ∈ C ∞ (a) | f with compact support}. Let Σ  be a root system in a∗ and k a multiplicity function on Σ  . For f ∈ Cc∞ (a)W we define its hypergeometric Fourier transform F = F(Σ  , k) by Ff (λ) :=

1 #W



f (H ) F(Σ  , k, −λ; H ) δ(Σ  , k; H )dH .

(7.9)

a

This makes sense when δ(Σ  , k) is locally integrable. Remark 7.3 The hypergeometric Fourier transform is introduced by Opdam [32] as the Cherednik transform. If Σ  = 2Σ and k2α = mα /2, then F(Σ  , k) equals (7.8) for the trivial K-type π, that is, the Harish-Chandra transform (cf. [13, Chap. 6], [22, Ch. IV], [41, Chap. 9]). If Σ  has real rank one, then F(Σ  , k) reduces to the Jacobi transform (cf. [25]). Theorem 7.4 Suppose (1.6) and (7.1) are valid for Σ π and kπ . Then δ(Σ π , kπ ) is locally integrable and it holds with F = F(Σ π , kπ ) and e(Σ π , kπ ) in (7.3) that  21 1 π π fˆ = 2e(Σ ,k ) F f δG/K δ(Σ π , kπ )− 2 for any f ∈ Cc∞ (a)W .

(7.10)

Proof The local integrability follows from (5.4), while (7.10) is direct from (1.6), (7.2), (7.8) to (7.9). 

Spherical Functions for Small K-Types

165

7.4 Inversion Formulas and Plancherel Formulas √ We normalize the Haar measure d λ on −1a∗ so that the Euclidean Fourier transform and its inversion are given by f˜ (λ) =





f (H )e−λ(H ) dH ,

f (H ) =

a

√ −1a∗

f˜ (λ)eλ(H ) d λ.

On the hypergeometric Fourier transform we have Theorem 7.5 ([32, 33]) Let Σ  be a root system in a∗ and k a real-valued multiplicity function on Σ  such that kα ≥ 0 for any α ∈ Σ  . Let F = F(Σ  , k). Then for any f ∈ Cc∞ (a)W we have the inversion formula 1 f (H ) = #W

 √ −1a∗

Ff (λ) F(Σ  , k, λ; H ) |c(Σ  , k, λ)|−2 d λ.

and the Plancherel-type formula 1 #W



|f (H )|2 δ(Σ  , k; H )dH = a

1 #W

 √ −1a∗

|Ff (λ)|2 |c(Σ  , k, λ)|−2 d λ.

Moreover F uniquely extends to the isometry √ 1 1 ∼ δ(Σ  , k; H )dH )W − |c(Σ  , k, λ)|−2 d λ)W . L2 (a, #W → L2 ( −1a∗ , #W From this the following result on the π-spherical transform is deduced: Corollary 7.6 Suppose (1.6) is valid for Σ π and kπ such that kπα ≥ 0 for any α ∈ Σ π . Then for f ∈ Cc∞ (a)W we have the inversion formula f (H ) =

1 #W

 √ −1a∗

fˆ (λ) Υ π (φλ )(H ) |cπ (λ)|−2 d λ

and the Plancherel-type formula 1 #W

 |f a

(H )|2 δ˜G/K (H )dH

1 = #W

 √ −1a∗

|fˆ (λ)|2 |cπ (λ)|−2 d λ.

Moreover the π-spherical transform f → fˆ uniquely extends to the isometry √ 1 ˜ 1 ∼ |cπ (λ)|−2 d λ)W . L2 (a, #W δG/K (H )dH )W − → L2 ( −1a∗ , #W

(7.11)

166

H. Oda and N. Shimeno

Proof Under (7.1) all the statements are immediate from (1.6), (7.2), (7.6), (7.10) to Theorem 7.5. So suppose (7.1) is not valid. Even in this case, one easily sees that δ(Σ π , kπ ) is locally integrable and that Formulas (7.2), (7.6) and (7.10) hold by π π replacing 2e(Σ ,k ) with some constant. Hence the corollary follows.  As we can see from the list in Sect. 2, there exists at least a pair of Σ π and kπ satisfying (1.6) and (7.11) unless (π, V ) is one of the following: (1) (2) (3) (4) (5)

(π, V ) for g = sp(p, 1) (p ≥ 1); (πs± , V ) in Sect. 2.4 for g = so(2r, 1) (r ≥ 2); (π, V ) in Sect. 2.5 2.5 for g = so(p, q) (p > q ≥ 3, p : even, q : odd); (π, V ) for a Hermitian G; (π2 , C2 ) in Theorem 2.2 for G˜ 2 .

For (5), the elementary π-spherical functions cannot be expressed by Heckman– Opdam hypergeometric functions. The harmonic analysis in this exceptional case is therefore to be studied separately. But we do not go further with it in this paper. For (1)–(4), every Σ π chosen in Sect. 2 is of type BC. Suppose first that G has real rank one and let Σ π = {±α, ±2α}. Then the inversion formula and the Planchereltype formula for the Jacobi transform F are available under the assumption 1 kπα , kπ2α ∈ R and kπα + kπ2α > − , 2 which is much weaker than that of Theorem 7.5 (cf. [12, Appendix 1], [25]). From this we can deduce a result on the π-spherical transform for (1), (2), and (4) with g = su(p, 1) corresponding to Corollary 7.6. In fact, such a result is shown by [8, π 37] for (1), by [3] for √ (2) ∗with s = 1, and by [12] for (4) with g = su(p, 1). If c (λ) ∗ has zeros in a+ + −1a then the inversion and Plancherel-type formulas contain discrete spectra in addition to the same continuous spectrum as in the formulas of Corollary 7.6. Next, let us consider (4) for G with higher rank. If the parameter ν of the onedimensional K-type π given in Sect. 2.6 satisfies 2|ν| ≤ max{m α2 , 1} for α ∈ Σlong , then we can apply Theorem 7.5 to deduce Corollary 7.6. The general inversion and Plancherel-type formulas are given by [19, Chap. 5] for |ν| < 21 m α2 (α ∈ Σlong ) and by [36] for an arbitrary ν. Both [19, 36] employ Rosenberg’s method [35] for the classical Harish-Chandra transform. If |ν| is sufficiently large, the Plancherel measure contains spectra with lower-dimensional support along with the most continuous spectrum in Corollary 7.6. Possible spectra with√lower-dimensional support are obtained by calculating residues of cπ (λ)−1 in a∗+ + −1a∗ (see [36] for details). Finally suppose (π, V ) is (3). In the notation of Sect. 2.5 one has from Theorem 7.2 that

Spherical Functions for Small K-Types

cπ (λ) = C

q  i=1

·

167

Γ (λ((2ei )∨ ) + 21 )

· Γ (λ((2ei )∨ ) + 21 (p − q + 1))  Γ (λ((ei − ej )∨ ))Γ (λ((ei + ej )∨ )) 1≤i 0. Then, v1 := v/r is an element of S(V ). Since G acts transitively on S(V ), we write v1 = g · v0 for some g ∈ G. Thus, we obtain v = g · r v0 , which is an element of G · (R≥0 v0 ).  √ Next, let VC = V + −1V diagonally, namely, √ G act on the complexification √ g · (v1 + −1v2 ) := gv1 + −1gv2 for g ∈ G, v1 , v2 ∈ V . We denote by (Rv0 )⊥ the orthogonal complement of Rv0 in V . Then, we have: Lemma 2.4 Retain the setting as in Lemma 2.3. Suppose that the isotropy subgroup G v0 of G at v0 acts transitively on the unit sphere S((Rv0 )⊥ ). Then, VC is expressed as VC = G · (R≥0 v0 +

√ −1(Rv0 ⊕ Rw0 ))

176

A. Sasaki

for an element w0 ∈ S((Rv0 )⊥ ). Proof Our proof is based on [14, Lemma 5.2]. Let us see it briefly. Applying Lemma 2.3 to the case where G v0 acts transitively on S((Rv0 )⊥ ), the vector space (Rv0 )⊥ is written as (Rv0 )⊥ = G v0 · R≥0 w0 for an element w0 ∈ S((Rv0 )⊥ ). On the other hand, it is clear that G v0 · Rv0 = Rv0 . As V = Rv0 ⊕ (Rv0 )⊥ , we obtain V = G v0 · Rv0 ⊕ G v0 · R≥0 w0 = G v0 · (Rv0 ⊕ R≥0 w0 ). Combining the above decomposition with Lemma 2.3, we conclude VC = V +

√ √ −1V = G · R≥0 v0 + −1(G v0 · (Rv0 ⊕ R≥0 w0 )) √ = G · (R≥0 v0 + −1(Rv0 ⊕ R≥0 w0 )).

In particular, since R≥0 w0 is contained in Rw0 , we obtain √ −1(Rv0 ⊕ R≥0 w0 )) √ ⊂ G · (R≥0 v0 + −1(Rv0 ⊕ Rw0 )) ⊂ VC .

VC = G · (R≥0 v0 +

Hence, Lemma 2.4 has been proved.



3 Cartan Decomposition for (G 2 (C), SL(3, C)) In this section, we give a proof of Theorem 1.1 for a non-symmetric reductive spherical pair (G 2 (C), S L(3, C)) (Type R-2). We begin this section with the outline of our proof. As mentioned in Lemma 2.2, the homogeneous space G 2 (C)/S L(3, C) is biholomorphic to the complex unit sphere S(Im(CC )). Then, we first find a real submanifold T1 which meets every G 2 orbit in S(Im(CC )) (Sect. 3.1). Second, we give an abelian group A1 such that T1 is an A1 -orbit (Sect. 3.2). After that, we prove Theorem 1.1 for this case (see Theorem 3.6 for detail).

3.1 G 2 -Action on S(Im(CC )) First, we give a decomposition of S(Im(CC )) into G 2 -orbits. We set T1 : = (R≥0 e1 +

√ −1(Re1 ⊕ Re2 )) ∩ S(Im(CC )).

Lemma 3.1 The complex unit sphere S(Im(CC )) is written as

(3.1)

A Cartan Decomposition for Non-symmetric Reductive Spherical …

177

S(Im(CC )) = G 2 · T1 . Proof Retain the notation as in Sect. 2.4. We observe that G 2 acts transitively on S(Im(C)) and the isotropy subgroup (G 2 )e1 = SU (3) acts transitively on S(W ) (see Lemma 2.1). By Lemma 2.4, we have Im(CC ) = G 2 · (R≥0 e1 +

√ −1(Re1 + Re2 )).

Since G 2 is a subgroup of S O(7), we obtain S(Im(CC )) = (G 2 · (R≥0 e1 + = G 2 · ((R≥0 e1 +

√ √

−1(Re1 + Re2 ))) ∩ S(Im(CC )) −1(Re1 + Re2 ))) ∩ S(Im(CC ))

= G 2 · T1 . 

Hence, Lemma 3.1 has been proved.

Next, we consider an explicit description√of an element of T1 in the coordinates. Let √ v be an element of T1 . As T1 ⊂ Re1 + −1(Re1 + Re2 ), we write v = x1 e1 + −1(y1 e1 + y2 e2 ) for some x1 ∈ R≥0 and some y1 , y2 ∈ R. Then, we have (v, v) = (x1 +

√

√ √ −1y1 )2 + ( −1y2 )2 = (x12 − y12 − y22 ) + 2 −1x1 y1 .

Since v ∈ S(Im(CC )), three real numbers x1 , y1 , y2 satisfy x12 − y12 − y22 = 1 and x1 y1 = 0. Hence, we get y1 = 0 and x12 − y22 = 1. In particular, x1 has to be a positive number. Therefore, T1 is of the form T1 = {(cosh θ)e1 +

√ −1(sinh θ)e2 : θ ∈ R}.

√ Here, the map R → T1 , θ → (cosh θ)e1 + −1(sinh θ)e2 is an embedding. Then, T1 is a one-dimensional real submanifold in S(Im(CC )).

3.2 G 2 -Action on G 2 (C)/SL(3, C) We recall from Lemma 2.2 that S(Im(CC )) is biholomorphic to G 2 (C)/S L(3, C). As S(Im(CC )) = G 2 · T1 , there exists a real submanifold S1 in G C /HC such that T1  S1 and G C /HC = G u · S1 via the biholomorphic diffeomorphism. To find S1 , we construct an abelian group A1 as follows. Let us define a matrix δ(x,y) by

178

A. Sasaki

√ ⎞ 0 0 − −1x √0 ⎟ ⎜ 0 0 ⎟. √ 0 − −1y =⎜ ⎠ ⎝ 0 −1y 0 0 √ −1x 0 0 0 ⎛

δ(x,y)

Now, we set a1 = {τθ := diag(δ(0,θ) , δ(−θ/2,−θ/2) ) : θ ∈ R}

(3.2)

A1 = exp a1 = {tθ = diag(exp δ(0,θ) , exp δ(−θ/2,−θ/2) ) : θ ∈ R}.

(3.3)

and

Then, A1 is a one-dimensional abelian group. We note √ ⎞ cosh x 0 − −1 sinh x √ 0 ⎟ ⎜ 0 0 ⎟. √ cosh y − −1 sinh y =⎜ ⎠ ⎝ −1 sinh y cosh y 0 √ 0 −1 sinh x 0 0 cosh x ⎛

exp δ(x,y)

Lemma 3.2 The abelian group A1 is contained in G 2 (C). Sketch of Proof Let us verify that any element tθ ∈ A1 satisfies (tθ ei )(tθ e j ) = tθ (ei e j ) (0 ≤ i, j ≤ 7) for our choice of the C-basis {e0 , . . . , e7 } in (2.1). In fact, the computation is straightforward from the followings: √ tθ e1 = (cosh θ)e1 + −1(sinh θ)e2 , √ tθ e2 = − −1(sinh θ)e1 + (cosh θ)e2 , √ tθ e4 = (cosh(θ/2))e4 − −1(sinh(θ/2))e7 , √ tθ e5 = (cosh(θ/2))e5 − −1(sinh(θ/2))e6 , √ tθ e6 = −1(sinh(θ/2))e5 + (cosh(θ/2))e6 , √ tθ e7 = −1(sinh(θ/2))e4 + (cosh(θ/2))e7 and tθ ei = ei for i = 0, 3.



As mentioned in the proof of Lemma 3.2, an element (cosh θ)e1 + ∈ T1 is written by tθ e1 . Then, the submanifold T1 is expressed as T1 = {tθ e1 : θ ∈ R} = A1 · e1 .

√ −1(sinh θ)e2

(3.4)

A Cartan Decomposition for Non-symmetric Reductive Spherical …

179

Hence, we set S1 := A1 S L(3, C)/S L(3, C).

(3.5)

Then, we have: Lemma 3.3 S1  T1 . Combining Lemma 3.1 with Lemma 3.3, we get the decomposition of the homogeneous space G 2 (C)/S L(3, C) as follows: Proposition 3.4 G 2 (C)/S L(3, C) = G 2 · S1 . Proof Let g be an element of G 2 (C). By Lemma 3.1, the element ge1 ∈ S(Im(CC )) is written as ge1 = k · v1 for some k ∈ G u and v1 ∈ T1 (see (3.4)). Moreover, v1 is given by v1 = tθ e1 ∈ A1 · e1 for some tθ ∈ A1 , from which ge1 = (ktθ )e1 . This means g −1 ktθ ∈ (G 2 (C))e1 = S L(3, C). Hence, we obtain gS L(3, C) = k · tθ S L(3, C) ∈  G 2 · S1 .

3.3 Lie Algebra a1 In this subsection, we observe the Lie algebra a1 . Let g = g2 (C), h = sl(3, C) and gu = g2 be the Lie algebras of G 2 (C), S L(3, C) and G 2 , respectively. The differential automorphism of the Cartan involution θ of G 2 (C) (see (2.3)), which we use the same letter to denote, is given by θ(X ) = X (X ∈ g). Since δ(x,y) = δ(−x,−y) =√−δ(x,y) , we have θ(τθ ) = τ(−θ) = −τθ for any τθ ∈ a1 . Hence, a1 is contained in −1gu . Next, let q be the orthogonal complement of h in g with respect to the Killing form on g. As S L(3, C) = (G 2 (C))e1 , we write h = {X ∈ g2 (C) : X e1 = 0}. Thus, a1 is not contained in h. In fact, we have: Lemma 3.5 a1 ⊂ q. Proof We will give a C-bases of h = sl(3, C) and q as follows, which is based on [22, Theorem 1.5.1]. Let E i j (0 ≤ i, j ≤ 7, i = j) be a C-linear transformation on CC satisfying E i j e j = ei and E i j ek = 0 for k = j. We shall also use the same letter E i j to denote the matrix of degree eight corresponding to the C-linear transformation E i j via the identification EndC (CC )  M(8, C) with respect to the C-basis {e0 , . . . , e7 }. For 0 ≤ i < j ≤ 7, we set X i j := E i j − E ji . Then, {−X 23 + X 45 , − X 45 + X 67 , X 24 + X 35 , − X 25 + X 34 , X 26 + X 37 , − X 27 + X 36 , X 46 + X 57 , − X 47 + X 56 } is a C-basis of sl(3, C), and

(3.6)

180

A. Sasaki

{2X 12 − X 47 − X 56 , 2X 13 − X 46 + X 57 , 2X 14 + X 27 + X 36 ,

(3.7)

2X 15 + X 26 − X 37 , 2X 16 − X 25 − X 34 , 2X 17 − X 24 + X 35 } √ √ is a C-basis of q. Since 2X 12 − X 47 − X 56 ∈ q is written by −1τ2 = 2 −1τ1 , an arbitrary τθ is of the form τθ = θτ1 ∈ q. Thus, the Lie algebra a1 is contained in q.  √ Therefore, we have shown a1 ⊂ −1gu ∩ q.

3.4 Proof of Theorem 1.1 for (G 2 (C), SL(3, C)) A Cartan decomposition for the reductive spherical pair (G 2 (C), S L(3, C)) is provided by Proposition 3.4. More precisely, we prove: Theorem 3.6 (Theorem 1.1 for Type R-2) Let (G C , HC ) be the reductive spherical pair of Type R-2, namely, (G 2√(C), S L(3, C)). Then, the one-dimensional abelian group A1 = exp a1 with a1 ⊂ −1gu ∩ q given by (3.2) satisfies G C = G u A1 HC . Proof Let g be an element of G 2 (C). By Proposition 3.4, there exists k ∈ G u = G 2 and tθ ∈ A1 such that g HC = k · (tθ HC ) = (ktθ )HC . Thus, we have (ktθ )−1 g ∈ HC . We write h := (ktθ )−1 g ∈ HC . Then, we obtain g = ktθ h ∈ G u A1 HC . Hence, we conclude G C ⊂ G u A1 HC . Clearly, G C ⊃ G u A1 HC . Therefore, we get G C =  G u A1 HC .

4 Cartan Decomposition for (Spi n(7, C), G 2 (C)) In this subsection, we give a proof of Theorem 1.1 for the non-symmetric reductive spherical pair (Spin(7, C), G 2 (C)). The proof of Theorem 1.1 for Type R-1 proceeds in parallel with that for Type R-2 which has been discussed in Sect. 3.

4.1 Spi n(7)-Action on S(CC ) In this subsection, we consider the Spin(7)-action on the complex unit sphere S(CC ) (Type R-1 ). The Spin(7)-action on S 7 = S(C) is transitive and the action of the isotropy subgroup Spin(7)e0 = G 2 on S 6 = S(Im(C)) is also transitive (see Lemma √ 2.1). It follows from Lemma 2.4 that the complexified Cayley algebra CC = C + −1C is written as

A Cartan Decomposition for Non-symmetric Reductive Spherical …

CC = Spin(7) · (R≥0 e0 +

√

181

−1(Re0 + Re1 )).

Then, we obtain S(CC ) = Spin(7) · (R≥0 e0 +

√

−1(Re0 + Re1 )) ∩ S(CC ).

0 in S(CC ) as Hence, we take a one-dimensional real submanifold T 0 : = (R≥0 e0 + T

√ −1(Re0 + Re1 )) ∩ S(CC ).

(4.1)

Then, we have: Lemma 4.1 The complex unit sphere S(CC ) is expressed as 0 . S(CC ) = Spin(7) · T

4.2 Spi n(7)-Action on Spi n(7, C)/ G 2 (C) In this subsection, we give a real submanifold which meets every Spin(7)-orbit in Spin(7, C)/G 2 (C). Let us take a matrix dθ as dθ :=

√

√ cosh θ − −1 sinh θ . −1 sinh θ cosh θ

(4.2)

0 of S O(8, C) by We define a subgroup A 0 := { aθ = diag(dθ , d(−θ/3) , d(−θ/3) , d(−θ/3) ) : θ ∈ R} A

(4.3)

0 is a subgroup of Spin(7, C). Lemma 4.2 The set A For the verification of Lemma 4.2, we prepare the notation as follows: ax := diag(I2 , dx , dx , dx ) ∈ S O(7, C).

(4.4)

0 , we take a(2θ/3) ∈ S O(7, C). Then, the Sketch of Proof For an element  aθ of A direct computation shows that aθ e j ) =  aθ (ei e j ) (0 ≤ i, j ≤ 7) (a(2θ/3) ei )( aθ ∈ Spin(7, C). for the C-basis {e0 , . . . , e7 } of CC . This implies that 

(4.5) 

182

A. Sasaki

0 is of the form By taking the same argument as for T1 , the real submanifold T 0 = {(cosh θ)e0 + T

√

−1(sinh θ)e1 : θ ∈ R}.

Thus, we write 0 · e0 . 0 = A T Hence, we set 0 G 2 (C)/G 2 (C).  S0 := A

(4.6)

0 . Lemma 4.3  S0  T Therefore, we get a decomposition of Spin(7, C)/G 2 (C) as follows: Proposition 4.4 Spin(7, C)/G 2 (C) = Spin(7) ·  S0 .

0 4.3 Lie Algebra of A Let g = spin(7, C), h = g2 (C) and gu = spin(7) be the Lie algebras of Spin(7, C), G 2 (C) and Spin(7), respectively, and q be the orthogonal complement of h in g with respect to the Killing √ form on g. In this subsection, we show that the Lie algebra  a0  of A0 is contained in −1gu ∩ q. The Lie algebra  a0 is given as follows. Let δθ be a matrix given by δθ :=

√

√ 0 − −1θ . −1θ 0

(4.7)

Then,  a0 is given by  a0 = { αθ = diag(δθ , δ(−θ/3) , δ(−θ/3) , δ(−θ/3) ) : θ ∈ R}.

(4.8)

0 = exp In particular, we have A a0 . We choose a Cartan involution θ of g given by θ(X ) = X (X ∈ g) (see Sect. 2.3). αθ ) = − αθ for any  αθ ∈  a0 because δθ = δ(−θ) = −δθ . This Then, we have√θ( αθ ) = ( implies  a0 ⊂ −1gu . Next, we show: Lemma 4.5  a0 ⊂ q. Sketch of Proof As mentioned in Sect. 3.3, the Lie algebra g2 (C) is decomposed into the direct sum of sl(3, C) and the orthogonal complement of sl(3, C) in g2 (C), denoted here by q . Moreover, we take a C-basis of sl(3, C) as in (3.6) and that of

A Cartan Decomposition for Non-symmetric Reductive Spherical …

183

q as in (3.7). It turns out that  a0 is orthogonal to both sl(3, C) and q , and then to a0 ⊂ q.  g2 (C). Hence, we obtain  √ Consequently, we have proved  a0 ⊂ −1gu ∩ q.

4.4 Proof of Theorem 1.1 for (Spi n(7, C), G 2 (C)) In this subsection, we will prove Theorem 1.1 for (Spin(7, C), G 2 (C)). Let G u := Spin(7) be a maximal compact subgroup of G C . Theorem 4.6 (Theorem 1.1 for Type R-1 ) Let (G C , HC ) be the reductive spherical pair of Type R-1 , namely, (Spin(7, √ C), G 2 (C)). Then, the one-dimensional abelian 0 = exp a0 with  a0 ⊂ −1gu ∩ q given by (4.8) satisfies group A 0 HC . GC = Gu A Proof The proof of Theorem 4.6 is the same as the proof of Theorem 3.6. Then, we omit its proof. 

5 Cartan Decomposition for (S O(7, C), G 2 (C)) In this section, we give a Cartan decomposition for the non-symmetric reductive spherical pair (G C , HC ) = (S O(7, C), G 2 (C)) (Type R-1). The key idea is to take the image of our Cartan decomposition for (Spin(7, C), G 2 (C)) given by Theorem 4.6 through the double covering group homomorphism π (see (2.2)). To carry out, we define a subgroup A0 of S O(7, C) = (S O(8, C))e0 by A0 := {aθ = diag(I2 , dθ , dθ , dθ ) : θ ∈ R}.

(5.1)

Here, an element aθ has already appeared in the proof of Lemma 4.2 and dθ is given by (4.2). Then, the Lie algebra a0 of A0 is of the form a0 = {αθ = diag(O2 , δθ , δθ , δθ ) : θ ∈ R}

(5.2)

where δθ is given by (4.7). In particular, we have A0 = exp a0 . Let g = so(7, C), h = g2 (C) and gu = so(7) be the Lie algebras of G C , HC and , respectively and q the orthogonal complement of h in g. Clearly, a0 is contained G u√ in −1g √ u . By the same argument as Lemma 4.5, we have a0 ⊂ q. Thus, we obtain a0 ⊂ −1gu ∩ q. 0 → Now, we return to the relation (4.5). This implies that π induces the map π : A aθ ) = a(2θ/3) . A0 given by π(

184

A. Sasaki

0 ) coincides with A0 . Lemma 5.1 The image π( A 0 satisfies π( Proof For any aθ ∈ A0 , the element  a(3θ /2) ∈ A a(3θ /2) ) = aθ . Hence,   we have proved π( A0 ) = A0 . Theorem 5.2 (Theorem 1.1 for Type R-1) Let (G C , HC ) be the reductive spherical pair of Type R-1, namely, (S O(7, √ C), G 2 (C)). Then, the one-dimensional abelian group A0 = exp a0 with a0 ⊂ −1gu ∩ q given by (5.2) satisfies G C = G u A0 HC . C ) of G C = Proof We observe that G C = S O(7, C) is realized as the image π(G Spin(7, C). It follows from Theorem 4.6 that u A u )π( A 0 )π(HC ). 0 HC ) = π(G C ) = π(G π(G

(5.3)

u ) = π(Spin(7)) coincides with S O(7), π(HC ) = π(G 2 (C)) Here, the image π(G 0 ) with A0 . Hence, (5.3) implies G C = G u A0 with G 2 (C), and by Lemma 5.1 π( A HC .  The following theorem is an immediate consequence of Theorem 1.1 or Proposition 4.4. Proposition 5.3 S O(7, C)/G 2 (C) = S O(7) · (A0 G 2 (C)/G 2 (C)). For the sake of our application given in the next section, we will explain that Proposition 5.3 follows from Proposition 4.4. The double covering group homomorphism π induces a double covering map  π : Spin(7, C)/G 2 (C) → S O(7, C)/G 2 (C), gG 2 (C) → π(g)G 2 (C).

(5.4)

In particular,  π (Spin(7, C)/G 2 (C)) coincides with S O(7, C)/G 2 (C). It follows from Proposition 4.4 that π (Spin(7, C)/G 2 (C)) S O(7, C)/G 2 (C) =  0 G 2 (C)/G 2 (C))) = π (Spin(7) · ( A 0 )G 2 (C)/G 2 (C)) = π(Spin(7)) · (π( A = S O(7) · (A0 G 2 (C)/G 2 (C)). Hence, we set S0 := A0 G 2 (C)/G 2 (C). Then, the above argument shows: Corollary 5.4 π( S0 ) = S0 .

(5.5)

A Cartan Decomposition for Non-symmetric Reductive Spherical …

185

6 Application to Visible Actions on Complex Manifolds The motivation of the study for a Cartan decomposition for non-symmetric reductive spherical pairs is to investigate a classification problem on strongly visible actions on reductive complex homogeneous spaces. The notion of (strongly) visible actions has been introduced by Kobayashi for giving an unified explanation for multiplicityfreeness of representations (cf. [7]). In this aspect, it plays a crucial role to find a real submanifold which meets every orbit. Once one can find a Cartan decomposition for a reductive spherical pair, one can also provide an explicit description of such a submanifold simultaneously. This section studies spherical homogeneous spaces of rank-one type from the viewpoint of (strongly) visible actions. Let us give a quick review on strongly visible actions. A holomorphic action of a Lie group G on a connected complex manifold D is called strongly visible if there exist a real submanifold S in D (called a slice) and an anti-holomorphic diffeomorphism σ on D satisfying the following conditions (see [7]): D = G · S,

(V.1)

σ| S = id S ,

(S.1)

σ(x) ∈ G · x (∀x ∈ D).

(S.2)

We note that the slice S is automatically totally real, namely, Jx (Tx S) ∩ Tx S = {0} for any x ∈ S (see [7, Remark 3.3.2]). Here, J stands for the complex structure of D. In Kobayashi’s original definition [7, Definition 3.3.1] of strongly visible actions, it allows a complex manifold D containing a non-empty G-invariant open set satisfying (V.1)–(S.2). For this paper, we shall adopt the above definition for simplicity. Now, we prove: Theorem 6.1 Let (G C , HC ) be a non-symmetric reductive spherical pair of rankone type. Then, the G u -action on D = G C /HC is strongly visible. In particular, one can find a one-dimensional slice S for the strongly visible action. In the following, we prove Theorem 6.1 for (G C , HC ) given in Table 1. More precisely, we will verify three conditions (V.1)–(S.2).

6.1 Verification of (V.1) We have already proved that there exists a one-dimensional real submanifold S := AHC /HC .

(6.1)

186

A. Sasaki

Table 2 Our choice of slice S satisfying (S.1) Type GC HC R-1 R-1 R-2

S O(7, C) Spin(7, C) G 2 (C)

G 2 (C) G 2 (C) S L(3, C)

S

D = Gu · S

S0  S0 S1

Proposition 5.3 Proposition 4.4 Proposition 3.4

satisfying D = G u · S, which implies the condition (V.1). We list our choice of S and the proposition showing D = G u · S in Table 2.

6.2 Verification of (S.1) In this subsection, we will verify the condition (S.1). First, let I1,1 := diag(1, −1) and I+− : = diag(I1,1 , I1,1 , I1,1 , I1,1 ) = diag(1, −1, 1, −1, 1, −1, 1, −1). Since (I+− ei )(I+− e j ) = I+− (ei e j ) for 0 ≤ i, j ≤ 7, the element I+− lies in G 2 . Here, we define an anti-holomorphic involution σ0 on S O(8, C) by σ0 (g) = I+− g I+− (g ∈ S O(8, C)).

(6.2)

Lemma 6.2 The involution σ0 stabilizes Spin(7, C), S O(7, C), G 2 (C) and S L(3, C). Proof As I+− ∈ G 2 , it is obvious that σ0 stabilizes Spin(7, C), S O(7, C) and G 2 (C). For the proof that S L(3, C) is σ0 -stable, it is necessary to verify the relation σ0 (g)e1 = e1 for any g ∈ S L(3, C). Let g be an element of S L(3, C). It is obvious that g ∈ S L(3, C). Thus, we obtain (σ0 (g))e1 = I+− g I+− (e1 ) = I+− g(−e1 ) = I+− (−e1 ) = e1 . Then, σ0 stabilizes S L(3, C).



By Lemma 6.2, the restrictions of σ0 to Spin(7, C), S O(7, C) and G 2 (C) becomes involutions on Spin(7, C), S O(7, C) and G 2 (C), respectively, which we use the same letter σ0 to denote. We choose a Cartan involution θ of S O(8, C) as in (2.3). Clearly, θ commutes with σ0 , from which σ0 stabilizes the maximal compact subgroup S O(8) = {g ∈ S O(8, C) : θ(g) = g} of S O(8, C). By definition, Spin(7), S O(7) and G 2 are also σ0 -stable. Let (G C , HC ) be a non-symmetric reductive spherical pair contained in Table 1. The anti-holomorphic involution σ0 on G C induces an anti-holomorphic diffeomorphism σ on the non-symmetric spherical homogeneous space G C /HC as follows:

A Cartan Decomposition for Non-symmetric Reductive Spherical …

187

σ(g HC ) := σ0 (g)HC (g ∈ G C ).

(6.3)

Now, let us show the condition (S.1) for our choice of σ in (6.3). The submanifold S in G C /HC given in (6.1) comes from the one-dimensional abelian subgroup in G C 0 (Type R-1 ), A1 (Type R-2). Then, it is necessary for the given by A0 (Type R-1), A verification of (S.1) to show the following: 0 , A1 . Lemma 6.3 σ0 | A = id A for A = A0 , A Proof First, let aθ = diag(I2 , dθ , dθ , dθ ) be an element of A0 . Then, the complex conjugation of aθ equals a(−θ) . Thus, we compute σ0 (aθ ) = I+− a(−θ) I+− = diag(I1,1 I2 I1,1 , I1,1 d(−θ) I1,1 , I1,1 d(−θ) I1,1 , I1,1 d(−θ) I1,1 ) = diag(I2 , dθ , dθ , dθ ) = aθ . Hence, σ0 | A0 = id A0 holds. 0 . Then, we Next, let  aθ = diag(dθ , d(−θ/3) , d(−θ/3) , d(−θ/3) ) be an element of A have aθ ) = diag(I1,1 d(−θ) I1,1 , I1,1 d(θ/3) I1,1 , I1,1 d(θ/3) I1,1 , I1,1 d(θ/3) I1,1 ) σ0 ( = diag(dθ , d(−θ/3) , d(−θ/3) , d(−θ/3) ) =  aθ . 0 . This implies that σ0 is the identity map on A Finally, let tθ = exp τθ = diag(exp δ(0,θ) , exp δ(−θ/2,−θ/2) ) be an element of A1 . We put J+− := diag(1, −1, 1, −1). Then, we have J+− δ(x,y) J+− = δ(−x,−y) , Hence, we obtain σ0 (tθ ) = I+− diag(exp δ(0,−θ) , exp δ(θ/2,θ/2) )I+− = diag(exp(J+− δ(0,−θ) J+− ), exp(J+− δ(θ/2,θ/2) J+− )) = diag(exp δ(0,θ) , exp δ(−θ/2,−θ/2) ) = tθ . Hence, σ0 | A1 = id A1 . Therefore, Lemma 6.3 has been proved.



Thanks to Lemma 6.3, the following equality holds for any a HC ∈ S = AHC /HC (a ∈ A): σ(a HC ) = σ0 (a)HC = a HC . Hence, we have verified σ| S = id S , namely, (S.1).

188

A. Sasaki

6.3 Verification of (S.2) In this subsection, we shall see that the condition (S.2) follows from (V.1) and (S.1) and the involution σ0 given by (6.2). Retain the setting as in Sect. 6.2. Let x be an element of the spherical homogeneous space G C /HC of rank-one type. By the condition (V.1), one can find elements k ∈ G u and a ∈ A such that x = k · a HC . As σ| S = id S (the condition (S.1)), we have σ(x) = σ0 (k) · σ(a HC ) = σ0 (k) · a HC = (σ0 (k)k −1 ) · x. Here, σ0 stabilizes G u (see Sect. 6.2). Then, σ0 (k)k −1 is an element of G u . Hence, (σ0 (k)k −1 ) · x lies in the G u -orbit through x, from which we have shown σ(x) ∈ G u · x. Hence, the condition (S.2) has been verified.

6.4 Proof of Theorem 6.1 For a non-symmetric reductive spherical pair (G C , HC ) of rank-one type, we have verified the condition (V.1) in Sect. 6.1, (S.1) in Sect. 6.2 and (S.2) in Sect. 6.3. Therefore, Theorem 6.1 has been proved.

6.5 Remark We end this paper by the observation of σ0 from the corresponding fixed point set of the Lie algebra as follows. Let (G C , HC ) be a non-symmetric spherical pair of rank-one type and g the Lie algebra of a complex simple Lie group G C . We use the same letter σ0 to denote the differential automorphism on g. We write gσ0 as the fixed point set of σ0 in g. Our choice of σ0 satisfies that (so(8, C))σ0 is isomorphic to so(4, 4). Then, its real rank, denoted by rank R (so(8, C))σ0 , equals four, which coincides with rank(so(8, C)). This means (so(8, C))σ0 is a normal real form of so(8, C). Further, we have (so(7, C))σ0  so(7, C) ∩ so(4, 4)  so(3, 4), (spin(7, C))σ0  spin(7, C) ∩ so(4, 4)  spin(3, 4), (g2 (C))σ0  g2 (C) ∩ so(4, 4)  g2(2) . It turns out that the Lie algebra gσ0 satisfies Proposition 6.4 rank R gσ0 = rank g.

A Cartan Decomposition for Non-symmetric Reductive Spherical …

189

We have found the same property as Proposition 6.4 in the non-symmetric spherical homogeneous spaces of line bundle case. Namely, we have prove that if G C /HC is a line bundle G C /[K C , K C ] over the complexification G C /K C of an irreducible Hermitian symmetric space G/K of non-tube type, then the G u -action on G C /[K C , K C ] is strongly visible and one can take a slice S and an anti-holomorphic diffeomorphism σ satisfying (V.1)–(S.2) and Proposition 6.4 (see [15, Lemma 2.2] and [18]). The key ingredient is to find a Cartan decomposition for (G C , [K C , K C ]) explicitly. We can show that for any reductive spherical pair (G C , HC ) we have a Cartan decomposition by giving an explicit description of the abelian part and that the G u -action on the spherical homogeneous space G C /HC is strongly visible with a slice coming from a Cartan decomposition for (G C , HC ) and an anti-holomorphic diffeomorphism coming from an involution on G C satisfying Proposition 6.4. In fact, we have shown the strong visibility in some cases, see [15–18], in particular, we have provided a slice explicitly (note that our choice of slice in [16, 17] is not abelian). The others will be explained in the forthcoming papers which contain how to find an explicit description of the abelian part for a Cartan decomposition (cf. [19]).

References 1. Borel, A.: Le plan projectif des octaves et les sphères comme espaces homogènes. C. R. Acad. Sci. Paris 230, 1378–1380 (1950) 2. Brion, M.: Classification des espaces homogènes sphériques. Compos. Math. 63, 189–208 (1987) 3. Danielsen, T., Krötz, B., Schlichtkrull, H.: Decomposition theorems for triple spaces. Geom. Dedicata 174, 145–154 (2015) 4. Flensted-Jensen, M.: Spherical functions of a real semisimple Lie group. A method of reduction to the complex case. J. Funct. Anal. 30, 106–146 (1978) 5. Helgason, S.: Differential geometry, Lie groups, and symmetric spaces. Graduate Studies in Mathematics, vol. 34. American Mathematical Society, Providence, RI (2001) 6. Knapp, A.W.: Lie groups beyond an introduction. Progress in Mathematics, vol. 140. Birkhäuser Boston Inc., Boston (2002) 7. Kobayashi, T.: Multiplicity-free representations and visible actions on complex manifolds. Publ. Res. Inst. Math. Sci. 41, 497–549 (2005) (special issue commemorating the fortieth anniversary of the founding of RIMS) 8. Kobayashi, T., Oshima, T.: Finite multiplicity theorems for induction and restriction. Adv. Math. 248, 921–944 (2013) 9. Krämer, M.: Sphärische Untergruppen in kompakten zusammenhängenden Liegruppen. Compos. Math. 38, 129–153 (1979) 10. Montgomery, D., Samelson, H.: Transformation groups of spheres. Ann. Math. 44, 454–470 (1943) 11. Murakami, S.: Exceptional simple Lie groups and related topics in recent differential geometry. Differential Geometry and Topology (Tianjin, 1986–87). Lecture Notes in Mathematics, vol. 1369, pp. 183–221. Springer, Berlin (1989) 12. Mykytyuk, I.V.: Integrability of invariant Hamiltonian systems with homogeneous configuration spaces. Mat. Sb. (N. S.) 129(171), 514–534 (1986) 13. Rossmann, W.: The structure of semisimple symmetric spaces. Can. J. Math. 31, 157–180 (1979)

190

A. Sasaki

14. Sasaki, A.: Visible actions on irreducible multiplicity-free spaces. Int. Math. Res. Not. IMRN 2009, 3445–3466 (2009) 15. Sasaki, A.: A characterization of non-tube type Hermitian symmetric spaces by visible actions. Geom. Dedicata 145, 151–158 (2010) 16. Sasaki, A.: A generalized Cartan decomposition for the double coset space SU (2n + 1)\S L(2n + 1, C)/Sp(n, C). J. Math. Sci. Univ. Tokyo 17, 201–215 (2010) 17. Sasaki, A.: Visible actions on the non-symmetric homogeneous space S O(8, C)/G 2 (C). Adv. Pure Appl. Math. 2, 437–450 (2011) 18. Sasaki, A.: Admissible representations, multiplicity-free representations and visible actions on non-tube type Hermitian symmetric spaces. Proc. Jpn. Acad. Ser. A 91, 70–75 (2015) 19. Sasaki, A.: A Cartan decomposition for spherical homogeneous spaces of Cayley type (in preparation) 20. Vinberg, É.B., Kimel’fel’d, B.N.: Homogeneous domains on flag manifolds and spherical subsets of semisimple Lie groups. Funct. Anal. Appl. 12, 168–174 (1978) 21. Wolf, J.: Harmonic analysis on commutative spaces. Mathematical Surveys and Monographs, vol. 142. American Mathematical Society, Providence, RI (2007) 22. Yokota, I.: Exceptional Lie group. arXiv: 0902.0431

Lagrangian Submanifolds of Standard Multisymplectic Manifolds Gabriel Sevestre and Tilmann Wurzbacher

Abstract We give a detailed, self-contained proof of Geoffrey Martin’s normal form theorem for Lagrangian submanifolds of standard multisymplectic manifolds (that generalises Alan Weinstein’s famous normal form theorem in symplectic geometry), providing also complete proofs for the necessary results in foliated differential topology, i.e., a foliated tubular neighborhood theorem and a foliated relative Poincaré lemma. Keywords Multisymplectic geometry · Lagrangian submanifolds · Foliated differential topology MSC (2010) Primary: 53D05 · 53D12 · Secondary: 53C12

1 Introduction It is well-known that Lagrangian submanifolds play a central role in symplectic geometry. This can easily be traced back to the search for so-called “generating functions” of (local) symplectomorphisms in the framework of the Hamilton–Jacobi method for integrating Hamilton’s equation (see the classical reference [1], Sects. 47–48). This method is closely connected to the observations that the graph of a diffeomorphism between two symplectic manifolds is Lagrangian if and only if the diffeomorphism is symplectic and that the image of a one-form is Lagrangian (inside the cotangent bundle) if and only if the form is closed. Alan Weinstein deduced from such classical facts his famous symplectic creed: “Everything is a Lagrangian submanifold”. Of course, from a modern perspective the main argument for this creed is ... Weinstein’s fundamental result from 1971 (see [11]). G. Sevestre (B) · T. Wurzbacher Institut Élie Cartan Lorraine, Université de Lorraine et C.N.R.S., 57000 Metz, France e-mail: [email protected] T. Wurzbacher e-mail: [email protected] © Springer Nature Switzerland AG 2019 A. Baklouti and T. Nomura (eds.), Geometric and Harmonic Analysis on Homogeneous Spaces, Springer Proceedings in Mathematics & Statistics 290, https://doi.org/10.1007/978-3-030-26562-5_8

191

192

G. Sevestre and T. Wurzbacher

Weinstein’s normal form theorem Let L be a Lagrangian submanifold of a symplectic manifold (M, ω). Then there exist open neighborhoods U and V of L in M respectively T ∗ L, and a diffeomorphism φ : U → V such that φ| L = id L and ∗ φ∗ (ω T L ) = ω on U . Classical mechanics is geometrized by the Hamiltonian approach on cotangent bundles and more generally on symplectic manifolds, whereas its higher dimensional analogue, classical field theory, can be formulated in a Hamiltonian way on multicotangent or jet bundles, and leads more generally to multisymplectic manifolds (cf. [10], Sect. 2 for a recent account of this). A multisymplectic manifold is a manifold together with a nondegenerate, closed (k+1)-form ω with k in N; k = 1 being the symplectic case. In a 1988 article ([8]) Geoffrey Martin extended Weinstein’s result to an important class of multisymplectic manifolds including multicotangent bundles. (Note that he reserves the term “multisymplectic” for the class of multisymplectic manifolds where his theorem applies.) The proof of his main result (Lemma 2.1) being rather cryptic, and in parts being reduced to mere hints for the reader, his precocious results fell into oblivion, not receiving the deserved attention. The spanish school on differential-geometric methods in mathematical physics revived multisymplectic geometry (in its modern definition) at the end of the last century, and Manuel de Léon, David Martín de Diego and Aitor Santamaría-Merino gave in [4] a rather detailed framework for Martin’s normal form theorem. Unfortunately, the necessary condition that a certain naturally associated subbundle of the tangent bundle of the ambient manifold should be integrable is not emphasised in their proof of Martin’s main result (see the proof of Lemma 3.24 in the cited article). Since multisymplectic geometry is by now emerging fast as the “right” (higher) geometric formulation of classical field theory, thanks to the advent of rather wellsuited homotopical and homological methods, the interest in Martin’s result is growing and we felt compelled to give a self-contained, detailed account of his result and techniques. It turns out that one crucially needs “folkloristic” extensions of two standard theorems in differential topology to a foliated setting (these being of independent interest, in fact). Once established, Martin’s ingenious idea that the path method of Jürgen Moser (see [9]) applies though a multisymplectic form of degree k+1 does not yield an isomorphism between the tangent bundle and the bundle of k-forms, goes through and yields the following result: Martin’s normal form theorem (Theorem 1 below) Let (M, ω) be a standard k-plectic manifold, with k > 1. Let the distribution W ⊂ T M be defined as W := ∪ p∈M Wω ( p) and let L be a k-Lagrangian submanifold of M complementary to W (that is T p M = T p L ⊕ W | p , ∀ p ∈ L). If W is integrable, there exist open neighborhoods U and V of L in M and k (T ∗ L), and a diffeomorphism φ : U → V such that: k ∗ φ| L = id L and φ∗ (ω  (T L) ) = ω on U. Results of a related but more global nature were obtained by Frans Cantrijn, Alberto Ibort and Manuel de Léon in 1999 (see Theorem 7.3 in [3]) and Michael

Lagrangian Submanifolds of Standard Multisymplectic Manifolds

193

Forger and Sandra Z. Yepes in 2013 (see Theorem 7 in [5]). In both cases the focus is shifted from the local situation near a Lagrangian submanifold to the foliation associated to an involutive Lagrangian distribution and its leaf space, implying an important role for regularity assumptions on the foliation, and for connections on the leaves. We conclude the introduction by summarising the paper’s content. In Sect. 2 we give the basic definitions, as multisymplectic vector spaces and manifolds and their isotropic and Lagrangian subspaces respectively submanifolds. We also give here some examples of isotropic and Lagrangian submanifolds of multisymplectic manifolds. Section 3 introduces the notions of “standard” multisymplectic vector spaces and manifolds, central for this article. We prove the fundamental properties of a standard multisymplectic vector space (V, ω) (with ω a (k+1)-linear form and k > 1), notably the existence of a unique subspace W ⊂ V that is isomorphic to k (V /W )∗ via the natural contraction map (compare Lemma 1 and Proposition 1). On the level of manifolds, we explain why multicotangent bundles are standard multisymplectic manifolds. In Sect. 4 we give a detailed proof of Martin’s normal form theorem (see above), expanding and explaining Martin’s extremely brief original proof. In an Appendix we give complete proofs for the extension of two classical differentialtopological results to foliated manifolds, more precisely, we show a foliated tubular neighborhood theorem and a foliated relative Poincaré lemma (see Theorems 2 and 3).

2 Multisymplectic Vector Spaces and Manifolds, and Lagrangian Submanifolds In this section, we give the basic definitions used in the paper, together with some examples. We will work over the real numbers and all manifolds will be smooth. The algebraic considerations for vector spaces hold true over fields of characteristic zero instead of the reals. Definition 1 Let V be a vector space, k ≥ 1 and ω ∈ k+1 (V ∗ ). We say that (V, ω) is a k-plectic vector space (or simply a multisymplectic vector space) if ω is nondegenerate, in the sense that: ω  : V → k (V ∗ ); v → ιv ω is injective. As in the symplectic case, we can define orthogonal subspaces with respect to ω, but in this setting we have more than just one “ω-orthogonal complement” for a given subspace of V : Definition 2 Let (V, ω) be a k-plectic vector space, U ⊂ V a subspace and 1 ≤ j ≤ k. We define the j-th orthogonal complement of U with respect to ω as follows:

194

G. Sevestre and T. Wurzbacher

U ⊥, j := {v ∈ V | ιv∧u 1 ∧...∧u j ω = 0, ∀u 1 , ..., u j ∈ U }. We say that U ⊂ V is a j-isotropic subspace (respectively, a j-Lagrangian subspace) if U ⊂ U ⊥, j (respectively if U = U ⊥, j ). Going to manifolds we have: Definition 3 Let M be a manifold and ω ∈ k+1 T ∗ M. We say that (M, ω) is a kplectic manifold, or simply a multisymplectic manifold, if the form ω is closed and nondegenerate, in the sense that for all q ∈ M, the map: ωq : Tq M → k (Tq∗ M); vq → ιvq ωq is injective. Analogously to the linear case, we will say that a regular submanifold L is a j-isotropic respectively j-Lagrangian submanifold of M, if, for each p ∈ L, T p L is a j-isotropic respectively j-Lagrangian subspace of T p M. Before studying a special class of multisymplectic manifolds in Sects. 3 and 4, we will give general examples of multisymplectic manifolds and isotropic submanifolds. Note that if N is a submanifold of M of dimension n, then N is j-isotropic for all j ≥ n in a trivial way. Thus in the following examples, we will only consider “interesting” isotropic and Lagrangian submanifolds, where this is not the case. Example 1 Let M be an orientable manifold of dimension m and ω a volume form on M. Then (M, ω) is a (m−1)-plectic manifold. In this case there are no non-trivial examples (in the sense stated above) of isotropic submanifolds of M. Example 2 Let Q be a manifold, k ≥ 1 and the dimension of Q being greater or equal to k+1. Then the manifold M := k (T ∗ Q) is naturally equipped with a k-plectic form. Indeed let θ ∈ k (M) be defined by: θα p (v1 , ..., vk ) := α p (π∗α p (v1 ), ..., π∗α p (vk )), where α p ∈ M, v j ∈ Tα p (M), and π : M → Q is the canonical projection. Then ω := −dθ is a k-plectic form on M. This construction is the generalization of the symplectic form on a cotangent bundle. The zero-section of k (T ∗ M) is a k-Lagrangian manifold, and the fibers of π are 1-Lagrangian. To see this, we can work in local coordinates. A direct computation shows then that if (q i ) are coordinates on an open subset U ⊂ Q and ( p I ) are coordinates on the fibers of k (T ∗ U ), we have: ω|U = −



dpi1 ,...,ik ∧ dq i1 ∧ ... ∧ dq ik .

i 1 ,...,i k

Using this local description it is easy to see that Q is k-Lagrangian and the fibers are 1-Lagrangian. More generally, for α ∈ k (Q), we have that im(α) ⊂ M is a kLagrangian manifold if and only if α is closed. This follows from α∗ θ = α (where, on the left-side, α is regarded as a map α : Q → M), implying α∗ ω = −dα.

Lagrangian Submanifolds of Standard Multisymplectic Manifolds

195

Example 3 Let (M, η) be a k-plectic manifold and ω ∈ k+1 (M × M) the form given by: ω = p1∗ η − p2∗ η, where for i = 1, 2, the map pi is the projection pi : M × M → M on the i-th factor. Then (M × M, ω) is a k-plectic manifold. Considering a diffeomorphism φ : M → M, we claim that φ , the graph of φ, is k-Lagrangian if and only if φ is a symplectomorphism in the sense that φ∗ η = η. Indeed T(q,φ(q)) (M × M) = {(u q , φ∗q (u q )) | u q ∈ Tq M}. Then for (u i , φ∗ (u i )) ∈ T(q,φ(q)) (M × M) (1 ≤ i ≤ k) we obtain: ω(q,φ(q)) ((u 1 , φ∗ (u 1 )), ..., (u k , φ∗ (u k )) = ηq (u 1 , ..., u k ) − ηφ(q) (φ∗ (u 1 ), ..., φ∗ (u k )) = ηq (u 1 , ..., u k ) − (φ∗ η)q (u 1 , ..., u k ), showing the claim. Example 4 Let M be a complex manifold with a holomorphic volume form . Then setting ω = (), the real part of , turns (M, ω) into a multisymplectic manifold. To get a feeling of how Lagrangian submanifolds may look in this case, we consider M = C3 = R6 and  = dz 1 ∧ dz 2 ∧ dz 3 = dz 123 . We find: ω = d x 123 − d x 156 − d x 246 − d x 345 , where we have omitted wedge products and x i are coordinates in R6 . Then the manifold {x 1 = x 2 = x 3 = 0} is 2-Lagrangian, and the manifold {x 2 = x 3 = x 5 = x 6 = 0} is 1-Lagrangian. Example 5 (Compare [3], Sect. 3.) Let M = R6 and ω = d x 145 + d x 246 + d x 356 + d x 456 . Then ω is a 2-plectic form, and L 2 = {x 1 = x 3 = x 4 = x 6 = 0} and L 3 = {x 4 = x 5 = x 6 = 0} are (linear) 1-Lagrangian submanifolds of (M, ω) of different dimensions. (Note that (R6 , ω) is symplectomorphic to the multicotangent bundle 2 (T ∗ R3 ) with the multisymplectic form defined in Example 2 above.) Example 6 Let G be a real semi-simple, compact Lie group. Consider the Cartan form ω ∈ 3 (G), which is the bi-invariant form defined at the neutral element e by: ωe (ξ, η, ζ) := [ξ, η], ζ, where ξ, η, ζ ∈ g (the Lie algebra of G) and < ., . > is the Killing form. The form ω is closed because it is bi-invariant, and it is nondegenerate because the Killing form is nondegenerate and [g, g] = g. Consider T ⊂ G, a torus. Its Lie algebra t is abelian and thus T is 1-isotropic. Thus if T is a maximal torus then it is 1-Lagrangian.

196

G. Sevestre and T. Wurzbacher

3 Standard Multisymplectic Vector Spaces and Manifolds In this section, we will be interested in a special class of multisymplectic vector spaces and manifolds, important in applications of multisymplectic geometry to classical field theories. Definition 4 Let V be a vector space and k > 1. We say that V is a standard k-plectic vector space, if (V, ω) is a k-plectic vector space and there exists a subspace W ⊂ V such that: (1) ∀u, v ∈ W, ιu∧v ω = 0 (2) dim(W ) = dim(k ((V /W )∗ )) (3) codim(W ) > k . Let us also consider the following condition: dim(W ) ≥ codim(W ) .

(3’)

Concentrating on the higher degree cases (k > 1) we then have the following relations between these conditions: Lemma 1 Let (V, ω) be a k-plectic vector space with k > 1. Then: (i) conditions (1) and (2) imply that the map ω  |W : W → k V ∗ induces a linear isomorphism: χ : W → k (V /W )∗ , (ii) if conditions (1) and (2) are satisfied, then condition (3) is equivalent to condition (3 ), (iii) if (V, ω) is standard, then dim(W ) ≥ 2. Remark In reference [4] these multisymplectic vector spaces are called of type (k+1, 0). Proof We denote the projection V → V /W by π. Then the subspace π ∗ (k (V /W )∗ ) ⊂ k V ∗ is given by: π ∗ (k (V /W )∗ ) = {η ∈ k V ∗ | ιv η = 0, ∀v ∈ W }. By condition (1), ω  (w) ∈ π ∗ (k (V /W )∗ ) whenever w ∈ W ; thus ω  induces a unique injective linear map χ : W → k (V /W )∗ such that π ∗ ◦ χ = ω  |W . Moreover, χ is a linear isomorphism by condition (2); thus proving the first assertion. Now put d = dim(W ), c = codim(W ). Then dim(V ) = c +d. Assume conditions (1)–(3) to hold, and d < c. By condition (2), d = kc , thus d = c(c−1)...(c−(k−1)) ≥ c. This contradiction shows that conditions (1)–(3) imply conk(k−1)...1  dition (3 ).

Lagrangian Submanifolds of Standard Multisymplectic Manifolds

197

Now assume conditions (1), (2), (3 ) to hold and c ≤ k. Since c < k is easily seen to contradict (2), then c = k and d = 1. By (3 ), dim(V ) = 2 and thus 1 = dim(k (V /W )∗ ), implying k = 1, contradicting the assumptions. Thus the conditions (1), (2), (3 ) imply conditions (1)–(3), and the second assertion is proven. the fact that Now assume that d ≤ 1. If d = 0, then k V ∗ = {0}, contradicting )−1 and therefore k+1= ω is nondegenerate. Now if d = 1, we have 1 = dim(V k dim(V ), implying c = k; this contradiction proves the last assertion.  If (V, ω) is a standard k-plectic vector space, with k > 1, then the subspace W satisfying Definition 2 is unique as shows the following: Proposition 1 Let (V, ω) be a standard k-plectic vector space, with k > 1, and  two subspaces satisfying Definition 2. Then W = W . W, W  has codimension at most 1 in W  . To do this, assume Proof First we show that W ∩ W  ) > 1. Then, there exists linearly independent vectors the opposite: codim W (W ∩ W  such that span(u, v) ∩ W = {0}; thus we can find η ∈ k (V /W )∗ such u, v of W that ιu∧v η = 0. But, for all w ∈ W , ιw ιu∧v ω = 0, so there cannot exist a w ∈ W such that η = ιw ω, and this contradicts the fact that the map χ is an isomorphism.  . Then there exists a non-zero vector z ∈ W  such that Now suppose W = W k−1  span(z) ∩ W = {0}. For all w ∈ W ∩ W , η ∈  (V /W ): χ∗ (π(z) ∧ η)(w) = (π(z) ∧ η)(χ(w)) = ω(w, z, η) = 0, where χ∗ denotes the dual of the map χ, and π : V → V /W is the canonical projection. The above equation is well-defined because for w ∈ W , ιw ω depends only on its evaluation on element of k (V /W ), because of condition (1) in Definition 2. Denote Z = span(z). The above computation shows that:  ), χ∗ (π(Z ) ∧ k−1 (V /W )) ⊂ ann W ∗ (W ∩ W  ) = {η ∈ W ∗ | η(w) = 0, ∀w ∈ W ∩ W  }. This implies, where ann W ∗ (W ∩ W ∗ k−1  together with codim W (W ∩ W ) ≤ 1, that dim(χ (π(Z ) ∧  (V /W ))) ≤ 1. Furthermore: dim(π(Z ) ∧ k−1 (V /W )) = dim(k−1 (V /W )) > 1, because codim(W ) > k. This shows a contradiction, and thus the Proposition.



The preceding proposition allows to denote such a subspace by Wω and motivates the next definition: Definition 5 Let M be a manifold, k > 1, and ω ∈ k+1 (T ∗ M). We say that (M, ω) is a standard k-plectic manifold if (M, ω) is a k-plectic manifold and if for each p ∈ M, (T p M, ω p ) is a standard k-plectic vector space. For all p ∈ M the unique subspace of T p M satisfying Definition 4 is denoted by Wω ( p) or simply W ( p).

198

G. Sevestre and T. Wurzbacher

The remainder of this section is dedicated to showing that standard multisymplectic vector spaces are in fact symplectomorphic to a canonical k-plectic model that we will describe now. Proposition 2 Let (V, ω) be a standard k-plectic vector space. Then the subspace Wω is 1-Lagrangian. Moreover, there exists a k-Lagrangian vector space L ⊂ V complementary to Wω and the map χ induces (for all choices of such L) an isomorphism: Wω ∼ = k (L ∗ ). Proof Condition (1) in Definition 2 implies that Wω is 1-isotropic. Now if w ∈ Wω⊥,1 but w ∈ / Wω , then we can find η ∈ k (V /Wω )∗ such that ιw η = 0. But, for all u ∈ Wω , ιu ιw ω = 0 and thus there cannot exist a u ∈ Wω such that η = ιu ω. This property contradicts the fact that the map χ is an isomorphism, and therefore proves that Wω is 1-Lagrangian. Now let  L be any subspace complementary to Wω . We may canonically identify V /Wω and  L since the restriction to  L of the projection π : V → V /Wω is an ∼ = isomorphism. Thus we have a canonical isomorphism χ : Wω −→ k ( L ∗ ). We will search for a k-Lagrangian complement of the form L = {v + Av | v ∈  L} for some linear map A :  L → Wω . For L to be k-Lagrangian, it has to verify that L ⊂ L ⊥,k , L, j = 1, ..., k+1: i.e., for all v j ∈  ω(v1 + Av1 , ..., vk+1 + Avk+1 ) = 0. This condition suffices here because assuming that there exists an element u ∈ L ⊥,k \L, we may write u = v + w, with v ∈ L and w ∈ Wω . For u 1 , ..., u k ∈ L we compute then: ω(u, u 1 , ..., u k ) = ω(v, u 1 , ..., u k ) + ω(w, u 1 , ..., u k ) = ω(w, u 1 , ..., u k ) = 0 because v ∈ L ⊂ L ⊥,k , and u ∈ L ⊥,k . Thus we obtain w ∈ Wω ∩ L ⊥,k . But then for all v j = x j + y j , v j ∈ V , x j ∈ L and y j ∈ Wω : ω(w, v1 , ..., vk ) = ω(v, u 1 , ..., u k ) + ω(w, u 1 , ..., u k ) = 0. By the nondegeneracy of ω, we obtain w = 0, thus u = v ∈ L which is a contradiction to u ∈ L ⊥,k \L. Now we return to the construction of the linear map A. We have: ω(v1 + Av1 , ..., vk+1 + Avk+1 ) = ω(v1 , ..., vk+1 )  (−1) j+1 ω(Av j , v1 , ..., vj , ..., vk+1 ). + j=1,...,k+1

Lagrangian Submanifolds of Standard Multisymplectic Manifolds

199

Let  := χ ◦ A, then the Lagrangian condition reads as follows: ω(v1 , ..., vk+1 ) = −



(−1) j+1 (v j )(v1 , ..., vj , ..., vk+1 ).

j=1,...,k+1 1 We denote by T the application T :  L → k ( L ∗ ), v → ιv ω. If  = − k+1 T , then −1 the above condition is verified. Thus the map A := (χ) ◦  has the property that  its graph L =  is a k-Lagrangian space, complementary to Wω .

Definition 6 Let V be a vector space and ωcan the canonical (k+1)-form on the space V := V ⊕ k (V ∗ ) given by: k  ωcan (v1 ⊕ α1 , ..., vk+1 ⊕ αk+1 ) = (−1)i+1 αi (v1 , ..., vi−1 , vi , vi+1 , ..., vk+1 ), i=1

for all v j ∈ V , and α j ∈ k (V ∗ ). Then ωcan is a k-plectic form. We call (V, ωcan ) a canonical k-plectic vector space. Lemma 2 Let (V, ω) be a k-plectic vector space with k > 1. Then (V, ω) is isomorphic to a canonical k-plectic vector space (L ⊕ k (L ∗ ), ωcan ) if and only if (V, ω) is standard. Proof Let (V, ω) be a standard k-plectic vector space, and L a k-Lagrangian subspace complementary to Wω . We define ωcan as above on the space L ⊕ k (L ∗ ). Let γ := id L ⊕ χ : L ⊕ Wω → L ⊕ k (L ∗ ), where we again canonically identified V /Wω and L. Then γ is a linear isomorphism and furthermore we find: ωcan (γ(u 1 ⊕ w1 ), ..., γ(u k+1 ⊕ wk+1 )) = ωcan (u 1 ⊕ ιw1 ω), ..., u k+1 ⊕ ιwk+1 ω))  (−1) j+1 ιw j ω(u 1 , ..., uj , ..., u k+1 ) = j=1...k+1

= ω(u 1 ⊕ w1 , ..., u k+1 ⊕ wk+1 ), i.e. γ ∗ ωcan = ω. Therefore, the k-plectic space (V, ω) is symplectomorphic to (L ⊕ k (L ∗ ), ωcan ). To show the converse, first note that if (L ⊕ k (L ∗ ), ωcan ) is a canonical kplectic vector space, then the space L identified with L × {0} ⊂ L ⊕ k (L ∗ ), is k-Lagrangian, and the space k (L ∗ ), identified with {0} × k (L ∗ ) ⊂ L ⊕ k (L ∗ ) is 1-Lagrangian. Indeed: ωcan ((v1 , 0), ..., (vk+1 , 0)) = 0, and if ωcan ((v, α), (v1 , 0), ..., (vk , 0)) = 0 for all v j ∈ L, then α(v1 , ..., vk ) = 0 and then α = 0. This shows that L is k-Lagrangian. Moreover:

200

G. Sevestre and T. Wurzbacher

ωcan ((0, α), (0, β), (v1 , γ1 ), ..., (vk−2 , γk−2 ) = 0, and if ωcan ((v, α), (0, β), (v1 , γ1 ), ..., (vk−2 , γk−2 ) = 0 for all v j ∈ V and γ j ∈ k (L ∗ ), then ιv β = 0 and thus v = 0. This shows that k (L ∗ ) is 1-Lagrangian. Thus, if a k-plectic linear space (V, ω) is symplectomorphic to a space (L ⊕ k (L ∗ ), ωcan ), then, pulling back the 1-Lagrangian space k (L ∗ ) to V with this symplectomorphism gives a linear subspace W ⊂ V satisfying Definition 2.  Remark Consider now a manifold Q, k > 1 and (k (T ∗ Q), ω  structure exposed in Example 2 of Sect. 2. We have:

k

(T ∗ Q)

) the k-plectic

T (k (T ∗ Q))| Q = T Q ⊕ k (T ∗ Q). Recall that each fiber of k (T ∗ Q) is 1-Lagrangian, and Q is k-Lagrangian. Thus, at k ∗ each point p ∈ Q, the form ω  (T Q) evaluated at the point p is in fact the canonical k ∗ form on the space T p Q ⊕  (T p Q). Using the coordinate expression of the form: ω

k

(T ∗ Q)

|U = −



dpi1 ,...,ik ∧ dq i1 ∧ ... ∧ dq ik ,

i 1 ,...,i k

with U ⊂ Q an open set, (q i ) coordinates on U and ( pi ) coordinates on the fibers of T ∗ U , we see that at any point αq of k (T ∗ Q), the fiber k (Tq∗ Q) is 1-Lagrangian and the tangent space at αq satisfies the conditions of Definition 4. This implies that k ∗ (k (T ∗ Q), ω  (T Q) ) is a standard k-plectic manifold.

4 Normal Forms of Lagrangian Submanifolds of Standard Multisymplectic Manifolds In this section we give a proof of the main result, that first appeared in equivalent form as Lemma 2.1 in Geoffrey Martin’s 1988 article [8]: Theorem 1 Let (M, ω) be a standard k-plectic manifold with k > 1. Let the distribution W ⊂ T M be defined as W := ∪ p∈M Wω ( p) and let L be a k-Lagrangian submanifold of M complementary to W (that is T p M = T p L ⊕ W p for all p in L). If W is integrable, there exist open neighborhoods U and V of L in M and k (T ∗ L), and a diffeomorphism φ : U → V such that: φ| L = id L and φ∗ (ω 

k

(T ∗ L)

) = ω on U.

Proof It follows from the definition of a standard k-plectic manifold and the results found in the linear case, that if L is complementary to W we have an isomorphism of vector bundles: χ : W | L → k (T ∗ L),

Lagrangian Submanifolds of Standard Multisymplectic Manifolds

201

which is given by contraction of vectors in W with ω, using the identification T M| L /W | L ∼ = T L. (Note that we write in the sequel often Wx for the fiber of the vector bundle W → M over x in M.) Using this map we construct the following vector bundle isomorphism:  : T M| L = T L ⊕ W | L ∼ = T L ⊕ k (T ∗ L) = T (k (T ∗ L))| L , acting as the identity on vectors of T L, and transforming vectors of W via χ (i.e.,  = idT L ⊕ χ). Furthermore, this map is for each x ∈ L an isomorphism between k ∗ the multisymplectic vector spaces (Tx M, ωx ) and (Tx (k T ∗ L), ωx T L ). We now wish to find a diffeomorphism f : U1 → U2 , where U1 , U2 are neighborhoods of L respectively in M and in k (T ∗ L), such that f | L = id L and for every x ∈ L, k ∗ Tx f = x . Such a map f then fulfills ( f ∗ ω  (T L) )| L = ω| L . By the foliated tubular neighborhood theorem, we may find a neighborhood U of L in M, a neighborhood V of the zero-section in W | L , and a diffeomorphism φ, which is the identity along L, maps each leaf of the foliation to a fiber of W | L → L, and has as its differential at any point of L the identity. Let f := χ ◦ φ. Then f maps L to the zero section of k (T ∗ L), and is a diffeomorphism onto an open subset U  ⊂ k (T ∗ L) which contains L (as the zero section). Furthermore we have for x ∈ L, Tx f |Wx = χ and Tx f |Tx L = idTx L (where we have identified Tx (Wx ) with k ∗ Wx ). Thus we obtain for x ∈ L, Tx f = x , and upon putting  ω := f ∗ ω  (T L) , we arrive, by the above said, at  ω | L = ω| L . We now want to show that for any vector fields X, Y (defined in U ) and tangent ω = 0. Let p ∈ U and F p be the leaf of the foliation defined by W, which to W , ι X ∧Y  passes through p. This leaf also passes through a point of L, say x. Then φ maps this leaf to the space Wx , and thus f maps F p to k (Tx∗ L). Thus if, X p , Y p are vectors in W p , we may consider that T p f (X p ), T p f (Y p ) are vectors of k (Tx∗ L). Since this k ∗ space is 1-Lagrangian with respect to ω  (T L) , we find ι X ∧Y  ω = 0. Working on an open neighborhood U of L in M, we adapt now the well-known ω − ω) for t ∈ [0, 1]. Moser path method (see [9]) to our situation. Let ωt = ω + t ( ω − ω =: ω  so ω  | L = 0. Thus for x ∈ L, Then we have ωt | L = ω| L and ∂t∂ ωt =  ωt (x) is nondegenerate for all t ∈ [0, 1] and the set of points (t, x) such that ωt (x) is nondegenerate is an open subset of R × M. So, shrinking U if necessary, we may suppose that ωt is nondegenerate in U for all t ∈ [0, 1]. We also have that dωt = dω  = 0. By the relative Poincaré lemma, there exist a neighborhood U of L in M and μ ∈ k (U ) with dμ = ω  and μ| L = 0. Moreover -upon using Theorem 3- we can choose μ such that ιv μ = 0 whenever v ∈ W , because ω  vanishes when contracted with two vectors of W (because both ω and  ω have this property). Therefore μ may be interpreted as a section U → k (T M/W )∗ . Let us now take a look at the map: 

ωt : W → k (T ∗ M), given by contraction of ωt with vectors of W . For u, v ∈ W , lying over the same  point, ιu∧v ωt = 0; so ωt may be seen as a map:

202

G. Sevestre and T. Wurzbacher 

ωt : W → k (T M/W )∗ ; u → ιu ωt , and, by the nondegeneration of ωt , this map is injective, and thus is an isomorphism for dimensional reasons. Then for each t ∈ [0, 1], there exists a unique vector field X t (which take values in W ) such that: ι X t ωt + μ = 0. The association (t, x) → X t (x) thus gives a (time-dependent) vector field tangent to W . But μ| L = 0 so we deduce that for x ∈ L, for all t ∈ [0, 1], X t (x) = 0 by the nondegeneration of ωt . Let φt be the curve of local diffeomorphisms tangent to X t . We have φt | L = id| L . So ∀t ∈ [0, 1] φt | L is defined. But if D(φ) ⊂ [0, 1] × U is the domain of φ, then [0, 1] × L ⊂ D(φ) so, by the openness of D(φ), we may suppose (again shrinking the domain U if necessary), that φt is defined in U for all t ∈ [0, 1]. Now we compute:   ∂ ∂  ∗  φt ωt = φ∗t ωt + L X t ωt ∂t ∂t ∗ = φt (ω  − dμ) = 0. ω = ω, so if φ = g ◦ f (where f is defined above), we obtain Let g := φ1 . Then g ∗ k ∗  φ∗ (ω  (T L) ) = ω, and maintain φ| L = id L , concluding the proof. Acknowledgements We wish to thank Camille Laurent-Gengoux for several useful discussions related to the content of this article.

Appendix: Two Results in Foliated Differential Topology In this appendix we give proofs for two “folkloristic” but subtle (and useful) extensions of well-known results in differential topology. Both are used in [8] but ask for a detailed proof. A brief sketch of a proof of the first result is given on the pages 88–89 in [2]. We begin with the tubular neighborhood theorem, in the presence of a foliation: Theorem 2 (Foliated tubular neighborhood theorem) Let M be a manifold, W ⊂ T M an integrable distribution, and N a submanifold complementary to W in the sense that W | N ⊕ T N = T M| N . Then there exist an open neighborhood U of N in M, and a diffeomorphism φ from U onto an open subset of W | N containing the zero section, such that φ| N = id N , the differential of φ at any point of N is the identity, and φ maps for all p in N the leaf of the foliation defined by W passing through it to the fiber φ(U ) ∩ (W | p ) of W | N → N , intersected with φ(U ). Proof Let g be a fixed (auxiliary) Riemannian metric on the manifold M.

Lagrangian Submanifolds of Standard Multisymplectic Manifolds

203

Given q ∈ M, the leaf Wq of the foliation W defined by the distribution W and containing q is given as an injectively immersed submanifold jq : Fq → M (with image jq (Fq ) = Wq ). The induced Riemannian metric jq∗ (g) defines an exponential map ex p W , notably one has ex pqW : Tq (Fq ) → Fq , defined on an open neighborhood of 0. Since Tq (Fq ) is canonically identified with Wq = Tq (Wq ) via the differential of jq , and jq is smooth, ex pqW is a smooth map from an open neighborhood of 0 in Wq to M, having values in Wq . Restricting q to be an element of N we obtain a map ex p W,N from a subset of W | N containing N to M. Let us now show that ex p W,N is, in fact, smooth on an open neighborhood of N ϕ → V1 × V2 ⊂ Rm−d × Rd , such in W | N . Fix q in N and a coordinate chart M ⊃ U − that the fibers of π : V1 × V2 → V1 are the leaves of the foliation W (d is the rank of this foliation). Furthermore, we can assume that ϕ(q) = 0 and denote the elements of Rm−d resp. Rd by x resp. z. We denote ϕ(U ∩ N ) by N and T ϕ(W ) by W if no ambiguities are possible. By the assumption T M| N = T N ⊕ W | N we have ∀q ∈ N ⊂ V1 × V2 that Rm = Tq (V1 × V2 ) = Tq N ⊕ Wq = Tq N ⊕ Rd and thus the natural projection πq : Tq N → Rm−d is a linear isomorphism. Thus π| N : N → V1 has everywhere maximal rank equal to the dimension of V1 . Shrinking V1 and V2 if necessary, we can assume that π| N : N → V1 is a diffeomorphism whose inverse is described by (idV1 , f ) : V1 → V1 × V2 , where f : V1 → V2 is smooth and N =  f , the graph of f . The map ψ given by ψ(x, z) = (x, z − f (x)) =: (x, y) is a diffeomorphism of V1 × V2 to an open subset of Rm . Restricting ψ to an appropriate open neighborhood of 0, the image of ψ is a product of open subsets of Rm−d and Rd . Furthermore, ψ(0) = 0, ψ preserves the leaves of W, and maps N =  f to {y = 0}. Post-composing ϕ with ψ yields a chart of M near q compatible with the foliation W and “adapted” to N . Obviously, we can construct a locally finite covering of N by open subsets of M that are domains of such charts, again denoted by ϕ : U → V1 × V2 for simplicity. W,N is given as the time-one value of the solution of the In these coordinates ex p(x,0) following differential equation: dy i dy j d 2 yk  k = 0 for 1 ≤ k ≤ d, + i, j (x, y) 2 dt dt dt i, j subject to the initial condition that x ∈ V1 , y(0) = 0 and dy (0) ∈ W(x,0) . Standard dt results on smooth ordinary differential equations depending smoothly on parameters  ⊂ W | N containing N , where ex p W,N is imply that there exists an open subset O uniquely defined and smooth.  = Tq N ⊕ Wq = Tq M, we obtain that Upon identifying, for q ∈ N , Tq O D(ex pqW,N ) = idTq M . By the below cited Proposition 3, it follows that there exists an  ⊂ W | N such that ex p W,N | O is a diffeomorphism open neighborhood O of N in O onto its image U , an open neighborhood of N in M. Calling its inverse φ, this latter map fulfills the conditions stated in Theorem 2. 

204

G. Sevestre and T. Wurzbacher

The last argument relies on a standard result in differential topology (cf., e.g., Proposition 7.3 in [6]): Proposition 3 Let Y and Y  be two manifolds, and X ⊂ Y , X  ⊂ Y  two regular submanifolds. Let f : Y → Y  be a smooth map satisfying: • f | X : X → X  is a diffeomorphism • Tx f : Tx Y → T f (x) Y  is an isomorphism for all x ∈ X Then there exists an open neighborhood V of X in Y such that f (V ) is open in Y  , and f |V is a diffeomorphism. Now we show the relative Poincaré lemma, again in the presence of a foliation: Theorem 3 (Foliated relative Poincaré lemma) Let M be a smooth manifold and N ⊂M a submanifold. Let ω be a closed (k+1)-form on M which vanishes when pulled back to N . Then there exists a neighborhood U of N in M, and a k-form μ defined on U , such that dμ = ω|U and μ| N = 0. Moreover, if there exists an integrable distribution W ⊂ T M complementary to N , and such that ιu∧v ω = 0 whenever x is in M and u and v are in Wx ⊂ Tx M, we may choose μ such that ι X μ = 0, for all vector fields X taking value in W and defined on an open subset of U . Proof By the (standard) tubular neighborhood theorem, there exist U and V neighborhoods of N in M respectively E (where E → N can be chosen to be any vector bundle such that E ⊕ T N = T M| N ), and a diffeomorphism φ : U → V fixing N pointwise. Thus in what follows, we can and will assume to be in a open neighborhood U of N in M, which is also a vector bundle π : E = U → N over N . Let us consider the map: H : [0, 1] × U → U , (t, x) → t · x = t x. If we denote Ht (x) := H (t, x) then H0 = ι ◦ π (where ι : N → U = E is the inclusion of N as the zero-section of E), and H1 = id E = idU . Let Yt (x) := dtd Ht (x). The smooth map Yt is not a vector field since H is not a flow, but the following formula still holds: d (H ∗ ω) = d(Ht∗ ιYt ω) + Ht∗ ιYt dω, (*) dt t where, for α a (k+1)-form, Ht∗ ιYt α is the following (well-defined!) k-form: (Ht∗ ιYt α)x (v1 , ..., vk ) = αt x (Yt (x), Tx Ht (v1 ), ..., Tx Ht (vk )), for x ∈ U and v j ∈ Tx U . For a proof of (∗) see, e.g., [7]. Since ι∗ ω = 0 and ω is closed we obtain:

Lagrangian Submanifolds of Standard Multisymplectic Manifolds

205

ω|U = H1∗ ω − H0∗ ω   d ∗ Ht ω dt = [0,1] dt = (d(Ht∗ ιYt ω))dt [0,1] =d (Ht∗ ιYt ω)dt [0,1]

= dμ,

where we set μ := [0,1] (Ht∗ ιYt ω)dt. Moreover μ| N = 0 because Yt | N = 0. To prove the last part of the theorem, we apply Theorem 2 in order to choose a foliated tubular neighborhood U of N with respect to W . We can thus assume that the fibers of U → N are the fibers of W | N → N . Then for x ∈ U , Yt (x) ∈ Wt x , implying for X a vector field tangent to W : (ι X Ht∗ ιYt ω)x (v1 , ..., vk−1 ) = (Ht∗ ιYt ω)x (X (x), v1 , ..., vk−1 ) = ωt x (Yt (x), Tx Ht (X (x)), Tx Ht (v1 ), ..., Tx Ht (vk−1 )) =0 since Yt (x) and Tx Ht (X (x)) are both in Wt x .



References 1. Arnold, V.: Mathematical Methods of Classical Mechanics. Graduate Texts in Mathematics, vol. 60. Springer, Berlin (1989) 2. Candel, A., Conlon, L.: Foliations I. Graduate Texts in Mathematics, vol. 23. American Mathematical Society (2000) 3. Cantrijn, F., Ibort, A., de Léon, M.: On the geometry of multisymplectic manifolds. J. Aust. Math. Soc. Ser. A. Pure Math. Stat. 66(3), 303–330 (1999) 4. de Léon, M., de Diego, D.M., Santamaría-Merino, A.: Tulczyjew’s triples and lagrangian submanifolds in classical field theories. In: Applied Differential Geometry and Mechanics. Universiteit Gent, Ghent, Academia Press (2003) 5. Forger, Michael, Yepes, Sandra Z.: Lagrangian distributions and connections in multisymplectic and polysymplectic geometry. Differ. Geom. Appl. 31, 775–807 (2013) 6. Golubitsky, M., Guillemin, V.: Stable Mappings and Their Singularities. Graduate Texts in Mathematics. Springer, New York (1973) 7. Guillemin, V., Sternberg, S.: Geometric Asymptotics. American Mathematical Society (1977) 8. Martin, Geoffrey: A Darboux theorem for multi-symplectic manifolds. Lett. Math. Phys. 16(2), 133–138 (1988) 9. Moser, Jürgen: On the volume elements on a manifold. Trans. Am. Math. Soc. 120, 286–294 (1965) 10. Ryvkin, Leonid, Wurzbacher, Tilmann: An invitation to multisymplectic geometry. J. Geom. Phys. 142, 9–36 (2019) 11. Weinstein, Alan: Symplectic manifolds and their Lagrangian submanifolds. Adv. Math. 6, 329–346 (1971)

The Poisson Characteristic Variety of Unitary Irreducible Representations of Exponential Lie Groups Ali Baklouti, Sami Dhieb and Dominique Manchon

Abstract We recall the notion of Poisson characteristic variety of a unitary irreducible representation of an exponential solvable Lie group, and conjecture that it coincides with the Zariski closure of the associated coadjoint orbit. We prove this conjecture in some particular situations, including the nilpotent case. Keywords Poisson characteristic variety · Representations · Coadjoint orbits · Exponential groups · Solvable Lie algebras MSC Classification (2000) 22E27 · 81S10

1 Introduction We introduced in [1], for any exponential solvable Lie algebra g and any coadjoint orbit  ⊂ g∗ , a topological module (πν , M) over the formal enveloping algebra: ν (gC ) = T (g)[[ν]]/x ⊗ y − y ⊗ x − ν[x, y], U identified to a deformed algebra A = (S(gC )[[ν]], ∗) via a C[[ν]]-module isomorphism (for example, the symmetrisation map or the Duflo isomorphism). This module is obtained by considering the differential of the associated irreducible unitary representation ρ of the group G = exp g. This representation is constructed by inducing a A. Baklouti (B) · S. Dhieb Faculté des Sciences de Sfax, Sfax, Tunisia e-mail: [email protected] S. Dhieb e-mail: [email protected] D. Manchon LMBP, CNRS, Université Clermont-Auvergne, 3 place Vasarély CS 60026, 63178 Aubière, France e-mail: [email protected] © Springer Nature Switzerland AG 2019 A. Baklouti and T. Nomura (eds.), Geometric and Harmonic Analysis on Homogeneous Spaces, Springer Proceedings in Mathematics & Statistics 290, https://doi.org/10.1007/978-3-030-26562-5_9

207

208

A. Baklouti et al.

unitary character of a polarization h of ξ ∈ , i.e. a maximal isotropic Lie subalgebra of g with respect to the bilinear form B = , [−, −]. Any polarization obviously contains the radical g() of the bilinear form B . The polarization h must satisfy Pukanszky’s condition: Ad∗ (exp h)ξ = ξ + h⊥ . Namely this construction gives rise for any real number  to a unitary representation ρ of the group G  = exp g , which is irreducible for  = 0. Here g is the Lie algebra obtained from g by multiplying the Lie bracket by . Differentiating each of these representations and setting ν = i we get the requested topological module by considering ν as an indeterminate [1, Sect. 3] : πν = iρ−iν . Let A = S(gC ) = A/νA. Then the topological module (πν , M) on A induces an A-module structure on M = M/νM. Let us denote by π0 the associated representation of A. We are interested in the annihilator I of π0 , as well as in the ideal J of A defined by: J = Ann πν /ν Ann πν . We now define the characteristic variety V (πν ) and the Poisson characteristic variety VA(πν ). The first (resp. the second) is the set of common zeroes of the elements of I (resp. J ). The inclusion J ⊂ I yields the reverse inclusion V (πν ) ⊂ VA(πν ). According to the unitarity, we can consider these two affine varieties as defined on the field of real numbers. We have proved in [1, Théorème 5.3.1], for any exponential solvable Lie group, that: V (πν ) = ξ + h⊥ . The present paper is devoted to the study of the Poisson characteristic variety. More precisely we state the following conjecture: Conjecture 1 Let G be an exponential solvable Lie group, with Lie algebra g, and let π be an irreducible unitary representation of G, associated to a coadjoint orbit (Ad∗ G) =  via the Kirillov orbit method. Then the Poisson characteristic variety VA(πν ) coincides with the Zariski closure in g∗ of the orbit . We have proved this result in the nilpotent case [1, Théorème 5.4.1], using the explicit description of the annihilator of the representation ρ due to Godfrey (cf. [5]) and Pedersen (cf. [8]). In this case, the orbit is Zariski-closed, and then VA(πν ) = . The method we used there relies on specific generators of the ideal J , which are, at the same time, via symmetrisation, generators of the annihilator of ρ. We propose here a direct method which does not call for specific generators. We prove Conjecture 1 in several other situations: in the case g = [g, g] + g(), the proof is similar to the one in the nilpotent case. The case when the chosen Pukanszky polarisation is normal is also possible to handle. We also treat several interesting low-dimensional examples. We have not yet been able to prove the conjecture in the general case.

The Poisson Characteristic Variety of Unitary Irreducible …

209

2 First Approach to Conjecture 1 We keep the notations of the introduction. Let d be the dimension of the coadjoint orbit (Ad∗ G). We choose a coexponential basis (X 1 , . . . , X d/2 ) of the polarization h in g. The Hilbert space of the representation ρ is thus identified to L 2 (Rd/2 ) via the diffeomorphism  : Rd/2 −→ G  /H t = (t1 , . . . , td/2 ) −→ exp t1 X 1 · · · exp td/2 X d/2 H. Recall from [1] that I is the ideal of those polynomials Q 0 such that πν (Q 0 ) = O(ν), whereas J is the ideal of the polynomials Q 0 such that there exists a sequence (Q j ) j≥1 of polynomials such that: Q = Q 0 + ν Q 1 + ν 2 Q 2 + · · · + ν k Q k + · · · ∈ Ker πν . In other words, I is the set of polynomials Q 0 such that πν (Q 0 ) is “small” (i.e. vanishes for ν = 0), and J is the set of polynomials Q 0 which can be deformed into an element of the annihilator. One clearly has the inclusion I ⊂ J . Conjecture 1 can be reformulated as follows: Conjecture 2 Any polynomial vanishing on the orbit can be deformed into an element of the annihilator of the associated topological module πν . Now let Q 0 ∈ S(g) such that Q 0 vanishes on the orbit  = (Ad∗ G). Recall that we look for Q 1 , Q 2 , . . . ∈ S(g) such that πν (Q 0 + ν Q 1 + ν 2 Q 2 + · · · ) = 0. We have Q 0 ∈ Ker π0 , hence πν (Q 0 ) = O(ν). Moreover ad X · Q 0 vanishes on  for any X ∈ g, hence πν (ad X · Q 0 ) = O(ν), i.e. [πν (X ), πν (Q 0 )] = νπν ([X, Q 0 ]) = O(ν 2 ). Applying this to X = X 1 , . . . , X d/2 and using πν (X j ) = −∂ j + O(ν) [1, Lemme d 5.1.4] we end up with the fact that π˙ 0 (Q 0 ) = dν | πν (Q 0 ) is a partial differential ν=0

operator with constant coefficients. Using the same lemma from [1] again, there exists an element Q 1 ∈ S(g) such that π˙ 0 (Q 0 ) = −π0 (Q 1 ). Hence we get: πν (Q 0 + ν Q 1 ) = O(ν 2 ),

(1)

thus making a first step towards deforming Q 0 as we would like. Unfortunately the next steps are much more harder, and we have been able to carry out the process only in the nilpotent case. Lemma 3 Let G be an exponential solvable Lie group with lie algebra g. Let Q 0 ∈ J . Then, the r -th derivative π0(r ) (Q 0 ) is a partial differential operator with polynomial coefficients of degree at most r − 1.

210

A. Baklouti et al.

Proof The result has been just proved for r = 1. We now detail the case r = 2. Let j, k ∈ {1, . . . , d/2}. Starting from the fact that ad X j ad X k Q 0 vanishes on  we get:   πν (X j ), [πν (X k ), πν (Q 0 )] = O(ν 3 ).

(2)

Vanishing of the coefficient of ν 2 in (2) yields: 

   π0 (X j ), [π0 (X k ), π¨ 0 (Q 0 )] + π˙ 0 (X j ), [π0 (X k ), π˙ 0 (Q 0 )]   + π0 (X j ), [π˙ 0 (X k ), π˙ 0 (Q 0 )] = 0,

hence: 

     − ∂ j , [−∂k , π¨ 0 (Q 0 )] + π˙ 0 (X j ), [−∂k , π˙ 0 (Q 0 )] + − ∂ j , [π˙ 0 (X k ), π˙ 0 (Q 0 )] = 0.

(3) The second term of the right-hand side vanishes because π˙ 0 (Q 0 ) is a constant coefficient partial differential operator. The last term also vanishes because of the following computation: 

     − ∂ j , [π˙ 0 (X k ), π˙ 0 (Q 0 )] = [−∂ j , π˙ 0 (X k )], π˙ 0 (Q 0 ) + π˙ 0 (X k ), [−∂ j , π˙ 0 (Q 0 )] .

The second term of this sum vanishes as π˙ 0 (Q 0 ) is a constant coefficient partial differential operator. The first term also vanishes because π˙ 0 (X k ) is a partial differential operator with coefficients of degree at most one: this comes from the formula d

2 

d

2 ∂a X,u (νt) ∂ ν a X,u (νt) + a X,0 (νt) + πν (X k ) = ∂t 2 ∂tu u u=1 u=1

(4)

where a X,0 . . . , a X,d/2 are analytic functions on Rd/2 (cf. [9]). Thus the first term of the sum (3) vanishes, which is equivalent to the fact that π¨ 0 (Q 0 ) is a differential operator with affine coefficients. The case r ≥ 3 is treated similarly, by induction on r . From   πν (X j1 ), [· · · [πν (X jr ), πν (Q 0 )] · · · ] ,

(5)

the vanishing of the coefficient of ν r in (5) implies that   ∂ j1 , [· · · [∂ jr , π0(r ) (Q 0 )] · · · ] = O(ν r +1 ) is a finite sum of Lie brackets involving only the operators π0(u) (Q 0 ), u = 1, . . . , r − 1 and π0(s) (X j ), s = 1, . . . , r − 1, j = 1, . . . , d/2. The operators of the first family are partial differential operators with coefficients of degree at most u − 1 by induction hypothesis, those of the second family are partial differential operators with coefficients of degree at most s by (4). More precisely we have:

The Poisson Characteristic Variety of Unitary Irreducible …

  (−1)r ∂ j1 , [· · · [∂ jr , π0(r ) (Q 0 )] · · · ]   (s1 )  =− π0 (X 1 ), [· · · [π0(sr ) (X r ), π0(u) (Q 0 )] · · · ] .

211

(6)

u+s1 +···+sr =r, s j ≥0, 0